Skip to content
Markdown

ADR 0053 — Governance catalogs: push quality results, pull column classifications

  • Status: Proposed
  • Date: 2026-10-25
  • Deciders: @TheurgicDuke771
  • Related: ADR 0031 (no SDK whose licence we cannot ship), 0034 (asset identity by the OpenLineage naming spec; OpenLineage emission with quality facets; the LineageProvider seam this mirrors), 0039 (where the catalog credential lives), 0041 (audited configuration), 0049 (the classification table and the additive-only rule this extends), 0052 (contracts, a later payload). CONTRIBUTING rule 45 (open standards first).

Context

Data consumers look for data in a governance catalog — DataHub, OpenMetadata, Collibra — not in DataQ. Two things are missing between them. First, a check's verdict never reaches the catalog, so the person choosing a table cannot see that its checks are failing. Second, the catalog often holds the organisation's authoritative column classifications (a PII tag, a glossary term), and DataQ's redaction ladder cannot read them: it reads warehouse tags (G3), the suite policy and its own profiler detections (ADR 0049), and nothing else.

DataQ already emits OpenLineage run events with data-quality facets (ADR 0034, dark by default) and already identifies every asset by the OpenLineage dataset-naming spec. Lineage is pulled through a LineageProvider seam with Marquez as the reference. The catalog work fits beside those, not inside the datasource or orchestration seams: a catalog is neither a place checks run nor a pipeline that triggers them.

The OpenMetadata Python SDK is published by Collate under terms we have not been able to clear for redistribution (ADR 0034 recorded it as licence-blocked), so any OpenMetadata support speaks its REST API directly. DataHub's REST and GraphQL APIs are Apache-2.0 like the server. Collibra's APIs are proprietary and there is no instance we can test against.

Decision

1. One CatalogProvider seam, configured as a workspace integration

A third integration seam beside LineageProvider and the alert publishers: one Admin-configured catalog per workspace (Admin → Integrations), its credential write-only in the SecretStore, its base URL under the credential-destination re-supply rule (moving the URL needs the credential again), every change audited. Each provider speaks the catalog's HTTP API with httpx — no vendor SDK. The seam has two capabilities a provider may implement independently: publish_results and fetch_classifications.

2. Push: one assertion per check, one result per run, off the critical path

After a run reaches a terminal state, a Celery task (not the run itself) publishes, for every check that produced a result, an assertion on the catalog dataset that matches the suite's asset and a run result for it: pass / fail (warn and critical map to the provider's nearest), the run time, the check's name, kind and dimension, metric_value, and a link to the run in DataQ. Never observed_value, failing samples or anything the redaction ladder could mask: the catalog is a wider audience than the suite. Delivery retries with backoff; a failure is recorded on a catalog health signal (the workspace_health pattern, alerting on the failing edge) and never fails or delays the run. Assertion identity is stable per check (dataq:<check_id>), so a re-run updates rather than duplicates.

Dataset matching uses the asset's OpenLineage identity, translated per provider (DataHub URN platform + name + environment; OpenMetadata FQN), with an optional per-connection override for the catalog's platform-instance name. An asset with no match is skipped and counted on the health signal, not guessed.

3. Pull: catalog classifications as a new, additive-only source

A daily task (and the Admin's "refresh now") reads, for each monitored asset, the column-level tags and glossary terms the catalog holds, and writes sensitive verdicts into asset_column_classifications with source = 'catalog' — the table ADR 0049 built for exactly this. The mapping is a fixed, documented convention, never a per-workspace mapping (the G3 argument: a configurable mapping is an unreviewed security control where one typo un-masks a column): a tag or term whose name, case-folded, is pii, sensitive, confidential, restricted, or a hierarchical name ending in one of those (Classification.PII, PII.Sensitive), is sensitive.

It is additive-only, like the profiler: a catalog can mask a column DataQ would have shown; it can never un-mask one (a catalog public tag is ignored in phase 1), it never satisfies fail-closed clearance, and the suite's identifier_column does not clear it. On the ladder it sits above the profiler and below the warehouse's own tags; its redaction reason is catalog_tag. A classification that disappears from the catalog is removed on the next pull (unlike a profiler detection, which needs an Admin dismissal: the catalog is the authority for its own tags).

4. Providers, in order

  1. DataHub — the reference. Push through GraphQL upsertCustomAssertion + reportAssertionResult (the external-assertion path that works on open-source DataHub, not only DataHub Cloud); pull through the dataset's schema fields' tags and glossary terms. Verified against a local DataHub quickstart (Apache-2.0) in the harness.
  2. OpenMetadata — REST only: a test case per check (/api/v1/dataQuality/testCases), results by PUT …/testCaseResult; pull from /api/v1/tables/name/{fqn}?fields=columns,tags. Verified against a local OpenMetadata server.
  3. Collibra — not built until a user names it: proprietary API, no test instance, so it would ship unverified. The seam makes it a provider module when the demand comes.

5. What stays out

Pushing data contracts (ADR 0052) to the catalog, pulling lineage or ownership from it, writing DataQ's own classifications back to it, and any per-workspace tag mapping. Each is a later, separate decision.

Consequences

  • One migration: the source value-set on asset_column_classifications widens by catalog, and a catalog_settings singleton (provider, base URL, secret pointer, enabled flags, last push/pull status). Additive.
  • The redaction ladder gains a rung and a reason (catalog_tag); every surface that shows redaction_reasons names it, and the pinned rung tests gain a row.
  • A new outbound data flow (results to the catalog) is disclosed in the sub-processor list and GET /admin/deployment, default off.
  • Two providers mean two local verification targets in the harness, both heavy (Kafka/OpenSearch for DataHub, Elasticsearch/MySQL for OpenMetadata); they run in a verification window, not in CI. CI tests the providers against recorded HTTP exchanges.

Alternatives considered

  • Rely on the OpenLineage data-quality facets DataQ already emits. The standard-first answer (rule 45), and it may suffice for push where a catalog ingests those facets as assertions. To be checked first in the build: if DataHub's OpenLineage ingestion turns them into assertions, push becomes configuration of the existing emitter and §2 shrinks to the health signal. It does not help pull, which has no standard.
  • Vendor SDKs. DataHub's is usable, but OpenMetadata's is licence-blocked, and one seam over plain HTTP keeps both providers the same shape.
  • Catalog classifications that can also clear a column. Rejected for phase 1: a catalog is edited by more people than a warehouse's tags, and an un-masking source needs the review a configurable mapping would bypass.
  • Per-workspace tag mapping. Rejected for the G3 reason above.