ADR 0038 — DQ-dimension classification on checks: seven canonical dimensions, derived default, stored and overridable¶

Status: Accepted (§5 amended 2026-07-19 — existing checks ARE backfilled; see the amendment in §5)
Date: 2026-07-19
Deciders: @TheurgicDuke771
Related: 0005 (severity tiers + the health-score formula the scorecard reuses), 0012 (check.kind — the axis this one is orthogonal to), 0015 (comparison kind), 0019 (custom SQL — the one authoring path with no derivable dimension), 0036 (§4 names this work; see Relationship to ADR 0036 below), 0037 (the workspace-true aggregate rule the scorecard must obey).

Context¶

A check today carries two classification axes, and neither is a data-quality dimension:

Field	Axis	Example
`kind` (ADR 0012)	how the monitor works	`expectation` / `freshness` / `comparison`
`expectation_type`	the specific GX rule	`expect_column_values_to_be_unique`

A DQ dimension is a third, orthogonal axis: the semantic category of quality a check measures. It is what makes "we have 47 checks" legible as "your Timeliness is uncovered" — and it is the input the planned asset scorecard aggregates by.

The value is not primarily the score. It is coverage: "this asset has zero Timeliness and zero Uniqueness checks" is immediately actionable in a way a pass-rate never is, and CLAUDE.md §5 already records that most real incidents are freshness/volume — i.e. exactly the dimensions teams are least likely to have covered.

Unlike the ADR-0005 severity columns or ADR-0012's kind/metric_value, a dimension column is pure widening — nullable, additive, no data rewrite, no two-step deploy — so there was never schema-timing pressure to rush it. That is why this work was deliberately held back from the Week-3 migration and gets a deliberate design pass now.

Decision¶

1. Seven canonical dimensions, closed vocabulary¶

accuracy, completeness, consistency, integrity, timeliness, uniqueness, validity.

The canonical six (completeness, uniqueness, validity, accuracy, consistency, timeliness) plus integrity — referential/relational correctness, which the six fold into "consistency" and which DataQ can express distinctly once cross-dataset comparison exists (ADR 0015).

Stored as VARCHAR(32) with a table CHECK over the vocabulary, matching the CHECK_KINDS idiom (models._in_check) — not a Postgres ENUM. An enum type needs ALTER TYPE to extend, which is exactly the migration friction this column was deferred to avoid; a CHECK constraint is a one-line ALTER and stays visible in the model file next to the other vocabularies.

Closed, not free-form. A free-text dimension makes the scorecard's coverage view meaningless the first time someone types "Timeliness " with a trailing space — you cannot report "you have no Timeliness checks" against a set you don't control. The cost is that adding an eighth dimension is a migration; that is the right trade for a reporting axis.

2. Derived default, stored value, user-overridable at any time¶

Three properties, and all three are load-bearing:

Derived — on create, if the check's type maps to a dimension, that is the default. Nobody should have to hand-classify expect_column_values_to_not_be_null as Completeness.
Stored — the resolved value is written to checks.dimension, not recomputed on read. This is what lets the scorecard aggregate with a SQL GROUP BY instead of joining through a Python map, and what makes an override survive.
Overridable at any time — at create and by later PATCH. Derivation is a good guess about intent, not a fact: the same expect_column_values_to_be_between is Validity when it bounds a percentage and Accuracy when it asserts a reconciled total.

Rejected: pure derivation (no column) — no override, and the scorecard loses SQL aggregation. Rejected: pure user-set — every existing check and most new ones land unclassified, so the coverage view is empty until someone backfills by hand, which nobody does.

3. Derivation is deliberately partial, and `NULL` is a real state¶

Check type	Dimension	Why
`expect_column_values_to_not_be_null`	`completeness`
`expect_table_row_count_to_be_between`	`completeness`	"is all the data here"
`monitor:volume`	`completeness`	a short load is missing data
`expect_column_values_to_be_unique`	`uniqueness`
`expect_column_values_to_be_between`	`validity`	conforms to a rule
`expect_column_values_to_be_in_set`	`validity`
`expect_column_values_to_match_regex`	`validity`
`expect_column_value_lengths_to_be_between`	`validity`
`expect_column_values_to_be_of_type`	`validity`
`monitor:freshness`	`timeliness`
`monitor:schema_drift`	`consistency`	structural stability over time
`comparison:records` / `comparison:columns`	`consistency`	cross-dataset agreement (ADR 0015)
custom SQL (`unexpected_rows_expectation`)	none	arbitrary predicate — unknowable

accuracy and integrity are never derived. No generic GX expectation can tell you whether data matches reality, or whether a relationship holds. They exist for the author to select — most often on a custom-SQL check, which is precisely the path with no derivable answer. Pretending to derive them would populate the scorecard with confident nonsense, which is worse than an honest gap.

So dimension is nullable, and NULL means "unclassified" — a state the scorecard must render as a gap, never silently bucket. Existing checks keep NULL (see §5).

4. Orthogonal to `kind` and to `engine`¶

kind is how the monitor works, dimension is what quality aspect it measures, engine (ADR 0036) is what evaluates it. All three vary independently: a freshness kind is Timeliness whether GX or DMF runs it; a comparison kind is Consistency by default but Accuracy when the baseline is a source of truth.

The derivation map is therefore keyed on expectation_type with a kind fallback, not on kind alone — expectation covers a dozen different dimensions, so keying on kind would collapse them.

5. Backward compatibility¶

Additive nullable column on checks and on check_versions (history must round-trip the field, or restoring an old version silently reclassifies a check).

~~Existing rows are left NULL, not backfilled.~~ (Superseded — see the amendment below.) The original reasoning: a backfill would be indistinguishable from a deliberate user classification, so a later correction to the derivation map could never tell "the map said so" from "a human said so".

Amendment (2026-07-19): existing checks ARE backfilled¶

Migration a7b8c9d0e1f2 fills dimension for every pre-existing check the map can classify.

The original position over-weighted its own concern and under-weighted the cost:

The ambiguity it feared is vacuous at the moment it matters. The column is introduced one revision earlier, so when the backfill runs the set of checks carrying a human-set dimension is exactly empty. The backfill cannot overwrite anyone's decision, because nobody has made one yet. The original text reasoned as if derived and user-set values were being mixed, when in fact the user-set set is empty by construction.
The cost of not doing it is the feature. Leaving them NULL makes the planned scorecard report "unclassified" for every existing check — useless on day one, on precisely the workspaces with the most history worth reporting on. "Classified on next edit" assumed people re-save old checks, which they do not.

The residual risk is narrower than §5 claimed: only that a future correction to the derivation map cannot distinguish a backfilled row from a user's override. Mitigation is recorded in the migration — re-derive only rows whose stored value still equals that migration's map output, which it inlines for exactly this purpose.

The backfill deliberately writes no check_versions rows: it is a system classification, not an edit, and minting a version per check would fill every history drawer with a change nobody made. Unmapped types (custom SQL) stay NULL — §3's "unclassified is a real state" is unchanged.

6. Surfaces¶

Check editor — a dimension select, pre-filled with the derived default, always editable. The derived default is shown as the selected value rather than as a placeholder, so what you see is what gets stored.
Export/import — dimension rides the check document, and the two cases are distinguished: absent (an older export) → derived, exactly as if freshly authored; present, including an explicit null → taken verbatim, so an unclassified check stays unclassified. Conflating them would re-create §5's forbidden backfill by another door — export always emits the key, so every pre-migration check would be silently classified on the way through. The export version does not bump: adding an optional field is backward-compatible in both directions, and EXPORT_VERSION is documented to bump only on an incompatible shape change.
Changing a check's expectation type re-derives the dimension in the editor (the create page already resets the whole form on a type change). Keeping the old value would leave the select showing a classification the help text simultaneously claims is "defaulted from the check type" — and it would then be sent explicitly, so a uniqueness check filed as completeness would look like a deliberate override forever. The backend does not re-derive on PATCH: it cannot tell a derived value from an override, so leaving it is the conservative choice, and the editor sends the corrected value.
Scorecard — consumes this; not built here.

Relationship to ADR 0036¶

ADR 0036 §4 asserts the dimension classification and an engine-aware backend expectation catalog are "one story… not two catalogs". This ADR narrows that: it lands the dimension axis and a backend derivation map — the seed of that catalog — without moving the frontend catalog server-side.

The reason is sequencing, not disagreement. The backend catalog's purpose is engine-awareness (which expectations a DMF or DQX connection offers), and no native engine exists yet — ADR 0036 §6 makes DMF the first build and trigger-gates the rest. Building a server catalog now would mean guessing at an interface with exactly one implementation, while the scorecard is blocked on nothing more than a column. When DMF lands, the derivation map is the thing it extends.

Consequences¶

Coverage reporting becomes possible at all, which is the scorecard's actual value.
One more field on the authoring form. Mitigated by the derived default: the common path is "don't touch it".
The derivation map is a judgement call that will be argued with (row_count → Completeness rather than a Volume dimension; schema_drift → Consistency rather than Validity). That is why it is overridable, and why the map lives in one module rather than being scattered.
Adding an eighth dimension is a migration. Accepted in §1.
Two axes on the wire look similar (kind and dimension) and will be confused. The editor labels them distinctly and this ADR is the reference.

Alternatives considered¶

Free-text dimension — rejected in §1; unbounded values destroy coverage reporting.
Postgres ENUM — rejected in §1; ALTER TYPE friction for no gain over a CHECK.
Derive at read time, no column — rejected in §2; no override, and the scorecard loses SQL aggregation.
Backfill existing checks from the map — rejected in §5; makes a derived guess indistinguishable from a human decision.
Multiple dimensions per check (a check as both Validity and Accuracy) — rejected. A many-to-many turns "% of Completeness checks passing" into a question with no single answer, and every scorecard number would need a double-counting rule. One dimension per check; split the check if it genuinely measures two things.