Dataplex) unlocked per connection¶

Status: Accepted (direction + seam shape; Snowflake DMF is the first native build, DQX/Dataplex trigger-gated — see §6)
Date: 2026-07-17
Deciders: @TheurgicDuke771
Amends: ADR-0003 (its v1.1 engine-swap shape only — the suite-level engine: gx | dqx toggle moves to the check grain and engines become connection-anchored; 0003's core decision, GX-only for v1, stands)
Related: 0003 (GX-only v1; the suite-level gx | dqx toggle this ADR supersedes), 0011 (CheckRunner/ConnectionAdapter seams), 0012 (check.kind + metric_value — the axis §4 keeps orthogonal), 0030 (registry dispatch shape), 0031 (license guardrail gating DQX), post-v1 roadmap gap G-g (GX-pin engine risk — this ADR is its abstraction answer).

Context¶

v1 is deliberately GX-only (ADR 0003): one result schema, one catalog, one editor. But the platforms DataQ monitors now ship their own DQ primitives — Snowflake Data Metric Functions (system + custom DMFs, Enterprise Edition), Databricks Labs DQX (rule engine over DataFrames/streams; the DLT/streaming cases batch-only GX can't serve), and GCP Dataplex data-quality scans (CloudDQ successor). Users on those platforms reasonably want DataQ to use the native capability rather than re-implement it — both for pushdown (checks run where the data lives, instead of pulling into the worker's pandas) and for meeting warehouse standards teams where they already are.

Two integration patterns must not be conflated:

Synchronous evaluators — DataQ calls the engine at run time and gets outcomes back (GX today; DQX; ad-hoc DMF invocation via SELECT SNOWFLAKE.CORE.NULL_COUNT(...); Dataplex on-demand scans). This ADR.
Warehouse-scheduled measurement ingest — the platform runs checks on its own cadence (DMF schedules attached to tables, recurring Dataplex scans) and DataQ provisions and ingests. Structurally a pull provider (like orchestration polling / lineage pull), not a CheckRunner. Deferred to a future ADR; its open schema question is that results.run_id is NOT NULL and a warehouse-scheduled measurement has no DataQ run (synthetic runs vs a sibling measurements table). Snowflake-scheduled DMFs are out of scope for DataQ (decided 2026-09-29): DataQ evaluates DMFs ad hoc and does not attach, schedule or ingest Snowflake-managed DMF runs. ROW_COUNT, ACCEPTED_VALUES and SCHEMA_CHANGE_COUNT, which exist only in the attached model, stay unsupported on the DMF engine; the GX engine covers row count, accepted values and schema drift on Snowflake.

ADR 0003 sketched DQX as a suite-level engine: gx | dqx toggle on UC suites. That shape predates this design and is superseded here (§2).

Decision¶

1. Engines are capabilities of connections¶

There is no global engine setting and no engine config page. Every datasource connection offers gx; native engines are unlocked by connection type and validated per connection instance:

Engine	Unlocked by connection type	Exists today?
`gx`	all datasource types	yes (sole engine)
`dmf`	`snowflake`	first native build (§6)
`dqx`	`unity_catalog`	trigger-gated (§6)
`dataplex`	`bigquery` (future type)	gated on BigQuery-as-datasource (§6)

No Snowflake connection → no DMF anywhere in the product. This extends the dispatch shape that already exists: registry.build_check_runner keys on connection.type; it becomes two-key (connection.type, engine) with a capability map answering "which engines does this connection offer."

2. Engine selection is per check, not per suite¶

check.engine (TEXT, default 'gx', additive migration), validated on save against the suite's connection capability set. Rationale: the realistic suite is mostly GX expectations plus a few native checks — a suite-level toggle would force artificial suite splits. The run path already partitions one run's checks across evaluators (expectation checks → CheckRunner, monitor kinds → MonitorRunner); partitioning by engine is the same move, inside one run with one result set. Supersedes ADR 0003's suite-level toggle sketch (CLAUDE.md §5 updated in this PR).

3. Type gates the offer; the connection instance validates the reality¶

Native availability is not implied by type alone (DMFs need Enterprise Edition + EXECUTE DATA METRIC FUNCTION / SNOWFLAKE.DATA_METRIC_USER; DQX needs workspace job/serverless execution rights). The connection test/re-auth flow probes actual engine availability and stores a per-engine capability flag on the connection, with an actionable remediation when unavailable — what the limitation is and what access to request (e.g. "requires Enterprise Edition", "grant the DataQ role EXECUTE DATA METRIC FUNCTION + usage on the target database"). The stored reason is classified guidance, never raw exception text (raw driver errors have carried credentials before — a standing lesson). Phasing: phase 1 may gate by type only, with run-time failures landing as classified error results; the capability flag + probe is phase 2, but the flag is designed into the schema from the start.

4. `kind` ⊥ `engine` — engines are alternate evaluators of existing kinds¶

check.kind says what is measured; check.engine says who evaluates it. Snowflake's FRESHNESS system DMF is an alternate evaluator of the existing freshness kind — never a new kind (no dmf_freshness). (Amended 2026-08-22, slice-2 build: ROW_COUNT was named here as the volume evaluator, but it cannot be invoked ad hoc — Snowflake documents "You can't call this function directly", live-confirmed in every argument spelling — so volume stays GX-only; its COUNT(*) pushes down identically. FRESHNESS additionally accepts only DATE/TIMESTAMP_LTZ/TIMESTAMP_TZ bare-column arguments — a TIMESTAMP_NTZ column gets classified run-time guidance pointing at the GX freshness monitor.) This keeps the ADR 0012 seam, the scorecard's dimension mapping, and trend queries engine-agnostic. Each engine carries a supported matrix (which kinds/check types it can evaluate) — per-check information, reinforcing §2. Consequence: the expectation catalog (frontend-only today, by design) must become an engine-aware backend catalog; that is one story with the dimension-classification work, not two catalogs.

DMF outcomes are metric-shaped and ride the existing result semantics unchanged: scalar → results.metric_value, thresholds applied by the run service outside the runner (as today), severity tiers unchanged.

5. Lifecycle: the check's engine is tied to the connection — validated, never silent¶

Save-time: creating/updating a check with an engine the suite's connection doesn't offer is a 422 naming the missing capability.
Run-time: an engine that was available at authoring but isn't now (privilege revoked, edition downgrade, connection re-pointed) lands the check as the existing error operational status with a classified reason — never a silent skip, never counted in pass rates (error is already excluded from severity denominators).
Export/import: a suite export carrying native-engine checks imports anywhere, but the user is warned (toast + per-check report) that N checks require platform-native capabilities the target connection may not offer; the same save-time validation marks them explicitly rather than dropping or silently converting them.
Connection delete already cascades suites/checks (ADR 0020) — no new orphan class.

6. Build order and triggers¶

Engine	Gate / trigger	Notes
DMF (Snowflake)	Build first	Ad-hoc `SELECT` invocation of system DMFs (`NULL_COUNT`, `NULL_PERCENT`, `DUPLICATE_COUNT`, `UNIQUE_COUNT`, `FRESHNESS`, `BLANK_COUNT`, `FUTURE_TIMESTAMP_PERCENT`) mapped onto existing kinds; custom DMFs shipped 2026-09-29 (amendment below). `ROW_COUNT` was on this list and is out — it has no ad-hoc form (2026-08-22 amendment in §4). `ACCEPTED_VALUES` and `SCHEMA_CHANGE_COUNT` are out for the same reason (Snowflake documents both as "can't call this function directly"; live-confirmed 2026-09-27: a direct call rejects `ACCEPTED_VALUES`'s lambda argument as a syntax error, and without it — or `SCHEMA_CHANGE_COUNT` with no argument — returns the sentinel `-1`, never a measurement. They need the attached, scheduled DMF model — warehouse-side attachment management, decided out of scope on 2026-09-29). Ad-hoc DMF arguments must be a bare column of a table or view: an expression, a `VALUES` list or a CTE is refused (`Invalid argument types` / `only supports table-like objects`). Constraint: needs a live Snowflake with Enterprise features to build against.
DQX (Unity Catalog)	A real Databricks/streaming user, plus two prereqs	(a) a remote-execution design — DQX's value is Spark-side evaluation, so DataQ must submit work to the workspace, a new architectural capability (today's UC runner pulls into pandas); (b) the Databricks Labs license check against ADR 0031's no-source-available guardrail before it stays on any roadmap.
Dataplex (BigQuery)	A BigQuery-as-datasource decision	BigQuery isn't a DataQ datasource; that decision dominates the cost and comes first.
Scheduled-native ingest (DMF schedules / Dataplex scans)	Separate future ADR	Pull-provider shape; must first settle synthetic-runs vs sibling-measurements-table for `results.run_id`.

Consequences¶

Additive migration: check.engine default 'gx'; per-engine capability flags on connections (phase 2 probe fills them). Backward-compatible; no existing row changes meaning.
registry gains the two-key dispatch + capability map; run dispatch partitions a run's checks by engine.
Backend engine-aware catalog lands with the dimension-classification work as one story; check editor filters check types by the selected engine's supported matrix.
The scorecard and all trend/aggregation SQL stay engine-agnostic — guaranteed by §4.
G-g (GX-pin risk) is discharged from "watch item" to "decided abstraction": GX becomes one engine behind the seam rather than the definition of a check.

Alternatives considered¶

Suite-level engine toggle (ADR 0003's sketch). Rejected: the realistic suite is mostly GX expectations plus a few native checks — a suite-level engine forces artificial suite splits, and the per-engine supported matrix (§4) is per-check information anyway. The run path already partitions one run's checks across evaluators, so the check grain costs nothing structurally.
Native metrics as a monitor kind (kind='native_metric') instead of an engine. Seriously considered (2026-07-17 discussion): it would ship ad-hoc DMF calls on the existing ADR 0012 seam with no engine machinery at all. Rejected because it dead-ends — it cannot represent DQX/Dataplex (full evaluators, not metrics), blocks future warehouse-side attachment management, and violates §4 by encoding the evaluator into the kind axis.
Engine-specific kinds (dmf_freshness, dqx_null_check, …). Rejected outright: forks the kind axis, so every consumer — scorecard, dimension mapping, trend SQL — would branch per engine forever.
A global/workspace engine setting. Rejected: engine availability is a property of a connection instance (edition, grants, workspace rights), not of the workspace. A global toggle cannot answer "is DMF available on this connection" and would reintroduce the config-page indirection §1 deliberately avoids.

Amendment (2026-09-28): DQX shipped, executed remotely¶

DQX is now the second native engine, offered on unity_catalog connections. The user chose to build it ahead of retiring the Databricks environment. Both §6 prerequisites are settled:

(b) The licence check failed for bundling. databricks-labs-dqx is published under the proprietary Databricks License, which permits use only with Databricks services. It is therefore never a DataQ dependency (ADR 0031, CONTRIBUTING rule 40). DataQ neither imports, installs nor redistributes it.
(a) Remote execution is the design. For each run, DataQ uploads a fixed notebook to the connection user's workspace folder and submits one serverless job (Jobs API runs/submit) carrying every DQX check of the run. The notebook installs a pinned DQX version, validates each rule (an invalid rule errors only itself), applies all valid rules in one Spark pass, and returns failing-row counts per rule, never row values. The run path gained an optional runner hook, run_native_checks, so an engine with per-job start-up cost is batched. DMF stays per check.

A security property this engine forced: DQX evaluates bare string arguments as Spark SQL (check_funcs.get_limit_expr → F.expr), live-confirmed when the list value 'SM CASE' resolved as a column named SM. DataQ therefore builds every rule from a closed vocabulary of eight types, with typed arguments: - columns are allowlisted identifiers, backtick-quoted - list values are emitted as escaped SQL literals - limits are numbers or ISO dates validated by DataQ

A user string never reaches DQX unquoted. The authoring gate and the run path share one rule builder, so they cannot disagree.

Capability probe: phase 1, gated by connection type only. A missing jobs or workspace permission lands as a classified per-check error at run time, as §2 permits. A probe would itself cost a job start-up.

Amendment (2026-09-29): custom DMFs shipped¶

§6's "custom DMFs later" is built. A Snowflake check of type dmf:custom runs a data metric function the customer created (CREATE DATA METRIC FUNCTION), on the existing dmf engine.

Shape. config is {"function": "DATABASE.SCHEMA.FUNCTION", "columns": [...]}. The name must be fully qualified. Each part goes through the shared SQL-identifier allowlist and is quoted by the usual folding rule (all-lower-case stays bare, anything else is quoted). The SNOWFLAKE database is refused, because the system DMFs have their own types and some return a sentinel -1 when called this way.
Invocation. The system functions' ad-hoc form, with 1 to 20 bare columns of the suite target, in the function's signature order: SELECT <db>.<schema>.<dmf>(SELECT <col>[, …] FROM <target>). The ad-hoc rule stands: no expressions, no second table. Authoring and the run path share one statement builder, so a config that saves is exactly one that runs.
Result. The return value is metric_value, banded by the check's thresholds (higher = worse, a positive fail or critical threshold required, like the other banded DMF types). It is shown unmasked, like every DMF metric. That is safe only because the function returns a number the customer chose to compute, so the user guide tells authors to return a count or a percentage, never a data value. The dimension is never derived (ADR 0038): NULL unless the author sets one.
Discovery. The connection-test probe runs SHOW DATA METRIC FUNCTIONS IN ACCOUNT and stores the non-SNOWFLAKE results in engine_capabilities.dmf.custom_functions (capped at 200) for the editor to suggest. SHOW was chosen over INFORMATION_SCHEMA.FUNCTIONS because it needs no warehouse and spans every database the role can see, not just the connection's. The list doubles as the USAGE check the issue asked for: Snowflake lists a custom DMF only to a role holding USAGE on it.
Errors. A DMF the role cannot use raises the same Unknown user-defined function as one that does not exist, so both get one message naming the grants needed. A column list that doesn't match the signature, an unknown column and a missing schema each get their own fixed message; driver text never reaches the result.

Live-verified 2026-09-29 on the Enterprise account with the existing least-privileged connection role, against a throwaway schema: - the probe listed exactly the custom DMFs the role held USAGE on; - a suite run through the worker path and a dry-run evaluated a one-column and a two-column DMF with the right values and warn / fail / critical bands; - an unknown DMF, an ungranted DMF, a wrong column count and an unknown column each landed as a classified per-check error; - injection-shaped names and columns were refused at authoring, and the run path refuses them without issuing any SQL.

Scheduled (attached) DMFs stay out of scope, as recorded in §6 and the Context.

Amendment (2026-09-29): a stream mode for DQX¶

Streaming is why DQX sits beside batch-only GX. A DQX check now has an optional mode: snapshot (the default, as before) or stream.

How a check targets a stream. The target is unchanged: the suite's catalog.schema.table. That can be a Lakeflow streaming table or any Delta table that grows by appends. stream mode evaluates only the rows appended since the check's last persisted result.

Where evaluation happens. Evaluation stays in the same one serverless job per run. The notebook reads the table with readStream and an availableNow trigger, applies the same DQX rules, and sums failing rows per rule from the stream's observed metrics. A noop sink means nothing is written. The stream stops once it has caught up. Stream checks that resume from the same version share one stream.

What DataQ persists. Failing-row counts only, as in batch mode. Each result also records a small stream report: the Delta version range it read, the rows it evaluated, the table's Delta id, and the number of rewrite commits it skipped. The resume point is that recorded version, held by DataQ and not by a Spark checkpoint. The next run starts from the latest evaluated result's next_version. A job whose result is never saved is therefore re-read, not skipped. A persistent checkpoint would already have advanced past rows whose counts were lost. The checkpoint Spark requires is a throwaway folder in a UC volume named on the connection (dqx_checkpoint_volume). It is deleted after each run. Serverless rejects both temporary and DBFS checkpoint locations, which was confirmed live.

It runs on the existing run and schedule path. There is no new ingest subsystem, table, migration or beat task. The batch hook gains a previous argument, the latest evaluated observed_value per check, read from results.

Invariants kept. DQX is still installed pinned in the workspace only. Rules still come only from the closed vocabulary, through the same builder, and mode never reaches DQX. The job still returns only counts: its collect() calls read grouped counts, table history, or a limit(0) pre-flight, never rows.

Honesty rules. - A run with no new rows is skip, not pass. - A stream report DataQ cannot trust makes that check error, so it never becomes a resume point. - A replaced table (its Delta id changed) or an expired history restarts the stream from the whole table, and the result says restarted. - Rewrite commits (UPDATE/DELETE/MERGE/overwrite) are skipped with skipChangeCommits and counted in change_commits_skipped, since the mode is defined over appends.

A platform fact that shaped the notebook: a streaming query that fails fails the whole Databricks command, even when the Python exception is caught, and even with dbutils.notebook.exit in the handler (live). So predictable stream failures are prevented up front instead of caught: - a resume version past the latest commit starts no stream - each rule is pre-flighted on an empty read - rewrite commits are skipped rather than raised

A stream failure that still occurs errors every DQX check in that run.

Rejected alternatives. - Reading a Lakeflow pipeline's expectation metrics from its event log. It covers only pipelines the user already runs and only the expectations defined inside them. It needs pipeline-id configuration and a second evaluator vocabulary, and it cannot evaluate DataQ's own rules. - Provisioning a DQX step inside the user's pipeline. That is pipeline management: DataQ would edit user infrastructure, which the orchestration model (monitor and trigger only) rules out. - A persistent checkpoint per check. It is simpler to state, but it loses counts on a failed persist, as described above.

Both rejected designs are heavier, and neither evaluates a stream with DataQ's rules.

Live-verified 2026-09-29 (Databricks Free Edition, serverless). The runs went through execute_run, the UC runner and the real job: - an append-only Delta table: a full first read, an increment-only second read, a no-new-rows skip, and an UPDATE skipped and counted - a Lakeflow streaming table (CREATE STREAMING TABLE … AS SELECT * FROM STREAM …): a first read, then only the rows a REFRESH appended