Skip to content
Markdown

Architecture Decision Records (ADRs)

Each ADR captures a single significant architecture decision: the context, the decision, the consequences, and the alternatives considered. New ADRs are append-only — supersede an old decision by adding a new ADR and marking the old one's status as Superseded by ADR-NNNN.

Format

  • File name: NNNN-short-kebab-slug.md (zero-padded 4-digit sequence)
  • Frontmatter fields:
  • Status — one of Proposed, Accepted, Deprecated, Superseded by ADR-NNNN (title-case)
  • Date — YYYY-MM-DD
  • Deciders — who made the call
  • Consulted (optional) — stakeholders whose sign-off the decision needed (e.g. product owner for ADR 0005). Omit when none.
  • Supersedes (optional) — ADR-NNNN this decision replaces. Omit when none.
  • Superseded by (optional) — ADR-NNNN that later replaced this one. Add when the status flips to Superseded by.
  • Amends (optional) — ADR-NNNN this decision partially overrides (whole-ADR replacement uses Supersedes instead). Pair it with an inline > **Amendment (date, ADR-NNNN):** … blockquote at the top of the amended ADR and an "(amended by NNNN — …)" note on its index Status, so the override is visible where readers actually look. Precedents: ADR 0028 (amends 0024), ADR 0012's amendment blockquote, ADR 0031 (amends 0013).
  • Sections: Context, Decision, Consequences, Alternatives considered, Related (optional)
  • Keep each ADR short — 1–2 pages. If it grows past that, the decision is probably two decisions.

Template

# ADR NNNN — <title>

- **Status:** Proposed
- **Date:** YYYY-MM-DD
- **Deciders:** @handle
- **Consulted:** <stakeholder>   <!-- optional; omit when none -->
- **Supersedes:** ADR-NNNN       <!-- optional; omit when none -->
- **Superseded by:** ADR-NNNN    <!-- optional; add when superseded -->

## Context

## Decision

## Consequences

## Alternatives considered

Index

# Title Status
0001 Trunk-based branching with squash-merge into main Accepted
0002 Conventional commits for PR titles and commit messages Accepted
0003 GX-only for v1; DQX deferred to v1.1 for DLT/streaming Accepted (amended by 0036 — suite-level engine: gx \| dqx toggle → check-level, connection-anchored)
0004 Unified OrchestrationProvider abstraction for ADF and Airflow Accepted
0005 Severity tier weights (warn / fail / critical → health score) Accepted
0006 ADF webhook authentication (shared secret in URL + hard-cutover rotation) Accepted
0007 Airflow callback model (HMAC-signed webhook + polling fallback) Accepted
0008 FastMCP server at /mcp — Azure AD token validated (same token as REST); everything exposed as tools rather than resources; thin wrappers reusing the service layer + per-suite authz; fail-closed without auth. Amended: 8 → 19 tools (Tier 1 reads), then 19 → 30 tools (Tier 2 edit-gated mutations + live-probe tools), then 30 → 33 → 38 (Tier 3A coherence), then 38 → 42 (Tier 3B batch 1 — assets + incidents reads) → 46 (Tier 3B batch 2 — incident lifecycle, list_columns, near-misses; Tier 3B complete) → 47 (get_pipeline_status, get_adf_pipeline_status kept as a deprecated alias) → 48 (get_doc, curated docs pages; 25 read-only / 18 state-changing / 5 live-probe) Accepted
0009 Repo layout — flat monorepo (backend/ + frontend/) Accepted
0010 Provider-agnostic infrastructure seams (Azure is the default, not the architecture) Accepted
0011 Extensibility seams for deferred connectors and integrations Accepted
0012 Monitor-kind seam (check.kind discriminator + numeric metric storage) Accepted
0013 Marketplace distribution (customer-deployed BYOL) and anti-vendor-lock-in guardrails Accepted (amended by 0031 — §5 licensing line + licensed-revenue framing)
0014 Cross-dataset reconciliation as a comparison check kind (reuse FastAPI_DataComparison engine) Accepted
0015 Two-connection comparison check model — suite stays single-connection and supplies the target under test; a comparison check adds one source ref (checks.source_connection_id FK) + suite-target-shaped config.source; execution via a new DatasetReader seam + the ported FDC engine, row-cap fail-fast; report files derived on-demand, never stored; no connection→connection generalization (0030 Option B stays deferred). Amended 2026-09-16: comparison sources do not support sampling — coherent key-set sampling is a different mechanism from the suite-target's positional sampling and was not built; COMPARISON_MAX_ROWS fail-fast + query-filter narrowing are the sanctioned alternatives Accepted (built 2026-07-12)
0016 Severity derivation semantics (band the unexpected-%, thresholds override GX success) Accepted
0017 Upgrade Python runtime 3.11 → 3.13 (3.14 deferred — GX-capped); bundled with the Snowflake 3→4 CVE refresh Accepted
0018 Results surface is an in-app page (suite-scoped authz + PII redaction); Grafana deferred to optional ops add-on Accepted
0019 Custom-SQL checks ride kind='expectation' via GX UnexpectedRowsExpectation (no new kind); read-only validation + SQL-datasource gating Accepted (amended 2026-08-08 — UC custom SQL needs a GX Databricks-SQL batch, not the pandas one; it had never worked before)
0020 History/audit: per-entity Type-4 snapshot tables (check_versions, connection_versions) where config history is needed; no SCD-2; credentials never snapshotted; cascade-delete accepted; cross-entity audit log deferred Accepted (amended by 0041 — the deferred audit log is now accepted; soft-delete re-deferred, SCD-2 closed)
0021 Live test/demo-data environment (retail model, 3 reference flows) lives outside the repo — IaC stack/mock-data/Databricks notebook not git-tracked; discharges the deferred live-warehouse/file smoke Accepted (amended 2026-07-27 — harness IaC CLI is OpenTofu, converted with the app stack; see ADR 0024)
0022 Week-6 prototype adoption — full 13-screen set as dedicated pages (Share is the only drawer; prototype wins on conflicts; Settings/Admin pulled into W6); chart library = recharts (lazy-loaded) Accepted
0023 Container image registry — GitHub Container Registry (GHCR) over ACR / Docker Hub (vendor-neutral, public package → anonymous ACA pull, reuses GITHUB_TOKEN); supersedes the earlier ACR scaffolding Accepted
0024 App deploy infra — in-repo IaC stack; shares subscription/RG/ACA-env/Postgres with the harness; ACA api+worker+migrate-job + SWA-Standard linked same-origin /api + self-hosted Redis + KV (UAMI) + App Insights + AAD-app-reg OIDC Accepted (amended 2026-07-27 — OpenTofu replaces Terraform as the IaC CLI, in-place amendment; ADR 0031 coherence, state/providers/path unchanged)
0025 Production image — multi-stage python:3.13-slim + pip (not conda; ~2.84GB→~1GB); conda stays the local-dev tool; amends the W1 conda lock Accepted
0026 DataQ-issued API keys (PATs) as a second authenticator behind the get_current_user seam — REST + /mcp identically; phase 1 = user-scoped PATs (dq_live_…, sha256-at-rest, show-once, uniform 401, mandatory expiry, owner-cascade); service-account principals = phase 2 (deferred); HTTP Basic rejected Accepted (phase 1 built 2026-07-04; amended by 0032 — email-identity slice of the phase-2 principal question)
0027 Suite permissions — workspace-admin is implicit admin on every suite (governance/break-glass); drop grantable suite-admin; normal users get owner/edit/view; workspace-admin gets workspace-wide visibility (supersedes earlier, narrower workspace-admin decisions) Accepted (amended by 0033 — admin source → stored users.role; Viewer share-cap)
0028 Cloud-neutral image — one multi-arch frontend image, nothing baked; auth config injected at runtime (window.__DATAQ_CONFIG__ via nginx envsubst) behind a generic DATAQ_AUTH_* contract; bypass fail-closed (explicit DATAQ_AUTH_MODE=bypass only); replace MSAL with a generic OIDC client validated against Azure; frontend SWA→Container App (amends 0024); AWS/GCP IaC post-v1 Accepted (amended by 0032 — otp mode + cookie session credential)
0029 dbt as a third OrchestrationProvider (mirrors the Airflow callback model 0007) — HMAC webhook + artifacts poll of run_results.json (adls/s3/file); binds to dbt's universal surface (no host API); job-level grain; migration widens the connection-type/provider/dedup value-sets Accepted
0030 Apache Iceberg — engine-level read (Snowflake/UC iceberg tables) is free & zero-code; the only new build is a native pyiceberg read (v2 baseline, v3 deferred) behind a thin IcebergCheckRunner (scan → DataFrame → gx_runner); new self-contained iceberg connection type (Option A: own catalog + storage credential — independent lifecycle, cascade-safe), Option B two-connection ref deferred to 0015; reads Delta UniForm too; native impl deferred Accepted (native read shipped; introspection built)
0031 Distribution licensing — free open-source (MIT) + customer-deployed BYOL; no entitlement/license-key (amends 0013: supersedes its §5 licensing-model line + licensed-revenue framing); marketplace listings are free offers of the OSS artifacts; THIRD-PARTY-NOTICES/SBOM in images + releases; standing no-strong-copyleft dependency guardrail (CONTRIBUTING rule 40) Accepted
0032 Email OTP sign-in — passwordless third authenticator behind get_current_user (dq_sess_ cookie sessions, PAT-style sha256-at-rest); auth-mode ladder bypass · otp · oidc, fail-closed startup; mandatory signup allowlist (no open registration); one user row per normalized email (aad_object_id → nullable, two-step); separate AUTH_EMAIL_* mailer + SMTP pre-flight; hard prereq = the rate-limiting auth slice; amends 0026 (email-identity slice) + 0028 (mode enum + cookie credential) Accepted (amended by 0033 — signup default role + widened trust statement)
0033 Workspace roles — Admin / Member / Viewer as stored users.role on the two-axis model (role × per-suite ladder, ladder untouched); connection mutations Admin-only (closes the workspace-global hole, breaking for Members); Viewer capped at view; WORKSPACE_ADMIN_EMAILS demotes to bootstrap/break-glass; in-app role management + last-admin guard; amends 0027 (admin source) + 0032 (signup default role) Accepted
0034 Assets & lineage (gap G-d design) — first-class asset entity keyed by the OpenLineage dataset-naming spec (namespace+name adopted verbatim, dbt-ol/Spark byte-compatible; DEV/QA = distinct assets, UI groups over env); lineage emitted/pulled, never built — OL emission w/ quality facets → dbt manifest.json → lineage_edges cache → LineageProvider seam (Marquez reference; OpenMetadata SDK license-blocked per 0031); incidents anchor to assets (one open per asset+check, evidence-card payload); asset/incident authz derived from suite grants Accepted
0035 Request rate limiting — app-level innermost HTTP middleware (portable per 0028/0013, covers the /mcp mount a dependency can't, principal-aware where nginx isn't); fixed-window Redis counter (window index in the key → no EXPIRE race), no new dep (INCR over slowapi/limits); fail-open; endpoint classes webhook(per-IP) · default(per-sha256(token)) · unauth(per-IP), raw token never keyed/logged; /healthz+OPTIONS exempt; headers on 429 only; nginx limit_req out of scope; hard prereq for 0032's OTP Accepted
0036 Connection-anchored check engines — engines are capabilities of connections (GX universal; dmf⇔snowflake, dqx⇔unity_catalog, dataplex⇔future bigquery); check.engine per check (supersedes 0003's suite-level gx \| dqx toggle); type gates the offer, connection probe validates reality (classified remediation, never raw errors); kind ⊥ engine (engines = alternate evaluators of existing kinds; engine-aware backend catalog lands with the dimension-classification work); lifecycle validated save/run/import-time (error status, never silent); DMF first build, DQX/Dataplex trigger-gated; scheduled-native ingest deferred to a future ADR Accepted (amends 0003; DMF first native build; DQX/Dataplex trigger-gated)
0037 Workspace-visible asset identity — asset identity + lineage topology (incl. column pairs) visible to every member; aggregate verdicts (health rollup, future scorecard) workspace-true (all composing suites, one number for all viewers); grant boundary moves to suite-derived detail (composing-suite names/runs/results/samples/incidents — ADR 0027 ladder unchanged, suite-grain 404-no-leak intact); asset detail 200s for all, restricted_suite_count replaces redaction; supersedes an earlier identity-redaction regime (amends 0034 decision 5) Accepted
0038 DQ-dimension classification on checks — seven canonical dimensions (accuracy/completeness/consistency/integrity/timeliness/uniqueness/validity) as a closed vocabulary via table CHECK, not a Postgres ENUM (no ALTER TYPE friction) and not free text (unbounded values destroy coverage reporting); derived default → stored → overridable at any time (stored so the scorecard aggregates in SQL and an override survives); derivation is deliberately partial — accuracy/integrity are never derived and custom SQL has no answer, so NULL means unclassified and the scorecard must render it as a gap; existing rows ARE backfilled (migration a7b8c9d0e1f2; §5 amended 2026-07-19 — the backfill cannot overwrite a human decision because none existed yet when it ran, while leaving them NULL made the scorecard useless on day one; unmapped types stay NULL); dimension ⊥ kind ⊥ engine; narrows 0036 §4 — lands the derivation map as the backend catalog's seed without moving the frontend catalog server-side, since no native engine exists yet Accepted
0039 Self-hosted secret backend — a fourth SecretStore speaking the KV v2 HTTP API, not a vendor SDK (one mode serves OpenBao / Vault Community / Enterprise / HCP; no HashiCorp lock-in traded for the Azure one); OpenBao (MPL-2.0, LF fork of pre-BUSL Vault 1.14) is the shipped/tested/pinned server — Vault Community is BUSL-1.1 and rule-40-forbidden as a distributed component, supported as a target; SECRET_STORE=openbao replaces redis and RedisSecretStore is deleted (plaintext credentials were the default of the published eval stack), with a loud one-cycle RuntimeError shim, Redis stays for broker + rate-limit counters; token auth phase 1 (AppRole deferred/filed), dev-mode token generated into the gitignored .env; Azure Key Vault remains the prod default (managed identity beats a held token), OpenBao is the BYOL/cloud-neutral option; an outage is never reportable as a state — new SecretStoreUnavailableError, deliberately NOT a SecretNotFoundError subclass, since every caller branches on the TYPE and none reads the message (a sealed vault otherwise renders "not set" on the admin page, skips alert delivery silently, and runs connections with the credential omitted); no status is silent (400=KV v1 mount, 401=gateway, 429=HCP throttle); AzureKeyVaultStore had the same defect since Week 2 and is fixed too (the same invisible-degradation shape seen before); delete purges via KV v2 metadata delete, fail-soft Accepted
0040 Warehouse inventory sync + the table-enumeration seam — one per-datasource enumerate_tables capability (Snowflake/UC information_schema, engine-native case = the identity-safe path, connection-database scoping, ephemera/system schemas excluded) feeding three consumers: inventory sync (opt-in per connection, daily crontab beat, upsert_assets(preserve_provenance=True) so last_seen advancement keeps live tables out of the orphan sweep and dropped tables age out through it — no new column, no new guard; no "discovered" badge, ADR 0037's neutral unmonitored rendering IS the marker), GET_LINEAGE seeds, and future pickers; ASSET_INVENTORY_MAX_TABLES cap (default 2000) with loud truncation; decided: S3 namespace stays s3://{bucket} for AWS, becomes s3://{host[:port]}/{bucket} only when endpoint_url is set Accepted
0041 History & audit posture for v1.x+ (amends 0020) — one append-only audit_events table (actor/action/entity/before/after/ts) recording deliberate acts by a principal only, with entity_id deliberately FK-less so the row outlives the entity it describes (an FK would either cascade the audit away or block the delete); the data-read audit (gap G1) is phase 2 on the same table, discriminated by action_class — not a second table — with the opposite latency contract (phase 1 writes inside the mutation's transaction, fail-closed; phase-2 reads must stay off the critical path); (a) preserving check_versions past delete rejected — RESTRICT makes checks undeletable, SET NULL orphans rows needing self-contained identity (a worse audit log scoped to one entity), tombstone = soft-delete for one table — the audit log's delete event answers it instead and cascade stands; (b) soft-delete re-deferred on a narrowed argument (its headline motivation is already met — an earlier 409 guard means a connection can no longer be deleted out from under its runs; the column is the easy part vs. "deleted means inert" for live credentials/bindings/schedules; and a delete-that-doesn't-delete fights the G2 Art-17 erasure work), mitigated by a follow-up blast-radius confirmation; (d) SCD-2 closed permanently, not deferred; Type-4 tables stay the product surface (version drawer/restore), the audit log is the durable record; payloads are per-entity allow-lists reusing the snapshot builders, never dict(row), never a secret value, never sample_failures/observed_value, and reuse only the credential subset of _PII_KEYS (the full set redacts name/display_name/user_id — the very content of a rename and the point of the actor record); append-only via REVOKE UPDATE, DELETE — a guard against accidental in-app mutation, explicitly NOT tamper-resistance (dataq_app owns the table and can re-grant; splitting the role is rejected because the single-role model is the whole mitigation for the unpatched Postgres RI owner-cast escalation, so real tamper-evidence needs the G1 external anchor); coverage guarded off FastAPI's route table, not an audit registry (the ADR-0039 tautology) Accepted (phase-1 build; follow-ups filed)
0042 LLMProvider seam — outbound LLM for authoring assistance (NL→SQL generation, check suggestions, future RCA narrative), admin-configured, BYO credential, default-OFF; two wire impls only (anthropic official SDK + openai_compatible httpx — the latter covers Azure OpenAI, Bedrock chat-completions, and local Ollama/vLLM/TGI; no model server in the image per 0025); structured output is a native \| prompt_json capability ladder and never a trust boundary — callers re-validate, generated SQL rides the 0019 validator + dry-run, suggestions the server-side allowlist; config = llm_settings DB singleton behind require_workspace_admin, API key write-only in SecretStore with the credential-destination re-supply rule on base_url; execution worker-side through llm_invocations (poll + audit + token/cost record; G4 posture reads it); prompt context = schema + masked aggregate profiler stats only, never sample rows; rejected: proxying a vendor key we own, bundling weights, MCP exposure, per-feature providers Accepted
0043 In-app workspace membership — one workspace_members table an admin manages from Admin → Members; empty table = today's behaviour, non-empty = member if in the env allowlist OR the table, checked on every request at four points (_upsert_user, OTP session resolve, PAT resolve, OTP eligibility) so removing someone stops their browser session and their PATs on the next request; env allowlists become a starting seed plus an emergency way in that can only grant (OTP mode still needs a non-empty one to boot); the first row imports every existing user as a provisional member shown for review, never a silent backfill; last-admin removal refused, self-removal confirmed; initial_role applies only when the user row is first created; the IdP account stays a prerequisite — DataQ controls its own door only; REST denies with 403, /mcp with 401 Accepted