# Architecture Decision Records (ADRs)

Each ADR captures a single significant architecture decision: the context, the decision, the consequences, and the alternatives considered. New ADRs are append-only — supersede an old decision by adding a new ADR and marking the old one's status as `Superseded by ADR-NNNN`.

## Format

- File name: `NNNN-short-kebab-slug.md` (zero-padded 4-digit sequence)
- Frontmatter fields:
  - **Status** — one of `Proposed`, `Accepted`, `Deprecated`, `Superseded by ADR-NNNN` (title-case)
  - **Date** — `YYYY-MM-DD`
  - **Deciders** — who made the call
  - **Consulted** *(optional)* — stakeholders whose sign-off the decision needed (e.g. product owner for ADR 0005). Omit when none.
  - **Supersedes** *(optional)* — `ADR-NNNN` this decision replaces. Omit when none.
  - **Superseded by** *(optional)* — `ADR-NNNN` that later replaced this one. Add when the status flips to `Superseded by`.
  - **Amends** *(optional)* — `ADR-NNNN` this decision partially overrides (whole-ADR replacement uses `Supersedes` instead). Pair it with an inline `> **Amendment (date, ADR-NNNN):** …` blockquote at the top of the amended ADR and an "(amended by NNNN — …)" note on its index Status, so the override is visible where readers actually look. Precedents: ADR 0028 (amends 0024), ADR 0012's amendment blockquote, ADR 0031 (amends 0013).
- Sections: Context, Decision, Consequences, Alternatives considered, Related (optional)
- Keep each ADR short — 1–2 pages. If it grows past that, the decision is probably two decisions.

### Template

```markdown
# ADR NNNN — <title>

- **Status:** Proposed
- **Date:** YYYY-MM-DD
- **Deciders:** @handle
- **Consulted:** <stakeholder>   <!-- optional; omit when none -->
- **Supersedes:** ADR-NNNN       <!-- optional; omit when none -->
- **Superseded by:** ADR-NNNN    <!-- optional; add when superseded -->

## Context

## Decision

## Consequences

## Alternatives considered
```

## Index

| # | Title | Status |
|---|---|---|
| [0001](0001-trunk-based-branching.md) | Trunk-based branching with squash-merge into `main` | Accepted |
| [0002](0002-conventional-commits.md) | Conventional commits for PR titles and commit messages | Accepted |
| [0003](0003-gx-only-for-v1.md) | GX-only for v1; DQX deferred to v1.1 for DLT/streaming | Accepted (amended by 0036 — suite-level `engine: gx \| dqx` toggle → check-level, connection-anchored) |
| [0004](0004-orchestration-abstraction.md) | Unified `OrchestrationProvider` abstraction for ADF and Airflow | Accepted |
| [0005](0005-severity-tier-weights.md) | Severity tier weights (warn / fail / critical → health score) | Accepted |
| [0006](0006-adf-webhook-authentication.md) | ADF webhook authentication (shared secret in URL + hard-cutover rotation) | Accepted |
| [0007](0007-airflow-callback-model.md) | Airflow callback model (HMAC-signed webhook + polling fallback) | Accepted |
| [0008](0008-mcp-server.md) | FastMCP server at `/mcp` — Azure AD token validated (same token as REST); everything exposed as tools rather than resources; thin wrappers reusing the service layer + per-suite authz; fail-closed without auth. **Amended:** 8 → 19 tools (Tier 1 reads), then 19 → 30 tools (Tier 2 edit-gated mutations + live-probe tools), then 30 → 33 → 38 (Tier 3A coherence), then 38 → 42 (Tier 3B batch 1 — assets + incidents reads) → 46 (Tier 3B batch 2 — incident lifecycle, `list_columns`, near-misses; Tier 3B complete) → 47 (`get_pipeline_status`, `get_adf_pipeline_status` kept as a deprecated alias) → **48** (`get_doc`, curated docs pages; 25 read-only / 18 state-changing / 5 live-probe) | Accepted |
| [0009](0009-flat-monorepo-layout.md) | Repo layout — flat monorepo (`backend/` + `frontend/`) | Accepted |
| [0010](0010-provider-agnostic-infrastructure-seams.md) | Provider-agnostic infrastructure seams (Azure is the default, not the architecture) | Accepted |
| [0011](0011-extensibility-seams-for-deferred-integrations.md) | Extensibility seams for deferred connectors and integrations | Accepted |
| [0012](0012-monitor-kind-seam.md) | Monitor-kind seam (`check.kind` discriminator + numeric metric storage) | Accepted |
| [0013](0013-marketplace-distribution-and-anti-lock-in.md) | Marketplace distribution (customer-deployed BYOL) and anti-vendor-lock-in guardrails | Accepted (amended by 0031 — §5 licensing line + licensed-revenue framing) |
| [0014](0014-reconciliation-comparison-check-kind.md) | Cross-dataset reconciliation as a `comparison` check kind (reuse FastAPI_DataComparison engine) | Accepted |
| [0015](0015-two-connection-comparison-check-model.md) | Two-connection comparison check model — suite stays single-connection and supplies the **target under test**; a `comparison` check adds one **source** ref (`checks.source_connection_id` FK) + suite-target-shaped `config.source`; execution via a new `DatasetReader` seam + the ported FDC engine, row-cap fail-fast; report files derived on-demand, never stored; **no** connection→connection generalization (0030 Option B stays deferred). **Amended 2026-09-16: comparison sources do not support `sampling`** — coherent key-set sampling is a different mechanism from the suite-target's positional sampling and was not built; `COMPARISON_MAX_ROWS` fail-fast + query-filter narrowing are the sanctioned alternatives | Accepted (built 2026-07-12) |
| [0016](0016-severity-derivation-semantics.md) | Severity derivation semantics (band the unexpected-%, thresholds override GX success) | Accepted |
| [0017](0017-python-313-runtime-upgrade.md) | Upgrade Python runtime 3.11 → 3.13 (3.14 deferred — GX-capped); bundled with the Snowflake 3→4 CVE refresh | Accepted |
| [0018](0018-results-surface-and-grafana-deferral.md) | Results surface is an in-app page (suite-scoped authz + PII redaction); Grafana deferred to optional ops add-on | Accepted |
| [0019](0019-custom-sql-check-kind.md) | Custom-SQL checks ride `kind='expectation'` via GX `UnexpectedRowsExpectation` (no new kind); read-only validation + SQL-datasource gating | Accepted (amended 2026-08-08 — UC custom SQL needs a GX Databricks-SQL batch, not the pandas one; it had never worked before) |
| [0020](0020-history-and-audit-strategy.md) | History/audit: per-entity Type-4 snapshot tables (`check_versions`, `connection_versions`) where config history is needed; no SCD-2; credentials never snapshotted; cascade-delete accepted; cross-entity audit log deferred | Accepted (amended by 0041 — the deferred audit log is now accepted; soft-delete re-deferred, SCD-2 closed) |
| [0021](0021-demo-test-data-environment-strategy.md) | Live test/demo-data environment (retail model, 3 reference flows) lives outside the repo — IaC stack/mock-data/Databricks notebook not git-tracked; discharges the deferred live-warehouse/file smoke | Accepted (amended 2026-07-27 — harness IaC CLI is **OpenTofu**, converted with the app stack; see ADR 0024) |
| [0022](0022-week6-prototype-adoption-and-chart-library.md) | Week-6 prototype adoption — full 13-screen set as dedicated pages (Share is the only drawer; prototype wins on conflicts; Settings/Admin pulled into W6); chart library = recharts (lazy-loaded) | Accepted |
| [0023](0023-container-image-registry-ghcr.md) | Container image registry — GitHub Container Registry (GHCR) over ACR / Docker Hub (vendor-neutral, public package → anonymous ACA pull, reuses `GITHUB_TOKEN`); supersedes the earlier ACR scaffolding | Accepted |
| [0024](0024-app-deployment-infrastructure.md) | App deploy infra — in-repo IaC stack; shares subscription/RG/ACA-env/Postgres with the harness; ACA api+worker+migrate-job + SWA-Standard linked same-origin `/api` + self-hosted Redis + KV (UAMI) + App Insights + AAD-app-reg OIDC | Accepted (amended 2026-07-27 — **OpenTofu** replaces Terraform as the IaC CLI, in-place amendment; ADR 0031 coherence, state/providers/path unchanged) |
| [0025](0025-production-image-pip-slim.md) | Production image — multi-stage `python:3.13-slim` + pip (not conda; ~2.84GB→~1GB); conda stays the local-dev tool; amends the W1 conda lock | Accepted |
| [0026](0026-auth-api-keys-and-principal-seam.md) | DataQ-issued API keys (PATs) as a second authenticator behind the `get_current_user` seam — REST + `/mcp` identically; phase 1 = user-scoped PATs (`dq_live_…`, sha256-at-rest, show-once, uniform 401, mandatory expiry, owner-cascade); service-account principals = phase 2 (deferred); HTTP Basic rejected | Accepted (phase 1 built 2026-07-04; amended by 0032 — email-identity slice of the phase-2 principal question) |
| [0027](0027-suite-permission-model-workspace-admin.md) | Suite permissions — workspace-admin is implicit `admin` on every suite (governance/break-glass); drop grantable suite-admin; normal users get owner/edit/view; workspace-admin gets workspace-wide visibility (supersedes earlier, narrower workspace-admin decisions) | Accepted (amended by 0033 — admin source → stored `users.role`; Viewer share-cap) |
| [0028](0028-cloud-neutral-image-runtime-config-generic-oidc.md) | Cloud-neutral image — one multi-arch frontend image, nothing baked; auth config injected at runtime (`window.__DATAQ_CONFIG__` via nginx envsubst) behind a generic `DATAQ_AUTH_*` contract; bypass fail-closed (explicit `DATAQ_AUTH_MODE=bypass` only); replace MSAL with a generic OIDC client validated against Azure; frontend SWA→Container App (amends 0024); AWS/GCP IaC post-v1 | Accepted (amended by 0032 — `otp` mode + cookie session credential) |
| [0029](0029-dbt-orchestration-provider.md) | dbt as a **third** `OrchestrationProvider` (mirrors the Airflow callback model 0007) — HMAC webhook + artifacts poll of `run_results.json` (adls/s3/file); binds to dbt's universal surface (no host API); job-level grain; migration widens the connection-type/provider/dedup value-sets | Accepted |
| [0030](0030-iceberg-native-read-path.md) | Apache Iceberg — engine-level read (Snowflake/UC iceberg tables) is free & zero-code; the only new build is a **native `pyiceberg` read** (v2 baseline, v3 deferred) behind a thin `IcebergCheckRunner` (scan → DataFrame → `gx_runner`); new **self-contained** `iceberg` connection type (Option A: own catalog + storage credential — independent lifecycle, cascade-safe), Option B two-connection ref deferred to 0015; reads Delta UniForm too; native impl deferred | Accepted (native read shipped; introspection built) |
| [0031](0031-oss-byol-distribution-licensing.md) | Distribution licensing — **free open-source (MIT) + customer-deployed BYOL**; no entitlement/license-key (amends 0013: supersedes its §5 licensing-model line + licensed-revenue framing); marketplace listings are free offers of the OSS artifacts; THIRD-PARTY-NOTICES/SBOM in images + releases; standing no-strong-copyleft dependency guardrail (CONTRIBUTING rule 40) | Accepted |
| [0032](0032-email-otp-signin.md) | Email OTP sign-in — passwordless **third authenticator** behind `get_current_user` (`dq_sess_` cookie sessions, PAT-style sha256-at-rest); auth-mode ladder `bypass · otp · oidc`, fail-closed startup; mandatory signup allowlist (no open registration); one user row per normalized email (aad_object_id → nullable, two-step); separate `AUTH_EMAIL_*` mailer + SMTP pre-flight; hard prereq = the rate-limiting auth slice; amends 0026 (email-identity slice) + 0028 (mode enum + cookie credential) | Accepted (amended by 0033 — signup default role + widened trust statement) |
| [0033](0033-workspace-roles-rbac.md) | Workspace roles — **Admin / Member / Viewer** as stored `users.role` on the two-axis model (role × per-suite ladder, ladder untouched); **connection mutations Admin-only** (closes the workspace-global hole, breaking for Members); Viewer capped at `view`; `WORKSPACE_ADMIN_EMAILS` demotes to bootstrap/break-glass; in-app role management + last-admin guard; amends 0027 (admin source) + 0032 (signup default role) | Accepted |
| [0034](0034-asset-entity-openlineage-identity-lineage-pull.md) | Assets & lineage (gap G-d design) — first-class **asset entity keyed by the OpenLineage dataset-naming spec** (namespace+name adopted verbatim, dbt-ol/Spark byte-compatible; DEV/QA = distinct assets, UI groups over `env`); **lineage emitted/pulled, never built** — OL emission w/ quality facets → dbt `manifest.json` → `lineage_edges` cache → `LineageProvider` seam (Marquez reference; OpenMetadata SDK license-blocked per 0031); incidents anchor to assets (one open per asset+check, evidence-card payload); asset/incident authz derived from suite grants | Accepted |
| [0035](0035-request-rate-limiting.md) | Request rate limiting — **app-level innermost HTTP middleware** (portable per 0028/0013, covers the `/mcp` mount a dependency can't, principal-aware where nginx isn't); fixed-window Redis counter (window index in the key → no EXPIRE race), **no new dep** (INCR over slowapi/limits); **fail-open**; endpoint classes `webhook`(per-IP) · `default`(per-`sha256(token)`) · `unauth`(per-IP), raw token never keyed/logged; `/healthz`+`OPTIONS` exempt; headers on 429 only; nginx `limit_req` out of scope; hard prereq for 0032's OTP | Accepted |
| [0036](0036-connection-anchored-check-engines.md) | Connection-anchored check engines — **engines are capabilities of connections** (GX universal; `dmf`⇔snowflake, `dqx`⇔unity_catalog, `dataplex`⇔future bigquery); **`check.engine`** per check (supersedes 0003's suite-level `gx \| dqx` toggle); type gates the offer, connection **probe** validates reality (classified remediation, never raw errors); **`kind` ⊥ `engine`** (engines = alternate evaluators of existing kinds; engine-aware backend catalog lands with the dimension-classification work); lifecycle validated save/run/import-time (`error` status, never silent); DMF first build, DQX/Dataplex trigger-gated; scheduled-native ingest deferred to a future ADR | Accepted (amends 0003; DMF first native build; DQX/Dataplex trigger-gated) |
| [0037](0037-workspace-visible-asset-identity.md) | Workspace-visible asset identity — **asset identity + lineage topology (incl. column pairs) visible to every member**; aggregate verdicts (health rollup, future scorecard) **workspace-true** (all composing suites, one number for all viewers); grant boundary moves to **suite-derived detail** (composing-suite names/runs/results/samples/incidents — ADR 0027 ladder unchanged, suite-grain 404-no-leak intact); asset detail 200s for all, `restricted_suite_count` replaces redaction; supersedes an earlier identity-redaction regime (amends 0034 decision 5) | Accepted |
| [0038](0038-dq-dimension-classification.md) | DQ-dimension classification on checks — **seven canonical dimensions** (accuracy/completeness/consistency/integrity/timeliness/uniqueness/validity) as a **closed** vocabulary via table `CHECK`, not a Postgres ENUM (no `ALTER TYPE` friction) and not free text (unbounded values destroy coverage reporting); **derived default → stored → overridable at any time** (stored so the scorecard aggregates in SQL and an override survives); derivation is **deliberately partial** — `accuracy`/`integrity` are never derived and custom SQL has no answer, so **NULL means unclassified** and the scorecard must render it as a gap; **existing rows ARE backfilled** (migration `a7b8c9d0e1f2`; §5 **amended 2026-07-19** — the backfill cannot overwrite a human decision because none existed yet when it ran, while leaving them NULL made the scorecard useless on day one; unmapped types stay NULL); `dimension` ⊥ `kind` ⊥ `engine`; narrows 0036 §4 — lands the derivation map as the backend catalog's seed without moving the frontend catalog server-side, since no native engine exists yet | Accepted |
| [0039](0039-openbao-self-hosted-secret-backend.md) | Self-hosted secret backend — a fourth `SecretStore` speaking the **KV v2 HTTP API, not a vendor SDK** (one mode serves OpenBao / Vault Community / Enterprise / HCP; no HashiCorp lock-in traded for the Azure one); **OpenBao** (MPL-2.0, LF fork of pre-BUSL Vault 1.14) is the shipped/tested/pinned server — Vault Community is BUSL-1.1 and rule-40-forbidden as a *distributed* component, supported as a *target*; **`SECRET_STORE=openbao` replaces `redis` and `RedisSecretStore` is deleted** (plaintext credentials were the default of the published eval stack), with a loud one-cycle `RuntimeError` shim, **Redis stays** for broker + rate-limit counters; token auth phase 1 (AppRole deferred/filed), dev-mode token generated into the gitignored `.env`; Azure Key Vault remains the prod default (managed identity beats a held token), OpenBao is the BYOL/cloud-neutral option; **an outage is never reportable as a state** — new `SecretStoreUnavailableError`, deliberately NOT a `SecretNotFoundError` subclass, since every caller branches on the TYPE and none reads the message (a sealed vault otherwise renders "not set" on the admin page, skips alert delivery silently, and runs connections with the credential omitted); no status is silent (400=KV v1 mount, 401=gateway, 429=HCP throttle); **`AzureKeyVaultStore` had the same defect since Week 2** and is fixed too (the same invisible-degradation shape seen before); `delete` purges via KV v2 **metadata** delete, fail-soft | Accepted |
| [0040](0040-warehouse-inventory-sync-table-enumeration-seam.md) | Warehouse inventory sync + the **table-enumeration seam** — one per-datasource `enumerate_tables` capability (Snowflake/UC `information_schema`, engine-native case = the identity-safe path, connection-database scoping, ephemera/system schemas excluded) feeding three consumers: **inventory sync** (opt-in per connection, daily crontab beat, `upsert_assets(preserve_provenance=True)` so `last_seen` advancement keeps live tables out of the orphan sweep and dropped tables age out through it — no new column, no new guard; no "discovered" badge, ADR 0037's neutral unmonitored rendering IS the marker), **GET_LINEAGE seeds**, and future pickers; `ASSET_INVENTORY_MAX_TABLES` cap (default 2000) with loud truncation; **decided**: S3 namespace stays `s3://{bucket}` for AWS, becomes `s3://{host[:port]}/{bucket}` only when `endpoint_url` is set | Accepted |
| [0041](0041-history-audit-strategy.md) | History & audit posture for v1.x+ (amends 0020) — **one append-only `audit_events` table** (actor/action/entity/before/after/ts) recording **deliberate acts by a principal only**, with `entity_id` deliberately **FK-less** so the row outlives the entity it describes (an FK would either cascade the audit away or block the delete); the data-**read** audit (gap G1) is **phase 2 on the same table**, discriminated by `action_class` — not a second table — with the opposite latency contract (phase 1 writes inside the mutation's transaction, fail-closed; phase-2 reads must stay off the critical path); **(a) preserving `check_versions` past delete rejected** — `RESTRICT` makes checks undeletable, `SET NULL` orphans rows needing self-contained identity (a worse audit log scoped to one entity), tombstone = soft-delete for one table — the audit log's delete event answers it instead and **cascade stands**; **(b) soft-delete re-deferred** on a narrowed argument (its headline motivation is already met — an earlier 409 guard means a connection can no longer be deleted out from under its runs; the column is the easy part vs. "deleted means inert" for live credentials/bindings/schedules; and a delete-that-doesn't-delete fights the G2 Art-17 erasure work), mitigated by a follow-up blast-radius confirmation; **(d) SCD-2 closed permanently**, not deferred; Type-4 tables stay the **product** surface (version drawer/restore), the audit log is the **durable record**; payloads are **per-entity allow-lists** reusing the snapshot builders, never `dict(row)`, never a secret value, never `sample_failures`/`observed_value`, and reuse only the **credential subset** of `_PII_KEYS` (the full set redacts `name`/`display_name`/`user_id` — the very content of a rename and the point of the actor record); append-only via `REVOKE UPDATE, DELETE` — **a guard against accidental in-app mutation, explicitly NOT tamper-resistance** (`dataq_app` owns the table and can re-grant; splitting the role is rejected because the single-role model is the whole mitigation for the unpatched Postgres RI owner-cast escalation, so real tamper-evidence needs the G1 external anchor); coverage guarded off FastAPI's **route table**, not an audit registry (the ADR-0039 tautology) | Accepted (phase-1 build; follow-ups filed) |
| [0042](0042-llm-provider-seam.md) | `LLMProvider` seam — outbound LLM for authoring assistance (NL→SQL generation, check suggestions, future RCA narrative), **admin-configured, BYO credential, default-OFF**; two wire impls only (`anthropic` official SDK + `openai_compatible` httpx — the latter covers Azure OpenAI, Bedrock chat-completions, and local Ollama/vLLM/TGI; no model server in the image per 0025); structured output is a `native \| prompt_json` capability ladder and **never a trust boundary** — callers re-validate, generated SQL rides the 0019 validator + dry-run, suggestions the server-side allowlist; config = `llm_settings` DB singleton behind `require_workspace_admin`, API key write-only in `SecretStore` with the credential-destination re-supply rule on `base_url`; execution worker-side through `llm_invocations` (poll + audit + token/cost record; G4 posture reads it); prompt context = schema + masked aggregate profiler stats only, never sample rows; rejected: proxying a vendor key we own, bundling weights, MCP exposure, per-feature providers | Accepted |
| [0043](0043-in-app-workspace-membership.md) | In-app workspace membership — one `workspace_members` table an admin manages from Admin → Members; **empty table = today's behaviour, non-empty = member if in the env allowlist OR the table**, checked on every request at four points (`_upsert_user`, OTP session resolve, PAT resolve, OTP eligibility) so removing someone stops their browser session and their PATs on the next request; env allowlists become a starting seed plus an emergency way in that can only grant (OTP mode still needs a non-empty one to boot); the first row imports every existing user as a **provisional** member shown for review, never a silent backfill; last-admin removal refused, self-removal confirmed; `initial_role` applies only when the user row is first created; the IdP account stays a prerequisite — DataQ controls its own door only; REST denies with 403, `/mcp` with 401 | Accepted |