Markdown

Features¶

Everything DataQ offers, by area. This is the readable tour — for the terse what-runs-where lookup see the feature matrix, and for how to turn each feature on the recommended way see Recommended usage.

DataQ is a single-tenant data-quality monitoring platform: it runs checks against your data stores, records every result, surfaces health on a dashboard, alerts the right people, and reacts to your pipelines — all behind SSO, with an AI-assistant surface.

Datasources¶

Stores you write checks against (see Datasources & checks):

Snowflake (DEV / QA / UAT) — key-pair or password auth.
Unity Catalog (Databricks) — three-level catalog.schema.table.
ADLS Gen2 and AWS S3 (or any S3-compatible store) — flat files, a single object or a batch pattern (orders_*.csv, latest-or-specific) in CSV or Parquet.
Apache Iceberg — native pyiceberg read straight from object storage, no query engine in front: namespace.table addressing, REST / SQL / Glue / Hive catalogs, credential-less catalogs supported, also reads Delta UniForm tables (ADR 0030).

Checks & authoring¶

A check is one data-quality rule, authored in the UI. Six kinds ship (check.kind): GX expectations (including custom SQL), and the freshness / volume / schema-drift / anomaly / comparison monitor kinds:

DQ dimension — every check is classified by the quality aspect it measures (accuracy, completeness, consistency, integrity, timeliness, uniqueness, validity). It is filled in for you from the check type and stays editable; it powers the asset scorecard below. See Datasources & checks.
GX expectations — column- and table-shape rules (not-null, unique, in-set, row count, value ranges, …) from the GX catalog, with a form editor. On a Snowflake connection, four of these (null count/percent, duplicate count, unique count) can instead run on Snowflake's own Data Metric Functions (DMF) engine — a separate category in the check editor, selected per check.
Custom SQL — an escape hatch for cross-column/join rules: rows returned = failures. Read-only, single-statement (enforced). SQL datasources only.
Freshness monitor — hours since the latest timestamp (is the data stale?). On a flat file you can leave the timestamp column blank, and it measures when the file last landed instead — which catches a producer that stopped sending files, something a timestamp inside the data cannot see.
Volume monitor — row count within an expected range (did the load land whole?). Freshness + volume are the auto-monitor kinds and run on every datasource: the SQL datasources (Snowflake / Unity Catalog), Apache Iceberg (computed natively via a pyiceberg scan rather than SQL), and ADLS Gen2 / S3 flat files.
Schema-drift monitor — flags a column added, dropped, or type-changed against a stored baseline (did the shape change under you?). Also runs on every datasource — introspection is per-store (SQL information_schema, a Parquet footer / CSV header sample for flat files, Iceberg table metadata), never a data scan.
Anomaly monitor — a rolling z-score baseline over a check's metric_value history, with optional seasonality (weekday-aware); flags a value that is abnormal for this dataset rather than one crossing a fixed threshold. Skips on cold start instead of false-alerting, and the trend view overlays the learned baseline.
Comparison check — cross-dataset reconciliation (ADR 0015): diff the suite's dataset against a baseline on any other connection (cross-type and cross-env both work), joined on key columns, producing matched / mismatched / additional-per-side buckets with a mismatch-% metric and an on-demand CSV/XLSX report — for migration / replication / promotion validation.
Column profiler — nulls, distinct count, min/max, and top values for a column, on any datasource — the baseline you set thresholds from.
Dry-run preview — run one check against live data without persisting, on every datasource with a runner (Snowflake / UC / flat files / Iceberg).
Severity thresholds — band a check into warn / fail / critical from the observed unexpected-% (ADR 0005/0016); leave blank for binary pass/fail.
Check version history — every edit is versioned and viewable per check.

Suites¶

A suite is a named set of checks bound to one connection + target (Datasources & checks):

Run target — one table (SQL) or one file/batch (flat file) per suite.
Sharing — per-suite view / edit grants (suite-level is the access model; no folder scopes). See access & auth.
Export / import — move a suite's definition between environments (author on DEV, import against PROD) instead of re-clicking checks.

Running checks¶

Four ways a suite runs (all the same authz — feature matrix):

Run now — with live per-check progress and cancel.
Cron schedule — 5-field cron + IANA timezone, DST-aware, no backfill (Scheduling).
Pipeline trigger — runs automatically when a bound ADF / Airflow / dbt run succeeds (Orchestration).
API / MCP — POST /suites/{id}/run, or the trigger_suite_run MCP tool.

Results, dashboard & history¶

Monitoring dashboard — leads with asset-level health (how many assets are monitored, need attention, or have a run in flight — click through to an asset or the full Assets view), then a severity-weighted health score, pass rate, run counts, average duration, per-day trends, and per-suite performance. Assets are the primary lens (ADR 0034 nav inversion): the sidebar leads with Assets above Suites, and every suite/run links back to its asset.
Warehouse inventory sync (ADR 0040) — an opt-in per-connection toggle (Snowflake / Unity Catalog connection forms) that enumerates every table the connection can see into the asset view daily, so a table with no suite, run, or lineage signal is visible as unmonitored instead of invisible. Synced tables age out automatically once dropped from the warehouse. Unity Catalog needs SELECT on system.information_schema for the connection's principal.
Asset DQ scorecard — per-dimension coverage and score on the asset page: which quality dimensions have checks, how they scored on the latest run, and — the part worth acting on — which dimensions have no checks at all. Three states are kept distinct on purpose: a score, "no signal" (checks exist but none evaluated), and "not covered" (no checks) — an asset nobody watches never shows a green tick. The numbers are workspace-wide, so two people looking at one asset see one verdict.
Asset browse — two lenses over the same assets: a By source drill-down (datasource → database/catalog → schema → table, with env + health on each leaf) and a flat, searchable All assets table (real server-side paging, 50 rows/page). Both open the asset detail (health across suites, lineage, incidents). A workspace bigger than the tree view can comfortably render (2000 assets) shows an explicit "showing N of M" note rather than a silently partial tree — switch to All assets to page through the rest.
Datasources read as names, not connection strings — an asset is identified by its OpenLineage namespace (the physical location, so DataQ's identifiers join byte-for-byte with dbt/Spark emissions), but it is displayed as a human label: Snowflake · ACCT, ADLS · account/container, iceberg_catalog. The raw namespace stays one hover away (and is what the copy button copies). Note the label is deliberately not the connection name: several connections can point at one datasource instance and all resolve to the same namespace — which is exactly what makes the same table reached two ways a single asset.
Lineage graph — an asset's provenance and blast radius render as one left-to-right graph (upstream → this asset → downstream), one column per hop, with clickable nodes. Depth ≥2 comes from the blast-radius BFS, which also returns the real edges — so a two-hop consumer is drawn hanging off the node it actually descends from, not off the asset it never touches.
Two health axes on an asset — Connection health ("could DataQ reach and execute against the datasource?" — a failed run, or a check the datasource threw on) and Data-quality health (the severity-weighted verdict of its suites). They are derived and shown separately, so a datasource DataQ cannot connect to never masquerades as a data failure — and a run that evaluated nothing reads No data, never a green "passing". Operational error/skip results feed only the connection axis. Both are read off the runs already recorded — there is no connection-probe polling loop.
Results page — every run with its per-check outcomes; drill into a run for observed-vs-expected values and redacted failing-row samples.
Severity + operational statuses — warn / fail / critical for data outcomes, plus error (evaluation threw) and skip (precondition unmet) kept distinct from failures.
Failure reasons — a run that fails to execute shows a redaction-safe reason (config / connectivity / permission), not a bare "failed".
Metric trend view — a per-check history chart of metric_value over time, with threshold bands and (for anomaly checks) a rolling z-score baseline overlay; the numeric history is kept for trend/baseline even after samples are purged by the retention sweep.
PDF report export — a one-click, zero-dependency PDF of a run's results, for sharing outside the app.
Usable on a phone — the whole app, not just the dashboard. On a narrow viewport the sidebar becomes an overlay drawer (it never squeezes the page), tables scroll inside their own container rather than the page, the lineage graph scrolls inside its card, and the panels that grant access or edit a suite reflow so their controls stay on screen.

Orchestration (monitor + trigger only)¶

DataQ observes pipelines and triggers suites on success — it never runs them, and these are not datasources (Orchestration):

Azure Data Factory — Azure Monitor alert → webhook (+ 10-min poll).
Apache Airflow — DAG callback → HMAC webhook (+ 10-min poll).
dbt — post-build callback → HMAC webhook (+ 10-min run_results.json artifact poll).

All three sit behind one provider interface; trigger bindings map (provider, pipeline/DAG/job, env) → suite.

Alerting¶

Delivered when a run breaches its threshold (Notifications & alerting):

Channels — Microsoft Teams, Slack, email and generic HMAC webhooks, defined once by an admin and linked per suite, with deep links + expected-vs-observed context.
Threshold — per suite: fail-only / warn-and-worse (default) / always.
Severity routing — urgency scales with severity; critical escalates.
Dedup — you hear about a breakage once (and again on escalation); a clean run resets.
Snooze — silence a known-broken check for N hours (history + re-fire preserved).
Per-suite config — channels + threshold + recipients set per suite.
Connection poll-health alerts — when an orchestration connection's poll keeps failing (an expired credential, a moved orchestrator), DataQ stops ingesting pipeline runs and firing the suites bound to them. After 3 consecutive failures it alerts — once on the way down, once on recovery — with the classified reason.

Data protection¶

Column-aware redaction — failing-row samples are the one place results can carry PII; a suite column policy (auto-derived by the classifier or set by hand) keeps non-sensitive breaches debuggable while PII columns stay masked.
PII-minimising retention — samples are purged after the retention window; the aggregatable metric history survives.

Access & auth¶

Two sign-in modes — email OTP (a one-time code, no Identity Provider required — the local/eval default, ADR 0032) and OIDC SSO (provider-neutral, Azure AD and AWS Cognito both validated in prod). Exactly one is active per deployment.
Personal access tokens (PATs) — dq_live_ tokens for headless / AI-client use, same authz as the user, on REST and /mcp (API keys, ADR 0026).
Suite sharing — per-suite view / edit grants, capped by workspace role (below).
Workspace roles — Admin / Member / Viewer — a stored, in-app-managed role per user (ADR 0033), not just an env allowlist. Admin has workspace-wide visibility over every suite/result/schedule and is the only role that can create, edit, delete, re-auth, or test a connection — connections are shared infrastructure holding credentials, so Members consume them but don't manage them. Member can create/own suites and receive edit shares; Viewer is capped at view everywhere, including on any share. Roles are managed under /admin/members by an existing Admin; WORKSPACE_ADMIN_EMAILS still works as a bootstrap/break-glass allowlist for a fresh or locked-out workspace.

AI-assisted authoring and root-cause analysis¶

An optional, off-by-default LLM integration (bring-your-own Anthropic or OpenAI-compatible provider + credential, admin-configured) powers three features: NL→SQL generation for the custom-SQL check editor, curated check suggestions driven by a live column profile constrained to a vetted expectation vocabulary, and a root-cause narrative for a failed check — ranked hypotheses citing real evidence-card layers, plus what the evidence snapshot could not see. SQL generation sends column names and counts; check suggestions add masked profile statistics, including each non-sensitive column's most frequent values; the RCA narrative additionally sends the triggering check's own observed value, routed through the same column-policy/warehouse-tag redaction floor every results surface applies. Raw sample rows never leave the deployment on any path. See AI features for a walkthrough with real output, and Security & data handling for the full transfer-vector accounting.

AI assistants (MCP)¶

A curated 48-tool MCP server at /mcp lets Claude / Copilot / Cursor list suites, read results, checks, runs, connections, schedules, trigger bindings and notification config, trigger and cancel runs, poll status, add/update/delete/snooze checks, dry-run a check preview, manage schedules and trigger bindings end-to-end (create / update / delete), import a suite and set its run target, read and restore a check's version history, read and set a suite's column policy, browse assets (the tables DataQ monitors, with health and lineage) and incidents (what is broken right now, with the evidence — acknowledge and resolve them, and surface near-miss triggers that silently never fire), list a target's columns, profile columns, suggest a PII policy, test a connection, read a curated docs page, and read the health score and pipeline status — in natural language, with the same per-suite authz as the UI. The 48 split three ways: 25 read-only, 18 that change state, and 5 that persist nothing but open a live datasource connection with stored credentials and so are gated like writes (AI assistants, ADR 0008 + its Tier 1–3B amendments).

Observability¶

Structured JSON logs with request_id correlation (FastAPI → Celery → GX), PII redaction at the logger, and OpenTelemetry traces + logs — exported to Azure Application Insights and/or a generic OTLP endpoint (Observability).

Deployment & portability¶

Runs live today on both Azure Container Apps and AWS ECS Fargate — two independent, equally-real deployments (API + worker + frontend; the frontend is the sole public surface on each). Every cloud dependency sits behind a seam — OIDC auth, the secret store, observability export, and the orchestration providers are each swappable behind a provider-agnostic interface, which is what let AWS ship as a second target with zero app code differing between clouds (Architecture, deployment parity).