# Features

Everything DataQ offers, by area. This is the readable tour — for the terse
what-runs-where lookup see the [feature matrix](../reference/feature-matrix.md), and for how to turn
each feature on the recommended way see [Recommended usage](recommended-usage.md).

DataQ is a single-tenant **data-quality monitoring platform**: it runs checks against your
data stores, records every result, surfaces health on a dashboard, alerts the right people,
and reacts to your pipelines — all behind SSO, with an AI-assistant surface.

## Datasources

Stores you write checks **against** (see [Datasources & checks](datasources-checks.md)):

- **Snowflake** (DEV / QA / UAT) — key-pair or password auth.
- **Unity Catalog (Databricks)** — three-level `catalog.schema.table`.
- **ADLS Gen2** and **AWS S3 (or any S3-compatible store)** — flat files, a single object or a **batch** pattern
  (`orders_*.csv`, latest-or-specific) in **CSV** or **Parquet**.
- **Apache Iceberg** — native `pyiceberg` read straight from object storage, no query
  engine in front: `namespace.table` addressing, REST / SQL / Glue / Hive catalogs,
  credential-less catalogs supported, also reads Delta UniForm tables (ADR 0030).

## Checks & authoring

A check is one data-quality rule, authored in the UI. Six kinds ship (`check.kind`): GX
**expectations** (including custom SQL), and the **freshness / volume / schema-drift /
anomaly / comparison** monitor kinds:

- **DQ dimension** — every check is classified by the quality aspect it measures
  (accuracy, completeness, consistency, integrity, timeliness, uniqueness, validity).
  It is filled in for you from the check type and stays editable; it powers the asset
  **scorecard** below. See [Datasources & checks](datasources-checks.md#dq-dimension-adr-0038).

- **GX expectations** — column- and table-shape rules (not-null, unique, in-set, row
  count, value ranges, …) from the GX catalog, with a form editor. On a Snowflake
  connection, four of these (null count/percent, duplicate count, unique count) can
  instead run on Snowflake's own **Data Metric Functions (DMF)** engine — a separate
  category in the check editor, selected per check.
- **Custom SQL** — an escape hatch for cross-column/join rules: rows returned = failures.
  Read-only, single-statement (enforced). SQL datasources only.
- **Freshness monitor** — hours since the latest timestamp (is the data stale?). On a
  flat file you can leave the timestamp column blank, and it measures when the **file
  last landed** instead — which catches a producer that stopped sending files, something
  a timestamp inside the data cannot see.
- **Volume monitor** — row count within an expected range (did the load land whole?).
  Freshness + volume are the auto-monitor kinds and run on **every** datasource: the SQL
  datasources (Snowflake / Unity Catalog), Apache Iceberg (computed natively via a
  `pyiceberg` scan rather than SQL), and ADLS Gen2 / S3 flat files.
- **Schema-drift monitor** — flags a column added, dropped, or type-changed against a
  stored baseline (did the shape change under you?). Also runs on **every** datasource —
  introspection is per-store (SQL `information_schema`, a Parquet footer / CSV header
  sample for flat files, Iceberg table metadata), never a data scan.
- **Anomaly monitor** — a rolling z-score baseline over a check's `metric_value` history,
  with optional seasonality (weekday-aware); flags a value that is abnormal *for this
  dataset* rather than one crossing a fixed threshold. Skips on cold start instead of
  false-alerting, and the trend view overlays the learned baseline.
- **Comparison check** — cross-dataset reconciliation (ADR 0015): diff the suite's
  dataset against a baseline on **any other connection** (cross-type and cross-env both
  work), joined on key columns, producing matched / mismatched / additional-per-side
  buckets with a mismatch-% metric and an on-demand CSV/XLSX report — for migration /
  replication / promotion validation.
- **Column profiler** — nulls, distinct count, min/max, and top values for a column, on
  any datasource — the baseline you set thresholds from.
- **Dry-run preview** — run one check against live data **without persisting**, on every
  datasource with a runner (Snowflake / UC / flat files / Iceberg).
- **Severity thresholds** — band a check into **warn / fail / critical** from the observed
  unexpected-% (ADR 0005/0016); leave blank for binary pass/fail.
- **Check version history** — every edit is versioned and viewable per check.

## Suites

A **suite** is a named set of checks bound to one connection + target
([Datasources & checks](datasources-checks.md)):

- **Run target** — one table (SQL) or one file/batch (flat file) per suite.
- **Sharing** — per-suite **view / edit** grants (suite-level is the access model; no
  folder scopes). See [access & auth](#access-auth).
- **Export / import** — move a suite's definition between environments (author on DEV,
  import against PROD) instead of re-clicking checks.

## Running checks

Four ways a suite runs (all the same authz — [feature matrix](../reference/feature-matrix.md#ways-a-suite-runs)):

- **Run now** — with **live per-check progress** and **cancel**.
- **Cron schedule** — 5-field cron + IANA timezone, DST-aware, no backfill
  ([Scheduling](scheduling.md)).
- **Pipeline trigger** — runs automatically when a bound ADF / Airflow / dbt run **succeeds**
  ([Orchestration](orchestration.md)).
- **API / MCP** — `POST /suites/{id}/run`, or the `trigger_suite_run` MCP tool.

## Results, dashboard & history

- **Monitoring dashboard** — **leads with asset-level health** (how many assets are
  monitored, need attention, or have a run in flight — click through to an asset or the
  full [Assets](../reference/feature-matrix.md#assets-lineage-datasources) view), then a
  severity-weighted **health score**, pass rate, run counts, average duration, per-day
  trends, and per-suite performance. Assets are the primary lens (ADR 0034 nav inversion):
  the sidebar leads with **Assets** above Suites, and every suite/run links back to its asset.
- **Warehouse inventory sync** (ADR 0040) — an opt-in per-connection toggle
  (Snowflake / Unity Catalog connection forms) that enumerates every table the
  connection can see into the asset view daily, so a table with no suite, run,
  or lineage signal is **visible as unmonitored** instead of invisible. Synced
  tables age out automatically once dropped from the warehouse. Unity Catalog
  needs `SELECT` on `system.information_schema` for the connection's principal.
- **Asset DQ scorecard** — per-dimension coverage and score on the asset page: which
  quality dimensions have checks, how they scored on the latest run, and — the part
  worth acting on — which dimensions have **no checks at all**. Three states are kept
  distinct on purpose: a score, "no signal" (checks exist but none evaluated), and
  "not covered" (no checks) — an asset nobody watches never shows a green tick. The
  numbers are workspace-wide, so two people looking at one asset see one verdict.
- **Asset browse** — two lenses over the same assets: a **By source** drill-down
  (datasource → database/catalog → schema → table, with env + health on each leaf) and a
  flat, searchable **All assets** table (real server-side paging, 50 rows/page). Both open
  the asset detail (health across suites, lineage, incidents). A workspace bigger than the
  tree view can comfortably render (2000 assets) shows an explicit "showing N of M" note
  rather than a silently partial tree — switch to **All assets** to page through the rest.
- **Datasources read as names, not connection strings** — an asset is *identified* by its
  OpenLineage namespace (the physical location, so DataQ's identifiers join byte-for-byte
  with dbt/Spark emissions), but it is *displayed* as a human label: `Snowflake · ACCT`,
  `ADLS · account/container`, `iceberg_catalog`. The raw namespace stays one hover away
  (and is what the copy button copies). Note the label is deliberately **not** the
  connection name: several connections can point at one datasource instance and all
  resolve to the same namespace — which is exactly what makes the same table reached two
  ways a single asset.
- **Lineage graph** — an asset's provenance and blast radius render as one
  left-to-right graph (upstream → this asset → downstream), one column per hop, with
  clickable nodes. Depth ≥2 comes from the blast-radius BFS, which also returns the
  real edges — so a two-hop consumer is drawn hanging off the node it actually
  descends from, not off the asset it never touches.
- **Two health axes on an asset** — *Connection health* ("could DataQ reach and execute
  against the datasource?" — a failed run, or a check the datasource threw on) and
  *Data-quality health* (the severity-weighted verdict of its suites). They are derived
  and shown separately, so a datasource DataQ cannot connect to never masquerades as a
  data failure — and a run that evaluated nothing reads **No data**, never a green
  "passing". Operational `error`/`skip` results feed only the connection axis. Both are
  read off the runs already recorded — there is no connection-probe polling loop.
- **Results page** — every run with its per-check outcomes; drill into a run for
  observed-vs-expected values and **redacted failing-row samples**.
- **Severity + operational statuses** — warn / fail / critical for data outcomes, plus
  `error` (evaluation threw) and `skip` (precondition unmet) kept distinct from failures.
- **Failure reasons** — a run that fails to execute shows a redaction-safe reason
  (config / connectivity / permission), not a bare "failed".
- **Metric trend view** — a per-check history chart of `metric_value` over time, with
  threshold bands and (for `anomaly` checks) a rolling z-score baseline overlay; the
  numeric history is kept for trend/baseline even after samples are purged by the
  retention sweep.
- **PDF report export** — a one-click, zero-dependency PDF of a run's results, for sharing
  outside the app.
- **Usable on a phone** — the whole app, not just the dashboard. On a narrow viewport the
  sidebar becomes an overlay drawer (it never squeezes the page), tables scroll inside
  their own container rather than the page, the lineage graph scrolls inside its card, and
  the panels that grant access or edit a suite reflow so their controls stay on screen.

## Orchestration (monitor + trigger only)

DataQ **observes** pipelines and triggers suites on success — it never runs them, and these
are **not** datasources ([Orchestration](orchestration.md)):

- **Azure Data Factory** — Azure Monitor alert → webhook (+ 10-min poll).
- **Apache Airflow** — DAG callback → HMAC webhook (+ 10-min poll).
- **dbt** — post-build callback → HMAC webhook (+ 10-min `run_results.json` artifact poll).

All three sit behind one provider interface; **trigger bindings** map
`(provider, pipeline/DAG/job, env) → suite`.

## Alerting

Delivered when a run breaches its threshold ([Notifications & alerting](notifications.md)):

- **Channels** — Microsoft Teams, Slack, email and generic HMAC webhooks, defined once by an
  admin and linked per suite, with deep links + expected-vs-observed context.
- **Threshold** — per suite: fail-only / warn-and-worse (default) / always.
- **Severity routing** — urgency scales with severity; critical escalates.
- **Dedup** — you hear about a breakage once (and again on escalation); a clean run resets.
- **Snooze** — silence a known-broken check for N hours (history + re-fire preserved).
- **Per-suite config** — channels + threshold + recipients set per suite.
- **Connection poll-health alerts** — when an orchestration connection's poll keeps failing
  (an expired credential, a moved orchestrator), DataQ stops ingesting pipeline runs and
  firing the suites bound to them. After 3 consecutive failures it alerts — once on the way
  down, once on recovery — with the classified reason.

## Data protection

- **Column-aware redaction** — failing-row samples are the one place results can carry PII;
  a suite **column policy** (auto-derived by the classifier or set by hand) keeps
  non-sensitive breaches debuggable while PII columns stay masked.
- **PII-minimising retention** — samples are purged after the retention window; the
  aggregatable metric history survives.

## Access & auth

- **Two sign-in modes** — **email OTP** (a one-time code, no Identity Provider
  required — the local/eval default, ADR 0032) and **OIDC SSO**
  (provider-neutral, Azure AD and AWS Cognito both validated in prod). Exactly one is active per deployment.
- **Personal access tokens (PATs)** — `dq_live_` tokens for headless / AI-client use, same
  authz as the user, on REST **and** `/mcp` ([API keys](api-keys.md), ADR 0026).
- **Suite sharing** — per-suite view / edit grants, capped by workspace role (below).
- **Workspace roles — Admin / Member / Viewer** — a stored, in-app-managed role per user
  (ADR 0033), not just an env allowlist. Admin has workspace-wide visibility over every
  suite/result/schedule and is the only role that can create, edit, delete, re-auth, or
  test a **connection** — connections are shared infrastructure holding credentials, so
  Members consume them but don't manage them. Member can create/own suites and receive
  `edit` shares; Viewer is capped at `view` everywhere, including on any share. Roles are
  managed under [`/admin/members`](admin.md) by an existing Admin; `WORKSPACE_ADMIN_EMAILS` still works as a
  bootstrap/break-glass allowlist for a fresh or locked-out workspace.

## AI-assisted authoring and root-cause analysis

An optional, off-by-default LLM integration (bring-your-own Anthropic or OpenAI-compatible
provider + credential, admin-configured) powers three features: **NL→SQL generation** for
the custom-SQL check editor, **curated check suggestions** driven by a live column profile
constrained to a vetted expectation vocabulary, and a **root-cause narrative** for a failed
check — ranked hypotheses citing real evidence-card layers, plus what the evidence snapshot
could *not* see. SQL generation sends column names and counts; check suggestions add masked profile
statistics, including each non-sensitive column's most frequent values; the RCA narrative additionally sends the triggering check's own
observed value, routed through the same column-policy/warehouse-tag redaction floor every
results surface applies. Raw sample rows never leave the deployment on any path. See
[AI features](ai-features.md) for a walkthrough with real output, and
[Security & data handling](../security/overview.md) for the full transfer-vector accounting.

## AI assistants (MCP)

A curated **48-tool MCP server** at `/mcp` lets Claude / Copilot / Cursor list suites, read
results, checks, runs, connections, schedules, trigger bindings and notification config,
trigger and cancel runs, poll status, add/update/delete/snooze checks, dry-run a check preview,
manage schedules and trigger bindings end-to-end (create / update / delete), import a suite
and set its run target, read and restore a check's version history, read and set a suite's
column policy, browse **assets** (the tables DataQ monitors, with health and lineage) and
**incidents** (what is broken right now, with the evidence — acknowledge and resolve them,
and surface near-miss triggers that silently never fire), list a target's columns, profile
columns, suggest a PII policy, test a connection, read a curated docs page, and read the
health score and pipeline status — in natural language, with the same per-suite authz as the
UI. The 48 split three ways: **25 read-only, 18 that change state**, and **5 that persist
nothing but open a live datasource connection with stored credentials** and so are gated like
writes ([AI assistants](mcp-setup.md), ADR 0008 + its Tier 1–3B amendments).

## Observability

Structured JSON logs with `request_id` correlation (FastAPI → Celery → GX), PII redaction at
the logger, and **OpenTelemetry** traces + logs — exported to Azure Application Insights
and/or a generic OTLP endpoint ([Observability](../operate/observability.md)).

## Deployment & portability

Runs live today on **both Azure Container Apps and AWS ECS Fargate** — two independent,
equally-real deployments (API + worker + frontend; the frontend is the sole public
surface on each). Every cloud dependency sits behind a seam — OIDC auth, the secret
store, observability export, and the orchestration providers are each swappable behind a
provider-agnostic interface, which is what let AWS ship as a second target with zero app
code differing between clouds ([Architecture](../architecture/overview.md), [deployment
parity](../operate/deployment-parity.md)).