# Notifications & alerting

DataQ alerts on run outcomes over **Microsoft Teams**, **Slack**, and **email** — all
behind one `ResultPublisher` seam, so every channel gets the same severity-aware
behaviour. Alerts fire from the worker as soon as a run reaches a terminal state.

## Channels

Alert destinations are **channels**: a Teams webhook, a Slack webhook, an email recipient
list or a generic HMAC-signed webhook, each defined **once by a workspace Admin** under
**Admin → Settings → Notification channels** and then linked to any number of suites. A
suite's own panel offers only the channels that exist — nobody types a webhook URL into a
suite. Webhook URLs are validated against a **per-channel** host allow-list (Teams:
`webhook.office.com` / `logic.azure.com`; Slack: `hooks.slack.com`) so a typo cannot
exfiltrate alerts to an arbitrary endpoint, and they are stored in the secret store, never
echoed back.

Workspace-level defaults still exist, enabled by environment configuration
(`TEAMS_WEBHOOK_SECRET_NAME`, `SLACK_WEBHOOK_SECRET_NAME`, `EMAIL_*` — see the
[env-var reference](https://github.com/TheurgicDuke771/DataQ/blob/main/.env.app.example)).
A suite with no channel linked alerts through those; a suite with channels linked alerts
through **both** — delivery is additive, not either/or.

**Legacy inline destinations.** Before channels existed a suite could carry its own Teams
or Slack webhook or recipient list. Those keep delivering, and the suite's panel shows them
as a separate *Legacy inline destinations* card with a **Clear** per entry so an editor can
move the suite onto a channel and retire the override. Setting a new inline destination
is not possible any more — not from the app and not from the API, for anyone: the request
is refused with the field named. A suite's destinations are the channels, full stop.

## Per-suite configuration

Open a suite → **Notifications** panel:

- **Send alerts for this suite** — on/off.
- **Alert threshold** — `On fail / critical` · `On warn and worse` (default) · `Always
  (every run)`.
- **Channels** — the admin-configured channels this suite alerts through; pick from the
  list. Viewers see the linked names only.

## Severity-aware routing

The run's **worst severity** decides how loudly the alert lands: `warn` renders quiet,
`fail` standard, and `critical` escalates (channel mention on Teams). A run that
**failed to execute** (the datasource was unreachable, the adapter raised) always
alerts regardless of the suite's threshold — an operational failure is never filtered
as "no warn-level breach".

## Dedup — first failure, not every run

A broken check on a 15-minute schedule would otherwise page you 96 times a day. DataQ
compares each run's failing checks to the suite's **previous terminal run** and alerts
only when something got **worse**: a check newly failing, or escalating severity
(warn → fail → critical). A clean run resets the baseline, so the *next* regression
re-fires. No configuration needed.

## Snooze / suppression

Snooze a specific check's alerts for N hours from the suite's check list (e.g. during a
known upstream incident). A run alerts only if at least one **un-snoozed** check is
failing; when every failing check is snoozed, the alert is suppressed. Snoozes expire
automatically.

## Connection poll-health alerts

Run alerts tell you a **check** broke. This one tells you the **pipe** broke.

An orchestration connection (ADF / Airflow / dbt) is polled every 10 minutes. When that
poll starts failing — an expired credential, a revoked token, an orchestrator that moved
— DataQ stops ingesting pipeline runs, stops firing the suites bound to them, and stops
refreshing any lineage the connection feeds. Nothing is *failing*; things are simply not
*happening*, which is far easier to miss. Prod lineage was dark for six days on exactly
this before the alert existed.

So after **3 consecutive failed polls** (~30 minutes — enough to ride out a restarting
orchestrator or a transient 502), DataQ pushes an alert through the same channels as run
alerts, carrying the connection, the classified reason, and how long it has been down.

- **It fires on the crossing, and only the crossing.** A connection dead for a week
  alerts once, not a thousand times — an alert you have to mute is an alert that stops
  working.
- **Recovery is signalled too**, so the loop closes without you going to look.
- **The reason is classified, never the raw error** (`auth_failed`, `not_found`, …). The
  real prod incident above carried the SAS token inside its exception message, and an
  alert is the one place that string would leave DataQ.
- No per-suite config applies — a connection has no suite, so these go to the
  **workspace** channel (`TEAMS_WEBHOOK_SECRET_NAME` / `SLACK_WEBHOOK_SECRET_NAME` /
  `EMAIL_TO`).

Tune with `ORCHESTRATION_POLL_FAILURE_ALERT_THRESHOLD` (default `3`; `0` disables the
push). Disabling the push does **not** blind the UI: the connections list still badges a
failing poll with its failure count, and the lineage panel still warns rather than
showing a confident empty graph.

## Troubleshooting

| Symptom | Check |
|---|---|
| No alert on a failing run | Suite panel: enabled? threshold covers the severity? Dedup: did the *same* checks already fail in the previous run? All failing checks snoozed? |
| Teams/Slack alert rejected | Webhook host must be on the allowed-hosts list; the URL secret must exist in the secret store. |
| Alert on every run wanted | Set the suite's threshold to **Always (every run)** — dedup still applies to failures, but clean runs report too. |