Skip to content

ADR 0007 — Airflow callback model (HMAC-signed webhook + polling fallback)

  • Status: Accepted
  • Date: 2026-05-30
  • Deciders: @TheurgicDuke771

Context

Per ADR 0004, Apache Airflow DAG run events reach DataQ via DAG-level on_success_callback / on_failure_callback hooks that POST to POST /api/v1/orchestration/events/airflow. We must authenticate these requests so a third party cannot forge DAG events (successful-completion events can trigger suite runs via trigger_bindings).

Unlike Azure Monitor (see ADR 0006), the Airflow callback is code we author as a copy-paste snippet for the user's DAGs, so it can set custom headers and compute a signature. This permits a stronger header-based HMAC scheme rather than a secret-in-URL.

Two constraints shape the design:

  • We cannot mutate users' DAGs. Adoption of the callback snippet is voluntary; some users will not add it. A polling fallback is therefore mandatory, not optional.
  • The signing key is a shared symmetric secret that must live in Key Vault on our side and in the user's Airflow environment.

Decision

Authenticate the Airflow webhook with an HMAC-SHA256 signature over the raw request body, carried in an X-DataQ-Signature header. Provide a copy-paste callback snippet; document the Airflow REST polling fallback for non-adopters. Rotate by hard cutover. No request-freshness check in v1.

Authentication

  • The callback computes HMAC-SHA256(signing_key, raw_body), hex-encoded, and sends it as X-DataQ-Signature: sha256=<hex> (GitHub-style prefix for forward compatibility with other digests).
  • The endpoint recomputes the HMAC over the raw, unparsed body and compares constant-time (hmac.compare_digest). Mismatch / missing signature → 401.
  • Because the signature covers the raw body, every field — including a timestamp the snippet includes — is authenticated even though v1 does not act on freshness (leaves the door open to add a replay window later with no protocol change).
  • After successful auth, the endpoint returns 200 for all well-formed events; malformed body → 422.

Callback snippet + polling fallback

  • Ship a copy-paste dataq_airflow_callback helper for users to wire into on_success_callback / on_failure_callback. It reads the signing key from the user's Airflow secret backend / env, never hardcoded.
  • For DAGs that don't adopt the snippet, the Airflow REST API dagRuns polling fallback every 10 minutes (ADR 0004) is the documented path to capture success runs. Webhook is the fast path; polling is the floor.
  • Per ADR 0004, only success events trigger suite runs; failure events alert the user but do not trigger.

Signing-key storage & config

  • Key lives in Key Vault, surfaced as AIRFLOW_WEBHOOK_SIGNING_KEY via the SecretStore abstraction (PR 3b). The same value is configured in the user's Airflow environment.

Rotation — hard cutover

  • One active signing key at a time. Rotation = update the Key Vault key and the user's Airflow-side key in lockstep.
  • A brief mismatch window may drop a callback delivery; acceptable because the 10-minute dagRuns polling fallback recovers missed success runs. No dual-key overlap logic in v1.

Replay / freshness — not enforced in v1

  • v1 validates the HMAC only; it does not reject stale timestamp values.
  • Duplicate / replayed deliveries are neutralised by the idempotent upsert keyed on (provider, provider_run_id) in pipeline_runs — no double-trigger.
  • Freshness enforcement is a documented hardening candidate (the timestamp is already signed), deferred until replay is shown to be a real risk.

Consequences

Positive - Header-based HMAC over the raw body is a standard, well-understood scheme — no secret in the URL (improves on the ADF constraint). - Voluntary adoption is de-risked by the mandatory polling fallback; webhook is a latency optimisation, not a correctness dependency. - Symmetric with ADR 0006 on the operational decisions (hard-cutover rotation, idempotent-upsert dedup) — one mental model for both providers.

Negative - Symmetric shared key means the user's Airflow environment holds signing material; a compromised Airflow could forge events. Bounded to that user's own DAGs in a single-tenant deployment; rotation is the mitigation. - Users who never adopt the snippet get 10-minute worst-case detection latency instead of near-real-time. Documented, and acceptable for the DQ-trigger use case. - Hard cutover can drop deliveries during rotation; accepted because polling backfills. - No replay window; impact bounded to idempotent no-ops (no double-trigger).

Alternatives considered

  • Secret in URL query param (as ADF uses) — rejected for Airflow. Since we author the callback, we can do proper header-based HMAC; no reason to accept the weaker secret-in-URL exposure here.
  • Webhook-only, no polling fallback — rejected. We can't guarantee users add the callback to every DAG; without polling, those runs would be invisible. ADR 0004 mandates the fallback.
  • Dual-key overlap rotation — rejected for v1, same rationale as ADR 0006: the polling fallback covers the cutover gap.
  • Asymmetric signing (user holds a public key) — rejected as over-engineered for single-tenant v1; symmetric HMAC + Key Vault is sufficient.
  • ADR 0004 — the OrchestrationProvider abstraction; AirflowProvider.parse_event consumes the authenticated payload, list_recent_runs drives the polling fallback.
  • ADR 0006 — the ADF sibling; uses secret-in-URL because Azure Monitor webhooks can't set custom headers. Hard-cutover rotation and idempotent-upsert dedup (no freshness check) are shared decisions.
  • Signing-key config (AIRFLOW_WEBHOOK_SIGNING_KEY), the receiver, the dataq_airflow_callback snippet, and the Airflow connection type land in the Week 2 Airflow orchestration PR.