# Getting started Three tracks, matched to why you're here: - **Production and enterprise teams** → deploy on a cloud platform with a reference stack: Azure Container Apps (primary) or AWS ECS Fargate. See [Production deployment](../operate/deployment.md). This is the recommended way to run DataQ for a team: managed Postgres and Redis, a cloud secret store, your identity provider, and the frontend as the only public surface. - **Evaluate or try it locally** → pull the prebuilt images (below). One Docker Compose file on your machine, no cloud account. - **Develop / contribute** → build from source with `scripts/setup.sh` ([further down](#develop-from-source)). ## Evaluate locally from prebuilt images ![After sign-in: the Connections page, where the first thing you add is a datasource](../assets/screenshots/connections-list.png){ .screenshot } **Prerequisite:** Docker (Compose v2). No source checkout, no conda, no Node, no Azure tenant. ```bash curl -O https://raw.githubusercontent.com/TheurgicDuke771/DataQ/main/docker-compose.ghcr.yml export OPENBAO_TOKEN=$(openssl rand -hex 16) # root token for the bundled vault export DATAQ_SIGNIN_EMAIL=you@example.com # the address allowed to sign in docker compose -f docker-compose.ghcr.yml up ``` This pulls the published images from GHCR and brings up Postgres + Redis + the API + Celery worker + the UI + a local mail catcher, runs migrations, and seeds demo data. Open **`https://localhost:3000`**, type the address you exported, and read the 6-digit code in the bundled inbox at **`http://localhost:8025`**. - **Sign-in works with no SMTP relay.** The stack bundles its own mailbox ([Mailpit](https://mailpit.axllent.org), MIT), so DataQ's real mailer runs its real SMTP path and the message lands in a web inbox on your machine instead of the internet. No mailbox has to exist; nothing leaves the host. - **Multi-arch:** the images are `linux/amd64` + `linux/arm64`, so Apple Silicon runs native (not emulated). - **HTTPS with a local certificate.** On first start the stack generates its own certificate authority (*DataQ Local Dev CA*) and a certificate for `localhost`, and the UI serves HTTPS with it, so the session cookie is `Secure` as it is in production. Your browser does not know this CA, so it warns once; accept the warning to continue, or trust the CA on this machine and the warning goes away: ```bash curl -O https://raw.githubusercontent.com/TheurgicDuke771/DataQ/main/scripts/local-ca.sh bash local-ca.sh install # asks for sudo; `uninstall` reverses it, `status` reports ``` It adds the CA to the macOS System keychain, or to the system CA store on Debian/Ubuntu and Fedora/RHEL, and to Firefox's own store when the NSS `certutil` tool is installed (otherwise it prints the manual step; on Windows it prints the `certutil` command). It trusts only a certificate named *DataQ Local Dev CA*, and if you decline `sudo` the stack keeps working with the warning. Run `uninstall` before `down -v`, or the old CA stays trusted after its stack is gone. To point a single client at the CA instead (`curl`, the CLI, an MCP client), copy the certificate out: ```bash docker compose -f docker-compose.ghcr.yml cp frontend:/certs/ca.pem dataq-local-ca.pem curl --cacert dataq-local-ca.pem https://localhost:3000/healthz ``` The CA's private key stays in a volume that only the generator mounts, and the CA survives restarts; `down -v` removes it. The `localhost` certificate lasts a year and is renewed when the stack is started within 30 days of its expiry; the same goes for the certificates Postgres, Redis, the mail catcher, the vault and the API hold. Containers that were already running keep the old ones, so after a renewal restart the stack (`docker compose -f docker-compose.ghcr.yml restart`). Plain `http://localhost:3000` redirects to HTTPS. The inbox on `:8025` stays plain HTTP on loopback. - **TLS inside the stack too.** The API, worker and scheduler reach Postgres with `sslmode=verify-full`, Redis over `rediss://` (its plain port is off), and the mail catcher over STARTTLS, each verified against the same local CA, and each server holds a certificate for its own hostname only, with a key no other container can read. The vault is reached over HTTPS and the UI's proxy talks to the API over TLS, both verified the same way, and Postgres refuses a network client that is not on TLS. These are the client code paths a managed database, cache, vault and mail relay use in production. Two things stay plain by design: the inbox page on `:8025`, and the demo profile's sample warehouse, which stands in for a warehouse of your own (the connection's TLS setting is the control there). - **Check it yourself.** `scripts/local-smoke.sh` (fetch it the same way as the compose file, and run it beside it) smokes the running stack the way a deployment is smoked after a roll: the certificate verifies against the stack's CA, plain HTTP redirects, the API and MCP refuse an anonymous caller, only the UI and the inbox have a host port, a real sign-in, read and sign-out work with a `Secure` cookie, Postgres and Redis are on TLS, and a burst is rate-limited. It exits non-zero if any check fails. - **One way in, as in production:** the UI on `:3000` is the only published surface. The API has no host port; it is reached through the UI at `https://localhost:3000/api`, and MCP clients connect to `https://localhost:3000/mcp/`. The database, Redis and the vault are reachable only inside the compose network. The interactive API page is off, as it is on a production deployment; the [API reference](../reference/rest-api.md) is published with these docs. - **Loopback-only:** both published ports (`:3000` and the inbox on `:8025`) bind to `127.0.0.1` — the stack is reachable from your own machine but never the LAN. That matters more than usual for `:8025`, which serves live sign-in codes to anyone who can reach it. **Not for production** — a real deploy uses the OpenTofu stack (`deploy/terraform/azure`, ADR 0024). - **Try it on real data:** add `--profile demo` to the `up` command. It also starts a sample warehouse (PostgreSQL) holding a small shop dataset with deliberate defects, and connects DataQ to it: a connection named *Demo warehouse (PostgreSQL)*, two suites (*Shop orders* and *Shop customers*) whose checks cover value rules, custom SQL, freshness, volume, an aggregate statistic, schema drift and anomaly detection, and an hourly schedule. Both suites run once as the stack starts, so your first sign-in shows real results: several checks fail on purpose, and the anomaly check waits for more history before it scores. No new orders arrive while the stack runs, so the freshness check warns after a day and fails after three, which is the monitor doing its job; restarting the stack makes the data current again. The sample warehouse has no host port and is reachable only inside the compose network. - **Pin a release** instead of the moving stable tags: `DATAQ_BACKEND_TAG=vX.Y.Z DATAQ_FRONTEND_TAG=vX.Y.Z docker compose -f docker-compose.ghcr.yml up`. Take the compose file from the same release (replace `main` with `vX.Y.Z` in the download URL) and pin both images: the file and the images change together, and a newer file with older images does not start. - **Your data lives in a directory.** The database, the sample warehouse and the vault write to `./dataq-data` beside the compose file; set `DATAQ_DATA_DIR` to an absolute path to keep it elsewhere. Connections, suites, results and the credentials you enter survive a restart, `down -v` and a Docker reset, and the directory can be backed up or moved while the stack is stopped. The vault unseals itself from a key file in that directory, so there is no unseal step, and `OPENBAO_TOKEN` may change between starts: the new token replaces the old one. **That convenience has a cost:** the key, the vault's root token and the data sit together, so anyone who can read the directory can read every stored credential. Treat it like a password file; a production deployment uses a managed secret store instead. - **Reset:** `docker compose -f docker-compose.ghcr.yml --profile demo down -v`, then delete the data directory (`rm -rf dataq-data`). `down -v` alone removes only the certificates. On Linux the files belong to the containers' users, so deleting, copying or moving the directory needs `sudo`. - Omitting `DATAQ_SIGNIN_EMAIL` stops the stack and says so — there is no no-sign-in default to fall into. ## Choosing an auth mode DataQ ships **two** ways to log a human in: | Mode | For | You must bring | Sign-in looks like | |---|---|---|---| | **`otp`** ← *the default* | A small team, or anyone evaluating locally | an **SMTP relay** in production — **nothing locally**, the compose stacks bundle a mailbox | you type your address, DataQ emails a 6-digit code, you type it back | | **`oidc`** | An organisation that already has an IdP | an **OIDC app registration** (Azure AD, Okta, Keycloak, Cognito, …) | your normal SSO redirect | **`otp` is what you boot into.** Both compose stacks used to start with no sign-in at all, which meant anyone who could reach the port administered a tool that stores warehouse credentials — and an eval stack's defaults are the security posture people actually run (the same reasoning that moved the default secret store off the plaintext one). What made `otp` unusable as a default was the SMTP relay it used to require; the bundled catcher removes that, so the only thing left to supply is *who* is allowed in — one variable, `DATAQ_SIGNIN_EMAIL`, which `scripts/setup.sh` asks for. Setting it empty is the downgrade; leaving it unset stops the stack rather than picking for you. The mode is **never inferred**. Anything unrecognised or half-configured renders an "authentication not configured" banner rather than quietly falling back to something permissive — and the backend refuses to boot on a half-configured OTP block rather than coming up unable to log anybody in. **Two selectors, set them together.** The frontend's `DATAQ_AUTH_MODE` (injected at runtime by nginx — ADR 0028) and the backend's own mode (inferred from its `AZURE_*` or `AUTH_EMAIL_*` settings) are separate contracts, and **neither can derive the other**. Backend OTP on with the frontend on `oidc` shows an SSO flow against an IdP that isn't configured; the reverse shows a code form whose endpoints 503. ### `otp` — email one-time codes (ADR [0032](../adr/0032-email-otp-signin.md)) The default, and the rung for teams that have email but no IdP. **Locally there is nothing to configure but your address.** Both compose stacks run a [Mailpit](https://mailpit.axllent.org) container (MIT) as the mailbox: DataQ performs a real SMTP submission against it — the same `connect → AUTH → send` path a production relay gets — and the message appears in a web inbox at `http://localhost:8025`. On the prebuilt-image stack the mailer uses STARTTLS and verifies the catcher against the stack's local CA. On the from-source stack it runs with `AUTH_EMAIL_TLS_MODE=none`, the plaintext downgrade `none` exists for, logged loudly on every send: it is correct against a container you started on your own machine and wrong against anything else. **In production you still bring your own relay** — bundling an outbound mailer is a deliberate non-goal (direct-to-MX from an arbitrary self-hosted IP gets sign-in codes spam-foldered, and a vendor relay would leak sign-in metadata to a third party). Frontend: ``` DATAQ_AUTH_MODE=otp ``` Backend — all four mailer values plus **at least one allowlist entry** (there is no open registration; DataQ holds failing-row samples, which are PII): ``` AUTH_EMAIL_SMTP_HOST=smtp.example.com AUTH_EMAIL_SMTP_PORT=587 AUTH_EMAIL_USERNAME=dataq@example.com AUTH_EMAIL_FROM=dataq@example.com AUTH_EMAIL_PASSWORD_SECRET_NAME=dataq-smtp # the VALUE lives in your secret store AUTH_OTP_ALLOWED_DOMAINS=example.com # and/or AUTH_OTP_ALLOWED_EMAILS=... WORKSPACE_ADMIN_EMAILS=you@example.com # bootstrap: your own address ``` Give this block to **both** the api and the worker: the api validates the request and mints the code, but the worker is what actually sends the email (the send runs off the request path so a slow relay cannot leak who is allow-listed). A worker without it logs `otp_send_task_failed` and no code ever arrives. First sign-in: put your own address in **both** the allowlist and `WORKSPACE_ADMIN_EMAILS`, then sign in to your own mailbox. There is no seeded password to rotate. The sign-in form is deliberately credential-only — no sign-up step, no name field, per ADR 0032 — so a first-time OTP user's row has no name yet. The app offers a one-time, skippable prompt for one right after that first sign-in. Skipping it is fine — the name is cosmetic, never an authz input — and it (or any later change) is always available from **Profile**, for every auth mode, not just `otp`. Before letting anyone else in, **prove the mailer works**: as a workspace admin, `POST /api/v1/admin/auth-email/test` sends a real message to your own address and, on failure, names the stage that broke (`connect` / `tls` / `auth` / `send`). Far better to find a bad relay here than at a teammate's first sign-in, when the only symptom is a code that never arrives. It exercises the **api's** transport block synchronously; the worker sends the real codes with its own copy of that block, so if the pre-flight passes and codes still never arrive, read the worker log for `otp_send_task_failed`. One consequence worth knowing up front: with no IdP there is no bearer token for `/mcp` to validate, so in this mode **a PAT (`dq_live_…`) is the only credential MCP accepts**. AI clients need [an API key](../guides/api-keys.md); a session cookie will not do. Read [Security & data handling](../security/overview.md) before enabling it — under OTP **the mailbox is the credential**, so mailbox compromise is account compromise. The session is an HttpOnly cookie with a fixed 24 h life and no refresh token; signing in again is the refresh. Codes expire in 10 minutes, are single-use, and allow 5 attempts. The mailer defaults to **SMTP + STARTTLS on 587**, verified against the system trust store. Two more transports are available via `AUTH_EMAIL_TLS_MODE`: `implicit` for a submission relay on **:465** (SMTPS), and `none` — a deliberate plaintext downgrade, logged loudly on every send, for a loopback relay or a throwaway test rig only. An internal relay signed by a **private CA** doesn't need the container's whole-process trust store touched: point `AUTH_EMAIL_CA_BUNDLE` at its PEM and only the mailer's own connection trusts it. There is no option to skip certificate verification — the bundle is the answer to "my relay's cert isn't publicly trusted", not a way around checking it. ### `oidc` — self-hosting with your own identity provider The compose eval runs the frontend with `DATAQ_AUTH_MODE=otp` against the bundled mailbox, which is fine for evaluation and for a team on a trusted network, but is not an IdP. The frontend is **one generic image** whose auth config is injected at **runtime** (nginx serves `/config.js` from the `DATAQ_AUTH_*` env), so the same image goes from eval to real SSO with **no rebuild** (ADR 0028): - As-pulled with no auth env it shows an "authentication not configured" banner. For real SSO, run the same image with `DATAQ_AUTH_MODE=oidc` + `DATAQ_AUTH_AUTHORITY` (e.g. `https://login.microsoftonline.com//v2.0`) + `DATAQ_AUTH_CLIENT_ID` (your SPA app registration) + `DATAQ_AUTH_API_SCOPE` (`api:///`), and run the **backend** with the matching `AZURE_*` settings. For a non-Azure IdP whose scope vocabulary differs, the optional `DATAQ_AUTH_SCOPE` replaces the requested scope string entirely — AWS Cognito rejects the default list's `offline_access`, so a Cognito deployment sets `DATAQ_AUTH_SCOPE="openid email profile"` (with the backend's `OIDC_ISSUER` + `OIDC_AUDIENCE` instead of `AZURE_*`). - The frontend reverse-proxies `/api` + `/mcp` to the backend at `DATAQ_API_UPSTREAM`, resolving it via the DNS server it **detects from the container's `/etc/resolv.conf`** at startup — so the one image works on Docker's embedded DNS (Compose) **and** cluster DNS (Kubernetes / Container Apps) without a rebuild. - **MCP** (`/mcp`) is **fail-closed**: it needs a working sign-in configuration. Under `oidc` it validates the bearer token; under the eval stack's `otp` it is served with a **PAT (`dq_live_…`) as its only credential** (there is no IdP to issue a token, and a session cookie is deliberately rejected) — see [API keys](../guides/api-keys.md). ## Develop from source **Prerequisites:** **conda** (the backend uses a conda env — not venv/poetry), **Docker** + Docker Compose, and **Node 24+ / pnpm 9+** for the frontend. ```bash git clone https://github.com/TheurgicDuke771/DataQ.git cd DataQ ./scripts/setup.sh # creates the `dataq` conda env, installs pre-commit, pulls # images, runs DB migrations, seeds dev data, writes a local .env conda activate dataq docker-compose up # Postgres + Redis + FastAPI (:8000) + React (:3000) + Celery # + Mailpit (:8025), the local inbox for sign-in codes ``` `setup.sh` also offers to trust the local HTTPS certificate the prebuilt-image stack uses (it asks for `sudo`; `DATAQ_TRUST_LOCAL_CA=0` skips it, and declining is harmless). `setup.sh` asks **which address may sign in** and writes the answer to your gitignored `.env` as `DATAQ_SIGNIN_EMAIL`. That address is allow-listed *and* made a workspace admin, and you sign in by reading the code at `http://localhost:8025` — no cloud account or IdP, no SMTP relay. You can change the address later by editing that one variable and re-running `docker compose up`. **Which file wins.** The compose stack sets the whole `AUTH_EMAIL_*` block in its own `environment:`, which beats `env_file:` unconditionally — so for the api/worker containers `.env.app` is **ignored for these keys in every state**, including when the switch is empty. Putting a real relay in `.env.app` and clearing the switch does not select it; the stack refuses to start until you choose a mode. To point the **compose** stack at a real relay, keep `DATAQ_SIGNIN_EMAIL` set and override the same key names in the **root `.env`**: ``` DATAQ_SIGNIN_EMAIL=you@example.com AUTH_EMAIL_SMTP_HOST=smtp.example.com # the from-source stack's catcher default is `none` AUTH_EMAIL_TLS_MODE=starttls # prebuilt-image stack only (see below); nothing may follow the `=`, not even a comment AUTH_EMAIL_CA_BUNDLE= ``` On the **prebuilt-image stack** the catcher is reached over STARTTLS and verified against the stack's local CA (`AUTH_EMAIL_CA_BUNDLE=/certs/ca.pem`), and that bundle is used *instead of* the usual certificate authorities. A public relay's certificate is not signed by the local CA, so set `AUTH_EMAIL_CA_BUNDLE=` (empty) alongside the host, or every sign-in mail fails certificate verification. #### Sending to real mailboxes from the prebuilt-image stack To have sign-in codes arrive in Gmail, Outlook or any other real mailbox, point the stack at an SMTP relay you have an account on. Put this in a `.env` file beside the compose file (or export the variables) and start the stack: ``` DATAQ_SIGNIN_EMAIL=you@example.com # Microsoft 365 / Outlook: smtp.office365.com AUTH_EMAIL_SMTP_HOST=smtp.gmail.com AUTH_EMAIL_SMTP_PORT=587 AUTH_EMAIL_USERNAME=you@example.com AUTH_EMAIL_FROM=you@example.com # empty, so the relay is checked against the public CAs. Nothing may follow the `=`: # a comment on this line would become the value. AUTH_EMAIL_CA_BUNDLE= # needed on the first start only DATAQ_SMTP_PASSWORD= ``` - **The password is handed over once.** `DATAQ_SMTP_PASSWORD` is read by a start-up step that writes it to the stack's vault and by nothing else; the API never has it in its environment. The vault keeps it across restarts, so remove the line after the first start. Supply it again only to change it. - **Gmail** needs an app password (an account with 2-step verification, then *App passwords*), not your sign-in password. **Microsoft 365 / Outlook** works where SMTP AUTH is enabled for the mailbox; many organisations turn it off. - The relay will only send **from** an address the account owns, so `AUTH_EMAIL_FROM` is normally the same as the username. - With a real relay configured the bundled inbox on `:8025` receives nothing. - If the code does not arrive, an Admin can use **Admin → Settings → Send test email**, which reports the relay's answer. With no password stored the start-up step says so in its log (`docker compose logs otp-mail-secret`) rather than inventing one. **Alert emails** use a separate mailer. Turn it on the same way: ``` EMAIL_SMTP_HOST=smtp.gmail.com EMAIL_USERNAME=you@example.com # comma-separated recipients EMAIL_TO=team@example.com # first start only DATAQ_ALERT_SMTP_PASSWORD= ``` `.env.app` is still the file for **host-side dev** (uvicorn on your own machine, which reads it directly) — point `AUTH_EMAIL_SMTP_HOST` at `localhost` if you want the catcher there too (Mailpit publishes `127.0.0.1:1025`). ## Configuration All runtime config is environment variables read by the backend's `Settings`. The **complete, commented reference** is [`.env.app.example`](https://github.com/TheurgicDuke771/DataQ/blob/main/.env.app.example) — copy it to `.env.app` (gitignored) and adjust. Never commit secrets; `scripts/setup.sh` generates local-dev credentials on first run. ## Running tests The DB-backed tests need a real Postgres (`gen_random_uuid()`/jsonb, which SQLite can't host); notification tests also need Redis. The suite is ~4,800 tests and growing — exact counts below drift with every PR, so treat them as shapes, not contracts. There are three ways to run: ```bash # 1. One command — brings up the compose Postgres + Redis, provisions a dedicated # `dataq_test` DB, runs the whole suite incl. the real-broker E2E (this is what # CI runs; the script sets DATAQ_E2E=1 for you — see the note below): scripts/test-backend.sh # → everything passes scripts/test-backend.sh -k notifications # extra pytest args pass through # 2. Plain pytest — incl. the VS Code / PyCharm test runner. With the compose # services up, conftest AUTO-DETECTS the local Postgres (from .env, on dataq_test) # so the DB tests run — no env vars, no wrapper. The one real-broker E2E test # still skips here (see the note below): docker compose up -d postgres redis conda run -n dataq python -m pytest backend/tests # → all pass, 1 skipped # 3. No services at all — the DB-backed tests skip, the pure-unit suite still runs green: conda run -n dataq python -m pytest backend/tests # → passes with the DB tests skipped # Frontend: cd frontend && pnpm test ``` > The auto-detect is safe: it only kicks in when `TEST_DATABASE_URL` is unset, targets a > **separate `dataq_test` database** (your dev DB + seed data are untouched), and is a > no-op in CI (which sets `TEST_DATABASE_URL` explicitly). The **one** test that needs an > extra, EXPLICIT opt-in is the real-infra E2E (`test_probe_e2e`) — it spins up a live > Celery worker + broker and does real commits/TRUNCATEs, so it requires `DATAQ_E2E=1` > alongside `DATABASE_URL` + `REDIS_URL` (not merely the latter two): conftest's > auto-detect above means `DATABASE_URL` alone is no longer a reliably deliberate signal, > so a fourth, conftest-never-sets-it-for-you flag keeps this one test a conscious choice. > `scripts/test-backend.sh` and CI both set it; a bare `pytest` does not. Before pushing, run the same gate CI does: Ruff, Black `--check`, mypy, Bandit, pytest (backend) and ESLint, Prettier `--check`, Vitest (frontend). See the **[Contributing guide](../architecture/contributing.md)**.