How alerting works. Every endpoint is probed on its own cadence — 30 s for infra,
60–120 s for app paths, up to 30 min for batch-aware freshness gates. A probe that fails its
conditions failure-threshold times in a row (typically 3, so a couple of minutes of
sustained failure — never a single blip) fires a Slack alert to #alerts-notification through
nq-alert-relay, enriched with the exact failed condition, a summary of the captured
response body (via nq-sidecar-probe), and the deploy provenance of what is running
(version · commit · deployer from /.nq) — plus a
View status button that deep-links back to this page (/s/<service>).
Recovery is announced automatically after success-threshold (2) consecutive passes.
Idle-by-design targets — scale-to-zero staging apps and batch jobs — are rendered as
Idle and never alerted on, so anything red below is a real problem.
Active alerts
What these are. Deep and synthetic checks that go beyond a service's /health
tile — they catch the failure modes a green health endpoint hides. Auth canaries: a
deliberately-bad login must return 401 with the sanitized error — a 5xx means the
auth backend is silently down while /health stays green. Data-freshness gates: the rebalancer's and
ingestion's own view of how current market/stock data is — the "Market data isn't ready" 503 incident
class, caught before the 09:35 ET rebalance trips over it. API smoke tests: real request
round-trips (public live-performance GET, analytics-ingest POST) asserting body shape and latency, not just
status. OTLP ingest auth: the telemetry gateway must reject unauthenticated traffic with
401 — "up and correctly refusing" is the healthy state. Deploy-provenance probes: each app's
/.nq sidecar must answer with a non-empty version, proving the provenance mesh
itself is alive even when an app is wedged. Groups are nq-app.yaml blueprint names —
one group = one deploy unit = "what to redeploy". Click any row for full check history, per-condition
results, and captured response bodies.