Skip to content

Observability

Rose currently uses several observability systems:

  • Google Cloud Logging for backend application logs
  • Sentry for error tracking and uptime monitoring (see Sentry Uptime Monitoring)
  • PostHog for product analytics
  • Langfuse for LLM tracing and prompt observability
  • Grafana Cloud for OpenTelemetry metrics (chat latency & per-LLM-call)

Sentry Uptime Monitoring

Sentry polls the backend deep status probe (GET /status on ixsearch_api) to detect outages in any of the widget's runtime dependencies — MongoDB, Redis, Neo4j, Supabase, Azure OpenAI, OpenAI, Cohere, Langfuse.

How the endpoint works

Endpoint Purpose Auth Probes
GET /health Liveness — Cloud Run probe none none
GET /ping Ultra-light readiness none none
GET /ready Cloud Run readiness — used by load balancer none MongoDB ping, Redis ping
GET /status Deep status probe (Sentry uptime) X-Status-Token header MongoDB dbStats, Redis DBSIZE, Neo4j RETURN 1, Supabase REST query, Azure OpenAI /openai/models, OpenAI /v1/models, Cohere reach, Langfuse reach

Response shape (200 all-ok, 503 if any critical service is down):

{
  "status": "ok",
  "version": "1.407.0",
  "environment": "production",
  "public_ip": "34.x.x.x",
  "timestamp": "2026-05-26T17:43:06.796311+00:00",
  "services": {
    "mongodb": {"status": "ok", "latency_ms": 157, "detail": "collections=14"},
    "redis": {"status": "ok", "latency_ms": 70, "detail": "keys=64036"},
    "neo4j": {"status": "ok", "latency_ms": 470, "detail": null},
    "supabase": {"status": "ok", "latency_ms": 90, "detail": null},
    "azure_openai": {"status": "ok", "latency_ms": 200, "detail": null},
    "openai": {"status": "ok", "latency_ms": 1366, "detail": null},
    "cohere": {"status": "ok", "latency_ms": 168, "detail": null},
    "langfuse": {"status": "ok", "latency_ms": 135, "detail": null}
  }
}

Critical services (mongodb, redis, neo4j, supabase, azure_openai, openai) flip the global status to down and trigger HTTP 503. Non-critical (cohere, langfuse) stay HTTP 200 but show up as degraded/down in the body — Sentry can alert on partial regressions via body-match rules.

Code: backend/packages/ixweb/ixweb/routes/health.py.

Authentication

/status is not public — guarded by a shared-secret header so only Sentry can hit it.

Header Form
X-Status-Token: <secret> preferred
Authorization: Bearer <secret> accepted (for Sentry compatibility)

Missing or wrong token → HTTP 404 (not 401) to avoid leaking endpoint existence to scrapers. Comparison uses hmac.compare_digest (timing-safe).

The secret is stored in GCP Secret Manager under the name STATUS_PROBE_TOKEN in project inboundx. The backend fetches it via ixinfra.utils.secret_manager.get_secret, which checks env first then falls back to Secret Manager (cached per process via @lru_cache). Cloud Run's default service account already has roles/secretmanager.secretAccessor at project level, so no per-secret IAM binding is required.

Rotating the token

gcloud secrets versions add STATUS_PROBE_TOKEN \
  --data-file=<(openssl rand -hex 32) \
  --project=inboundx

The backend caches the token via @lru_cache for the lifetime of each worker process. Restart the Cloud Run service (or trigger a new revision) to pick up the new version. Then update the corresponding Sentry monitor's header value.

Configuring a Sentry Uptime Monitor

  1. Read the token out of Secret Manager:
gcloud secrets versions access latest \
  --secret=STATUS_PROBE_TOKEN --project=inboundx

  1. In Sentry: Alerts → Uptime Monitors → Create Monitor.

  2. Fill in:

  3. URL: pick per environment

    • production: https://api.userose.ai/status
    • staging: https://api-staging.userose.ai/status
    • test: https://api-test.userose.ai/status
  4. Method: GET
  5. Interval: 1–5 minutes
  6. Timeout: 10 seconds (each backend probe is capped at 2 s; nine in parallel + network overhead fits comfortably)
  7. Headers:
    • Name: X-Status-Token
    • Value: paste the token from step 1
  8. Expected status code: 200

  9. Body match (optional): "status":"ok" to alert when the endpoint returns 200 but a non-critical dependency is degraded

  10. Alert routing: wire to the same channel as other backend Sentry alerts.

Local testing

The local backend respects the same token. Set it in backend/.env.local:

echo 'STATUS_PROBE_TOKEN=<value-from-secret-manager>' >> backend/.env.local

Restart just dev <environment>, then preview the response in playground at http://localhost:3001/status — paste the same token into the page (stored in localStorage under rose:status_probe_token). Pick the endpoint with the API endpoint selector at the top.

Adding a new probe

In backend/packages/ixweb/ixweb/routes/health.py:

  1. Add _probe_<service>() -> ServiceStatus that wraps the check in asyncio.wait_for(..., timeout=STATUS_PROBE_TIMEOUT_S) and returns ServiceStatus(status, latency_ms, detail).
  2. Add the name to the names list and the coroutine to probes inside _run_status_probes.
  3. If the service is widget-critical, add its name to the CRITICAL_SERVICES set so a failure flips HTTP to 503.
  4. Use _http_auth_probe for endpoints requiring a key, _http_reach_probe for unauthenticated reachability.

OpenTelemetry Metrics (Grafana Cloud)

Backend services emit native OpenTelemetry metrics to Grafana Cloud over OTLP/HTTP. Traces and logs are intentionally not exported: LLM traces go to Langfuse via its own SDK, and infra APM (HTTP request/health spans) is not collected at all.

Superlog and the FastAPI/HTTPX/Requests/logging auto-instrumentors were removed (IX-4127). They were fanning every /health and /api/version span into Langfuse via the shared global TracerProvider, doubling the Langfuse observation bill for zero observability value. Metrics were repointed from Superlog to Grafana Cloud; the browser backoffice OTel bootstrap was deleted (backoffice keeps Sentry + PostHog).

Which service exports what:

  • ixsearch-api — chat latency and per-LLM-call metrics (see Chat Latency & LLM-Call Metrics).
  • knowledge-api / knowledge-content-worker — operation/dispatch/task counters.

The Grafana Basic-auth token lives in Secret Manager under GRAFANA_OTLP_AUTH_BACKEND; the shared reader factory is ixinfra.utils.grafana_otlp.grafana_metric_reader().

Manual start_as_current_span calls are inert

Several services still wrap domain operations in tracer.start_as_current_span (ixknowledge_api/service.py, ixknowledge_api/auth.py, ixknowledge_content_worker/runner.py + knowledge_client.py, ixadmin_api/knowledge_*.py). Since no service sets a TracerProvider any more, these resolve to the no-op tracer and record nothing — they are harmless but produce no telemetry. Do not add new ones expecting spans to appear; delete them opportunistically when touching those files.

Chat Latency & LLM-Call Metrics (Grafana Cloud)

Native OTel metrics emitted by the Website Agent and shipped to Grafana Cloud. They answer "how fast is Rose, end-to-end and per model call?" — latency lives here; token/cost analytics stay in Langfuse.

Measurement tiers

Tier What Where measured
A. Whole-turn request → first/last token, spanning every node (intent → retrieval → answer/redirect/booking) ixchat/chatbot.py (graph stream anchor)
A′. Request-boundary HTTP-in → response done; includes pre-graph preamble ixsearch_api/routes/chat.py
B. Per-LLM-call one record per model invocation, by use case (node) ixchat/nodes/streaming_utils.py + ixllm.metrics.timed_llm_call

Instrument inventory

Names carry the rose_chat_ prefix and no unit suffix — Grafana Cloud's OTLP→Prometheus normalization appends _seconds / _total and the _bucket/_sum/_count series. Explicit second-scale histogram buckets are set via Views in observability.py (default OTel buckets are tuned for counts).

Instrument Type Unit Tier Notes
rose_chat_replies_total Counter 1 A silent-Rose canary — one per reply leaving the graph, by outcome (success/empty/error) and site
rose_chat_inflight UpDownCounter 1 A in-flight chat requests (live concurrency), by site
rose_chat_answer_ttft Histogram s A whole-turn time to first token (streaming only)
rose_chat_answer_duration Histogram s A whole-turn end-to-end, request → last token (TTLT; streaming + non-streaming)
chat.query.duration Histogram s A′ request-boundary end-to-end (both endpoints)
rose_chat_llm_call_duration Histogram s B per model call; _count = call rate
rose_chat_llm_call_ttft Histogram s B per-call first-token latency (streaming calls)
rose_chat_llm_tokens Counter tokens B best-effort token usage (see caveat)

Temporality & per-instance series

All instruments export with cumulative temporality (the OTel default). Do not switch to delta — Grafana Cloud / Mimir's OTLP gateway rejects delta counters/histograms with HTTP 400 (invalid temporality and type combination), which silently drops every metric batch and leaves dashboards empty.

The search API runs 2–3 Cloud Run instances. Each stamps a unique service.instance.id (observability.py:_resource), so every instance is its own Prometheus series — this, not delta, is what keeps cross-instance counters from collapsing into one bouncing series (which would make rate()/increase() read each dip as a counter reset). Consequence: always aggregate across instances in queries — sum(rose_chat_inflight) for the fleet total, sum(rate(rose_chat_replies_total[5m])), sum by (le, …) (rate(..._bucket[5m])) before histogram_quantile.

Tagging plan

Discipline: bounded, low-cardinality only. Never session_id, raw IDs, turn_number, or exception messages. Resource attrs already carry service.name, deployment.environment.name, service.version, vcs.ref.head.revision — do not duplicate env/version on metrics.

Whole-turn (A) attributes — chosen to join with rose_chat_replies_total:

Attribute Example Source
site mayday.fr state["site_name"] (matches the canary's site)
outcome success / empty / error mirrors rose_chat_replies_total
response_node answer_writer / redirect_handler / booking_handler node that produced the answer
streaming true / false streaming vs non-streaming entry point

Per-LLM-call (B) attributes — OTel GenAI semantic conventions:

Attribute Example Source
app.gen_ai.use_case answer_writer, intent_classifier the node / route use case (ResolvedChatHandle.use_case)
gen_ai.request.model gpt-5.4, gpt-4.1-mini route policy model
gen_ai.provider.name openai / azure / cerebras route policy provider
outcome success / error call result
error.type timeout / rate_limited / upstream_5xx only on outcome=error (short, bounded)
token_type input / output rose_chat_llm_tokens only

Dots become _ as Grafana labels (gen_ai_request_model, app_gen_ai_use_case). Per-call metrics are deliberately not tagged with site (avoids site × model × node series blowup) — site-level latency lives in tier A.

Why call-site instrumentation (not a callback handler)

A LangChain callback handler is not used for per-call metrics because:

  1. Several aux nodes pass config={"callbacks": []} (a Langfuse Omit-bug dodge) — a graph-config handler would be stripped on exactly those calls.
  2. No-fallback use cases (redirect_handler, booking_handler) stream through a raw client — wrapping it in a proxy risks breaking astream_events.
  3. Langfuse traces via OTEL spans, not the callbacks list.

Instead: astream_accumulate (one chokepoint for all streaming answer calls) takes use_case/request_model/provider kwargs, and structured ainvoke sites are wrapped with ixllm.metrics.timed_llm_call(...).

Token caveat: reliable token usage is unavailable at call sites — with_structured_output(...) returns a parsed object with no usage_metadata, and streamed chunks only carry usage when stream_usage is enabled. rose_chat_llm_tokens is best-effort; full token/cost analytics stay in Langfuse.

Example PromQL (Grafana Explore)

# p95 whole-turn TTFT per site
histogram_quantile(0.95, sum by (le, site) (rate(rose_chat_answer_ttft_seconds_bucket[5m])))

# p95 per-call latency by model and use case (node)
histogram_quantile(0.95, sum by (le, gen_ai_request_model, app_gen_ai_use_case)
  (rate(rose_chat_llm_call_duration_seconds_bucket[5m])))

# token throughput by model and type
sum by (gen_ai_request_model, token_type) (rate(rose_chat_llm_tokens_total[5m]))

Code map

File Role
backend/packages/ixchat/ixchat/metrics.py tier-A histograms + record_answer()
backend/packages/ixllm/ixllm/metrics.py tier-B instruments + timed_llm_call, record_*, extract_usage
backend/packages/ixchat/ixchat/nodes/streaming_utils.py per-call emit for streaming answer calls
backend/apps/api/search/ixsearch_api/.../observability.py bucket Views on the MeterProvider
backend/apps/api/search/ixsearch_api/.../routes/chat.py tier-A′ chat.query.duration (both endpoints)

PostHog Batch Pipeline Stall Alerting (IX-4046)

The posthog_batch_processor Cloud Run job (every 5 min) transforms posthog_events_raw → the live analytics tables and advances posthog_batch_cursor. Its original alerting was crash-only: a dead or partial export returns 0 silently ("No raw events to process — clean skip"), indistinguishable from "no traffic". On 2026-07-18 that let a 12h hole in posthog_events_raw pass with zero alerts.

Two silent-failure detectors now run on every batch invocation — including the clean-skip path, which is the exact signature of a stalled export (check_pipeline_health in ixposthog_batch/monitoring.py, called from run_batch's finally).

1. Cursor lag (primary)

now() - posthog_batch_cursor.last_processed_timestamp. Catches export stall, processor stall, and a frozen cursor at once.

  • Emitted as the posthog_batch_cursor_lag_seconds gauge to Grafana Cloud OTLP (same fan-out + GRAFANA_OTLP_AUTH_BACKEND token as the chat metrics above), labelled environment and deployment_environment_name.
  • The job also Slacks itself past CURSOR_LAG_ALERT_SECONDS (default 7200 = 2 h, env-overridable) via ixinfra.notifications.send_slack_notification — same channel as job-failure alerts.

Why the threshold is 2 h and not 30 min (IX-4143)

The PostHog batch export's shortest interval on our plan is hourly, so a healthy cursor lag is a sawtooth, not a flat line: it drops to a few minutes when the hour's batch lands (~5–16 min past the hour), then climbs past 70 min before the next one. The original 1800 s threshold sat in the middle of that normal range, so production Slacked a false "cursor stalled" for roughly half of every hour. Any threshold below one full export interval is a guaranteed hourly false alarm. 7200 s means "a whole hourly batch went missing" — the real failure.

If the export interval is ever shortened, lower this threshold with it.

1b. Raw ingest lag (IX-4211)

now() - max(posthog_events_raw.inserted_at), emitted as posthog_batch_raw_ingest_lag_seconds, Slacking past INGEST_LAG_ALERT_SECONDS (same 2 h default and the same sawtooth reasoning as cursor lag).

Cursor lag alone cannot tell you which side failed: a stalled export freezes the cursor too, so "export down" and "processor down" look identical on it. Ingest lag moves only with the export, so the pair disambiguates:

cursor lag ingest lag verdict
high high export down — nothing is arriving
high normal processor down — rows are arriving, nobody consumes them
normal high impossible in practice; suspect a hand-edited cursor

The probe is an index-backed ORDER BY inserted_at DESC LIMIT 1 on the partial index idx_posthog_events_raw_inserted_at. It must keep its inserted_at IS NOT NULL filter — the index is partial, and without the filter the planner falls onto a 34 GB sequential scan every five minutes.

2. Export coverage (secondary)

PostHog's own count of one anchor event (rw_posthog_initialized) vs the rows that landed in posthog_events_raw for the same event over a settled 60-min window (ending 2 h ago). Slacks below EXPORT_COVERAGE_MIN_RATIO (default 0.99). This is the only detector for "the export is running but incomplete" — cursor lag misses it, because the cursor keeps advancing.

3. Processing coverage (secondary)

The same anchor in posthog_events_raw vs session_events (rw_-stripped name, posthog_initialized), emitted as posthog_batch_processing_ratio and Slacking below PROCESSING_COVERAGE_MIN_RATIO (default 0.99).

Why both floors are 0.99, and why they used to be 0.5 (IX-4211)

Both legs are 1:1 by construction and read 1.000 in health. Measured hour by hour on 2026-07-27: leg 2 read ~1.000 on 21 hours and 0.621 / 0.615 / 0.979 on three — and all three were real data loss, confirmed by heap position, not noise. The shared 0.5 floor let a 38%-loss hour pass silently for weeks. There is no healthy value below 1.000 on either leg; 1% of headroom is the whole budget.

COVERAGE_MIN_RATIO still overrides both legs when set, so existing deployment overrides and the COVERAGE_MIN_RATIO=2 force-a-fire validation below keep working.

The processing leg settles 6 h back, not 2 h

Since IX-4211 the cursor follows ingestion order, so a chunk PostHog delivers 5 h late is processed 5 h late — correctly. A window evaluated sooner reads that as a deficit. _PROCESSING_WINDOW_END_LAG is therefore 6 h, sized on measured ingestion lateness (p99 1 h 08 m, max 1 h 08 m 28 s on the first post-migration batch).

If this leg produces false fires, widen the window — never lower the threshold. Lowering the threshold is exactly what caused the blindness this detector exists to end.

Detector 2 stops at the landing zone. This covers the rest of the chain: rows that landed in raw but were never transformed. Write failures already crash the run into a Slack alert and poison rows already land in posthog_batch_dlq, so what this adds is the happy-path case — a filter or transform bug silently dropping rows. That is the segment the retired session_events_backup comparison used to cover, since the backup was an independent capture of the final table.

Reads ~1.000, but not exactly: only $pageview and web_vitals are skipped on the way into session_events (processor.py), so the anchor is essentially 1:1 — measured 6224 / 6225 = 0.9998 in production. The occasional single-row shortfall is a raw row with no resolvable session id, which _group_by_session skips by design (it still advances the cursor). That drift is a handful of rows in thousands and nowhere near the threshold, but it is why this leg is not asserted as an exact equality. Skipped entirely when raw holds zero rows for the window — nothing landed means nothing could be processed, and detector 2 has already alerted.

Both ratios are graphed

The ratios are emitted as gauges on every evaluation, healthy or not — same Grafana Cloud OTLP path and environment label as the cursor-lag gauge. Graph it before touching the 0.5 threshold: that number is a guess, and the gauge is what turns it into an observed band. It also exposes slow decay, which a fixed threshold cannot see.

Unlike cursor lag, this one does not query PostHog on every 5-min invocation. The window is snapped to the clock hour, and a Redis claim keyed by environment + window lets the first available run evaluate it while later runs skip it. A short lease covers the active evaluation; after normal completion the claim is retained for two hours. If the process fails mid-check, the short lease expires and a later run retries. This keeps the normal rate at 24 PostHog queries/day without losing an hour when the minute-zero invocation cannot acquire the processor lock.

PostHog is queried directly through the HogQL query API (POST https://eu.posthog.com/api/projects/80620/query/), authenticated with a dedicated monitoring key (query:read scope only) read from POSTHOG_MONITORING_API_KEY — a standalone Secret Manager secret, env-overridable for local runs. See the operator-setup section below.

A missing key silently disables this check — it logs a warning and returns, and nobody reads job logs. That is the same failure shape as the < 50 rows skip this check replaced, so it needs the same treatment: the gauge below is emitted only when the check actually ran, so a missing/rotated/expired key, a failing query, or a dead job all stop the series. Alert on its absence (rule 2 below) — do not rely on the warning.

Why one anchor event rather than the whole export: the batch export is configured with a 19-entry include_events allowlist, and mirroring that list in code would drift silently the moment someone edits it in the PostHog UI. A partial export drops rows across all event types, so a single event detects it just as well. rw_posthog_initialized fires once per session init — ~850/h at the overnight trough, ~20k/h at peak — so the ratio stays meaningful with no absolute row floor. That matters: the previous check (session_events vs the webhook's session_events_backup) skipped below 50 reference rows, so decommissioning the backup webhook would have silently retired the detector instead of failing loudly (IX-4170).

Why the window ends 2 h ago and not 30 min (IX-4147)

Same sawtooth as above, one layer down. PostHog's own event count is complete for the window immediately; posthog_events_raw can only be as complete as the last hourly export. With a 30-min settle lag the window's tail simply had not been exported yet, and coverage read as 90 - cursor_lag_minutes out of 60 minutes — 33% at the sawtooth peak, ~48% mid-cycle. Production Slacked "Partial export suspected" every hour on data that was 100% complete two hours later. The settle lag tracks DEFAULT_CURSOR_LAG_ALERT_SECONDS, so both detectors share one definition of "a whole hourly batch went missing".

Grafana alert rules (belt-and-suspenders — not redundant)

The job's own Slack is fast and independent of Grafana, but it cannot detect its own absence — "the job never ran" (dead scheduler, crash-loop, image won't boot) or "the check returned early" produce no Slack, because nothing ran to send one. Both rules below exist to alert on silence, which is why No Data → Alerting is the load-bearing setting in each.

Rule 1 — cursor stalled / job dead

  1. Alerting → Alert rules → New, on the Grafana Cloud Prometheus datasource.
  2. Query: max(posthog_batch_cursor_lag_seconds) (add by (environment) to alert per-env).
  3. Condition: > 7200 for 5m (match the job-side threshold — see the hourly-sawtooth note above; > 1800 false-fires every hour).
  4. Set No Data handling to Alerting — a stalled gauge (job dead) is itself the alert.
  5. Contact point → the same Slack channel as the job alerts.

Rule 1b — export stalled (which side is down)

  1. Query: max(posthog_batch_raw_ingest_lag_seconds).
  2. Condition: > 7200 for 5m.
  3. No Data → Alerting.
  4. Contact point → same Slack channel.

Pair this with Rule 1 on a single dashboard row: firing together means the export is down, Rule 1 alone means the processor is.

Rule 2 — coverage gap, or the coverage check going silent

  1. Query: min(posthog_batch_export_coverage_ratio). Add a second rule on min(posthog_batch_processing_ratio) with the same no-data window for the raw → session_events leg.
  2. Condition: < 0.99 for 15m, on both rules. Not 0.5 — see the why-0.99 note above; at 0.5 a 38%-loss hour reads as healthy.
  3. No Data → Alerting, evaluated over a window of at least 3 h. The gauge is written once an hour (see the cadence note above), so a shorter window false-fires between emissions.
  4. Contact point → same Slack channel.

If the processing rule fires on a legitimately-late export chunk, widen _PROCESSING_WINDOW_END_LAG in monitoring.py — do not relax this condition.

Rule 2's No-Data arm is the one that matters most: it is the only thing that catches a missing, rotated, or expired POSTHOG_MONITORING_API_KEY, a PostHog API outage, or a query that started failing — all of which make the check return early and log a warning nobody reads. Without it, the coverage detector can be disabled by a credential change and stay disabled indefinitely, which is precisely the failure mode IX-4170 was filed to remove.

Expect a rare double-fire (job-side + Grafana) when a threshold is crossed while the job still runs; harmless for a silent-failure guard. These rules are the durable alerts that survive backup-webhook decommission, and rule 1 is the exit criterion for IX-4035. Both job-side detectors now reference PostHog or posthog_events_raw only, so the decommission removes no monitoring.

Operator setup for the export-coverage check

The credential is a dedicated monitoring key, deliberately not folded into the rose-backend-env-* bundle: it is read-only, single-purpose, and independently rotatable.

  1. Create the PostHog key. Settings → Personal API keys (EU, not us.). Label rose-posthog-batch-monitoring, scoped to the Rose app (80620) project only, with exactly one scope: Query → Read (query:read). No write scopes, no batch_export:read — the check never reads the export config.
  2. Store it as a standalone Secret Manager secret named POSTHOG_MONITORING_API_KEY:
printf %s "$KEY" | gcloud secrets create POSTHOG_MONITORING_API_KEY \
  --project=inboundx --data-file=- --replication-policy=automatic
# rotation later:
printf %s "$NEW_KEY" | gcloud secrets versions add POSTHOG_MONITORING_API_KEY \
  --project=inboundx --data-file=-

printf rather than echo — a trailing newline lands in the secret payload and breaks the Authorization header. 3. Grant the Cloud Run service account access. The Terraform in infrastructure/secret-manager-iam.tf only grants secretAccessor on the rose-backend-env-* / rose-frontend-env-* bundles, so a standalone secret is unreadable until granted — the job would log PermissionDenied and skip.

The binding is declared in local.standalone_secret_names there, but do not run a bare tf-with-env.sh apply for it: the root module also covers VPC, static IPs, Cloudflare and MongoDB Atlas, so a full apply reconciles all of that to add one binding. Grant it directly instead (after step 2 — the binding needs the secret to exist):

gcloud secrets add-iam-policy-binding POSTHOG_MONITORING_API_KEY \
  --project=inboundx \
  --member="serviceAccount:$(gcloud projects describe inboundx --format='value(projectNumber)')-compute@developer.gserviceaccount.com" \
  --role=roles/secretmanager.secretAccessor

google_secret_manager_secret_iam_member is additive and idempotent, so the next full apply reconciles to the same binding rather than conflicting with it.

get_secret() checks the environment first, so an entry in backend/.env.<env> still overrides the secret for local runs — but production reads the standalone secret.

Until all three steps are done the job logs POSTHOG_MONITORING_API_KEY unavailable — export-coverage check disabled each run and never emits posthog_batch_export_coverage_ratio. That log is the only in-band signal, which is why Grafana rule 2 below is mandatory rather than optional: a key that is never created — or is rotated, expires, or loses its IAM binding later — otherwise leaves you silently down to cursor lag alone.

Forced validation

CURSOR_LAG_ALERT_SECONDS=0 just _run-posthog-batch <env> forces a lag fire → expect a Slack message + a posthog_batch_cursor_lag_seconds log line (and the gauge in Grafana Explore).

COVERAGE_MIN_RATIO=2 just _run-posthog-batch <env> does the same for the coverage detector — any real ratio is then below threshold. The first successful invocation for that settled window retains its Redis claim, so delete the lock:posthog-batch:coverage:<env>:<window> key before repeating this forced check within the same hour.

Two warnings on both commands. Slack + Supabase are shared across environments, so this posts a real message to the job-alerts channel. More importantly, _run-posthog-batch runs the full processor, not just the health checks: it writes session_events / visitor_sessions to production and advances the cursor, which makes the next scheduled run skip that work. To exercise only the read-only coverage inputs (secret → PostHog query → raw count → ratio), call _posthog_anchor_count and count_raw_events_in_window from a throwaway script instead — no writes, no Slack, no cursor movement.