Skip to content

ADR: PostHog Cost — Phase 1 Reduce, Phase 2 Remove (First-Party Self-Capture)

Status

Proposed

Tracking: IX-3999 (master). Phase 2 spike: IX-3997.

Visual explainer (interactive diagrams): 2026-07-17-posthog-cost-reduction-and-removal.html

Date

2026-07-17

Context

PostHog is billed on ingested event volume (~50M events/mo, project 80620, all from the widget $lib=web). Three cost centers were identified in IX-3879:

Cost center ~$/mo Status
Realtime destination ("Sync events to supabase hook") ~1,789 Killed — batch export (IX-3881 / IX-3885)
Feature flags (/flags per page load) ~674 Killed — IX-3969
Event ingestion ~1,529 Still alive

Event ingestion is the last untouched line. Two forces pull in different directions:

  • Cheap + safe: trim ingested volume while keeping PostHog-the-product (insights, explore, experiment UI on rw_* events). Reversible, low risk.
  • Zero + irreversible: replace posthog-js ingestion entirely with a first-party path. Kills the bill but drops PostHog as a product.

The frontend analytics layer is already a generic multi-provider event bus (frontend/shared/src/analytics/event-bus.ts); PostHog is one provider. Autocapture and session replay are already off — posthog-js is a thin event pipe. IX-3885 already built a batch pipeline: PostHog Postgres Batch Export → posthog_events_raw → Python posthog_batch_processor → live tables (session duration + GA4 is_engaged + traffic + conversion). This pipeline is the key asset: whatever writes posthog_events_raw is interchangeable.

Decision

Drive the bill to zero in two deliberate phases, gated on each other.

Phase 1 — REDUCE (keep PostHog alive)

Trim ingested volume with no product loss. Already mostly shipped; remaining work:

  • [IX-3996] — fix the batch processor btree bug (oversized page_url overflows the shadow INCLUDE index). Pipeline must be healthy before Phase 2 reuses it.
  • [IX-3963] — full $set elimination. $set is 41.8% of volume (~20.8M/mo), fired by direct posthog.people.set() calls (not the event bus). Killing it requires migrating ~38 conversion insights off person-props onto event-level / Supabase visitor profiles first.

Phase 2 — REMOVE (first-party self-capture)

Replace posthog-js ingestion with a first-party SelfCaptureProvider on the existing bus that emits rows in the exact posthog_events_raw shape, reusing the posthog_batch_processor verbatim (Option A). Strategy = dual-run → parity → cutover:

  1. Emit self-capture alongside PostHog into isolated *_selfcap_shadow tables.
  2. Diff shadow vs live on session_id (widget-generated, shared by both streams).
  3. Sign off on parity, then cut over (self-capture becomes sole writer, remove posthog-js).

Locked sub-decisions:

  • Option A — worker lands raw events; reuse the Python processor. No per-event JS math for duration/traffic/conversion (that logic lives once, in the processor).
  • Parity join key = session_id. Event-level falls back to (session_id, event_name, event_at) (self-capture uuids ≠ PostHog uuids).
  • Geo = Cloudflare edge request.cf.country, mapped ISO→full name at the worker.
  • Rollout gate = per-client SiteConfig analytics.self_capture (default off).

Phase gate

Phase 2 Workstream A is blocked by [IX-3996] — self-capture reuses the exact processor, so the pipeline must be green first. $set ([IX-3963]) is a Phase 1 cost-trim that dies for good at the Phase 2 cutover ([IX-4004]); run it now regardless — it saves money every month until cutover.

Diagrams

Current state (as of 2026-07-17) — Phase 1 pipeline live, $set still firing

flowchart LR W[Widget<br/>posthog-js] -->|capture| PH[PostHog<br/>ingest ~50M/mo] W -.->|posthog.people.set<br/>$set 41.8%| PH PH -->|Batch Export<br/>5 min| RAW[(posthog_events_raw)] RAW --> PROC[posthog_batch_processor<br/>Cloud Run Job] PROC --> LIVE[(visitor_sessions<br/>session_events<br/>visitor_page_views<br/>conversations)] LIVE --> BO[Backoffice KPIs] PH --> PROD[PostHog product<br/>insights / explore / experiments] style PH fill:#ffe0b2,stroke:#e65100 style RAW fill:#e1f5fe,stroke:#0277bd style PROC fill:#e8f5e9,stroke:#2e7d32

Now: IX-3996 (btree fix) in progress; IX-3963 ($set) not started. Everything else in Phase 1 is done.

Phase 2 dual-run — self-capture emits alongside PostHog into shadow tables

flowchart LR W[Widget] -->|posthog-js| PH[PostHog ingest] W -->|SelfCaptureProvider<br/>rw_-mapped batch| CF[cloudflare-selfcapture-ingest<br/>+geoip +referring_domain] PH -->|Batch Export| RAW[(posthog_events_raw)] CF -->|INSERT on_conflict=uuid| SRAW[(selfcapture_events_raw)] RAW --> PROC1[processor<br/>default env] SRAW --> PROC2[processor<br/>RAW/CURSOR/target=selfcap] PROC1 --> LIVE[(live tables)] PROC2 --> SHADOW[(*_selfcap_shadow)] LIVE --> PARITY{parity diff<br/>on session_id} SHADOW --> PARITY PARITY -->|iterate to match| SIGN[sign-off] style CF fill:#f3e5f5,stroke:#6a1b9a style SRAW fill:#f3e5f5,stroke:#6a1b9a style SHADOW fill:#f3e5f5,stroke:#6a1b9a style PARITY fill:#fff9c4,stroke:#f9a825

Same processor image runs twice — the second job just points RAW/CURSOR/target at the selfcap tables via env. No logic fork = the parity signal is real.

Phase 2 post-cutover — PostHog removed

flowchart LR W[Widget<br/>posthog-js REMOVED] -->|SelfCaptureProvider| CF[selfcapture-ingest] CF --> SRAW[(selfcapture_events_raw)] SRAW --> PROC[processor] --> LIVE[(live tables)] --> BO[Backoffice KPIs] style W fill:#c8e6c9,stroke:#2e7d32 style CF fill:#f3e5f5,stroke:#6a1b9a

Ingestion cost → ~0. $set gone. PostHog-the-product retired.

Migration steps + status + test

flowchart TD subgraph P1[Phase 1 — REDUCE] T1[IX-3879/3881/3885/3969<br/>flags + realtime→batch ✅] T2[IX-3996 btree fix 🔄] T3[IX-3963 $set elimination 📋] end subgraph P2[Phase 2 — REMOVE] A[IX-4000 A backend landing 📋] B[IX-4001 B ingest worker 📋] C[IX-4002 C frontend provider 📋] D[IX-4003 D parity + sign-off 📋] E[IX-4004 E cutover 📋] end T2 --> A A --> B A --> C B --> D C --> D D --> E style T2 fill:#fff9c4,stroke:#f9a825 style T3 fill:#ffe0b2,stroke:#e65100

Testing strategy (per step)

Step What it does How we test
IX-3996 btree fix cap page_url before batch insert; poison scanner URLs no longer abort the chunk processor runs clean on the 52 oversized rows; no 54000 index error; real URLs lossless
IX-3963 $set kill migrate ~38 insights off person-props, then drop the $set calls rebuild each of the ~38 insights on event-level/Supabase props → parity vs the person-prop version before deleting the emit
A [IX-4000] backend landing migration (raw+cursor+4 shadow tables) + parameterize processor + 2nd Cloud Run job unit: env table-selection + default = unchanged live behavior (regression guard). Preview branch: seed selfcapture_events_raw, run processor, assert shadow session_duration_seconds / is_engaged / traffic_* / conversion flags match hand-computed
B [IX-4001] ingest worker decode batch, add geoip + referring_domain, insert on_conflict=uuid wrangler dev → POST sample batch → row lands with $geoip_country_name mapped (FR→"France") + $referring_domain derived; dup POST = no-op; malformed rejects clean; /health+CORS
C [IX-4002] frontend provider SelfCaptureProvider + shared property-mapping + $pageview/$pageleave + nav beacons, gated flag vitest: SelfCapture output == posthog output for same event (shared-mapping proof); pageview/pageleave emission; sync-beacon on CTA/form/login; eventFilter rejects conversation/evaluation; gate off = silent; just lint/just check
D [IX-4003] parity + sign-off diff shadow vs live, ramp staging+1 client, iterate drive widget E2E (impression→message→CTA→form) → per matched session_id: session count, duration, is_engaged, traffic_channel, per-event-name counts, conversion flags all match LIVE → document sign-off
E [IX-4004] cutover self-capture sole writer, remove posthog-js, kill PH ingest post-cutover backoffice KPIs unchanged vs baseline at 100% traffic; no event loss across the repoint window; PH ingestion volume → ~0; bundle drops posthog-js; rollback verified (flag off → posthog-js returns) before decommission

Consequences

Positive

  • Eliminates ~$1,529/mo ingestion cost (Phase 2) on top of the ~$2,463/mo already cut (Phase 1).
  • Reuses the proven posthog_batch_processor verbatim — duration/traffic/conversion logic lives once; parity signal is trustworthy.
  • Per-client flag = safe incremental ramp, instant rollback, no redeploy.
  • Isolated shadow tables during dual-run → no double-count, clean comparison, cutover = a repoint.

Negative

  • Phase 2 loses PostHog-the-product: insights, ad-hoc explore, experiment UI on rw_*. Product decision, not just cost.
  • New surface to own: Cloudflare ingest worker (delivery durability — in-request write, no CDP retry) + a second Cloud Run job.
  • posthog-js wire format is not decoded (we emit our own batch shape) — but we now own the capture contract end to end.

Neutral

  • posthog-js stays authoritative for person_id during dual-run; session_id is the join key, so divergence is safe.
  • Supabase is one shared DB across envs — the selfcap job, like the PostHog one, is production-scoped; isolation is by table, not by env.

Alternatives Considered

  • Edge-intercept full bypass at the unified proxy (original IX-3997 idea, "State Y"). Deactivate PostHog forwarding at handleAnalytics and write straight to Supabase. Rejected as the primary path: it would have to decode posthog-js's compressed wire format (brittle, PH can change it) and starves the [IX-3885] batch pipeline that the backoffice now depends on. The self-capture provider approach owns the capture contract instead of reverse-engineering it.
  • Edge event-filtering only (drop $set/flag events before forwarding). Halves the bill with zero product loss, keeps PostHog fully alive. Kept as a fallback / complement, but it is a partial win — it does not remove the vendor. Effectively subsumed by Phase 1 ($set elimination happens at the source instead).
  • Per-event JS math for duration/traffic/conversion in the provider. Rejected — rebuilds the buggy incremental duration estimator [IX-3881] replaced, is order-sensitive, and forks the logic three ways, destroying the parity signal.