phase 1: make it cheap (keep PostHog) → phase 2: first-party self-capture, remove it · 2026-07-17
$set (IX-3963)
Phase 1 pipeline is live: widget → PostHog → Batch Export → posthog_events_raw → processor → live tables → backoffice.
Two things are still open: IX-3996 (batch processor btree bug, in progress) and IX-3963
($set = 41.8% of volume, fired by direct posthog.people.set(), not started).
graph LR
W[Widget
posthog-js] -->|capture| PH[PostHog ingest
~50M/mo]
W -. "posthog.people.set
$set 41.8% — IX-3963" .-> PH
PH -->|Batch Export 5min| RAW[(posthog_events_raw)]
RAW --> PROC["posthog_batch_processor
btree bug — IX-3996"]
PROC --> LIVE[(live tables:
visitor_sessions · session_events
page_views · conversations)]
LIVE --> BO[Backoffice KPIs]
PH --> PROD[PostHog product
insights · explore · experiments]
class PH,PROD ph;
class RAW,LIVE db;
class PROC now;
class W wid;
Trim ingested volume, no product loss. Reversible. Mostly shipped — two tickets remain.
| Ticket | What it does | Status |
|---|---|---|
| IX-3879 | Disable per-load flag fetch + trim junk events | done |
| IX-3881 | Realtime destination → batch export (+ fix duration estimator) | done |
| IX-3885 | Activate batch export, repoint KPI consumers, decommission webhook | done |
| IX-3969 | Remove PostHog feature flags (~$674/mo) | done |
| IX-3918 / 3913 | Traffic-source funnel · batch deploy fix | done |
| IX-3996 | Fix batch processor btree overflow (oversized page_url) | in progress |
| IX-3963 | Full $set kill — migrate ~38 insights off person-props first | not started |
$set gone → ingestion cost roughly halved, PostHog fully functional. This is the gate into Phase 2.
New SelfCaptureProvider on the existing event bus emits the exact posthog_events_raw shape,
reusing the same processor verbatim (Option A). Strategy: dual-run → parity → cutover.
During dual-run both streams run side by side into isolated shadow tables.
graph LR
W[Widget] -->|posthog-js| PH[PostHog ingest]
W -->|"SelfCaptureProvider
rw_-mapped batch"| CF["CF worker: selfcapture-ingest
+geoip +referring_domain
IX-4001 (B)"]
PH -->|Batch Export| RAW[(posthog_events_raw)]
CF -->|"INSERT on_conflict=uuid"| SRAW[(selfcapture_events_raw
IX-4000 A)]
RAW --> P1["processor · default env"]
SRAW --> P2["processor · selfcap env
IX-4000 (A)"]
P1 --> LIVE[(live tables)]
P2 --> SHADOW[(*_selfcap_shadow)]
LIVE --> DIFF{"parity diff on session_id
IX-4003 (D)"}
SHADOW --> DIFF
DIFF -->|iterate to match| SIGN[sign-off]
class CF,SRAW,SHADOW,P2 sc;
class DIFF gate;
Once parity signs off, cutover makes self-capture the sole writer and removes posthog-js:
graph LR
W["Widget
posthog-js REMOVED · IX-4004 (E)"] -->|SelfCaptureProvider| CF[selfcapture-ingest]
CF --> SRAW[(selfcapture_events_raw)]
SRAW --> PROC[processor] --> LIVE[(live tables)] --> BO[Backoffice KPIs]
X[PostHog ingest
~$1,529/mo → $0]:::gone
classDef gone stroke-dasharray:5 5,opacity:0.5;
class W done2; class CF,SRAW sc;
graph TD
subgraph P1[Phase 1 — Reduce]
T1["IX-3879/81/85/69
flags + realtime→batch ✓"]
T2["IX-3996
btree fix"]
T3["IX-3963
$set elimination"]
end
subgraph P2[Phase 2 — Remove]
A["IX-4000 · A
backend landing + processor"]
B["IX-4001 · B
ingest worker"]
C["IX-4002 · C
frontend provider"]
D["IX-4003 · D
parity + sign-off"]
E["IX-4004 · E
cutover, remove posthog-js"]
end
T2 --> A
A --> B
A --> C
B --> D
C --> D
D --> E
T3 -. "dies at cutover" .-> E
class T1 done2; class T2 now; class T3 todo2;
IX-3996 — self-capture reuses the exact processor, so the pipeline must be green first. $set (IX-3963) is a Phase 1 trim that dies for good at the Phase 2 cutover (IX-4004) — run it now anyway; it saves money every month until then.| Step | What it does | How we test |
|---|---|---|
| IX-3996 now | Cap page_url before batch insert; poison scanner URLs stop aborting the chunk | Processor runs clean on the 52 oversized rows; no 54000 index error; real URLs lossless |
| IX-3963 todo | Migrate ~38 insights off person-props, then drop the $set calls | Rebuild each insight on event-level / Supabase props → parity vs the person-prop version before deleting the emit |
| A · IX-4000 | Migration (raw + cursor + 4 shadow tables) + parameterize processor + 2nd Cloud Run job | Unit: env table-selection; default = unchanged live (regression guard). Preview branch: seed raw, run processor, assert shadow duration / is_engaged / traffic / conversion match hand-computed |
| B · IX-4001 | CF worker: decode batch, add geoip + referring_domain, insert | wrangler dev → POST sample batch → row with $geoip_country_name (FR→"France") + $referring_domain; dup POST = no-op; malformed rejects clean; /health+CORS |
| C · IX-4002 | SelfCaptureProvider + shared mapping + $pageview/$pageleave + nav beacons, gated flag | vitest: SelfCapture output == posthog output (shared-mapping proof); pageview/pageleave emission; sync-beacon on CTA/form/login; eventFilter; gate off = silent; just lint/check |
| D · IX-4003 | Parity diff shadow vs live; ramp staging + 1 client; iterate | Drive widget E2E (impression→message→CTA→form) → per matched session_id: count, duration, is_engaged, traffic, per-event counts, conversion flags all match LIVE → sign-off |
| E · IX-4004 | Self-capture sole writer, remove posthog-js, kill PH ingest | Post-cutover KPIs unchanged vs baseline at 100% traffic; no event loss across repoint; PH volume → ~0; bundle drops posthog-js; rollback verified before decommission |
docs/src/adr/2026-07-17-posthog-cost-reduction-and-removal.md ·
Plan: .context/plans/first-party-self-capture-provider-dual-run-to-shad.md