October 2025 - Visitor Identity Inference and the First Production Baseline¶
Context¶
First production month: real traffic across four B2B SaaS clients (experimentation, accounting, banking, HR). The retained R&D was learning who a visitor is well enough to personalize, and standing up the production measurement baseline that every later improvement is measured against. The month's analytics-delivery and form-detection work is the conversion-measurement apparatus that makes the closed-loop experiments measurable; only operations and product delivery are excluded.
Maps to 2026 R&D projects: Compounding Context Engine (visitor identity); Closed-Loop Agent Evaluation and Optimization (first production-measurement baseline).
Visitor Identity Inference and Profile Merging¶
Project and lock¶
The lock (verrou technologique) is identifying the company and role behind an anonymous B2B visitor well enough to personalize, when the only reliable signal — IP-based enrichment — covers a minority of visitors and the richer signal — what the visitor says — is unstructured and may contradict the enrichment. The uncertainty is whether inference from conversation content can be fused with enrichment data into one coherent profile without the two sources corrupting each other.
This month's work¶
The hypothesis was that LLM inference of sector and role from conversation content, merged with IP enrichment under explicit conflict-resolution rules, raises identification beyond what either source gives alone. The core of the work was the merge algorithm: it prefers a known value over "unknown", preserves a completed enrichment status over a pending one, unions feature lists, and reconciles sector/sub-sector disagreements between the inferred and enriched views. Adopting a particular IP-enrichment vendor was ordinary integration.
Results, proof, and next step¶
In production, identification rose from roughly 35% on IP enrichment alone to roughly 72% with inference and merging — a measured production figure, not an estimate. Inference-only coverage is high but lower-confidence; the value of the merge is combining high coverage with the higher-confidence enriched fields. The research core was the inference-plus-merge conflict-resolution logic, built on an off-the-shelf vendor enrichment integration. Next step: turn single-source enrichment into a multi-source cascade (November) and detect buying intent on top of the resulting profile.
Proactive Page-Aware Questioning (engagement experiment, continued)¶
Project and lock¶
September showed a passive widget rarely engages and stood up a generic dynamic-questions framework whose relevance was weak. The open question was whether tailoring the opening questions to the page the visitor is on raises the initial-engagement rate — a hypothesis measurable only against production traffic.
This month's work¶
Opening-question generation was made page-aware: questions are conditioned on the page's URL pattern and content so a pricing-page visitor and a docs-page visitor see different prompts. The per-page configuration store and the self-service admin surface around it were ordinary product work; the research was the page-context-conditioned generation, run as an engagement experiment.
Results, proof, and next step¶
October established the production baseline against which the effect is read — an initial engagement rate of 1.24% — with the measured improvement reported in November once the per-page rollout completed. This is the engagement experiment; its effect is measured the following month, where the forward chain closes.
First Production-Measurement Baseline¶
October is the first month with production metrics, which makes it the start of the closed-loop measurement line in production. The baselines recorded for Q4 tracking were: initial engagement 1.24%, engaged conversations (two or more turns) 29.24%, and visitor identification rising to ~72% over the month. A conversion reference of ~2.89% (interaction to form submission) was first observed in the October–November launch window and is used as the single pre-improvement conversion baseline in later months — not re-declared as a fresh baseline each month. This is enabling Closed-Loop work: the substrate that lets November's and December's changes be measured rather than asserted. Making those measurements trustworthy was itself part of the work: conversion events (CTA clicks, form submissions) are captured with dual-path delivery so they survive the page navigation that immediately follows a conversion, and with detection that handles multi-step and varied client form implementations — the instrument without which the conversion metric cannot be read. Widget display was made deterministic in the same spirit, so the engagement experiment gets clean exposure instead of measurements corrupted by intermittent display bugs.
Prior-Month Results Review (September 2025 shipped work)¶
September's retained engagement work — the proactive opening-question mechanism — shipped with a controlled experiment attached to it, and that experiment is the first on the platform to return a significant, grounded result. Because the mechanism reached production in mid-September and the comparison ran on live desktop traffic into early October, the result matured here rather than in the September entry, and is reported as a quasi-randomized A/B test on the number of opening questions shown.
The investigation ran in two stages on the same engagement funnel — a widget impression, an opening question displayed, and a first message sent, restricted to desktop sessions longer than ten seconds so idle impressions do not dilute the denominator. The first stage compared showing a single opening question against showing several. Across roughly 4,400 exposed visitors per arm over eight days, the single-question control converted impressions to first messages at 1.5%, while three questions reached 3.2% and four questions 3.7%; both multi-question arms were statistically significant against the single-question control (p ≈ 0.016 and p ≈ 0.008). Showing more than one opening question roughly doubled the rate at which an impression became a conversation. The second stage then asked whether more is strictly better, comparing two questions against three across roughly 10,400 exposed visitors per arm over eleven days. Here the difference vanished: two questions converted at 3.1% and three at 3.3%, with neither separating from the other (p ≈ 0.34 and p ≈ 0.22). Two opening questions capture the full effect of multiple questions without the added intrusion of a third.
| Impression → first message | Single Q | Two Q | Three Q | Four Q |
|---|---|---|---|---|
| Conversion rate | 1.5% | 3.1% | 3.2–3.3% | 3.7% |
| Significant vs single Q | baseline | not tested vs control | yes (p ≈ 0.02) | yes (p ≈ 0.01) |
The design is a sequential pair of experiments rather than one clean randomized test, and the confounds are stated plainly: the two stages ran back to back rather than concurrently; a feature-flag assignment fault zeroed the two-question arm in the first stage, so the two-versus-three contrast rests on the second stage only; and the client and traffic mix shifted across the three-week window. Within those limits the finding is firm and twofold: opening questions raise initial engagement against a passive widget, and the optimum is two — the value that was carried into production. This closes the forward chain opened in September, where the mechanism was built but its effect was not yet measurable, and it is the first controlled confirmation that the proactive-questioning hypothesis moves the metric it was designed to move.
Not retained as R&D¶
Sentry integration, production release workflow, and widget performance tuning — operations and product delivery with no nexus to a named experiment.
Open threads carried from the foundation phase¶
For honesty about continuity: the remaining tenant-isolation query-pattern coverage from July–August was left at its production-sufficient state and not pursued further once production needs were met; and the September reranking-versus-baseline comparison on the new evaluation harness was not separately reported in Q4. Both are recorded here as open rather than silently dropped.
Next Work (November)¶
- Multi-source enrichment cascade to raise identification further.
- LLM-based interest-signal detection for automated qualification.
- Dialog-state extraction for workflow triggers.
- Begin the multi-agent routing investigation (November–December).