September 2025 - Retrieval Quality Measurement and Reranking

Context

Final pre-production month before the October launch. The retained R&D was the hybrid-retrieval quality investigation finally getting its instrument — a way to measure quality — and a reranking experiment to run on it; plus a first, partial visitor-engagement experiment. The month's large engineering items — scaling the vector store for valid production-scale experiments, per-operation instrumentation, and latency mechanisms — are the apparatus that investigation runs on; only cloud-networking plumbing and product UI are excluded.

Maps to 2026 R&D projects: Compounding Context Engine (retrieval quality); Closed-Loop Agent Evaluation and Optimization begins here with the evaluation harness; the engagement experiment seeds Adaptive Multi-Tenant Conversation Orchestration.

Hybrid Graph-Vector Retrieval at B2B Production Quality and Latency (continued)

Project and lock

July and August advanced the quality/latency investigation on qualitative grounds only — there was no harness to score retrieval quality, so every prior quality claim was an impression. The lock this month was twofold: closing that measurement gap, and testing whether reranking retrieved documents by query relevance improves answer quality, a question that cannot be posed numerically until the harness exists.

This month's work

Two things were built. First, the measurement substrate: a modular dataset-evaluation harness that scores responses for hallucination, relevancy, faithfulness, correctness, and contextual recall against labeled datasets, with reproducible protocols. This is what converts "quality" from a pilot impression into a measurable quantity, and it is the precondition for every later closed-loop evaluation. Second, a reranking stage was integrated into the retrieval pipeline behind a configuration flag, with reranked depth separated from the number of chunks retrieved so the two tune independently. The experiments also needed a platform valid at production scale and instrumented to read: the retrieval store was moved to a managed vector database so quality and latency results hold at production document volumes rather than on a toy store, and per-operation tracing was added across the multi-step pipeline to locate where latency and quality are lost. The same line of work completed the provider/model experiment harness for embeddings — extending the configuration-driven provider abstraction begun in July to the embedding path so embedding-provider comparisons run without code changes — and added batching and caching of embedding calls to hold the latency budget, with environment separation so experimental conditions run without disturbing production. The cloud-networking plumbing around the database, and the mobile UI shipped this month, are the only ordinary parts.

Results, proof, and next step

The harness was stood up only late in the month, so no scored before/after reranking comparison was completed within the period — the decimal eval scores an earlier draft showed were not produced by a September run and have been removed as ungrounded. The reranking quality effect is therefore to be measured on the new harness. The measurement substrate and the reranking experiment form one measurement-driven investigation of hybrid-retrieval quality; the scored comparison is the pending next step.

Visitor Engagement via Proactive Questioning

Project and lock

The widget was passive and most visitors never engaged. The uncertainty was whether LLM-generated, page-and-context-aware opening questions would raise the initial-engagement rate over a passive widget — a hypothesis testable only against production traffic.

This month's work

A dynamic-questions component was built: it generates opening questions from page content and visitor context, rotates them with timing control, follows the site language, and reads its per-page question count from configuration. September stood the framework up and validated it qualitatively in pilot.

Results, proof, and next step

A controlled comparison of the number of opening questions shown was launched in September on live desktop traffic; it ran into early October, and its significant result — that opening questions raise initial engagement and that two questions are the optimum — is reported in October's prior-month results review, where this forward chain is closed. The first production engagement baseline (an initial engagement rate of 1.24%) was also established this month for Q4 tracking. The engagement mechanism was built and the hypothesis set, and the experiment was launched here, but its measured result matured in October and is reported there rather than in this entry.

Not retained as R&D

Cloud-networking plumbing around the managed database (VPC peering, IP whitelisting) and mobile UI — ordinary integration and product delivery with no nexus to a named experiment.

Next Work (October)

1. Complete the remaining tenant-isolation query-pattern coverage carried from the foundation phase.
2. Run the reranking quality comparison on the new harness.
3. Visitor profiling — understand who is asking — and the first multi-agent intent-handling exploration.
4. CTA tracking and form detection for conversion measurement.