September 2026 - R&D Journal: The Agent's Measured Effect on Conversion, a Forced Removal of the Ranking Stage, and Silent Knowledge Loss¶
Context¶
This entry covers September 1-30, 2026, and folds in work from the last four days of August that the August entry does not record.
The central result came from live traffic. Of three randomized tests started in mid-July, each withholding the agent from a random share of visitors, the largest became readable in September: on that site the agent lowered demo-form submissions. Reading the conversations behind that number turned the question from whether the agent helps into how it should behave, and a randomized comparison of two conversation policies launched on September 15. Its result, read at its fixed end date in November, is reported in that month's entry.
On September 1 the hosted service that ranked retrieved passages stopped answering. The ranking stage was removed and measured, partly answering August's open question of whether it earns its cost. Retrieval work in the same weeks exposed failures in which the knowledge reported as indexed was not the knowledge retrieval returned.
The measurement substrate moved to a single first-party path, with the gap between the two paths measured before the old one was cut. Several evaluation instruments were found failing silently. Outbound publishing gained a quality gate. The first post-deployment read of August's query-resolution work, in production since August 31, is in the Prior-Month Results Review.
Adaptive Multi-Tenant Conversation Orchestration¶
Project and lock¶
The agent is embedded on business-software vendors' websites, and each vendor judges it by the leads it produces: a submitted demo form, or an email address left in the chat. No prior work established which conversation policy produces more leads. An agent that answers in full may satisfy a visitor who would otherwise have filled in the form; one that pushes the demo early may lose a visitor who is not ready. Transcripts cannot show either effect, because visitors who did not chat, or who converted without the agent, are invisible there.
The second lock carries over from earlier months. Business contracts (qualification gates, content-gating offers, support routing) must hold on every turn while the model decides wording and timing. Enforced in code, a rule can override a good decision by the model; left to the prompt, it can be ignored. Where that line sits is decided case by case, by measurement.
This month's work¶
Hypotheses¶
- H1. On the largest test site, showing the agent changes the share of visitors who submit the demo form. The direction was not assumed.
- H2. If the agent lowers form submissions, the cause is frustration: visitors who fail to get what they want leave.
- H3, the alternative to H2. The cause is deflection: the agent answers the question in the chat and lets the visitor go without proposing a next step.
- H4. Splitting deterministic routing rules from model judgment along the line of what was asked, rather than who is asking, removes misroutes without creating new ones.
Experimental setup: the on/off tests¶
Each test assigns a visitor at random, on first visit, to see the agent or not, and keeps that assignment on later visits. Three tests ran in September (Table 1). On the largest site, a staff-scheduling software vendor, the target minimum detectable effect was 11% relative. On the accounting-software site, enrollment of new visitors stopped at its configured end date of August 25 while enrolled visitors' conversions kept accruing.
The primary metric is the share of enrolled visitors who submitted a tracked demo form after enrollment, each visitor counted once. A combined metric also counts a visitor who left an email in the chat; held-out visitors never see the chat, so only the exposed arm can gain from email. Reads were taken on September 14, when the follow-up was decided, and on September 30, each counting visitors whose first session started before the cut-off. The test was not stopped at either read.
Table 1. On/off tests running in September 2026.
| Site | Holdout share | Started | Planned end | Enrolled by Sept 30 |
|---|---|---|---|---|
| Staff-scheduling vendor | 20% | July 17 | October 13 | 356,022 |
| Accounting-software vendor | 50% | July 16 | August 25 (enrollment) | 91,506 |
| Rose's own website | 10% | July 15 | October 3 | 2,458 |
To test H2 against H3 on the largest site, the conversations of the test window were read from the chat side: whether the agent proposed a demo, whether the visitor booked one in the chat, and, in a sample of forty conversations where a proposed demo was declined, what the visitor had come for.
Experimental setup: the playbook test¶
An on/off test says whether the agent helps, not how it should behave. The follow-up compared the agent's two existing conversation policies, called playbooks, on the same site. The information playbook answers first and proposes a demo only when an accumulated interest score crosses a threshold that single-turn visitors rarely reach by construction. The conversion playbook answers towards a next step, holds its first proposal while discovery questions run, lets a language-model judge decide when the visitor is ready, and never proposes two turns in a row. The test flips one existing configuration value; no new prompt was written.
Three design choices came from the on/off read. Only visitors who start a chat are enrolled, since a visitor who never chats never meets a playbook. The primary metric is form or email, because a form-only metric penalizes a policy that moves leads into the chat, which is exactly what the on/off test could not see. The duration was fixed before launch at eight weeks, September 15 to November 10.
The test needed new apparatus. The agent's backend assigns each visitor deterministically from a hash of the test and visitor identifiers, before the playbook is chosen, and stores it once, so a visitor keeps the same playbook across visits and every downstream consumer reads the same server-resolved value. Staff previews and non-production requests are not enrolled. Results join the stored assignments to the site's own form and email events between enrollment and the end date, each visitor counted once.
Routing boundaries¶
Four intent-classifier misroutes were reproduced, each changing what the visitor saw. "How do I contact support?" was classified as a general question because the support category required evidence that the visitor was a customer; the answer path, which has no list of allowed destinations, then improvised a support route, in one run a link that did not exist. The adopted rule: customer evidence settles who is asking, never what was asked. A soft decline after a demo offer was routed as off-topic to the node that injects the client's contact links, pushing a link at the moment to back off. It now routes to the answer path, whose prompt lacks that menu, so the push disappears by construction rather than by instruction. Each new rule was paired with a counter-rule and dataset counter-examples so that it could not over-fire.
Two arbitration defects were investigated between the skill that offers gated content in exchange for an email and the skill that recommends a content card. Both triggers legitimately match some turns, and the rule that strips the card acts only when the selector returns both candidates. A prompt-level exception telling the selector to prefer the gated offer was measured first. The fix that held names the gated offer as a co-candidate in the card skill's description, so the selector returns the pair and the existing deterministic rule decides. The second defect concerned triggers that fire when a visitor reveals context, such as their sector, rather than asking for content.
A rule forcing a profiling question onto every turn while an optional profile field was missing was removed: it overrode the model's chosen ending, so a yes/no question about tax treatment closed with an unrelated profile prompt and buttons. The field is now asked when the model judges there is a natural opening; questions that gate a call to action stay enforced in code. A reordering of skill composition tried in the same change regressed other augmenting skills in review and was withdrawn in favour of a later single post-composition resolution step.
Three model-facing nodes were moved to a schema-enforced output mode after one provider route was found not to enforce the declared schema in the default mode.
Latency of the pre-answer gate¶
Intent classification, skill selection, interest detection and the readiness judge run in parallel before the answer, so the gate lasts as long as its slowest branch. A smaller, faster constrained-output classifier from an external provider was benchmarked against the production intent and skill-selection nodes on the same datasets (Table 4).
Workspaces resolved per turn¶
Some clients serve several markets from one domain under different path prefixes, with different offers, pages and rules; resolving the client from the hostname alone made a market-specific configuration impossible. Each turn now resolves its workspace from trusted request metadata (exact host, then the longest enabled path prefix, then the host default) and drives both the configuration and the retrieval corpus from that one resolution, so the two cannot diverge within a turn; the result is recorded on each message. Client skills can be layered per path, and a route whose display is switched off is now off rather than falling back to the root. The behaviour is gated per client and defaults to off.
Results, proof, and next step¶
Measurements¶
Table 2 reports the on/off tests. On the staff-scheduling site the form-only difference was -11.2% at the September 14 read; counting emails left in the chat, which adds 65 exposed visitors out of about 220,000, it was -9.3%. By September 30 these had narrowed to -8.4% and -6.5%, the latter's interval ending just short of zero (p = 0.044). The accounting-software site, at a conversion base near 6%, showed no detectable difference. Rose's own site enrolled too few visitors to read.
Table 2. Demo-form conversion by arm, on/off tests. Intervals are 95% Wald intervals on the log of the rate ratio; p is a two-sided two-proportion test at a fixed horizon.
| Site | Cut-off | Metric | Agent shown | Agent withheld | Relative difference | 95% interval | p |
|---|---|---|---|---|---|---|---|
| Staff-scheduling | Sept 14 | Form | 3,144/219,700 (1.431%) | 882/54,745 (1.611%) | -11.2% | -17.5% to -4.3% | 0.002 |
| Staff-scheduling | Sept 14 | Form or email | 3,209/219,700 (1.461%) | 882/54,745 (1.611%) | -9.3% | -15.8% to -2.4% | 0.009 |
| Staff-scheduling | Sept 30 | Form | 4,136/285,138 (1.451%) | 1,122/70,884 (1.583%) | -8.4% | -14.2% to -2.2% | 0.009 |
| Staff-scheduling | Sept 30 | Form or email | 4,220/285,138 (1.480%) | 1,122/70,884 (1.583%) | -6.5% | -12.4% to -0.2% | 0.044 |
| Accounting software | Sept 30 | Form | 2,726/45,514 (5.99%) | 2,823/45,992 (6.14%) | -2.4% | -7.3% to +2.7% | 0.35 |
| Accounting software | Sept 30 | Form or email | 2,735/45,514 (6.01%) | 2,823/45,992 (6.14%) | -2.1% | -7.0% to +3.0% | 0.41 |
| Rose's own site | Sept 30 | Form | 27/2,218 (1.22%) | 5/240 (2.08%) | not read | too few events | 0.26 |
Table 3 reports the chat side of the staff-scheduling test window. Of the forty declined proposals read, about two-thirds were visitors whose price or how-to question had already been answered in the chat; most of the rest were spam, abuse or existing customers. In those forty, about four to six prospects were lost to a failed request, too few to explain the difference in Table 2. The sample excludes the 3,544 conversations with no demo proposed, so it cannot rule out frustration there.
Table 3. Demo proposals and bookings in the chat, staff-scheduling site, July 17 to September 13. 95% Wilson intervals.
| Conversations where… | Count | Share | 95% interval |
|---|---|---|---|
| A demo was proposed | 289/3,833 | 7.5% | 6.7% to 8.4% |
| A demo was booked, when proposed | 33/289 | 11.4% | 8.2% to 15.6% |
| A demo was booked, when not proposed | 145/3,544 | 4.1% | 3.5% to 4.8% |
The playbook test launched on September 15. By September 30 it had recorded 826 enrollments, 408 on the information playbook and 418 on the conversion playbook, consistent with the designed 50/50 split (z = 0.35). Its outcome was not read in September, by design.
Table 4 reports the routing and latency measurements. With the intent set at 113 of 113 after the four rules, it no longer discriminates: it is now a regression guard, not a signal, and needs harder cases. The prompt-level arbitration exception was reverted; with the co-candidate wording, all 108 items across the eight feature datasets passed and the skill selector kept a recall of 1.000. A first wording for the reveal-shaped triggers, describing the reveal case as matching "even though they asked for no content", read as an override of refusals and was replaced by a wording that only names what matches. The malformed-output count compares the default mode with the schema enforced.
Table 4. Routing and latency measurements.
| Measurement | Before | After or candidate |
|---|---|---|
| Intent classification set, correct | 104/113 | 113/113 (two runs) |
| Gated offer selected on the worst-case overlap turn | prompt exception: 0/5 | co-candidate wording: 5/5 |
| Gating dataset | 14/15 | 15/15 |
| Explicit refusal selected for gating, first reveal wording | 0/3 | 3/3 (rejected) |
| Malformed structured output, one node, direct calls | 4/16 | 0/16 |
| Intent accuracy, production classifier vs candidate | 0.982, mean 1.06 s | initial 0.796, 0.26 s; tuned 0.965, about 0.25 s |
| Skill selection micro-F1 and exact match, production vs candidate | 0.833 and 0.600, 0.50 s | 0.909 and 0.800, 0.25 s |
Interpretation¶
H1 is supported on the staff-scheduling site. The evidence favours H3 over H2: the agent proposed a demo in fewer than one conversation in thirteen, bookings were higher when it did, and the declined proposals read as answered questions rather than failures. The bookings comparison is observational, since the agent proposes to visitors who already show intent; it indicates the direction of an intervention without measuring it, which is why the follow-up was a randomized playbook comparison rather than a policy change.
The on/off read also exposed a construct problem: a form-only metric attributes to the agent a loss that is partly a transfer of leads into the chat. Here the transfer, 65 visitors at the September 14 read and 84 by September 30, does not reverse the result, but it changes the metric a policy test must use.
On routing, H4 held where it was tested. The two negative results are the more reusable learning: told to choose between two matching skills, the selector kept choosing the wrong one whatever the instruction said, and a phrasing written to widen a trigger silently overrode a refusal. Both durable fixes moved the decision into the deterministic layer, which the model only has to feed.
The alternative classifier closed most of its quality gap and ran about four times faster on intent classification and twice as fast on skill selection in isolation, but was not promoted. Its intent criteria were tuned by reading errors on the measured dataset, so 0.965 is a development score, not an estimate of unseen quality. And because the gate waits for its slowest branch, replacing skill selection alone would not shorten the turn; the estimated saving of about 0.7 seconds from replacing both nodes was not measured in the real graph.
Threats to validity¶
- Repeated looks. The staff-scheduling test was read twice before its planned end, and fixed-horizon p-values overstate significance under repeated looks. The effect also narrowed between the two reads.
- One site. The effect is measured on one vendor. The second site shows no detectable effect at a much higher conversion base; the third is too small.
- Construct. A form or an email is a lead, not a sale. Whether the forms the agent displaced would have closed was not measured, because sales outcomes were not joined to the test.
- Chat-side reading. The transcript sample is forty conversations, read by one reviewer, and the proposal and booking flags come from the agent's own pipeline.
- Routing datasets. The classification set is small, now saturated, and partly built from the same investigation that tuned the rules.
Next step: read the playbook test at its fixed end date, and replicate it on other sites if it shows an effect.
Compounding Context Engine for Company, Industry, and Buyer Intelligence¶
Project and lock¶
The retrieval path ends with a cross-encoder that scores every candidate passage, entity and relation against the visitor's message and keeps the best. August showed that the context-sufficiency metric cannot judge that stage, being blind to passage order, and left open whether the stage earns its cost.
The second lock is integrity on top of an open-source graph-retrieval library that has no notion of tenant, no reference counting on the passages it stores, and an index that becomes searchable asynchronously. The platform must guarantee that what a client's knowledge base is reported to contain is what retrieval actually returns; the library does not provide that guarantee, and its failures are silent.
The third lock is fidelity and precedence: what the scraper keeps from a web page, and which source wins when a curated answer and a page disagree, decide what the agent can say. Neither had been measured against the page itself.
This month's work¶
The ranking stage, removed by an outage¶
On September 1, after about forty hours of clean operation, the hosted reranking service stopped answering when the provider suspended the account, and every retrieving query waited up to three seconds for it before falling back. The stage was removed rather than disabled. The retrieval library already applies the limits that matter before documents reach the agent (a passage cap and graph-side token budgets); a second cap existed only to truncate the reranker's mixed pool, and on one inspected query cut 17 retrieved passages to 7 and 19 entities to 9. Without it, a small graph sends every node and relation to the answer writer. Curated answers are placed first.
The hypothesis was that the reranker's value was selection, removing noise that would distract the answer writer; if so, a cap on context volume should reproduce its effect. The removal was measured against four reranked runs taken on the outage day, before the cut-over, on one client's 81-item end-to-end set, and against current runs on a 42-item cross-client set. Three alternatives were then tried: disabling the curated-first ordering, halving the passage cap, and a cap by document type.
Three days later the passage cap was lowered from 18 to 10, close to what the reranker had kept in production (a mean of 10.7 passages per query, median 11, over 520 traced queries). Each value was evaluated on the same full datasets, and latency on seven interleaved pairs of requests.
Indexed but not retrievable¶
A retrain of seven uploaded documents for one client ran for more than twenty-five minutes without completing a document, and would have ended as succeeded with vectors that carried no embeddings. The chain: the fallback embedding call had no time bound, the library's worker killed it with an opaque timeout, the vector store inserted the documents without embeddings, and the library recorded a per-document failure while the batch call returned normally. Each step was locally reasonable; together they produced an index that looked complete and could not be searched.
A search for the same failure class found five more. A curated-answer sync counted an insert as soon as it classified a row as new, although the library silently drops an insert whose identifier it already holds. The library's delete returns a refusal as a value, rather than raising, when another task holds its pipeline; the sync discarded the value, re-inserted, read the stale status as success, and stamped the new content's hash onto the old indexed content, so the document read as unchanged forever after (the library's documentation names a failure value its own type does not contain). Identical text produces identical passage identifiers and the library deletes every passage a document lists, so removing one page can delete passages a retained page still needs. A section of retrieved context was closed at the first code fence anywhere in a passage, dropping every later passage before the answer. And a retrain was reported complete when the documents were written, although the vector index serving retrieval is rebuilt asynchronously, so a tester could still receive the old answer.
Each fix confirms by reading back. Syncs re-read what the library recorded after every insert and replacement and count only what is confirmed. Deletion statuses are checked against an allow-list, so an unknown status fails closed. Replacements are serialized. Shared passages are removed from a deleted document's list before the delete, so the library removes only the page's own. A retrain stays in progress until passages written in that run appear in vector search. An audit command compares three narrowing counts per client: rows in the source of truth, documents the library records as processed, and documents retrieval can actually return.
Fidelity and precedence¶
On one client, broad product questions retrieved the same boilerplate block many times. Repeated website blocks are now deduplicated before ingestion, each page keeping its own source snapshot and canonical owner. Scraping had used a hosted main-content extraction that removed page heroes, where several clients state their value proposition and case-study figures, and an optional hosted model-based cleaner that kept a different subset of the page on each run. Both were turned off in favour of excluding navigation and footers, and the platform's own cleaner was given compact image references and a rule to keep label-and-value cards.
When a curated answer and a website page stated different facts, the answer writer saw both as undifferentiated context. Curated answers are now a separate block with a rule that they win on conflict, without taking over questions they do not answer. A deterministic feature dataset was built for it: paraphrased exact matches, loose matches, and one conflict item.
Identity and its provenance¶
One visitor-identification provider resolves a browsing session only about forty seconds after it starts, so the lookup made at widget load failed for about 98% of visitors: production identified 24 companies out of 1,354 lookups a day, and in a sixteen-minute production sample, 0 of 119 chat-path lookups identified anyone. Lookups are now queued and resolved by a scheduled sweep; the provider keeps a session resolvable for thirty days, so a late lookup loses nothing. The same work found that 42% of stored visitor emails came from de-anonymization providers rather than from the visitor. Since the agent reads a stored email as consent and stops asking for one, a provider's guess is now kept in a separate field, available to rate provider accuracy, and never treated as the visitor's own.
Results, proof, and next step¶
Measurements¶
Table 5 reports the removal of the ranking stage. Only conversational flow separated from noise: it fell by 0.019 on the client set, 7.3 times the run-to-run standard deviation, while the other dimensions moved by 1.3 to 1.7 standard deviations. At item level, 5 of 80 compared answers were materially worse, 74 flat and 1 better. Context grew from 30.0 to 54.5 documents per query on the client set and from 29.9 to 48.0 on the cross-client set, but total input per turn grew only 15%, from 59,029 to 67,914 tokens, because retrieval is 6-9% of a prompt dominated by instructions and history. Time to the last token fell from 7.18 to 6.00 seconds on the cross-client set and from 4.49 to 4.39 on the client set.
Table 5. End-to-end scores with and without the ranking stage. Ranges are over runs; the reranked arm cannot be re-run.
| Dataset | Dimension | Reranked | Removed | Current develop |
|---|---|---|---|---|
| One client, 81 items | Answer quality | 0.811-0.825 (4 runs) | 0.804-0.814 (2 runs) | 0.817 (1 run) |
| One client, 81 items | Correctness | 0.827-0.853 | 0.822-0.835 | 0.840 |
| One client, 81 items | Completeness | 0.825-0.844 | 0.819-0.831 | 0.827 |
| One client, 81 items | Flow | 0.768-0.774 | 0.747-0.754 | 0.758 |
| Cross-client, 42 items | Answer quality | not available | 0.825 (1 run) | 0.833, range 0.828-0.843 (3 runs) |
| Cross-client, 42 items | Flow | not available | 0.756 | 0.775, range 0.764-0.786 |
| Cross-client, 42 items | Share of items passed | not available | 0.732 | 0.770, range 0.714-0.833 |
None of the three alternatives restored flow (Table 6), and disabling the curated-first ordering cost 0.05 on the share of items passed, so it stayed.
Table 6. Attempts to reproduce the reranked flow score, one client, 81 items.
| Condition | Flow |
|---|---|
| Reranked (reference) | 0.768-0.774 |
| Curated-first ordering disabled | 0.752 |
| Passage cap 18 halved to 9 | 0.755-0.767 |
| Cap by document type at 30 | 0.758 |
On the passage cap, moving from 18 to 10 changed the mean answer score by -0.009 on the client set and +0.005 on the cross-client set, against a noise floor of about 0.005 measured from identical runs: the signs disagree, so there is no consistent effect at one run per cell. Seven interleaved latency pairs gave 4,481 and 4,573 milliseconds, with the sign depending on which arm ran second: the measurement read cache warmth, not the cap. The slow turns lay elsewhere. Retrieval took 50-85% of every slow turn, most of it in vector search on the turns inspected, and its 90th percentile had risen from 3.2-4.3 seconds in mid-August to 5.6-7.0 seconds while the median had not moved. That tail belongs to the shared database cluster and was not fixed this month. A graph-side count cap was also tried and removed: it saved about 750 tokens of a 60,000-token prompt.
Table 7 reports the knowledge-integrity failures found in production.
Table 7. Silent knowledge-loss failures found in September, production.
| Failure | Measured extent |
|---|---|
| Curated answers recorded as ingested but not retrievable | 79 answers, 12 of them on two live clients; all active clients at 867/867 after repair |
| Replacements whose delete was refused, leaving old content under a new hash | 58 documents, 49 of them on seven live clients |
| Code fence closing the retrieved-context section | one inspected page: 10 retrieved passages reduced to 2, the curated answer ranked fifth never reaching the model |
| Deleting a page that shared passages with a retained page | one client's index blocked at revision 40 of 167 from September 11 |
| Retrain reported complete before the vector index served it | observed by testers; no count |
The fidelity changes were measured on fixed pages. Deduplication removed 74.5% of duplicate block copies on one client and lowered the share of repeated text in retrieved context from 9.9% to 6.0%. In a before-and-after on 29 pages from 12 clients, scraped fresh and cleaned with the production prompt, page heroes were kept and no menu text survived; on the 26 pages that could be compared side by side with the stored version, cleaned content grew from about 105,000 to 120,000 characters. A label-and-value card that the cleaner had dropped in 3 of 5 runs was kept in 5 of 5.
On curated precedence, the feature dataset passed 5 of 5 items across three runs with the change, while the baseline failed the conflict item. On one client page, eight runs of three questions repeated a contradicting page fact 3 times in 16 before the change and 0 times in 16 after, and the loose question was never taken over by the curated answer. The cross-client set fell from 0.843 to 0.824 over three repeats, above its pass mark of 0.78. The largest drops were answers that now follow a client's curated answer against an older expected answer, which points at the expected answers rather than the change; these had not been refreshed by the end of the month.
On identity, the sweep queued 499,860 sessions in September (Table 8); the 32,516 matched to a company are 9.8% of those the provider resolved.
Table 8. Deferred identification, September 2026.
| Outcome | Sessions | Share of queued |
|---|---|---|
| Matched to a company | 32,516 | 6.5% |
| Resolved, no company | 299,653 | 59.9% |
| Expired without becoming resolvable | 167,691 | 33.5% |
| Total queued | 499,860 | 100% |
Interpretation¶
The removal argues against the selection hypothesis: context nearly doubled, total input rose only 15%, and no cap on volume brought flow back. The reranker's measurable value looks like ordering, which passage the answer writer reads first, but no run held the selected passages fixed while changing their order, so the mechanism is tentative. That fits August's finding that the sufficiency metric is blind to order, and makes August's outstanding instrument, a rank-aware retrieval measure, also the instrument needed to choose a replacement. Until it exists, the loss is bounded to one conversational dimension, about 0.02, and the stage stays out.
The integrity failures share one mechanism: the library reports what it intended to do, not what it did, and every consumer built on that report inherits the gap. None of the six was visible from job outcomes, dashboards or answer quality before it was found; at least two were found by people watching a run or testing an answer. Reading back what was written is the only check that held in every case. It turns failures that passed as success into visible failures, so retrain failure rates are expected to rise.
A hosted extraction step that decides what a page's "main content" is removes, by design, the parts of a business-software page that state its offer. Position in the context was not enough to make a curated answer win a conflict; an explicit rule was.
The identification comparison is not controlled. Before the change, the denominator was every lookup made at widget load; after it, only sessions where free sources had found nothing were queued, and from September 3 engaged visitors were queued on every domain rather than on four. The rise from about 1.8% to 6.5% of lookups therefore mixes the timing fix with a different population.
Threats to validity¶
- One dataset per decision. The reranker decision rests on one client's 81 items plus one run on the cross-client set. The reranked arm could not be re-run after the outage, so the comparison crosses a short time gap, mitigated by a current run that reproduced the historical baseline closely.
- Judge-scored flow. Flow is a judge's dimension, not a visitor outcome.
- Counts of integrity failures are what the new audits found, not a population rate; former and test clients were excluded from repair.
- Fidelity measurements are on selected pages, and character counts are a proxy for retained facts.
Next step: build the rank-aware retrieval measure, use it to choose a replacement for the ranking stage, and run the integrity audit on a schedule as a monitor.
Closed-Loop Agent Evaluation and Optimization¶
Project and lock¶
August consolidated product events onto a first-party path but deferred making it the only source of page events, and closed on the need to run a randomized experiment through the new substrate. A measurement system cannot support causal claims if two of its sources can disagree, if its metrics count different populations as one flow, or if test traffic leaks into the series being read.
The same lock applies to the evaluation harness. An evaluation that fails silently, a dataset that every candidate passes, or a score tuned on the set that reports it all produce agreement that is not evidence, and none of these failures announces itself.
This month's work¶
Cutting page events to one source¶
Before the old path was removed, the two paths were compared on identical ground: three production cohorts on September 8-10, one hour each, matched over five hours around each window. The processor then stopped reading page events from the old path, and the widget stopped sending them there, in changes merged on September 16 and 18. With one path authoritative, a second question arose: which copy of an event, arriving twice with different timestamps, defines a session's start, end and duration. The adopted rule: the first persisted copy of an event identity wins, and an incoming duplicate may never push a session boundary outward. Already-inflated historical sessions are repaired only from an explicit, backed-up list, never by a blanket recomputation.
Metric constructs¶
Three metrics counted different units as one. The qualification funnel counted converted people in one step and sessions in the others, so its "did not start" figure subtracted incompatible units; it was rebuilt as one tree in which each step is a subset of the one before, with the reasons a visitor did not see the widget split out. The impact view comparing visitors who chatted with those who did not counted a visitor who converted in two months twice, and counted visitors whose forms preceded their first chat as assisted conversions. Each person now counts once, in the month of their last conversion, and conversions before the first chat are reported separately and excluded from the assisted share. On one client, a single self-serve signup counted as seven conversions because every step of a multi-step signup was configured as a completion page; completions now fire once per matched page pattern per session.
Three sources of contamination were closed: test conversations from the administrative preview were counted in client statistics; evaluation and test traffic could trigger the client's external webhooks and notifications; and abusive turns were mixed into the conversation pools that evaluations sample from.
Evaluation instruments¶
The retrieval evaluation built in August had errored on all 50 items of every arm since a tracing-client upgrade on August 28 removed a method it called; nothing ran it, so nobody saw. It was repaired and added to the merge gate as a completion check. The same investigation found that every evaluation type silently discarded a pass threshold of zero, treating zero as absent. Lessons from other lines were recorded as rules: a dataset that scores perfectly is a guard, not a signal; the share of items passed is not a decision metric when it swings across identical runs; criteria tuned on a set make its score a development score; and a judge that decides a merge from one sample at temperature zero fails on one unlucky sample, so the agent-facing end-to-end gate now takes the majority of three. Multi-turn items were added where a single-turn item structurally cannot catch the defect, such as the agent re-asking a question the visitor has just answered.
The classifier benchmark of the orchestration project produced the protocol now required for a production claim: a fresh set of production turns, labelled by an independent annotator, frozen with hashes before either arm runs, with the arms named only after the verdict, and retired to development data as soon as it causes a change.
The client-graded evaluation loop¶
Clients can now build a set of questions with expected answers, replay them through their live agent, and see each answer graded by a judge. Accepting an answer makes it the new expectation; rejecting it records where the fix belongs, such as the knowledge base, a guideline or support. Client feedback, which earlier months treated as anecdotal, becomes a labelled, repeatable set per client, with a human verdict to compare the judge against.
Results, proof, and next step¶
Measurements¶
Table 9 reports the comparison of the two page-event paths before the cut-over. The gap was accepted; it is the gap the substrate now carries.
Table 9. Page events recorded by the old path with no first-party counterpart, three production cohorts, September 8-10, 2026.
| Event | Unmatched | Share |
|---|---|---|
| Page view | 13/24,486 | 0.053% |
| Page exit | 93/21,739 | 0.428% |
On a fixed sample of 296 sessions from a production snapshot of September 15, 16 had a duration that differed between the two copies of their events: eleven by one second, two by three, one each by four, 21 and 145 seconds. The 21-second case moved the session across the ten-second engagement threshold. This is a deterministic sample, not a population rate.
On the repaired retrieval evaluation, the full 50 items completed with no error and scored 0.477 for the production arm. The first five items, the deterministic sample a fast merge gate would use, scored 0.325 over six runs, against a floor of 0.40: a scored gate at that sample size would fail every merge of healthy code, which is why the gate checks completion only.
The client-graded loop recorded 69 runs on 11 sites between August 31 and September 30, and 252 questions on 13 sites by the end of the month, including staff use. Of 681 graded answers, only 9 carried both a judge verdict and a human verdict, and they agreed on 5. That is too few to estimate agreement between the judge and the client.
Interpretation¶
Retiring a measurement path needs its own measurement: the residual gap is small but not zero, and concentrated on exit events, which feed duration and engagement. The timestamp rule matters for the same reason: a metric with a threshold, such as engagement at ten seconds, can change category from a one-second disagreement between sources.
The metric-construct corrections change definitions, so figures computed before and after them are not comparable. The evaluation findings share one pattern: each instrument reported agreement, a pass or a complete run when the right reading was "not measured". The playbook test is the first experiment read entirely through the consolidated substrate.
Threats to validity¶
- The parity comparison covers three one-hour windows on three sites.
- The timestamp sample was chosen deterministically from those cohorts.
- The judge-and-human overlap is nine items.
Next step: read the playbook test through the substrate at its end date, and build a judge-agreement measure once enough answers carry both verdicts.
Synthesized Knowledge Publishing for Generative-Engine Discoverability¶
Project and lock¶
Generated answer pages are published so that search and generative engines can find and cite them. A page that publishes a refusal, an off-topic answer or a near-duplicate harms the client's site in a way no engine reports back. The lock this month was whether generated answers could be judged for publication without a human reviewing each page, and how to detect duplicates when the obvious judge, a language model shown the whole page, proved unreliable.
This month's work¶
Each generated answer is now judged on six scored dimensions with a typed verdict (publish, rewrite or reject) whose internal consistency is enforced in code, so a contradictory verdict is itself rejected. An answer gets at most two rewrites driven by the judge's feedback, and any reviewer error, timeout or exhausted budget fails closed. Rejected answers are replaced by lower-ranked question clusters, and a final selection step reviews the survivors together.
Duplicate detection was first given to a language model shown the whole page. On a real French-language run it merged distinct topics into "the same theme" and reduced 47 candidates to 19. Duplicates are now detected by cosine similarity between title embeddings: above 0.85 a candidate is rejected, between 0.70 and 0.85 one pair goes to an adjudicator, and below 0.70 nothing is questioned, with thresholds calibrated on that run. A batch that shrinks by more than 30% against the published one is held and the live page kept. Two latent defects surfaced: the embedding cache had never produced a hit, because a whole run's lookup overflowed a request-size limit, and a tokenizer that only handled unaccented letters had collapsed similarity scores for every non-English client.
The first page for a new client exposed three more defects. Generation and the final selection shared one deadline, so generation used all of it, the selection step failed closed, and all 44 candidates, 24 of them already accepted, were discarded. Every page title rendered the domain rather than the brand. And enabling the feature to preview a page also enrolled the domain in unattended republishing; the two are now separate switches.
Results, proof, and next step¶
After the deadline fix, the first client's page went from no published answers to 18. The quality gate has an evaluation against a frozen batch for the selection step. No crawl, indexing or citation outcome was read for any published page in September, so the gate's effect on discoverability is a pending null. The negative learning is that a model asked to judge similarity across a whole page over-merges, while a geometric threshold calibrated on real candidates does not.
Next step: define an outcome measure for published pages, starting with crawl and indexing of the gated pages, before tuning the gate further.
Non-R&D / Productization Context¶
Product releases included office-format document upload, client-editable pricing guidance and guidelines, rescan page selection, plans, usage and subscription billing, compliance tabs, a redesigned knowledge area, Google sign-up with a guided first-day setup, and richer conversation exports. A per-client switch restricting every model call to European deployments was built for data residency; building it found that query-time embeddings had been leaving the region, which was corrected. Two agent-facing surfaces were built as product work: a tool that lets an AI browser query a client's agent from the page, and a programme in which buyer-side AI agents can find and sign up with vendors.
The rest was client-specific skills and configuration, onboardings, administrative interface changes, a consolidated database migration baseline, CI, dependency and security maintenance, and documentation. The last days of August added the tracing-client upgrade recorded above, the first client-graded evaluation loop, a corrected conversion whitelist, a switch of primary embedding provider, and client-scoped sites and roots.
Prior-Month Results Review (August shipped work)¶
Two of August's research changes reached production: the answer-prompt layout for prefix caching on August 26, and the gated query resolution on August 31. The resolution work was merged on August 28 with the switch enabled for production.
August's replay of the resolution gate over production traffic predicted that 19-23% of retrieval turns would be resolved, depending on the sample, that a third of turns would no longer pay for a rewrite measured to hurt them, and that supplying keywords would spare the retrieval library its own keyword-extraction call. Table 10 compares two weeks before the release with the four weeks after, from production counts of retrieval operations and of query-path model calls.
Table 10. Query-path model calls before and after the resolution release. Counts of production retrieval operations; 95% Wilson interval on the pooled share.
| Window | Retrievals | Resolution or rewrite calls | Library keyword extraction | Query-path calls per retrieval |
|---|---|---|---|---|
| Aug 17-31 (old rewriter) | 5,670 | 3,163 (55.8%) | 4,790 (84.5%) | 1.40 |
| Aug 31-Sept 14 | 9,181 | 2,299 (25.0%) | 5,501 (59.9%) | 0.85 |
| Sept 14-30 | 11,946 | 2,171 (18.2%) | 7,544 (63.2%) | 0.81 |
| Aug 31-Sept 30, pooled | 21,127 | 4,470 (21.2%, 20.6-21.7%) | 13,045 (61.7%) | 0.83 |
The pooled resolved share, 21.2%, falls inside the predicted range, and the share of retrievals that paid for the library's own extraction fell by about 23 points, roughly the share now supplied with keywords. This confirms the predicted population and the cost side of the change. It does not measure retrieval quality, for which no live-traffic signal exists; the quality effect remains August's apparatus result.
The read also changes which August estimate governs the production effect. Of the 4,470 resolutions after the release, 4,448 went to the keyword channel of graph-capable clients and 22 to the passage-only tier. August confirmed the effect on the passage-only tier and left it unresolved on the graph tier, so almost all production firing lands where the benefit was not established. The resolved share also moved from 25.0% to 18.2% between the two post-release windows with no change to the gate, so it depends on the period's traffic mix. Retrieval latency changed too, but the ranking-stage removal, the passage cap, embedding connection pooling and the vector-search tail all overlap it, so no latency effect is attributed to resolution.
The prompt-cache layout stays a pending null: the per-generation records read for this entry carry no cached-token counts, and the fleet-wide hit-rate series was not read. The content-card ordering change shipped in August had no randomized comparison and remains unmeasured.
Research Outcome¶
On the largest test site the agent lowered demo-form submissions by 11.2% at the mid-September read, with an interval excluding zero, narrowing to 8.4% by month end. The conversations read point to deflection more than frustration; a second site showed no detectable effect. That read redefined the metric for conversation policy as form or email among chat starters, and produced a randomized playbook test launched on September 15 with a fixed eight-week horizon. On routing, the durable fixes moved arbitration into the deterministic layer after two measured prompt-level attempts failed, and a faster classifier was not promoted because its score was a development score.
A vendor outage removed the ranking stage and measured it: the loss was confined to conversational flow, about 0.02, and no cap on context volume reproduced it, so the stage's value looks like ordering rather than selection. Six silent knowledge-loss failures were found in production, all sharing one mechanism, and were closed by reading back what the retrieval library actually recorded.
The measurement substrate now has one source of page events, with a measured residual gap of 0.05% of views and 0.43% of exits. Several evaluation instruments were found reporting agreement that was not evidence, and rules replacing them were adopted. Publication of generated pages is now gated, with no discoverability outcome yet measured. In production, the August resolution gate fired on 21.2% of retrieval turns, inside its predicted range, and cut query-path model calls from 1.40 to 0.83 per retrieval; nearly all of that firing is on the graph-capable tier, where its quality effect remains unresolved.