October 2026 - R&D Journal: An Agent Holdout's Final Read, Retrieval Decisions That Can Decline, and What External Assistants Gain From a Client's Agent¶
Context¶
This entry covers October 1-31, 2026, across four research lines.
- Conversation policy on live traffic. A randomized holdout, in which part of one client's visitors did not see the agent, reached its fixed end date on October 13 (Tables 1 and 2). A second randomized test comparing two conversation policies on the same client ran all month and ended on November 10; its result belongs to the November entry. The same protocol was launched on three further websites on October 22 (Table 3).
- Retrieval. Resolving a follow-up question that points at a product named earlier, without inventing a product when the question is searchable as written (Tables 5 to 7).
- Evaluation apparatus. Two offline-judge defects corrected, one conversion definition across every surface, and a first loop from a live answer-quality evaluator to shared prompt fixes (Tables 8 and 9).
- Outbound publishing. Extended from crawled pages to the general-purpose AI assistants buyers now ask directly (Table 10), with the page generator made subordinate to human review (Table 11).
Each result is labelled by source: randomized live-traffic tests, offline apparatus run on real client knowledge, or replays of production traffic that changed nothing in production.
Adaptive Multi-Tenant Conversation Orchestration¶
Project and lock¶
The agent runs on the websites of business-software vendors, and each vendor judges it on leads: a submitted demo form or an email address left in the chat. The open lock is which conversation policy produces more leads without degrading the answers. A policy that answers in full may satisfy a visitor who would otherwise have filled in a form; one that steers towards a demo too early may lose a visitor who is not ready. The effect depends on the interaction between the agent's behaviour, the vendor's sales motion and how the vendor's site measures a conversion, so the state of the art offers no answer.
The question was opened by a holdout on one client, a vendor of staff-scheduling software: from July 17, a fixed fifth of visitors were assigned to a condition in which the agent was not displayed, with a fixed end date of October 13. At the mid-September read the agent condition converted fewer visitors through the demo form. Conversations showed the agent answering the question and proposing no next step, so visitors left with the answer instead of the form. That motivated the conversation-policy test launched on September 15, comparing an information-first and a conversion-first policy among visitors who start a chat.
This month's work¶
Hypotheses¶
- H1. At its fixed end date, the holdout test confirms that showing the agent lowers demo-form conversion on this site.
- H2. The deficit is concentrated among visitors enrolled before the conversion-first policy was introduced into half of the agent's chats on September 15, and is smaller among visitors enrolled afterwards.
H2 was formed after the mid-September read and before this final read. It is reported as an exploratory split, not as a pre-registered comparison.
Experimental setup¶
Holdout. Visitors were assigned on their first eligible visit, sticky per visitor, 80% to the agent condition and 20% to the no-agent condition. The read counts each enrolled visitor once. A visitor converts if a tracked demo form is submitted in any session from their first eligible session up to the end date; the secondary metric also counts an email left in the chat. No eligible session was recorded after October 13. The test's own results store was not closed at the end date and kept counting later conversions, so the bounded read below was computed from the per-session records. Rates are compared with a two-proportion test; relative differences carry a 95% interval on the log ratio.
Replications. The protocol was written as a reusable procedure and launched on three further websites on October 22: an HR software vendor, a payment provider for restaurants, and the platform's own website. Each splits chat starters 50/50 between the information-first and conversion-first policies, server-side and sticky per visitor, with the same primary metric (form or email after the first chat) and a first checkpoint on December 31. Arms, split and metric are locked once a test runs, and only an operator can launch one. Before launch, each site's baseline was measured and every call-to-action destination checked for tracking; the baselines set the expected duration (Table 3). The procedure also narrowed the "email asked" guardrail, which had been read too broadly, to the offer of gated content in exchange for an email, excluding in-chat booking requests.
Policy changes during the running test. Three policy-independent changes, applied to both arms, shipped while the September test ran. The first stops repetition: links, discovery questions and suggested replies already shown in the session are not offered again, and a bare acknowledgement ("ok", "thanks") is its own intent, so the agent moves forward instead of restating its last answer. The second drops qualification questions that do not change the answer: when the knowledge cannot confirm a capability, the agent points to the team instead of asking for the visitor's sector or size. The third moves the support/learning boundary, so an existing customer asking how to use what they own is answered, not routed to support.
Routing under latency and cost constraints. The provider retired the smallest model tier on October 23; it served retrieval keyword extraction and three handler routes, and its replacement was measured before the switch (Table 4). A newer flagship model was evaluated as the answer writer against a pre-set gate: no more than 0.3 s added to the 90th-percentile time to first token.
Isolation. August established that matching tenants by normalised root domain leaks
data when two clients share a root. A review found the same class of rule in two further
paths, writing conversation summaries into a client's CRM and a visitor-field lookup for
enrichment, where comparing the last two domain labels makes every site under a shared
public suffix one tenant. Both now use a nested rule: same host, its www twin, or a
parent-child pair on a label boundary. Separately, the edge rate limiter and the agent
API disagreed on whether a question made only of an invisible character was empty, so a
request could skip the limiter and still run a full agent turn.
Results, proof, and next step¶
Measurements¶
Table 1. Holdout final read, staff-scheduling vendor, July 17 to October 13, 2026.
| Metric | Agent shown | Agent hidden | Difference | 95% interval | Relative | Two-sided p |
|---|---|---|---|---|---|---|
| Demo form | 4,547/306,101 (1.485%) | 1,231/76,306 (1.613%) | −0.128 pts | −0.227 to −0.029 pts | −7.9% | 0.010 |
| Form or email | 4,640/306,101 (1.516%) | 1,231/76,306 (1.613%) | −0.097 pts | −0.197 to +0.002 pts | −6.0% | 0.050 |
The split was 80.05% to 19.95% against a planned 80/20 (z = 0.7).
Table 2. Holdout by enrollment cohort, demo-form conversion.
| Enrolled | Agent shown | Agent hidden | Relative | 95% interval | Two-sided p |
|---|---|---|---|---|---|
| July 17 to September 15 | 3,275/224,271 (1.460%) | 913/55,892 (1.634%) | −10.6% | −16.9% to −3.9% | 0.003 |
| September 15 to October 13 | 1,272/81,830 (1.554%) | 318/20,414 (1.558%) | −0.2% | −11.7% to +12.7% | 0.97 |
With emails counted, the two cohorts read −8.8% (p = 0.013) and +1.9% (p = 0.76). The difference between the cohorts' effects is not significant (z = 1.5, p = 0.14).
Figure 1. Relative difference in conversion, agent shown against agent hidden, with 95% intervals.
Table 3. Replications launched on October 22, 2026.
| Site | Pre-launch baseline, form or email among chat starters | Expected duration for a twofold lift | Enrolled by October 31, conversion / information |
|---|---|---|---|
| HR software vendor | about 1.7% | about 10 weeks | 152 / 142 |
| Payment provider for restaurants | about 3.9% | 4 to 6 months | 44 / 52 |
| Platform's own website | about 4.9% | 4 to 6 months | 42 / 24 |
No outcome was read in October, by design. The 42/24 split on the platform's own site would arise by chance about three times in a hundred at that size; it is a watch item for the sample-ratio check at the first checkpoint.
Table 4. Routing comparisons run before a model change.
| Change | Measure | Incumbent | Candidate | Decision |
|---|---|---|---|---|
| Retiring the smallest tier, retrieval keyword step | Retrieval sufficiency, 43 production turns × 2 runs | reference | −0.012 (−0.038 to +0.013) | switch |
| Keyword step duration | about 1.25 s | about 0.85 s | ||
| End-to-end answer quality, 46 items × 2 runs | 0.833 | 0.825 | ||
| Newer flagship model as answer writer | Estimated uncached writer cost | reference | −20.6% | keep incumbent |
| Pipeline time to first token, p90 | 2.544 s | 4.093 s |
For the policy changes, the end-to-end evaluation moved from 0.837 to 0.846 over 43 items and three repeats with the repetition guards, and the intent classifier held its macro F1 of 1.0 on 121 items. On staging replays, a bare "ok" after an unanswered question restated the previous answer in 3 of 3 runs before the change and moved the conversation forward in 3 of 3 after it; a reported turn that drew an unrelated qualification question in 6 of 6 runs drew none in 6 of 6 after the second change.
Interpretation¶
H1 is supported on the demo-form metric: on this site, showing the agent lowered demo-form conversion by about 8% relative. Counting emails left in the chat narrows the deficit to an interval that touches zero, consistent with the September reading that part of the agent's effect moves leads from the form into the chat.
The cohort split is suggestive and no more: visitors enrolled after September 15 show no deficit, but their interval contains the earlier cohort's effect and the difference between cohorts is not significant. It is also confounded in time. The conversion-first policy reached at most half of the few percent of agent-condition visitors who chat, which alone could not close a gap of this size; the later cohort also had a shorter window to convert, a different seasonal mix, and the policy changes above. Only the policy test, which randomizes the policy among chat starters, can attribute an effect to the policy.
For routing, the smallest tier was retired without measurable loss. The flagship candidate failed its latency gate by a wide margin despite being cheaper, so the incumbent was kept: the answer writer's time to first token, not its price, is the binding constraint.
Threats to validity¶
- One client. The holdout measures one vendor's site, sales motion and tracking; the replications test other buyers.
- Metric construct. A form or an email is a lead, not a sale; sales outcomes were not joined to the test.
- Exploratory split. The cohort comparison was chosen after the mid-September read.
- Results store. The bounded read differs from the stored one by under 2% in counts and does not change the conclusion, but a read from the stored rates would not have been the planned one.
- Concurrent changes. Policy and routing changes shipped during the window and applied to all agent-condition visitors.
Next step: read the conversation-policy test once, at its planned end in November; read the three replications at their first checkpoint; close every randomized test's record at its end date so the stored read is the planned read.
Compounding Context Engine for Company, Industry, and Buyer Intelligence¶
Project and lock¶
August's query-resolution research established a gate: resolve a visitor's turn against the conversation only when it cannot be searched as written, defined as a reply of at most four words not ending with a question mark. That left one hole. A full question that points back at a product the conversation named ("Which retailers already run it?") is searchable in form but not in content, so the gate declines it and retrieval never sees the product. A pattern rule for a few French and English phrasings reopened the gate for some of these turns, missed every paraphrase outside its list, and on graph-capable clients also fired when nothing had been named.
The lock is the one August identified: asked to resolve, a language model cannot decline, so it invents a frame. Asked how many offices the company has, the August resolver turned the question into office space and capacity. A resolver for full questions must therefore decide whether to resolve at all, and that decision sits on the critical path before retrieval.
This month's work¶
Hypotheses¶
- H1. A one-line decision,
RESOLVEwith a product name copied from the conversation,KEEP, orAMBIGUOUS, recovers the referenced product on reference turns without adding a product to standalone or ambiguous turns. - H2. Appending the resolved name to the visitor's words improves the sufficiency of the retrieved context.
- H3. A model-free alternative, appending a line of conversation history, does as well at no cost.
The design is additive, following August's finding that substitutive rewriting loses: the visitor's text is never replaced. Deterministic guards discard a name that does not occur in the conversation, that is the company itself, that is already in the question, or that runs over six words.
Experimental setup¶
Material. Two labelled sets: 36 items on which the decision prompt was tuned, and 21 held-out items from other clients, written after the prompt was frozen. Histories are real production exchanges; visitor messages were written, because paraphrase and agreement variants are too rare to harvest. Standalone and ambiguous turns serve as negative controls.
Procedure. Each strategy runs through real retrieval on real client knowledge, three paired runs per item, on both knowledge tiers: graph-capable clients and the lite tier, which retrieves passages only.
Measures. Context sufficiency is scored 0-10, reported as 0-1, by the August language-model judge, which sees the conversation, the visitor's information need and the retrieved passages. Two deterministic measures: whether the product reached retrieval, and whether a product was added to a standalone or ambiguous turn. Comparisons are paired per item, with bootstrap 95% intervals and an exact sign test on items that changed.
Decision model. A three-billion-parameter model defaulted to AMBIGUOUS, recovering
25 of 44 references and still adding a product to 2 of 20 standalone turns. A larger
small-tier hosted model, reasoning disabled, recovered 41 of 44 and altered none; it was
retained.
Latency. Timed on the real retrieval node, each version twice per item in rotated order after a warm-up pass, on the 80 graph follow-ups the decision sees.
Results, proof, and next step¶
Measurements¶
Table 5. Decision against the pattern rule, three paired runs per item.
| Tier | Items | Sufficiency | Difference | 95% interval | Improved / degraded | Product reached retrieval | Standalone or ambiguous turns altered |
|---|---|---|---|---|---|---|---|
| Graph | 57 | 0.585 → 0.621 | +0.036 | −0.006 to +0.082 | 17 / 7 (p = 0.064) | 17 → 76 of 108 runs | 6 → 0 of 63 |
| Graph, held-out only | 21 | 0.629 → 0.673 | +0.044 | +0.003 to +0.089 | 7 / 1 | 3 → 25 of 36 | 0 → 0 of 27 |
| Lite | 57 | 0.573 → 0.582 | +0.009 | −0.034 to +0.056 | 11 / 13 | 2 → 80 of 108 | 0 → 0 of 63 |
Table 6. Model-free alternatives against the decision, graph tier, 57 items.
| Appended text | Against the decision | 95% interval | Improved / degraded | Product reached retrieval |
|---|---|---|---|---|
| Previous visitor message | −0.070 | −0.132 to −0.013 | 12 / 18 | 51 of 108 |
| Last agent answer, first 220 characters | −0.119 | −0.171 to −0.071 | 4 / 24 | 90 of 108 |
| Six-word summary of the conversation | −0.039 | −0.092 to +0.011 | 8 / 17 | 87 of 108 |
Table 7. Retrieval latency on graph follow-ups, 160 timings per version.
| Version | Mean | p50 | p90 |
|---|---|---|---|
| Retrieval only, no decision | 1.57 s | 1.38 s | 2.34 s |
| Decision, then retrieval | 2.04 s | 1.96 s | 2.80 s |
| Decision in parallel with retrieval | 1.69 s | 1.64 s | 2.67 s |
On the August regression set of real production turns, which contains the office-count item, the decision kept every gate-declined turn in every run, so retrieval input was identical to the pattern rule's.
Interpretation¶
H1 is supported: the product reached retrieval four times as often and no product was invented on any standalone or ambiguous turn. H2 is supported only partly: the gain is positive on both graph sets, with an interval excluding zero on the held-out set alone. On the lite tier the decision's two effects cancel: the name lets retrieval find the passages about that product, but when the knowledge has nothing on the question it pulls in product-description passages instead (one item fell from 0.40 to 0.00 on a price that is not public). The decision was therefore wired on graph-capable clients only. H3 is rejected: every model-free alternative trails the decision and alters every standalone turn that has history; the visitor's previous line often lacks the product, while the agent's last answer carries the product and everything around it.
The latency result decided the integration. The decision costs a median 415 ms and retrieval is the slowest branch before the answer on almost every checked turn, so deciding before retrieval would add its full time to a third of all turns to change a few percent of them. The retrieval node now searches the visitor's words immediately and searches again only when the decision resolves; kept turns cost the same as plain retrieval.
The generalisation: a retrieval-side resolver must be a decision that can decline, with deterministic guards on what it may add, not a generator asked to improve every input.
Threats to validity¶
- Construct. Sufficiency is judged by a model without human labels; pairing protects the comparison between arms, not the absolute score.
- Material. Visitor messages were written, not harvested; the share of real turns like these is estimated in the Prior-Month Results Review.
- Scale. The effect sits on the turns the decision changes, about +0.05 each, and is invisible in any all-traffic aggregate; answer quality was not measured.
A second retrieval hypothesis, shipped at the end of the month¶
On October 30 the chunker changed on the hypothesis that splitting a short document at every heading separates sibling facts, such as two subscription plans on one pricing page, so a question about one product retrieves only one of them. Documents that fit the 512-token chunk limit are now kept whole. In an offline ranking of one restaurant-technology client's 37 documents with the production embedding model, the pricing page went from four chunks to one; for two price questions the whole page ranked first with both plans, where previously one plan ranked first or eighth and the other was absent from the top twenty. The client's corpus fell from 270 chunks to 196. Because chunk identifiers are content hashes, pages indexed under the old splitter carry identifiers the new one no longer produces; the deletion path was rebuilt so a shared chunk survives while any live page still owns it. The change applies only to documents ingested after it, so its first production read belongs to the following month.
Next step: settle the graph-tier sufficiency gain on a few hundred harvested reference turns rather than written ones, and read the single-chunk rule on re-ingested knowledge.
Closed-Loop Agent Evaluation and Optimization¶
Project and lock¶
Every claim in this journal rests on two kinds of instrument: an offline judge that scores the agent's answers on curated datasets, and conversion metrics computed from the event stream. The lock is that both can be wrong in ways that agree with the change under test. August found two such defects in the retrieval harness; October looked for the same class of defect in the answer judge and the conversion definitions.
This month's work¶
Leakage in multi-turn items. In 19 of the 20 multi-turn items of the cross-client end-to-end dataset, the stored history included the turn under test and its reference answer, so replay and judge saw the expected answer before scoring. Both now share one function that cuts the history before the turn under test, and the judge receives that history for every multi-turn item.
A rubric that rewarded the wrong behaviour. The judge's "flow" criterion marked down a complete factual answer that closed without a follow-up question, rewarding the unrelated qualification questions the conversation work above was removing. Rubric version 3 scores flow as progress toward the visitor's request, keeps unrelated qualification and loops low, and accepts an honest limitation only when the reference confirms the gap. It was calibrated before adoption against labelled cases with expected score bands, five of them held out.
From a live evaluator to shared fixes. Since mid-September a language-model evaluator scores a sample of production turns. The lowest-scored turns were ranked and each routed to the layer that should change: the client's knowledge, the client's instructions, or the shared prompt. The first pass found 13 low turns out of 528; the shared-prompt fixes were stated as principles (confirm a specific case only when the context states it; never invent how-to steps; keep a follow-up to a declined off-topic request declined) and replayed at least three times per case. The open question was whether a shared fix detectably moves the live score.
One window for conversion attribution. Three attribution rules coexisted. The impact view credited a conversion to the agent only if the chat started inside the selected date range, so a visitor who chatted the day before the range and converted on its first day counted as converting without the agent. Form submissions outside the chat were credited for seven days after a chat. A demo credited by the batch processor to the visitor's last chat had no limit, so through a one-year cookie a booking could be credited to a conversation months older. All surfaces now credit a conversion within 30 days of the chat's last message, apply the same window to non-converting visitors so the rate is not inflated, and show an estimated final count, projected from the client's own conversion pace, while recent chatters can still convert.
Uncounted widget loads. Between about 13% and 22% of widget script starts, depending on the site, never reached analytics initialisation and so produce no event; the content-delivery network's request logs located the loss in the boot sequence but could not explain it. A temporary diagnostic was built: a page that leaves before initialisation sends one beacon with its last boot step, timing buckets and page-state flags, which the edge answers without storing anything in product analytics. Its reading belongs to the following month.
A repair-verification apparatus. Staff can send edited prompt and skill files with a test question to the deployed agent, which answers from the edits layered over its own files against the client's real knowledge, in a request-scoped overlay that is never cached and is kept out of client analytics. A repair can thus be verified on the real agent before it is merged.
Results, proof, and next step¶
Table 8. Corrections to the offline judge.
| Correction | Measure | Before | After |
|---|---|---|---|
| History cut before the turn under test | Multi-turn items, mean score change, 2 repeats | reference | +0.022 ± 0.016 |
| Rubric version 3 | Labelled calibration cases within their band | 21 of 21 | |
| Flow score on a supplied complete answer, 6 trials | 6/10 in 6 of 6 | 10/10 in 6 of 6 | |
| Cross-client dataset, 43 items, processed without error | 43 of 43, mean 0.851 |
Scores under rubric version 3 are not comparable with earlier runs, so the agent changes made alongside it were judged by their responses and question counts, not by a difference in mean score.
Table 9. Live evaluator before and after the release carrying the shared fixes.
| Period | Scored production turns | Mean score | Scored below 0.5 |
|---|---|---|---|
| September 14 to October 5 | 1,797 | 0.940 | 61 (3.39%) |
| October 7 to October 31 | 1,553 | 0.938 | 51 (3.28%) |
The difference in the low-score share is −0.11 points, 95% interval −1.33 to +1.11 points (p = 0.86): the shared fixes did not detectably move the live rate. Their scope predicts this: they target classes of turn that made up 13 of 528 ranked turns, while most of the population rate comes from turns no shared rule touches. The evaluator also scores 62% of turns at the maximum, which compresses its mean. The evidence for each fix is therefore its replay.
The attribution change altered history for every client. On one client, the 30-day conversions tile fell from 50 to 18 once conversions recorded long after the chat stopped being credited to it. Figures from before and after October 31 are not comparable. The same release introduced a regression in which some dashboards showed zero conversions; its detection and repair fall in the following month.
As in August, both judge defects and the attribution drift were found by reading the instrument's definition, not by running it; the holdout's open results store is a third instance.
Next step: rescore the cross-client dataset under the corrected history and rubric to fix a new baseline; read the widget-boot diagnostic; and evaluate a fix for a class of turns by replaying that class, since the live population rate cannot resolve it.
Synthesized Knowledge Publishing for Generative-Engine Discoverability¶
Project and lock¶
Discoverability by generative engines depends on what they can reach and trust, which the generator does not control. For crawled pages the levers are hosting, canonical addresses and crawlability. For an assistant answering a buyer directly, the question is whether facts a client has not published, or has published where the assistant does not look, reach the assistant at all. The gap depends on what each client's public pages say, so the state of the art cannot answer it for a given client.
A second lock concerns the generator itself. A generated FAQ run is not deterministic: re-running it replaced a published page with a different selection, and a final selection step occasionally discarded valid questions.
This month's work¶
What an assistant gains from the client's agent. Thirty-three questions from the cross-client evaluation dataset that a prospect could put to an outside assistant, covering eight vendors, were asked to two general-purpose coding assistants from different providers, each with live web search, alone and with a connector that lets them question the client's agent. Each question was prefixed with the vendor's name and domain, since an outside assistant does not know which site the visitor was on. Grading was blind and by meaning: answers to each question were shuffled, and graders unaware of assistant or arm scored correctness and completeness from 0 to 10 against a reference, combined into a facts score. Assistants alone were graded against the curated reference answer; assistants with the connector, against the agent's live answer to the same question in the same run. Arms were compared per question with an exact sign test. The two arms therefore had different references, which limits the comparison: the facts scores are not measured against one common answer.
A generator that proposes. A generation run no longer writes the published page; it writes suggestions. A new question is suggested when no published entry covers it and no person has removed or rejected it. A refreshed answer is suggested only when the knowledge has moved past the revision stamped on that answer, and only if a number or the substance changed, or when a person asks; hand-written or hand-edited answers change only on request. A person accepts or rejects each suggestion, and only accepted content is published.
Stable final selection. The final selector reviews a batch of candidate questions and marks duplicates. Sharing a route with the per-candidate reviewer, with no reasoning, it occasionally marked a valid question as a duplicate of an unrelated one, which a priority rule then dropped. It was given its own route with a low reasoning budget. A related failure that could empty a whole language page was removed: a question marked as a duplicate of one that was not kept is now kept.
Pages per language and section. Visitor questions are collected and clustered once, then generated, reviewed and proposed per language; each language page carries its own canonical address and alternate-language links, with a sitemap listing the alternates. A staff-only region code keeps same-language sibling sites from declaring each other as alternates. Published questions can be grouped into four to eight sections by one model call whose plan is rejected in code if it drops or repeats a question. A feasibility study rejected hosting a text copy of each client's website for agents: content-delivery platforms already convert pages for agents, and a copy drifts from the live site.
An instrument for agent traffic. Reaches by AI agents that ask nothing are recorded per client, apart from conversations so nothing is counted twice: fetches of the published FAQ pages by AI user agents, split into user-initiated fetches, search indexing and training crawls, and visits through the agent link. No buyer, question or answer is stored.
Results, proof, and next step¶
Table 10. General-purpose assistants with and without the client's agent, 33 questions, eight vendors.
| Assistant | Fully right alone, against curated answers | Facts score, alone → with agent (each arm against its own reference) | With agent: better / same / worse | Exact sign test | Answers with a major factual error, alone → with agent |
|---|---|---|---|---|---|
| Assistant A | 12 of 33 | 0.73 → 0.80 | 16 / 12 / 5 | p = 0.027 | 1 → 1 |
| Assistant B | 7 of 33 | 0.65 → 0.84 | 25 / 4 / 4 | p = 0.0001 | 5 → 1 |
Alone, both assistants stated most facts correctly but missed the main point on 10 of 33 questions each, where the answer is not public or is buried: rules a client had confirmed but not yet published, what a service covers, a client's own figures. On 4 of the 33 questions the agent's live answer contradicted the curated reference, and in those cases it misled both assistants; excluding them, the gains were larger (Assistant A 0.77 → 0.85, p = 0.001; Assistant B 0.67 → 0.88, p = 0.0005). Assistant B relayed the agent and searched the web on 3 answers of 33; Assistant A blended the agent's answer with web findings and its own estimates and gained less.
Table 11. Final-selection stability, runs with no valid question wrongly dropped.
| Setting | Batch A, 15 runs | Batch B, 15 runs |
|---|---|---|
| No reasoning (before) | 13 | 13 |
| Low reasoning budget | 15 | 15 |
| Medium reasoning budget | 15 | 15 |
The low budget was retained as equally clean and cheaper, at about one to two seconds more per selection call; five consecutive full runs of the selection evaluation then passed all four batches. Section planning, evaluated against a hand grouping of the 53 published questions on the platform's own page, passed 6 of 6 runs with a mean agreement of 89%. On an isolated copy of one client's production data, five real pipeline runs proposed refreshes for 23 of 25 stale answers and left a hand-edited answer untouched; a two-language run proposed 14 English and 16 French questions, and single-language output remained byte-identical to the previous behaviour.
The review-gated pipeline produced no production suggestions before the end of October, so its effect on published pages is a pending null. The agent-traffic instrument began recording on October 27; in five days it recorded 125 fetches of published FAQ pages by AI agents across two sites (110 user-initiated, 10 search indexing, 5 training) and 40 visits through the agent link. That is too short for a trend; it shows the instrument separates the three kinds of fetch.
The study's limits are one answer per question and arm, 2 to 8 questions per vendor, a reference drawn from the same knowledge the agent answers from (the benefit measured, not a bias), and agent answers served from a staging environment. Its most useful negative finding is that, on 4 of 33 questions, a regression of the agent since curation carried over into what the external assistant told the buyer.
Next step: rerun the same questions after each agent change and compare; recheck the four contradicted references against a current cited page; and read the review-gated pipeline and the agent-traffic instrument once both have a month of production data.
Non-R&D / Productization Context¶
Product work in October: a client connector letting AI assistants manage a client's FAQs and documents, summarize conversations, report impact and file support requests; scheduled and whole-day exports; new conversation and intent filters; workspace views; the guided FAQ editor; and marketing pages.
A substantial share of the month went into an agent-facing commerce surface (agent endpoint and discovery files, agent sign-up, claimable accounts, a vendor kit, trial mode, signed vendor calls, offer sheets, and a score of how far an AI agent gets through a vendor's sign-up), recorded as product work.
Also: visitor identification that waits for cookie consent; retrieval-adapter and rescan hardening; a reversible migration to packed binary vectors that halved vector storage per database node; client-specific fixes, dependency and security updates, cost reductions, and maintenance.
Prior-Month Results Review (September shipped work)¶
This section reads only the September changes October's research built on.
Gated query resolution. The August gate reached production at the turn of the month. August predicted from late-August traffic that about 44% of retrieval turns are a visitor's first message, 33-35% are later turns searchable as written, and 19-23% are eligible for the gate. An offline replay of three days of production retrieval turns up to October 7 through the deployed gate gives the realised population (Table 12).
Table 12. Production retrieval turns by gate outcome, 1,628 turns, 20 clients, three days to October 7.
| Bucket | Share of turns |
|---|---|
| First turn, nothing to resolve | 52.0% |
| Short reply, gate fires | 14.3% |
| Later turn on a graph-capable client, searchable as written | 33.1% |
| Later turn on a lite client | 0.6% |
Graph-capable clients carried 97.6% of retrieval turns. The gate fires on fewer turns than predicted, 14.3% against 19-23%. The September entry counted production resolution calls instead and found 21.2% over the month after release, falling from 25.0% to 18.2% between its two windows; the two reads differ in method and period. More importantly, the lite tier, where August confirmed the effect (+0.030 on 325 held-out items), carries only a few percent of production retrieval, while almost all of the gate's firing lands on the graph tier, where August's estimate remained unresolved; the gate's production benefit is bounded by the unresolved estimate. Its cost side holds: the legacy rewriter no longer runs on the third of turns searchable as written. No quality read is possible: the gate reached all clients at once, without randomization, and the live evaluator began scoring two weeks after deployment, so there is no before window.
Removal of the reranking stage. August left the cross-encoder ranking stage in place pending a rank-aware measurement. On September 2, before that instrument was built, it was removed because the provider stopped serving it and every query spent its timeout waiting. The live evaluator started after the removal, and first-token latency moves with too many overlapping changes to isolate this one, so whether the stage earned its cost on live traffic remains open; the September entry's offline measurement bounded the loss from removing it to conversational flow.
Conversation-policy test. Launched on September 15 with a single planned read at its fixed end date; it was not read in October.
Research Outcome¶
Under Adaptive Multi-Tenant Conversation Orchestration, the holdout showed at its planned end that, on one client's site, displaying the agent lowered demo-form conversion by about 8% relative (p = 0.010), and by 6% when emails left in the chat are counted (p = 0.050). The deficit was concentrated in visitors enrolled before the conversion-first policy existed, but that split is exploratory and not significant. The policy test that can attribute an effect to the policy is read in November; three replications on different buyers started on October 22.
Under Compounding Context Engine for Company, Industry, and Buyer Intelligence, a decline-capable reference decision recovered the referenced product four times as often with no invented product on 63 standalone and ambiguous runs. Its sufficiency gain was positive, with an interval excluding zero on the held-out set only, and cancelled out on the lite tier, where it is not used. The August gate fires on 14.3% of production turns, almost all on the tier where its effect is unresolved.
Under Closed-Loop Agent Evaluation and Optimization, the offline judge was corrected for expected-answer leakage and for a rubric that rewarded unrelated follow-up questions, and conversion attribution was unified on a 30-day window. Shared fixes driven by the live evaluator did not move its low-score rate (3.39% to 3.28%, p = 0.86): the population rate is too coarse to verify them, so they are checked by replaying the class of turn they target.
Under Synthesized Knowledge Publishing for Generative-Engine Discoverability, access to the client's agent raised the facts score of both assistants tested, against the agent's live answers (p = 0.0001 and p = 0.027). The page generator now proposes rather than publishes, and its final selection was stabilized; the review-gated pipeline's production effect is a pending null.
