August 2026 - R&D Journal: Retrieval Query Resolution, Measurement Substrate Consolidation, and Prompt-Cache Latency¶
Context¶
This entry covers August 1-27, 2026. The month's dominant research line was a systematic investigation into how a conversational message should be turned into a retrieval query — an uncertainty that June and July had implicitly assumed away. Alongside it, the measurement substrate that feeds every other experiment was consolidated onto a first-party ingestion path, and the answer-generation prompt was restructured around provider-side prefix caching.
No controlled experiment was live in the period. The randomized display experiment designed earlier in the year remained in draft and was not launched, so no randomized result matured this month. The measured results below come from a purpose-built evaluation apparatus rather than from live traffic.
Compounding Context Engine for Company, Industry, and Buyer Intelligence¶
Two faces this month: the query-resolution research that occupied most of the period, and the knowledge-intake tier migration that turned out to define the boundary the research had to respect.
Project and lock¶
The agent answers by retrieving from a per-client knowledge base. Retrieval, however, consumes a single query string: the conversation history is deliberately not forwarded to the retriever, because the retrieval path returns context without generating an answer and would never read it. The consequence is that a visitor turn carrying no standalone meaning — a one-word reply such as a sector name, an unresolved pronoun, or a bare "yes" answering a question the agent had just asked — reaches the index with no context whatsoever.
The platform has always addressed this by rewriting the visitor's message into a self-contained question before retrieval. The lock this month was whether that premise is correct at all: does a language model rewriting a conversational message into standalone prose improve the knowledge retrieved, and if not, what mechanism would? The state of the art offers query rewriting and hypothetical- document expansion as named techniques, but offers no answer to which of them helps a short, multilingual, mid-conversation B2B message against a small per-tenant knowledge base — and, critically, no way to tell whether the technique is helping or harming without an instrument that measures retrieval itself.
A second, related lock surfaced from the intake side. Clients are served under two knowledge tiers, one of which builds a knowledge graph over the ingested corpus and one of which does not. Whether a retrieval strategy transfers across that boundary was unknown, and unmeasured.
This month's work¶
The starting hypothesis was the ordinary one: the rewriter's prompt is poor, so fixing the prompt will improve retrieval. Auditing five hundred production rewrite operations established the defect rates first. Seventy percent of turns never reached the language model at all, falling back to a pattern-matching rule that handles English possessive pronouns only and is therefore inert on French, Spanish and Italian traffic. Of the rewrites that were generated, twenty-four percent were truncated mid-word or mid-number by an output-length ceiling that is adequate for English and not for token-dense languages; fifty-seven percent carried emphasis markup copied out of the assistant's own formatted messages; seventy-four percent were more than four times longer than the visitor's input. The feature added roughly one hundred and fifty milliseconds to the retrieval critical path on a blended per-turn basis.
Those defects turned out not to propagate. Joining each rewrite to the retrieval it produced and to the answer that followed showed no correlation between rewrite quality and either the volume of retrieved material or the rate at which the final answer disclaimed knowledge. Reading the worst cases end to end, the answers were correct: truncation removes the tail of an over-expansion the rewriter itself invented, and the answer writer receives the conversation independently and repairs the rest. That reframed the question from "fix the rewriter" to "does it earn its cost", and it established that final-answer quality is a damped, lagging signal — useless as a metric for this uncertainty.
The investigation therefore required an instrument that did not exist: an evaluation that scores retrieval rather than the answer. The apparatus built for it runs each candidate strategy through real retrieval against real per-client knowledge bases and scores the returned passages with a judge that is shown the conversation, the visitor's actual information need, and the passages — asking only whether those passages let you answer. Three labelled datasets were constructed from real production turns so that every history-and-message pair is one a visitor genuinely produced: a fifty-item general set spanning ten client sites and tagged by turn type; a one-hundred-and-twenty-three-item set restricted to elliptical turns across twenty-two sites; and a one-hundred-and-ten-item held-out set drawn from the same harvest and never used to choose a threshold or a prompt rule.
Building the instrument produced the first negative learning, and it was about the instrument. The initial judging rubric contained an escape clause allowing a message that "needs no knowledge lookup" to score full marks. The judge applied it to elliptical replies — exactly the turns where expansion is mandatory — awarding the do-nothing control a perfect score for retrieving nothing useful. Correcting the rubric to state that an elliptical message still carries an information need supplied by the conversation dropped the control's score on those items from 0.62 to 0.34. Only after that correction did the instrument discriminate in the direction the acceptance criterion required.
With a working instrument, strategies were compared paired per item across repeated runs rather than by comparing means, because the per-item variance is of the same magnitude as the effects being measured. Seven strategies were tested, across two model sizes and both knowledge tiers: the production rewriter; a hygiene-corrected prompt; a deliberately telegraphic rewrite; hypothetical-answer expansion; resolution into retrieval keywords; the same keywords appended to the visitor's own text; and a two-call variant producing both.
Results, proof, and next step¶
The headline result contradicted the starting hypothesis. No prose rewrite of any shape beat simply sending the visitor's raw message. The hygiene-corrected prompt eliminated every defect it targeted — markup failures and truncations both fell to zero — and moved retrieval quality by nothing. Hypothetical-answer expansion lost. Most informatively, running the expansion through a model roughly forty times larger made the result worse, not better: a stronger model writes more fluent invented specifics, and on an embedding channel a fluent wrong passage is more damaging than a crude one. That falsified the second of two mechanistic theories the investigation had advanced, the first being that verbose rewrites inflate the retriever's own keyword extraction — a hypothesis discarded when the correlation between rewrite length and extracted keyword count measured +0.17, essentially none.
The mechanism that finally explained the failures was structural rather than linguistic. The retrieval library exposes two channels: the query text drives vector matching over passages, while a separate pair of keyword lists drives graph matching over entities and relations. When keywords are not supplied, the library spends its own model call deriving them from whatever text it is given. The rewriter was therefore composing prose for a consumer that immediately re-extracts keywords from it — a lossy round trip whose output the platform never controlled. Every prompt improvement was being applied on the wrong side of that conversion.
Supplying resolved keywords directly, and leaving the visitor's own words to drive the vector side, initially measured +0.050 against the control with sixteen items improved against three degraded. That figure did not survive scrutiny of the instrument. A review found the judge was being shown the resolved keywords appended to the query text for the keyword strategies, while the control had no equivalent annotation — a difference between strategies that is not a difference in the retrieved passages the judge is asked to score. With the annotation removed and every strategy shown identical text, resolving each turn measured indistinguishable from sending the raw message.
Recovering where the effect had come from produced the more durable finding. The gain was concentrated entirely on elliptical turns; the loss entirely on turns whose raw message was already a well-formed query, including a group whose only pronoun is a possessive naming the company, which retrieval already supplies. On those the resolver cannot decline, so it invents an interpretation: asked how many offices the company has, it resolved the question to office space and capacity, and the passage stating the office count left the result set altogether. Applying the same short-reply gate developed for the graph-less tier to the keyword channel recovered most of that loss while keeping the elliptical gain intact.
Running the two resolution routes against each other then settled which half of the design carries the effect. On graph-capable clients, resolving into keywords for the graph channel and resolving the query text for the passage channel measured within a thousandth of each other, with wins and losses evenly split. Several rounds of work had treated the channel as the discovery and the gate as a refinement; the comparison says the reverse. The gate is the intervention. The channel decides only what a resolution costs — supplying keywords spares the retrieval library its own extraction call — not whether it helps.
The losses themselves turned out to originate after retrieval rather than in it. Passage candidates are identical between the control and the keyword strategy by construction, since the visitor's raw message drives that channel in both. What differs is the graph candidates, and the platform reranks entities, relations and passages together in a single pool against one query before truncating to a fixed budget. A graph result that ranks well therefore evicts a passage that would otherwise have been kept. On one turn, where a visitor stated they were importing a spreadsheet of contacts, both arms returned the same thirty documents and the step-by-step import guide that answered the question was displaced by accurate but unhelpful entities about the surrounding product area; the score fell from 0.80 to 0.30. Identical volume, different content — which is also why the count of retrieved documents cannot be used as a quality signal, a check that had looked reassuring in the opening audit.
The intuitive account of that harm was tested and failed. The story had been that resolution dilutes by adding terms the visitor did not ask about; grouping the resolved turns by how many resolved terms were absent from the visitor's own message shows the opposite, with the most heavily augmented turns scoring best. The harm is not a function of how much is added but of how well the raw message already worked: on turns whose unresolved query was already scoring well the intervention is net negative, and on turns carrying no searchable content of their own it is positive. There is nothing to gain when retrieval is already succeeding, and whatever is added competes for a fixed budget.
The tier boundary then invalidated that result for part of the client base. On the tier without a knowledge graph, retrieval consists of passage matching alone, so supplied keywords are ignored outright. Forcing the evaluation into that mode showed the production rewriter to be actively harmful there — the tier paying least was getting the worst of it — and appeared to produce a calibration of the instrument: a strategy that provably emits a byte-identical query on every item still drifted by +0.012, which was read as the judge's noise floor. It was not. The drift was the display annotation described above, and removing it removed the drift. The floor is now established from turns the gate declines, whose retrieval input is identical to the control's by construction; those move by +0.003. The distinction matters because the earlier, larger figure had been used to argue that a competing result was several times the noise.
Solving the graph-less tier required isolating its worst regime. Elliptical turns there score 0.28-0.37 against a general-set baseline above 0.50 — not a weak spot within retrieval but a separate, harder regime. Reading those turns showed that every strategy tested had resolved the subject of the reply and discarded the request: when the agent offers a customer example and the visitor answers "yes", the rewrites all produced a question about the topic rather than a request for an example. Resolving the request rather than the topic helped substantially on the dominant sub-mode, and harmed a second sub-mode badly — turns where the visitor ignored the question and raised something new, where the raw message is already a well-formed query that resolution damages.
That split turned out to be predictable from reply length alone: one-word replies gain most, five-word replies lose, and a five-word elliptical reply is usually a real question. Gating resolution on a short reply that does not end in a question mark — two lines, no classifier, no additional model call — tripled the measured effect and, by construction, closed the harmful sub-mode entirely rather than relying on the model to behave. Rewriting the resolver's prompt against the measured failure features, and enforcing its constraints in code rather than merely requesting them, halved the variance and cut mean output length from 7.6 to 3.7 words.
The final read is on the held-out set, after four methodology corrections: the display annotation was removed so every strategy is judged on identical text; bootstrap confidence intervals and an exact sign test replaced standard-error reporting; the noise floor is measured by re-running the whole retrieval and judging path on identical input rather than by re-scoring the same passages, which prices only the judge and understates it; and the turns the gate declines are read as a control built into each run, since their retrieval input is identical to the control's by construction.
Under the corrected instrument the gated resolver scored +0.046 on eighty-three judged items, with a confidence interval excluding zero. That figure could not be trusted on its own: a power calculation showed eighty-three items resolve only an effect of about 0.05, while the effect under test was near 0.03, so the estimate sat exactly at its own detection threshold — the regime in which effects are systematically overstated, because they are only reported when noise happens to push them over.
Additional runs could not fix this. Decomposing the variance showed run-to-run noise contributes four percent and item-to-item difference ninety-six, so doubling or quadrupling the runs changes the standard error in the third decimal. Only more items help. A second held-out set of two hundred and forty-two judged turns was therefore harvested and de-duplicated against every earlier set.
Harvesting evaluation material from real conversations is what makes the instrument faithful, and it is also what makes the corpus a personal-data surface. Reviewing the first three sets found sixteen visitor email addresses carried verbatim into the corpus, one belonging to a visitor who had stated in the conversation that they were ten years old. The addresses were replaced with synthetic equivalents on reserved domains that preserve the shape of the original — dotted, run-together, digit-laden — so each item still exercises the same path; the minor's message was rewritten to a neutral request that preserves the item's label; and a mechanical check was added so that a future harvest fails rather than leaks. The larger harvest was then prepared under the stricter rule that failure taught: substitutions applied consistently, so a repeated correspondent remains one person across the corpus; turns disclosing an age or a telephone number dropped rather than rewritten; and turns carrying a visitor-typed personal name removed by reading them, since no pattern detects those reliably. Preparing the sample this way is part of the experiment rather than an administrative step beside it — it is what allows real conversational material to serve as the test corpus at all. From roughly three thousand production resolution operations the harvest found a thousand distinct elliptical turns, of which two hundred and sixty-two were selected across twenty-seven client sites, capped per site so that no single client could dominate, and de-duplicated against the earlier sets.
Both estimates fell on the larger sample, as the power analysis predicted. The graph-tier effect halved and remains unresolved; a third labelled set of a hundred and five judged turns had already returned +0.004 for that tier, with the control built into that same run drifting further than the effect it was meant to qualify. Three independent sets — one positive, two null — put the honest estimate for graph-capable clients close to zero before the powered measurement was run, and the powered measurement did not overturn that. The chunks-only effect fell from +0.046 to +0.025 but survived: pooled across three hundred and twenty-five held-out items it is +0.030 with a confidence interval of [+0.014, +0.047], seventy-three improvements against thirty-six, and an exact sign test at p=0.001. The effect now exceeds the detection threshold rather than sitting on it. Where the gate fires it is +0.054. Two independent controls support the reading: turns the gate declines are byte-identical to the control by construction and moved -0.001, and the sub-mode where resolution previously cost -0.23 now moves by zero across forty turns.
Four hypotheses were falsified in sequence, each with its own measurement: that the prompt was the problem; that verbose rewrites fan out the graph budget; that resolver precision was the ceiling; and that the format of the enrichment was the ceiling. The surviving generalisation is that additive strategies beat substitutive ones — the visitor's own words are the highest-precision signal available, and every rewrite trades that away for context, a trade that never pays on the passage channel. A complementary mechanism was recorded alongside it — that a wrong keyword on the graph channel merely fails to match an entity and is therefore harmless, while the same keyword on the embedding channel drags the vector off target. Re-measurement falsified the harmless half. Graph search spends a fixed budget of entities and relations, so a wrong keyword does not fail quietly; it matches something else and displaces what the raw message was already retrieving. The corrected statement is narrower and more useful: on both channels resolution pays only where the visitor's message cannot be searched as written, which is exactly what the gate encodes.
The merge described above pointed at one last mechanism, and at the most promising hypothesis of the period. If the final selection is decided by a cross-encoder scoring every candidate against the visitor's raw message, then on a two-word turn that ranker has almost nothing to discriminate with, and it holds the last word over what survives. Probing it directly on real turns confirmed the premise: on the most contentless messages its scores across a fixed document set spread by less than five hundredths, and giving it the resolved context instead roughly tripled that spread and changed eight of the twenty documents it selected. The prediction followed cleanly — enrich the ranking query while leaving the embedded text untouched, since that channel punishes imprecision and the ranking stage evidently does not.
It did not pay. Enriching the ranking query alone measured +0.019 against the control, and adding it on top of keyword resolution moved the result by a thousandth with wins and losses exactly tied. Three mechanically unrelated interventions — resolving into graph keywords, resolving the query text, enriching only the ranking query — all land between +0.01 and +0.03 with intervals crossing zero, and they do not compose. The generalisation that survives is about none of them: once the conversation's context has been injected anywhere in the retrieval path, injecting it somewhere else as well buys nothing. The ceiling is how much a two-word turn can be made to carry, not which channel carries it. That was the fifth mechanistic account falsified in the period, and it ended the search for a sixth. The plumbing was kept, defaulting to the existing behaviour, because it is the apparatus that makes the negative result reproducible.
The same probe raised a question the apparatus cannot answer. Disabling reranking altogether, which falls back to a fixed allocation across document types, moved the score by -0.011 to -0.022 across two sets with every interval crossing zero, and the split by turn type reversed the intuition: the ranking stage helped most on exactly the elliptical turns where it had been shown to be nearly blind, and was mildly negative on well-formed questions. The reading is not that reranking is worthless but that the metric is the wrong shape for the question. Context sufficiency asks whether the material needed to answer is somewhere among the roughly thirty documents returned; ranking decides in what order it appears, and a passage moved from last place to first scores identically here while mattering a great deal to an answer writer reading top-down under a token budget. Settling whether that call earns its cost requires a rank-aware measurement — the position of the first relevant passage, or precision within the first few — which does not exist. Building it is the outstanding instrument, and until it exists the ranking stage is left in place rather than removed on evidence incapable of supporting the removal.
A second blind spot was structural, and sat in the instrument's own design. Turns that need no retrieval at all — an address typed into a capture step, a farewell, a nonsense string — are excluded from every judged mean by construction, since there is no information need to score. They are therefore the one class the apparatus never measured. Reading them directly showed the resolver inventing plausible topical keywords for roughly two-thirds of them, drawn from the conversation rather than from the message, and the retrieval task is started on every turn regardless of which route the conversation takes, so those turns pay for the call as well. The retrieval library's own extraction prompt carries an explicit rule for exactly this case, instructing the model to return nothing for a greeting or a nonsense string; the resolver built here does not, and its fallback triggers only on an unparsable response, so a confidently wrong resolution passes through unchallenged. Closing that gap alters every resolution and therefore needs its own measurement; it is recorded as an open defect rather than patched at the end of a measurement campaign.
Because the decision to enable the path turns on the tail rather than the mean, the resolver's output was audited for the failure modes a mean cannot show. Three hundred and sixty real resolutions produced by the gate were scanned for content that must never reach a retrieval query — addresses, links, instruction text, fragments of the resolver's own reasoning — and none appeared. Short adversarial inputs, including attempts to override the model's instructions, a database statement and a script fragment, all pass the gate by shape, all reach the model, and all returned ordinary topical keywords: the blast radius of a bad resolution is one turn of worse retrieval, and the output is never shown to the visitor. One genuine defect did surface — emphasis markers around a single word survived a cleaning rule written for the doubled form, so about three percent of resolutions carried punctuation into the graph query — and was fixed with a test.
A measured effect on a held-out corpus says nothing about how often the intervention would fire, so the gate was replayed over production retrieval turns to size its eligible population. Slightly under half of retrieval turns are a visitor's first message, where there is no conversation to resolve from and nothing changes. A third are later turns whose message is already searchable as written — the population on which the existing rewriter runs and on which it measures harmful — and there the gate's effect is to stop the call entirely. About one turn in five is gate-eligible. That replay also corrected a figure from the opening audit, which had put first turns at seventy percent; over a fourteen-day window it is closer to forty-four, and the earlier figure should not be relied on. Weighting the held-out win and loss rates by that population puts the expected effect at roughly seven turns in a hundred retrieving better material and two retrieving worse, with a third of all retrieval turns no longer paying for a call measured to hurt them. The largest single block is subtraction, not addition, which is the honest case for the change and a different case from the one the investigation set out to make.
Two findings from outside the measurement bear on what can be claimed next. The first is a defect exposed by unrelated work on an internal preview surface that displays which knowledge fed an answer: the map from commercial tier to retrieval mode routed the higher tier to a mode that fetches graph context only and no document passages at all, so a client with an explicit tier retrieved strictly less than a client with none, which falls back to the environment default. Correcting the map restored passages for those clients — one site went from none to eighteen on the same question — and the provenance of graph facts, which the library strips while truncating, was re-attached from the untruncated originals the same call returns. The correction landed after most of the graph-tier measurement had been taken at each client's own default, which is a further reason that tier's estimate is treated as unresolved rather than merely small.
The second is that an assumption held throughout the investigation was wrong. The switch governing this work is read from a per-environment configuration file rather than from the per-client configuration store, and no environment-variable override exists for it. There is therefore no per-client canary and no runtime rollback: enabling it is a deployment, and it applies to every client in that environment simultaneously. The staged rollout assumed in every earlier plan — expose one client, read its traces, widen — is not available until the switch is moved into the per-client store. The switch was nonetheless turned on in the configuration of every environment, production included, and the path exercised end to end in a live local session — the configuration change sits on an undeployed branch, so no visitor traffic has reached it. That session confirmed the mechanism and not the benefit: four turns in five were declined by the gate and reached retrieval byte-identical to what the visitor typed, and the turn that fired produced two keyword lists that drove the graph queries while the visitor's own two words drove the passage query. A handful of conversations sits far inside the noise of an effect this size, so nothing about quality follows from it.
One methodological result deserves recording on its own. Both defects found in the instrument — the rubric escape clause and the strategy-specific annotation — were found by reviewing the harness, not by running it, and both had inflated the result in the direction of the hypothesis under test. The rule adopted from that is that any result supporting the thing being built is re-measured after any change to the harness, and it was written into the team's standing evaluation guidance rather than left as an observation in one investigation.
Two limits bound all of it. The metric is self-consistent rather than ground-truth: no human labelled these items, so pairing protects the comparison between strategies but not the absolute score, and a subset should be hand- labelled before any sufficiency figure is treated as an operational target. And the work stayed on a development branch. Its environment configuration was enabled but not deployed, so it did not reach production; there is no post-deployment window to read and no production effect can be claimed. The measured retrieval improvements are results from the experiment apparatus, not from live traffic.
Running alongside, the knowledge-intake work made migration between the two knowledge tiers safe, retired the staged rollout gates that had guarded ingestion, extended access to migrated sources for domain-scoped users, and added website source rescans with propagating deletions. That boundary is no longer an operational detail: the retrieval research established that it is behaviourally load-bearing, because it decides which retrieval channels exist at all. What it does not decide is the intervention. The strategy that wins was initially thought to be tier-specific — one for each side of the boundary — and re-measurement showed instead that a single rule works on both, with the tier affecting only which channel carries it and what that channel costs.
Next step: read the first post-deployment window for the resolution path, accepting that it will be an environment-wide read rather than a canary until the switch moves into the per-client configuration store; build the rank-aware retrieval measurement that would let the ranking stage be decided on evidence instead of left in place by default; close the no-retrieval gap and re-measure, since it changes every resolution; and hand-label a subset of the held-out items to check judge agreement before any sufficiency figure is treated as an operating target.
Closed-Loop Agent Evaluation and Optimization¶
Project and lock¶
Every experimental conclusion the platform draws rests on the event stream that records what visitors did. June established precomputed experiment marts on top of that stream; the lock that remained was the stream itself. Product events were being delivered through a third-party webhook path that was lossy in ways that were hard to characterise, expensive in storage, and — because a shadow copy was maintained in parallel — carried two representations of the same events that could disagree. A measurement substrate that can silently disagree with itself cannot support causal claims, and the marts built on it inherit the defect.
A second, narrower uncertainty concerned the definition of conversion. Attributing a conversion to the interaction that caused it is not mechanical when several signals can follow a single session, and a definition that drifts between surfaces reproduces the metric-drift problem the marts were built to remove.
This month's work¶
The hypothesis was that a first-party ingestion path with canonical projections would be both more faithful and materially cheaper than the webhook-plus-shadow arrangement, and that the two could be compared before the old path was retired. The work proceeded in that order: first-party ingestion of page events was built and run in parallel, canonical projections of those page events were derived so that the two representations could be compared on identical ground, and only once the projections were trusted was the webhook path retired and the shadow system removed. Retiring it reclaimed roughly eighteen gigabytes on the session event store — the cost side of the same decision.
Two hardening items belong to the same substrate. Client-side event capture was found to poison its own rate-limit state under some conditions, dropping events silently; the recovery path for that was added. And conversion attribution was corrected so that a conversion is credited to the interaction responsible for it and client-specific conversion filters are enforced consistently rather than applied on some surfaces only.
Instrumentation cost was addressed in the same period: metric export was made cheaper and histogram resolution corrected, and export was blocked entirely on local runs so that development traffic can no longer contaminate the series that experiments are read from.
The evaluation harness on which the month's retrieval investigation depended was hardened in the same period, and two of its defects were of the kind that silently manufactures agreement. A run whose individual items raised exceptions still reported its mean as a pass, so a partial run could be read as a complete one; runs now carry the number of items they were expected to cover and are read as failed unless every one completed. And the single pass threshold applied to every dataset was uncalibrated and unreachable, because the judged score is a property of a dataset's difficulty rather than of the strategy under test — the hardest held-out set's do-nothing control sits far below the general set's, so one number is simultaneously vacuous on one and impossible on the other. Thresholds are now set per dataset, each about one standard error below that dataset's own measured control, which answers the only question a single run can honestly decide: did this strategy retrieve worse than not intervening at all? Ranking two strategies against each other still requires paired per-item comparison across runs, and that limitation is now stated wherever the threshold is read. Runs and their scores are also persisted rather than printed, so a claimed measurement leaves a durable record instead of living in a terminal.
Results, proof, and next step¶
The substantive learning is that a shadow representation maintained for safety becomes a liability once it is trusted: two copies of the same events invite disagreement, and the cost of keeping both is not only storage. Running the first-party path in parallel long enough to compare projections, rather than switching over, is what made retirement defensible — but the comparison itself was possible only because canonical projections were built specifically to put both representations on the same footing.
The negative learning is that client-side capture can fail silently through its own protective mechanism, which means an absent metric is not evidence of absent behaviour. That failure mode was invisible to every dashboard built on the stream.
No controlled experiment was live in the period, so the substrate work has not yet been exercised by a randomized measurement — its value remains prospective. The conversion-attribution correction changes a metric definition, so figures computed before and after it are not comparable, and any trend crossing that boundary must be read with the change stated.
Next step: run the randomized display experiment that has remained in draft, now that the substrate feeding it is single-sourced, and confirm that the marts reproduce its result independently.
Adaptive Multi-Tenant Conversation Orchestration¶
Project and lock¶
The answer-generation call dominates perceived latency and cost, and the platform has no control over the provider's inference speed. The remaining lever is the structure of the prompt itself. Providers cache a prompt prefix and charge less for a cache hit, but the cache keys on an exact prefix, so anything variable placed early — a per-visitor detail, a timestamp, a per-turn state field — invalidates the cache for every request that follows it. Which parts of a multi-tenant conversational prompt are genuinely stable across turns and across visitors, and how much is recoverable by reordering rather than by shortening, was not established.
The month also carried a multi-tenant isolation lock. Configuration and knowledge are scoped per client, but the scoping key is not always the obvious one: several clients can share a root domain, so any access decision that normalises to the root grants one client's data to another.
This month's work¶
The prompt was restructured so that stable content precedes variable content, with cache-hit telemetry added at the same time — without which a reordering cannot be verified, since a cache hit is otherwise indistinguishable from a fast response. Conversation history was moved from an inlined transcript to a replayed message array positioned ahead of the variable segment, which serves the same prefix-stability goal.
Model capability checks were re-keyed on provider family rather than on individual model identifiers, so that routing decisions survive model version changes instead of silently falling back when an unrecognised identifier appears — a recurring failure mode when several providers are in rotation.
On the isolation side, configuration writes were made to fail rather than overwrite when the underlying record has changed, closing a path by which a stale editor could write one tenant's values over another's; team management in the administrative interface was scoped to the selected client; access to migrated knowledge sources was granted on exact domain rather than normalised root; and at the storage layer, anonymous read access to client configuration and a set of stored procedures reachable across tenant boundaries were closed, so that the scoping decided in the application is not bypassable underneath it.
A change to the ordering of content cards relative to questions was also shipped this month; it is a behavioural change to the conversation surface, and no randomized comparison was run for it.
Results, proof, and next step¶
The learning from the prompt work is that prefix caching converts prompt layout into a cost and latency variable independent of prompt length — the same tokens, reordered, price differently. Its telemetry is in place but the change shipped late in the period, so the post-deployment window is too short to read; the cache-hit rate is the metric that will settle it and is not yet reportable.
The isolation learning is that tenant scoping must resolve the exact identifier that owns the data, and that root-domain normalisation is an attractive convenience which silently becomes a cross-tenant leak as soon as two clients share a root. That case is not hypothetical in the current client base.
The content-card ordering change has no measured effect and is carried as a pending null; it is a candidate for the randomized comparison the substrate work above is intended to support.
Next step: read the cache-hit telemetry once a full post-deployment window has accumulated, and quantify what share of the answer prompt is genuinely stable across turns.
Non-R&D / Productization Context¶
The August product releases include per-language qualification answer options, additional widget placeholder translations, regional-locale and alias-specific call-to-action configuration, a switch of administrative sign-in from magic links to one-time email codes, bot protection on self-serve signup, a staff insights page with a weekly report workflow, per-developer activity charts, and a series of onboarding and administrative-interface layout fixes.
The month also carried routine operational work: client-specific agent skill and configuration edits for individual accounts, chatbot triage fixes scoped to single clients, dependency and formatting maintenance, documentation and changelog updates, and administrative-interface polish. The closing days added a consolidated knowledge-source menu, simplified brand and competitor configuration screens, automatic activation of the free plan after onboarding, a single switch governing which environment tests and evaluations resolve against, and a deployment fix for the marketing site. The internal preview panel that lists the knowledge retrieved for each answer is a product surface in its own right; the two retrieval defects it exposed are recorded above, in the project they belong to.
Prior-Month Results Review (July shipped work)¶
No controlled experiment was live in July or August, and the randomized display experiment designed earlier in the year remained in draft, so no randomized result matured for review in this period.
The July journal entry has not yet been written, so July's shipped work carries no stated hypothesis for a post-deployment read to be tied back to. Constructing a before-and-after on July's changes without that stated hypothesis would produce numbers without a claim to test, so none are reported here. Completing the July entry, and reading its shipped changes against the substrate consolidated in August, is the outstanding item for the next entry.
Research Outcome¶
As of August 27, the month's deepest investigation is the retrieval query-resolution work under Compounding Context Engine for Company, Industry, and Buyer Intelligence. It produced a measurement instrument that did not previously exist, four labelled datasets including two held-out sets, five explicitly falsified mechanistic hypotheses, a reproducible noise floor, and a replicated out-of-sample effect on the hardest class of conversational turn — together with two general findings: that additive query strategies outperform substitutive ones on embedding-based retrieval, and that once conversational context has been injected anywhere in the retrieval path, injecting it elsewhere as well adds nothing. It also produced two bounded negatives worth as much as the positive result: the graph-capable tier's effect is indistinguishable from zero across three sets, and whether the reranking stage earns its cost cannot be decided by this apparatus at all, because context sufficiency is blind to ordering. A rank-aware measurement is the instrument that would settle it and does not yet exist. The work did not reach production, so its production effect is a pending null, and the per-environment scope of its switch means the first production read will be environment-wide rather than a canary.
The measurement-substrate consolidation under Closed-Loop Agent Evaluation and Optimization is complete on the engineering side and unexercised on the experimental side: it removes a known source of disagreement between two representations of the same events, but no randomized measurement has yet run through it. Prompt-cache layout under Adaptive Multi-Tenant Conversation Orchestration is instrumented and shipped too late in the period to read.