November 2026 - R&D Journal: Retrieving Facts Documented Away From the Subject Named¶
Context¶
This entry for November 2026 is in progress and will be completed, with the month's other research lines, when the month closes. Its first research line asks whether an agent can answer a factual question from a client's own documents, without a curated answer added after the error was noticed.
No controlled experiment was live for this work, and none of it reached production. The measured results below come from an isolated copy of one client's knowledge base and from test runs against that client's live knowledge. They are apparatus results, not results from live traffic. Tables 1 to 5 describe the prototype. Table 6 and the update that follows it describe the later version.
Compounding Context Engine for Company, Industry, and Buyer Intelligence¶
Project and lock¶
The test case was a restaurant-technology client that sells two subscription plans, both listed on one pricing page. Asked what a given product costs, the agent often failed to name both plans. A curated FAQ entry and a client-specific instruction containing the prices had been added to correct this. Both were scheduled for removal once better document chunking (the splitting of documents into the passages that retrieval searches) had shipped. Such copies carry a cost of their own: when the client changes a price, each copy goes stale.
The lock is more general than this one page. Vector retrieval embeds the visitor's whole question as one point, so a question that names a subject and asks for a fact is pulled towards whichever passage best matches the combined meaning. When the fact is documented away from the subject's own page, the subject can outweigh the fact. The August query-resolution work found that additive query strategies beat substitutive ones and that the graph-capable knowledge tier's resolution effect was indistinguishable from zero. Neither finding says how to retrieve a fact that sits apart from the subject named, nor whether graph retrieval, which searches on separate keywords, already solves that.
This month's work¶
Hypotheses¶
Three explanations were tested in turn, each after the previous one fell short.
- H1, chunking. The two plans sit in separate passages, so keeping sibling sections together will make both reach the agent. The page measured about 800 tokens, above the 512-token limit for keeping a document whole. A sibling-section merge was built: sections that share a parent heading and fit the limit together form one passage.
- H2, retrieval limits. The pricing page is cut off by a limit on graph retrieval. An offline ranking with the production embedding model and vector search only placed the pricing page's best passage 15th for "What is the price of the digital menu?", with similarities of 0.66 for the product's page and 0.46 for the pricing page. Only ten passages reach the agent.
- H3, query composition. The fact asked for needs its own search. Chunk searches were added alongside the existing searches, in the same parallel step, first using the keywords the retrieval library already extracts for its graph. Those keywords are worded for the graph, so "Pricing" found the page while "Price inquiry" and "Payment question" often did not. The final variant adds one field to the existing keyword-extraction call: the kind of information requested, in the plain words a document would use as a heading, with the subject excluded. It needs no separate language-model call, and the prototype used no word list. A later version maps six English price words to one topic (Table 6).
Experimental setup¶
Material. The client's 89 production documents for one market, loaded into an isolated copy with the knowledge graph enabled and the curated FAQ absent. The client instruction containing prices was blanked in the first chunking test and replaced by a price-free one in all later tests. Each chunking change required a rebuild, so builds are identified in Table 2. The live-knowledge test in Table 1 read production knowledge in preview mode with the FAQ withheld and saved nothing.
Questions. Twelve price questions, most naming a product, plus four questions on other facts documented on the pricing page (go-live time and support hours). Twelve new price phrasings served as an out-of-sample check on the final variant.
Procedure. Each run was a new single-turn conversation. Questions were repeated three times per condition, or twice in the first chunking test and the live-knowledge test.
Measures. Early experiments scored only the answer text, which hid the cause of each failure. A dedicated harness was then built for this experiment. It records the exact documents handed to the agent on each turn and scores two outcomes separately:
- Retrieved: both price amounts appear in the passages handed to the agent.
- Cited: both amounts appear in the answer.
Analysis. Table 2 gives 95% Wilson intervals over runs. These treat runs as independent, which repeats of one question are not, so they understate the uncertainty. Table 3 therefore compares conditions at question level with a two-sided exact sign test, dropping ties. Evaluation scores in Table 5 are the mean over the client's thirty-six-item end-to-end set, scored by a language-model judge with a pass mark of 0.78.
Results, proof, and next step¶
Measurements¶
Table 1 reports the experiments scored on answer text. H1 was only partly supported, and the limits in H2 had no effect, with each setting confirmed live in the logs. Widening the graph search raised the entities in the final context from 21 to 33 on one inspected turn, but relationships were cut to the same 24 by their token budget. Adding an explicit search term raised the live-knowledge score from 12 of 24 to 24 of 24, though that term was also visible to the answer writer. Searching all keywords together did no better than searching none, because the combined query mixed the product back in. Reciprocal rank fusion, the usual way of combining ranked lists, scored below fixed slots.
Table 1. Answers naming both plans, by experiment.
| Experiment | Condition | Price questions | Other questions |
|---|---|---|---|
| H1, chunking | No merge | 10/24 | not run |
| Merge, first build | 16/24 | not run | |
| Merge, second build | 14/24 | not run | |
| H2, graph limits | Search width 10, cutoff 0.6 | 15/36 | 9/12 |
| Search width 20 | 15/36 | 9/12 | |
| Cutoff 0.4 | 15/36 | 9/12 | |
| H3, live knowledge | Question as asked | 12/24 | not run |
| Question plus "pricing plans" | 24/24 | not run | |
| H3, search variants | Search off | 14/36 | 9/12 |
| All keywords as one query | 12/36 | 9/12 | |
| Each keyword alone, reserved slots | 27/36 | 9/12 | |
| Each keyword alone, 15 passages | 27/36 | 9/12 | |
| Each keyword alone, rank fusion | 21/36 | 9/12 |
Table 2 and Figure 1 report the separated measures. Across all 288 harness runs, the answer cited both plans exactly when retrieval had supplied them, with no exception in either direction. Retrieval therefore accounts for every failure in this set.
Table 2. Retrieval and citation of both plans, twelve questions, three runs each. Build A has no merge; builds B and C have the merge.
| Condition | Build | Retrieved | 95% interval | Cited | Only one plan retrieved | Questions at 3/3 |
|---|---|---|---|---|---|---|
| Existing configuration | A | 15/36 | 27-58% | 15/36 | 9 | 5/12 |
| Sibling merge | B | 18/36 | 34-66% | 18/36 | 0 | 6/12 |
| Graph keywords searched alone | A | 29/36 | 65-90% | 29/36 | 1 | 9/12 |
| Graph keywords searched alone, plus merge | B | 27/36 | 59-86% | 27/36 | 0 | 9/12 |
| Plain topic | A | 35/36 | 86-100% | 35/36 | 0 | 11/12 |
| Plain topic, repeat | A | 35/36 | 86-100% | 35/36 | 0 | 11/12 |
| Plain topic, new phrasings | A | 33/36 | 78-97% | 33/36 | 2 | 10/12 |
| Plain topic, plus merge | C | 36/36 | 90-100% | 36/36 | 0 | 12/12 |
Figure 1. Runs with both plans retrieved, per condition, with 95% Wilson intervals.
At question level (Table 3), the plain-topic search improved 7 of the 12 questions against the existing configuration and degraded none; the other 5 already passed every run. The result is p = 0.016 unadjusted and p = 0.09 after Bonferroni correction for the six comparisons, so it does not reach the conventional threshold on this sample. Table 4 shows where each condition gained and lost.
Table 3. Paired comparison by question, 12 questions.
| Comparison | Improved | Degraded | Tied | Exact sign test, p |
|---|---|---|---|---|
| Existing configuration → sibling merge | 1 | 0 | 11 | 1.00 |
| Existing configuration → graph keywords | 5 | 0 | 7 | 0.06 |
| Existing configuration → plain topic | 7 | 0 | 5 | 0.016 |
| Graph keywords → plain topic | 3 | 0 | 9 | 0.25 |
| Graph keywords → graph keywords plus merge | 2 | 2 | 8 | 1.00 |
| Plain topic → plain topic plus merge | 1 | 0 | 11 | 1.00 |
Table 4. Runs out of 3 with both plans retrieved, by question. Product names are generalized.
| Question about | Existing | Merge | Graph keywords | Graph keywords + merge | Plain topic | Plain topic + merge |
|---|---|---|---|---|---|---|
| Five questions, each | 3 | 3 | 3 | 3 | 3 | 3 |
| Loyalty integration | 0 | 3 | 2 | 3 | 3 | 3 |
| Tips feature | 0 | 0 | 3 | 3 | 3 | 3 |
| QR code ordering | 0 | 0 | 3 | 3 | 3 | 3 |
| Digital bill | 0 | 0 | 3 | 0 | 3 | 3 |
| Digital menu | 0 | 0 | 3 | 0 | 3 | 3 |
| Online reputation | 0 | 0 | 0 | 3 | 3 | 3 |
| Guest data collection | 0 | 0 | 0 | 0 | 2 | 3 |
Table 5 reports the regression evidence, which predates the plain-topic variant. On the real knowledge, with the FAQ still present, the graph-keyword search moved the mean score by less than the variation between repeats on a single item (up to 0.15). The isolated copy scored lower in every arm because it holds one market and no FAQs. An alternating latency test on the graph-keyword version, six questions three times each, gave median time to first token of 7.7 seconds off and 6.8 seconds on. Cached keywords may favour the second arm, so this shows no visible cost rather than a gain. Latencies in Table 2's runs ranged from 4.0 to 6.6 seconds but were not alternated.
Table 5. Mean end-to-end evaluation score, thirty-six items.
| Change tested | Knowledge | Repeats per item | Off | On |
|---|---|---|---|---|
| Sibling merge | Isolated copy | 1 | 0.700 | 0.682 |
| Graph keywords, reserved slots | Isolated copy | 1 | 0.688 | 0.692 |
| Graph keywords, repeat | Isolated copy | 1 | 0.686 | 0.722 |
| Graph keywords, 15 passages | Isolated copy | 1 | 0.688 | 0.696 |
| Graph keywords, reserved slots | Real, FAQ present | 2 | 0.864 | 0.854 |
Table 6 reports the later version, which caches topics with their keywords and maps six English price words to one topic. Each piece was removed in turn on the isolated copy, over thirty-six price questions, twelve of them written before these runs. The keyword cache was cleared between the three passes, so the passes are independent extractions.
Table 6. Both plans retrieved and cited, later version, 108 runs per condition.
| Condition | Sibling merge | Retrieved | Cited |
|---|---|---|---|
| All pieces | on | 108/108 | 103/108 |
| All pieces | off | 105/108 | 103/108 |
| No word mapping | on | 105/108 | 101/108 |
| No word mapping | off | 102/108 | 99/108 |
| Three hits per topic, no word mapping | off | 97/108 | 98/108 |
| Graph keywords instead of topics | on | 88/108 | 91/108 |
| A phrase with alternatives per topic, no word mapping | on | 73/108 | 80/108 |
| No extra search | on | 66/108 | 72/108 |
One run of the no-word-mapping condition with the merge on failed to complete, and it is counted as not retrieved and not cited. With all pieces on, the uncited runs came from two ambiguous questions that the agent read as guest fees. Citing and retrieving no longer matched exactly as in Table 2: some conditions cited more often than they retrieved, for a reason not established, possibly graph text.
The merge was removed: it left citation unchanged and could not be switched off. On ten evaluation sets on real knowledge, with the curated answer still present, the later version without the merge raised the mean score on seven and lowered it on three, in a range of -0.059 to +0.018 per set. The largest fall came from a set of eight items, mostly from one off-topic question. Ranking offline with seven embedding models, the words "fee" and "fees" placed the price passage between 15th and 55th on every model, so a newer model would not remove the need for the mapping.
Interpretation¶
The data contradict H2 and support H1 only narrowly. The merge did what it was built to do, since one-plan retrievals fell from 9 to 0 in Table 2, but its effect on retrieving both plans is not distinguishable from zero. It reduced the client's passages from 1,032 to 873 and changed 31 of the 89 documents. With graph keywords it traded failures rather than removing them (Table 4). Its added value on top of the plain-topic search is not established, because 36 against 35 is within the variation between rebuilds seen in Table 1.
The data favour H3. Searching for the kind of fact asked gave the largest gain and held on new phrasings. Rank fusion scored below fixed slots, against expectation. The likely reason, unverified, is that the library still takes turns between passages and graph sources. Citation and retrieval behaved as separable problems for this client, so the price-free instruction appears sufficient once retrieval succeeds.
Threats to validity¶
- Construct. Retrieved and cited are string matches on the two amounts, not judgements of the answer. Only the Table 5 scores involve a judge.
- External. One client, one market and one kind of fact. Every plain-topic run was a price question, so generality to other facts is a hypothesis. The question on handhelds also passed here but had failed on the live knowledge, for a reason not established.
- Leakage. The plain-topic instruction uses "pricing" as one of its own examples, and every question tested is about price. The new phrasings test wording, not new kinds of fact, so the high scores may overstate the mechanism.
- Internal. Questions were not sampled at random. Conditions on different builds are not strictly paired, since each rebuild changes the graph. Both prices appeared in graph text in none of the 36 runs on build C, against 12 and 13 in the plain-topic runs on build A, which is unexplained. The merged condition was not run on the new phrasings, the only runs where one-plan retrievals remained.
- Side effects. The plain-topic prompt shifts the keywords the graph receives. Median graph context held at about 22 entities and 22 relationships, but one turn fell to 4 relationships. The prototype also bypassed the keyword cache, so repeated questions paid an extra language-model call. The later version caches topics.
- Scope of the regression check. The cross-client scores in the update above ran with the curated FAQ present, so the result without it is measured only on the isolated copy.
Next step: remove the curated FAQ once the search is live and check the answers. Latency needs alternating measurements on the later version. Non-price facts need their own test set, with an instruction whose examples do not name the fact under test.
Non-R&D / Productization Context¶
The client-specific instruction was rewritten without prices, so it can no longer go stale. It tells the agent to name every plan when the retrieved knowledge lists them, and not to state a price that was not retrieved. The test environment's database replay was also repaired, because new isolated copies were failing on an unrelated migration. Archived legal and careers pages that crowd this client's retrieval have not been removed.
Prior-Month Results Review (October shipped work)¶
The October journal entry has not yet been written, so this section reads only the one October change this investigation directly tested. That change, shipped on October 30, keeps short documents in a single chunk. It was expected to make the curated pricing answer unnecessary, on the assumption that the pricing page was a 215-token document. The re-scraped page was about 800 tokens, so the change did not apply to it, and with the FAQ and client instruction disabled, one of four test questions lost the plans entirely. This is a single-page read over days, with no randomization. No live-metric read was attempted, because the window is too short.
Research Outcome¶
So far this month, the investigation has falsified the graph-limit explanation, found chunking insufficient, and identified a mechanism that retrieves a fact documented away from the subject named: a separate search on the kind of information asked for. On one client's price questions, retrieval of both plans rose from 15 of 36 runs to 35 of 36, and to 33 of 36 on new phrasings. Question-level improvement was 7 of 12 with none degraded, which does not reach conventional significance after correction. For this client, citation and retrieval proved separable, and the remaining failures sat in retrieval. Rank fusion and wider limits are recorded negatives on this test.
On other clients the later version showed no clear gain or loss, and the clear gain remains this one client's price questions. The mechanism is untested on other kinds of fact, and the work has not reached production, so its production effect is a pending null. The curated answer that prompted the investigation is still live.