Query rewriting vs retrieval quality¶

Date: 2026-08-26 · Ticket: IX-4447 · Data: 500 production Langfuse tool-query-rewriter spans (2026-08-18 → 2026-08-25); three purpose-built eval datasets (query-rewriter-v1, query-rewriter-elliptic-v1, query-rewriter-elliptic-holdout) run against staging LightRAG.

Question¶

The chat pipeline rewrites the visitor's message before retrieval (packages/ixchat/ixchat/utils/query_rewriter.py, policy in chat_routing.py). Does that rewrite improve knowledge retrieval, and if not, what would?

The rewriter's only consumer is LightRAG retrieval. langgraph_retriever.py confirms retrieval sees only the query string — conversation history is explicitly not forwarded, and invoke() drops it if a caller passes it. So a visitor turn like Business or Oui reaches the index with no context whatsoever unless something puts it there. That is the case the rewriter exists to serve.

Method¶

Three instruments, built in sequence as each one proved insufficient.

1. Production span audit. 500 tool-query-rewriter spans joined to their sibling tool-lightrag-retrieval spans and to the final answer, to check whether span-level defects propagate downstream.

2. A retrieval eval, rose-eval query-rewriter. Neither existing eval kind measures the right thing: e2e judges the final answer (which the answer writer repairs, because it receives full history independently), and feature runs on one synthetic site. The new kind runs rewrite → real LightRAG retrieval → context-sufficiency judge: an LLM judge reads the conversation, the visitor's real information need, and the chunks LightRAG actually returned, then scores 0–10 whether those chunks let you answer. Deterministic hygiene checks (truncation, markdown, language preservation, graft) run alongside for free. Code in packages/ixevaluation/ixevaluation/rewriter/.

3. Datasets. All items are real production turns lifted from Langfuse, so every (history, message) pair is one a visitor actually produced.

dataset	n	composition
`query-rewriter-v1`	50	10 clients × 5, tagged `elliptical` / `pronoun` / `self_sufficient` / `topic_shift` / `no_retrieval`
`query-rewriter-elliptic-v1`	123	elliptical turns only (≤5-word reply to a question the agent just asked), 22 sites, capped 14/site, tagged `qualification` / `case_request` / `topic_shift` / `no_retrieval`
`query-rewriter-elliptic-holdout`	110	same 401-turn harvest, held out — never used to pick a threshold or a prompt rule

Comparison discipline. Per-item delta stdev is ~0.20, so a 43-item run carries SE ≈ 0.03 — the same size as the effects being measured. Arms are compared paired per item over N≥2 runs, never mean-vs-mean. Later rounds added bootstrap CIs, an exact sign test, and a per-run noise floor measured by re-judging identical input.

Findings¶

1. The production rewriter is defective, and the defects are absorbed¶

Path split: 70% (349/500) of turns never reach the LLM at all — first turn falls to a Python regex fallback that handles English you/your only, so it is a no-op on FR/ES/IT traffic. 30% (151) take the LLM path (ministral-3b-latest); 2.6% of those time out at the 1s ceiling with max_retries=0.

Of the 143 non-empty LLM rewrites:

24% (34/143) truncate mid-word, comma or digit. max_tokens=50 is token-dense-language-hostile (~230–275 chars in French). The only guard is len(rewritten) < 2, so the fragment goes straight into the embed call.
57% (81/143) contain **bold**, 19% are wrapped in quotes — copied from the assistant's formatted messages, which _format_conversation_history passes verbatim. Pure embedder noise.
74% (111/151) are >4× longer than the input, median 144 chars for inputs of 2–15 chars.
Language flips within a single conversation; greetings and gibberish get a fabricated query (hello → a 200-char product question, sffs → a full client query, todo → a Spanish travel itinerary).

Cost: p50 451ms / p90 740ms on the LLM path, ≈147ms blended per turn on the retrieval critical path.

But downstream the damage is not visible. Retrieved document count is flat across every rewrite category (27.5–29.5 docs, zero zero-doc retrievals) — it saturates at chunk_top_k and cannot discriminate. KB-miss phrasing in the final answer does not correlate with rewrite quality (truncated 0/31, full LLM 1/72, Python fallback 1/196). Reading the worst cases end to end, the answers are correct: truncation removes the tail of an over-expansion the rewriter itself invented, and the answer writer repairs the rest from history.

So the defects are real but the pipeline absorbs them. The rewriter is sloppy, not harmful — which reframes the question from "fix it" to "does it earn its 147ms at all".

2. No prose rewrite beats sending the raw query¶

query-rewriter-v1, 2 runs per arm, 43 judged items, paired:

arm	sufficiency	vs off (paired)	better / worse / tie
off (raw query)	0.430	—	—
current (production)	0.413	−0.017 ± 0.030	9 / 15 / 19
candidate (hygiene prompt)	0.405	−0.026 ± 0.021	7 / 11 / 25
minimal (telegraphic)	0.416	−0.014 ± 0.030	12 / 13 / 18

The candidate prompt fixes every defect it targets — markdown failures 29 → 0, truncation 23 → 0, inflation 21 → 5 — and retrieval does not move. Its one regression explains why: the pass-through rule ("already stands on its own → output unchanged") is too easy for a 3B model to reach for, so 11 of 26 elliptical/pronoun items come back unexpanded.

The eval independently reproduced the production defect rate (58% markdown here vs 57% measured on production spans), which is the dataset validating itself.

3. The metric had a bias, and finding it was the point¶

The v1 judge rubric contained an escape hatch — "if the message needs no knowledge lookup, score 10". The judge applied it to elliptical replies: Logiciel interne (a bare answer to "which payroll tool do you use?") scored the do-nothing control 1.00 with the reasoning "no knowledge lookup was needed". That inflated off on exactly the items where rewriting is mandatory.

v2 removes the hatch, states that an elliptical message still carries an information need supplied by the conversation, and drops no_retrieval items from the judged mean. Elliptical off fell 0.62 → 0.34 — the control is now penalised where expansion is required, which was the acceptance criterion.

Of 43 judged items, 27 discriminate; 9 are floor items no arm can win (competitor questions, pure qualification turns, real KB gaps) and 7 are ceiling items.

4. Prose was being written for a consumer that throws it away¶

LightRAG has two retrieval channels (operate.py:_build_query_context):

the query text drives the vector / chunk side,
hl_keywords / ll_keywords drive the graph side (entities, relations).

When keywords are not supplied, get_keywords_from_query spends its own LLM call deriving them from the query text. The rewriter was therefore writing prose for a consumer that immediately re-extracts keywords from it — a lossy round trip we never controlled. All the prompt work was on the wrong side of it.

Same client, same message Santé:

LightRAG's own keywords  ['Santé', 'Skello']
  → top chunk: "Odento is a client of Skello in the health sector"    (a client name)

supplied  ['santé', 'tarifs'] / ['pricing']
  → top chunk: "Santé et Bien-Être is a sector where Skello provides
                optimized scheduling solutions…"                       (the answer)

A rival hypothesis — that verbose rewrites inflate the extracted keyword count and fan the graph budget out — did not hold: correlation between rewrite length and extracted keyword count is +0.17. The extractor normalises. The problem was never keyword volume, it was having no say in which keywords.

So: stop rewriting the query, resolve straight into keywords. The visitor's own message keeps driving the vector side; resolved keywords drive the graph side.

arm	sufficiency	vs off (paired)	better / worse / tie
keywords	0.477	+0.047 ± 0.016	16 / 4 / 23
fused (raw + keywords appended)	0.494	+0.064 ± 0.033	18 / 11 / 14
both (2nd LLM call for prose)	0.460	+0.030 ± 0.030	15 / 10 / 18

This table did not replicate — see section 10

These arms were scored while the judge was shown each arm's graph keywords appended to the query text, which the control arms had nothing equivalent to. With that annotation removed, keywords scores −0.015 ± 0.028 against the raw query on the same dataset. The mechanism described below is real and the mode split holds; the headline number does not.

By mode:

mode	n	off	current	keywords
elliptical	21	0.32	0.36	0.42
pronoun	5	0.72	0.48	0.64
self_sufficient	14	0.53	0.50	0.55
topic_shift	3	0.27	0.27	0.33

keywords wins on elliptical — the mode that justifies having a rewriter at all. fused has the higher mean but loses head-to-head against keywords (+0.014 ± 0.028, 10 better / 13 worse): more items get worse than better, so the headline number is noise. keywords is the arm with only 3 losses out of 43.

both confirms the through-line: paying a second LLM call for resolved prose on the vector side scored below free keyword-appending. Prose loses on both channels.

Side effect: supplying keywords skips LightRAG's own extraction call. Median retrieval 1.86s → 1.62s (n=6), roughly offsetting the resolver's own call.

5. Lite-tier clients have no graph, and today's rewriter hurts them¶

TIER_TO_RAG_MODE = {"lite": "naive", "advanced": "global"}, and naive_query calls _get_vector_context(query, …) and nothing else — no keyword extraction, no graph. On a lite client, supplied keywords are ignored outright.

--rag-mode naive forces the mode so the same turns stand in for a lite client:

arm	sufficiency	vs off	better / worse
off	0.505	—	—
current (production)	0.460	−0.044 ± 0.039	8 / 14
keywords	0.516	+0.012 ± 0.018	6 / 4
fused	0.514	+0.009 ± 0.029	10 / 8

Two things fall out. Today's rewriter is actively hurting the tier paying least. And this looked like a free calibration of the instrument: keywords produces a byte-identical query on 43/43 items under naive (as it must, with no graph) and still drifted +0.012, suggesting anything under ~0.02 was judge noise.

That floor is too low, and section 10 shows why. Identical query text does not give identical passages: when no keywords are supplied, LightRAG derives its own with an LLM call, so the control arm's retrieval is itself non-deterministic. Measuring the floor by re-judging the same passages (0.009) prices only the judge, not the retrieval it is judging.

6. Additive beats substitutive — six strategies, two model tiers¶

Testing whether better prose could beat the raw query on the chunks-only path:

arm (naive mode)	vs off
minimal (telegraphic)	−0.067
current (production prose)	−0.052
candidate (clean resolved question)	−0.043
HyDE, gpt-oss-120b	−0.044
HyDE, ministral-3b (declarative)	−0.019
fused (raw + keywords)	+0.001
off (raw)	0.000

A 40× bigger model made it worse, not better. It writes better prose and more confident invented specifics; on an embedding channel a fluent wrong passage is worse than a crude one. That falsified the second mechanistic theory (after keyword fan-out): the ceiling was not resolver precision either.

Everything that replaces the visitor's text loses. The only thing reaching parity is the one that keeps their text and adds to it. Consistent with the graph result, which won by adding a channel rather than by altering what the visitor wrote.

The mechanism behind the asymmetry: a wrong keyword on the graph side simply fails to match an entity (harmless); the same keyword on the embedding side drags the vector off-target (harmful). Same input, opposite risk profile — which is why the identical resolver is worth +0.050 on graph and 0.000 on chunks.

7. Elliptical turns are a different, worse regime — and the fix is a gate¶

Elliptical turns are where the raw query is most useless, and every arm so far sat flat there. Isolating them (query-rewriter-elliptic-v1, 123 items) gives a baseline of 0.369 vs 0.513 on the general set: not a weak spot within retrieval, a separate regime.

The failing mechanism turned out to be speech act, not topic. When the agent asks "shall I share a customer example?" and the visitor types Oui, every arm resolved the subject (healthcare HR) and dropped the request (give me a customer example) — answering "what is this about" when the question is "what are they asking for".

An intent arm that resolves the speech act splits hard by mode:

mode	n	off	intent	better / worse
qualification	66	0.27	0.35	24 / 5
case_request	8	0.35	0.45	2 / 0
topic_shift	20	0.63	0.40	1 / 10

Large win on the dominant mode (70% of elliptical turns), disaster on topic_shift — where the visitor ignored the question and asked something new, the raw query is already a real question and resolving damages it.

A length heuristic separates the modes almost as well as a classifier, for none of the cost:

mode	mean words	median
qualification	2.8	3
case_request	2.6	3
topic_shift	4.0	5

reply length	n	delta (intent − off)	better / worse
1 word	19	+0.189	9 / 2
2 words	19	+0.074	7 / 1
3 words	14	−0.014	0 / 1
4 words	21	+0.038	9 / 3
5 words	21	−0.186	2 / 8

gate	delta	resolves
none (always resolve)	+0.018	94/94
≤4 words	+0.060	73/94
skip if reply ends with `?`	+0.068	74/94

Both gates triple the effect and select nearly the same population (73 vs 74) — they are two probes of the same thing, since a 5-word elliptical reply usually is a question. Shipped rule: ≤4 words AND does not end with ? (query_keywords.should_resolve). Two lines, no classifier, no extra LLM call.

8. A prompt defect was suppressing the signal, and fixing it exposed a smaller one¶

The intent prompt asked the model to classify into four numbered cases and then write a query. ministral-3b sometimes emitted the classification instead:

'5 à 10 ans' → '2. The assistant asked for the duration of the curre…'
'Beauté'     → '2. The assistant asked for a specific type of exampl…'   (−0.40)

4 of 20 gate firings were scaffold leakage — 20% garbage output. Making the classification implicit and validating the output fixed it 4 → 0, and the broken items recovered (Beauté 0.00 → 0.60). The aggregate did not move, because the prompt edit changed every resolution, not just the broken three — including removing a lucky accident (Around 500 leads → "2000 credits", the literal answer, which happened to sit on top of the right chunk). At 16 fired items a single 0.80 swing is ±0.05 on the mean, so both the before and after numbers were noise.

Rewriting the prompt against the measured features — mandate the company name and the visitor's own words (enforced in code, not merely requested), target 5–7 words instead of "at most 12" — doubled the effect and halved the variance:

prompt	mean delta	better / worse	SE
intent (old)	+0.018	27 / 15	±0.032
gated (new)	+0.036	18 / 6	±0.017

Mean rewrite length dropped 7.6 → 3.7 words, and topic_shift returned to 0.63 with 0 better / 0 worse — the gate leaves those turns untouched by construction, closing the worst failure mode structurally rather than by hoping the model behaves.

9. Held-out replication¶

Methodology hardening before the final read: a held-out dataset never used for tuning; the judge display leak fixed (graph keywords are now shown only when the graph actually ran — under naive they were annotated but ignored, which was inflating an identical-input arm); bootstrap CI and an exact sign test replacing mean ± SE; and the noise floor measured in the same run rather than inferred.

query-rewriter-elliptic-holdout, 86 judged items, 2 runs per arm:

off     0.281
gated   0.322
delta  +0.041    95% CI [+0.005, +0.078]    sign test 21+ / 10−  p = 0.071

judge run-to-run on identical input: mean |diff| 0.009, moved on 4/86 items

The tuning set gave +0.036; the held-out set gives +0.041 — the effect did not shrink, which is what usually happens to an in-sample number. The CI excludes zero; the sign test is marginal (the mean is carried partly by size-of-win, not only count-of-wins). The effect is ~4.5× the directly measured noise floor.

Baseline here is 0.281, lower than the tuning set's 0.353, because the holdout is pure qualification — the hardest mode. 0.28 → 0.32 on the worst band in the product.

10. Re-measured after the instrument was corrected — the graph result needed a gate¶

A code review found that the judge was being shown each keyword arm's resolved keywords appended to the query text, while the control arms had no equivalent annotation. That is a difference between arms which is not a difference in the passages being scored, and it is the arm-selection signal the whole comparison rests on. The keywords are now reported on the result object only; the judge sees the same query text for every arm.

Everything in section 4 was re-run under the corrected judge. query-rewriter-v1, 43 judged items, 2 runs per arm, paired:

arm	sufficiency	vs off	better / worse / tie
off (raw query)	0.462	—	—
current (production)	0.434	−0.028 ± 0.033	8 / 16 / 19
keywords (resolve every turn)	0.447	−0.015 ± 0.028	10 / 11 / 22
keywords-gated	0.470	+0.008 ± 0.017	9 / 8 / 26

The +0.047 does not survive. Resolving every turn is no better than sending the raw query. What the corrected measurement shows instead is a clean mode split:

mode	n	off	keywords	keywords-gated
elliptical	21	0.357	0.414	0.414
topic_shift	3	0.267	0.367	0.333
self_sufficient	14	0.546	0.507	0.507
pronoun	5	0.780	0.460	0.680

The gain is entirely on elliptical turns; the loss is entirely on turns whose raw query was already the best in the dataset. pronoun collapses from 0.780 to 0.460 — and those five items are not what the label suggests. Their only pronoun is a possessive naming the company (votre logiciel, my traffic, avez-vous), which retrieval already binds, because it queries Question about "X": …. There is nothing to resolve, but the resolver cannot decline, so it invents a frame:

Combien de bureaux avez-vous ?   →   bureaux, capacité, espace | infrastructure, scale

It read bureaux as office space, not a company's office count. Score 1.00 → 0.00, identically in both runs, and the judge is explicit about why: on the raw query "the passages directly provide that: Orisha has 20 offices"; on the resolved keywords "the passages are about product/platform concepts… nothing here helps".

This falsifies the asymmetry claimed in section 6. A wrong keyword on the graph side was described as harmless — it simply fails to match an entity. It is not: the graph spends a fixed top_k, so wrong keywords match something and displace the entities the raw query was already retrieving. Retrieved document count moved 16 → 21 on that item while the answer left the result set.

Applying should_resolve — already shipping on the chunks-only path — to the graph channel recovers most of it: pronoun −0.320 → −0.100, elliptical gain kept in full, and the arm becomes the best of the four. It fires on 16 of 43 items and scores +0.056 where it fires, 6 better / 1 worse.

Out-of-sample, on a gate-dense dataset. query-rewriter-v1 contains only 16 gate-eligible turns, which is why its overall number stays inside the noise. The held-out elliptical set is the right instrument — 83 judged qualification items, graph-capable clients, 2 runs per arm:

off              0.294
keywords-gated   0.326
delta           +0.032 ± 0.020    18 better / 11 worse / 54 tie
gate fired on 68/83 items;  on those:  +0.038   17 better / 10 worse

A better noise control. The 15 items where the gate declines are a control built into the run itself: their retrieval input is byte-identical to off. They moved by +0.003, on 2 of 15 items. On query-rewriter-v1 the same control moved 10 of 27 items by up to 0.20 — worth knowing, because it means identical query text does not give identical passages. With no keywords supplied LightRAG derives its own with an LLM call, so the control arm's retrieval is itself non-deterministic, and the 0.009 floor quoted in section 9 prices only the judge.

The lite-tier result re-verified. Section 9's +0.041 was measured before the display leak was fully closed, so it was re-run from scratch under the corrected judge, same held-out dataset, retrieval forced to naive:

	section 9	re-measured
off	0.281	0.293
gated	0.322	0.339
delta	+0.041	+0.046 ± 0.016
95% CI	[+0.005, +0.078]	[+0.018, +0.078]
better / worse	21 / 10	20 / 7

It replicates and tightens: the confidence interval still excludes zero and its lower bound moved up. The gate fires on 52 of 83 items and scores +0.071 where it fires, 18 better / 6 worse. The 31 items it declines are the in-run noise control — they moved +0.003, on 3 items.

And the two channels turn out to be interchangeable. If the gate is what matters, then on a graph-capable client the text resolver built for the lite tier should do just as well as the keyword channel. It does — same 83 held-out items, same two runs, client-default retrieval modes:

arm	sufficiency	vs off	95% CI	better / worse
off	0.294	—	—	—
gated keyword channel	0.326	+0.032 ± 0.020	[−0.004, +0.073]	18 / 11
gated text resolver	0.327	+0.033 ± 0.020	[−0.006, +0.073]	19 / 11
gated text vs gated keywords	—	+0.001 ± 0.023	[−0.045, +0.046]	18 / 16

Head to head they are a coin flip. The gate is the intervention; the channel it resolves into is not. That reverses the framing of section 4, which treated the graph channel as the discovery and the gate as a later refinement. The graph channel's remaining advantage is cost, not quality: supplying keywords skips LightRAG's own extraction call, which the text resolver still pays.

So the two claims came out of the correction differently. The chunks-only path is confirmed: +0.046, CI excluding zero, on the mode that dominates elliptical traffic. The graph path is directionally positive by the same amount through either channel, but its confidence interval includes zero (sign test p = 0.265), and two runs of 83 items cannot close that — the per-item spread, not run-to-run noise, is what limits it, so the fix is more held-out items rather than more runs.

11. Powered properly, on 325 held-out items¶

Every number up to here was underpowered. With a per-item delta stdev of ~0.15, 83 items resolve only an effect of ~0.05 at 80% power, while the effect under test is ~0.03 — so "CI includes zero" was never evidence of absence. And more runs cannot fix it: decomposing the holdout variance, run-to-run noise is 4% and item-to-item difference 96%, so two runs give SE 0.0203 and eight give 0.0201. Items are the only lever.

query-rewriter-elliptic-v2 adds 242 judged items harvested fresh from production and deduped against every earlier set (154 candidates were already seeded). Pooled with the holdout, the detectable effect drops to ~0.023.

Lite tier (chunks only) — confirmed:

set	n	off → gated	delta	95% CI	better / worse	p
holdout	83	0.293 → 0.339	+0.046	[+0.018, +0.078]	20 / 7	0.019
elliptic-v2	242	0.316 → 0.342	+0.025	[+0.006, +0.045]	53 / 29	0.011
pooled	325	—	+0.030 ± 0.008	[+0.014, +0.047]	73 / 36	0.001

The effect (0.030) now exceeds the detectable threshold (0.023), the CI excludes zero on both sets independently, and the sign test clears 0.05 on each. Where the gate fires — 116 of 242 turns — it is +0.054, 47 better / 19 worse.

Graph tier — positive but still unresolved:

set	n	delta	95% CI	better / worse	p
holdout	83	+0.032	[−0.004, +0.073]	18 / 11	0.265
elliptic-v2	242	+0.014	[−0.006, +0.034]	46 / 33	0.177
pooled	325	+0.018 ± 0.009	[+0.000, +0.037]	64 / 44	0.067

Both headline numbers shrank on more data, which is what a marginal detection does: the graph tier halved (+0.032 → +0.014) and the lite tier fell by nearly half (+0.046 → +0.025). The lite effect survived that shrinkage; the graph one did not clear the bar, and settling it would take ~635 items.

The instrument is sound. The turns the gate declines are byte-identical to the control by construction, and they moved −0.001 (lite, 126 items) and −0.004 (graph, 78 items). And topic_shift — the mode where resolution used to cost −0.23 — sits at −0.000 on 40 items. That failure is closed by construction.

Verdict¶

tier	retrieval mode	strategy	effect (325 held-out items)	status
lite	`naive` (chunks only)	gated text resolver	+0.030, CI [+0.014, +0.047], 73 / 36, p=0.001	confirmed
advanced	`mix` / `global` (graph)	gated keyword channel	+0.018, CI [+0.000, +0.037], 64 / 44, p=0.067	positive, unresolved

Both beat today's production rewriter, which is negative on the lite tier (−0.044) and negative on the advanced tier (−0.028 on query-rewriter-v1). That part is not in question: the current rewriter is worse than doing nothing on both tiers, and turning it off is a strict improvement independent of anything that replaces it.

One intervention, not two. The gate is what works. Resolving only turns the visitor could not have searched as written helps on both tiers; the channel it resolves into does not matter (+0.001 ± 0.023 head to head). Resolving every turn is worthless-to-harmful, which is what nine rounds of prompt work kept rediscovering without naming.

How much traffic this touches. Production tool-query-rewriter spans carry the visitor's message and the history length, so the gate can be replayed over real turns. Two pulls of recent production spans (3000 spans, and 1450 spans over 1442 distinct traces, 2026-08-25 → 08-27):

bucket	share
first turn — nothing to resolve, never fires	44%
multi-turn, searchable as written — gate declines	33–35%
gate fires	19–23%

So roughly one retrieval turn in five is eligible, and a third of all turns are ones where today's rewriter runs, measurably hurts, and the gate would stop it running at all. Traffic-weighted, the retrieval win lands on about 7% of turns and the regression on about 2%; the cost saving lands on every declined turn.

Caveat on transferring the effect size: the held-out set is elliptical qualification turns specifically, while the production gate-eligible population is a broader mix of one- to four-word replies. Direction should transfer, magnitude is not guaranteed to.

Scale of it, stated honestly. +0.046 is not every item improving a little. On the lite tier: 56 of 83 items unchanged, 20 improved, 7 regressed; the gate declines a third of turns outright and those are identical by construction. Where it fires, +0.071. So roughly one turn in four gets meaningfully better retrieval, one in twelve gets worse, two-thirds are untouched — real, replicated, and not a visibly better bot. It is worth shipping because the cost is negative: the gate removes today's rewriter call on every turn it declines.

Falsified along the way, each with its own measurement:

The prompt is the problem — the hygiene prompt fixed every defect and moved retrieval by nothing.
Verbose rewrites fan out the graph budget — length/keyword-count correlation +0.17.
Resolver precision is the ceiling — a 40× larger model scored worse.
Enrichment format is the ceiling — four forms tested under naive, all substitutive forms lost.

The surviving generalisation: additive beats substitutive. The visitor's own words are the highest-precision signal in the system; every rewrite trades that away for context, and on the chunks channel the trade never pays.

What this does not measure¶

Not answer quality. Measured separately at the outset and flat — the answer writer receives history independently and repairs bad retrieval, which is exactly why the metric stops at retrieval.
The judge cannot exceed the KB. 9 of 43 items are floor items no arm can win. Being paired, they dilute the mean without biasing the comparison.
Self-consistency, not ground truth. No human labelled these items. A judge systematically wrong about a client's domain is wrong for every arm equally — pairing protects the comparison, not the absolute 0.51. Before anyone treats a sufficiency score as a KPI, hand-label a subset and check judge agreement.
Staging KBs, two runs, one judge. Enough to rank arms; not enough to declare a production number.

Shipping state¶

Production behaviour remains unchanged until this branch is merged and deployed. The branch sets query_rewriter.enabled = true in every environment TOML; deployment activates tier-aware resolution for every tenant in that environment. Turning the sole switch off sends the visitor's message through untouched; it does not restore the legacy prose rewriter.

ixchat/utils/query_keywords.py — resolves the message into SPECIFIC: / THEME: lists plus the should_resolve gate. Falls back to LightRAG's own extraction on first turn, on LLM failure, on an unparsable response, and on a half-filled response: global mode reads hl_keywords alone and local reads ll_keywords alone, so returning one channel would silently strip the graph side for whichever mode needs the other.
ixchat/nodes/knowledge_retriever.py — routes on has_graph = retriever.rag_mode != "naive". Graph-capable clients get the keyword channel; lite clients get the gated resolver (ixchat/utils/query_intent.py), which resolves only turns that cannot be searched as written and leaves everything else untouched.
ixrag/lightrag/langgraph_retriever.py — ainvoke(query, ll_keywords=…, hl_keywords=…). Additive; callers passing nothing are unaffected.
ixevaluation/rewriter/ — the eval kind and its arms (production, off, legacy / current, candidate, minimal, keywords, fused, both, hyde, intent, gated). The default production arm mirrors the shipping tier branch; legacy reproduces the retired prose rewriter. Runs are recorded as Langfuse dataset runs with per-item scores, and a run that drops or errors on any item is reported incomplete rather than scored — a partial run is a different dataset, not a weaker result.
Three seed commands under apps/cli/cli/langfuse/commands/; datasets live in Langfuse and are re-seedable. Every item is a real production turn with visitor PII and credential-bearing URLs replaced with synthetic values, enforced by apps/cli/tests/unit/test_seed_datasets_pii.py.

Both tier strategies are enabled in development, test, staging, and production configuration on this branch. Lite's powered effect is confirmed; the graph estimate remains positive but unresolved and should be monitored after deploy.

Reproducing¶

# shipping behavior (the default arm); force naive to model a lite tenant
rose-eval query-rewriter --arm production --env staging
rose-eval query-rewriter query-rewriter-elliptic-holdout --arm production --rag-mode naive --env staging

# control, ~50s on the 50-item set
rose-eval query-rewriter --arm off --env staging

# graph channel: `keywords` resolves every turn, `keywords-gated` is what ships
rose-eval query-rewriter --arm keywords --env staging
rose-eval query-rewriter --arm keywords-gated --env staging

# the held-out set is the instrument for anything gate-related — the 50-item set
# has only 16 gate-eligible turns, too few to move a mean
rose-eval query-rewriter query-rewriter-elliptic-holdout --arm keywords-gated --env staging
rose-eval query-rewriter query-rewriter-elliptic-holdout --arm gated --rag-mode naive --env staging

Two runs per arm, minimum, compared paired per item. Read the items the gate declines as the in-run noise control: their retrieval input is identical to the control's by construction, so whatever they drift by is the resolution limit of that run.

Implementation notes and per-round detail: docs/plans/query-rewriter-eval.md.