Embedding models for multilingual retrieval¶

Date: 2026-11-04 · Ticket: IX-5976 · Data: 90 widget questions and 30 staff questions from four clients, ranked with seven embedding models; 36 questions run through the real chat graph on Bonx's test corpus, in four setups. No production data, index or configuration changed.

Question¶

Bonx reported that French questions ignore its English FAQs and its English Q&A document. Would a newer embedding model fix this, and would it help other clients? Every workspace uses text-embedding-3-small ("Small") today.

Result¶

Final (2026-11-05): no embedding migration is needed for Bonx. On current develop, which has 20 chunks for lite clients (IX-5968) and the topic chunk search (IX-5899), adding 3 reserved slots for the client's own content, placed first, gives the best Bonx result tested, on the EU route, with no client work. See Final comparison on current develop. The findings below were measured on the older base and explain how we got here.

On Bonx, Cohere Embed 5 Pro retrieves clearly more, but its answer gain is modest. It retrieves the right English content on 30 of 33 non-English questions, against 24 for Small. Across three blind reviews it gives 0, 0 and 1 wrong answers against 4 each for Small, but fully right answers vary too much between reviewers to separate them (28, 27 and 21 against 24, 24 and 21).
Raising the chunk limit does not improve answers on this set. Small at 20 or 30 chunks retrieves 27 of 33, but a blind review rates fewer answers right than at 10 (18 against 21).
Voyage 4 Large does not beat Small's answers on Bonx. It retrieves 27 of 33, but the blind review rates 21 answers fully right and 3 wrong, mostly because it misses the customer count.
text-embedding-3-large, the tested model with an EU route, does not improve Bonx's answers. It retrieves 28 of 33, but a blind review rates 22 answers right and 3 wrong, against 24 and 4 for Small.
The IX-5968 candidate fix alone does not solve this. Small with 20x candidates retrieves 22 of
It restores the API FAQ, but it is not a cross-language fix.
Other clients show encouraging retrieval gains, with no real-bot evidence yet. Across Eskimoz, PayFit and sunday, Cohere finds a direct answer on 47 of 51 questions, against 41 for Small.
No model is proven best for every client. Voyage leads overall retrieval across clients by one question and wins clearly on sunday's pricing questions. Gemini Embedding 2 is close and has an EU processing route.

Recommendation¶

Decided (2026-11-04): the target is one global model with an EU provider. That rules out the tested Cohere and Voyage routes, despite Cohere's result on Bonx.

Pause the embedding migration and keep Small. Ship the 3 reserved slots, placed first, for every client. Rescan Bonx's website, because its stale /about page still produces the wrong customer count. Asking self-serve clients to maintain translated FAQs is not an option. Test Gemini Embedding 2 through Vertex AI only if real conversations show that this is not enough.

Method¶

The work ran in three stages, each prompted by what the previous one could not show.

1. Single-client offline test. 1,032 sunday public-page chunks and 36 questions, ranked by five models. It measured English pricing questions and French questions against French pages, so it says nothing about cross-language retrieval.

2. Four-client comparison. 120 recorded questions, 30 each for Bonx, Eskimoz, PayFit and sunday, plus 12 authored Bonx regressions. Bonx's recorded questions are staff playground traffic; the other 90 are widget questions. Seven models were ranked with exact cosine search, which excludes the IX-5968 approximate-search defect. Model-blind and rank-blind agent curators reviewed every top-10 passage on the 90 widget questions. That verified a direct target for 51 questions. The other 39 are unresolved in the reviewed pool, not unanswerable.

3. Bonx pilot on the real bot. Bonx's test corpus was re-embedded with Cohere and Voyage (1,739 chunks each) and served through the real retriever, chat graph and answer model. Four setups were compared: Small as it runs today, Small with 20x candidates (what the IX-5968 fix gives), Cohere and Voyage.

A first set of 23 questions could not separate Small, Small with 20x candidates and Cohere: they scored 22, 20 and 22 on a keyword check, because Bonx's French and Italian pages already hold many of the same facts. A harder set of 36 questions then targeted facts found mainly in the English FAQs and Q&A: 33 non-English and 3 English controls, three runs each. A question counts as retrieved when the specific FAQ or Q&A section appears in the retrieved context.

A separate model reviewer then scored one run per non-English question for Small, Cohere and Voyage, against Bonx's FAQs and Q&A. Labels were shuffled per question, so it could not tell which setup wrote which answer. Answers were scored as right, partly right or wrong.

Findings¶

Retrieval across clients¶

Share of questions with a verified target where a direct answer appears in the top results (exact search, stage 2).

Model	Top 3, of 51	Top 10, of 51	Eskimoz cross-language, top 10, of 13	Bonx staff cross-language, top 10, of 27
Small (today)	32	41	7	9
text-embedding-3-large	34	45	9	20
BGE-M3	34	40	9	22
Qwen3 Embedding 8B	38	44	10	21
Gemini Embedding 2	40	47	11	22
Voyage 4 Large	42	48	10	19
Cohere Embed 5 Pro	43	47	12	24

Per client, Small against Cohere:

Client	Top 3	Top 10	Verified questions
Eskimoz	13 → 17	15 → 19	20
PayFit	9 → 12	12 → 13	13
sunday	10 → 14	14 → 15	18

Cross-language evidence on real visitors rests almost entirely on Eskimoz. PayFit's questions are all same-language, and sunday has one cross-language question, retrieved only by Gemini. Same-language ranking also improves: across the three clients, the top 3 goes from 26 to 35 of 41 questions.

The single-client test points the other way on pricing. On 12 English pricing questions, Voyage reached 12, Large 11, Cohere 8, Small 7 and Gemini 6. Cohere is not the right default for every kind of content.

Bonx pilot, real bot¶

Right FAQ or Q&A section retrieved, per question. Retrieval gave the same result in all three runs.

Setup	French, of 15	Italian, of 8	German, of 10	Non-English total, of 33
Small today	9	5	10	24
Small + 20x candidates	10	4	8	22
text-embedding-3-large	13	5	10	28
Cohere	13	7	10	30
Voyage	12	6	9	27

Blind review of the answers, on 33 non-English questions:

Setup	Right	Partly right	Wrong	Judged best	Free of stray English
Small today	24	5	4	7	27
Cohere	28	5	0	9	21
Voyage	21	9	3	7	24

The reviewer called 10 questions a tie. It marked English words with a common native equivalent, such as "self-serve" or "forecast", as stray English, so this column is strict.

A second reviewer, with fresh labels and more lenient on jargon, scored Small, Large and Cohere:

Setup	Right	Partly right	Wrong	Judged best	Free of stray English
Small today	24	5	4	7	29
text-embedding-3-large	22	8	3	7	32
Cohere	27	6	0	9	27

A third reviewer, stricter on claims about other vendors, scored Small at 10, 20 and 30 chunks against Cohere at 10:

Setup	Right section retrieved, of 33	Right	Partly right	Wrong	Judged best	Context, median characters
Small, 10 chunks (today)	24	21	8	4	6	not recorded
Small, 20 chunks	27	18	11	4	4	11,824
Small, 30 chunks	27	18	11	4	7	18,595
Cohere, 10 chunks	30	21	11	1	10	not recorded

More chunks retrieve three more sections in French, none in Italian or German, and time to first token stayed about 1.01 s. On this set the answers did not improve: sections arrive low in the list (rank 13 to 18), and the German customer-count answer got worse with more context.

The reviewers disagree by more than the setups do. Cohere scored 28, 27 and then 21 answers right, and Small 24, 24 and 21. Cohere's consistent lead is fewer wrong answers (0, 0, 1 against 4, 4, 4), which comes from a few questions.

Large runs on the same EU OpenAI route as Small. It retrieves more than Small, but its answers are no better on this set: some still say "quelques semaines" or 60+ customers, and some planning answers are weaker.

German retrieval is high for Small although Bonx has almost no German pages (6 of 1,738 chunks). That fits the explanation that same-language pages crowd out English content: where a language has its own pages, as French and Italian do, English sections lose out. This is consistent with the data, not tested directly.

Cohere gains over Small: SAP Business One in French and Italian, NetSuite and MRPeasy in French, the customer count in Italian, and a French penetration-test question.
Cohere misses: Divalto and Zucchetti, as Small does, and it loses the VIF section Small finds. The Divalto answer was still judged right, from other content.
Cohere answers that are worse:
- Zucchetti answers can describe Bonx as a layer on top of the ERP, which contradicts the Q&A.
- French and Italian hosting answers retrieve the Q&A first but never say Google Cloud, while Small's mostly do. In German the result reverses: Cohere always says it and Small never does.
- More answers contain stray English, plausibly because more English content reaches the answer.
A side effect: Cohere pulled Italian pages into French results on two questions.
Voyage: it ranks some vendor sections higher than Cohere (SAP in French at rank 2 against 7), but misses the customer count in French, Italian and German. Its French and German answers then say the figure is unavailable.

Latency, cost and storage¶

Query embedding round trips from a laptop after ingestion finished, 16 queries per model. These are not full-chat latency.

Model	Median ms	Dimensions	Storage vs Small	USD / M tokens	Monthly scenario
Small	213	1,536	1.00×	0.02	$2.09
text-embedding-3-large	237	3,072	2.00×	0.13	$13.58
Gemini Embedding 2	289	3,072	2.00×	0.20	$20.90
Voyage 4 Large	342	1,024	0.67×	0.12	$12.54
Cohere Embed 5 Pro	349	2,048	1.33×	0.12	$12.54
BGE-M3	884	1,024	0.67×	0.01	$1.04
Qwen3 Embedding 8B	1,343	4,096	2.67×	0.01	$1.04

The scenario is 14,944 monthly retrievals at 300 query tokens each, plus a hypothetical 100M ingestion tokens that were not measured. It excludes answer generation, infrastructure and migration.

Re-embedding Bonx cost an estimated $0.03 per model. Building three indexes on about 660k records took about 4 minutes on the test cluster for each model.

Time to first token on the real bot, Bonx lite, 72 paired turns in random order from a laptop:

Measure	Small, median	Cohere, median	Paired difference, median	Middle half of differences
Retrieval	0.27 s	0.42 s	+156 ms	+94 to +218 ms
Time to first token	1.11 s	1.11 s	-4 ms	-201 to +204 ms
Full answer	2.40 s	2.30 s	-70 ms	-359 to +353 ms

Large, measured the same way in a separate run, showed no slowdown: time to first token 0.99 s against 1.01 s for Small, and retrieval 0.22 s against 0.23 s.

For Cohere, retrieval is measurably slower, but time to first token shows no measurable change: variation in the rest of the turn is larger than the gain or loss, and why the extra retrieval time does not show up was not traced. mix workspaces make several embedding calls per turn and were not measured. Production latency may differ, because the server, not a laptop, would call OpenRouter.

Isolated answer comparison¶

Separately from the real-bot pilot, on 30 fixed cases generated outside the chat graph, a blind model judge compared Cohere and Voyage: it preferred Cohere 17 times, Voyage 8, with 5 ties. Useful answers were 29 of 30 and 26 of 30. The sample is small and cross-language heavy, and Small was not in this comparison. It points the same way as the Bonx blind review.

EU processing¶

Today's production embeddings use the EU OpenAI route. The tested OpenRouter routes for Cohere and Voyage are global. As of 2026-11-04:

Option	EU status	Catch
Cohere on OpenRouter	None	Global only
Cohere on Microsoft Foundry	No EU Data Zone	No date, per a Microsoft moderator on 2026-10-23
Cohere on SageMaker, own region	Possible, regions not confirmed	Always-on endpoint, about $1,700 to $2,450 a month in software fees before AWS costs
Cohere Model Vault, Standard	EU region available; Embed 5 Pro there not confirmed	Single-tenant, listed from $3.00 an hour, about $2,190 a month per instance
Cohere Embed v4 on Amazon Bedrock	EU routing listed (`eu.cohere.embed-v4:0`)	Older model, not benchmarked here
Gemini Embedding 2 on Vertex AI	EU multi-region processing listed	Not tested through Vertex

The operator approved the global route for the Bonx pilot. That approval does not extend to other workspaces.

Problems the embedding does not fix¶

Found while reading the pilot answers:

English in French answers. Phrases such as "Biggest difference:" appear in all three setups.
Wrong product name. One answer called the product by the client record's name instead of Bonx.
Stale website data. The indexed /about page says 60+ manufacturers and the live page says 80. Small repeated the old figure in 5 of 108 answers, and Cohere and Voyage in none.

Final comparison on current develop¶

All earlier Bonx results ran on a base that predated IX-5968 and IX-5899. This one reran the same 36 questions, three runs each, on one develop commit. A fresh reviewer scored all four setups together.

Setup	Right section retrieved, of 33	Answer check, of 99	Right	Partly right	Wrong	Judged best
Small, older base	24	68	22	8	3	5
`develop`	28	80	24	7	2	5
`develop` + 3 reserved slots, first	30	88	27	5	1	10
Cohere, older base	30	74	24	9	0	8

Cohere was not rerun on develop, so that row is not a like-for-like comparison.

Order matters. On the older base, adding the reserved chunks at the end of the context made answers worse (19 right, 6 wrong), and placing them first fixed that (23 right, 3 wrong).
Speed. Median time to first token was 1.22 s with slots and 1.23 s without, in parallel runs. A paired run on the older base measured +48 ms, from about 140 ms more retrieval.
Other clients. rose-eval e2e on main-dataset, 43 items, three repeats, same commit, on test: 0.848 without slots and 0.864 with them. 6 items improved by more than 0.03, 3 got worse, 34 stayed the same. The scorer swings by up to 0.3 on one item between runs, so this shows no regression rather than a gain.
Remaining miss. The Italian customer count still says 60+, from the stale indexed page.
Minimum score. 0.68 does not filter much on some clients: loosely related Pennylane FAQs scored 0.83. It stops greetings on Bonx, but was calibrated on that client only.
Coverage. Only FAQs and managed documents get slots. Legacy Drive documents do not until migrated.

Limits¶

Small samples. 51 verified questions across three clients and 33 on Bonx, chosen rather than sampled, so these are not population estimates. Voyage's gain over Small has a paired p-value of 0.065 before correction.
The hard Bonx set was written with the English knowledge in view. It deliberately targets what French and Italian pages lack, so it measures the reported failure, not typical traffic.
The blind reviews are model reviewers on one run per question. No human checked them, and answers vary between runs. Five reviewers gave materially different scores to the same Small and Cohere answers (Cohere 21 to 28 right, Small 21 to 24). Their reference is Bonx's own knowledge, which favours setups that retrieve it.
Gemini has not run on the real bot, and neither Cohere nor Voyage has on any other client.
Lite only on Bonx. Bonx is lite, so the blind reviews do not cover mix retrieval. Only the e2e comparison on main-dataset exercises the reserved slots on advanced clients.
Not production. The pilot used test vectors, plus 67 chunks copied from production so the corpus matched, and one older chunk remains there. Its route lookup asked for production's route because Bonx's corpus is bound to production.
Stored-data findings. The diagnostics on Eskimoz and other workspaces reported lost chunks and drift, which were preserved rather than repaired.

Open decisions¶

Rescan Bonx's website in production. This fixes the stale figures with any model.

Reproducing¶

Raw questions, passages and answers contain client data and stay out of the repository, under .context/cross-language-eval/ and .context/ix5976-pilot/ in the workspace that ran them. The full four-client report is the completed seven-model comparison.

The per-workspace embedding migration code used for the model tests was not shipped. rose-chat only serves a site: workspace in the environment its corpus is bound to, so testing a production-bound workspace needs a small harness that reads the route for that environment and sends deployment_target="evaluation", which skips conversation storage and lead webhooks. A branch-versus-base rose-eval comparison must run on one commit: an older branch mixes in everything merged since.