Skip to content

AI assistants alone vs with our clients' Rose agents

Date: 2026-10-16 · Ticket: IX-5606 · Data: Langfuse main-dataset, 33 questions on 8 brands · Plan: docs/plans/2026-10-16-ix-5606-client-stats.md

TL;DR. Rose is the reference: its answers were curated with the clients. Asked the same 33 main-dataset questions with web search alone, Codex is fully right on 12 and Claude Code on 7; they miss the main point on 5 and 8 and make a major factual error on 1 and 3. What they lack is what public pages do not say: rules the client has not published yet (Pennylane's new pricing model), what a service covers (PayFit's payroll specialists, supported collective agreements), the client's own figures and positioning. Given the client's Rose agent through the connector, both get closer to Rose's answer: Claude Code's facts score goes from 0.65 to 0.84 (better on 25 questions, worse on 4, p = 0.0001), Codex's from 0.73 to 0.80 (better on 16, worse on 5, p = 0.027).

Question

When a prospect asks a general AI assistant about one of our clients, how much closer to the client's own answer does it get when it can question the client's Rose agent (Rose for Agents, ask_vendor) than with web search alone? The assistants are the four prospects use: Codex, Claude Code, ChatGPT and Claude chat.

Method

  • Truth. main-dataset (Langfuse, the website agent's e2e regression set), whose reference answers were curated with the clients: the 33 of its 46 items a prospect could ask an outside assistant. Left out: follow-ups that depend on earlier turns, qualification replies ("Business", an email address) and chat tests (greeting, prompt injection). Pennylane 8, Rose 7, Sunday 4, Lucca 3, Hexa 3, PayFit 3, AB Tasty 3, Skello 2. The backoffice answer-check datasets were tried and dropped: half their golds are past agent answers accepted by an unrecorded reviewer, and two clients make up most of them.
  • Wording. Each question is prefixed with the vendor's name and domain, since an outside assistant does not know which website the visitor was on.
  • Arms. Each assistant alone (live web search) and with the Rose connector (the r4a-mcp stdio server and a short instruction, bench/AGENTS.vendors.md): Codex gpt-6.1-sol medium, Claude Code claude-opus-5-5 with no user settings. Plus the client's Rose agent asked the question directly, as on its website. Rose agents answered from staging.
  • ChatGPT and Claude chat have no headless runner: the 33 main-dataset questions are pasted into the apps in two batches (.context/ix5606/prompts/); results below once in.
  • Reference. Rose is right by definition. Assistants alone are graded against main-dataset's curated answer; assistants with the connector against the Rose agent's live answer to the same question in this run (the curated answer then only chooses the question).
  • Grading, by meaning. Each question's four answers (Codex and Claude Code, alone and with Rose) are shuffled as A-D, so the grader does not know which assistant or arm wrote which, and graded against the reference by Claude agents briefed with bench/GRADER.md: correctness (0-10: no claim that contradicts Rose's answer or is clearly false; extra facts are fine) and completeness (0-10: gives Rose's facts that answer the question, main point first). Style, length, language, formatting, attributions and follow-up questions are ignored: the assistants answer in their own words and are not trying to be the client's chat assistant. Facts combines the two with rose-eval's weights renormalized: correctness^0.615 × completeness^0.385. A major factual error is correctness 4/10 or less (smaller inaccuracies, 6/10, are not counted); main point missed is completeness 4/10 or less.
  • Earlier gradings, dropped. rose-eval's own judge against the curated answers compared wording more than meaning (Rose itself, asked again, scored only 0.75 on correctness) and its flow and format scores grade the chat style; a first 0-3 judge demanded every detail of the reference.

Results: alone, against Rose's curated answers

The reference here is main-dataset's curated answer; the two assistants' answers to each question are shuffled as A and B and graded blind (bench/GRADER.md).

Surface Correctness Completeness Facts Fully right Major factual error Main point missed
Rose (curated reference) 1.00 1.00 1.00 33 of 33 0 0
Codex alone 0.88 0.67 0.78 12 of 33 1 5
Claude Code alone 0.75 0.60 0.67 7 of 33 3 8

"Fully right" is correctness and completeness both 8/10 or more. On Pennylane's "Tarif pour 12 employés ?", Codex quoted the public per-user grid and was graded wrong: Pennylane's new pricing model (PME from 10 employees, custom pricing) is confirmed with Rose but not public yet (see backend/apps/shared_data/prompts/website-agent/skills/clients/pennylane.com/AGENTS.md).

On four extra PayFit questions about collective agreements (truth: PayFit's ccn_coverage lookup table), both assistants answered "can't confirm, ask sales" on the two lesser-known ones (IDCC 3016 covered, IDCC 1875 not covered); Rose answered yes and no.

Results: with the client's Rose agent, against Rose's live answers

Assistant Correctness Completeness Facts Major factual error Main point missed Web searched Median time
Rose agent (reference) 1.00 1.00 1.00 5 s
Codex alone 0.82 0.62 0.73 1 10 13 s
Codex + Rose 0.89 0.69 0.80 1 6 18 of 33 24 s
Claude Code alone 0.74 0.55 0.65 5 10 28 s
Claude Code + Rose 0.88 0.80 0.84 1 3 3 of 33 22 s

Each assistant with and without Rose, question by question, on the facts score:

Assistant With Rose: better / same / worse Sign test Mean change
Claude Code 25 / 4 / 4 p = 0.0001 +0.19
Codex 16 / 12 / 5 p = 0.027 +0.07

On 4 of the 33 questions the Rose agent's answer today contradicts the curated main-dataset answer (see Findings); taking Rose as the truth there marks correct assistant answers wrong. On the other 29 questions the gain is larger: Claude Code 0.67 → 0.88 (better on 22, worse on 4, p = 0.0005), Codex 0.77 → 0.85 (better on 16, worse on 2, p = 0.001).

Every Rose answer called the client's agent.

Findings

  • Rose mostly adds what the web does not have. Alone, both assistants get most facts they state right (correctness 0.82 and 0.74) but miss the main point on 10 questions of 33: the answer is not public or is buried (PayFit's payroll specialists, Pennylane's users per plan, Skello's customer sizes). With Rose they give it.
  • Claude Code gains most. It relays the agent (it searched the web on 3 answers of 33) and its answers with a major factual error fall from 5 to 1: alone, it repeats third-party pages (PayFit's specialists "in principle" give no regulatory help; Rose's answer is yes, for payroll matters).
  • Codex gains less because it already answers well alone and, with Rose, blends the agent's answer with web findings and its own estimates (a "two months" rollout estimate for Lucca).
  • The Rose agent is sometimes wrong, and then it misleads both assistants. On 4 questions its answer today contradicts the curated answer: it asked for a profile instead of quoting Pennylane's PME and TPE prices ("Tarif pour 12 employés ?", "C'est combien pour 3 salariés ?"), gave a sign-up link as Sunday's support page, and gave no link for Lucca's sales terms (that run did not keep the agent's cited sources, so a cited link would not show; run keeps them now). These look like regressions of the agent since curation.

Decisions it supports

  1. Have the agent answer mcp-server questions as a source: give the figures it has (Pennylane's PME and TPE prices) instead of asking for a profile first. Measured with this bench.
  2. Re-check the main-dataset references the assistants contradicted with a current, cited page (AB Tasty's pricing, PayFit Copilot vs PayFit AI) before trusting those items.
  3. Re-run the same questions into a new folder after each change, and compare the tables.

Limits

  • One answer per question and arm, one judge; 33 questions, 2 to 8 per brand.
  • The references come from the same knowledge the agent answers from: that is the benefit measured, not a bias to remove.
  • Staging, not production, answered the Rose calls; their conversations are stored with environment = 'staging', outside the clients' production analytics.

Reproduce

Answers and grades are kept outside the repository, in ~/rose-data/client-bench/main1/ on Benoit's machine; the export is the Langfuse main-dataset items as JSON. The by-meaning grading is in blind/ (alone, curated reference) and blind-vs-rose/ (the batches given to the graders, their results and the A-D mapping), unblinded into grades-blind.json and grades-blind-vs-rose.json. From backend/, with Codex and Claude Code signed in and Secret Manager access:

export IX_ENVIRONMENT=test
.venv/bin/python apps/r4a_mcp/bench/clients.py prepare <main-dataset items.json> <dir>
.venv/bin/python apps/r4a_mcp/bench/clients.py run <dir>          # Codex, Claude Code, Rose agent
.venv/bin/python apps/r4a_mcp/bench/clients.py import-app <dir> <pasted app replies>
# Alone, against the curated answers: blind batches, graded by Claude agents briefed with GRADER.md
.venv/bin/python apps/r4a_mcp/bench/clients.py blind <dir>
.venv/bin/python apps/r4a_mcp/bench/clients.py unblind <dir>
.venv/bin/python apps/r4a_mcp/bench/clients.py stats <dir> --blind
# With the connector, against Rose's live answers
.venv/bin/python apps/r4a_mcp/bench/clients.py blind <dir> --vs-rose --arms codex-base,codex-rose,claude-code-base,claude-code-rose
.venv/bin/python apps/r4a_mcp/bench/clients.py unblind <dir> --vs-rose
.venv/bin/python apps/r4a_mcp/bench/clients.py stats <dir> --blind --vs-rose