Skip to content

Conversion influence: LLM judge vs staff labels

Date: 2026-11-19 · Tickets: IX-6345 (this study), IX-6334 (journal) · Data: 109 conversations labelled by one staff annotator (conversations of 2026-05-25 to 2026-06-22), scored by the production judge ConversionInfluenceScorer.

Question

The goal is to prove value: show that Rose has a decisive impact on conversions, and how often. Staff can label conversations (decisive / positive / neutral / negative) but that does not scale, so the plan was to let an LLM judge do the labelling and calibrate it against the staff labels (target: at least 80% agreement on at least 50 labelled items, plus a chance-corrected statistic).

Can the judge replace the human labels today?

Answer

No. The best condition tested reaches 69.8% raw agreement (95% interval 56.6 to 81.1) and kappa 0.45 to 0.49 on 53 held-out items, below the 80% target, and kappa in that range is moderate agreement, not a substitute for a grader. Until the judge is reworked and re-measured, impact figures have to come from human labels (or from the randomized holdouts), not from the judge.

Method

  • Material: 110 staff-labelled conversations from 4 client sites (97, 9, 2 and 1 items), one annotator, latest label per conversation wins. One conversation had no readable messages, so n = 109 (23 decisive, 61 positive, 25 neutral, 0 negative). 97 items come from one site, and all 23 decisive labels are from it. Selection of the conversations is unknown.
  • Judge input: production parity: transcript, the four conversion booleans (demo proposed, demo booked, CTA clicked, email captured), topics.
  • Conditions: the prompt as it stood (v2, prompt-v2-baseline.txt) and a rewritten prompt (v3, prompt-v3-rewrite.txt), each on gpt-4.1-mini (production default, temperature 0), gpt-5.4, and gpt-6.2-sol (temperature 1, low reasoning effort).
  • Split: items split once, stratified by label and main site vs other, seed 7: tuning half (56) and held-out half (53). The v3 prompt was written from the tuning half only; the held-out half was scored once afterwards.
  • Metrics: raw agreement, per-class precision and recall, Cohen's kappa unweighted and linear-weighted (the scale is ordered), 95% bootstrap intervals (5000 resamples). Raw agreement alone misleads: always answering "positive" scores 56.0% on these 109 items.
  • Langfuse runs (dataset feature-conversion-influence-judge): judge-gpt-4.1-mini-prompt-v2-2026-11-19, judge-gpt-5.4-prompt-v2-2026-11-19-real, judge-gpt-5.4-prompt-ix6345-v3-2026-11-19, judge-gpt-6.2-sol-low-prompt-ix6345-v3-2026-11-19. The run judge-gpt-5.4-prompt-v2-2026-11-19 is mislabelled: it actually ran gpt-4.1-mini with the v2 prompt (script bug) and could not be deleted, because Langfuse removed the run-delete API on 2026-11-16. Do not cite it.
  • No Supabase writes. The prompt is read from the repository file at runtime, not from Langfuse.

Findings

1. The production judge over-calls "decisive"

Staff \ judge (v2, gpt-4.1-mini) decisive positive neutral negative total
decisive 17 6 0 0 23
positive 41 9 11 0 61
neutral 16 3 6 0 25

29.4% raw agreement (21.1 to 37.6%), kappa 0.03 (-0.07 to 0.13). The judge answers "decisive" on 74 of 109 conversations (staff: 23), including 71 of the 88 conversations with a booked demo (staff decisive: 22). It treats a conversion signal as proof the agent caused the conversion, which its own prompt says not to do.

2. A stronger model alone does not fix it

Held-out half, v2 prompt: gpt-4.1-mini 28.3% / kappa 0.05; gpt-5.4 47.2% / kappa 0.04. The error flips instead: gpt-5.4 answers "decisive" on 3 of 109 (staff: 23).

3. A better rubric helps a lot, but not enough

What the tuning-half reading showed, and what v3 encodes: staff "decisive" conversations had several answered evaluation questions (median 4 visitor turns, against 1 for staff "positive"); the booking often happens outside the chat, so it must not be required in the transcript; some staff "neutral" conversations are test sessions or non-buying support questions. v3 defines a buying-evaluation question, makes "decisive" = conversion signal + at least two answered such questions, makes test and non-buying conversations "neutral", and fixes the order of the decision.

Held-out half (n = 53) Raw agreement Kappa Linear kappa
v2, gpt-4.1-mini 28.3% 0.05 0.11
v2, gpt-5.4 47.2% 0.04 0.13
v3, gpt-4.1-mini 60.4% 0.36 0.36
v3, gpt-5.4 69.8% (56.6 to 81.1) 0.45 (0.24 to 0.65) 0.44
v3, gpt-6.2-sol 69.8% (56.6 to 81.1) 0.49 (0.27 to 0.69) 0.53

On all 109 (includes the tuning half, so optimistic): v3 gpt-5.4 64.2% / kappa 0.37; v3 gpt-6.2-sol 63.3% / kappa 0.40. Held-out per class, v3 gpt-5.4 (precision / recall): decisive 0.86 / 0.55, positive 0.73 / 0.90, neutral 0.57 / 0.33. gpt-6.2-sol: decisive 0.82 / 0.82 on held-out and recall 0.87 on all 109; neutral recall about 0.32 to 0.42.

4. The remaining gap looks like label inconsistency

Near-identical single-question conversations (for example "what is included in the free trial?") carry different staff labels: neutral in some cases, positive or decisive in others, and no notes were recorded. More prompt tuning on this set would mostly fit that noise.

What this means for proving value

  • The staff labels remain the only usable evidence of "decisive" impact today: 23 of 109 labelled conversations (21%). That is not a platform rate: the selection of the labelled conversations is unknown, one annotator, and all decisive labels come from one site.
  • The judge cannot yet scale that evidence. Showing impact across clients needs either more human labelling or significantly more work on the judge.
  • The randomized holdouts remain the causal evidence of impact (see Skello widget holdout); a judge label is a per-conversation attribution, not a causal estimate.

Options for more work on the judge (not tested)

  1. Measure the ceiling first: a second annotator on 30 to 50 of the same conversations gives an inter-annotator kappa to compare with 0.45 to 0.49; if humans agree no better than that, the judge cannot be pushed further.
  2. Labels with notes, so inconsistent cases can be explained and the rubric corrected from them.
  3. More sites: at least one held-out site, because 89% of the items come from one site and the rubric text encodes patterns seen there.
  4. Repeat runs to measure judge variance (one run per condition so far).
  5. Judge as a filter, human as the grader: on held-out data v3 gpt-5.4 judges "decisive" with precision 0.86; sending only its "decisive" calls to a human to confirm would cut labelling effort (precision 0.86 on 7 judged items is not enough to rely on; measure on more items).
  6. Model and cost: gpt-4.1-mini is retired from the API on 2027-01-14 (IX-5581); gpt-6.2-sol is not yet in the model catalogue. Cost and latency were not measured.

Threats to validity

One annotator and no inter-annotator measure (construct). Unknown selection of conversations (selection). 89% of items from one site; no negative labels, so the negative class is untested; 91 of 109 conversations have a completed conversion (demo booked, link clicked or email left; 7 more only had a demo proposed), so conversations that did not convert are few (external). The v3 rubric was written after reading tuning-half conversations next to their labels, by the same agent that ran the evaluation; the held-out half protects the reported numbers but not the rubric from site-specific patterns. One run per condition at one temperature; the held-out half scored higher than the tuning half (69.8% vs 58.9% for v3 gpt-5.4), consistent with noise at n = 53 and 56. Intervals are wide: 80% is not excluded at n = 53.

Reproduce

Files in 2026-11-19-conversion-influence-judge/:

  • prompt-v2-baseline.txt, prompt-v3-rewrite.txt: the two judge prompts (v3 is the working-tree edit of backend/apps/shared_data/prompts/conversation-processing/conversation-conversion-influence.md on branch feature/ix-6345-improve-the-conversion-influence-judge-so-it).
  • results-per-item.csv: per conversation, site, split, signals, staff label, and the judge label for four conditions. No conversation text or visitor data.
  • export_items.py.txt (writes /tmp/judge_run/items.json from the dataset and split.json from the CSV; run first), run_judge_experiment.py.txt (logs a Langfuse dataset experiment), local_eval.py.txt (scores a split locally), metrics.py.txt (confusion matrix, kappa, bootstrap). Scratch scripts, run from backend/ with IX_ENVIRONMENT=test, after source backend/scripts/load-dev-secrets.sh; each calls load_main_env() itself. Run them under .venv/bin/python after renaming to .py.
  • The dataset is created by cd backend; poetry run python -m cli.langfuse.commands.sync_feature_conversion_influence (idempotent).