Jev System One for intent and skill classification

Date: 2026-09-19 · Ticket: IX-5096 · Data: intent-classifier (113 items), a frozen 48-item multilingual check set, a frozen 32-item boundary set, and skill-selector (10 items).

Question

Can TypeSafe System One (jev-latest) replace Rose's general-purpose LLM calls for intent classification or skill selection while preserving behavior and reducing the latency before answer generation?

The answer is not yet for production. Jev is fast and can express both a single intent choice and multiple independent skill decisions, but the available quality evidence is not independent enough for a cutover. The retained code is an evaluation adapter; production routing and labels are unchanged.

Method

The existing rose-eval harness was extended with evaluation-only classifiers:

• Intent uses one Jev Choice over the production intent labels.
• Skill selection sends one independent Jev Noul question for every visible skill, applies the neutral 0.50 threshold, then reuses deterministic condition, requirement, and dependency handling.
• Both candidates consume the same dataset inputs as their production controls.
• Latency benchmarks run items sequentially so provider concurrency does not hide per-request latency.

Intent criteria were changed one boundary at a time and measured on the 113-item dataset. This makes the final intent score a development score. Likewise, alternative skill formulations and a threshold sweep were examined on the same 10 skill items. The 0.80 threshold that scored perfectly was rejected; the prototype retains the untuned 0.50 threshold.

The full experiment history, including every retained and reverted intent change, is recorded in docs/plans/20260919095828-ix-5096-jev-intent-classifier-eval.md.

Results

Intent classification

Classifier	Accuracy	Macro-F1	Mean latency	p95 latency
Production azure/gpt-4.1-mini	0.982	—	1.06s	1.24s
Jev, initial one-line criteria	0.796	0.763	0.26s	0.35s
Jev, retained nine-label criteria	0.9646	0.9571	~0.25s	—

The tuned Jev arm closed most of the original quality gap while remaining about four times faster in isolation. It still did not match the production classifier's accuracy. More importantly, the criteria were developed by reading errors from the measured dataset, so 0.9646 does not estimate unseen production quality without optimism.

The additional multilingual and boundary sets caught regressions during the experiment, but they do not repair that independence problem: they were created and interpreted inside the same investigation rather than supplied as unseen production gold by an independent annotator.

Taxonomy experiment

Merging out_of_scope_task into offtopic simplified classification and did not hurt the normalized evaluation. It was not promoted because the two labels have different runtime effects:

• offtopic briefly redirects a harmless aside and preserves an active booking.
• out_of_scope_task exits that flow and activates a stronger refusal guardrail.

The labels therefore remain separate. The merged arm exists only to measure the classification cost of that distinction.

Skill selection

Classifier	Skill recall	Micro-F1	Exact match	Jaccard	Mean latency
Production cerebras/gpt-oss-120b	1.000	0.8333	0.600	0.800	0.501s
Jev Noul per visible skill	1.000	0.9091	0.800	0.900	0.246s

The Jev result repeated across three complete runs. It missed no expected skill and added two false positives. This establishes API and composition feasibility, not production quality: the dataset has only 10 items and its expected labels cover only clean_ending, demo_offer, and content_gating.

A higher-scoring formulation used one Choice for the required category. It was rejected because production deliberately permits multiple matching skills from one category for downstream arbitration. Improving the metric by removing that behavior would make the comparison invalid.

Latency on the answer critical path

Intent classification and skill selection run in parallel with interest-signal detection, demo-readiness judging, and conditional early profile extraction. The pre-answer gate is therefore the slowest branch, not the sum of all branches:

pre-answer gate = max(intent, skills, interest, readiness, early profile)

Replacing only the skill selector reduces that call by about 255ms but is unlikely to reduce normal answer latency because the current intent classifier is slower. Replacing both intent and skills could reduce the ordinary analysis gate from roughly 1.06s to the approximately 0.38s sibling-node range, but the estimated ~0.7s saving has not been measured in a real graph A/B. Early profile extraction can still dominate turns where it runs.

Jev is not a general replacement for early profile extraction. Its outputs are constrained to Choice, Score, and Noul; it cannot return an arbitrary company name, role, problem, or other freeform value. It could handle predefined qualification fields, while deterministic parsing could cover simple email, URL, or numeric values. That is a separate experiment.

Why the next dataset must be hidden

A hidden holdout is a frozen set whose expected labels are unavailable to the person or process tuning the candidate. Once individual errors are inspected and used to change criteria, that set becomes development data even if its filename still says holdout.

A production decision requires a new set built as follows:

1. Sample fresh, anonymized production turns with the full state used by each classifier.
2. Cover every intent and important skill, multi-skill overlaps, booking and qualification state, several clients and languages, and ambiguous negative cases.
3. Have an independent annotator assign gold labels and resolve disagreements; do not use either evaluated arm as the answer key.
4. Freeze inputs, labels, rubric, and hashes before running either arm.
5. Name the arms A and B, run both once, lock metrics, and reveal the mapping only after the verdict.
6. If the result causes another prompt change, retire that set to development data and use a new hidden set for the next production claim.

As a starting floor, use 200–300 intent turns with at least 20 examples for each rare label, and 150–250 skill turns covering every important skill and state combination. Report both a distributional sample and a deliberately stratified boundary slice so common traffic does not hide rare safety failures.

Decision

IX-5096 answers its feasibility question:

• Keep the evaluation adapters so the result is reproducible.
• Do not change production routing or merge production labels from this data.
• Treat Jev intent classification as a candidate for a separately tracked, independently labeled holdout evaluation.
• Treat Jev skill selection as underpowered until the skill dataset covers the real label and state space.
• Measure whole-turn time to first token in the real graph only after quality clears the hidden holdout.

Reproduce

From backend/ with evaluation credentials loaded:

uv run rose-eval intent-classifier intent-classifier --classifier llm --json
uv run rose-eval intent-classifier intent-classifier --classifier jev --json
uv run rose-eval skill-selector skill-selector --classifier llm --json
uv run rose-eval skill-selector skill-selector --classifier jev --json

TYPESAFE_API_KEY is required for the Jev arms. jev-merged remains available for the evaluation-only eight-label taxonomy comparison.