Skip to content

Analysis

Data-driven investigations into agent behavior and product metrics. Each entry states a question, method, findings, and reproducible queries.

  • Demo-offer timing vs conversation drop-off — offering the demo CTA early kills sessions (~15pp lower continuation at every turn) with no conversion upside (IX-4063).
  • Jev System One for intent and skill classification — Jev cuts isolated classifier latency substantially, but development-set tuning and sparse skill coverage block a production cutover (IX-5096).
  • Query rewriting vs retrieval quality — prose rewriting never beats the raw query; resolving into LightRAG graph keywords does (+0.050), and a gated resolver lifts elliptical turns on graph-less clients (+0.041 held out) (IX-4447).
  • Codex alone vs Codex + Rose for Agents — after three rounds, on 50 held-out tool-buying questions fact-checked blind, Codex + Rose is wrong on 1 answer against 3 and fully right on 86% against 76%, at 46 s against 39 s and about twice the tokens; full transcript (IX-5364).
  • AI assistants alone vs with our clients' Rose agents — against Rose's curated answers, Codex alone is fully right on 12 of 33 questions and Claude Code on 7, missing what public pages do not say; with the client's Rose agent, Claude Code goes from 0.65 to 0.84 on facts (p = 0.0001) and Codex from 0.73 to 0.80 (p = 0.027) (IX-5606).