January 2026 - Orchestration Strategy Selection and First Multi-Agent Implementation¶
Context¶
December 2025 closed with a working proof-of-concept of intent-based routing to specialized handlers and a validated belief that the architecture was feasible. It left open the harder question that shaped January: which orchestration strategy is correct for production-scale, multi-tenant traffic, and how to build the first real version of it without paying an unacceptable latency penalty. January was therefore two things at once — a structured literature review that turned the open question into a small set of testable hypotheses, and the first month in which the chosen direction was actually implemented rather than only designed.
Three uncertainties carried over from December and framed both halves of the month: how to minimize time-to-first-token without losing routing intelligence; how to defend the agent against indirect prompt injection carried inside retrieved documents and tool outputs; and how to overlap useful work (retrieval, tool calls, reasoning) with token generation instead of serializing those steps. Active deployments during the period spanned four B2B SaaS clients — in experimentation, accounting, banking, and HR — plus the Rose website itself. Client identities are withheld; nothing in this entry depends on them.
January's work maps onto three of the four standing 2026 R&D projects. The orchestration review and the first multi-agent/skill implementation belong to Adaptive Multi-Tenant Conversation Orchestration. The retrieval-latency, knowledge-freshness, and pre-chat enrichment work belongs to the Compounding Context Engine. The evaluation-dataset and observability-hardening work belongs to Closed-Loop Agent Evaluation and Optimization. The fourth project, outbound knowledge publishing, did not advance this month and is noted as such in the closing section. No new project was created; the orchestration "research areas" of the prior draft are folded back into the standing projects rather than tracked as one-off names.
Adaptive Multi-Tenant Conversation Orchestration¶
This project had two faces in January: a literature review that selected an orchestration strategy and recorded one strategy as structurally unavailable, and the first implementation pass that turned the December design into running code on the development branch.
Project and lock¶
The objective is to select skills, models, and deterministic business contracts live during a multi-tenant conversation while keeping the first token fast. The central lock is that no single orchestration strategy optimizes all the competing objectives at once — time-to-first-token, the ability to stream useful work concurrently, resistance to indirect prompt injection, token and call efficiency, and answer quality as a function of how much context the model must read. December committed to dedicated agents with LLM-based intent routing, but that is one point in the design space, not a demonstrated optimum: the state-of-the-art studies each technique in isolation and offers no framework for choosing and combining strategies against a concrete constraint set. A second, sharper lock surfaced during the review — the combinatorial-explosion problem. When behavioral dimensions multiply (response style × domain expertise × per-client customization × conversation phase), a dedicated-agent-per-combination design explodes: four styles × three domains × ten clients is already 120 distinct agent configurations, each needing its own maintained prompt, which defeats the modularization it was meant to provide.
This month's work¶
The review compared five orchestration strategies — a monolithic mega-prompt, dedicated agents, a dynamic skill-injection system, streaming function calls, and training-free statistical routing — against the objectives above, drawing latency and behavior figures from the source literature rather than from Rose measurements. It produced one firm negative result worth retaining: the cheapest known routing optimization, training-free statistical routing that reads the main model's own prefill activations to classify intent at near-zero added latency, is structurally closed to Rose. Those methods require access to hidden-layer activations, and Rose runs on black-box LLM APIs with no such access. That conclusion redirected the latency work toward the two black-box-compatible routes — embedding-similarity matching against skill descriptions, and a small fine-tuned external classifier — both of which need labeled conversation data to train, a different cost profile than the barred training-free methods.
The review also reframed quality as a function of context size: smaller, relevant context produces fewer hallucinations and better instruction adherence than a long prompt full of conditional branches. That reasoning, plus the combinatorial lock, made a skill system the leading hypothesis over dedicated agents, on the condition that skill selection itself does not reintroduce a second time-to-first-token penalty. The review's answer was a two-layer routing idea — cheap keyword/embedding pre-filtering for the bulk of turns, escalating to a premium model only for execution, with a decision-token pattern that lets a simple question skip skill injection entirely and begin answering immediately.
The implementation half of the month turned this into code on the development branch. The agentic structure was reworked and expanded to support multiple specialized agents rather than a single handler, and intent-classification state management was simplified so the classifier could run as one of several parallel analysis nodes feeding a deterministic router. A skill package was built alongside it: skills are file-based units carrying YAML frontmatter (name, description, category, activation conditions, dependencies, a flag to hide a skill's existence from the model, and a category that controls how its instructions enter context) plus markdown instructions. A skill-selector node uses a fast model to choose candidate skills, then deterministic heuristics refine the selection — exclusivity rules, frequency limits, and auto-injection of system skills. The deliberate departure from the literature is single-context aggregation: rather than isolating each skill in its own LLM call as the modular and multi-agent constructions in the literature generally do, every selected skill is aggregated into one call so the skills can inform one another within a single reasoning pass. The open uncertainty this creates is exactly the instruction-collision risk that per-agent isolation avoids by construction — whether one call can carry several composed skills while keeping reasoning coherent. The business-contract layer also advanced: the in-chat booking flow was made a globally configurable capability that defaults off, keeping CTA and qualification arbitration a per-tenant contract rather than a hard-coded behavior.
Results, proof, and next step¶
The retained outcome is a selected strategy, a documented negative result, and a first running implementation — not a measured quality or latency win. The skill-selection system, the expanded multi-agent structure, and the booking-arbitration contract exist on the development branch; none was rolled out to production in the period, and no A/B test was run. No controlled experiment was live in production at any point in January — the most recent prior experiment had ended in mid-December 2025, and the next was not created until well after the period. A measured result is intended to come either from a launched experiment or from a post-production before/after on the live conversion and engagement events once the work ships; neither was possible in January, because the only orchestration change that reached production was a single client's migration onto the new agentic system in the last week of the month — a post-deployment window too short and too confounded by concurrent changes to read as a result. The four working targets for the eventual skill-system test (latency at or below the monolithic prompt, instruction adherence above ninety percent, skill-selection accuracy above eighty-five percent, and user-satisfaction signals) are stated as targets, not results — with one now partially grounded: the skill-selection accuracy target has a first offline measurement against a small labeled dataset (about 0.83, just below the eighty-five-percent bar), reported under Closed-Loop Agent Evaluation and Optimization below; the other three, and any production comparison, remain pending. The skill runtime, the selector node, and the deterministic refinement layer are the substrate without which the dedicated-agents-versus-skill-system comparison cannot be run at all. Per-client skill content authored on top of this runtime is ordinary configuration. Next step: run the skill-system versus dedicated-agents versus monolithic-prompt comparison on production traffic with statistical-significance analysis, and build the embedding-based router as the black-box-compatible latency alternative.
Compounding Context Engine for Company, Industry, and Buyer Intelligence¶
Project and lock¶
This project owns the knowledge and state that flow into the agent at answer time. Two locks were active in January. The first is latency: retrieval sits on the critical path, so the time spent fetching documents is time the visitor waits before the first token, yet retrieval is not always needed and cannot simply be removed. The second is freshness and coverage: company knowledge drifts as clients update their sites and collateral, and the agent has little context about a visitor before the first message, so both the knowledge base and the per-visitor context must be kept current without manual intervention.
This month's work¶
To attack the latency lock, speculative document retrieval was implemented in the retrieval node. The mechanism fires retrieval as a background task at graph start without blocking on it; downstream the result is either awaited on the answer path or cancelled if the conversation is redirected before retrieval is needed. This removes retrieval from the critical path when it matters and discards the wasted work when it does not. The complementary idea of speculative generation — starting to generate a response on partial context and folding in full retrieval results as they arrive — was deliberately left unbuilt and is recorded as an open item rather than a claim.
To attack freshness and coverage, a scheduled job was built to refresh the knowledge base from clients' source documents automatically, with hardened error handling and operator notifications so a failed refresh surfaces rather than silently leaving stale knowledge in place. On the visitor side, a pre-chat enrichment path was added: an impression endpoint captures and enriches visitor context before the first message is sent, and behavioral-event capture and session-event tracking were extended (including forwarding events to the parent page's analytics) so the buyer-level state available to the agent compounds across the visit rather than starting empty at each turn. Retrieval scoping was also tightened — document tagging now pre-filters by the client's taxonomy and fetches additional categories dynamically only when the first pass skips them, keeping retrieved context relevant to the tenant rather than drawing broadly.
Results, proof, and next step¶
The speculative-retrieval change is implemented but its latency saving on Rose traffic was not measured in the period; the figure is pending the same observability pass that the orchestration work needs. The knowledge-refresh job and the pre-chat enrichment path are running enabling infrastructure — they keep the inbound context fresh and non-empty, which is the precondition for any later measurement of context quality. No experiment isolated their effect during January, so no causal number is claimed. Next step: measure the speculative-retrieval latency saving from production traces, and quantify whether pre-chat enrichment changes first-turn answer quality.
Closed-Loop Agent Evaluation and Optimization¶
Project and lock¶
This project builds the measurement substrate that decides whether any orchestration or context change is safe to ship. The lock in January was that there was no systematic way to evaluate the new multi-agent and skill behavior before rollout: changes were validated by hand, observability was noisy, and prompt/skill versions could drift between the prompt manager and the running system without a reliable way to detect the conflict.
This month's work¶
The central piece was a conversation dataset and evaluation harness — a curated set of conversations and the scaffolding to run candidate agent configurations against them, so a skill or routing change can be scored against known cases rather than judged anecdotally. Alongside it, an investigation traced empty-answer cases that appeared in observability traces and in the test environment, attributing and fixing the conditions that produced silent non-answers — a failure mode that any eval is blind to unless it is first made visible. The prompt/skill synchronization layer was hardened to detect version conflicts between the managed prompt store and the running configuration and to manage versions explicitly, with an explicit conflict handler added to the synchronization tooling so a divergence is surfaced and resolved rather than silently overwritten. Observability itself was made trustworthy: error reporting was de-duplicated to emit one event per failure instead of one per retry, session replay was disabled where it added noise without signal, and event filtering was tuned so the traces feeding any future eval reflect real failures rather than retry storms.
Results, proof, and next step¶
The retained outcome is the evaluation and observability substrate and — for the first time in the period — a scored run on it. The conversation dataset and harness exist, and in the second half of January the skill-selection node was scored against a small multi-label dataset of roughly two dozen labeled cases: the skills the agent selected matched the expected set at a micro-averaged F1, a skill-recall, and a Jaccard overlap all near 0.83, but only about half of the cases were fully correct on every label, an exact-match pass rate near 0.50. This is a first, small-sample baseline rather than a mature benchmark — the dataset is small and the runs were exploratory item-level probes — but it is a grounded measurement, not an assertion, and it sits just below the working target of eighty-five percent skill-selection accuracy, which quantifies how far the single-context skill aggregation still has to go and exposes that full-case correctness lags the per-label overlap. The empty-answer fix and the synchronization conflict handler are corrective hardening that the eval depends on — without de-duplicated, faithful traces and a reliable version-conflict signal, an eval score cannot be trusted. The measurement substrate is the apparatus the whole closed loop depends on; the first scored run now also gives it an output. Next step: expand the dataset beyond the first two dozen cases, add an LLM-as-judge for instruction adherence, and gate the production rollout on those scores.
Non-R&D / Productization Context¶
Substantial standard development shipped in the period and is excluded from the R&D scope above. Widget and delivery work included AI Sections with CSS scoping to prevent style conflicts, glassmorphism and chat-view styling, Shadow DOM CSS-loading fixes, a CDN cache-purge and stale-while-revalidate path with version verification, and a guard preventing the widget from loading inside iframes. Reliability and platform work included Redis connectivity fixes, async Supabase access methods to avoid blocking the event loop, Cloud Run startup-probe and VPC configuration, and a Supabase backup/restore CLI. Maintenance work included an admin-panel QA pass, centralizing backoffice local storage, splitting the agent guide and symlinking it, client-facing documentation builds, dependency bumps, and lint/CI configuration. One client was migrated onto the new agentic system during the month; that migration is productization of the retained orchestration work, not new R&D. No outbound knowledge-publishing (GEO) work occurred in January, so the Synthesized Knowledge Publishing project did not advance this month.
Prior-Month Results Review (December 2025 shipped work)¶
Because production changes need a post-deployment window before they can be read, each month reviews the prior month's shipped work as a quasi-experiment once enough live data has accumulated. December 2025's retained R&D that reached production was the parallel-convergence graph architecture and the intent-router proof-of-concept. The router's POC routed every detected intent to the same answer handler, so it changed the system's internal structure without yet changing user-facing behavior — the design predicts no measurable shift in engagement or conversion from December's work alone.
The live data is consistent with that prediction. The widget-engagement rate — the share of widget impressions that produce at least one sent message, the cleanest Rose-owned metric for this question — was essentially flat across the December-to-January boundary at roughly two percent, after rising through the autumn and before rising again in February. A separate presentation-style experiment that concluded in mid-December also returned a null on engagement (reported in full in December's prior-month review), so the flatness at this boundary reflects two converging reads — a structural routing change with no behavioral surface and a display-style change with no measured effect — rather than a single inconclusive signal. The month-to-month movement at that boundary sits inside the ordinary noise band of the surrounding months, so no engagement lift can be attributed to December's architecture change. A naive interaction-to-form-submission ratio was examined and discarded as invalid: the client-form-submitted signal counts site-wide form submissions tracked across each client's pages, not chat-gated conversions, so it vastly exceeds the chat-message volume and cannot serve as a conversion denominator here.
This is a deliberately weak, observational read, not a randomized test: there is no control group, the clients and traffic mix shifted over the window, and other changes shipped in the same period. Its value is the honest confirmation that December's structural work behaved as designed — invisible at the behavior level — which is the expected result for an architecture POC that does not yet differentiate handling. The first measurable user-facing effect should appear only once intent-specialized handling and the skill system reach production; that is the change the corresponding future retrospective will test.
Research Outcome¶
January resolved the orchestration-strategy question enough to act on it: a five-strategy comparison, a retained negative result barring training-free statistical routing on black-box APIs, and a decision to pursue a single-context skill system guarded by two-layer routing, expressed as four testable Q1 hypotheses (embedding-based routing for latency; skill system versus dedicated agents versus monolithic prompt; streaming function calls; and context reduction for quality). It also moved from design to implementation: the expanded multi-agent structure, the file-based skill runtime and selector, speculative document retrieval, automated knowledge refresh, pre-chat enrichment, and the conversation-evaluation harness all exist on the development branch. None of this shipped to production or ran as an experiment in the period — no controlled experiment was live in January, and the work remained on the development branch rather than in production — so every production quality and latency claim remains a working target pending the Q1 measurement pass. The one grounded measurement is offline: a first scored run of the skill-selection node against a small labeled dataset returned about 0.83 on per-label overlap with only half of cases fully correct, a baseline just below the eighty-five-percent target. The honest state at month end is a selected strategy, a built substrate, and a single small-sample eval baseline, with the comparative production results still to come.
Sources¶
A literature-review month rests on external sources; the two highest-risk are verified with identifiers, and items marked (citation to confirm) are real techniques cited informally during the review whose author/venue/identifier should be confirmed before any external CIR/JEI submission.
Academic papers¶
- "Fast Intent Classification for LLM Routing via Statistical Analysis of Representations" (NeurIPS 2025; OpenReview UMuVvvIEvA) — NormStat and VecStat, training-free intent classification during prefill with negligible latency overhead; the methods barred by Rose's black-box-API constraint.
- "GhostShell: Streaming LLM Function Calls for Concurrent Embodied Programming" (arXiv:2508.05298) — issuing function calls incrementally as tokens stream; reports large response-time gains in an embodied-robotics setting whose transfer to a conversational API context is the untested Q1 hypothesis.
- "Intent Detection in the Age of LLMs" (citation to confirm) — evaluation of LLMs for intent detection; uncertainty-based hybrid routing.
- "Modular and Hybrid Frameworks for LLM-Based Agents" (citation to confirm) — task decomposition and specialized prompts outperforming monolithic agents on complex tasks.
- Route0x (citation to confirm) — embedding-similarity routing handling most intent detection at minimal latency, avoiding LLM routing overhead.
- Decision-token pattern (citation to confirm) — forcing a decision token as the first output so simple queries answer immediately without skill injection.
- Bottom-Up-Then-Top-Down skill orchestration (citation to confirm) — building systems from atomic tasks orchestrated by a supervisor, parallelizing independent skill calls.
Technical industry documentation¶
Named, real organizations cited descriptively rather than by linkable reference: agentic design-pattern catalogs (chaining, routing, parallelization, orchestrator-worker, evaluator-optimizer); vendor inference-performance guides defining TTFT, time-per-output-token, and inter-token latency; agent-router guides on intent classification and semantic matching; layered defenses against direct and indirect prompt injection; and modular-prompting guidance on HTML-like tags that delimit trusted instructions from untrusted input.