Skip to content

July 2026 - R&D Journal: Experiment Validity, Demo-Offer Timing, and Evaluation-Harness Integrity

Context

This entry covers July 1-31, 2026. Rose moved from building measurement infrastructure to running controlled experiments on it, and most of the month's research came from what that exposed. Three randomized experiments, each holding out a share of a client site's visitors from the agent, launched between July 15 and July 17. Getting them to a valid read surfaced a sequence of defects in the experiment apparatus, one of them a statistical bias that manufactured a lift. In parallel, the rule deciding when the agent proposes a demo was replaced, after an observational study associated early offers with less continuation and no clear conversion benefit, and the evaluation harness guarding those changes was found to pass items it should have failed.

Closed-Loop Agent Evaluation and Optimization carries the experiment-validity, metric-definition and harness-integrity work; Adaptive Multi-Tenant Conversation Orchestration the demo-timing policy and its deterministic contracts; Compounding Context Engine for Company, Industry, and Buyer Intelligence conversation identity across visits and the correctness of knowledge intake. Synthesized Knowledge Publishing for Generative-Engine Discoverability had no research activity in July. Much of the month's engineering was productization, summarized separately.

None of the July experiments reached its planned end within the month, so no verdict was read; their designs and allocation checks are reported below. The June staff-wide holdout stopped on July 15; its read is in the Prior-Month Results Review.

Evaluation runs from this period are no longer retained (the store keeps records from late August onward). The evaluation scores below are those recorded at the time of each change and could not be re-read for this entry.


Closed-Loop Agent Evaluation and Optimization

Project and lock

June materialized experiment results into precomputed tables (marts) with explicit metric descriptors. July was the first month those marts had to support a live decision for a named client, and the lock moved from describing a metric to validating a causal comparison.

The agent's effect on a visitor is delayed: a visitor who chats may convert days later, on another page or subdomain, and the journey itself changes shape when the agent is present. Several standard practices of online experimentation therefore break in ways not documented for this setting: filtering the cohort, segmenting by engagement, choosing the read date, and trusting a sample-ratio check. Whether a holdout comparison on this product can be made valid, and with which rules, was open.

This month's work

Hypotheses

  • H1. Carving a holdout group from a client's traffic, with assignment by a stable hash of the visitor identifier, yields arms whose composition matches the design and whose comparison is unbiased for the agent's effect on conversion.
  • H2. The metric definitions inherited from the June marts measure the same quantity across the staff and client surfaces and across arms.
  • H3. The evaluation harness that gates prompt and skill changes fails an item when, and only when, the behaviour under test is wrong.

Experimental setup

Table 1 lists the three experiments launched in July. Each holds out a fixed share of a site's visitors from the agent. Duration was fixed at launch from a target minimum detectable effect at 95% confidence and 80% power. Conversion is the site's own tracked conversion events, filtered by each client's conversion rules.

Table 1. Randomized holdout experiments launched in July 2026. Sites are described generically.

Site Business Control share Planned window Minimum detectable effect
A Accounting software with a self-serve tier 50% July 16 - August 25 9%
B Staff-scheduling software, demo-led 20% July 17 - October 13 11%
C Rose's own website 10% July 15 - October 3 24%

Work performed

The allocation itself was the first defect. The control group was drawn with a second hash independent of the one deciding the share of traffic shown the agent; the two compounded, so a site configured at 50% traffic with a 10% control showed the agent to 45% of visitors rather than 50%. The control is now carved from the non-displayed remainder with a single hash partition, and the sample-ratio check's expected control share was corrected to match.

The mart refresh then failed silently twice. From July 1 the nightly refresh stopped completing: the recompute had grown to 25-45 minutes and hit its time limit, and from mid-month overlay views added to keep stopped experiments stable held a dependency on the tables being swapped, so every run rolled back. The marts stayed frozen until July 16. Separately, a filter on the event-delivery path had made a required anchor event conditional on membership of a staff experiment; when that experiment stopped on July 15, every other experiment stopped receiving anchors. Neither failure raised an error. The detector that caught the first compares the newest session in the marts with the clock, not the time of the last rebuild, because a rebuild that recomputes nothing still updates its own timestamp.

A third defect emptied the cohort. The eligibility rule excluded visitors with a prior staff-experiment assignment by aggregating a boolean over their assignments. On a site with no overlapping staff experiment every value was null, the aggregate was null rather than false, and the negated condition dropped every visitor: the marts for that experiment held zero rows.

A fourth defect was methodological, and is the month's most transferable finding. The cohort rule excluded visitors previously exposed to Rose. Before the start, the agent had been ramped up on part of the site's traffic, so prior exposure was a marker only treatment-assigned visitors could carry. Excluding on it removed returning treatment visitors and no equivalent control visitors. The read showed a false sample-ratio mismatch, with 63% of the cohort in control, and a +32% lift in call-to-action clicks that did not exist. The rule now excludes on a prior page view, an arm-independent superset, so the same visitors are removed from both arms.

The same error recurred twice more in the first interim reads. A comparison of "engaged" against "bounced" visitors across arms produced an apparently significant gap in completion, which was retracted: bounce is a session of at most one page view, and the agent replaces page navigation with conversation turns, so decisive visitors migrate into the "bounced" segment in the treatment arm only. Chatters against non-chatters is confounded the same way, because chatting is self-selected. The rule adopted is that arm comparisons condition only on pre-treatment covariates (entry page, referrer, campaign parameters, time); anything downstream of exposure is diagnostic only and labelled as such.

The interim reads also exposed conversion lag. In a journey-type classification outside the experiment (2,146 chatting visitors on one site, fourteen-day outcome window), visitors evaluating features converted at 10.3% within fourteen days; the same journey type read 0% in a day-four snapshot of the experiment. An early read counts the treatment arm's clicks before its conversions and penalizes treatment by construction.

The statistical reads were made consistent across surfaces. The call-to-action click rate, previously counts only, now carries an anytime-valid confidence sequence (a normal-mixture bound that stays valid under repeated looks), and the same pooled statistics serve the staff and client views, so the two cannot drift apart. A stopped experiment's results are now frozen at the stop; before, completed experiments were recomputed nightly and kept moving as late conversions arrived. The rebuilt refresh cut the nightly recompute from 25-45 minutes to about 18 seconds.

Metric definitions

Four metrics were found to measure something other than their name.

  • Demo timing. Demos were windowed by the start of the preceding conversation, so a demo booked days after a chat outside the window was dropped: on one client the thirty-day count read 195 against 223 in the event stream. Re-anchoring on the form-submission time brought it to 200.
  • Client call-to-action clicks. Clicks on the client's own links were counted for every visitor, engaged or not; they are now counted only in sessions with a started conversation on the client-facing surfaces. The experiment marts stay ungated, since gating there would zero the control arm.
  • Form navigation counted as intent. A click on a link pointing to the page already open, with no section anchor, is multi-step form navigation rather than a call to action. Excluding it removed about 60% of the click inflation seen on high-volume clients.
  • Session duration. A performance-sampling event, about 17% of event volume, extended apparent session length. The estimator now prefers the page-leave event as session end, falls back to a capped gap sum, and ignores the sampling event. At month end a cost-reduction change suppressed the page-leave event on blocked pages, collapsing the estimator to zero for those sessions (on one site, zero-duration sessions rose from about 35% to about 92%); the suppression was reverted for in-scope hosts the same week.

Measurement substrate

The experiments depend on a copy of the analytics event stream inside Rose's own database. In July that copy moved from a per-event webhook to a batch export, built in parallel before switching: the batch processor first wrote only to shadow tables compared with the live path, then became the writer with the old path kept as a backup. After cutover, four writes the old path made were found missing from the new one, including the timeline row that displays a booked demo and the timestamp that conversion-time windowing reads. All four were restored and backfilled.

The coverage monitor then raised false partial-export alerts on every run. The check waited 30 minutes after a window closed, but the export runs hourly, so coverage read as ninety minutes minus the cursor lag, out of sixty, which reproduced the observed 33% and 48%. Re-counted once settled, the same windows covered 100.2% to 100.4%. The monitor was rebuilt on two ratios, source to raw copy and raw copy to session table, with no minimum row count (the old floor would have let the check skip silently once the backup copy stopped filling). Both ratios are emitted on every evaluation, so a missing value raises its own alert. A live probe on a settled hour counted 6,225 events at the source, 6,225 in the raw copy and 6,224 in the session table; the remaining event had no resolvable session, which the processor drops by design.

Evaluation-harness integrity

The harness that gates prompt and skill changes had three faults of the same kind: it agreed when it should not have.

  • The retry rule re-sampled. When the gate failed, it re-ran the suite on a fresh random sample, which blocked merges on noise and could let a real regression pass on a luckier draw. Now only failed items are re-run, and an item that passes on retry passes the gate but is reported as flaky.
  • The agent's decisions were invisible. Importing the application reset the logging configuration and dropped every handler, so no record of which skill, intent or route was applied reached any log. Each run now reads a snapshot of those decisions from the conversation state and attaches it to failures.
  • Stale state was unreachable. Conversation history in an evaluation item is rebuilt from text, so a value computed by one turn was always empty on the next, and a bug where a step skips and a later step reads its stale output could never be reproduced. Items can now seed such values. The new item fails when a real stale-state defect fixed that month is reverted, and passed under the old harness.

Two smaller faults were fixed with them. Profiling answers were matched to fields by position, which recorded an answer about automation as a team size. Runs whose items raised errors still reported a passing mean; they now fail.

Two judged datasets were added. One tests whether the agent states that it is an AI when asked, as the EU AI Act requires: 28 items across 24 client configurations and six languages, including multi-turn cases and one negative case about a client's own AI product. The other judges whether the agent could actually answer each follow-up question it suggests, which the four existing judges could not see.

Results, proof, and next step

Measurements

Table 2 reports allocation by July 31, recomputed under the corrected cohort rule; each split agrees with its design. A count of raw assignment anchors for Site A on July 20 showed 51.5% in control, a residual left open for the final read.

Table 2. Visitors enrolled by July 31, 2026, by arm.

Site Control Treatment Control share Design z
A 21,071 20,992 50.09% 50% 0.39
B 11,186 45,264 19.82% 20% -1.09
C 66 651 9.21% 10% -0.71

Table 3 reports the evaluation measurements recorded at the time. The ten feature-suite failures left after the harness correction were kept as deliberate coverage for three known defects, two closed later in the month (Orchestration, below). The disclosure figures are before and after the disclosure rule. On a nine-item trap set, the groundedness judge scored the revised suggester at a mean of 0.86.

Table 3. Evaluation results recorded at the time of change.

Measure Before After
Feature suite, items passing, harness corrected, no product change 87/103 93/103
AI-disclosure dataset, mean judged score (28 items) 0.61 0.95
AI-disclosure dataset, deflections 11 0
Follow-up groundedness probe, ungrounded suggestion sets not measured 0/15

Interpretation

H1 held only after four corrections, each of which, uncorrected, would have produced a read that looked valid: the allocation hash compounded with the traffic gate, the null aggregate emptied the cohort silently, the frozen refresh left the dashboards plausible, and the treatment-marker filter produced a confident, wrong lift. The lesson is a property of the setting rather than a coding error: the agent changes the visitor's journey, so any variable it can influence becomes a collider when used as a filter or a segment. H2 was falsified on four metrics, each defined before the product changed what a journey looks like; bounce, corrected the same month, shows the mechanism most plainly, counting a visitor who settled everything in the chat and then converted as a bounce. H3 was falsified in its "only when wrong" direction: the harness both blocked correct changes on noise and passed incorrect ones.

Threats to validity

  • Interim reads. The day-three and day-four reads that triggered the investigation were not significant in either direction and were not used as verdicts.
  • Remaining validity items. The raw anchor split on Site A, a small leakage of agent-exposed sessions into its control arm, and a hygiene table not yet rebuilt after the fix were all open at month end.
  • Historical comparability. Every metric-definition correction above breaks the comparability of series across its deploy date.

Next step: read Site A at its planned end in August, and Sites B and C at theirs. Add a pre-registered, fixed-horizon attribution row per arm so the read is fair to delayed chat-driven conversions. Resolve the Site A anchor split before that read.


Adaptive Multi-Tenant Conversation Orchestration

Project and lock

The agent's commercial purpose is to turn a visitor's questions into a lead. Until July it answered each question in full and proposed a demo once an interest score, accumulating monotonically over the conversation, crossed a threshold. July opened with a counter-example from a low-volume, sales-led client that had left the platform: 438 conversations averaging 1.9 turns, 64% ending after one, a demo proposed in 4.6%; the client had observed booked demos recover after removing the agent. The hypothesis drawn was that a fully answered question is a terminal state: the agent satisfies exactly the curiosity that would otherwise have driven the visitor to contact sales.

The lock is which policy turns more visitors into leads, and when a demo proposal helps rather than ends the conversation. Sales heuristics such as qualifying before pitching do not say how to make a per-turn decision by a language-model agent across tenants with different sales motions, or how to tell readiness from conversation length.

The second lock is the contract layer. Business rules — offer gated content for an email on configured triggers, never quote a retrieved price outside the client's rules, never pitch a support conversation — are applied by selecting skills with a language model, so a rule that depends on a selection can silently fail to apply.

This month's work

Conversation policy

A second, conversion-oriented policy was built beside the existing one: it answers toward a next step, paces discovery questions before any proposal, and suppresses repeated pitches. The two became the information-first and conversion-first playbooks. A per-request override lets staff run either playbook on any client in preview without changing its configuration.

The demo-timing rule was then tested observationally over ninety days of production conversations, as recorded on July 21: Table 4 reports outcome by the depth of the first demo offer, and Table 5 compares, at each depth, turns with and without an offer.

Table 4. Outcome by turn of first demo offer, ninety days to July 21.

Turn of first offer Conversations Converted Continued, not converted Ended, not converted
1 2,623 25.3% 17.8% 56.9%
2 296 27.0% 24.7% 48.3%
3 413 23.7% 35.8% 40.4%
4 434 22.1% 34.8% 43.1%
5 or later 633 20.5% 42.0% 37.4%

Table 5. Share of turns followed by another visitor message.

Turn Demo offered Not offered Difference
1 31.9% 49.3% -17.4 pts
2 41.8% 57.8% -16.0 pts
3 47.6% 60.0% -12.4 pts
4 48.2% 63.0% -14.8 pts
5 or later 58.2% 73.9% -15.7 pts

Among conversations with an offer, early first offers did not convert less (25.3% and 27.0% at turns 1 and 2, against 20.5% to 23.7% later), and the study shows no offered-versus-unoffered conversion comparison, so the effect of an early offer on conversion is unresolved. An offer was followed by a continuation 12 to 17 points less often at every depth. The interpretation drawn was that the accumulating score conflated conversation length with buying readiness. The score was replaced by a per-turn readiness judge: a language-model call, run in parallel with the other steps, that requires two criteria together — a problem or goal the visitor states on their own initiative, and a request to move forward. Each verdict is recorded with its confidence so the judge can later be validated against what visitors did. On the conversion-first playbook the judge gates both the route to the booking flow and the demo button on the answer; on other clients it runs in shadow, recording verdicts without acting.

The judge failed three ways in its first week.

  • Lost verdicts. For two days every verdict was silently lost, because the storage it wrote to was not reachable from the service. Verdicts now go through a dedicated write path.
  • Stale verdicts. When the judge skipped a turn, it left the previous verdict in the conversation state; restored from the checkpoint, that verdict had lost its type, and reading it raised an error that bypassed the booking flow for every visitor who clicked the main demo button. A skip now clears the verdict.
  • Overlapping criteria. The rubric's first criterion listed examples ("see it", "start a trial", "get a quote") that were really instances of the second, so a single opening message such as "How would it look on my website?" satisfied both and the judge pitched on the first turn at 0.95-0.97 confidence. The examples were replaced by the principle that a request to move forward never counts as a volunteered need. A held-out behavioural test now fails against the old rubric; the earlier tests had mocked the model away, so reverting the rubric left them green.

Deterministic contracts

The email-for-content offer did not fire on turns clients had configured for it. The filed explanation was a priority conflict between two candidate endings; in fact the selector never returned the gated-content skill, so priority never applied. Three defects sat upstream of the priority table:

  • The skill description was enriched with exclusions that contradicted the client's configured content types.
  • It assumed every trigger is a content request, although "visitor mentions their industry" is a context signal.
  • The intent classifier routed "send me a case study by email" to a path that never applies ending skills.

Three earlier attempts had treated it as a priority problem and failed because they arbitrated over candidates that did not contain the skill. In the same layer, a closing skill had never been registered among the mutually exclusive endings, so it could reach the answer writer alongside the mandatory email request.

Guardrail selection showed a quieter version of the same dependency. A guardrail's description said when a message is on its topic, so on a follow-up reply inside a pricing thread the selector dropped the pricing guardrail about two times in five, and the answer quoted a figure retrieved from the client's public pricing page. A clause scoping each guardrail to the open thread, derived from the skill's own topic, is now rendered onto every guardrail's description. Two details were measured before fixing it: spliced into the scope sentence it worked better than appended after the closing exclusion, and a long form repeated on every guardrail grew the selection table by about a fifth and lowered selection below the baseline, so skills that declare nothing get a one-line form.

At month end the models that run intent classification and skill selection became unavailable on one client's onboarding test, through a billing error at their provider. Both failed open: the agent answered an off-topic request in full and exceeded the widget's time limit. The resulting design is asymmetric. Skill selection now fails closed, keeping the out-of-scope guardrail in its fallback, and an always-applied rule keeps the agent inside the client's domain without any guard-model call. Routing stays fail-open, because sending every turn to a refusal during an outage would deflect legitimate product questions.

Two contracts were added at the same layer. A new intent covers requests for the agent to do the visitor's work (draft a letter, write code, give personal legal or tax advice), with an answer-side fallback for classifier misses. A cap on how often the agent asks optional follow-up questions now holds across profiling while preserving the questions an answer requires.

Results, proof, and next step

Measurements

Table 6 reports the readiness judge's production verdicts from July 25, when verdicts were first recorded, to July 31.

Table 6. Readiness-judge verdicts in production, July 25-31, 2026.

Mode Site domains Conversations Verdicts Ready
Gating the demo 2 180 602 78 (13.0%)
Shadow, interest score still gating 30 2,337 5,019 601 (12.0%)

Table 7 reports the contract measurements recorded at the time of each change, on the evaluation datasets and live reproductions used to diagnose them.

Table 7. Contract measurements recorded at the time of change.

Contract Measure Before After
Gated-content offer Selector recall on the industry-mention turn, 5 runs 0/5 5/5
Gated-content offer Feature suite, items passing 104/106 106/106
Pricing guardrail in a thread Guardrail selected on the follow-up turn 7/12 12/12
Pricing guardrail in a thread Retrieved price quoted against the client's rules 2/9 0/12
Price asked with another answer Price addressed in the same turn, one client 4/6 16/16
Skill selection, all datasets Micro-F1 over 3 runs before and after 0.857 0.871
Intent classification with the new intent Macro-F1, 97 items not measured 0.970

Interpretation

The observational study supports the claim that an early demo offer costs continuation, with no conversion benefit measured; it does not establish the cause. A ready visitor may leave the chat precisely to book, which the conversion column controls only partly. The judge is therefore a design response to a correlation, and Table 6 measures adoption, not effect. Whether a "ready" verdict predicts conversion better than the score requires joining the recorded verdicts to outcomes, which had not been done by month end.

Each failed contract depended on a language-model selection that could silently omit it, and each fix moved the rule to where it applies by construction: an always-injected block, a deterministic exclusivity set derived from one registry, a fail-closed fallback. In the gated-content case, measuring selector recall directly, rather than reading end-to-end answers, located the defect in a component three earlier attempts had assumed was working.

Threats to validity

  • Observational design. Tables 4 and 5 come from production traffic across clients, with no randomization of offer timing.
  • Small evaluation sets. Table 7's sets are small, and several measures come from one client's configuration. The thread clause was measured on one client but renders for every client.
  • Short window. Table 6 covers seven days.

Next step: join the recorded verdicts to conversion outcomes to validate the judge. Compare the two playbooks under randomization rather than observationally: the per-request override already provides the switch, and the assignment apparatus built for the holdout experiments provides the method.


Compounding Context Engine for Company, Industry, and Buyer Intelligence

Project and lock

The context the agent consumes includes the conversation itself: earlier turns let the agent resolve a follow-up and avoid asking twice. That continuity rests on a session identity that must survive page reloads, re-entry from search results or email links, and, new in July, a conversation that starts inside a third-party AI assistant and continues on the client's site. Which re-entries should continue a conversation was not specified; the wrong answer splits one conversation into several, losing context for the agent and inflating the repeat-conversation metrics.

On intake, June's lock of keeping retrieval knowledge fresh, scoped and complete became a question of silent loss: can the platform tell when knowledge that should reach retrieval did not?

This month's work

The widget cleared the conversation and rotated the session identifier whenever a page was entered with a cross-domain referrer or none, so a visitor arriving from a search result, an email link or a typed address lost a live conversation mid-visit. Re-entry now keeps the transcript for up to 24 hours in the same tab, and stored state expires with the backend's conversation memory, so a restored transcript never outlives the context the agent holds for it. A companion mechanism continues a conversation started in a third-party AI assistant: a single-use, tenant-scoped identifier carries the earlier turns into the widget, and the site adopts it as the session identifier before analytics start, so chat, server memory and measurement see one identity.

Coverage questions are answered from lookup tables, structured lists of the industries, regions or cases a client covers. The agent often gave the right coverage verdict without naming the entry it came from, and an unnamed verdict cannot be told from a guess. The rule requiring the name lived in a client skill the selector might not pick; it now lives in the always-injected lookup block. A shortlist whose entries all share one verdict is answered directly rather than turned into a clarifying question, since the choice cannot change the answer.

On intake, two silent truncations were found. Reads of the document table were unpaginated, so a server-side row cap returned an arbitrary slice: one tenant read as having no documents, and another lost 14 of 1,014 eligible pages on every ingestion. A run for one tenant that ingested nothing also reported success. Reads now walk a cursor over an immutable key, and such runs fail. Migrated documents were classified as website pages from a source field that internal documents also carry; classification now depends only on the folder an operator places the document in. Scraping decisions are scoped to the real host, so two subdomains sharing a path no longer share a decision.

The company-level identity-resolution waterfall was revised: two third-party sources were withdrawn from it, one kept for person-level identity. The ground-truth measure built in late June, which compares a provider's answer with the visitor's own first-party email address, gained visible denominators and a matrix of agreement between providers.

Results, proof, and next step

Table 8 reports the re-entry pattern the continuity change targets, on production conversations for July 1-30, before the change shipped on July 31. A third of repeat conversations began within five minutes of the previous one ending, consistent with a fragmented single visit rather than a return.

Table 8. Repeat conversations by gap since the visitor's previous conversation, July 1-30, 2026, before the continuity change.

Measure Count Share of repeat conversations
Conversations 11,937
Visitors 10,427
Repeat conversations 1,510 100%
Started under 5 minutes after the previous one ended 497 32.9%
Started under 30 minutes after 775 51.3%

The change will lower the repeat-conversation rate mechanically; that fall removes an artefact and is not a regression. Any rate crossing July 31 must be read with the change stated, and a time-gap definition of a new conversation is the more stable basis.

The lookup change raised the lookup feature set from 14 to 17 of 17 items passing in a single run, as recorded at the time; the item it was filed for passed 3 of 3 runs, against 0 of 3 before. That single-run comparison hid an intermediate regression on one client's coverage dataset, whose mean moved from 0.88 to 0.80 and back to 0.86 over three to four runs per state.

At the person level the ground-truth measure returned a null: by July 31 only three visitors had both a person-level provider match and a first-party business email address, too few to estimate any provider's accuracy. Table 9 reads the larger company-level sample retrospectively, at the time this entry was written, over company identifications made before August 1 for visitors whose first-party email address is on a business domain; a provider agrees when the company domain it returned equals the email's domain or one is a subdomain of the other.

Table 9. Agreement of company-level providers with the visitor's own business email domain, identifications made before August 1, 2026. Providers are anonymized; B and D are the two sources withdrawn from the company-level cascade in July. Intervals are 95% Wilson intervals.

Provider Visitors compared Agree Agreement 95% interval
A 118 59 50.0% 41.1-58.9%
B (withdrawn) 45 6 13.3% 6.3-26.2%
C 13 7 53.8% 29.1-76.8%
D (withdrawn) 10 2 20.0% 5.7-51.0%
E 4 4 100% 51.0-100%

In total 188 distinct visitors carried at least one company identification to compare. The two withdrawn sources agree least, and source B's interval does not overlap source A's, which supports the withdrawal on accuracy. Three limits bound the read: exact domain matching counts a parent company, a brand alias or a regional domain as a disagreement, so every share is a lower bound on true agreement; the email addresses are read in their current state, not as they stood in July; and sources C to E are too small to rank.

Next step: read the repeat-conversation rate after the continuity change under a time-gap definition, and let the person-level ground-truth sample accumulate before revising the waterfall again.


Non-R&D / Productization Context

Most of July's releases were productization: self-serve signup with guided first-day setup and plan activation; a redesigned configuration studio; a playground running the real widget; team management for client administrators; a support hand-off to a third-party help-desk messenger; display targeting from the site's tag-manager data layer; a webhook for newly identified companies; per-provider in-chat booking; four more conversation languages; service notice banners; and analytics surfaces (funnel by traffic source, feature health, conversion by label, turns per visitor, a call-to-action column) whose metric-definition changes are recorded above. The knowledge-ingestion control plane, a durable orchestration of mapping, scraping and loading with explicit run identities, was built with established patterns for idempotent asynchronous work; its silent-loss findings are recorded above. A connector exposing a client's agent inside a third-party AI assistant was built on that vendor's application kit; the continuity mechanism it required is recorded above. Routine operations included client onboardings and skill fixes, a primary and fallback swap for the main answer model, regional data residency for model calls, warm instances against cold starts, offsite backups and daily configuration snapshots, a tenant deletion command, dependency updates, continuous-integration maintenance and documentation.


Prior-Month Results Review (June shipped work)

Two results on June's work matured in July: the read of the staff-wide holdout and a detect-and-repair cycle on the marts.

The staff-wide holdout. From June 10, 7% of visitors on fourteen sites were held out from the agent, with a planned minimum detectable effect of 10% at 95% confidence and 80% power; the experiment stopped on July 15. Table 10 reports its stored final read.

Table 10. Staff-wide holdout, June 10 to July 1, 2026, fourteen sites pooled.

Arm Visitors Conversions Conversion rate
Holdout (agent not shown) 12,016 151 1.257%
Agent shown 160,313 1,626 1.014%

The holdout share was 6.97% against 7% planned, and no holdout session recorded an exposure to the agent. Pooled, the agent-shown arm converted 19% less often in relative terms (two-proportion z = -2.54, p = 0.011). Because sites differ widely in traffic and conversion rate, Table 11 stratifies by site: the Mantel-Haenszel rate ratio is 0.82, with a 95% interval of 0.69 to 0.96 (Cochran-Mantel-Haenszel z = -2.41, p = 0.016). The direction is consistent but not uniform: treatment converted less often on 10 of the 13 sites with any conversion (exact sign test, p = 0.09).

Table 11. Conversion by site and arm, June staff-wide holdout. Sites are numbered by traffic; one site with fifteen visitors and no conversions is omitted.

Site Holdout Agent shown
1 57/5,219 (1.09%) 535/68,781 (0.78%)
2 38/1,356 (2.80%) 450/18,016 (2.50%)
3 9/1,258 (0.72%) 167/17,560 (0.95%)
4 3/1,284 (0.23%) 16/17,164 (0.09%)
5 10/1,118 (0.89%) 97/14,938 (0.65%)
6 2/530 (0.38%) 22/7,337 (0.30%)
7 9/330 (2.73%) 100/4,638 (2.16%)
8 0/286 (0.00%) 17/3,922 (0.43%)
9 16/266 (6.02%) 154/3,064 (5.03%)
10 2/127 (1.57%) 29/1,952 (1.49%)
11 4/136 (2.94%) 31/1,735 (1.79%)
12 1/62 (1.61%) 6/715 (0.84%)
13 0/42 (0.00%) 2/478 (0.42%)

The read must be qualified heavily. It covers June 10 to July 1, about three of the five planned weeks: from July 1 the mart refresh had been failing silently (Closed-Loop, above), and when the experiment stopped on July 15 the freeze-on-stop mechanism correctly preserved values that were by then two weeks old. It was computed under the cohort rule later corrected for its treatment-marker bias; its observed split matches the design, which argues against that distortion here, but it was not recomputed under the corrected rule. Its conversion metric predates July's metric-definition corrections. With those caveats, the read and its stratified analysis point the same way as the hypothesis that independently motivated the conversion-first playbook: an agent that fully answers may absorb demand that would otherwise have reached the client's forms. It is a negative result on the agent as configured in June and does not test the policy built in July.

The marts, detect and repair. The June marts failed twice without signalling it. A first freeze of about nine days was noticed only because a staff panel read zero visitors for the past week while the source data were healthy; the data-recency check described above was added on July 2. The second freeze ran from July 1 to July 16 and was resolved together with the filter that had starved every experiment of its anchor event. Silent failure, a detector designed around why the obvious signal had looked healthy, then repair: the closed loop the June work was meant to provide, applied to its own apparatus.

June's productized scraping produced no measurement in July. The provider ground-truth check built in June for enrichment-backed qualification is read retrospectively in Table 9.


Research Outcome

As of July 31, the month's deepest research was under Closed-Loop Agent Evaluation and Optimization. Running randomized holdouts on a conversational agent showed that several standard experimentation practices produce confident, wrong reads in this setting: filtering or segmenting on anything the agent can influence creates a collider, early reads penalize treatment through conversion lag, and metrics defined before the product, such as bounce, misdescribe the journeys it creates. Four apparatus defects, one of them a manufactured lift, were corrected before any verdict was read; the July experiments' allocations match their designs, and their verdicts await their planned end dates. The evaluation harness no longer agrees by default.

Under Adaptive Multi-Tenant Conversation Orchestration, an observational study associated early demo offers with less continuation and no measured conversion benefit, which led to replacing the accumulating interest score with a per-turn readiness judge, live on two site domains and in shadow on thirty, its predictive validity unmeasured. Business rules that depended on a language-model selection were moved to places where they apply by construction.

Under Compounding Context Engine for Company, Industry, and Buyer Intelligence, a third of July's repeat conversations began within five minutes of the previous one, the fragmentation the continuity change removes; its effect remains to be read.

The matured result of June's work is a stratified negative read for the agent as configured in June, a 0.82 conversion rate ratio across fourteen sites, qualified by a truncated window and an unrecomputed cohort rule, and consistent with the hypothesis behind July's conversion-first policy.