Feature eval test sites¶

The feature-* datasets run against two synthetic sites, testfeatures.com and testfeatures2.com. Their config is not arbitrary: each value exists so that some dataset can assert a behavior deterministically. Changing one silently re-scopes whatever asserts against it, so this page records the values, why they are what they are, and what breaks if they move.

Supabase is the source of truth, not the migrations

supabase/migrations/20260325140833_testfeatures_enable_all_features.sql sets engagement.suggested_answers.frequency = "high" with the comment "so the eval can reliably trigger it". That is stale — the live resolved value is moderate, so a later config write superseded it. Always read the live value with rose-config get engagement -s testfeatures.com before reasoning about behavior; a migration only tells you what was true the day it ran.

Live values (verified 2026-07-27)¶

Site	Key	Value	Why
`testfeatures.com`	`engagement.suggested_answers.frequency`	`moderate`	→ `(window=4, max=2)`. Keeps `_rule_force_context_gathering_very_high` off: that rule only fires for `high`/`very_high`, and when it fires it injects `context_gathering` on every turn, which makes any "must not ask 👉" assertion unsatisfiable.
`testfeatures.com`	`qualification.profiling.forms.qualification.cta_blocking_fields`	`["team_size", "use_case"]`	Drives the hard gate: the agent must collect these before offering the 👇 CTA. `feature-qualification` asserts that interplay.
`testfeatures.com`	`qualification.profiling.field_definitions`	`industry`, `use_case`, `team_size`, `main_challenge`	The two non-blocking fields (`industry`, `main_challenge`) are what make `has_pending_profile_field` true, which drives soft elicitation — see below.
`testfeatures2.com`	`qualification.profiling.forms`	`demo_booking` (`in_chat_booking`) + `demo-qualification` (`qualification`, no `cta_blocking_fields`)	`feature-inchat-booking` needs the booking form; the qualification form has no blocking fields, so it exercises soft elicitation with no hard gate.
`testfeatures2.com`	`engagement.suggested_answers.frequency`	`high`	→ `(window=4, max=3)`, so `_rule_force_context_gathering_very_high` is active here. Deliberate: this site is where forced context gathering is exercised, while `testfeatures.com` stays at `moderate`.

The two ways a 👉 gets forced¶

Both are code-level enforcement ("LLMs don't reliably follow prompt guidance"), and they are independent. A dataset item that asserts "no 👉" must silence both:

_rule_force_context_gathering_very_high (nodes/skill_applier_rules.py) — fires only for frequency high/very_high. Silenced by keeping sites at moderate.
_rule_force_context_gathering_when_profiling_pending (same file) — not frequency-gated. Fires whenever a non-blocking field is still uncollected. Silenced per item by injecting collected profiling state.

Negative items must inject profiling state¶

Any item asserting that a 👉 is absent has to declare the visitor's fields as already collected, otherwise soft elicitation legitimately asks for one and the item fails every run:

_item(
    query="Does TestFeatures support SAML SSO?",
    expected={"marker_present": False},
    inject_profiling_state={
        "collected_values": {
            "industry": "Software",
            "main_challenge": "Manual workflows",
            "team_size": "11-50",
            "use_case": "Automation",
        }
    },
)

Measured A/B on that exact query (testfeatures.com, no history):

	decision snapshot	👉 rendered
without injection	`skills=[content_recommendation, knowledge_retrieval, profiling_assistant] ctx_gathering=allowed(0/2) soft_elicit`	yes
with injection	same skills, `soft_elicit` absent	no

This is also the stronger contract: "a visitor we already know asks a specific factual question → don't ask another follow-up". Against a brand-new visitor with pending fields, asking one qualifying question is intended behavior, so an item that asserts silence there is asserting against the product.

Known defect: soft elicitation is an ungoverned question path (IX-4154)¶

Two failing items, one cause. profiling_assistant can ask a qualifying question independently of the context_gathering skill, and it obeys neither the cap nor the marker convention:

Item	Symptom
`feature-context-gathering` "I also need reporting capabilities"	window full (`forbidden(3/2)`), `context_gathering` correctly dropped — 👉 rendered anyway, so the configured cap does not hold
`feature-qualification` "Which plan should I pick?"	the question is asked ("How many people on your team would use it?") but with no 👉, because `content_gating` won ending exclusivity and dropped `context_gathering` where the marker instruction lives

The second one compounds: context_gathering_count is derived by scraping 👉 out of the response, so an unmarked question never increments the counter. The window under-counts, and the visitor gets asked more often than configured. Unmarked questions also skip the suggested-answer chips, since suggestion_router routes on 👉.

Both items are correct and are left red on purpose, with comments in their seed files telling future readers not to relax or relocate them.

Original note: soft elicitation ignores the frequency cap¶

The saturation item in feature-context-gathering supplies three prior 👉 turns, so the window is full. The snapshot confirms ctx_gathering=forbidden(3/2) and context_gathering is correctly dropped from the applied skills — but soft_elicit stays active and a 👉 is still rendered via profiling_assistant. A client who configures a low follow-up frequency therefore still gets questions past the cap. Tracked separately; the dataset item is correct and should stay failing until the product is fixed.

Reading the decisions¶

rose-eval features run <dataset> --item "<substring>" prints a decision snapshot for every failing item — applied skills, context-gathering availability and window counts, whether the hard gate or soft elicitation is active, intent, route, lookup verdict and collected values — plus a Langfuse trace link. Runtime logs are not usable for this: importing the API app runs setup_logging(), which drops every handler, so --debug shows nothing from ixchat. Read the snapshot, not the log.

Changing these values¶

Use rose-config, not a migration (migrations are for schema, config data lives in Supabase — see backend/AGENTS.md):

rose-config get engagement -s testfeatures.com
rose-config set engagement.suggested_answers.frequency moderate -s testfeatures.com

Before changing anything, check which datasets assert against the knob you are moving: testfeatures.com backs five datasets, and 12 items assert "no 👉" while 6 require one.