Holdout A/B tests to revenue: why Skello and Pennylane read flat or negative¶
Date: 2026-11-18 · Ticket: IX-6309 ·
Clients: skello.io (keen-pasteur-232), pennylane.com (rapid-galileo-288), production
Question: the two client holdout tests measure qualified form submissions only and read flat or negative. Is the measurement fair to Rose, where does the loss come from, and can the readout go down to pipeline and revenue?
Sources: Supabase "Rose PROD (EU)" (drtzxyuvppalvgczwhne), read-only:
analytics.experiment_sessions_read, session_events, visitor_page_views, conversations,
messages, experiments.behavior_enrollments. Vault notes Clients/Skello.md and
Clients/Pennylane.md for configuration dates. Earlier analyses:
IX-4924
(2026-09-14),
IX-4055
(2026-07-20). The earlier Skello 80/20 holdout (2025-10-20 to 2026-07-16) is read in
Skello widget holdout (IX-6308).
Status: internal. Share with Benoit before any client use: the published Skello case study claims +30% demo bookings, which these tests do not support.
TL;DR¶
- The metric is fair to Rose on bookings.
convertedcounts qualifiedclient_form_submittedonly, but in-chat demo bookings also fire that event: 357 of 358 Pennylane treatment bookers and 216 of 216 Skello bookers are already counted. - Skello's −8% most likely comes from a setup error, not from Rose cutting leads everywhere. The loss is concentrated in the period when Rose showed on the demo-form page: −10.6% before 2026-09-18, +2.4% after Rose was hidden there. The later estimate is not significant and still allows a loss; the cohort and the playbook also changed that week.
- Pennylane's −2% sits on one page group. Visitors landing on the e-invoicing pages convert
−20% with Rose; everywhere else +1.8%. The signup copilot (Form Assistant) on
start.pennylane.comreads positive (+11%, borderline). - Common lesson: Rose hurts on pages where the visitor is already on their way to a form, when it interrupts (pop-up questions) or adds a step (a qualifying question instead of the signup button).
- Revenue-level A/B is out of reach at these volumes, and clients do not share final conversion. The proposed readout is value-weighted leads per visitor plus support time saved.
1. What converted counts¶
analytics.experiment_sessions sets converted from client_form_submitted passed through the
client's conversion filters (form_id_wl, form_action_wl, url_inc, url_exc); see
supabase/migrations/20260928180000_baseline.sql (session_event_rollup). demo_booked and
email_captured are not in it.
In practice, bookings made in the chat also submit the client's form:
| Experiment | Treatment visitors who booked in chat | Not counted as converted |
|---|---|---|
Pennylane rapid-galileo-288 |
358 | 1 |
Skello keen-pasteur-232 |
216 | 0 |
So the treatment group is not penalized on demo bookings. Email-only captures stay outside the metric (IX-4924 measured them: form OR email moves Skello only from −11.5% to −9.7%).
Contamination: 4 Pennylane control visitors booked a demo through the chat and 45 control visitors saw the signup copilot (0.3%). Small, but the holdout is not perfectly sealed.
2. Skello: the loss follows the widget on the demo form¶
Effect before and after 2026-09-18¶
Person-level, split by the visitor's first-seen date (pre-treatment):
| Visitors first seen | Control | Treatment | Effect (95% CI) |
|---|---|---|---|
| Before 2026-09-18 | 1.66% (972 / 58,573) | 1.48% (3,490 / 235,197) | −10.6% (−16.7% to −4.0%) |
| From 2026-09-18 | 1.52% (270 / 17,733) | 1.56% (1,105 / 70,905) | +2.4% (−10.3% to +16.8%) |
| Whole test | 1.63% (1,242 / 76,306) | 1.50% (4,595 / 306,102) | −7.8% (−13.3% to −1.9%) |
Weekly, the gap is widest in the weeks of 08-17 (−35%) and 08-24 (−21%) and closes from the end of August onwards.
The two periods are not the same population. The mart only enrolls visitors on a session that
reaches a URL where Rose may show (url_allowed, applied to both arms). From 2026-09-18,
visitors who only saw the demo page are no longer enrolled, so the later cohort holds fewer of
the highest-converting visitors. The after-period estimate is consistent with no loss, but its
interval (down to −10.3%) does not rule one out.
Cause: Rose was displayed on skello.io/landing/demo¶
/landing/demo is the page with the most qualifying form submissions (4,197 between 07-17 and
10-13). Widget impressions on that page:
| Week of | Impressions on the demo page | Share of all Skello impressions |
|---|---|---|
| 07-13 to 09-07 | 5,700–9,300 per week | 10–12% |
| 09-14 | 4,669 | 6% |
| 09-21 onwards | 960 → 360 | about 1% |
Clients/Skello.md, 2026-09-18: Skello found the Rose bar on its demo form and asked to exclude
it again; it had been set to minimised instead of excluded since its removal in October 2025.
Rose hid it through the display rules.
By landing page (pre-treatment), before 2026-09-18 (recounted 2026-11-18; totals differ from the table above by a few visitors after a mart refresh):
| First page | Control | Treatment | Effect (95% CI) |
|---|---|---|---|
| Demo page | 6.50% (117 / 1,800) | 4.97% (367 / 7,378) | −24% (−37% to −6%) |
| Other pages | 1.88% (679 / 36,031) | 1.65% (2,380 / 144,095) | −12% (−20% to −5%) |
| Home | 1.01% (175 / 17,278) | 1.06% (734 / 69,082) | +5% (−11% to +24%) |
| No page view recorded | 0.03% (1 / 3,461) | 0.06% (9 / 14,640) | too few conversions |
From 2026-09-18, "other pages" are flat (2.25% vs 2.29%); demo-page landers are too few to read (99 control). Visitors landing elsewhere reach the demo page later in their visit, so the loss on "other pages" is consistent with the same display problem.
Confounder: the Skello playbook A/B test started on 2026-09-15¶
skello-playbook-v1 (behavior experiment, 2026-09-15 to 2026-11-10) split chatters between an
information and a conversion playbook. Forms within 14 days of enrolment:
| Playbook | Chatters | Forms | Rate |
|---|---|---|---|
| Conversion | 1,019 | 59 | 5.79% |
| Information | 969 | 38 | 3.92% |
+48%, z ≈ 1.95 (borderline; IX-6307 covers this result). That is about 19 extra forms over the whole period (95% CI roughly 0 to 38), against roughly 150 conversions recovered after 2026-09-18. It compares two Rose playbooks, not the playbook change against the setup before it, so it only sizes the playbook's contribution roughly. Even at its upper bound it explains a minority of the recovery; the demo-page hide is the most likely main cause. Not proven: both changes landed the same week.
Reconciling with IX-4924¶
IX-4924 (2026-09-14, before the hide) attributed the −11.5% to deflection: Rose answers in chat and rarely proposes the next step. It ruled out "UI blocking" by comparing conversations where Rose pushed the free trial, not by page. The time split above shows the loss is no longer measurable once the widget leaves the demo form, which IX-4924 could not see. Deflection may still exist, but it does not produce a measurable loss once the demo page is clean.
3. Pennylane: the loss sits on the e-invoicing pages¶
Test window: visitors first seen 2026-07-16 to 2026-09-19. The widget was effectively off in the weeks of 08-17 and 08-31 (and partly 08-10 and 08-24) while visitors were still assigned to treatment, so every Pennylane effect below is diluted.
Both tables below were recounted on 2026-11-18 from one query (variant from the visitor's first
session, a visitor "reaches" the signup site when any of their sessions has a page view on
start.pennylane.com).
Funnel split¶
| Step | Control | Treatment | Effect (95% CI) |
|---|---|---|---|
Reach start.pennylane.com |
29.8% (13,692 / 45,987) | 28.6% (12,951 / 45,336) | −4.1% (−6.0% to −2.1%) |
| Convert, once there | 21.0% (2,874 / 13,692) | 21.4% (2,771 / 12,951) | +1.9% (−2.7% to +6.8%) |
| Overall | 6.25% (2,874 / 45,987) | 6.11% (2,772 / 45,336) | −2.2% (−7.0% to +2.9%) |
Every conversion but one goes through the signup site, so the two steps multiply to the overall rate. "Once there" conditions on a step Rose can influence (IX-4055, L1), so it is diagnostic only. In a first count, about 80% of treatment visitors who reached the signup site saw the copilot.
By landing page (pre-treatment)¶
| First page | Control | Treatment | Effect (95% CI) |
|---|---|---|---|
| E-invoicing pages | 10.76% (516 / 4,794) | 8.61% (404 / 4,694) | −20% (−29% to −10%) |
| All other pages | 5.72% (2,358 / 41,193) | 5.83% (2,368 / 40,642) | +1.8% (−4% to +8%) |
| …home | 5.08% (1,282 / 25,261) | 5.07% (1,265 / 24,937) | 0% (−7% to +8%) |
| …pricing | 7.60% (137 / 1,803) | 7.81% (146 / 1,869) | +3% (−18% to +29%) |
| …landing directly on signup (copilot) | 20.6% (667 / 3,233) | 22.9% (725 / 3,171) | +11% (+1% to +22%) |
E-invoicing pages: logiciel-facturation-electronique (17.7% → 13.5%),
facturation-electronique-gratuite (33.4% → 28.0%), lp-facturation-electronique-* (paid
landing pages). Page titles promise "Facture électronique gratuite" and "Logiciel Facturation
Électronique à 0€". The loss there (about 100 signups) is larger than Pennylane's whole net gap.
What Rose does on those pages¶
Over the test, 665 conversations started on e-invoicing pages:
- 60% (398) started from a question Rose popped up (
first_message_source = dynamic_question). - 1.7 messages on average; 10% end with a CTA click.
- Top visitor questions: "Suis-je concerné par la réforme ?" (80), "Est-ce que je peux tester Pennylane gratuitement ?" (58), "Êtes-vous compatible avec la réforme ?" (45), "Je suis micro-entrepreneur, y a-t-il une offre gratuite ?" (25).
- Answers during the test were correct in substance but narrowed the page's promise ("essai gratuit de 15 jours", "offre gratuite réservée aux micro-entreprises non assujetties à la TVA") and ended with a qualifying question ("Quel est votre profil d'entreprise ?") instead of the signup button. The free offer still existed during the test (July–September 2026), so this diagnosis holds for that period.
- Treatment visitors on those pages who saw Rose without chatting converted 8.85% against 10.79% for control. Self-selected group, suggestive only.
Current answers (backoffice test mode, 2026-11-18, after the October knowledge cleanup):
- "Je suis micro-entrepreneur, y a-t-il une offre gratuite ?" → "Non, il n'y a pas d'offre
gratuite. Les offres démarrent avec Starter à 7 € HT/mois…". Settled: this answer is correct
now. Pennylane retired its Free plan on 2026-11-04 and Rose removed it from Pennylane's skills
that day (#2917, IX-5966). The free
e-invoicing pages are gone too:
/fr/facturation-electronique-gratuitewas last viewed on 2026-11-04, and the paid/fr/lp-facturation-electronique-sml-*pages return 404 since 2026-11-06. - "Suis-je concerné par la réforme ?" → correct, but ends with "En savoir plus" linking to the page the visitor is already on.
Rose has been live on 10% of Pennylane traffic since 2026-10-21 (Clients/Pennylane.md).
4. Revenue-level readout: what is possible¶
- A revenue A/B test is underpowered. Skello's −8% breaks even only if treatment leads close about 9% better. At an assumed 15% lead-to-deal rate (assumption, not data), detecting that needs about 11,000 leads per group; the Skello control group produced 1,242.
- Clients do not share final conversion. Skello's HubSpot receives the Rose session ID for leads, but no closed-won outcome comes back to Rose, and control visitors' leads carry no person-level ID.
- Our own data cannot compare lead quality between groups. Form events carry no field values
(only
form_id,form_name,page_url,detection_method), and company identification by IP runs only where the widget loads (102 Skello treatment visitors identified, 0 in control).
Proposed readout instead:
- Value-weighted leads per visitor, both groups. The form tracker reads one or two qualifying fields (company size, segment) on each client's lead form; each client gives one table of average deal value per segment. Readout: Σ(form submissions × segment value) per visitor. Needs a consent and data-protection decision, since Rose would store form content.
- Support time saved, outside the A/B test: support conversations answered × minutes per ticket from the client. Feeds the ROI calculator.
- Keep the holdout test as a no-harm guardrail on qualified lead volume, not as proof of revenue.
5. Recommendations¶
- No standard widget on conversion form pages by default, with a per-client override. Form Assistant deployments stay (Pennylane signup, PayFit signup).
- Pennylane e-invoicing pages: Form Assistant layout (small button, no pop-up questions), a signup CTA limited to those pages, and wording that matches the pages. Change to the client's live setup: needs Benoit's and Pennylane's approval. Test it against no Rose rather than assume.
- Rerun both tests on the fixed setup. The Skello post-fix period has too few visitors to claim a gain.
- Flag display-rule changes in experiment readouts. The Skello result mixes two configurations without saying so.
- Correct or qualify the Skello case study (+30% demo bookings) before a client or prospect sets it against these tests.
- Page playbooks (page pattern → goal, how answers end, which button) only if existing settings cannot stop answers ending on a qualifying question.
Open items¶
- Skello: the demo-page hide and the playbook test overlap by three days; not fully separable.
- Pennylane: device and traffic source cannot be compared (only recorded when the widget loads).
- Pennylane: no proactive-question test has run (
teaser_test_resultsis empty), so the pop-up's effect alone is not measured. - Pennylane: the paid e-invoicing landing pages return 404 but still got about 28 views from old links up to 2026-11-18.
Reproduce¶
Skello: person-level effect by first-seen period and landing page:
with f as (
select distinct on (person_id) variant, person_id, session_id, experiment_first_seen_at fs
from analytics.experiment_sessions_read
where experiment_id = 'keen-pasteur-232'
order by person_id, session_started_at),
c as (
select person_id, bool_or(converted) c
from analytics.experiment_sessions_read
where experiment_id = 'keen-pasteur-232' group by 1)
select case when f.fs < '2026-09-18' then 'before' else 'after' end per,
case when lp.u is null then 'no page view'
when lp.u like '%/landing/demo%' then 'demo page'
when lp.u ~ '^https?://(www\.)?skello\.io/?([?#].*)?$' then 'home'
else 'other' end landing,
f.variant, count(*) n, sum(c.c::int) k, round(100.0 * avg(c.c::int), 2) cr
from f join c using (person_id)
left join lateral (
select page_url u from public.visitor_page_views v
where v.session_id = f.session_id order by viewed_at limit 1) lp on true
group by 1, 2, 3 order by 1, 2, 3;
Pennylane: landing page and funnel in one pass (sum the landing rows for the funnel table):
with s as (
select person_id, variant, session_id, session_started_at, converted,
experiment_first_seen_at fs
from analytics.experiment_sessions_read
where experiment_id = 'rapid-galileo-288'),
f as (
select distinct on (person_id) person_id, variant, session_id, fs
from s order by person_id, session_started_at),
c as (select person_id, bool_or(converted) c from s group by 1),
r as (
select distinct s.person_id from s
join public.visitor_page_views v on v.session_id = s.session_id
where v.page_url like '%start.pennylane.com%'),
lp as (
select f.person_id,
(select page_url from public.visitor_page_views v
where v.session_id = f.session_id order by viewed_at limit 1) u
from f)
select f.variant,
case when lp.u is null then 'no page view'
when lp.u like '%start.pennylane.com%' then 'signup'
when lp.u like '%facturation-electronique%' then 'e-invoicing'
when lp.u ~ '^https?://(www\.)?pennylane\.com/(fr/?)?([?#].*)?$' then 'home'
when lp.u like '%/tarifs%' or lp.u like '%/pricing%' then 'pricing'
else 'other' end landing,
count(*) n, sum(c.c::int) k, count(r.person_id) reached,
sum((c.c and r.person_id is not null)::int) k_reached
from f join c using (person_id) join lp using (person_id)
left join r using (person_id)
where f.fs >= '2026-07-16' and f.fs < '2026-09-20'
group by 1, 2 order by 1, 2;
Widget impressions on a page, by week (filter on site_domain and event_name so the
idx_session_events_widget_impression_site index is used; unfiltered scans of session_events
time out):
select date_trunc('week', event_at) w, count(*) n,
count(*) filter (where properties->>'page_url' like '%/landing/demo%') demo_page
from public.session_events
where site_domain in ('www.skello.io', 'skello.io') and event_name = 'widget_impression'
and event_at between '2026-07-13' and '2026-10-13'
group by 1 order by 1;
Confidence intervals are log relative risk: exp(ln(RR) ± 1.96·sqrt(1/kt − 1/nt + 1/kc − 1/nc)).