Skip to content

Skello widget holdout: does showing Rose raise conversion?

Date: 2026-11-18 · Ticket: IX-6308 · Data: PostHog project Rose app (80620), www.skello.io, 2025-10-20 → 2026-09-18.

Verdict

No measurable uplift. Over the whole randomized holdout (2025-10-20 → 2026-07-16, about 1.02 million visitors), visitors who could see Rose converted at 0.75 % against 0.79 % for those who never could: −5 %, 95 % CI −10 % to +1 %, p ≈ 0.07. The test cannot rule out a small gain, and its estimate leans negative.

The "about +20 %" quoted in the JEI rescrit dossier comes from December 2025 alone (+19 %, p ≈ 0.12). The dashboards of the time showed +48 % because their control group was biased. On the demo-request pages, Rose lowered form completion, and the widget was removed there on 2026-09-18.

Date Step Number
2025-10-20 80/20 holdout starts on skello.io —
Dec. 2025 First reading, visitors who arrived in December +19 % (p ≈ 0.12), the dossier's "≈ 20 %"
2025-12-30 Dashboard table, biased control group ×1.48
2026-07-16 End of the 80/20 holdout, replaced by the backoffice experiment —
2026-09-14 Backoffice experiment keen-pasteur-232 (IX-4924), interim reading at day 59 of 88 −11.5 % form conversion, significant
2026-09-18 Widget removed from the demo-request pages ~1,000 → ~30 displays a day
This page Corrected reading of the whole holdout −5 %, not significant

The test

The question: does Rose's mere presence on a site make more visitors request a demo? Comparing chatters with non-chatters cannot answer it, since visitors who start a conversation were already more interested. It needs a control group drawn at random.

  • Assignment: traffic control gives each visitor a fixed bucket from 0 to 99 (traffic control). Buckets 0–79 may see Rose; 80–99 never do. In the week of 2025-11-03 the threshold was 90. The holdout ran until 2026-07-16; the backoffice experiment keen-pasteur-232 took over the next day.
  • Metric: a Skello form submission (rw_client_form_submitted, nearly all on /landing/demo) within 14 days of the visitor's first counted visit.
  • Reading: intention to treat. A treatment visitor counts whether or not they talked to Rose.

The first reading and why it was wrong

December 2025, read alone, gave the figure later quoted in the dossier:

Arm Visitors Conversions Rate
Without Rose 16,259 98 0.60 %
With Rose 65,359 468 0.72 %

It had four flaws:

  1. One month, not significant. With 98 control conversions the gap is within noise (p ≈ 0.12).
  2. A window chosen after the fact. The other periods read −6 % (Oct.–Nov.), −4 % (Jan.–Mar.), −10 % (Mar.–May) and −7 % (May–July). December is the only favourable one.
  3. A biased control group in the dashboards. Traffic control runs before the page display rules. Control therefore kept visitors of blog posts where Rose is never shown, while the treatment arm only counts visitors Rose was actually allowed to show to. Blog readers convert less, so control looked worse. The uplift table of 2025-12-30 read ×1.48, and December alone would read +86 %. The with Rose and without Rose funnels have the same flaw.
  4. An ignored warning. The arms split 73/27 instead of 80/20. A sample ratio mismatch like this flags a broken comparison before any lift is read.

Corrected method

  1. Whole holdout: 2025-10-20 → 2026-07-16, from its first day to the day before the backoffice experiment replaced it.
  2. Same pages for both arms: a visitor counts only on pages where Rose may display. A (path, device) pair is excluded when blocked decisions (Display rules: URL or device blocked, renamed URL not allowed by site configuration on 2026-03-24) outnumber displays. In practice, most blog posts.
  3. Split check before the result: the corrected arms split 80.3/19.7 (818,988 / 201,119), against 72.7/27.3 uncorrected. The small excess over 80 % is consistent with the 90 % week, which sends buckets 80–89 to treatment. The treatment arm barely moves (818,985 uncorrected): the correction drops 106,064 control visitors seen only on pages without the widget.
  4. Ambiguous visitors out: visitors seen in both arms (the 90 % week), visitors in the staff holdout that overlapped from 2026-06-08, and traffic outside www.skello.io.

Result

First visit Without Rose: visitors / conversions / rate With Rose: visitors / conversions / rate Relative gap p
2025-10-20 → 2025-11-30 25,668 / 212 / 0.83 % 129,724 / 1,002 / 0.77 % −6 % ≈ 0.37
Dec. 2025 16,259 / 98 / 0.60 % 65,359 / 468 / 0.72 % +19 % ≈ 0.12
2026-01-01 → 2026-03-23 54,582 / 421 / 0.77 % 216,551 / 1,597 / 0.74 % −4 % ≈ 0.41
2026-03-24 → 2026-05-17 40,090 / 419 / 1.05 % 160,659 / 1,515 / 0.94 % −10 % ≈ 0.06
2026-05-18 → 2026-07-16 64,520 / 437 / 0.68 % 246,695 / 1,562 / 0.63 % −7 % ≈ 0.21
Whole holdout 201,119 / 1,587 / 0.79 % 818,988 / 6,144 / 0.75 % −5 % (CI −10 % to +1 %) ≈ 0.07

Uncorrected, the same whole period reads 0.52 % against 0.75 % (+45 %).

Four periods of five lean negative. This test found no uplift from displaying Rose; a small effect in either direction is not ruled out. The later backoffice experiment keen-pasteur-232, with its own arms over a later period, read −11.5 % on form conversion and −9.7 % counting emails at its interim reading of 2026-09-14 (IX-4924 report).

Limits of the corrected method

  • Approximate page classification: "without widget" is inferred from observed display decisions, not from the configuration at each date.
  • Diluted effect: most treatment visitors never talk to Rose, so an effect on chatters can vanish in the average.
  • Limited power: 1,587 control conversions cannot separate an effect of a few percent from zero.
  • Partial conversion: only forms on www.skello.io count; free trials, bookings elsewhere and signed deals do not.
  • One client, with the Rose version of the time.
  • Internal traffic is not filtered by IP; negligible at this volume.

Demo-request pages

On /landing/demo and its IT, EN, BE and CH versions, the form is filled on the page itself, so the metric here is a form sent within a day of the first visit. Control pools the randomized control cohorts in turn: the 80/20 holdout until 2026-07-16, the staff holdout from 2026-06-08, then the keen-pasteur-232 control. The cohort stops on 2026-09-17, the day before the widget was removed.

Period Without Rose: visitors / forms / rate With Rose: visitors / forms / rate Relative gap p
Oct. 2025 → Jan. 2026 4,556 / 379 / 8.32 % 19,399 / 1,758 / 9.06 % +9 % ≈ 0.11
Feb. → Aug. 2026 25,725 / 1,730 / 6.73 % 92,600 / 5,707 / 6.16 % −8 % ≈ 0.001
2026-09-01 → 2026-09-17 3,233 / 174 / 5.38 % 12,568 / 710 / 5.65 % +5 % n.s.
Oct. 2025 → 2026-09-17 33,514 / 2,283 / 6.81 % 124,567 / 8,175 / 6.56 % −4 % ≈ 0.10

From February to August 2026 Rose sat below control every month. That split was read from monthly data, not fixed in advance; over the whole span the gap is not significant. With the keen-pasteur-232 interim result, it led to removing the widget from these pages on 2026-09-18. The IX-4924 report traces the mechanism to deflection: Rose answers the question in chat, and fewer visitors fill the form.

What this changes

  • Do not cite a Skello sales lift from the holdout. The defensible statement is no measurable uplift from a randomized test, plus a significant negative interim reading on form conversion in the later experiment.
  • Check the arm split before reading any lift. The 73/27 split was visible from the first week.
  • Compare arms on the same pages. Traffic control is decided before display rules, so the control arm always includes pages the treatment arm cannot reach.

Reproduction

Run with the PostHog MCP execute-sql tool (HogQL), project 80620.

-- Corrected holdout reading, by period of first visit
WITH keys AS (
  SELECT concat(path(properties.$current_url), '|', toString(properties.$device_type)) AS k,
    countIf(properties.rw_display_reason ILIKE 'Display rules%'
      OR properties.rw_display_reason = 'URL not allowed by site configuration') AS b,
    countIf(properties.rw_display_allowed = true) AS a
  FROM events
  WHERE event = 'rw_posthog_initialized' AND properties.$host = 'www.skello.io'
    AND timestamp >= '2025-10-20' AND timestamp < '2026-07-17'
  GROUP BY k),
blocked AS (SELECT k FROM keys WHERE b >= a),
exp AS (
  SELECT person_id, min(timestamp) AS first_seen,
    max(properties.rw_display_allowed = true) AS saw_rose,
    max(properties.rw_display_reason ILIKE 'Traffic control: User in bucket%') AS in_ctrl
  FROM events
  WHERE event = 'rw_posthog_initialized' AND properties.$host = 'www.skello.io'
    AND (properties.rw_display_allowed = true
      OR properties.rw_display_reason ILIKE 'Traffic control: User in bucket%')
    AND concat(path(properties.$current_url), '|', toString(properties.$device_type)) NOT IN (SELECT k FROM blocked)
    AND timestamp >= '2025-10-20' AND timestamp < '2026-07-17'
  GROUP BY person_id
  HAVING saw_rose + in_ctrl = 1),
forms AS (
  SELECT person_id, timestamp AS ts
  FROM events
  WHERE event = 'rw_client_form_submitted' AND properties.$host = 'www.skello.io'
    AND timestamp >= '2025-10-20' AND timestamp < '2026-08-01'),
per AS (
  -- a visitor converts if any form falls in their 14-day window
  SELECT exp.person_id AS pid, any(exp.in_ctrl) AS in_ctrl, any(exp.first_seen) AS first_seen,
    max(forms.ts >= exp.first_seen AND forms.ts <= exp.first_seen + INTERVAL 14 DAY) AS converted
  FROM exp LEFT JOIN forms ON forms.person_id = exp.person_id
  GROUP BY exp.person_id)
SELECT if(in_ctrl = 1, 'control', 'rose') AS arm,
  multiIf(first_seen < '2025-12-01', 'p0 oct-nov', first_seen < '2026-01-01', 'p1 dec',
    first_seen < '2026-03-24', 'p2 jan-23mar', first_seen < '2026-05-18', 'p3 24mar-17may',
    'p4 18may-16jul') AS period,
  count() AS visitors, sum(converted) AS converters
FROM per
GROUP BY arm, period ORDER BY period, arm;

-- Uncorrected reading (the biased one): drop the `NOT IN (SELECT k FROM blocked)` filter.

-- Demo-request pages, up to the removal
WITH exp AS (
  SELECT person_id, min(timestamp) AS first_seen,
    max(properties.rw_display_allowed = true) AS saw_rose,
    max(properties.rw_display_reason ILIKE 'Traffic control: User in bucket%'
      OR properties.rw_display_reason IN ('Staff holdout control cohort', 'Client experiment control cohort')) AS in_ctrl
  FROM events
  WHERE event = 'rw_posthog_initialized' AND properties.$host = 'www.skello.io'
    AND match(path(properties.$current_url), '/landing/demo')
    AND timestamp >= '2025-10-20' AND timestamp < '2026-09-18'
  GROUP BY person_id
  HAVING saw_rose + in_ctrl = 1),
forms AS (
  SELECT person_id, timestamp AS ts
  FROM events
  WHERE event = 'rw_client_form_submitted' AND properties.$host = 'www.skello.io'
    AND timestamp >= '2025-10-20' AND timestamp < '2026-09-20'),
per AS (
  SELECT exp.person_id AS pid, any(exp.in_ctrl) AS in_ctrl, any(exp.first_seen) AS first_seen,
    max(forms.ts >= exp.first_seen AND forms.ts <= exp.first_seen + INTERVAL 1 DAY) AS converted
  FROM exp LEFT JOIN forms ON forms.person_id = exp.person_id
  GROUP BY exp.person_id)
SELECT toStartOfMonth(first_seen) AS month, if(in_ctrl = 1, 'control', 'rose') AS arm,
  count() AS visitors, sum(converted) AS forms_1d
FROM per
GROUP BY month, arm ORDER BY month, arm;

p-values are two-proportion z-tests on the counts above; the confidence interval on the whole-holdout gap uses the unpooled standard error.