Demo-offer timing vs conversation drop-off

Date: 2026-07-21 · Ticket: IX-4063 · Data: PostHog project Rose app (80620), production, trailing 90 days, bots excluded.

Question

Does the Website Agent push the demo CTA (👇) too soon? Concretely: when the agent offers a demo, does the visitor stop engaging more often than when it does not — and does offering early cost conversions?

Motivation: the conversion playbook (behavior.playbook, formerly sales_led) fires the demo on a low, monotonically accumulating interest score (cumulative_score), so any sufficiently long discovery conversation eventually auto-pitches even without a real buying signal. See packages/ixchat/ixchat/utils/demo_proposal.py.

Method

Two Rose events carry the signal:

• rw_message_sent — one visitor turn (rw_message_number = ordinal, rw_message_source = typed / suggested_answer / booking_cta / …).
• rw_cta_rendered with rw_had_demo_offer = true — the agent rendered the demo offer.

Conversion = rw_cta_clicked / rw_client_cta_clicked / rw_email_captured after the offer.

Three cuts, all keyed on rw_session_id.

Findings

1. Outcome by conversation depth at first demo offer

Offer depth	sessions	converted	continued (no conv)	dead
turn 1	2623	25.3%	17.8%	56.9%
turn 2	296	27.0%	24.7%	48.3%
turn 3	413	23.7%	35.8%	40.4%
turn 4	434	22.1%	34.8%	43.1%
turn 5+	633	20.5%	42.0%	37.4%

Conversion is flat (~20–27%) regardless of when the offer fires — offering early wins no extra conversions — but early offers leave 57% of sessions dead (no conversion, no further message) vs 37% for late offers.

2. Continuation, offered vs not offered, controlled for turn depth

Turn	offered → continue	not offered → continue	delta
1	31.9%	49.3%	−17.4pp
2	41.8%	57.8%	−16.0pp
3	47.6%	60.0%	−12.4pp
4	48.2%	63.0%	−14.8pp
5+	58.2%	73.9%	−15.7pp

Offering a demo suppresses continuation by ~15pp at every depth — a robust effect, not a turn-1 artifact.

Verdict

Hypothesis holds. A mistimed demo offer buys no conversion upside and a large engagement/abandon cost. Conversion depends on visitor readiness, not on pitching sooner.

Caveat (observational, not causal): "stopped talking after an offer" is partly by design — a ready visitor leaves the chat to go book. The conversion split in Finding 1 controls for this (conversion does not rise for early offers), but a definitive causal number needs the playbook A/B experiment mart (conversion rate conversion vs information).

Design implication

The additive, monotonic interest score conflates engagement length with buying readiness. Because a wrong offer has measurable cost and delaying costs no conversions, readiness gating should fire on genuine buyer pull / qualification state, not accumulated turn count. Tracked in IX-4063.

Reproduction

Run via the PostHog MCP execute-sql tool (HogQL), project 80620.

-- Finding 1: outcome by depth at first demo offer
WITH offers AS (
  SELECT properties.rw_session_id AS sid, min(timestamp) AS ts0
  FROM events
  WHERE event='rw_cta_rendered' AND properties.rw_had_demo_offer=true
    AND properties.rw_environment='production'
    AND timestamp>now()-INTERVAL 90 DAY AND coalesce($virt_is_bot,false)=false
  GROUP BY sid),
msgs AS (
  SELECT properties.rw_session_id AS sid, timestamp AS ts
  FROM events WHERE event='rw_message_sent' AND properties.rw_environment='production'
    AND timestamp>now()-INTERVAL 90 DAY),
conv AS (
  SELECT properties.rw_session_id AS sid, min(timestamp) AS tsc
  FROM events WHERE event IN ('rw_cta_clicked','rw_client_cta_clicked','rw_email_captured')
    AND properties.rw_environment='production' AND timestamp>now()-INTERVAL 90 DAY
  GROUP BY sid),
per AS (
  SELECT o.sid AS sid,
    countIf(m.ts<=o.ts0) AS depth,
    countIf(m.ts>o.ts0) AS msgs_after,
    max(if(c.tsc>=o.ts0,1,0)) AS converted
  FROM offers o LEFT JOIN msgs m ON m.sid=o.sid LEFT JOIN conv c ON c.sid=o.sid
  GROUP BY o.sid)
SELECT multiIf(depth<=1,'1',depth=2,'2',depth=3,'3',depth=4,'4','5+') AS depth_at_offer,
  count() AS sessions,
  round(avg(converted)*100,1) AS pct_converted,
  round(avg(if(converted=0 AND msgs_after>0,1,0))*100,1) AS pct_continued_noconv,
  round(avg(if(converted=0 AND msgs_after=0,1,0))*100,1) AS pct_dead
FROM per GROUP BY depth_at_offer ORDER BY depth_at_offer;

-- Finding 2: continuation offered vs not, per turn
WITH m AS (
  SELECT properties.rw_session_id AS sid, timestamp AS ts,
    row_number() OVER (PARTITION BY properties.rw_session_id ORDER BY timestamp) AS n,
    leadInFrame(timestamp,1,toDateTime(0)) OVER (
      PARTITION BY properties.rw_session_id ORDER BY timestamp
      ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS next_ts
  FROM events WHERE event='rw_message_sent' AND properties.rw_environment='production'
    AND timestamp>now()-INTERVAL 90 DAY AND coalesce($virt_is_bot,false)=false),
o AS (
  SELECT properties.rw_session_id AS sid, timestamp AS ots
  FROM events WHERE event='rw_cta_rendered' AND properties.rw_had_demo_offer=true
    AND properties.rw_environment='production' AND timestamp>now()-INTERVAL 90 DAY),
j AS (
  SELECT m.sid AS sid, m.n AS n, (m.next_ts>toDateTime(1)) AS continued,
    max(if(o.ots>=m.ts AND (m.next_ts<=toDateTime(1) OR o.ots<m.next_ts),1,0)) AS offered
  FROM m LEFT JOIN o ON o.sid=m.sid GROUP BY m.sid, m.n, m.next_ts)
SELECT multiIf(n>=5,'5+',toString(n)) AS turn,
  countIf(offered=1) AS offered_n, round(avgIf(continued, offered=1)*100,1) AS cont_if_offered,
  countIf(offered=0) AS notoffered_n, round(avgIf(continued, offered=0)*100,1) AS cont_if_not
FROM j GROUP BY turn ORDER BY turn;