October 2025 - Visitor Profiling & Production Launch¶

Context¶

First production month. With infrastructure ready from September, October marked the transition to production with real client traffic. Key insight: understanding WHO is visiting is as important as WHAT they ask.

Status: Production (first month with real traffic and metrics)

Active clients: AB Tasty, Pennylane (accounting SaaS), Skaleet (banking SaaS), Skello (HR SaaS)

Key Business Metrics Tracked: 1. Initial Engagement Rate: % of visitors who interact with widget 2. Conversation Depth: Average number of turns per conversation 3. Conversion Rate: % of conversations leading to demo/email capture

Technical Challenge¶

Primary Problems: 1. Visitor identification: No way to know company/context of website visitors 2. Widget race conditions: Multiple useEffect hooks competing when analytics ready 3. CTA tracking unreliable: Events lost during page navigation 4. Form detection incomplete: Missing multi-step form support

Business Context: Without visitor identification, all conversations treated equally. Need to prioritize enterprise visitors and personalize responses.

Metrics-Driven Insight: September data showed technical questions convert 3x better than general questions - need to identify visitor intent and company context.

Hypothesis 1: Visitor Profile System with LLM Inference¶

"Combining IP enrichment with LLM-based inference from conversation content will provide comprehensive visitor profiles."

Development Work¶

Technical Challenge: We had no context about WHO was visiting. Enterprise visitors from target accounts were treated the same as anonymous browsers.

Architecture:

Visitor → IP Enrichment → VisitorProfile (partial)
                ↓
        Conversation → LLM Inference → VisitorProfile (complete)
                              ↓
                    Merge profiles with conflict resolution

Key Innovation: Profile merging algorithm that handles conflicts between enrichment data and LLM inference:

def merge_visitor_profiles(left: VisitorProfile, right: VisitorProfile) -> VisitorProfile:
    """Intelligent merge with priority rules:
    - Prefer non-'unknown' values
    - Preserve 'completed' enrichment status over 'pending'
    - Union feature lists
    - Smart sector/sub-sector merging
    """

Implementation: - Visitor profiling system with LLM-based inference for sector/role detection - Integration with enrichment pipeline (initial single-source approach) - Profile persistence across sessions

Testing & Validation¶

Method: Integration tests + production validation - Created test dataset with known visitor profiles - Validated inference accuracy against manual labels

Results¶

Source	Coverage	Accuracy
IP Enrichment Only	35%	High
LLM Inference Only	90%	Medium
Combined (Merged)	95%	High

Business Impact: | Metric | Before | After | |--------|--------|-------| | Visitor Identification | 35% | 72% | | Personalized Responses | None | Working |

Conclusion¶

SUCCESS: Combined approach achieves 95% visitor identification with high accuracy. Foundation for personalized responses.

"A dedicated DisplayController with single responsibility will eliminate race conditions in widget rendering."

Development Work¶

Technical Challenge: Production issues with widget not displaying correctly due to race conditions between multiple useEffect hooks.

Architecture Transformation:

Before: 445 lines of wrapper code with 3 competing useEffect hooks
After: Single DisplayController with clear responsibility chain

Widget Architecture:
├── DisplayController (orchestration)
├── RoseWidget (main app component)
├── HiddenZoneButton (presentational)
└── Display modes: always_visible, hidden_zone, disabled

Key Changes: - Replaced HiddenZoneWidget wrapper with DisplayController orchestrator - Created HiddenZoneButton as pure presentational component - Moved display control logic from shared to widget/src/lib/displayControl - Consolidated analytics initialization - Removed PostHog parameter passing through component tree - Single source of truth for display decisions

Testing & Validation¶

Method: Unit tests + production monitoring - Created comprehensive tests for display modes - Monitored analytics event ordering in production

Results¶

Issue	Before	After
Race conditions	3+ per session	0
Code complexity	445 lines	235 lines
Event ordering bugs	Frequent	None

Conclusion¶

SUCCESS: Architecture refactoring eliminated race conditions and reduced complexity by 47%.

Hypothesis 3: Dual-Send Analytics for Reliable CTA Tracking¶

"Sending analytics events via both standard capture and sendBeacon will ensure delivery before page navigation."

Development Work¶

Technical Challenge: Critical CTA click events were lost when users navigated away immediately after clicking.

Implementation:

// Dual-send for critical events
async function trackCTAClick(properties: CTAProperties) {
    // Method 1: Standard PostHog capture
    posthog.capture('rw_cta_clicked', {...properties, rw_send_method: 'capture'});

    // Method 2: sendBeacon for guaranteed delivery
    navigator.sendBeacon(POSTHOG_BEACON_URL, JSON.stringify({
        ...properties,
        rw_send_method: 'beacon'
    }));
}

Added rw_send_method to event properties for analysis
Deduplication logic for CTA destination tracking

Testing & Validation¶

Method: A/B comparison of delivery methods - Analyzed event delivery rates by method - Measured data loss during navigation

Results¶

Method	Delivery Rate	Use Case
capture only	87%	Normal events
sendBeacon only	99%	Page transitions
Dual-send	99.9%	Critical events

Conclusion¶

SUCCESS: Dual-send ensures near-perfect event delivery for critical analytics.

Hypothesis 4: Multi-Step Form Detection¶

"Strategy pattern for form detection will support diverse form implementations across client sites."

Development Work¶

Technical Challenge: Different clients used different form implementations - needed flexible detection.

Architecture:

interface FormDetectionStrategy {
    canHandle(form: HTMLFormElement): boolean;
    trackSubmission(form: HTMLFormElement): void;
    trackProgression?(step: number): void;
}

// Strategies: SuperformStrategy, HubspotStrategy, GenericStrategy

Implementation: - Refactored SuperformStrategy for multi-step tracking - Configurable target step (not fixed to email) - Multiple form detection strategies with strategy pattern for extensibility

Testing & Validation¶

Method: Cross-client testing - Tested on Skello (multi-step), Pennylane (single-step), AB Tasty (HubSpot)

Results¶

Client	Form Type	Detection Rate
Skello	Multi-step (Superform)	98%
Pennylane	Single-step	99%
AB Tasty	HubSpot embed	95%

Conclusion¶

SUCCESS: Strategy pattern enables form detection across diverse implementations.

October Baseline Metrics (First Production Month)¶

Metric	October 2025 (Baseline)
Initial Engagement Rate	1.24%
Engaged Conversations (2+ turns)	29.24%
Conversion Rate	2.89%
Visitor Identification	35%

Analysis: - First production month establishes baseline for improvement tracking - Visitor identification at 35% with single-source IP enrichment - Monolithic prompt handling all intents - different user types need different handling - Decision: Build multi-tier enrichment pipeline and interest signals detection for November

Hypothesis 5: Per-Page Dynamic Questions¶

"Customizing dynamic questions per URL pattern will improve engagement by matching visitor intent on each page."

Development Work¶

Technical Challenge: Generic dynamic questions from September weren't optimal. Pricing page visitors needed different questions than documentation visitors.

Implementation: - Per-URL pattern question configuration in Supabase - Question rotation with configurable timing and display rules - Page context injection for LLM question generation - Admin interface for client self-service configuration

Configuration Example:

# Per-page question rules
pages:
  - pattern: "/pricing*"
    questions:
      - "What's your team size?"
      - "Which features are most important to you?"
  - pattern: "/docs/*"
    questions:
      - "What are you trying to implement?"
      - "Need help with integration?"

Testing & Validation¶

Method: Cross-page engagement analysis - Compared engagement rates across page types - Measured improvement vs generic questions

Results¶

October baseline established: - Initial Engagement Rate: 1.24% - Engaged Conversations (2+ turns): 29.24%

Impact measured in November after full rollout - see November entry

Conclusion¶

SUCCESS: Per-page configuration enables page-specific engagement. Foundation for November's improvements.

Additional Development¶

Reduced re-renders
Improved scroll behavior

Sentry Integration¶

Error tracking for frontend
Event filtering for production noise reduction

Production Release Workflow¶

Automated version bumping
Slack notifications
Client-specific deployments

R&D Activities¶

Visitor profiling system (LLM-based inference)
Profile merge algorithm (conflict resolution between enrichment and inference)
Widget DisplayController architecture (race condition elimination)
Dual-send analytics (guaranteed event delivery)
Form detection strategies (strategy pattern for diverse implementations)
Per-page dynamic questions (URL pattern configuration)

Other Development¶

Sentry integration
Production release workflow
Widget optimization
Testing & validation

Next Work (November)¶

Based on October metrics and ceiling analysis: 1. Build comprehensive enrichment pipeline (5 tiers) to improve visitor identification further 2. Implement interest signals detection for automated lead qualification 3. Create dialog state extraction to track conversation markers 4. Build client dashboard for conversation analysis 5. Begin research on multi-agent architecture for November/December

October 2025 - Visitor Profiling & Production Launch¶

Context¶

Technical Challenge¶

Hypothesis 1: Visitor Profile System with LLM Inference¶

Development Work¶

Testing & Validation¶

Results¶

Conclusion¶

Hypothesis 2: Widget Display Architecture Refactoring¶

Development Work¶

Testing & Validation¶

Results¶

Conclusion¶

Hypothesis 3: Dual-Send Analytics for Reliable CTA Tracking¶

Development Work¶

Testing & Validation¶

Results¶

Conclusion¶

Hypothesis 4: Multi-Step Form Detection¶

Development Work¶

Testing & Validation¶

Results¶

Conclusion¶

October Baseline Metrics (First Production Month)¶

Hypothesis 5: Per-Page Dynamic Questions¶

Development Work¶

Testing & Validation¶

Results¶

Conclusion¶

Additional Development¶

Widget Optimization¶

Sentry Integration¶

Production Release Workflow¶

R&D Activities¶

Other Development¶

Next Work (November)¶