Online experimentation & conversion-rate optimisation (A/B testing, bandits, causal impact)

Teams ship website and product changes on intuition, then cannot tell whether conversion, retention or revenue really moved. Experimentation platforms randomise users between variants, measure the difference statistically and increasingly use AI to propose hypotheses, shorten tests (variance reduction, bandits), explain who benefits (heterogeneous effects) and write up results. For changes that cannot be randomised (pricing in one country, an outage, a TV campaign), causal-impact methods estimate the counterfactual from time series.

Problem & value

• Today: marketing and product teams run few tests, stop them early when a dashboard looks good ("peeking"), test many metrics without correction, and write results up by hand. Most ideas are neutral or negative, so a low test throughput means mostly guessing. Tests on low-traffic SMB sites often never reach significance.
• AI value: (1) more and better hypotheses and variant copy from analytics and session data; (2) shorter tests through variance reduction such as CUPED, which uses pre-experiment behaviour as a covariate [3]; (3) sequential testing so results can be monitored continuously without inflating false positives [2]; (4) bandits that shift traffic to better variants during short-lived campaigns; (5) automatic result summaries and a searchable archive of past learnings.
• Evidence: large platforms report that only a minority of tested ideas improve the target metric, which is the core argument for a systematic programme; see the practitioner reference by Kohavi, Tang and Xu [1]. Do not expect fixed uplift percentages: they depend on traffic, baseline and idea quality.

Variants

• Marketing / CRO on websites and shops: landing pages, checkout, pricing display, copy and images, run client-side by a visual editor.
• Product and feature experiments: server-side feature flags with randomised rollout, guardrail metrics (latency, errors, churn) and warehouse-native metric computation.
• Mobile apps and onboarding flows: retention and activation as primary metrics.
• Bandit-style optimisation: headline/promotion testing for short campaigns, recommendations ranking (compare REC-003).
• Quasi-experiments: difference-in-differences, synthetic control and Bayesian structural time series for geo tests, price changes and policy changes.
• Public sector / non-profit: testing letter wording, form design or service pages (nudge units), with ethics review.
• Private use: not applicable.

How it works

1. hypothesis (analytics, session insights, past tests, LLM suggestions)
2. design (primary + guardrail metrics, MDE, sample size, randomisation unit)
3. variant creation (visual editor / feature flag / LLM-drafted copy)
4. assignment + exposure logging (consent-aware, sticky bucketing)
5. monitoring (sample-ratio-mismatch check, guardrails, sequential tests)
6. analysis (CUPED, multiple-testing control, segment/HTE estimates)
7. decision + write-up (human sign-off)
8. learning archive non-randomisable change
9. causal-impact model on control series
10. effect with interval

• Statistics engine: frequentist sequential tests (always-valid p-values) or Bayesian posteriors; CUPED/CUPAC regression adjustment; sample-ratio-mismatch (SRM) detection as a data-quality alarm [1][2].
• Bandits: Thompson sampling or contextual bandits allocate traffic adaptively; they maximise in-test reward but give weaker inference, so use them for short-lived decisions and A/B tests for learning.
• Heterogeneous effects: causal forests, meta-learners (T/X/R-learner) and uplift models (EconML, CausalML) estimate who responds. Treat subgroup findings as hypotheses unless pre-registered, and correct for multiple comparisons.
• Causal impact: synthetic control and Bayesian structural time series (CausalImpact) compare the treated series to predicted counterfactuals [5].
• LLM role: ideation, variant drafting, experiment-brief and result narratives, natural-language querying of past tests. The LLM must never decide significance: numbers come from the statistics engine.
• Human-in-the-loop: experiment owner approves design and ship decision; analysts review SRM, novelty effects and segment claims.

Data preparation

Experimentation needs three data streams: exposure logs (who saw which variant), outcome events (orders, sign-ups) and metric definitions. They typically live in the web shop or app backend, the web analytics tool, the ERP or shop system (revenue, returns) and the consent platform. Typical gaps: no sticky user ID, exposure not logged, inconsistent conversion definitions between marketing and finance, bots and test orders in the data, returns and cancellations not linked to orders, and missing consent flags.

Preparation steps: 1. Agree one metric catalogue with primary and guardrail metrics (PREP-012). 2. Connect orders and revenue from shop or ERP with the event data (PREP-005). 3. Log exposure with a stable randomisation unit; test the pipeline with an A/A test and check sample ratio mismatch. 4. Remove bot and internal traffic; reconcile figures with finance. 5. Keep an archive of past tests and results as a basis for hypotheses (PREP-016).

Tools: SQL/dbt, the experimentation platform's SDK and metric layer, a warehouse if available. Effort is about 30-40% of the project, mostly metric definitions and exposure logging. See also GUIDE-003 and DATA-026.

Implementation paths

Which path for whom

• SMB (A): Kameleoon, AB Tasty, Convert or GrowthBook cloud are mature EU-hostable products with built-in statistics, and the process is standard.
• Enterprise (B): Experiments need custom metrics from the warehouse, integration with feature flags and product data, and high traffic makes seat and event fees large.
• Developer (C): GrowthBook or PostHog self-hosted plus EconML and statsmodels run on HW1 and keep exposure data in-house.

Path A — Buy (recommended for most SMBs)

• Web/CRO suites: Kameleoon (FR, EU), AB Tasty (FR, EU), Optimizely Web/Feature Experimentation (with Opal AI assistant and Stats Engine), VWO, Convert (EU-based), Adobe Target.
• Product / feature experimentation: Statsig, Eppo (Datadog), LaunchDarkly experimentation, GrowthBook (open core, EU cloud option), Amplitude Experiment, PostHog (EU cloud), Split (Harness).
• Check: EU data residency, cookieless or consent-mode operation, server-side SDKs, warehouse-native metrics (Snowflake, BigQuery, Databricks), CUPED and sequential testing, SRM alerts.
• Choose A when traffic is under a few million monthly visitors and the team lacks data-science capacity.

Path B — Build on cloud APIs

• Feature flags from a managed service or OpenFeature SDKs; metrics computed in your warehouse with SQL/dbt; analysis service in Python (statsmodels, scipy, pymc) on a cloud function.
• LLM APIs (Claude, Gemini, GPT, Mistral, EU regions) for hypothesis backlog, variant copy and result narratives, fed only aggregated results rather than raw user rows.
• Choose B when you already have a warehouse and need custom metrics or integration.

Path C — Self-host (open source)

• Platforms: GrowthBook (MIT core, self-hostable, warehouse-native) [4], PostHog (self-host), Unleash/Flagsmith for flags with OpenFeature.
• Libraries: EconML, CausalML, DoWhy, pymc/pymc-marketing, CausalImpact (R) and tfcausalimpact/pycausalimpact [5], statsmodels, Ax/BoTorch for adaptive and Bayesian optimisation.
• LLM: a local 7-30B open-weight model (Qwen, Mistral, Llama class) via vLLM or Ollama for drafting; statistics need only CPU. Hardware: HW1 for the stats stack, HW2 if an LLM is added.

Path D — Custom training

Worth it only at high traffic: in-house contextual bandits, uplift models or ranking-aware experiment designs (interleaving, switchback tests, network-effect corrections) on millions of exposures. Needs reliable exposure logging and a stable metric layer first.

State of the art (2026)

• Sequential and always-valid inference are standard in major engines, enabling continuous monitoring [2]; Bayesian and frequentist options are both common.
• Variance reduction (CUPED and ML-based extensions) is built into most product-experimentation platforms and often shortens runtime noticeably when pre-period data correlates with the metric [3].
• Warehouse-native experimentation (Eppo, Statsig, GrowthBook) keeps metrics in the data warehouse and reduces data copies.
• AI assistants (e.g. Optimizely Opal, platform copilots) generate ideas, variants, test plans and summaries; quality depends on the underlying data, and they do not replace statistical review.
• Limits and failure modes: peeking without sequential methods, SRM from bot traffic or redirects, novelty and primacy effects, interference between users (marketplaces, social features), low-traffic sites, many metrics without correction, over-interpreting segment effects, bandit bias in later analysis, LLM-generated variants that conflict with brand or law, and long-term effects that short tests miss. Cookie-consent rejection also biases samples towards consenting users.

Shared resources & combinations

• Shared stack: causal-inference, tabular-ml and time-series libraries and a warehouse metric layer are shared with DATA-024 (scenario modelling) and REC-005 (marketing mix modelling and attribution, which complements incrementality tests).
• Upstream: DATA-026 (session-replay and UX friction findings feed hypotheses), DATA-001 (conversational BI for metric exploration).
• Downstream/adjacent: REC-003 and REC-006 (personalisation policies validated by experiments), REC-007 (ad creative testing), DATA-008 (data quality monitoring for event streams).
• Open-source LLM server can be shared with other text use cases at HW2.

Effort, cost & KPIs

• Path A: days to weeks to launch the first test; pricing typically by monthly tracked users or events, with free tiers on several tools.
• Path B: 1-3 months incl. event pipeline, metric definitions and dashboards.
• Path C: 2-4 months and part-time data-engineering effort; low compute cost.
• Path D: quarters; justified by high traffic.
• KPIs: experiments per quarter, share of tests with a decision, win rate and cumulative validated uplift on primary metrics, time to decision, SRM incidence, guardrail violations caught, share of shipped changes that were tested, repeated-holdout agreement.
• Cost drivers: traffic volume pricing, engineering for server-side flags, analyst time, consent-rate loss.

People & change

Involve management and the marketing or product lead first to agree on goals, risk appetite for tests and who may ship. IT or developers integrate the SDK and feature flags; data owners (shop, ERP, finance) must confirm metric definitions. The data protection officer checks consent, cookieless operation and the AVV with the platform vendor (GUIDE-005). Works council: usually no, because the tests randomise customer-facing changes and do not evaluate employees; it becomes relevant (§87(1) no. 6 BetrVG) if experiments run on internal tools or if results are used to measure individual staff performance, and §90(1) no. 3 requires information if work processes change. Keep that boundary explicit in a short policy (GUIDE-004).

AI literacy: support the team with training on statistics basics (peeking, multiple testing, novelty effects) and on the fact that LLMs draft hypotheses and texts but never decide significance (GUIDE-006). Communicate that tests are routine, not a judgement on people's ideas. Pilot with one page or funnel and a small team of two to four, a pre-registered hypothesis and a go/no-go rule (GUIDE-009). Typical concern: "tests will override our expertise". Address it by treating expert ideas as hypotheses and having a named experiment owner make the ship decision.

Risks, regulation & compliance

• EU AI Act: typically minimal risk; AI literacy (Art. 4) applies. No Annex III use unless experiments are used to decide access to essential services, employment or credit, or deploy manipulative techniques. Art. 5(1)(a) prohibits AI that deploys subliminal or purposefully manipulative techniques causing significant harm, and 5(1)(b) exploiting vulnerabilities; avoid dark patterns and optimisation targeting vulnerable users [6].
• GDPR / ePrivacy: assignment IDs, cookies and behavioural events are personal data. Client-side testing generally needs consent under ePrivacy rules (Art. 5(3) ePrivacy Directive); document legal basis, minimise identifiers, prefer first-party hosting, EU data residency and processor agreements (Art. 28), define retention and no cross-site tracking [7].
• Consumer law: price experiments and personalised pricing must respect transparency duties (Consumer Rights Directive price-personalisation disclosure) and the DSA ban on dark patterns on platforms; test pricing on groups should avoid discriminatory outcomes.
• Research ethics: experiments on sensitive services or health apps need ethical review; inform users in terms of service.
• Mitigations: pre-registration of primary metric, guardrail metrics, multiple-testing control, holdout groups, audit log of decisions, human sign-off.

Getting started

1. Fix event tracking first: verify exposure logging and run an A/A test to check SRM and metric noise.
2. Choose one primary metric and two guardrails per test; compute the minimum detectable effect and runtime before launch.
3. Pick a tool (Path A or GrowthBook) with EU hosting and a consent strategy agreed with your DPO.
4. Keep a hypothesis backlog; use an LLM to draft ideas and variants, but prioritise by evidence and expected impact.
5. Review every test with a template (result, intervals, segments, learnings) and store it in a searchable archive.

Sources

1. Kohavi, Tang, Xu — Trustworthy Online Controlled Experiments (Cambridge University Press) — https://experimentguide.com/
2. Johari, Koomen, Pekelis, Walsh — Always valid inference: continuous monitoring of A/B tests — https://arxiv.org/abs/1512.04922
3. Deng, Xu, Kohavi, Walker — Improving the sensitivity of online controlled experiments using pre-experiment data (CUPED, WSDM 2013) — https://dl.acm.org/doi/10.1145/2433396.2433413
4. GrowthBook — open-source experimentation platform — https://github.com/growthbook/growthbook
5. Brodersen et al. — Inferring causal impact using Bayesian structural time-series models (Annals of Applied Statistics 2015) — https://research.google/pubs/inferring-causal-impact-using-bayesian-structural-time-series-models/
6. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
7. EDPB — Guidelines 2/2023 on technical scope of Art. 5(3) of the ePrivacy Directive — https://www.edpb.europa.eu/our-work-tools/documents/public-consultations/2023/guidelines-22023-technical-scope-art-53-eprivacy_en
8. Microsoft — EconML — https://github.com/py-why/EconML