> For the complete documentation index, see [llms.txt](https://partner-docs.covergenius.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://partner-docs.covergenius.com/optimizations/experiments.md). # Experiments ## What is experimentation? Experimentation lets you run randomized controlled trials, commonly known as A/B tests, to measure the impact of product changes on the metrics that matter to your business. In practice, you can test two or more versions of a product against your current experience and let real user behaviour tell you which one performs best. Cover Genius handles the randomization, data collection, and statistical analysis, surfacing the results in plain terms so you can act with confidence. Experiments are the validation step that follows an Exploration. Where an Exploration shows you the shape of the opportunity, an Experiment confirms whether a specific change is genuinely worth shipping. ## Why experiment? Controlled experiments are the most reliable way to establish **causality** (not just correlation) between a product change and its effect on user behaviour. Historical data can tell you what happened. Experiments tell you why. Running experiments helps you to: * **Ship with confidence.** Only release changes that have been proven to improve performance against your primary metric. * **Catch unintended impacts.** Measure how a change affects your full metric suite, not just the one you're optimising for. * **Move faster.** Real-time feedback on product performance means you can iterate quickly rather than relying on a gut feel. Without experiments, external factors such as seasonal trends, concurrent product changes, user cohort shifts, can easily be mistaken for the effect of your change. Experiments control for those factors so results are attributable to your change, not to chance ## Key concepts and definitions | Term | Definition | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Control** | The current, unchanged version of your product or feature. Acts as the baseline for all comparisons. Every variant's performance is measured relative to the control. | | **Variant** | A modified version of your product being evaluated. An experiment typically has one control and one or more variants, labelled by their adjustment over the control (e.g. +10%, +20%). | | **Sample** | The number of unique users enrolled in the experiment. Users are randomly assigned to either the control or a variant group and always see the same version for the duration. | | **Duration** | How long the experiment has been running. Longer experiments accumulate more data, which narrows uncertainty and makes results more reliable. | | **Primary metric** | The single most important measure of success for your experiment. Specified before the experiment starts. All winning/losing decisions are anchored to this metric. | | **Leading variant** | The variant currently showing the best performance relative to the control. Displayed prominently in the experiment header. | | **Chance to beat control (P(beat))** | The Bayesian probability that a variant genuinely outperforms the control. See the [Chance to beat control section](#chance-to-beat-control-p-beat) below. | | **Variant vs. control** | The absolute difference in the primary metric value between a variant and the control. For example, if the control GWP per order is $1.44 and the variant shows $1.68, the value is +$0.24. | | **Uplift range** | The 95% Bayesian credible interval for the relative improvement or decline of a variant. A range that doesn't cross zero indicates a clear directional effect. | | **Projected uplift** | Total monthly GWP gain from variants that have already won (p(beat) ≥ 95%). Summed across all experiments, running + closed, and normalized to 30 days. Cumulative running tally of proven wins. | ## Before you start: understanding your experiment setup Before an experiment goes live in your Portal, it is configured by the Cover Genius team in partnership with you. This includes: * **Product.** The Cover Genius protection product being tested (e.g. Travel Cancellation Protection). * **Variants.** The specific configurations being compared against the control. * **Primary metric.** The key measure of success (typically GWP per order). To propose a new experiment or adjust an existing one, contact your Cover Genius PGM. ***Finding Experiments in the Partner Portal:** Log in → left-hand sidebar → Optimizations → Experiments. Select any Experiments name to open its detailed view.* ### The Experiments overview All active and historical Experiments are available in your Portal, starting with an at-a-glance summary of the experiment's current state. Each entry contains:
FieldWhat it shows
Experiment nameThe name of the experiment and the protection product being tested.
Running experimentsThe total number of experiments currently live and collecting data.
Projected upliftThe combined estimated monthly business impact across all running experiments with a leading variant.
Ready to callThe number of experiments where the leading variant has reached ≥ 95% Chance to beat control on the primary metric and is ready for a rollout decision.
At riskThe number of experiments where a variant has reached ≤ 5% Chance to beat control on the primary metric, showing evidence of an underperforming variant.
StatusWhether the Experiment is currently 'Running', 'Scheduled', or 'Closed'.
Start dateWhen the Experiment began collecting data.
DurationHow long the Experiment has been running.
ResultsThe leading variant's current P(beat) expressed as a percentage, and whether the experiment is Inconclusive, Ready to call, or At risk.
### Performance Trend Performance Trend shows how the primary metric has evolved over time for the control and each variant. Use it to: * Check that all lines start in a similar range. This tells you the randomization is working. * Spot when a variant begins to diverge from the control. * Assess whether results are stable or still fluctuating. A pattern that holds consistently over time is more meaningful than a short-lived spike. ### Variant comparison table Variant Comparison is where your experiment results come to life. Each variant is measured against the control (the unchanged baseline experience) across four dimensions: how it performed on your primary metric, how different that performance was from the control, how confident the platform is that the difference is real, and what that difference could mean for your business if you ship it. Each variant row contains:
FieldWhat it means
VariantThe name of the variant. The control row is labelled baseline. Winning variants show a green Winning tag.
Primary metricThe variant's average value for the primary metric (e.g. GWP per order).
Variant vs. controlThe absolute difference between the variant and the control for the primary metric.
Chance to beat controlThe probability the variant genuinely outperforms the control.
Uplift rangeThe 95% Bayesian credible interval for the estimated relative improvement or decline.
Projected upliftEstimated monthly business impact of shipping the winning variant.
Each variant can be expanded in your Portal for a more detailed breakdown. ### Metrics Overview Metrics Overview shows performance across all tracked metrics — both primary and secondary. Use it to check whether a change that improves your primary metric is also creating unintended effects elsewhere. A variant that boosts GWP per order but reduces attach rate may not be a straightforward win. ### Metric Trends Metric Trends shows how each tracked metric has evolved over time, broken down by variant. Use it to: * **Check for consistency.** A metric that improves on some days but not others may not reflect a real effect. * **Identify novelty effects.** A spike in the first few days that fades may indicate users are reacting to the change itself, not its ongoing value. * **Spot data anomalies** that could affect your interpretation. ### Dimension Analysis Dimension Analysis breaks down experiment results by customer segment — country, device type, booking value, or any customized customer cohorts that your business cares about. A variant may perform strongly for one group but be neutral or negative for others. Use this to decide whether to ship to all customers or target a specific segment first. Remember: a segment-level result is only actionable if that segment was defined before the experiment started. A pattern identified by scanning results after the fact is a hypothesis to validate in a future experiment, not a confirmed finding to act on. ## How we calculate results Understanding the statistical method used helps you read results correctly and make better decisions. ### Chance to beat control (P(beat)) The results an Experiment produces are calculated using a Bayesian statistical framework to calculate Chance to beat control (P(beat) for each variant. Rather than asking "could this result be explained by chance?," this approach asks a more direct question: given all the data collected, how likely is it that this variant is genuinely better? That probability is P(beat). P(beat) tells you how confident you can be that the variant wins, but not by how much. Read it alongside the expected uplift (your best estimate of the size of the effect, where a negative value means the variant is likely worse) and its 95% credible interval (the range the true lift most plausibly falls in). A variant can post a high P(beat) on top of an uplift too small to be worth shipping, so all three numbers together drive a rollout decision. The full methodology is covered in the Statistical Methodology section. | Result | Chance to beat control | | ------------ | ---------------------- | | Winning | ≥ 95% | | Inconclusive | Between 5% and 95% | | Losing | ≤ 5% | A variant tagged Winning at 96.4% means there's a 96.4% probability it outperforms the control and only a 3.6% probability that it doesn't (that control is actually equal or better). {% hint style="info" %} **Note:** Some metrics are traffic-dependent and may show "P(beat) not applicable" until enough data has accumulated. This is expected behaviour and not a sign of a problem with your experiment. {% endhint %} ### Uplift range The uplift range is the 95% Bayesian credible interval for the relative improvement or decline of a variant. It gives you a range within which the true effect likely falls given the data collected so far. As the sample grows, this range narrows. * An uplift range of +8% to +22% means the true improvement is likely somewhere in that band. The entire range being positive is a good sign. * An uplift range that crosses zero (e.g. -2% to +14%) means you can't yet rule out that the variant has no effect or a small negative one. More data is needed. ### Projected uplift Projected uplift translates a statistical result into a business number. It estimates the ongoing monthly impact of shipping the winning variant to all users, based on the variant's lift and your current traffic volume. {% hint style="info" %} **Important:** This figure is an estimate, not a guarantee. It assumes the variant's performance holds across your full user base and that traffic volumes remain stable. Use it directionally, not as a precise forecast. For the full formula and its confirmed limitations, see the [Technical Reference](/optimizations/technical-reference.md) section. {% endhint %} ## Experiment status
StatusWhat it means
ScheduledThe experiment has been configured but hasn't started yet.
RunningThe experiment is live and collecting data.
ClosedThe experiment has ended and a decision has been made.
## Reading your results Here's how to approach your results in practice. 1. **Check the experiment snapshot.** Confirm the sample size looks reasonable for the experiment's duration. A very small sample relative to run time may indicate a targeting or launch issue. 2. **Look at the Performance Trend chart.** Do the control and variant lines start at a similar point? If they diverge immediately from day one, something may be wrong with the randomisation. A gradual divergence over time is the healthy signal. 3. **Read the Variant Comparison table.** Find the row with the highest Chance to beat control. Check whether: the chance to beat control is ≥ 95%; the uplift range is fully positive; and the projected uplift is meaningful relative to your business context. 4. **Check Metrics Overview for unintended effects.** A variant winning on GWP per order but showing declines in attach rate warrants a conversation before shipping. 5. **Review Metric Trends for stability.** A variant showing consistent lift over the full experiment window is more trustworthy than one where the lift is concentrated in a single day or week. 6. **Review Dimension Analysis if the result is mixed.** Segment breakdowns often reveal that a variant works well for a specific subset of users, which can still be a valuable, actionable insight. ### Inconclusive results An inconclusive result doesn't mean the change has no effect — it means the experiment couldn't establish one. The width of the uplift range tells you which situation you're in: a wide range means there isn't enough data yet to distinguish a real effect from noise; a narrow range centred near zero means the effect has been measured with reasonable precision and is unlikely to be commercially meaningful. #### What to do with an inconclusive result * **Extend the experiment.** If the uplift range is wide or traffic fell short of plan, more data may resolve the question. Extend for pre-specified reasons like these, not because P(beat) is hovering just below the threshold. Repeatedly extending until a result crosses the line inflates the risk of a false positive. * **Redesign the variant.** If the uplift range is narrow and centred near zero, the effect has been measured with reasonable precision, and this specific change is unlikely to move the metric in any meaningful way. Return to Explorations to identify a more promising direction. * **Accept the outcome.** If the range has narrowed to where even the upper end of the uplift range wouldn't justify rollout, deprioritising the change is a valid outcome. #### Inconclusive overall, but conclusive at segment level An overall inconclusive result does not mean there is no signal. It may mean the variant works well for some customer segments and poorly for others, and those effects cancel each other out in the aggregate. A blanket inconclusive result that hides a strong segment-level winner means a blanket rollout is not justified but a targeted rollout to the winning segment may be. {% hint style="info" %} **Pre-specified segments only:** A segment-level result is valid for decision-making only when the segment was defined before the experiment started, not identified by scanning the results. Post-hoc segments are exploratory at best and noise at worst. Do not use a segment identified from experiment results to justify a rollout. Treat it as a hypothesis and run a dedicated segment-level experiment to confirm it. {% endhint %} ## Experiment best practices * **Wait for a clear signal.** An experiment showing 76% chance to beat control still has a 24% chance the result is noise. Give experiments time to reach ≥ 95% before making a shipping decision. * **Check both direction and magnitude.** A 95% chance to beat control with an uplift range of +0.1% to +0.9% is statistically clear but may not be worth shipping. Weigh the projected uplift against the cost of the change. * **Look beyond the primary metric.** A variant that improves one metric while degrading another may do more harm than good. * **Trust consistency over spikes.** A variant that performs strongly over many weeks is a better bet than one that shows a big lift in a single week. Novelty effects are real. * **Don't call experiments too early.** The longer an experiment runs (within reason), the narrower the uplift range becomes and the more confident the P(beat) figure is. * **Use Dimension Analysis when results are mixed.** A globally inconclusive experiment can still reveal segment-level insights that are worth acting on. * **When in doubt, talk to your PGM.** Your Cover Genius PGM can help you interpret unusual results, propose experiment changes, or design the next test. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://partner-docs.covergenius.com/optimizations/experiments.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.