> For the complete documentation index, see [llms.txt](https://partner-docs.covergenius.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://partner-docs.covergenius.com/optimisations/experiments.md). # Experiments This section covers how to run and interpret controlled experiments through the Partner Portal. By the end, you'll understand what experiments are, how to set one up, and how to make confident decisions based on your results. ## What is experimentation? Experimentation lets you run randomised controlled trials, commonly known as A/B tests, to measure the impact of product changes on the metrics that matter to your business. In practice, you can test two or more versions of a feature against your current experience and let real user behaviour tell you which one performs best. Cover Genius handles the randomisation, data collection, and statistical analysis, surfacing the results in plain terms so you can act with confidence. Experiments are the validation step that follows an Exploration. Where an Exploration shows you the shape of the opportunity, an Experiment confirms whether a specific change is genuinely worth shipping. ## Why experiment? Controlled experiments are the most reliable way to establish **causality** (not just correlation) between a product change and its effect on user behaviour. Historical data can tell you what happened. Experiments tell you *why*. Running experiments helps you to: * **Ship with confidence.** Only release changes that have been proven to improve performance against your primary metric. * **Catch unintended impacts.** Measure how a change affects your full metric suite, not just the one you're optimising for. * **Move faster.** Real-time feedback on product performance means you can iterate quickly rather than relying on a gut feel. Without experiments, external factors such as seasonal trends, concurrent product changes, user cohort shifts, can easily be mistaken for the effect of your change. Experiments control for those factors so results are attributable to your change, not to chance. ## Key concepts and definitions | Term | Definition | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Control** | The current, unchanged version of your product or feature. Acts as the baseline for all comparisons. Every variant's performance is measured relative to the control. | | **Variant** | A modified version of your product being evaluated. An experiment typically has one control and one or more variants, labelled by their adjustment over the control (e.g. +10%, +20%). | | **Sample** | The number of unique users enrolled in the experiment. Users are randomly assigned to either the control or a variant group and always see the same version for the duration. | | **Duration** | How long the experiment has been running. Longer experiments accumulate more data, which narrows uncertainty and makes results more reliable. | | **Primary metric** | The single most important measure of success for your experiment. Specified before the experiment starts. All winning/losing decisions are anchored to this metric. | | **Leading variant** | The variant currently showing the best performance relative to the control. Displayed prominently in the experiment header. | | **Chance to beat control (P(beat))** | The Bayesian probability that a variant genuinely outperforms the control. See the Chance to beat control section below. | | **Variant vs. control** | The absolute difference in the primary metric value between a variant and the control. For example, if the control GWP per order is $1.44 and the variant shows $1.68, the value is +$0.24. | | **Uplift range** | The 95% Bayesian credible interval for the relative improvement or decline of a variant. A range that doesn't cross zero indicates a clear directional effect. | | **Projected uplift** | The estimated ongoing monthly business impact of shipping the winning variant to all users, calculated from the variant's lift and current traffic volume. | ## Before you start: understanding your experiment setup Before an experiment goes live in your Partner Portal, it is configured by the Cover Genius team in partnership with you. This includes: * **Product.** The Cover Genius protection product being tested (e.g. Travel Cancellation Protection). * **Variants.** The specific configurations being compared against the control. * **Primary metric.** The key measure of success (typically GWP per order). * **Target sample and duration.** How many users and how long the experiment will run. To propose a new experiment or adjust an existing one, contact your Cover Genius Partner Growth Manager (PGM). **Finding Experiments in the Partner Portal:** Log in → left-hand sidebar → Optimisations → Experiments. Select any Experiments name to open its detailed view. ### The Experiments overview When you open an experiment, you'll see a header bar that gives you an at-a-glance summary of the experiment's current state. | Field | What it shows | | --------------- | -------------------------------------------------------------- | | Product | The Cover Genius protection product being tested. | | Sample | The total number of unique users enrolled so far. | | Duration | How long the experiment has been running. | | Leading variant | The variant currently showing the strongest lift over control. | | Primary metric | The main metric being used to judge success. | Below the header, the experiment is organised into four main areas. ### Performance Trend The Performance Trend chart shows how the primary metric has evolved over time for the control and each variant. The X-axis shows the date range; the Y-axis shows the metric value (e.g. GWP in USD). Use this chart to: * Check that all lines start in a similar range. This confirms the randomisation is working. * Spot when a variant begins to diverge from the control. * Assess whether results are stable or still fluctuating. ### Variant comparison table The Variant comparison table is where you read your experiment results. Each row is a variant, with the control as the baseline. The Winning variant row is highlighted in green. | Field | What it means | | ---------------------- | --------------------------------------------------------------------------------------------------------- | | Variant | The name of the variant. The control row is labelled baseline. Winning variants show a green Winning tag. | | Primary metric | The variant's average value for the primary metric (e.g. GWP per order). | | Variant vs. control | The absolute difference between the variant and the control for the primary metric. | | Chance to beat control | The probability the variant genuinely outperforms the control. | | Uplift range | The 95% Bayesian credible interval for the estimated relative improvement or decline. | | Projected uplift | Estimated monthly business impact of shipping the winning variant. | Each row has an expand button on the right to drill into more detail for that specific variant. ### Metrics Overview The Metrics Overview section shows performance across all tracked metrics, both primary and secondary. Use this section to check whether a change that improves your primary metric is also creating unintended effects elsewhere. A variant that boosts GWP per order but reduces the attach rate may not be a straightforward win. ### Metric Trends The Metric Trends section displays time-series charts for each tracked metric, broken down by variant. Use these charts to: * **Monitor consistency.** A metric that improves on some days but not others may not reflect a real effect. * **Identify novelty effects.** A spike in the first few days that fades may indicate users are reacting to the change itself, not its ongoing value. * **Spot data anomalies** that could affect your interpretation. ### Dimension Analysis The Dimension Analysis section lets you break down results by user segment to identify which groups are driving the overall experiment outcome. For example, a variant may perform strongly for users in one country or on one device type, but be neutral or negative for others. This can inform whether to ship to all users or to a targeted segment first. ## How we calculate results Understanding the statistical method used helps you read results correctly and make better decisions. ### Chance to beat control (P(beat)) The Partner Portal uses a Bayesian statistical model to calculate Chance to beat control (P(beat)) for each variant. Rather than asking "could this result be explained by chance?," this approach asks a more direct question: given all the data collected, how likely is it that this variant is genuinely better? | Result | Chance to beat control | | ------------ | ---------------------- | | Winning | ≥ 95% | | Inconclusive | Between 5% and 95% | | Losing | ≤ 5% | A variant tagged Winning at 96.4% means there's a 96.4% probability it outperforms the control and only a 3.6% chance the apparent improvement is down to random variation in the data. {% hint style="info" %} **Note:** Some metrics are traffic-dependent and may show "P(beat) not applicable" until enough data has accumulated. This is expected behaviour and not a sign of a problem with your experiment. {% endhint %} ### Uplift range The uplift range is the 95% Bayesian credible interval for the relative improvement or decline of a variant. It gives you a range within which the true effect likely falls given the data collected so far. As the sample grows, this range narrows. * An uplift range of +8% to +22% means the true improvement is likely somewhere in that band. The entire range being positive is a good sign. * An uplift range that crosses zero (e.g. -2% to +14%) means you can't yet rule out that the variant has no effect or a small negative one. More data is needed. ### Projected uplift Projected uplift translates a statistical result into a business number. It estimates the ongoing monthly impact of shipping the winning variant to all users, based on the variant's lift and your current traffic volume. {% hint style="info" %} **Important:** This figure is an estimate, not a guarantee. It assumes the variant's performance holds across your full user base and that traffic volumes remain stable. Use it directionally, not as a precise forecast. For the full formula and its confirmed limitations, see the Technical Guidance section. {% endhint %} ## Experiment status | Status | What it means | | --------- | ---------------------------------------------------------- | | Scheduled | The experiment has been configured but hasn't started yet. | | Running | The experiment is live and collecting data. | | Closed | The experiment has ended and a decision has been made. | ## Reading your results Here's how to approach your results in practice. 1. **Check the experiment header.** Confirm the sample size looks reasonable for the experiment's duration. A very small sample relative to run time may indicate a targeting or launch issue. 2. **Look at the Performance Trend chart.** Do the control and variant lines start at a similar point? If they diverge immediately from day one, something may be wrong with the randomisation. A gradual divergence over time is the healthy signal. 3. **Read the Variant comparison table.** Find the row with the highest Chance to beat control. Check whether: the chance to beat control is ≥ 95%; the uplift range is fully positive; and the projected uplift is meaningful relative to your business context. 4. **Check Metrics Overview for unintended effects.** A variant winning on GWP per order but showing declines in attach rate warrants a conversation before shipping. 5. **Review Metric Trends for stability.** A variant showing consistent lift over the full experiment window is more trustworthy than one where the lift is concentrated in a single day or week. 6. **Review Dimension Analysis if the result is mixed.** Segment breakdowns often reveal that a variant works well for a specific subset of users, which can still be a valuable, actionable insight. ### Inconclusive results An experiment is inconclusive when the probability of beating control does not reach the decision threshold and the uplift range spans both positive and negative values after the full recommended duration has elapsed. Inconclusive results are not failures. They tell you that the tested change does not produce a detectable effect across your overall traffic, within the tested time window, or at the hypothesised effect size. #### What to do with an inconclusive result * **Extend the experiment.** If traffic was lower than expected or the true effect appears smaller than assumed, extending the duration gives more data and allows P(beat) to stabilize. Review P(beat) stability over consecutive days before deciding whether extension is worthwhile. * **Redesign the variant.** If the uplift range is centred near zero with low variance, the evidence suggests this specific variant has no meaningful effect on overall traffic. Return to Explorations to identify a more promising direction. * **Accept the null.** If the experiment has been running long enough and consistently shows no meaningful difference, deprioritising the change is a valid outcome. #### Inconclusive overall, but conclusive at segment level An overall inconclusive result does not mean there is no signal. It may mean the variant works well for some customer segments and poorly for others, and those effects cancel each other out in the aggregate. A blanket inconclusive result that hides a strong segment-level winner means a blanket rollout is not justified but a targeted rollout to the winning segment may be. {% hint style="info" %} **Pre-specified segments only:** A segment-level result is valid for decision-making only when the segment was defined before the experiment started, not identified by scanning the results. Post-hoc segments are exploratory at best and noise at worst. Do not use a segment identified from experiment results to justify a rollout. Treat it as a hypothesis and run a dedicated segment-level experiment to confirm it. {% endhint %} ## Best practices * **Wait for a clear signal.** An experiment showing 76% chance to beat control still has a 24% chance the result is noise. Give experiments time to reach ≥ 95% before making a shipping decision. * **Check both direction and magnitude.** A 95% chance to beat control with an uplift range of +0.1% to +0.9% is statistically clear but may not be worth shipping. Weigh the projected uplift against the cost of the change. * **Look beyond the primary metric.** A variant that improves one metric while degrading another may do more harm than good. * **Trust consistency over spikes.** A variant that performs strongly over many weeks is a better bet than one that shows a big lift in a single week. Novelty effects are real. * **Don't call experiments too early.** The longer an experiment runs (within reason), the narrower the uplift range becomes and the more confident the P(beat) figure is. * **Use Dimension Analysis when results are mixed.** A globally inconclusive experiment can still reveal segment-level insights that are worth acting on. * **When in doubt, talk to your Partner Growth Manager (PGM).** Your Cover Genius PGM can help you interpret unusual results, propose experiment changes, or design the next test. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://partner-docs.covergenius.com/optimisations/experiments.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.