> For the complete documentation index, see [llms.txt](https://partner-docs.covergenius.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://partner-docs.covergenius.com/optimizations/technical-reference.md).

# Technical Reference

This section covers the statistical methodology, data quality controls, and known limitations of the Optimizations platform. It is intended for data scientists, engineers, and technical reviewers who need to understand how results are produced and what constraints apply.

## Workflow design

Explorations use uniform random allocation across all price variants, not adaptive allocation. This is a deliberate choice. Adaptive methods shift traffic toward better-performing variants as data accumulates, surfacing apparent winners faster, but at the cost of introducing selection bias into the signal. Because the purpose of an Exploration is to produce an unbiased foundation for a subsequent controlled Experiment, optimizing during the Exploration phase would undermine the very thing it's meant to enable.

This reflects the exploration-exploitation trade-off: the Exploration phase accepts short-term sub-optimality to gather representative signal; the Experiment phase uses that signal to make a causal, exploitation-ready decision. The two phases are not interchangeable, and the statistical validity of an Experiment depends on the Exploration having run without interference.

## Statistical framework

The platform uses a Bayesian statistical framework throughout. Rather than relying on binary significance thresholds, Bayesian analysis estimates the probability that a variant genuinely outperforms the control given the observed data and prior assumptions.

For each metric type, the platform defines a prior distribution representing background beliefs about the parameter before experiment data is observed. As data accumulates, the prior is updated to produce a posterior distribution. Probability of beat and credible intervals are computed from the posterior. The use of conjugate priors ensures closed-form posteriors for proportion and mean metrics, enabling efficient computation at scale.

### Prior specification

The platform uses weakly informative priors. For conversion (ratio) metrics, the prior is an **empirical prior derived from the experiment's own control arm**: centred on the control's observed conversion rate with the strength of a single pseudo-observation. This provides light regularisation while letting the observed data dominate almost immediately.

<table><thead><tr><th>Metric type</th><th width="149.572021484375">Prior</th><th width="194.037353515625">Parameters</th><th>Rationale</th></tr></thead><tbody><tr><td>Ratio metrics (e.g. conversion rate)</td><td>Beta-Binomial</td><td>Empirical Beta prior: <code>α₀ = 1 + p̂ᶜ</code>, <code>β₀ = 2 − p̂ᶜ</code>, where <code>p̂ᶜ</code> is the control arm's observed conversion rate (rounded to 2 dp)</td><td>Weakly informative, centred on the control rate with the weight of <code>~1</code> pseudo-observation. Provides mild regularisation for small samples without materially influencing results.</td></tr><tr><td>Revenue metrics (e.g. GWP per order)</td><td>Two-part compound model: Beta-Binomial for conversion × Gamma for revenue-per-conversion</td><td>Conversion component uses the empirical Beta prior above; revenue-per-conversion component uses Gamma (shape k = <code>1</code>, scale θ = <code>1</code>)</td><td>The Gamma (<code>1</code>, <code>1</code>) prior places minimal assumption on the revenue-per-conversion distribution before data is observed.</td></tr></tbody></table>

Importantly, **no historical data from other experiments, products, or markets is used.** The only data that informs the prior is the control arm of the experiment itself. Each experiment is evaluated on its own evidence.

#### **Why an empirical prior rather than external history?**

Anchoring the prior to the current experiment's control arm eliminates the risk of prior misspecification — a situation where a prior calibrated on one partner's history is applied incorrectly to another, or where historical data doesn't reflect current conditions. Because the prior carries the weight of only a single pseudo-observation, it acts as light regularisation for very small samples and is quickly overwhelmed by observed data.

#### **Small sample caution**

Because no prior is very weak, early experiment results are driven entirely by whatever data has accumulated, which may be small and noisy. Do not treat early results as reliable regardless of how strong they appear.

For experiments with small per-variant sample sizes, P(beat) estimates should be interpreted with significant caution. If you’re operating in a low-volume market, you should extend experiment durations wherever operationally feasible.

### Chance to beat control

P(beat) is the posterior probability that the treatment outperforms the control given observed data. It is computed via Monte Carlo sampling: 100,000 draws from each variant's posterior, with P(beat) taken as the proportion of samples in which the treatment's relative uplift over control is positive — equivalent to `P(θT > θC | data)`.

P(beat) is computed for every configured metric, but the two roles differ:

* **Primary metric — drives the call.** An experiment is considered ready to call, and a variant is flagged as winning, when the leading variant's P(beat) on the primary metric reaches ≥ 0.95. This is the threshold that gates rollout decisions and confirmed-uplift totals.
* **Secondary metrics — evidence only.** Secondary metrics display P(beat) against the same 0.95 benchmark for interpretation and guardrail checks, but they do not determine the winning call.
* **At risk:** a variant with P(beat) < 0.05 on the primary metric shows strong evidence of underperforming control and is flagged as at risk.

GWP lift % × (GWP control ÷ percentage traffic of control) × (30 ÷ experiment duration in days)

### Credible intervals

The uplift range represents the 95% Bayesian credible interval for the estimated effect size, computed as the 2.5th and 97.5th percentiles of the posterior distribution of relative uplift. A narrow interval indicates high precision; a wide interval indicates high uncertainty regardless of the P(beat) value.

### Multiple metrics and multiple comparisons

Evaluating multiple metrics simultaneously increases the risk of observing spurious positive results by chance. The platform applies a single P(beat) threshold of 0.95 across all metrics. Pre-specifying the primary metric before the experiment starts is critical, changing it after observing results inflates the false discovery rate regardless of the threshold used.

Partners evaluating several secondary metrics should treat individual metric signals as directional rather than confirmatory. A consistent pattern across multiple metrics is more meaningful than a single metric crossing the threshold.

### Sequential monitoring

The Bayesian framework supports continuous monitoring throughout the experiment lifecycle. Unlike frequentist approaches, Bayesian posterior estimates remain valid regardless of when they are evaluated. However, acting on very early results still increases the risk of acting on noise. Allow sufficient observations to accumulate before treating P(beat) as stable.

### Sample size and duration

The platform does not calculate a minimum detectable effect or a statistically derived sample size or duration. There is no automated duration recommendation. The team running the experiment must use judgement based on expected daily traffic volume and how long is needed for the posterior P(beat) estimate to stabilize.

{% hint style="info" %}
**Practical heuristic:** Monitor P(beat) daily and look for stability rather than a single threshold crossing. A P(beat) that fluctuates significantly between days indicates insufficient data. A P(beat) that has been consistently above 0.95 for several consecutive days is more trustworthy than one that briefly crossed the threshold. High-variance metrics such as revenue per user require materially larger samples to achieve equivalent precision compared to binary conversion metrics.
{% endhint %}

### Projected uplift formula

Projected uplift is calculated as:

{% code expandable="true" %}

```
GWP lift % × (GWP control ÷ percentage traffic of control) × (30 ÷ experiment duration in days)
```

{% endcode %}

This is a linear scaling formula. It extrapolates the observed GWP uplift to a full month of traffic and makes no adjustments for:

* **Novelty decay.** The formula overstates long-term impact if the experiment captured a novelty spike.
* **Seasonal confounding.** If the experiment ran during atypical traffic, the projection inherits that distortion.
* **Non-additivity.** The formula assumes each variant's effect is independent; simultaneous rollouts may produce compounding or interfering effects.
* **Partial rollout.** The projection assumes full rollout to all eligible traffic.

{% hint style="info" %}
**Do not use in financial projections:** The projected uplift figure is a mechanical output of this formula applied to experiment results. It does not represent a modelled forecast. The gap between projected uplift and actual post-rollout impact is routinely material and the figure should not be used in financial projections without significant manual adjustment.
{% endhint %}

## Randomization and subject\_id

The unit of randomization for all experiments is the `subject_id`, a field your integration supplies in the XCover API request. Assignment to variants is performed using a hashing function applied to the `subject_id`, which guarantees that the same `subject_id` always maps to the same variant throughout the experiment.

You decide what `subject_id` represents. It may be a user ID, session ID, or device ID depending on your implementation. The quality of randomization depends directly on how your integration populates this field.

| Risk                     | Description                                                                                                                                   | Impact                                                                                                                           |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `subject_id` collision   | A large proportion of traffic shares the same `subject_id` (e.g. a session identifier that resets frequently, or a shared device identifier). | Variant allocation percentages are still honored, but the effective sample size is smaller than the quote volume suggests.       |
| `subject_id` instability | The same end user generates different `subject_id`s across sessions (e.g. a session token rather than a stable user identifier).              | The same person may be assigned to different variants across visits, contaminating measurement and producing unreliable results. |

### **Example: unstable vs. stable subject\_id**

```json
// ✗ Unstable — do not use
// A session token changes every visit. The same user gets assigned
// to different variants across sessions, contaminating results.
"partner": {
  "subject_id": "sess_x9k2m7p1"   // resets each session
}

// ✓ Stable — recommended
// A persistent account ID is the same across every session for the
// same user. Variant assignment stays consistent.
"partner": {
  "subject_id": "usr_a1b2c3d4"    // stable account identifier
}
```

{% hint style="info" %}
**Before running an experiment:** Confirm that your `subject_id` implementation uses a stable, unique identifier for the unit you intend to randomise. If you are unsure, contact your Cover Genius partner team.
{% endhint %}

## Carry-over risk

When customers are exposed to variants in an Exploration, their behaviour during that period may influence how they respond at the start of a subsequent Experiment. If the Experiment begins immediately, the baseline in the control group may be inflated or deflated by residual behavioural effects from the Exploration, making the Experiment results unreliable.

{% hint style="info" %}
**No automated enforcement:** The platform does not enforce a waiting period between Explorations and Experiments. If you start an Experiment immediately after an Exploration on the same feature, the platform will not warn you. Managing this risk is the responsibility of the team configuring the experiment.
{% endhint %}

### How long to wait

The appropriate waiting period depends on your traffic patterns, specifically how frequently your users return and how long the Exploration ran. As a general principle, wait long enough that the majority of users exposed to Exploration variants have had at least one additional session under normal conditions before the Experiment begins. Contact your Cover Genius PGM to confirm the recommended waiting period for your specific traffic context.

### Consequences of insufficient waiting

| Risk                     | Effect                                                                                                                                                           |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Inflated baseline        | Users exposed to a high-performing Exploration variant may exhibit elevated behaviour in the Experiment control group, compressing apparent uplift.              |
| Deflated baseline        | Users exposed to a poor-performing Exploration variant may exhibit suppressed behaviour, artificially inflating apparent uplift for Experiment variants.         |
| Misleading early results | Even if the Experiment eventually stabilises, early results will be unreliable and increase the risk of acting on noise before the recommended duration elapses. |

## Data quality

The reliability of experiment results depends on the quality of the data flowing into the platform. This section documents what controls exist, which are visible to partners, and where the responsibility for data quality sits with the partner rather than the platform.

### Null subject\_id filtering

Requests with a null `subject_id` are automatically filtered from experiment analysis. These records are excluded from all metric calculations and variant comparisons. No other filtering is applied automatically. High repetition rates or suspicious patterns remain in the analysis unless identified and filtered manually.

### Bot detection

The Partner Portal includes a `BotFilter` field that identifies `subject_id`s with more than 200 quotes. This provides a visible signal for detecting automated or non-human traffic. `Subject_id`s flagged by `BotFilter` should be reviewed before drawing conclusions from experiment data.

Internal alerts for bot-like patterns and traffic anomalies exist within Cover Genius systems but are not currently surfaced in the Partner Portal. If you suspect bot traffic is affecting your experiment, contact your partner team.

### Event deduplication

Event logs are deduplicated by `quote_id`. If the same conversion event fires more than once for a given quote, it is counted once in the analysis. A single `subject_id` can legitimately generate multiple distinct quotes and policy sales. These are counted as separate events.

### Traffic imbalance

Traffic is allocated to variants using a hashing function that guarantees consistent assignment. However, the platform does not currently provide a check in the Partner Portal to confirm that the configured traffic allocation percentages match the actual traffic received per variant.

{% hint style="info" %}
**Note:** Internal alerts exist for significant traffic imbalances between variants, but these are not surfaced in the Partner Portal. If variant traffic volumes look materially unequal, contact your partner team before acting on the results.
{% endhint %}

### Traffic anomaly monitoring

Cover Genius monitors for sudden spikes or drops in attach rate, price, quotes, and other key signals per variant. These alerts are internal and not currently surfaced in the Partner Portal. If you observe an unexpected pattern in your experiment charts such as a sudden discontinuity, an implausible value, or a metric moving in an unexpected direction, contact your partner team.

### Audit trail

BrightWrite maintains a log of all experiment events, configuration changes, and variant assignments. This audit trail supports internal investigation of data quality issues and experiment history. Access is internal. Partners can request a review through their partner team if they have concerns about the integrity of an experiment's data.

## Known limitations

| Limitation                                  | Detail                                                                                                                                                                                                                                                             |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Local optima                                | Experiments measure whether a specific change outperforms the current experience within the tested parameter space. They cannot determine whether a fundamentally different approach would outperform all tested variants.                                         |
| Audience validity                           | Results are valid for the traffic composition observed during the experiment window. Significant shifts in audience mix may mean conclusions do not generalize to future traffic.                                                                                  |
| Interaction effects                         | The platform does not model interaction effects between simultaneously running experiments. Overlapping experiments on the same user population may produce compounding or interfering effects not captured by individual experiment results.                      |
| Causal inference scope                      | Experiments establish evidence of a causal relationship between a specific change and an observed outcome under the conditions of the test. They do not establish causality for downstream business outcomes influenced by factors outside the experiment's scope. |
| Flat prior sensitivity (small samples)      | At small sample sizes, flat priors provide no stabilizing influence, and the posterior can fluctuate significantly. Revenue metrics are particularly sensitive because they carry higher variance than binary conversion metrics.                                  |
| Inconclusive aggregate / conclusive segment | An overall inconclusive result does not preclude a valid segment-level conclusion, provided the segment was pre-specified. Post-hoc segment results must be treated as hypotheses and validated in a dedicated follow-up experiment.                               |

***


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://partner-docs.covergenius.com/optimizations/technical-reference.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.