> For the complete documentation index, see [llms.txt](https://partner-docs.covergenius.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://partner-docs.covergenius.com/optimisations/technical-reference.md). # Technical Reference This section covers the statistical methodology, data quality controls, and known limitations of the Optimisations platform. It is intended for data scientists, engineers, and technical reviewers who need to understand how results are produced and what constraints apply. ## Workflow design Explorations use uniform random allocation across all price variants, not adaptive allocation. This is a deliberate choice. Adaptive methods shift traffic toward better-performing variants as data accumulates, surfacing apparent winners faster, but at the cost of introducing selection bias into the signal. Because the purpose of an Exploration is to produce an unbiased foundation for a subsequent controlled Experiment, optimising during the Exploration phase would undermine the very thing it's meant to enable. This reflects the exploration-exploitation trade-off: the Exploration phase accepts short-term sub-optimality to gather representative signal; the Experiment phase uses that signal to make a causal, exploitation-ready decision. The two phases are not interchangeable, and the statistical validity of an Experiment depends on the Exploration having run without interference. ## Statistical framework The platform uses a Bayesian statistical framework throughout. Rather than relying on binary significance thresholds, Bayesian analysis estimates the probability that a variant genuinely outperforms the control given the observed data and prior assumptions. For each metric type, the platform defines a prior distribution representing background beliefs about the parameter before experiment data is observed. As data accumulates, the prior is updated to produce a posterior distribution. Probability of beat and credible intervals are computed from the posterior. The use of conjugate priors ensures closed-form posteriors for proportion and mean metrics, enabling efficient computation at scale. ### Prior specification The platform uses flat, uninformative priors for all metric types. This is a deliberate design choice, the most conservative approach available within a Bayesian framework. | Metric type | Prior | Parameters | Rationale | | ----------------------------- | ----- | ------------------------ | ----------------------------------------------------------------------------------------------------------------- | | Conversion rate (proportions) | Beta | α0 = 1, β0 = 1 | Uniform distribution over \[0,1]. No assumption about the partner's conversion rate before the experiment begins. | | Revenue-per-user (means) | Gamma | shape k = 1, scale θ = 1 | Places no informative assumption on the revenue distribution before data is observed. | Because the priors are flat, the posterior distribution is driven almost entirely by the observed experiment data. The system does not draw on historical partner data, vertical benchmarks, or any external information. Each experiment is evaluated independently on its own evidence. #### **Why flat priors?** Flat priors eliminate the risk of prior misspecification, a situation where a prior calibrated on one partner's history is applied incorrectly to another, or where historical data doesn't reflect current conditions. The trade-off is that flat priors require more data to produce precise estimates. This is why experiment duration and minimum sample size recommendations are conservative. #### **Small sample caution** Because no prior knowledge is incorporated, early experiment results are driven entirely by whatever data has accumulated, which may be small and noisy. Do not treat early results as reliable regardless of how strong they appear. For experiments with fewer than 500 observations per variant, P(beat) estimates should be interpreted with significant caution. Partners operating in low-volume markets should extend experiment durations wherever operationally feasible. ### Chance to beat control P(beat) is defined as P(θT > θC | data), the posterior probability that the treatment parameter exceeds the control parameter given observed data. It is computed via Monte Carlo sampling from the posterior distributions of each variant. A single P(beat) threshold of 0.95 applies to all metrics, both primary and secondary. A result is considered to have strong evidence of a genuine effect when P(beat) ≥ 0.95. ### Credible intervals The uplift range represents the 95% Bayesian credible interval for the estimated effect size, computed as the 2.5th and 97.5th percentiles of the posterior distribution of relative uplift. A narrow interval indicates high precision; a wide interval indicates high uncertainty regardless of the P(beat) value. ### Multiple metrics and multiple comparisons Evaluating multiple metrics simultaneously increases the risk of observing spurious positive results by chance. The platform applies a single P(beat) threshold of 0.95 across all metrics. Pre-specifying the primary metric before the experiment starts is critical, changing it after observing results inflates the false discovery rate regardless of the threshold used. Partners evaluating several secondary metrics should treat individual metric signals as directional rather than confirmatory. A consistent pattern across multiple metrics is more meaningful than a single metric crossing the threshold. ### Sequential monitoring The Bayesian framework supports continuous monitoring throughout the experiment lifecycle. Unlike frequentist approaches, Bayesian posterior estimates remain valid regardless of when they are evaluated. However, acting on very early results still increases the risk of acting on noise. Allow sufficient observations to accumulate before treating P(beat) as stable. ### Sample size and duration The platform does not calculate a minimum detectable effect or a statistically derived sample size or duration. There is no automated duration recommendation. The team running the experiment must use judgement based on expected daily traffic volume and how long is needed for the posterior P(beat) estimate to stabilise. {% hint style="info" %} **Practical heuristic:** Monitor P(beat) daily and look for stability rather than a single threshold crossing. A P(beat) that fluctuates significantly between days indicates insufficient data. A P(beat) that has been consistently above 0.95 for several consecutive days is more trustworthy than one that briefly crossed the threshold. High-variance metrics such as revenue per user require materially larger samples to achieve equivalent precision compared to binary conversion metrics. {% endhint %} ### Projected uplift formula Projected uplift is calculated as: {% code expandable="true" %} ``` (GWP lift % × GWP control) ÷ (percentage traffic of control) × (30 ÷ experiment duration in days) ``` {% endcode %} This is a linear scaling formula. It extrapolates the observed GWP uplift to a full month of traffic and makes no adjustments for: * **Novelty decay.** The formula overstates long-term impact if the experiment captured a novelty spike. * **Seasonal confounding.** If the experiment ran during atypical traffic, the projection inherits that distortion. * **Non-additivity.** The formula assumes each variant's effect is independent; simultaneous rollouts may produce compounding or interfering effects. * **Partial rollout.** The projection assumes full rollout to all eligible traffic. {% hint style="info" %} **Do not use in financial projections:** The projected uplift figure is a mechanical output of this formula applied to experiment results. It does not represent a modelled forecast. The gap between projected uplift and actual post-rollout impact is routinely material and the figure should not be used in financial projections without significant manual adjustment. {% endhint %} ## Randomisation and subject\_id The unit of randomisation for all experiments is the `subject_id`, a field your integration supplies in the XCover API request. Assignment to variants is performed using a hashing function applied to the `subject_id`, which guarantees that the same `subject_id` always maps to the same variant throughout the experiment. You decide what `subject_id` represents. It may be a user ID, session ID, or device ID depending on your implementation. The quality of randomisation depends directly on how your integration populates this field. | Risk | Description | Impact | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | subject\_id collision | A large proportion of traffic shares the same subject\_id (e.g. a session identifier that resets frequently, or a shared device identifier). | Variant allocation percentages are still honoured, but effective sample size is smaller than quote volume suggests. | | subject\_id instability | The same end user generates different subject\_ids across sessions (e.g. a session token rather than a stable user identifier). | The same person may be assigned to different variants across visits, contaminating measurement and producing unreliable results. | ### **Example: unstable vs. stable subject\_id** ```json // ✗ Unstable — do not use // A session token changes every visit. The same user gets assigned // to different variants across sessions, contaminating results. "partner": { "subject_id": "sess_x9k2m7p1" // resets each session } // ✓ Stable — recommended // A persistent account ID is the same across every session for the // same user. Variant assignment stays consistent. "partner": { "subject_id": "usr_a1b2c3d4" // stable account identifier } ``` {% hint style="info" %} **Before running an experiment:** Confirm that your `subject_id` implementation uses a stable, unique identifier for the unit you intend to randomise. If you are unsure, contact your Cover Genius partner team. {% endhint %} ## Carry-over risk When customers are exposed to variants in an Exploration, their behaviour during that period may influence how they respond at the start of a subsequent Experiment. If the Experiment begins immediately, the baseline in the control group may be inflated or deflated by residual behavioural effects from the Exploration, making the Experiment results unreliable. {% hint style="info" %} **No automated enforcement:** The platform does not enforce a waiting period between Explorations and Experiments. If you start an Experiment immediately after an Exploration on the same feature, the platform will not warn you. Managing this risk is the responsibility of the team configuring the experiment. {% endhint %} ### How long to wait The appropriate waiting period depends on your traffic patterns, specifically how frequently your users return and how long the Exploration ran. As a general principle, wait long enough that the majority of users exposed to Exploration variants have had at least one additional session under normal conditions before the Experiment begins. Contact your Cover Genius partner team to confirm the recommended waiting period for your specific traffic context. ### Consequences of insufficient waiting | Risk | Effect | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Inflated baseline | Users exposed to a high-performing Exploration variant may exhibit elevated behaviour in the Experiment control group, compressing apparent uplift. | | Deflated baseline | Users exposed to a poor-performing Exploration variant may exhibit suppressed behaviour, artificially inflating apparent uplift for Experiment variants. | | Misleading early results | Even if the Experiment eventually stabilises, early results will be unreliable and increase the risk of acting on noise before the recommended duration elapses. | ## Data quality The reliability of experiment results depends on the quality of the data flowing into the platform. This section documents what controls exist, which are visible to partners, and where the responsibility for data quality sits with the partner rather than the platform. ### Null subject\_id filtering Requests with a null `subject_id` are automatically filtered from experiment analysis. These records are excluded from all metric calculations and variant comparisons. No other filtering is applied automatically. High repetition rates or suspicious patterns remain in the analysis unless identified and filtered manually. ### Bot detection The Partner Portal includes a BotFilter field that identifies `subject_id`s with more than 200 quotes. This provides a visible signal for detecting automated or non-human traffic. `Subject_id`s flagged by BotFilter should be reviewed before drawing conclusions from experiment data. Internal alerts for bot-like patterns and traffic anomalies exist within Cover Genius systems but are not currently surfaced in the Partner Portal. If you suspect bot traffic is affecting your experiment, contact your partner team. ### Event deduplication Event logs are deduplicated by `quote_id`. If the same conversion event fires more than once for a given quote, it is counted once in the analysis. A single `subject_id` can legitimately generate multiple distinct quotes and policy sales. These are counted as separate events. ### Traffic imbalance Traffic is allocated to variants using a hashing function that guarantees consistent assignment. However, the platform does not currently provide a check in the Partner Portal to confirm that the configured traffic allocation percentages match the actual traffic received per variant. {% hint style="info" %} **Note:** Internal alerts exist for significant traffic imbalances between variants, but these are not surfaced in the Partner Portal. If variant traffic volumes look materially unequal, contact your partner team before acting on the results. {% endhint %} ### Traffic anomaly monitoring Cover Genius monitors for sudden spikes or drops in attach rate, price, quotes, and other key signals per variant. These alerts are internal and not currently surfaced in the Partner Portal. If you observe an unexpected pattern in your experiment charts such as a sudden discontinuity, an implausible value, or a metric moving in an unexpected direction, contact your partner team. ### Audit trail BrightWrite maintains a log of all experiment events, configuration changes, and variant assignments. This audit trail supports internal investigation of data quality issues and experiment history. Access is internal. Partners can request a review through their partner team if they have concerns about the integrity of an experiment's data. ### Dimension analysis > **Internal Note:** This section is a placeholder pending confirmation from the data science team. Yang Wei has flagged Dimension Analysis as one of the most important features in the current portal. Full documentation will be added once the methodology, available dimensions, and any statistical caveats have been confirmed. ## Known limitations | Limitation | Detail | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Local optima | Experiments measure whether a specific change outperforms the current experience within the tested parameter space. They cannot determine whether a fundamentally different approach would outperform all tested variants. | | Audience validity | Results are valid for the traffic composition observed during the experiment window. Significant shifts in audience mix may mean conclusions do not generalise to future traffic. | | Interaction effects | The platform does not model interaction effects between simultaneously running experiments. Overlapping experiments on the same user population may produce compounding or interfering effects not captured by individual experiment results. | | Causal inference scope | Experiments establish evidence of a causal relationship between a specific change and an observed outcome under the conditions of the test. They do not establish causality for downstream business outcomes influenced by factors outside the experiment's scope. | | Flat prior sensitivity (small samples) | At small sample sizes, flat priors provide no stabilising influence, and the posterior can fluctuate significantly. Revenue metrics are particularly sensitive because they carry higher variance than binary conversion metrics. | | Inconclusive aggregate / conclusive segment | An overall inconclusive result does not preclude a valid segment-level conclusion, provided the segment was pre-specified. Post-hoc segment results must be treated as hypotheses and validated in a dedicated follow-up experiment. | *** --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://partner-docs.covergenius.com/optimisations/technical-reference.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.