From d51704a859316950aff65c4d3c8ea56439cb05db Mon Sep 17 00:00:00 2001 From: Jonas Alves Date: Mon, 26 Jan 2026 14:21:40 +0000 Subject: [PATCH] docs: clarify outlier limits are calculated across all variants --- .../goals-and-metrics/metrics/create.mdx | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/web-console-docs/goals-and-metrics/metrics/create.mdx b/docs/web-console-docs/goals-and-metrics/metrics/create.mdx index 4fff5c13..7e1b222e 100644 --- a/docs/web-console-docs/goals-and-metrics/metrics/create.mdx +++ b/docs/web-console-docs/goals-and-metrics/metrics/create.mdx @@ -100,9 +100,20 @@ You can choose one of four methods to manage outliers: :::caution Outlier limits can reduce variance and make metrics more stable, but they also change the underlying data. When used incorrectly, they may hide meaningful effects or introduce bias in experiment results. -Always ensure your limits reflect real-world constraints rather than engineering convenience, and avoid tuning them in response to specific experiment outcomes. +Always ensure your limits reflect real-world constraints rather than engineering convenience, and avoid tuning them in response to specific experiment outcomes. ::: +#### How limits are calculated + +When using the **Quantile** or **Standard Deviation** methods, the outlier limits (lower and upper bounds) are calculated across **all participants in the experiment combined**, not separately for each variant. + +This means: +- The same thresholds apply uniformly to all variants (control and treatments) +- The calculation uses the larger combined sample size, resulting in more stable threshold estimates +- All variants are treated identically, preventing any bias that could arise from variant-specific capping + +This approach ensures fair comparisons between variants while benefiting from the statistical power of the full experiment population. + #### Unlimited This is the default. This option applies no outlier treatment. All values are kept exactly as they appear in the event data.