𝐴𝐵Synthesis

The useful metric for 𝐴𝐵 tests is uplift, which is a continuous measure (where no effect = 0) rather than an odds ratio measure (where no effect = 1).

Based on conventional statistical assumptions that the success rates are sampled from an underlying continuous measure (e.g., uplift) that follows a logistic distribution, and that the variability of the success rate is the same in both groups 𝐴 and 𝐵, the log odds ratios can be converted to a continuous mean difference score, called standardized mean uplift:

$${𝑌}_{𝑖}=\frac{\surd 3}{𝜋}{\mathrm{𝑙𝑛𝑂𝑅}}_{𝑖}$$

Where:

• 𝑌_𝑖 = Standardized Mean Uplift for 𝐴𝐵 test 𝑖, which is the difference in mean uplift between groups 𝐴 and 𝐵 expressed in standard deviation units
• 𝑙𝑛𝑂𝑅_𝑖 = The natural logarithm of the odds ratio effect size of 𝐴𝐵 test 𝑖

The variance of the standardized mean uplift is then: $${𝑉}_{𝑖}={𝑈}_{𝑖}\frac{3}{{𝜋}^2}$$

Where:

• 𝑈_𝑖 = the variance of the odds ratio effect size of 𝐴𝐵 test 𝑖
• 𝑙𝑛𝑂𝑅_𝑖 = The natural logarithm of the odds ratio effect size of 𝐴𝐵 test 𝑖

The synthesized effect size is a summary standardized mean uplift, calculated as the weighted mean of the standardized effect sizes from all individual tests.

$$\mathrm{SMU}=\frac{{\sum }_{\mathrm{𝑖=1}}^{\mathrm{𝑘}}{𝑌}_{𝑖}{𝑤}_{𝑖}}{{\sum }_{\mathrm{𝑖=1}}^{\mathrm{𝑘}}{𝑤}_{𝑖}}$$

Where:

• 𝑌_𝑖 = the effect size (standardized mean uplift) of the ith test
• 𝑤_𝑖 = the weight given to the ith test
• 𝑘 = the total number of experiments

Each experiment is assigned a weight based on the inverse variance of its effect size. Inverse variance weighting means that experiments with higher precision, from larger sample sizes or from more precise measurement, receive more weight in the summary effect.

𝐴𝐵Synthesis uses a random effects model to assign weights. A random effects model does not assume that every experiment estimates exactly the same quantity, but instead assumes that the experimental effect follows a distribution. Statistically this is achieved by adding a scaling factor to the experiment variance that measures the excess variability among the group of experimental results and compares it to that expected from normal sampling variance.

$${𝑤}_{𝑖}=\frac{1}{{𝑉}_{𝑖}+{\tau }^2}$$

Where:

• 𝑉_𝑖 = the variance of the standardized mean update of the ith test
• τ² = the scaling factor measuring the excess variability among the group of experimental results

$${\tau }^2=\frac{Q-\mathrm{df}}{C}$$

The Q statistic represents the total weighted variance and is defined as:

$$𝑄={\sum }_{\mathrm{𝑖=1}}^{\mathrm{𝑘}}{𝑤}_{𝑖}{\mathrm{𝑌}}_{\mathrm{𝑖}}^2-\frac{({\sum }_{\mathrm{𝑖=1}}^{\mathrm{𝑘}}{𝑤}_{𝑖}{𝑌}_{𝑖}{)}^2}{{\sum }_{\mathrm{𝑖=1}}^{\mathrm{𝑘}}{𝑤}_{𝑖}}$$

with degrees of freedom:

$$\mathrm{𝐷𝑓}=𝑘-1$$

Using a scaling factor to ensure that tau-squared is in the same metric as the variance within-studies:

$$𝐶=\sum {𝑤}_{𝑖}-\frac{\sum {𝑤}_{\mathrm{𝑖}}^2}{\sum {𝑤}_{𝑖}}$$

The variance of the combined effect is the reciprocal of the sum of the weights:

$$𝑉=\frac{1}{{\sum }_{\mathrm{𝑖=1}}^{\mathrm{𝑘}}{𝑤}_{𝑖}}$$

The standardized mean uplift can be placed on a normal distribution using the variance:

$$𝑍=\frac{\mathrm{𝑆𝑀𝐷}}{\surd 𝑉}$$

The two-tailed p value is returned based on the normal distribution:

$$𝑝=2[1-\phi (\left|𝑍\right|\left)\right]$$

Having performed these calculations, 𝐴𝐵Synthesis returns the summary standardized mean uplift (SSMU) effect size, and the two-tailed 𝑝 value for the group of experiments.