A/B Test - Sample Size Calculator
Basic
Advanced
Total number of exposures per day
The number of exposures is the number of opportunities to convert. If you're measuring conversion rate from ad impression to click, the number of exposures is the number of ad impressions.
Conversion Rate (%) of Version A [Control]
Current/expected conversion rate of Version A. This will be used as a baseline to measure uplift over.
Assumed uplift (%) of Version B [Treatment]
How much better conversion rate you expect Version B to have.
Example: Conversion Rate A: 10%, Assumed uplift: 25%, the expected conversion rate of B is implied to be 25% higher than that of A (i.e: 12.5%).
Experiment Win Margin (%)
If you require that one version is better with a margin, you may specify it here. If you don't want to use this, you can leave it at 0%
Example 1: Experiment Win Criteria = 15% means B needs to have an uplift of at least 15% in order for it to win the experiment
Example 2: Experiment Win Criteria = -15% means B wins as long as its conversion rate is not more than 15% worse than A
The default parameters on this tab are sane, and are most likely what you want if you do not understand what they do.
% of Exposures assigned to Version A
How large proportion of the total number of exposures to expose to Version A
Unless you are using non-default priors, a 50/50-split requires the fewest number of total exposures except in extreme cases.
Prior: Version A
Distribution
Beta
alpha
beta
Prior: Version B
Distribution
Beta
alpha
beta
Computation quality
fast
accurate
very accurate
Controls accuracy of simulation. "accurate" should be sufficient for except for extremely low baseline conversion rates. Increase this if plots appear jagged
Experiment Simulation
Insights