You ran a test. Version B came out ahead by about a point. You shipped it, and nothing in your revenue changed.
I have done this. Most people reading this have done this. And the usual conclusion, that the change was too small to matter, is almost always the wrong one.
The real problem is that the test never had enough people in it to separate a result from noise. And nothing in the tooling tells you so: App Store Connect will happily show you a winner on 400 users.
So this article is mostly one table: how many users you actually need, worked out from your own conversion rate. Then what to do when the honest answer is that you do not have them, which for most indie apps it is.
This is the thing I wish somebody had put in front of me before my first test. It is standard statistics, nothing proprietary, and you can reproduce it in any sample size calculator.
| Your current rate | To detect +10% | +20% | +50% |
|---|---|---|---|
| 2.1% (median freemium) | 76,800 | 20,100 | 3,600 |
| 5% | 31,200 | 8,200 | 1,500 |
| 10.7% (median hard paywall) | 13,700 | 3,600 | 600 |
| 25% | 4,900 | 1,300 | 200 |
| 32.5% (median trial-to-paid) | 3,300 | 900 | 100 |
Read the top-left cell first. At the median freemium rate, which is the rate for an app you can use free and pay to unlock more, proving a 10% improvement takes about 154,000 people.
Now read down the first column. The same 10% lift costs 76,800 per version at 2.1% and 3,300 at 32.5%. A rate twenty times higher needs twenty-three times fewer people.
And read across the top row. Going from a 10% target to a 50% target drops the requirement from 76,800 to 3,600, because the difference between the two versions is squared in the formula. Doubling the size of the change you are looking for cuts the traffic you need by roughly four.
Those two readings are the whole strategy for a small app: test where the rate is high, and test changes big enough to find.
If you have a few thousand users a month, the table says something uncomfortable. You cannot run a conventional paywall A/B test to a conclusion. Not slowly, not eventually. The arithmetic does not get there.
That is not a reason to change nothing. It is a reason to be honest about which of these you are doing.
Three honest options when the numbers do not work
All three are legitimate. Calling the third one a test is what causes the damage.
Move up the funnel
Test paywall view to trial start rather than download to paid. The rate is bigger, so the test is smaller. You learn less, and you learn it.
Test the store listing
App Store Connect and Play run experiments on store traffic, not your users. Impressions you already have, with no app release needed.
Decide, and say so
Make the change because you have a reason. Write the reason down. This is a judgement call, and a judgement call recorded honestly beats a test you misread.
What goes wrong is option three wearing option one’s clothes: a 400-user test, a two-point gap, and a decision presented to everyone as data.
This is the one I would start with, because it needs no users at all. Store traffic is traffic.
On the App Store, product page optimisation runs up to three treatments against your current page, splitting real store visitors between them. Google Play store listing experiments do the same thing.
And because screenshots carry big differences rather than small ones, the lifts tend to sit in the part of the table where the numbers are survivable. The six redesigns I have documented all ran this way.
If version B has new screenshots and a new first caption and a different order, and it wins, you know the set is better. You do not know which change did it, which means you cannot do it again.
The exception is when you are replacing a whole set at once, which is a legitimate thing to test. Just be clear with yourself that the result is “this set beats that set”, not “outcome captions beat feature captions”.
Weekends do not behave like Tuesdays. A test that starts on a Thursday and ends on a Monday has sampled a different population at each end.
Run in multiples of seven days. And give it at least a month for anything downstream of a trial, because a 7-day trial means a week of your test has not finished converting yet.
Write it down. One metric, chosen in advance.
Otherwise you will find something: a test has a dozen numbers attached to it, and if you go looking after the fact, one of them will be up. That is not a finding, it is arithmetic doing what arithmetic does.
This one comes from somebody who has run them at scale. Michal Parizek tested both a 50% and a 100% price rise at Mojo. The 100% version made more new revenue outright. They shipped the 50% one, because seven-day cancellations jumped on the expensive version and that projected out to a worse year.
A revenue number at week two can be the opposite of the answer at month twelve.
Three things to check before you believe it.
Did it reach the number? Go back to the table. If the test did not carry enough people, the winner is not a winner, whatever the dashboard says.
Is the gap bigger than the gap you were looking for? If you powered the test for a 20% lift and saw 6%, you have not found a small effect. You have found nothing, measured imprecisely.
Does it survive the next renewal? A change that lifts conversion and lowers retention is a loss with a nice first week. The metrics that catch that are a month or more downstream.
It depends on your current conversion rate and how big a change you want to detect. At the median freemium rate of 2.1% converting by day 35, detecting a 10% relative lift needs roughly 76,800 users per version, so about 154,000 through the test. At a 32.5% trial-to-paid rate, the same 10% lift needs about 3,300 per version.
Usually because it never had the traffic to show one. A low conversion rate needs far more users than a high one, which is the opposite of most people's intuition. If the test did not reach the sample size for the lift you were looking for, a one or two point gap is noise and the tool will still display it as a winner.
The store listing. App Store Connect product page optimisation runs up to three treatments against your current page using real store visitors, and Google Play store listing experiments do the same. Both use store traffic rather than your installed base, so they work before you have one.
In whole weeks, because weekends do not behave like weekdays, and for at least a month if what you are measuring sits downstream of a trial. A 7-day trial means the last week of your test has not finished converting when you look at it.
Conversion design for subscription apps.
Screenshots, onboarding and paywalls, connected around one goal: getting more users to pay.