Google Play has a built-in A/B testing tool most developers never use
Google Play Console includes a feature called Store listing experiments that lets you A/B test your screenshots, icon, short description, and promo video against your live listing. The experiment runs on real traffic, splits visitors between variants, and produces a statistically significant winner.
It's one of the highest-leverage tools available to indie developers — and the majority of apps never run a single experiment.
What you can test
- Screenshots: Different images, different order, different captions
- Short description: Different copy, different emphasis
- App icon: Different designs, colors, styles
- Feature graphic: The wide banner shown on your store page
- Promo video: Different cuts or thumbnails
You can run one experiment per element at a time. Screenshots experiments are the most impactful starting point.
How to set up a screenshot experiment
- Open Google Play Console
- Select your app
- Go to Grow → Store listing experiments
- Click New experiment
- Select the element you want to test (screenshots)
- In the experiment setup, you'll see your current "Control" (live listing) and a "Variant A" slot
- Upload your variant screenshots — these should be meaningfully different from the control, not just slightly tweaked colors
- Set your traffic allocation — 50/50 is the default and recommended for most experiments
- Set an experiment name and click Start experiment
Google will split your listing impressions between the control and variant automatically.
What to test first
The highest-impact test: Screenshot 1 alone.
Because screenshot 1 is the only screenshot visible in search results without expanding, it has a disproportionate effect on install rate. A 10% improvement in screenshot 1 conversion compounds across every search impression your app receives.
Good variants to test for screenshot 1:
- Text vs. no text: Some apps convert better with a bold headline, others with a clean UI-only shot
- Dark vs. light background
- Feature benefit vs. social proof ("Build habits that stick" vs. "Joined by 50,000 users")
- Portrait vs. landscape orientation for tablet-first apps
Reading the results
After your experiment has been running for 2–4 weeks, go back to Store listing experiments to see the results.
Google shows:
- Install rate for control vs. variant
- Statistical confidence — Google recommends waiting for 90%+ confidence before declaring a winner
- Store listing installs from each variant
A result with less than 90% confidence is not reliable. Let the experiment run longer, or accept that the difference between variants is too small to be meaningful.
How long to run an experiment
The minimum useful duration is 2 weeks. This is because:
- App traffic has weekly patterns (weekends vs. weekdays behave differently)
- You need enough impressions to reach statistical significance
For apps with fewer than 100 installs per day, experiments may take 4–6 weeks to reach significance. This is normal. Do not end the experiment early just because one variant looks better after 5 days.
Applying the winner
When the experiment shows a clear winner at 90%+ confidence:
- Click Apply on the winning variant
- Google replaces your live listing with the winning version immediately
- The experiment ends
Your store listing is now updated without you having to manually republish.
Common mistakes
Testing too many things at once: Only change one meaningful thing between control and variant. If you change the background color, the caption, and the order all at once, you can't know which change drove the difference.
Ending experiments too early: The winning screenshot at day 7 may not be the winner at day 21. Weekly traffic patterns can skew early results significantly.
Testing trivial differences: A 5px border change or a slightly different shade of blue will not produce a statistically meaningful result. Make the variants genuinely different.
Not acting on results: Experiments that end with a clear winner but are never applied to the live listing are wasted data. Build a habit of applying winners immediately.
What to do after your first experiment
Once you've found your best-performing screenshot 1, run a second experiment on screenshot 2 with the winning screenshot 1 as your new control. Work through the screenshot set methodically.
Over 3–6 months of consistent experimentation, most apps can improve their listing conversion rate by 15–30%. For an app with 1,000 search impressions per day, a 20% improvement in conversion is 200 extra installs per day — from the same traffic.
Design your screenshot variants with Apporb's screenshot editor — device frames, gradients, all required sizes. Free plan at apporb.com.