Opal · B2C · screen time app
Split users with Firebase and compare groups in Amplitude
Answers in about 8 days; ran 121 tests with a team of 7
Workedexperimentationanalytics-attribution
What they did
Firebase assigns A/B variants; a user property in Amplitude marks baseline vs variant, and uplifts and statistical significance are computed from Amplitude cohorts. Mostly binary baseline-vs-variant tests because of limited volume (sometimes a few variants, e.g. paywall at screen 3 vs 5); adopt the winner and repeat. Focus on early-funnel changes because they reach significance fastest; with a 7-day trial most answers arrive within 8 days. In hindsight, should have prioritized bigger swings by estimating likely uplift first.
What happened
121 A/B tests run by a seven-person team.
“we use Firebase for A/B testing, but we run our tests through Amplitude, the data, cohorts, data analysis through Amplitude, basically with a user parameter on Amplitude for baseline versus variants, and then we look at uplifts and figure out statistical significance.”
“The earliest in the funnel, the easiest it is to get statistical significance quite fast.”
“Mostly just like a binary test, because we didn't have a ton of volume.”
Related topics
Get tactics like this every Monday
New tactics from the week's founder interviews, each linked to where it was said.
More from Opal
All 29 Opal tactics- Test how many app blocks free users get; settle where free users still love it3 free blocks gave free users a great experience while power users still pay
- License a stranger's viral video about phone overuse and run it as your adSeen by dozens of millions; still running 1-2 years later
- Hire a user who posted about you to run her own educational channel for you0 to ~700K followers and millions of views in a few months
- Drop the hard paywall and give the core app away for freePaying share of users fell 20% to 9%, but organic growth, 1M+ DAUs and revenue rose
- Sell the app to schools as a phone policy after students asked for itGrew from one school to many US high schools; front-page LA Times coverage
- Add focus streaks with an animated flame and gemstone rewards at milestonesBiggest win of the year for retention and growth; no numbers
More from this episode
- Opal: Judge everything by 8-day ad return; raise spend weekly while above 100%Near zero to $5M ARR in about a year, no separate marketing budget
- Opal: Ask new users where they heard about you, and split results by answerFounder calls it the most reliable attribution, beating paid tools
- Opal: Ship a new app version every Monday, without exceptionSet the rhythm behind 121 A/B tests
- Opal: Charge about $100 a year from the start, after launching at $69Credited with reaching 100% 8-day ad return quickly
- Opal: Pay a creator to run their existing video about your cause as an adTook over all campaigns; mission-led ads keep working
- Opal: Hire one person to make ads with creators who already love the appWorks better than the agencies tried before; no numbers
- Opal: Hide or shrink the close button on the onboarding paywallA winning test when optimizing for 8-day ad return
- Opal: Make the first paywall easier to skip so more people use the free appMore people activated; ARPU fell but stayed manageable, net positive
- Opal: Show the paywall right after a report on the years lost to phonesDownload-to-trial rose from 7% to 17% across iterations
- Opal: Show a trial timeline on the paywall: reminder on day 6, charge on day 7Dozens of challenger paywalls never beat it
- Opal: Set up a 9-to-5 weekday app block automatically during onboardingA huge win in tests; no numbers given
- Opal: Let new users choose how they want to block apps during setupFailed every time; one simple default won
- Opal: Try to bring people back after their first dayMost tests failed, except the offer to trial cancellers
- Opal: Give a free year for 5 invites and lifetime access for 2080% more users sent at least one invite
- Opal: Ask people's age and show a different setup or paywall by age
- Opal: When someone cancels their trial, send a 50% off offer by pushWorks quite well; the one re-engagement test that won
- Opal: Let users see their friends' screen time, with phone number sign-inSlight uplift in tests; some users invite friends
- Opal: Build for iPhone only and hold off on Android and MacReached $5M ARR without an Android app