Flo · B2C · period and ovulation tracker
Re-check test wins 3, 6 and 12 months after rollout
Finds that not all reported conversion wins became business wins; no numbers
Unknownexperimentation
What they did
Flo re-analyzes shipped experiments 3, 6 and 12 months after rollout to see whether a reported win was a business win. Year-end exercise: add up the relative uplifts reported from tens of winning experiments (5-10% each) and ask whether the business grew by that much, i.e. whether the effects compounded.
What happened
Mike says long-run analysis shows not all paywall conversion wins translate into business wins.
- Stage
- scale
In their words
Sub Club by RevenueCat · What 80M users taught Flo about A/B testing — Mike Charniatsou“you have a possibility to analyze all those things, you know, 3 months after the rollout, 6 months after the rollout, 12 months after the rollout, and figure out whether the win that you reported as a win is actually like a business win”
“check, did your business grow by the same percentage that you've reported over the course of the year? You know, did the effect compound over the time or not, you know?”
Related topics
Get tactics like this every Monday
New tactics from the week's founder interviews, each linked to where it was said.
More from Flo
All 21 Flo tactics- Test many prices in each country; keep the one with the most revenue per userHalf-price variants sometimes won: 3x retention and much higher revenue per user
- Set separate country prices for iPhone and Android usersAndroid's best price is ~15-20% below iPhone in US/UK; far bigger gap in Brazil
- Lower the local price so ads get enough sales to scale profitably in BrazilUnlocked profitable paid installs in Brazil; not possible at the higher price
- Give the full app free in 50+ countries where people can't pay20M+ free subscriptions given; no direct revenue or growth, but motivates staff and users
- Give the core app away free, then sell add-on modules as users' needs change60M+ monthly users; thesis behind a $200M raise at a $1B+ valuation
- Pitch the app as #1 in women's health instead of competing in all of healthBecame the first women's health unicorn: $200M+ raised at a $1B+ valuation
More from this episode
- Flo: Give your best-performing ad angle its own onboarding path that continues the ad's storyBecame a great revenue driver; no numbers given
- Flo: Hold a weekly sync where marketing and product share every change they makeSource of experiment ideas like the ad-angle onboarding path; no numbers
- Flo: Keep the core app free and offer paid extras when users' needs change40% of subscriptions start more than 6 months after install
- Flo: Judge free-trial tests on an early event that predicts who will payLets the team make early calls; no numbers given
- Flo: Add an onboarding question that tells users which stage they're in, never 'sorry, we couldn't tell'Lost ~10% twice; the evidence-based rewrite won 23%, confirmed by a blackout test
- Flo: Remove a step that just won to check the win was realRemoving the 23%-winning step lost, confirming the uplift
- Flo: Copy another app's winning paywall pixel for pixelDidn't work, despite a 10% uplift reported by the other teams
- Flo: Promise a reminder before the free trial ends on your paywallDidn't work at either of two apps
- Flo: Add a free-trial on/off switch to your paywallWorked for other teams, apparently not here; barely described
- Flo: Introduce your most popular feature during onboardingHurt conversion; biggest fail of the year
- BoldVoice: Remove the X button to make your paywall hardConversion looked great, but refunds spiked and it drained revenue
- Loóna: Sell through a web onboarding funnel instead of only the app storesPart of growth to almost $6M ARR; infrastructure proved costly