What 80M users taught Flo about A/B testing — Mike Charniatsou

Flo's user acquisition team noticed Meta ads with a narrative about enhancing your sex life gaining traction, performing well even into the existing generic funnel. Product dissected it as an unexplored user need: added a separate user goal 'enhance sex life' and enriched the web onboarding with questions tailored to it, so users clicking that ad land in a funnel that acknowledges the ad they came from and continues the narrative seamlessly.

Weekly syncs between Flo's marketing and product teams. Practice: every change product introduces and every change marketing makes to ad campaigns is shared, even if it seems not to matter, because you never know which insight the other team will value. Mike frames it as the fix for the usual tension (marketing says the funnel doesn't convert, product says marketing brings the wrong audience).

Flo re-analyzes shipped experiments 3, 6 and 12 months after rollout to see whether a reported win was a business win. Year-end exercise: add up the relative uplifts reported from tens of winning experiments (5-10% each) and ask whether the business grew by that much, i.e. whether the effects compounded.

Flo positions itself as a lifelong partner: core needs such as cycle tracking are free, and the team avoids intrusive paywalls and doesn't push to monetize on day one or day seven. It tracks how needs evolve across life stages (understanding periods, trying to get pregnant, following pregnancy, perimenopause) and, when it sees something of value beyond the core offering, approaches users with a tailored product offering at that time, converting free users 3, 6 or 12 months later.

Flo's web onboardings still offer free trials instead of asking users to pay upfront, which delays the real conversion event. To call experiments early, the team picks proxy events that correlate with future user value and are informative about outcomes 30-40 days out. Mike notes this needs lots of historical data; early-stage startups have to rely on gut feeling.

In Flo's perimenopause web onboarding funnel (users arriving from ads promising to explain their phase), added a step to determine the user's perimenopause phase. Test 1: almost 10% drop in conversion. Test 2, after design guesses: same drop. A post-mortem segmenting the audience found the questions were hard to answer and almost 75% of users got no phase, seeing an 'Oops, sorry, we were not able to pinpoint your perimenopause phase' screen. Test 3: question rewritten with the medical team, logic slightly changed, and users who still can't be classified just go to the next section with no failure message.

Common practice at Flo: after a win, run a blackout test that removes the new element. For the perimenopause phase step (a 23% win that seemed too good to be true), a fourth iteration removed the new section; it lost, proving the uplift. Blackouts are also used to test removing existing onboarding steps, since funnels tend to keep accumulating steps.

Product leaders from several companies told Flo a paywall pattern had given them a 10% conversion uplift. Flo implemented exactly the same thing, pixel to pixel. Mike's takeaway: borrow the idea, not the solution, and understand the underlying problem first.

Tested the Blinkist-style paywall (reassuring users they will be reminded before the trial ends, to make starting a free trial feel safe) at both Loóna and Flo.

Mentioned right after the Blinkist paywall failures at Loóna and Flo: 'the same thing with the trial toggle, it worked for some teams'. No details on the test.

Partner Mode is one of Flo's most requested features, used by more than 2 million pairs. Flo added it into the onboarding funnel expecting a conversion boost.

Removed the X (close) button from the paywall, making it fully hard. Initially 'crushing it' and about to roll out to everyone, until the team looked at refunds.

Loóna explored web acquisition and web onboardings when few apps did it (Noom was the trailblazer) and with no sleep/meditation player to copy, so the team worked out from scratch the onboarding's goal, length and friction points. Building the payment and funnel infrastructure was far more expensive than expected; the usual pitch was bypassing App Store and Google Play commissions.