Lessons from 121 A/B Tests - Kenneth Schlenker, Opal

Paid ads (with a paywall and RevenueCat) from the first week on the App Store to get a steady ~100 downloads a day. Dropped other goals and focused on one metric: Day 8 ROAS, chosen because the 7-day free trial converts by day 8, so a change shipped Monday is readable 8 days later. Tested with small spend until there was a path to 100%, then scaled. Every week review last week's D8 ROAS: above 100% expand budgets, below 100% cut back until back at 100%. No growth target or marketing budget: revenue reinvested on the 8-day cycle. Started with simple spend this week vs revenue 8 days later; moved to install-week cohorts (D8 and D30) once late conversions skewed numbers.

An onboarding question asks where the user heard about Opal (e.g. Facebook, word of mouth). Cohorts are segmented by the answer and unblended ROAS is calculated per cohort. Attribution software is also in place, but the survey answer is trusted most.

Since the first week on the App Store, one app version released every Monday. Work is organized around the release cycle; changes range from tiny to large. Pairs with the 8-day ROAS read so each weekly change gets a result about 8 days later.

Annual subscription with a 7-day free trial. Launched at $69/year and later raised to about $99-100/year. High price is part of why paid ads paid back within 8 days.

Found a creator's organic video about the screen time problem that was not about Opal. Reached out because it matched Opal's mission; the creator turned out to use Opal. Made a deal to use it as an ad and tweaked it slightly. The ad talks about the mission and does not show the product. Second time doing this; now doing more mission-focused ads.

Tried fully outsourced creative agencies that make and run ads for a percentage of spend, and concept-only creative agencies. Moved to in-house: a single creative partnerships manager plus some contractors, finding creators in the space who care about the mission and ideally use and love the app, and producing ads with them. About 80% of paid marketing work is creative. Planning to hire a separate technical UA/campaign person.

Tested how easy the onboarding paywall is to dismiss; hiding the dismiss button or making it as discreet as possible won, during the phase optimized for Day 8 ROAS.

Part of the move to freemium. Made the original onboarding paywall easier to dismiss so more people start using the free version. Judged on two metrics together: retention (usage, paid or not) and first-year ARPU, which may fall but should stay above $2.

Tested many placements. Paywall goes early but not on the first screen: a few onboarding questions (e.g. current screen time), then a mini report projecting lifetime phone use (US average 22 years, based on smartphone at 12, life expectancy 77, 5h08m a day over waking hours), then 'the good news is we can save up to 3, 4, 5 years of your life', then the free-trial paywall. Moving the paywall two screens earlier alone can be worth +10% trial starts.

Main onboarding paywall is the Blinkist-style transparent paywall: start the free trial now, get a reminder in 6 days, pay the full subscription on day 7, which reduces anxiety about starting a trial. Dozens of alternative paywall designs tested against it, and testing continues.

During onboarding users pick their most distracting apps (e.g. Instagram, TikTok); Opal automatically creates a Monday-Friday 9:00-5:00 work-hours schedule that blocks them. Next day blocking starts without the user remembering to do anything. Editing or cancelling the schedule is made very easy.

Tried several ways of giving people a choice of how to disconnect during onboarding instead of the default work-hours block.

Ran many tests to re-engage people after the initial onboarding on the first day. No specifics given.

Existing program: invite 5 people and get a free year. Redesigned the referral screen to look better and added tiers: 5 invites = free year, 20 = lifetime subscription, 50 = special prize, shown as goals in a clear UI.

Onboarding asks the user's age and differentiates the onboarding, or even the paywall, based on the answer (e.g. professionals toward the paid plan, younger users toward referrals, as the cohost suggested and the guest confirmed).

RevenueCat tracks subscription status; when a trial is cancelled, OneSignal sends a push notification with an offer (currently 50% off, with the offer itself being A/B tested). An in-app modal repeats the offer when the user comes back. No email: onboarding uses phone number sign-in.

Replaced email collection with phone number sign-in, which supports a social feature where users see friends' screen time, as positive peer pressure and a support group. The feature was A/B tested.

Stayed iOS-only to figure out product and onboarding on one platform with limited resources, timed to Apple's new Screen Time API (iOS 16), which Opal was first to build on. Android, Mac and other devices deferred to a later expansion phase.

Firebase assigns A/B variants; a user property in Amplitude marks baseline vs variant, and uplifts and statistical significance are computed from Amplitude cohorts. Mostly binary baseline-vs-variant tests because of limited volume (sometimes a few variants, e.g. paywall at screen 3 vs 5); adopt the winner and repeat. Focus on early-funnel changes because they reach significance fastest; with a 7-day trial most answers arrive within 8 days. In hindsight, should have prioritized bigger swings by estimating likely uplift first.