Duolingo · B2C · language learning app
Ship every change as an A/B test, even back-end tweaks
Credited for ~50% yearly DAU growth a decade after launch; no test-level numbers
Workedexperimentationanalytics-attribution
What they did
Every change, down to back-end changes with no UX impact, is released as an A/B test to catch regressions elsewhere in the app. In-house experimentation platform built by a data infrastructure/experimentation team: engineers set up tests once a feature is built, and reports on new data are generated every morning; PMs check the experiments dashboard first thing each day. One retention team runs ~20 experiments at a time. Each test yields 5-6 learnings that are turned straight into new tests. Different experiments may run on the same surface on iOS and Android, then the winner is aligned across platforms.
What happened
Guest says the formula works; Duolingo still grows DAU ~50% year over year about 10 years after launch.
In their words
Sub Club by RevenueCat · Why You Should Test Everything and How To Do It — Osman Mansur, Duolingo“Even if it's just like a back-end change that doesn't change the UX experience at all, we'll just run it as an A/B test just to make sure there's no regressions where it impacts something else in the app.”
“every morning, every day, we generate reports on the new data that comes in from experiments, and as a PM, that's like my first thing I do when I wake up is check the experiments dashboard”
“So every time we run an A/B test, we will analyze the data, and from that data, we'll learn like five or six different things that we can then turn around immediately and test again.”
Related topics
Get tactics like this every Monday
New tactics from the week's founder interviews, each linked to where it was said.
More from Duolingo
All 61 Duolingo tactics- Cut the free trial from 14 days to 7More conversions overall and experiment results twice as fast
- Sell a one-time ad-free month at $3, $5 or $10 to test demandUptake far above expectations; led to the subscription
- Tell users on the paywall you'll remind them on day 5 of a 7-day trialWon A/B tests by easing fear of subscription traps
- Change the buy button to say 'Try for $0.00'Beat 'Try for free' on both trial starts and trial-to-paid
- Send tomorrow's reminder 23.5 hours after today's lesson, not at a time users pickBeat every other scheduling strategy tested, including user-chosen times
- Let trial users pick which day they get the reminder before being chargedTested; added friction but built trust and delayed cancel decisions. No numbers given
More from this episode
- Duolingo: Never run two tests on the same screen; email the team before and after each testTeams rarely step on each other; no numbers
- Duolingo: Keep each test running 3-4 weeks, even if results look clear the next day
- Duolingo: After a big launch, keep 1-2% of users on the old version for 3-4 months
- Duolingo: Have every employee use the app and post feedback in one Slack channelA key source of test ideas; no numbers
- Duolingo: Let an algorithm pick from 250 reminder messages, and A/B test adding new onesNotifications called a top retention driver; no numbers shared
- Duolingo: Send a calm daily reminder, then an urgent 10 PM 'save your streak' messageGuest calls distinct purposes 'really important'; no numbers
- Duolingo: Mix cheerful and passive-aggressive reminders in the mascot's voice over 7 daysA quirky day-4 message can win back users who ignored the first three
- Duolingo: Keep reminders to about three words for newer users; say more to loyal onesThree-word messages 'worked really well'; no numbers shared
- Duolingo: Put an emoji at the start of the notification title, not in the bodyTitle-leading emoji beat emoji in the body in tests
- Duolingo: Tell users to 'tap here to start a quick lesson' in remindersUnderperformed; read as an ad rather than the mascot talking
- Duolingo: Let users skip a day without losing their streak, up to two days'Amazing for retention'; later tweaks kept adding retention gains
- Duolingo: Let users buy back a lost streak, or get it free with the subscription
- Duolingo: Add a home screen widget where the mascot looks more worried until you practiceBeat expectations as a retention lever; no numbers shared
- Duolingo: After a finished lesson, show a card asking users to add the widgetSurprisingly many installs; widget DAU went 'hockey stick'
- Elevate: Send reminders as local notifications that users can't switch off