Duolingo · B2C · language learning app
Never run two tests on the same screen; email the team before and after each test
Teams rarely step on each other; no numbers
Workedexperimentationteam-process
What they did
Rule: no multiple tests on the same real estate at once; when unavoidable, analyze the intersection of the two experiments for interaction effects. When a product review council (CEO and others) approves a test, the PM emails the product org what is being tested and what was discussed; at launch decision, another email shares data and learnings, so other teams can spot impacts on their features or metrics. Teams also cross-analyze overlapping experiments.
What happened
Guest says teams manage to avoid collisions pretty seamlessly.
In their words
Sub Club by RevenueCat · Why You Should Test Everything and How To Do It — Osman Mansur, Duolingo“the one principle we try to abide by is not running multiple tests on the same real estate at once. And if we do, then we will look at the intersection of those two experiments to make sure that there's no weird effects with those experiences being combined.”
“So whenever a product change is approved by our product review council of like the CEO and others, the PM will send an email out to the product org”
Related topics
Get tactics like this every Monday
New tactics from the week's founder interviews, each linked to where it was said.
More from Duolingo
All 61 Duolingo tactics- Cut the free trial from 14 days to 7More conversions overall and experiment results twice as fast
- Sell a one-time ad-free month at $3, $5 or $10 to test demandUptake far above expectations; led to the subscription
- Tell users on the paywall you'll remind them on day 5 of a 7-day trialWon A/B tests by easing fear of subscription traps
- Change the buy button to say 'Try for $0.00'Beat 'Try for free' on both trial starts and trial-to-paid
- Send tomorrow's reminder 23.5 hours after today's lesson, not at a time users pickBeat every other scheduling strategy tested, including user-chosen times
- Let trial users pick which day they get the reminder before being chargedTested; added friction but built trust and delayed cancel decisions. No numbers given
More from this episode
- Duolingo: Ship every change as an A/B test, even back-end tweaksCredited for ~50% yearly DAU growth a decade after launch; no test-level numbers
- Duolingo: Keep each test running 3-4 weeks, even if results look clear the next day
- Duolingo: After a big launch, keep 1-2% of users on the old version for 3-4 months
- Duolingo: Have every employee use the app and post feedback in one Slack channelA key source of test ideas; no numbers
- Duolingo: Let an algorithm pick from 250 reminder messages, and A/B test adding new onesNotifications called a top retention driver; no numbers shared
- Duolingo: Send a calm daily reminder, then an urgent 10 PM 'save your streak' messageGuest calls distinct purposes 'really important'; no numbers
- Duolingo: Mix cheerful and passive-aggressive reminders in the mascot's voice over 7 daysA quirky day-4 message can win back users who ignored the first three
- Duolingo: Keep reminders to about three words for newer users; say more to loyal onesThree-word messages 'worked really well'; no numbers shared
- Duolingo: Put an emoji at the start of the notification title, not in the bodyTitle-leading emoji beat emoji in the body in tests
- Duolingo: Tell users to 'tap here to start a quick lesson' in remindersUnderperformed; read as an ad rather than the mascot talking
- Duolingo: Let users skip a day without losing their streak, up to two days'Amazing for retention'; later tweaks kept adding retention gains
- Duolingo: Let users buy back a lost streak, or get it free with the subscription
- Duolingo: Add a home screen widget where the mascot looks more worried until you practiceBeat expectations as a retention lever; no numbers shared
- Duolingo: After a finished lesson, show a card asking users to add the widgetSurprisingly many installs; widget DAU went 'hockey stick'
- Elevate: Send reminders as local notifications that users can't switch off