Duolingo · B2C · language learning app
Keep each test running 3-4 weeks, even if results look clear the next day
Unknownexperimentation
What they did
At Duolingo's scale, retention-team tests reach statistical significance on top-line metrics (DAU, D1 retention) the morning after launch, but experiments still run at least 3-4 weeks on average to rule out novelty effects. Teams with small user bases (e.g. Duolingo for Schools, a teacher dashboard) calculate required test length up front.
In their words
Sub Club by RevenueCat · Why You Should Test Everything and How To Do It — Osman Mansur, Duolingo“Generally, we make sure we run experiments for at least three or four weeks on average, just so we're ruling out novelty effects”
Related topics
Get tactics like this every Monday
New tactics from the week's founder interviews, each linked to where it was said.
More from Duolingo
All 61 Duolingo tactics- Cut the free trial from 14 days to 7More conversions overall and experiment results twice as fast
- Sell a one-time ad-free month at $3, $5 or $10 to test demandUptake far above expectations; led to the subscription
- Tell users on the paywall you'll remind them on day 5 of a 7-day trialWon A/B tests by easing fear of subscription traps
- Change the buy button to say 'Try for $0.00'Beat 'Try for free' on both trial starts and trial-to-paid
- Send tomorrow's reminder 23.5 hours after today's lesson, not at a time users pickBeat every other scheduling strategy tested, including user-chosen times
- Let trial users pick which day they get the reminder before being chargedTested; added friction but built trust and delayed cancel decisions. No numbers given
More from this episode
- Duolingo: Ship every change as an A/B test, even back-end tweaksCredited for ~50% yearly DAU growth a decade after launch; no test-level numbers
- Duolingo: Never run two tests on the same screen; email the team before and after each testTeams rarely step on each other; no numbers
- Duolingo: After a big launch, keep 1-2% of users on the old version for 3-4 months
- Duolingo: Have every employee use the app and post feedback in one Slack channelA key source of test ideas; no numbers
- Duolingo: Let an algorithm pick from 250 reminder messages, and A/B test adding new onesNotifications called a top retention driver; no numbers shared
- Duolingo: Send a calm daily reminder, then an urgent 10 PM 'save your streak' messageGuest calls distinct purposes 'really important'; no numbers
- Duolingo: Mix cheerful and passive-aggressive reminders in the mascot's voice over 7 daysA quirky day-4 message can win back users who ignored the first three
- Duolingo: Keep reminders to about three words for newer users; say more to loyal onesThree-word messages 'worked really well'; no numbers shared
- Duolingo: Put an emoji at the start of the notification title, not in the bodyTitle-leading emoji beat emoji in the body in tests
- Duolingo: Tell users to 'tap here to start a quick lesson' in remindersUnderperformed; read as an ad rather than the mascot talking
- Duolingo: Let users skip a day without losing their streak, up to two days'Amazing for retention'; later tweaks kept adding retention gains
- Duolingo: Let users buy back a lost streak, or get it free with the subscription
- Duolingo: Add a home screen widget where the mascot looks more worried until you practiceBeat expectations as a retention lever; no numbers shared
- Duolingo: After a finished lesson, show a card asking users to add the widgetSurprisingly many installs; widget DAU went 'hockey stick'
- Elevate: Send reminders as local notifications that users can't switch off