Duolingo · B2C · language learning app
After a big launch, keep 1-2% of users on the old version for 3-4 months
Unknownexperimentationanalytics-attribution
What they did
For bigger features, after launch keep ~1-2% of users in control for about 3-4 months to measure long-term impact, especially for features (like social features) whose effect on DAU builds slowly via network effects. After fixing a big bug, may run a holdout that gives the bug back to ~1% of users to measure its impact. Each launched feature also gets a feature dashboard tracking engagement over time to spot novelty decay.
In their words
Sub Club by RevenueCat · Why You Should Test Everything and How To Do It — Osman Mansur, Duolingo“So basically, after we launch an experiment, we'll keep maybe 1% or 2% of users in the control group and leave that running for like 3 or 4 months, something like that.”
“If it if it's a huge bug fix, then we will like patch it, and then maybe run a holdout experiment, like giving the bug back to like 1% of users,”
“So for every feature that we we build, we'll have a feature dashboard that tracks some of the more ad hoc metrics, feature engagement-level things that we might be interested in”
Related topics
Get tactics like this every Monday
New tactics from the week's founder interviews, each linked to where it was said.
More from Duolingo
All 61 Duolingo tactics- Cut the free trial from 14 days to 7More conversions overall and experiment results twice as fast
- Sell a one-time ad-free month at $3, $5 or $10 to test demandUptake far above expectations; led to the subscription
- Tell users on the paywall you'll remind them on day 5 of a 7-day trialWon A/B tests by easing fear of subscription traps
- Change the buy button to say 'Try for $0.00'Beat 'Try for free' on both trial starts and trial-to-paid
- Send tomorrow's reminder 23.5 hours after today's lesson, not at a time users pickBeat every other scheduling strategy tested, including user-chosen times
- Let trial users pick which day they get the reminder before being chargedTested; added friction but built trust and delayed cancel decisions. No numbers given
More from this episode
- Duolingo: Ship every change as an A/B test, even back-end tweaksCredited for ~50% yearly DAU growth a decade after launch; no test-level numbers
- Duolingo: Never run two tests on the same screen; email the team before and after each testTeams rarely step on each other; no numbers
- Duolingo: Keep each test running 3-4 weeks, even if results look clear the next day
- Duolingo: Have every employee use the app and post feedback in one Slack channelA key source of test ideas; no numbers
- Duolingo: Let an algorithm pick from 250 reminder messages, and A/B test adding new onesNotifications called a top retention driver; no numbers shared
- Duolingo: Send a calm daily reminder, then an urgent 10 PM 'save your streak' messageGuest calls distinct purposes 'really important'; no numbers
- Duolingo: Mix cheerful and passive-aggressive reminders in the mascot's voice over 7 daysA quirky day-4 message can win back users who ignored the first three
- Duolingo: Keep reminders to about three words for newer users; say more to loyal onesThree-word messages 'worked really well'; no numbers shared
- Duolingo: Put an emoji at the start of the notification title, not in the bodyTitle-leading emoji beat emoji in the body in tests
- Duolingo: Tell users to 'tap here to start a quick lesson' in remindersUnderperformed; read as an ad rather than the mascot talking
- Duolingo: Let users skip a day without losing their streak, up to two days'Amazing for retention'; later tweaks kept adding retention gains
- Duolingo: Let users buy back a lost streak, or get it free with the subscription
- Duolingo: Add a home screen widget where the mascot looks more worried until you practiceBeat expectations as a retention lever; no numbers shared
- Duolingo: After a finished lesson, show a card asking users to add the widgetSurprisingly many installs; widget DAU went 'hockey stick'
- Elevate: Send reminders as local notifications that users can't switch off