Duolingo · B2C · language learning app
Let an algorithm pick from 250 reminder messages, and A/B test adding new ones
Notifications called a top retention driver; no numbers shared
Workednotificationsexperimentationretention
What they did
Daily practice reminder sent every day; stops after 7 days of inactivity. ~250 copy templates are eligible, and a multi-armed bandit learns which to send to each user. Templates carry eligibility criteria (e.g. streak length) instead of feeding many user features into the model. PMs keep writing new templates; each batch is A/B tested: control = bandit with 250 templates, variant = bandit with 260 (10 new). Measured locally (click-through of the new ones) and globally (top-line metrics). Bandit tuning changes can be A/B tested the same way. CTR is also drilled down by UI language; weak languages (e.g. Romanian) go to the localization team.
What happened
Guest says notifications are a big driver of retention and user growth for many years.
In their words
Sub Club by RevenueCat · Why You Should Test Everything and How To Do It — Osman Mansur, Duolingo“and we have a machine-learning algorithm that optimizes the best ones to send to a user. What we do as PMs is iterate on those 250 copy templates, adding more, changing existing copy templates”
“Yeah, the A/B test would be, and control is the multi-armed bandit with 250 templates, and then the experiment is the multi-armed bandit with 260, um, adding the 10 new ones.”
“So we send you a reminder to practice every day, and if you if you've been inactive for seven days, then we'll stop sending it to you.”
“So the bandit will just optimize for the eligible templates that a given user is eligible for.”
Related topics
Get tactics like this every Monday
New tactics from the week's founder interviews, each linked to where it was said.
More from Duolingo
All 61 Duolingo tactics- Cut the free trial from 14 days to 7More conversions overall and experiment results twice as fast
- Sell a one-time ad-free month at $3, $5 or $10 to test demandUptake far above expectations; led to the subscription
- Tell users on the paywall you'll remind them on day 5 of a 7-day trialWon A/B tests by easing fear of subscription traps
- Change the buy button to say 'Try for $0.00'Beat 'Try for free' on both trial starts and trial-to-paid
- Send tomorrow's reminder 23.5 hours after today's lesson, not at a time users pickBeat every other scheduling strategy tested, including user-chosen times
- Let trial users pick which day they get the reminder before being chargedTested; added friction but built trust and delayed cancel decisions. No numbers given
More from this episode
- Duolingo: Ship every change as an A/B test, even back-end tweaksCredited for ~50% yearly DAU growth a decade after launch; no test-level numbers
- Duolingo: Never run two tests on the same screen; email the team before and after each testTeams rarely step on each other; no numbers
- Duolingo: Keep each test running 3-4 weeks, even if results look clear the next day
- Duolingo: After a big launch, keep 1-2% of users on the old version for 3-4 months
- Duolingo: Have every employee use the app and post feedback in one Slack channelA key source of test ideas; no numbers
- Duolingo: Send a calm daily reminder, then an urgent 10 PM 'save your streak' messageGuest calls distinct purposes 'really important'; no numbers
- Duolingo: Mix cheerful and passive-aggressive reminders in the mascot's voice over 7 daysA quirky day-4 message can win back users who ignored the first three
- Duolingo: Keep reminders to about three words for newer users; say more to loyal onesThree-word messages 'worked really well'; no numbers shared
- Duolingo: Put an emoji at the start of the notification title, not in the bodyTitle-leading emoji beat emoji in the body in tests
- Duolingo: Tell users to 'tap here to start a quick lesson' in remindersUnderperformed; read as an ad rather than the mascot talking
- Duolingo: Let users skip a day without losing their streak, up to two days'Amazing for retention'; later tweaks kept adding retention gains
- Duolingo: Let users buy back a lost streak, or get it free with the subscription
- Duolingo: Add a home screen widget where the mascot looks more worried until you practiceBeat expectations as a retention lever; no numbers shared
- Duolingo: After a finished lesson, show a card asking users to add the widgetSurprisingly many installs; widget DAU went 'hockey stick'
- Elevate: Send reminders as local notifications that users can't switch off