Duolingo · B2C · language learning app
Run 3-6 month holdouts for changes whose effects show up late
New push lifts can turn negative after about 6 weeks
Workedexperimentationnotifications
What they did
Ran a 1-2 year ads holdout, but no longer recommends year-long holdouts (they get forgotten and polluted by bugs). Now runs 3-6 month holdouts or long experiments on things like leaderboards (10 weeks to climb from lowest to top league) and push notifications, where novelty lifts fade.
What happened
Found push notification wins that looked great early turned negative after about six weeks.
In their words
Sub Club by RevenueCat · Why Your Free Users Are Your Real Growth Engine – Cem Kansu, Duolingo CPO“But we do way more like three month, six month holdouts on things that we know have different short, or we assume will have short to long term effects that might be different.”
“But then if you wait out six weeks, sometimes it doesn't just come back to neutral, it actually turns negative because you're sending too many push notifications that turns people off.”
Related topics
Get tactics like this every Monday
New tactics from the week's founder interviews, each linked to where it was said.
More from Duolingo
All 32 Duolingo tactics- Cut the free trial from 14 days to 7More conversions overall and experiment results twice as fast
- Let trial users pick which day they get the reminder before being chargedTested; added friction but built trust and delayed cancel decisions. No numbers given
- Ask new users in onboarding if they want to learn with or without adsSignificantly higher uptake; one of the biggest wins of the year
- Change the paywall button from 'Try for free' to 'Try for $0.00'Meaningful conversion lift over 'Try for free'; no numbers given
- Give 3 free days of the paid tier, no card needed, even to current subscribersEffective for free-to-paid and for upgrading subscribers to the top tier, per guest
- Let people pick which day you remind them before their free trial endsLifted conversion; no numbers given
More from this episode
- Duolingo: Keep lessons free and sell learners' translations to news sitesSmall revenue; dropped by 2016 (price race and low translation quality)
- Duolingo: Charge $1-3 to fix a broken streakA good share of users paid; showed how much users value their streak
- Duolingo: Show one ad only after a lesson ends, and test it against retentionRetention mostly neutral, at most a slight hit, across ad tests
- Duolingo: Count ad complaints per million users and compare them to other complaintsMissing-course complaints were 3x ad complaints; ads launched
- Duolingo: Block gambling and sketchy ad types, even if it costs revenueAccepted lost ad revenue for better experience; no numbers
- Duolingo: Sell ads directly to brands and have app characters introduce themUsers like the format; slow to sell because it's custom
- Duolingo: Use the end-of-lesson ad slot to advertise your own paid planGuest says house ads beat programmatic ads; no numbers
- Duolingo: Sell a one-time ad-free month at $3, $5 or $10 to test demandUptake far above expectations; led to the subscription
- Duolingo: Launch a plan with no ads, offline use, and 'support our mission'About half of early buyers paid to support the app; started a run rate
- Duolingo: Offer a free trial even though the app is already freeOne of the biggest unlocks; guest had expected it to flop
- Duolingo: Tell users on the paywall you'll remind them on day 5 of a 7-day trialWon A/B tests by easing fear of subscription traps
- Duolingo: Change the buy button to say 'Try for $0.00'Beat 'Try for free' on both trial starts and trial-to-paid
- Duolingo: Give subscribers unlimited hearts so mistakes never stop their lessonsBig turning point: raised subscription conversion and retention
- Duolingo: Make paid features free when a test shows they earn littleTook a small revenue hit on purpose to grow free users
- Duolingo: Add new subjects inside the main app instead of launching new appsSeparate apps were 'hard mode'; in-app works better, but many users still don't know
- Duolingo: Offer live human tutors on demand in a higher-priced planFailed: users joined, panicked and left; later replaced by AI video call