Why You Should Test Everything and How To Do It — Osman Mansur, Duolingo

Every change, down to back-end changes with no UX impact, is released as an A/B test to catch regressions elsewhere in the app. In-house experimentation platform built by a data infrastructure/experimentation team: engineers set up tests once a feature is built, and reports on new data are generated every morning; PMs check the experiments dashboard first thing each day. One retention team runs ~20 experiments at a time. Each test yields 5-6 learnings that are turned straight into new tests. Different experiments may run on the same surface on iOS and Android, then the winner is aligned across platforms.

Rule: no multiple tests on the same real estate at once; when unavoidable, analyze the intersection of the two experiments for interaction effects. When a product review council (CEO and others) approves a test, the PM emails the product org what is being tested and what was discussed; at launch decision, another email shares data and learnings, so other teams can spot impacts on their features or metrics. Teams also cross-analyze overlapping experiments.

At Duolingo's scale, retention-team tests reach statistical significance on top-line metrics (DAU, D1 retention) the morning after launch, but experiments still run at least 3-4 weeks on average to rule out novelty effects. Teams with small user bases (e.g. Duolingo for Schools, a teacher dashboard) calculate required test length up front.

For bigger features, after launch keep ~1-2% of users in control for about 3-4 months to measure long-term impact, especially for features (like social features) whose effect on DAU builds slowly via network effects. After fixing a big bug, may run a holdout that gives the bug back to ~1% of users to measure its impact. Each launched feature also gets a feature dashboard tracking engagement over time to spot novelty decay.

All 600+ employees use the app daily. A Slack channel (#feedback-product) lets anyone dump feedback on any part of the app; PMs watch it for insights and test ideas. Junior PMs are also empowered to own roadmap areas and ideate bottom-up.

Daily practice reminder sent every day; stops after 7 days of inactivity. ~250 copy templates are eligible, and a multi-armed bandit learns which to send to each user. Templates carry eligibility criteria (e.g. streak length) instead of feeding many user features into the model. PMs keep writing new templates; each batch is A/B tested: control = bandit with 250 templates, variant = bandit with 260 (10 new). Measured locally (click-through of the new ones) and globally (top-line metrics). Bandit tuning changes can be A/B tested the same way. CTR is also drilled down by UI language; weak languages (e.g. Romanian) go to the localization team.

Multiple notifications per day each get a distinct purpose and CTA. Practice reminder: clear CTA ('Hey, it's time for your daily practice ... study Spanish for 5 minutes'). For users on a streak who haven't practiced by ~10 PM, a 'streak saver' with a more alarming tone ('Last reminder', 'Hey, it's getting late'). Framed as a narrative so it feels like the mascot is having a conversation with the user.

Notifications are written in the voice of the mascot (Duo), mostly by PMs with content designers reviewing. Tone varies across the up-to-7-day schedule: encouraging ('Amazing job. You're on a 10-day streak. Come back and do Spanish.') and passive-aggressive ('You can practice Spanish if you want.'). The bandit learns which ones work over the schedule.

After an iOS update started truncating notification text after ~4-5 words, tested very short copy (titles under 30 characters, e.g. 'Got 3 minutes? Study Spanish now.'), especially for new or less engaged users on short streaks. For highly engaged users (e.g. 50-day streak), more complex copy referencing mechanics works too ('Hey, you're out of streak freezes. Don't lose your 50-day streak.'). Targeting is done via eligibility criteria on each template.

Tested emoji placement in push notifications. Emoji in the title, as the leading character, performed better than emoji in the body or at the end of the text. Hypothesis: a title emoji grabs attention; body emoji make the message look noisy.

Goal: get users to tap the notification itself, since it deep-links into a lesson and raises lesson completion. Tested copy templates explicitly instructing users to tap, e.g. 'Tap here to start a quick lesson.'

Losing a streak is a major churn point. Added a streak freeze item: equipped in advance, it keeps the streak if a day is missed. Bought with the in-app currency (gems) or earned as a reward (e.g. completing a daily quest). Iterated on gem pricing and reward drop rates, and raised the cap so users can equip two at a time. Guardrail: protect the 'sanctity of the streak' so it stays meaningful.

Streak repair restores a streak that was actually lost after all streak freezes were used. Historically sold as an in-app purchase and included as a perk of the Super subscription. Team works to keep streak features aligned and not too complicated.

iOS home screen widget showing current streak length and the mascot, who gets angrier/more worried through the day if the user hasn't practiced (Tamagotchi-like). Feels less spammy than a push because the user opted in, yet it is a constant reminder. Educational content in the widget was on the roadmap.

When a user finishes a session or lesson, show an in-app card promoting the home screen widget ('Hey, we have a widget. You should install it.'). Tested as an experiment. Separately optimize what the widget looks like.

Around 2015, iOS let users disable push notifications but not local notifications, so apps (Elevate, and likely Duolingo) sent reminders as local notifications; also cheaper than building push. Apple later unified notification controls, closing the loophole.