Run 3-6 month holdouts for changes whose effects show up late

Cem KansuSo even though you have a year long holdout, you realize at some point you actually polluted it because you like introduced bugs there. Anyway, so very long holdouts. We are anti just because of a practical perspective.

Cem KansuBut we do way more like three month, six month holdouts on things that we know have different short, or we assume will have short to long term effects that might be different.

Cem KansuOne of those, for example, this is not exactly monetization, but anything that has to do with our leader boards, we try to run long period experiments or long holdouts because on the Duolingo leaderboard set, if you start from the lowest league, it takes you 10 weeks to get to the highest one.

Cem KansuIf we change the dynamics of how the leaderboard works, we might not see the actual result in two weeks. So we let these run for longer. Another one is push notification experiments where every, of course, new push notification you introduce has amazing returns because it has amazing novelty.

Cem KansuBut then if you wait out six weeks, sometimes it doesn't just come back to neutral, it actually turns negative because you're sending too many push notifications that turns people off. So some stuff that we assume, not just monetization, might have longer term adverse effects, we run long holdouts or just run a long experiment.

David BarnardThe way Duolingo test things is fast. We could talk hours just on that, but I did want to get back to the monetization journey. So at what point did it become clear that you needed to introduce a subscription?