Use each user's signals to pick between your own upgrade ads and paid ads

Anmol TiwariYeah. We've really ramped up, I think, our investments in personalization and using machine learning to try and optimize a whole host of things. As you know, of course, our freemium or free tier is with ads, and those ads are a mix of kind of ads that are promoting our subscription products and also third party ads; promoting websites, mobile, other mobile apps, et cetera.

Anmol TiwariAnd one of the first few things that we tried starting to optimize was, "Well, what is the right mix of promoting our own product versus serving other external ads?" And we saw good wins on it. And then that translates also into now, "Well, okay. Now if we know that you're likely to subscribe, we can probably predict how likely are you to subscribe to which plan."

Anmol TiwariWe used to have fixed ratios on how many ads we show that are for each tier, but that's quite an inefficient way to operate because we probably have a lot of signals that we can leverage. "Did you ever tap on that plan in the past? Did you try to learn more about a video call when we showed you the video call as your next lesson?" So for us, we've seen actually a lot of great wins, and it's super important to keep collecting signals that can help you personalize not just basically all aspects of upsells. And I think we've just barely scratched the surface of it. And I think there's a lot of wins that we can go just getting all the way into, for example, we have many, many of creatives that run to promote our Super product as an ad at the end of a lesson. And we've had some wins in having a Bandit and trying to optimize which creative will work best for who, and we're …

David BarnardYeah, that's awesome. And always fun to hear the app at Duolingo scale is still early in some of the optimizations that we all wish we had time and budget and teams to work on. It's really cool to hear the wins you have seen on personalization in the app.