Let a bandit algorithm pick which upgrade ad each user sees after a lesson

Anmol TiwariAnd one of the first few things that we tried starting to optimize was, "Well, what is the right mix of promoting our own product versus serving other external ads?" And we saw good wins on it. And then that translates also into now, "Well, okay. Now if we know that you're likely to subscribe, we can probably predict how likely are you to subscribe to which plan."

Anmol Tiwari… each tier, but that's quite an inefficient way to operate because we probably have a lot of signals that we can leverage. "Did you ever tap on that plan in the past? Did you try to learn more about a video call when we showed you the video call as your next lesson?" So for us, we've seen actually a lot of great wins, and it's super important to keep collecting signals that can help you personalize not just basically all aspects of upsells. And I think we've just barely scratched the surface of it. And I think there's a lot of wins that we can go just getting all the way into, for example, we have many, many of creatives that run to promote our Super product as an ad at the end of a lesson. And we've had some wins in having a Bandit and trying to optimize which creative will work best for who, and we're continuing to see wins from that.

David BarnardYeah, that's awesome. And always fun to hear the app at Duolingo scale is still early in some of the optimizations that we all wish we had time and budget and teams to work on. It's really cool to hear the wins you have seen on personalization in the app.