Use a prediction model to show each user just the one plan they'll likely buy

David Barnard… it's led the way in this hybrid monetization, but having three different levels of plan and then y'all tested out a fourth level of plan, we'll talk about that toward the end, hopefully if we have time, but it does get super confusing. And then you have the boosts and the other in app purchases and things like that. So then what did you do to actually to solve that and to make it more accessible to folks and getting people to the right plan?

Shawn GongSo that was challenging, but luckily we had a brilliant machine learning team. So when I talked to the team members and then they told me, "Hey, we can try to build ML models and then use that to predict users willing to pay and then we can surface the best product they're most likely to buy." So that's our solution. Because based on the insight we talk about it's like people, customers don't need a lot of option. They need the right one.

Shawn GongThink about like Netflix, how many users are like, "Oh my God, I don't know what to watch." There's so many choices, they end up spending hours scrolling and then now they can't watch anything. That's why Netflix have something like top picks for you, right? Based on your previous behavior and your ratings, they predict what you're most likely to watch. So solve that problem. So very similar.

Shawn GongYeah, spot on. Yes. But obviously to reduce the risk, we couldn't just start it from all the paywalls because we have a lot of different paywall. And then the same times that's very expensive and it takes a lot of time and then effort to train the model, to test the model. So we start with something small. So we test with a couple features and then just start with not all the scope.

Shawn GongSo we can see, hey, based on this machine model to historically, a version, a control will be same as we used to show the paywall and the products. To the new one, this is dynamic. We will change basically when they show a user, let's say for David, and then our paywall will ask the ML model, "Hey, which paywall should we show?" And then the paywall will decide, okay, which product we should recommend to David. And then that's how we test the treatment and then we'll be able to measure the conversion and then the total revenue.

David BarnardAnd how did it do?

David BarnardWhat about counter metrics? Were there anything you were watching to make sure things didn't go south in retention or user experience, that people weren't happy with the plans that they were presented?

Shawn GongTinder has a really good process in place and then we are required to also measure our counter metrics. This is great because you cannot only focus on one area. For example, for this case, we cannot just only focus on the first time conversion and then only the revenue amount. So we have to measure, hey, let's say David, based on our model, maybe prior to that, he will buy a Plus. Now he bought Platinum. So we want to check, hey, is David going to come back? Is David going to buy Platinum again? Is he going to cancel?

Shawn GongSo we definitely measure those long-term success metrics to make sure this model really served correctly, because that's nothing that we wanted to continue to work on. Most product features, it's not like launch and done, that's it. We have to optimize user rate. So we wanted to learn, hey, we actually have different models. We build a different ones we want to compare, so we want to see how they perform between those models, same time, how they …