Run separate paywall tests at once in US, Europe and Latin America every week or two

David BarnardThe last thing I wanted to touch on was experiment velocity. Y'all did a lot of tests. How do you keep up with that? How do you isolate variables? How do you think about testing velocity?

Michal ParizekOh, yeah, it was something super important. I actually think that it's the number one growth asset as everyone who actually wants to optimize paywalls and pricing. If you have good research, good prioritization, if velocity is, I think, it's even maybe more important because there's more shots you take. There's essentially a higher chance you'll win, there's a shorter feedback loop, better learning cycle. So it was a key. And yeah, I did a lot of experiments, and I think maybe the number one thing which allowed me to do that was having a third-party paywall platform allowed me to being autonomous and iterate fast, and really shorten the cycle of actually coming up with the experiment, developing it, launching it from months or weeks to essentially day. So it was so important that I think without it I would definitely wouldn't be able to get that amount of experiments live in that short time.

David BarnardThat makes a ton of sense and Mojo is a pretty big app, and y'all were doing a ton of user acquisition, so you did have a lot of users coming in to be able to do the test on, but did you have a particular testing velocity? Would you be launching it on a weekly cadence or even three days and then wait for the data to make a final decision, but while you're already kicked off another test?

Michal ParizekIt was more weekly or bi-weekly cadence. But what was also important to say that we mostly run experiments. We typically broke down the audience, the new users, to three buckets according to geo. It was like a US, or our US, Australia, Canada, English, US English bucket. Then there was the Europeans and then there was a Latin America. And we typically had these streams of tests for each, the specific geo bucket. And we've run tests in all of them free in parallel. And each test lasted usually one or two weeks because in those three geo, it was these three key geo segments we had, and we were able to get the statistical significant results usually in those one or two weeks for each of those buckets. So we were able to conduct experiments there.

David BarnardI think a great way to summarize this whole episode is you should be testing a lot. If you're not, you're leaving revenue on the table. Again, it's just such a clear example that you were able to increase average revenue per user by 60% in five months. That is such a huge win for the company ability to acquire new users and then over the long haul building that subscriber base over time. So such a fascinating lesson and thanks for sharing your …