Judge paywall tests on revenue after refunds, waiting 3+ weeks past trial start

Anada LakraSo I've kind of gotten a little disillusioned with these ... Again, almost get rich quick schemes, it feels like paywall optimization is becoming. And it's like at the end of the day, just make a better product for people, and that's how you drive more revenue.

David BarnardYeah. And to be clear, I mean, some of those hacks work and you're experimenting with them, you try stuff. And I'm just going to read it from my notes because I love the way you framed this. The ultimate judge, the ultimate metric you should be measuring against is not trial starts and some of those early funnel ... Not even initial conversions and stuff, but it's net revenue after refunds per exposed user. And then here's the kicker on a matured cohort. How did you decide on that specifically as your kind of gold standard metric to measure any experiment against?

Anada LakraIt came from the experience of running a lot of these and implementing some changes that looked really good. And in hindsight, again, at that time. Maybe we just didn't know that we had to wait and see all these other metrics mature. And also we didn't have instrumentation. It was really easy day one to see start trial rate and that's where you get the most data. That's where you're most likely to get a p-value that is statistically significant …

Anada Lakra… worse experience because those people who ask for refunds are generally pissed. They're not happy that they have to go through that process. You're also creating this negative user sentiment around the product that you don't want. And then of course, the matured cohort piece is important because people won't ask for the refund necessarily on day one. They'll ask for the refund after they've actually seen that the payment hit their bank account.

Anada LakraSo you have to think about, "Okay, trials start on this date, give it an extra week for them to convert, give it at least two extra weeks for the dust to settle from refunds." Again, it's very tempting to make a quick decision on trials, but you have to give it more time to see the entire story play out.

David BarnardYeah, it makes so much sense, but it's hard to do and it's hard to find the discipline. Like you were saying, the tooling, I mean revenue cap, we just in the last year or two finally expanded the experimentation window, where you can actually go back and look at three months and six months. Now it's over a year that you can look back. We've tried to make that really easy because it's one of those things in ... If you're running your own …