Check 7-day cancellations before keeping a higher price that won on early revenue

David BarnardDid you look back on some of those experiments and see the impact on retention? So did you knowingly sacrifice some long-term revenue for that quick return on ad spend?

Michal ParizekTypically where we look at retention or at least something like a proxy to retention, we use typically seven-day cancellation rate as a proxy for retention rate or for renewal rate, what it could look like. And we typically look at this proxy when we did price testing because I've seen data that, do you have any tests at different prices, particularly the higher prices? Usually you see higher cancellation rates and lower renewal rates at high prices. So it's quite reasonable. And I remember a couple of tests where we actually tested a different price and the price actually, mostly a high price, actually turned out to be the winner on the new revenue, but when we actually modeled having the new price for a year long, calculating a bit long-term revenue, and we saw that we would actually sacrifice in the long-term mainly because the renewal rates just dropped because of the proxy.

Michal ParizekSeven-day cancellation rate just was way higher than for the baseline price. So then we decided not to do that and kept the original price, and sacrifice a bit of less new revenue, but I think more new revenue in the long term.

David BarnardOne of the things I talked about in another one of these State of Subscription Apps podcast was how a lot of times experiment results don't stack. So you get a 10% win here, and a 20% win there, and a 15% win here, and then you look at it at the end and you actually haven't raised average revenue per user by the sum total of all those experiments. You're getting 10% here but losing 5% there, and getting 20% here but losing 10% there. What do you …