GTM Sauce

Yousician · B2C · music learning app

Set calendar reminders to recheck price tests at 3 and 6 months

Catches price 'wins' that flip once renewals come in; no numbers

Unknownexperimentationpricing

What they did

After rolling out a winning experiment, put a recurring 'experiment recheck' in Google Calendar with the data analyst. Prioritize anything touching price or discounts (not layout/copy changes, unless the lift was surprisingly large or top-line growth isn't matching the stacked test wins). Recheck the cohort at three and six months depending on what is sold, or up to a year. Keep the cohort set up so the recheck is just a quick pull.

What happened

Guards against cases like a price increase that looked like a win but whose control group renewed better six months later.

In their words

Stop Celebrating Conversion Wins Before Checking Renewals – Sara Grana, Yousician

Play the episode

David BarnardAnd this gets back to what we started with was keeping a detailed history. Is that when you do run an experiment, do you set an alarm like three months from now?

Sara GranaYeah, my Google Calendar up there has this experiment recheck with my live data analyst. Yeah.

David BarnardGotcha. So you are looking back at specific intervals, and do you do that for... I guess you wouldn't do that for every test. What's the criteria for the things that you're most care to look back on?

Sara Grana… is price changes, I really care about. Because for example, if it's about, oh, the layout is different or the copy is different, then sure, maybe there's a different in renewal by why would it, right? So there's certain things that are probably not. So I don't know. You shouldn't be spending a lot of time. The thing is also, the more you do it, the easier it gets in a way. It's just like another part of the analysis, so to say. It's already set up that way. So you have already a cohort, it's like, okay, you pull back, okay, here. And sometimes it's even with some tools, it's even there that you can just recheck this test how it performed. So it doesn't need to be really time-consuming. But usually, everything that is with price or discounting or thing like this, I would always recheck after, depending on what we are selling three months, six months. And then if you have to wait a year, it's... Yeah.

David BarnardIt seems like too, maybe the priority would be to make sure you're checking back on the things that made the biggest move. So maybe there was a copy change on the paywall button that did a 25% lift, which sounds crazy, but it happens sometimes where you have these crazy lifts. And so maybe those kind of things would also fall into that bucket of anything that made a surprisingly big lift mark that one to go back on because-

Get tactics like this every Monday

New tactics from the week's founder interviews, each linked to where it was said.

More from this episode

Tags: cohort-recheck, price-test, renewals, calendar-reminder