List every reason a page fails before brainstorming fixes, and write each test down

Ethan EthierYeah, so I think one of the biggest shifts we've had over the last few years of experimentation is we're obsessed with the problem and not the solution. And what I mean by that is rather than going ahead and just simply testing out ideas for experiments, we take our time to really truly understand what actually is a problem we're trying to solve, and the way we do that is through three main steps.

Ethan EthierSo first we'll look at the data, we track everything on Amplitude, and we know where there's drop-offs on our web, on the app onboarding, and we'll look at what certain areas have the biggest drop-off, but also that we feel have the biggest area to improve. And once we have an idea of what we actually want to experiment on, for example, let's say it's a pricing page, then we'll go ahead and look at that pricing page and we'll list out as many problems as to why that pricing page isn't converting before anything else.

Ethan EthierSo maybe it's a lack of trust on that page, maybe the value proposition isn't clear. We'll list out as many problems as possible, and as a team, we'll decide on what we think the biggest problem is. And then only once we have that problem locked in, then we'll start listing out solutions. So if it's trust, then maybe we need testimonials, we need more credibility, and based off of the solution, then we'll just go and roll it and experiment.

Ethan EthierSo yeah, that's been the main process, and I think the biggest thing from all of that is making sure you document down everything just because it's easy to send a message to your engineer on Slack and say, "Hey, let's roll this out." But once you actually take the time to write things down, you understand the problem so much deeper. And not only that, but if an experiment fails, you can go back to the drawing board and understand what exactly you tested, why it failed, and how you can iterate on it.

Ethan EthierBecause there's been, probably 70% of the experiments that we run fail on the first go, and that doesn't mean it's a failure. It just means that maybe there's something wrong with the execution. Maybe we covered the wrong problem. So often we're iterating on these failures to uncover a winner. So that's been a really helpful process for us.

David Barnard… kind of things can move the needle. Some little things can be dramatically. Especially when it does come to CTA texts and things like that, the subtle aspects of how people are thinking about what action happens after they tap a button. I can see some of these really high leverage points being able to test more granular stuff. But yeah, it's like if you can test a big swing and try more dramatically new stuff, you just get clear results too.

Ethan EthierYeah, no, no, I agree. Not to say that a little text change won't make a difference. I think for us it's just been when we have a problem that we're trying to solve and we have these different hypotheses for why, we'll usually test it as a batch just because again, it also depends on how much traffic you have coming in. We won't have enough to do a bunch of mini tests, so we'd rather focus on a bigger swing. If that doesn't work, okay, maybe we'll strip things down a little bit, but that's usually how we've been approaching experimentation.

David BarnardYeah. And then because you, as you said earlier, have looked through the funnel and estimated the likely impact, you're also looking for those high leverage places anyways, places where people drop off the funnel, places where you feel like conversion should be dramatically higher than it is, and not testing as much the places where you're already seeing really good results.