Write down the reason behind every A/B test so AI agents can learn from it

Joseph Choi… that when you run an A/B test, you have to have a hypothesis for like the reason why you're running this test. And then if the test proves you right, then now you have like instead of a hypothesis, now you have like a proven, you know, insight that, okay, it seems like roughly like if we control for the other variables, like this thing that I thought was true is actually true and we should do like more of those types of things down the line.

Pablo SánchezThat is exactly that. Like you tag your decision. You know, when something is the conclusion of, "It feels better," or "I prefer it," that's not tangible. Like, there is a reason for everything, no? When we define an A/B test, we try to give it a reason. And that's also even more important nowadays that we try to feed all that information to an agent.

Pablo SánchezIf I tell an agent, "I have been winning this A/B test," means nothing. If I tell the agent, "Here we have our backlog of A/B test. These were all the hypothesis, this was all the results, this were all the learnings," we give a system with important information. We will be able to better build the next iterations. Because me, I have been in those A/B tests, if I get an app, I know what I have to do. But it's very hard to tell an agent or a new person coming into the team the day one what they have to do, unless you have everything perfectly tracked. They can learn from that.

Pablo SánchezReducing friction on the onboarding in Celebs was about removing the paywall. If what matters is removing friction, in another app, it's going to be something different. So if now our strategy is removing friction in onboarding, we have 5, 6, 7 things that we tried that made it happen, and then we have to find the equivalent in the other apps, but under the premise of: I want to remove friction because I'm converting very good down the funnel.