Sociaaal · B2C · AI-native app studio
Write down the reason behind every A/B test so AI agents can learn from it
Unknownexperimentationteam-process
What they did
Every A/B test gets a stated reason (hypothesis), result and learning in a backlog, never 'it feels better'. Learnings are generalized into strategies (e.g. 'remove onboarding friction when down-funnel conversion is strong') with the 5-7 concrete changes that achieved it, so an agent or new hire can find the equivalent in other apps.
In their words
The Superwall Podcast · Patrick Stuart-Constant & Pablo Sánchez - $16M AI CEOs: How to Make $1.3M/Month With AI Apps“When we define an A/B test, we try to give it a reason. And that's also even more important nowadays that we try to feed all that information to an agent.”
“If I tell the agent, "Here we have our backlog of A/B test. These were all the hypothesis, this was all the results, this were all the learnings," we give a system with important information.”
Related topics
Get tactics like this every Monday
New tactics from the week's founder interviews, each linked to where it was said.
More from this episode
- Sociaaal: Buy apps that went viral and are now declining, then grow them againBought 6-year-old Celebs mid-decline; now bigger than ever, ~$300K+/month
- Sociaaal: Run the same style of test ads for apps you might build or buyStrong ad results count as a key data point in buy decisions; no numbers
- Celebs: Drop the paywall after onboarding; show the lookalike result fast and ask laterMore revenue per user; later paywalls converted the users who used to drop
- Sociaaal: Rerun old A/B tests that lost once the rest of the app has changedOnboarding paywall removal lost several times, then won big
- Sociaaal: Run about 100 A/B tests a month, starting small and scaling the winnersPortfolio ARPU roughly 3x in 18 months, 2x in 12; about 1 in 10 tests wins
- Sociaaal: Make about 4,000 AI video ads a month, with people coming up with the ideasProfitable at 3x LTV/CAC; creative team tripled in 6 months to keep up
- Sociaaal: Keep testing brand-new ad ideas instead of endless versions of one winnerCredited with staying market leader in their niches; no numbers
- Sociaaal: Use AI to find TikToks from other industries to inspire your ads
- WiseSam: Ask ad visitors many questions, reflect answers back, then offer a $1 first weekFounders say the long quiz works very well; no numbers yet
- WiseSam: Give new users one-tap task buttons so the AI assistant learns about them fastUsers get more engaged once the agent knows them; still needs more guidance