App Optimization Through Experimentation — Hannah Parvaz, Aperture

A/B tested where to place the iOS App Tracking Transparency prompt, expecting that showing it early would hurt sign-up or trial starts. Tested several placements (control vs multiple variants in Firebase/Mixpanel-style in-house tests). Showing it immediately on first launch had the least impact on everything, including conversion to paying and hard activation rate. Now rolled out across Aperture's apps.

For a new app, first run app install campaigns only (minimum ~$6K budget: ~$5 CPI x 50 conversions/week, about 7/day, enough for a few test variants plus one prospecting campaign) so the algorithm and team learn which messages and creatives work. Once a portfolio of working creatives exists, duplicate the campaign and optimize copies for registration, trial, and a custom event (e.g. trial plus content views) to see which performs. Post-install optimization on iOS needs ~120 installs/day per campaign, because each Meta campaign maps to ~10 SKAdNetwork campaign IDs that each need installs for postbacks. Raise budgets gradually and check the new level is sustainable.

Every creative test is one control plus variants that differ in one element. Message test: same design, four different messages vs control. Take the winning message and put it into four design concepts (SMS conversation, iMessage conversation, fake iOS Notes, UGC style, etc.) to see if it holds up. Iterate on the best format with message variations. Repeat continuously; move winners into always-on campaigns and touch those as little as possible.

Use an MMP (AppsFlyer, Adjust or Singular) as the source of truth for scaling decisions, because post-ATT Meta reporting under-counts. Also educate clients on SKAdNetwork and IDFA rules so they know when a platform's trial count is unreliable due to low install volume.

For completely free products (no need to monetize), run paid campaigns only on Android. Android has no SKAdNetwork-style minimum installs per day; you still need 50 conversions/week on Meta or 10/day on Google to optimize, but not 120+ installs/day. Also suggested as a place to learn when budget is low.

Defined a first meaningful action and a core action (listening to a story), then found the number of core actions after which users are 'hard activated': six stories on average. Gave every user 10 free stories (not 6, since the average is not the truth) while still offering the trial in onboarding. Then studied the step-by-step funnel: most listened to one story, so the focus became getting people to the second (autoplay, tooltips, rewards, light gamification).

Experimented with moving the trial offer later than the initial onboarding. Kept it in onboarding (not forced, closable) because far more people start trials there even if many cancel, then gave users further chances to start a trial later once they were warmer.

In a bar app where users got a free drink per visit, the average gap between visits was 21 days, but each user has their own cycle (e.g. 18 vs 35 days). A user is treated as starting to churn once they pass their own average interval, not the global one. Used to grow and track the hard-activated pool.

First check whether results are weak on a blended basis. Then walk every step against internal benchmarks: CPM, click-through rate, click-to-install (if weak, fix ads: saturation, new directions), true App Store installs vs first app opens, registration, trial start, trial-to-paid. If ads look fine, move into product. Consider retargeting installed-but-not-opened users. Sometimes pause ads until the problem is fixed.

Add up spend across all paid channels (Meta, TikTok, Google...), then divide by total installs, trials and purchases from every channel including organic. Track these blended costs over time to capture the untracked halo effect of ads (word of mouth, later searches).

Never launch five channels at once. Launch one, get baselines, reach good results, then introduce the next. Run incrementality tests on each channel at both the direct response and awareness level.

Spend a small amount on engagement campaigns boosting organic Instagram, Facebook and TikTok posts ($1-2 CPM) to reach the same kind of people who also see direct response ads ($7-12 CPM), so they see the brand several times. Evaluate awareness and direct response spend together and watch overall blended costs.

When Meta costs rose dramatically over summer 2023, the client routed money to Google, which remained more than 2x profitable, expecting to shift some back later. Long-term goal is an equilibrium with large shares on several channels rather than 95% in one.

For digital subscription apps: November 1-30 performance is poor because e-commerce bids up ads, so put more budget into brand awareness and boosted posts that month. Costs drop on December 1; from about December 15 (last shipping dates) to the end of January ('Q5') is a strong window. Use New Year messaging, run a sale (in-app via RevenueCat or on the web) and push gifting.

Competitors on the same expensive keywords set daily spend caps (around $10,000/day) that ran out by about noon. The app stopped spending in the morning and ramped up spend after noon.

Duolingo showed the ATT prompt after the user had completed a few lessons, so tracking focuses on users who stick past the first sessions (the audience worth modelling).