Run 1,000+ A/B tests a year, each product manager owning one area

David BarnardHow does all this play out in testing? Because I know you and the team there at LinkedIn, there's a culture of testing, of wanting to try lots of things and huge team now and a lot of users to experiment on. So I'd love to start diving into testing, and at the highest level, how does that value-driven growth shape how you do the testing?

Ora LevitWithin the Premium team alone, prior to LinkedIn, it would be a lot higher, but within our Premium subscriptions, we run over a thousand A/B tests a year. We essentially test anything that we would launch into market, whether it's a new feature, any layout, a new way to talk about the value that we provide, it would all go through an A/B test process where we would evaluate what resonates better for our members. Ultimately, I don't believe in sitting in the office just ourselves and figuring out what it should be because we have millions of members. They know better what works for them.

Ora LevitAnd so the more we can minimize time internally and maximize time where we learn and go to market and get that signal, the better we are at serving our customers. And so that's how we go about things. We have on our team a very bottoms-up driven A/B testing culture. Each product manager owns a certain area or domain and they work towards growing that area. And so they run their tests and make sure that they reach the goals and the roadmap that they set and we try to go for forward momentum. And so the more we go and try and test, the faster we get to delivering more value to members.

Ora LevitI would definitely say there are some sacred things in sense of we wouldn't want to put an experience that we're not proud of in front of members or on our feed, right? We wouldn't want to put something that is disruptive, that we feel like members won't enjoy. And so those would be things that we often just not even discuss, not consider, not launch. And sometimes if the debate would come of, "It has a business opportunity," we would lean on the …

Jacob EitingHow do you even just manage a thousand tests per year because that's like three plus a day? Is there a single air traffic controller that's making sure these aren't conflicting or do you just send it and just you'll hear back if it did something bad between ... I'm talking to cross or maybe you have them all isolated, because you have so many users, maybe you can keep it all isolated. What is that process like as a growth product team?

Ora LevitYeah, we have a system that measures however experiment does. It does look to the bias experiment overlaps. We do a time sequence things. And so if we know that two things might clash and mix each other's reads in some way that we can't disentangle, we would stagger things. Sometimes our tech would also demand some type of sequencing. And so we do do those things, but ultimately, we operate across a large ecosystem that has many surfaces and many members and many experiments don't need that sequencing.

Jacob EitingRight. I guess that's the flip side benefit of the complexity of LinkedIn as a product, is you have all these different, somewhat isolated places you can test and it can be done independently. I'm just thinking that's like three tests a day. That's a lot.