Adopting growth experimentation in marketing culture

When you’ve heard about growth hacking, you’ve heard about its bread and butter: growth marketing experiments. This means testing newfound ideas that can spark growth opportunities next to day-to-day marketing activities.

In business, most new ideas and innovations don’t result in an increase in results, some maybe even counterproductive. But they are absolutely necessary for growth. Good and bad ideas are hard to distinguish at first, and that’s why it is critical to run growth experiments or A/B tests. Expertise in a changing landscape that is marketing is one thing. Proving strategies with data is another.

Company-wide curiosity

But what is the effect of implementing all these experiments? Companies like Netflix, Booking, LinkedIn, and IBM have their own growth team running many many A/B tests. They are setting up small scale experiments that reduce the risk of implementing them on their products. Failing is not bad. Look at venture capital companies investing in 20 companies, to just find that one unicorn startup to reimburse and grow their investment.

In the case of Booking, they run more than 1000 tests simultaneously and run by estimates about 25000 tests a year. In less than 2 decades, the company grew from a small Dutch startup to the world’s largest online accommodation booking platform.

At Booking, everyone in the company has the ability to propose and run an experiment. About 75% of its 1800 technology and product staffers actively use the company’s growth experimentation platform. Why? Because nothing stalls innovation more than the so-called ‘highest-paid person’s opinion’. Too many innovations would get blocked by management while it is data that needs to validate ideas. About 10% of all experiments generate positive results. This low success rate still translates into a significant number of successes with impact.

Digital giants like Facebook, Google and Microsoft found running many A/B tests to be a game-changer, a vital part of marketing and innovation. Microsoft’s Bing department, for instance, makes dozens of monthly improvements that collectively have boosted revenue by 10% to 25% a year.

Firms without digital roots, like FedEx and H&M, have also embraced digitalization and online testing. Identifying touch points, design choices, discounts, etc.

“In an increasingly digital world, if you don’t do large-scale experimentation, in the long term—and in many industries the short term—you’re dead,” Mark Okerstrom, the CEO of Expedia Group

Evidence over opinions

A gut feeling can steer ideas in the right direction but as a marketer, we need to keep into account that we are incredibly biased by our own experiences. Growth experiments can fuel healthy debates and become instrumental in decision-making.

Bookings democratized growth experimentation approach starts with filling out an electronic experimentation form and ticking the necessary boxes: the purpose, the beneficiaries, the modifications, etc. The experimentation team keeps a close eye on the experiments for the first few hours.

If an important metric starts performing worse, they immediately stop the experiment. In their case, they get enough visitors to quickly see their effect. If you’re a smaller startup, you’d probably test it longer until your test group is big enough. If something performs well, Booking scales the test for a longer period, eventually implementing its changes. Every test is centralized in a searchable repository of past experiments in full detail. For the company’s decision-makers, this approach was something to get used to but was proven to be a valid approach.

In 2015 at IBM, experimentation was not a core activity. Growth experiments were conducted, but they were costly and followed a rigid process. They only had one specialist who was the gatekeeper of all experiments. At the time, the company ran 97 tests a year. By 2018, IBM’s head of marketing surged the tests to a total of 2822. 5500 marketers worldwide would be able to conduct their own tests. IBM installed easy-to-use tools for experimentation, created a support center and introduced the growth experimentation framework. Testing units had to run 30 online experiments in 30 days. They would continue with the most scalable experiments and repeat the process.

For big companies, experimentation is crucial to stay alive, or better, to keep growing. A small startup or marketing team can implement this way of working to come to results more quickly. When things need to go fast, numbers need to be proven, growth needs to be established and proof for a B or C funding series needs to be provided, your marketing experiments show tangible evidence of success or failure.

Five pitfalls of experimentation and A/B testing

1️⃣ Forgetting you’re communicating with all kinds of humans.

In the case of booking, try not to lose personal touch with too much automation. As a consumer, I am a bit annoyed with too many things going on in my accommodation search. 10 people looking at this room, 7 booked it today, only 3 left, a special discount to book NOW. Maybe it’s because I am a marketer. Looking at the right metrics here is important. Maybe there are more clicks, maybe more interactions or short term sales. But keep a good overview of what you are doing when communicating with real humans and how it affects your brand.

That being said, there is nothing wrong with commercializing your product. In Belgium, we’re very reserved in sales. We hate putting ads on a creative piece of work. Somehow the creative idea needs to get the required funding. It’s hard to be a good salesperson. But you can’t have contempt for the sales and marketing process. B2C, you can create a quick feedback loop with your customers. B2B often has a longer process of testing, but it’s important to learn the truth about the effects of your experiments.

2️⃣ Growth experimentation ethics

When testing new growth experiments, companies must think carefully about what tests the user would consider unethical. It’s a hard question to answer. In 2012, Facebook ran a weeklong experiment on whether emotional states were contagious on the platform. Facebook curated a test where they showed a group of people more positive news or negative news, to see if it had an effect on their own postings. It involved nearly 690 000 randomly selected users where about half were unknowingly exposed to manipulated emotional expressions in their newsfeed. Hence the increasing outrage on public privacy.

3️⃣ Risks for decision-makers

A profiling software tool had the ability to hire people 50 times more efficiently. By curating the right profiles for the available jobs, resulting in more productive teams and fewer wrong hires, this innovative product seems like a no brainer. In 2 years’ time, the product would repay itself. Yet the middle manager, who was the decision-maker, could not make this decision. Firstly, there is the issue of budget. Secondly and more importantly, implementing the software would be something he would not want to risk his job for. The team would not be rewarded for their successes but would be punished if something went wrong. On an individual level, he could lose his job to propose a new way of working.

This comes back to the culture your company needs to implement. Risks should be allowed, or at least be talked about. Innovation doesn’t ‘just happen’.

4️⃣ Not looking beyond the average.

Imagine your new experiment results in all users spending an increased average of 5 EUR. At first glance, we would assume each user is spending an extra 5 EUR. However, that increase could also occur if a few users start spending a lot more and a lot of users stop using your product. Typical dashboards report the global average. It’s important to deep dive into the data. The devil is in the details.

5️⃣ Only having a short term focus

For growth experiments and A/B tests to be successful, they have to run long enough to prove results. Only focussing on the short term can shift results, as it takes time for users to grow used to changes. When Facebook changed its interface, everyone was frustrated. But it took time for us to get used to these changes. How long to run your growth experiment depends on the test? Run it until user behavior stabilizes. Both LinkedIn and Netflix monitor how users interact and results typically stabilize after about a week. Ensure that you’re measuring steady captured data of a new feature, as opposed to a short-term novelty effect.

To succeed at A/B testing, people have to commit to making decisions based on data. For most established companies, that requires transformational change.

A/B testing sticks

It’s a powerful way to gain insights on potential changes, on different customer segments and markets in the right hands. A/B testing has been around for a long time. At Pinterest, more than 65% of employees look weekly at the big data system summarizing all learnings and tests.

To make A/B testing or experimentation successful for your company, or clients, it will first need a change in culture: company politics and human decision. Everyone needs to embrace newfound confidence in online testing to scale growth experiments to a number for testing to be effective and result, ultimately, a growth in sales.