BLOG

The Business Case for Running Growth Experiments

Peter Sharkey on 25th of Sep 2017

September 25, 2017

Growth Hacking Techniques

Why do we even need to run growth experiments? If we’re experienced marketers, shouldn’t we have some elevated skill, superior intuition, or hard-fought knowledge to nail everything the first time around?

The truth is, you can be the greatest marketer in the world but it still makes business sense to run experiments.

In fact, ideas are overrated. Running solely on intuition is a good way to both curb innovation and open yourself up to risk. Experimentation, on the other hand, allows you to learn far more than you would otherwise. In this article, we’ll focus on the business case for experimentation, why you should run experiments, and how to convince your teammates and execs to invest in optimization.

What’s the big deal about growth experiments?

Growth is never done, and our efforts are never optimal. Remember that there’s always room for improvement. As Treehouse put it:

“Don’t think your website is good as it is. Always test, always improve. It’s a slow process but worth doing. Our concrete test results show that a clear modern design improves sales. Especially in the sales process, trust building is very important.”

There are tons of case studies that could convince you of the power of optimization:

There are hundreds of these examples, and while they can certainly help make the case for running tests, one single A/B test win is not the point of growth experiments. Rather, it’s about discovery, mitigating risk, empowering innovation, and embodying a culture of experimentation and a growth mindset.

We’re all wrong a lot of the time

Let’s start off with a simple truth: it’s hard to predict the value or outcome of any given idea.

Don’t believe it? Try out this exercise: Look at a series of 10 A/B tests (just the creative—the control and the variation) and try to guess the winners. (Image Source)

It’s unlikely you’ll break 70% and if you averaged out a large enough sample size, it would probably rest right around 50% accuracy. A coin flip.

We all tell ourselves stories and we mislead ourselves into thinking we can predict outcomes, but it’s simply not true. Industry data ranges a bit, but most estimates put the A/B test success rate at around 10-20%.

Now, why do you think that you’re any different? Why do you think that your ideas will be right more often than anyone else’s? We’re all wrong a lot of the time. Even when we do correctly guess the winner to a single A/B test, our solution may not be the optimal one (have you tried testing all competing hypotheses?). Therefore, to mitigate risk—and more so, to turn being wrong from a human weakness into a competitive advantage—we must run experiments. It’s the best way discover what’s working and what’s not. Observational insights simply don’t do the trick. Controlled experiments are the gold standard of insight.

Celebrate discovery

Testing leads to failure, and failure leads to understanding.
Burt Rutan
American aerospace engineer

So you’re wrong? Good. As Andrew Anderson puts it:

“The key here is that it is not about their opinion or a single person. It is about the constant future discovery of what works and what works best, which means that every time someone was wrong, we found something better. Hammer that message home and celebrate the amazing returns you get from being wrong, and you will open up people’s eyes to just how little they really understand about what changes user behavior and makes your company money.”

Most companies are predicated on a culture that celebrates individuals with “good ideas” who are “right.” While this sounds like a perfectly logical way to manage a company, it’s actually a perfectly good way to mislead yourselves into thinking you’re productive while missing out on undervalued ideas and overvaluing ideas that may not work. This leads to HiPPOs making decisions in spite of the data and actual good ideas being buried and never seeing the light of day.

Experimentation breaks you free from those chains. As Erik Johnson wrote: (Image Source)

“Most have been trained to think they must have answers, have been rewarded when they were right, and punished when they were wrong. Sell testing as a relief to this stress: none of us have the answers, but all of us have great ideas. All of those ideas are equal and we can follow this process to solve them together. No more HiPPOs.”

Sometimes our ideas produce good results, and sometimes they don’t, but testing it is a safe way to find out what’s working. Michael Schrage railed against the cult of great ideas in his wonderful book, The Innovator’s Hypothesis. As he put it:

Results matter. Innovation is value obtained from actual use, not the promise of potential.
Michael Schrage

He explained further that the best cultures value experimentation, not big ideas or strategic planning:

“The best innovators typically run the best experiments. The best experiments are designed to generate the greatest opportunities for insight in the least amount of time. Amazon’s transcendentally successful recommendation engines, for example, weren’t born of strategic planning. They rapidly evolved from unauthorized experiments that looked at how shoppers might respond to unsolicited suggestions on-screen. Simple technical tests launched a profitable multi-billion dollar business platform.”

Let’s solve real business problems

Depending on the blogs you may read, you could have the wrong idea about growth or conversion optimization. If you think it’s about testing button colors, you’re missing the point. Experimentation lets us solve tough business problems in a way that mitigates risk and empowers innovation and creativity.

It’s about discovery. As Adam Avramescu from Optimizely wrote in CXL’s enterprise email course, you should reframe the questions you’re asking:

Instead of “What should I test?” ask, “What problems are my users having?”
Instead of “What are my competitors doing?” ask, “What makes my favorite online experiences so effective?”
Instead of “What’s the best color for this button?” ask, “Does this button even matter? Is there a more radical way to approach this landing page?”

When you pull together an inquisitive analytics team and tend to a creative and sharp growth team, you find solutions that your competitors would never have dreamed of. While they’re worrying about what everyone else is doing or what worked in the past, you’re asking “what if?” and “why?” and “what if we did x y and z?”

Asking business questions is a crucial skill for an analyst. Asking them in a way that may pose an answer and a decision/action is something a growth experiment can do very well.

For example, if we find in our data that those that use the search bar on the homepage convert at a much higher rate than those that don’t, we may want to explore why. We could, of course, tell ourselves stories and simply commit to action. In that case, we would simply make the search bar bigger and more emphasized and call it a day.

Instead, though, we could run an experiment, and see what’s actually going on. (Image Source)

In that case, we may run the experiment (make the search bar bigger/more emphasized) and find that nothing changes. Or maybe it converts worse. What happened?

For one, you just mitigated the risk of simply acting because you believed you saw something in the data. But two, you learned an incremental insight that will help you in your continuous process to improve the user experience.

It’s only through well-designed growth experiments that we can get trustworthy answers to our most important business questions.

How can growth experiments move the needle in terms of revenue and profit?

Conversion optimization is like retirement planning. The earlier you start, the better off you are.

As the saying goes, compound interest is the 8th wonder of the world. If you can get a 5% lift each month, that adds up. The best time to start is yesterday. The second best time is today.

The earlier you start and the more wins you can accumulate, the better off you are. (Image Source)

Let’s be clear here: this is a simplified chart. Experiment wins are perishable for many reasons, perhaps due to trends, seasonality, audience behavior changes, regression to the mean, or many other unexplainable phenomenon. But there’s no doubt that consistently iterating and getting small (or big) wins adds up.

As I mentioned, sometimes the wins aren’t 5%. Sometimes you get huge lifts with relatively little work.

Looking beyond lifts: why culture drives all else

We can easily quantify the value of an experimentation program, but it’s tough to quantify the impact it has on the company culture.

Apart from the simple fact that experimentation mitigates risk, and therefore encourages creativity and innovation, it’s simply more fun to work at a company that values experimentation. Here’s how Wyatt Jenkins put it in an HBR article:

“[In an experimentation culture] more of your ideas will see daylight in the form of tests, instead of being killed off on whiteboards and in presentations. When it’s easy to try your ideas, your team can stop speaking in the abstract about things that haven’t happened yet, and instead speak about results and next steps. Lastly, co-workers will be highly motivated, because they get to see their ideas live in the real world.”

One of the key features of innovative work cultures is that the companies reduce the cost of experimentation. They make it easy to conduct experiments and grant the resources required to launch and analyze them. Indeed, experimentation is encouraged and supported in these places.

“This is a new kind of management, where instead of viewing the boss’s role as the Caesar to make decisions, the boss’s role is to put in a system whereby junior people can run fast and cheap experiments, so that the ideas can prove themselves.” - Scott Cook, Founder of Intuit

We can clearly see that the most innovative companies in the world—Google, Netflix, Facebook, Amazon, and Airbnb—all value experimentation. Why shouldn’t you?

Conclusion

Growth and experimentation are hot topics right now, and for good reason. Businesses are starting to realize that “good” ideas from executives don’t have a 100% success rate (not even close), and that if we empower experimentation, we not only get clearer answers, but we get more ideas that rise to the top.

None of us are right all the time. It’s about finding answers together. Running experiments is not more work. We can both mitigate risk and encourage innovation and creativity.

Last but not least, it’s much more fun working in a culture that values experimentation. That’s something you’ll just have to trust me on. Growth experimentation produces clear business value while creating a more open and fun work environment. What’s holding you back?