Many people think of testing and optimization as selecting one tactic and then seeing if it has a positive or negative impact.
But that’s just validating ideas, and not the best way to maximize outcomes or profit. In fact, that approach has a less than 10% success rate in terms of winning experiments.
So, what is the best way?
According to Andrew Anderson, head of website optimization at ZenBusiness, it’s moving from a validation mindset to a discovery and exploitation mindset.
An explore / exploit mindset increases your chances of success
Discovery is superior to validation.
That’s because you start with fewer assumptions, broadening your experimentation approach to include more than your preconceived notions, ideas, and biases. You start by learning what matters.
One way to think of it is the idea of a casino with many different slot machines with different payouts.
“Every tactic you do is going to have a different payout,” Andrew said. “Some are better, some are worse, but everything has an outcome.”
The question becomes, how do we put money in enough slot machines to figure out which one pays the most, while keeping resources back to exploit that information and put your eggs into the winning basket?
When you take that approach it changes how you do testing. Instead of thinking there’s one way to solve a problem and just testing that, you instead focus on what you’re trying to improve.
It might be website navigation or improving landing pages.
“We’re gonna look at all the different tactics that can do that, and then deconstruct what the best way to tackle that in different phases and different steps are,” Andrew said.
For websites, real estate trumps everything, he says. This includes the size, location, existence, or relative position of items. After that, it’s presentation, function, and then copy.
Learn from what doesn’t work, not just what does
There is another major benefit to the discovery approach.
When validating, you don’t get to learn from being wrong. With exploitation, you get to see what works instead and you can apply it to other areas.
Say you want to maximize your copy. Instead of just testing out using a social proof statement that “7,000 businesses used your services,” you’d experiment with a few different types of messaging.
“I might end up with a direct value statement versus social proof versus FOMO-based messaging and test out three or four executions of each of those,” Andrew said.
You have a much better chance of finding a successful strategy when you have a whole pool of tactics than when looking at only one.
With validation, the only thing you learn is whether your hypothesis is wrong or right. But with exploitation, if you’re wrong, you also gain a new idea.
“When I’m right, the best case scenario is I just get what I thought was gonna happen, but all the gain, everything I get to learn about what matters, what doesn’t and going different directions is when I go, ‘Hey, I think 7,000 works best, but instead this FOMO messaging worked much better,’” he said.
Then you get to take that lesson and apply it to other pieces you want to optimize.
This only happens if you’re testing in a way that results in multiple data points.
The importance of discipline-based testing
Regardless of what you’re testing, having a disciplined approach is necessary.
Being disciplined means not making assumptions that influence your testing and making rational decisions about the results.
“Discipline is instead of saying, ‘here’s this concept or this validation’ or something, it’s the discipline of always looking at what is the pool of feasible alternatives,” Andrew said.
It’s taking into account the cost of measuring those things and also acting in rational ways where you let the proof guide your decisions.
“Rational here simply meaning the math two is greater than one. And when two is greater than one, I always choose two,” he said. “ And then how do I apply that to everywhere else I do that?”
You see this in conversion rate optimization, but it can also be used in a content strategy.
“Instead of just putting all my eggs and making 40 articles of the same thing, can I build four different pools of 10 articles and see what performs best and then put my resource in that in the future?” he said.
It could also look like trying four or five different strategies for paid acquisition or affiliate groups and applying what works best.
No matter what you’re experimenting with, it means following the data, and not your guesses.
“It’s trying to find ways to break up any concept, to look at the assumptions and to have the discipline to say, even if I feel 100% confident this is the best way, until I have the data, I have to follow through with things,” he said.
Only measure what is relevant to your business
It’s easy to get caught up in different metrics. Sometimes you chase a metric that doesn’t actually get you where you want to go.
Before you start measuring something, pinpoint your goal, whether that’s leads or something else. Then, measure the pieces that actually have a direct connection to that.
“If my measure of content is that it needs to lead to leads, or it needs to lead to people entering another part of a long tail purchase cycle, as long as I can agree that there’s a direct piece and later measure how well that truth is, I can use that,” Andrew said.
With content it can get tricky if you’re testing SEO because it’s a longer term strategy. But it’s also easy to get distracted when using other channels to generate more leads.
“Another classic example of that is I add a module to a page, or I add an engagement thing, or I add an email capture or something like that, the success is not people interacting with it. The success is what its impact is to the bottom line,” Andrew said.
Don’t measure the wrong thing, because good numbers on traffic or engagement don’t mean anything if your goal is to get more leads.
“You may want less people to interact with it,” he said. “Just because you have a piece of copy and it gets more people from point A to point B, that may be good. You may actually want less people to pay attention to an area. But if you’re only measuring one small subset of things instead of the whole, you’re always going to get trapped on that.”
Tie experimentation to growth
Experimentation for growth relies on discovery of different channels and tactics that can be maximized.
“It could be discovering the efficiency of different channels, could be discovering different tactics within those channels,” Andrew said.
It’s important to take as much time as possible to figure out where you want to go and not to have any predetermined ideas. Don’t just go after one group with one tactic because then you’re just doing what you’ve always done.
“It might be, Hey, which of these five groups responds best?” he said. “Or which of these five content strategies work best?”
There will be cost overhead to do that, but ideally you’ll end up with two or three channels that work and you can then split those out for testing.
“From a growth point, I’m always trying to get to two or three primary channels that can scale and two or three that don’t and two or three tactics in each one,” he said. “The only way I’m gonna do that is from constantly exploring and measuring.”
With growth, you want to be creating decision points wherever you can so that you can maximize your resources around what works best.
The first part of this process is always discovery, then exploitation.
And if you stay disciplined in your approach, you’ll be able to find the ideas that actually work best and lean into them as opposed to coming in with a presumption that could limit your growth.
