Many people think of testing and optimization as selecting one tactic and then seeing if it has a positive or negative impact. But that’s just validating ideas, and not the best way to maximize outcomes or profit.
In fact, that approach has a less than 10% success rate in terms of winning experiments. According to Andrew Anderson, Head of Website Optimization Programs at ZenBusiness, the better way to experiment is to move from a validation mindset to a discovery and exploitation mindset.
In this episode he explains what this tactic looks like and how it can be applied to websites, content, and growth. Andrew has been described as “The Most Passionate Optimizer in the World”.
He also touches on how and why he got into writing and blogging, and how he constructs articles which is completely different from the typical SEO model. It’s a really fascinating discussion. We hope you enjoy.
Show Topics
- How an exploitation mindset increases your chances of success
- Learn from what doesn’t work, not just what does
- The importance of discipline-based testing
- Measure what’s relevant to your business
- Tie experimentation to growth
Show Links
- Check out ZenBusiness
- Follow Andrew Anderson on LinkedIn or Twitter
- Connect with Omniscient Digital on LinkedIn or Twitter
Listen to the podcast
Watch the video
Key Takeaways
03:30 – The downside of validation mindset
A validation mindset limits your chances of maximizing outcomes because it’s only a binary: positive or negative.
“For a lot of people, testing or optimization is really just a way for some different group to go in. I have this one magical solution or this other site is doing this or this. I heard about this one tactic. Let’s go try this out and see what the impact is. And so from that point, you’ve already decided what path you’re going to go, how are you going to tackle something, and you just want to see if it’s positive or negative. And that’s where you’re just validating ideas. And the problem with this is that that’s how we’re trained our entire lives. You’ve gone through a school system. You’re either right or wrong, and you get a gold star or you don’t. That’s not the best way to maximize outcomes or maximize profit, if you’re looking at it from an economic standpoint. Really what you’re doing is limiting your chances to a binary of this was better or worse. And if you look at industry averages, it comes out to slightly under 10% success rate for most people, in terms of winning experiments, ignoring all the problems with how most people do data.”
4:34 – An exploitation mindset increases your chances of success
In website optimization as well as other things in life, an exploitation mindset exponentially increases your chances of success because you’re trying several different things.
“Exploitation is very much looking at this from using bandit theory, for those who don’t know. And that’s really where you talk about you enter a casino and there’s all these different slot machines and every slot machine has a different payout. That’s the same thing that you do in real life. Like every tactic you do is going to have a different payout. Some are better, some are worse, but everything has an outcome. And so how do we put money in enough slot machines to figure out which one pays the most while keeping resources back to exploit that information, to put the most in the winning? And so what that does is really changes how you do stuff. Instead of looking at this is the one way I think I can solve this, you can start with that, but really what you’re saying is we’re going to focus on improving our navigation or improving our landing pages. And we’re gonna look at all the different tactics that can do that, and then deconstruct what the best way to tackle that in different phases and different steps are.”
08:57 – Learn from what doesn’t work, not just what does
When validating, you don’t get to learn from being wrong. With exploitation, you get to see what works instead and you can apply it to other areas.
“Let’s say we’re even going to do the copy. Let’s take that concept of social proof. What other types of concepts could I have and what other executions of that? I might end up with a direct value statement versus social proof versus FOMO based messaging and test out three or four executions of each of those. Now I still could be right. There’s still the possibility of that one message trumps it. But if you start looking at the permutations, all those you’re less than 1% chance, where the pool, the whole is going to have a much higher chance. More importantly, when I’m wrong, I also get to learn something. See when I’m right, the best case scenario is I just get what I thought was gonna happen, but all the gain, everything I get to learn about what matters, what doesn’t and going different directions is when I go, Hey, I think 7,000 works best, but instead this FOMO messaging worked much better. Okay. Where else can I apply that? What other ways can I tackle that? Where does this fit into other parts of the user flow? And so those types of statements you can only do if you’re testing in a way that not only challenges that concept, but also tests multiple executions of those things to get multiple data points.”
12:44 – The importance of discipline-based testing
Having a disciplined approach means you’re always looking at the range of alternatives for how you do something and choosing the one with the greatest impact, instead of making assumptions.
“Discipline is instead of saying, here’s this concept or this validation or something, it’s the discipline of always looking at what is the pool of feasible alternatives? What is the cost of measuring those? How can I act in a rational way, which is by far the hardest step? And again, rational here simply meaning the math two is greater than one. And when two is greater than one, I always choose two. And then how do I apply that to everywhere else I do that? And so that happens obviously in conversion rate optimization or optimization of any kind, but the same thing could be from a content strategy. Instead of just putting all my eggs and making 40 articles of the same thing, can I build four different pools of 10 articles and see what performs best and then put my resource in that in the future? Or can I look at four or five different strategies for my paid acquisition side or four or five possibilities for affiliate groups and different ways to tackle that and find out what applies and what works best? It’s trying to find ways to break up any concept to look at the assumptions, and to have the discipline to say, even if I feel 100% confident this is the best way, until I have the data, I have to follow through with things.”
16:34 – Measure what’s relevant to your business
Pinpoint your goal, then measure the pieces that actually have a direct connection to that.
“If my measure of content is that it needs to lead to leads, or it needs to lead to people entering another part of a long tail purchase cycle, as long as I can agree that there’s a direct piece and later measure how well that truth is, I can use that. But for most SEO you really are talking longer term time strategies. So you can say after two months, if these things or after three months, I’m going to do that. And that’s why you don’t just all do just SEO. You’re going to do some SEO. You’re going to do some extra content. You gotta do a little bit of everything because all these things work on different time schedules. And so you need to know when you apply resource or not. If I know I’m gonna create a whole bunch of content and wait for two months, well, then I can plan my resources. If I’m just going to keep creating content, what’s the point?”
18:37 – Don’t get distracted by other measures
People often get distracted by positive metrics that seem like they’re related to your goal, but actually aren’t. Keep your eye on the big picture and what’s affecting the bottom line.
“It doesn’t matter if I get 1,000 new page views if I’m only getting one new lead, because the lead’s what I make money off of. And so being able to say, it’s not what I think I want to happen, or I think this is how it’s going to happen, but what’s going to happen. So another classic example of that is I add a module to a page, or I add an engagement thing, or I add an email capture or something like that. The success is not people interacting with it. The success is what its impact is to the bottom line. You may want less people to interact with it. That same thing is true when we go back to the copy piece. Just because you have a piece of copy and it gets more people from point A to point B, that may be good. You may actually want less people paying attention to an area. But if you’re only measuring one small subset of things instead of the whole, you’re always going to get trapped on that.”
29:49 – CRO requires creating decision points
Experimentation is how you find out what users need. By creating decision points you can figure out what’s most effective and what’s feasible, while keeping costs low.
“Product is really the discipline to look at three things and balance them. It’s what users want, what users need, and what the business needs. And companies get out of control, get out of success, when they focus on only one or two of those and they don’t balance those three. So to me, what users want, I can get very easy. But what users need, that’s where experimentation comes in. That’s where I can actually measure those things and do that. And the more ways I can create decision points, where I can discover and go in different directions and feature things and to figure out what’s best, that helps shape that entire strategy and helps me view it as part of that holistic whole. I can’t be successful without that. I could interview a million users and they could all tell me that they like red over blue, but it does not mean that red is better than blue. It just tells me that they say that red is more than blue. I mean, Malcolm Gladwell talks about this all the time. The need part there is 100% down to how do I discover what they need. How do I figure out, give them the option to see what they need, and then do that.”
33:20 – Tie experimentation to growth
Experimentation for growth relies on discovery of different channels and tactics that can be maximized.
“It could be discovering different channels. It could be discovering the efficiency of different channels, could be discovering different tactics within those channels. The more steps I can take before that and figure out where I want to go and not be predetermined, like I’m going to do this tactic and I’m gonna go after this group. It might be, Hey, which of these five groups responds best? Or which of these five content strategies work best? I know there’s some cost overhead on that, but the worst case, I end up where I was. Best case, I end up with two or three that work and I can split those out. So from a growth point, I’m always trying to get to two or three primary channels that can scale, and two or three that don’t ,and two or three tactics in each one. Well, only way I’m gonna do that is from constantly exploring and measuring. If I just have one or two and one or two tactics, or I’m just doing the same thing I did before, it either succeeds or fails. It’s the same validation problem as before. And so I’m creating those decision points and trying to maximize my resources around that, as opposed to trying to plan my resources to this long tail strategy. In our bandit theory, the first part of the coin you spend is always on discovery. It’s not on the exploitation.”
35:36 – Testing for growth versus website optimization
For website optimization you want to have 10-11 different channels that you’re testing, whereas with growth you want to have three or four.
“If I’m looking at growth channels, it comes down to what’s the minimum viable cost to get there and how many ways can I split that up? So I may end up with only three or four, which is very suboptimal to 10 or 11 on a website. But still the more things I can compare, the higher chance of the outcome I get. And so the first question I have is what resources do I have? And what’s the absolute minimum viable I can get for each channel and test that out. It may be, I do two or three simple ads on a different channel or a simple thing before I even hire a contractor and spend money. And just see what the initial reaction is. I don’t need it to be perfect, but I need to see if it’s going to be within say 80% as a heuristic of successful. Because then I know I can optimize it and grow it. But if it’s coming in at 30%, every dollar I spend further on that to keep building out, to even have what could be a minimal viable, it’s a waste of time.”
44:31 – Any team can have a discovery mindset
Andrew used to think that every company needed an optimization team, but he’s realized that the team doesn’t matter. What’s important is the discipline and discovery mindset.
“I think the biggest thing I’ve changed is I used to be a believer that no matter what size organization, you need an optimization team and optimization discipline I’ve since changed that to kind of what I described before, which is you need to have the discovery mindset in everything, the team that does that is irrelevant. By the way, where they sit is irrelevant. But if you don’t have the discipline, you don’t have the discovery mindset and try to build that into everything you do, that’s where your failure is. So I used to call the tactic itself the problem instead of the actual core thing that’s causing the symptoms. And that’s really where I’ve shifted.”
Connect with Omniscient Digital on social:
Twitter: @beomniscient
Linkedin: Be Omniscient
Listen to more episodes of The Long Game podcast here: https://beomniscient.com/podcast/
