Skip to main content
Field NotesSEO

Field Notes #109: SEO Epistemology

By February 3, 2025No Comments15 min read
Field Notes #109_ SEO epistemology

Last Updated on February 3, 2025

This essay is for the nerds who love decision theory, which I hope is the vast majority of marketing and revenue leaders. 

Particularly in 2025, where we’re dealing with platform and consumer behavior shifts, as well as a media environment that makes it more challenging to parse signals from noise, it’s more important than ever to understand how to make prudent and high expected value decisions.

Primer: The Hierarchy of Evidence

The hierarchy of evidence provides a useful framework for understanding the different types and reliability of knowledge:

  • Anecdotes and Personal Opinions: Foundational but often subjective. These are akin to “bro science” in fitness or case studies in SEO.
  • Observational Studies: The bulk of SEO research, where patterns emerge over time but lack rigorous control of variables.
  • Quasi-Experiments: When natural divisions (e.g., different strategies in two regions) allow some level of controlled analysis.
  • Randomized Controlled Trials (RCTs): The gold standard for isolating variables, though challenging in SEO’s complex systems.
  • Meta-Analyses: Aggregated insights across studies, a rarity in our field but invaluable when available.

Ronny Kohavi has done a wonderful job popularizing this in the fields of experimentation and product management, where, comparatively, it is somewhat easier to parse out causal signals than it is in the dynamic and often opaque arena of SEO or organic growth more broadly. 

There’s also the dimension of mechanistic or theoretical understandings, on which we base a lot of our knowledge in SEO. Things like Google patents or leaks inform our actions, or at the very least, they inform further observational studies and experiments to build more credibility.

Image Source

In health, we rely on mechanisms like muscle protein synthesis as a theoretical basis, but we confirm this with RCTs. Similarly, SEO mechanistic insights (think Google’s patents) should inform strategies but are best coupled with real-world testing.

Finally, we rely on first principles thinking, or in other words, critical thinking or common sense. This is actually more important in dynamic (i.e. constantly changing) and opaque (i.e. we don’t understand the true mechanics and there’s a delayed feedback loop) arenas like SEO. 

Fitness Analogies (…You Knew It Was Coming)

If your goal is to build muscle, some might advocate bodyweight exercises and a carnivore diet, while others swear by heavy lifting and plant-based nutrition.

The especially arcane may advise you to visualize and manifest your muscle growth, or undergo regular hypnosis or craniosacral therapy sessions to induce gainz. 

Observational studies suggest resistance training works (those who lift weights have bigger muscle on average than those who don’t, yeah?), but only controlled trials quantify effects like the superiority of 20 sets per week per muscle group over 10 and the isolated effects of certain nutrients like Vitamin D or hormones like testosterone in contributing to these outcomes.

In SEO, we face analogous debates: backlinks, content velocity, technical SEO, schema markup, AI-driven content creation, engagement signals, and click-through-rates on title tags.

But unlike fitness, our field is opaque, dynamic, and adversarial. Platforms like Google keep us guessing, competitors evolve, and algorithms (and consumer behavior) change. We have delayed feedback loops between inputs and outputs, and even the length of that delay is opaque (i.e. in some cases you may be able to quantify intervention effects in days whereas other cases require years, and it’s hard to tell up front when the effects will take place if at all). 

First Principles in an Uncertain World

Critical thinking, or reasoning from first principles, is vital. 

Consider the explosion of unedited AI content flooding SERPs. At first, it seems logical—more pages equal more traffic, a purely mathematical bit of common sense. It’s also cheaper to produce, so the expected value calculation is great. 

But second- and third-order effects reveal over-saturation, content devaluation, and an inevitable push toward new filtering signals that emerge as a result of the oversupply of content and the constrained demand for it (both from a platform and a consumer standpoint). 

Smart marketers will think longer than 25 seconds about the first order effect, and they will eventually land on and spend most of their time thinking about the third order effects, which will result in a more robust strategy that skates where the puck is going. 

Quick Personal Example:

Years ago, I mass-produced shallow affiliate content for my personal site. Traffic soared and I made a bunch of affiliate revenue, which coincided nicely with my expensive cross-country move to New York City—until the day it all went away. 

Of course, my gains were completely wiped out, and now I’m lucky if I make $200 a month from my affiliate programs, reinforcing the idea that long-term strategy beats short-term hacks.

I wrote an essay on this effect, which I called “chicken bones in the garbage disposal.”

A short term hack or convenience resulting in a lot of pain and expenses to clean up later. Not good for what is typically a longer term channel that delivers compounding results (not a short term arbitrage).

The first-principled thinker examines not just the initial benefit of mass production but also:

  • Second-order effects: If I can do this so easily, can’t everyone else? And won’t they? What happens when everyone adopts the same approach? Supply saturation, increased competition, and demand dilution.
  • Third-order effects: How will platforms or users adjust their behavior? Platform incentives dictate utility (as well as the ability to monetize) for users, thus inferring they will clean up and build new filtering signals for relevancy and credibility. Users may do the same, because time (and rather, attention) is a finite commodity. 

Counterfactuals and Mitigating Cognitive Biases

Many of our strongest signals in marketing come from case studies and observational studies.

We identify winners in various domains and then we apply analysis techniques, typically simple ones like linear regressions, to reverse engineer what made them successful. 

The smartest of these analysts understand that just because you have a trend line does not mean you have a causal factor (see: Anscombe’s quartet).

The even smarter ones understand that this type of analysis is rife with survivorship bias by the mere fact that we’re typically only analyzing winners. 

In this realization, we turn to counterfactuals. In essence, thought exercises, or even better, counter case studies, to identify if a variable truly correlates with an outcome. 

For example, several billionaires dropped out of college. You know who they are without me even telling you.

Does this mean that dropping out of college makes it more likely that you will become successful financially, even a billionaire? 

Of course not. There’s a much more complicated analysis that needs to occur where we identify those who dropped out but did not become billionaires as well as isolating confounding variables, and eventually, looking at the distribution of outcomes to see what patterns (if there even are any) of how college completion impacts financial outcomes. 

This image has an empty alt attribute; its file name is image-114.png

Image Source

We can consider running the same thought experiments when we see case studies or observational studies in SEO. For instance, I could run a quick analysis on brand inclusions in LLMs and run a regression on domain rating, number of backlinks, and Surround Sound SEO saturation. 

And let’s say I do identify a strong correlation with brands who appear for LLM product queries (“what are the best AI design tools”) and both the number of backlinks to the site as well as Surround Sound SEO saturation

This is a very interesting starting point, but it does not parse out confounding variables nor does it solve for survivorship bias. 

To start to dig deeper, I would need to run a counterfactual: are there brands with high Surround Sound SEO saturation (or referring domains) who do NOT appear favorably in LLMs? And are there confounding variables, like market share or brand equity, that would contribute causally to both links | surround sound as well as LLM inclusions?

Don’t Lose Your Shirt

In most cases, we’ll be operating from a good degree of uncertainty. In the above case, we have two strong correlate to LLM inclusions: number of links and Surround Sound SEO mentions. 

(If you’re unfamiliar with Surround Sound SEO, it’s the amount of real estate you occupy on the top 1-2 SERPs for product-related queries). 

My (quite strong) heuristic here is to take that path that will allow me to win even if I lose. And by that, I mean, I’m not going to venture out on a weak branch and risk falling to my death if that branch can’t hold my weight. 

So if I were to believe Surround Sound SEO mentions were to lead to LLM inclusions, and I were wrong, what’s the worst case scenario? 

Well, essentially, I’ll still drive a ton of BOFU leads through those placements. So I’ll still win even if it’s not a causal factor. 

But what if I choose to believe that number of backlinks is the key and I run a massive spam campaign or buy 10,000 links? 

Not only may I lose my money and have that be ineffective or net neutral, but I may incur a penalty and lose my existing traffic. 

So even if we don’t understand the exact variable weighting or causal pathways, we can still make smart decisions that balance evidence and common sense risk analysis. 

Don’t Be Fooled by Randomness

In Fooled by Randomness, Taleb highlights how noise often masquerades as insight:

“It takes a huge investment in introspection to learn that the thirty or more hours spent ‘studying’ the news last month neither had any predictive ability during your activities of that month nor did it impact your current knowledge of the world…

…Over a short time increment, one observes that variability of the portfolio, not the returns. In other words, one sees the variance, little else. I always remind myself that what one observes is at best a combination of variance and returns, not just returns (but my emotions do not care about what I tell myself).”

One can identify this behavior in those who are swayed too violently by algorithm updates and news, but also those who are addicted to dashboards and look at trendlines under a magnifying glass. 

Implement a technical SEO intervention today (Monday) and look at the data Thursday, and you see an increase in performance (on one of the 312 metrics you could use to track it). What happened – was it your intervention, a holiday or seasonality, a piece of news related to your industry or topic, algorithm updates, personalization in search behavior, competitive fluctuations, or mere randomness?

Some data – like investment portfolio performance, health and wellness, and SEO – requires a bit more time to fully “cook.” 

Sherlock Holmes and the Dog That Didn’t Bark

Sometimes, what doesn’t happen is as telling as what does. Sherlock Holmes famously identified the absence of a barking dog as a suggestion that the dog was familiar with the perpetrator of the crime.

What’s visible is obvious, and is therefore salient. What’s invisible may matter, but it is much more difficult to measure let alone identify.

For instance, I talked about the importance of a preeminent “why” behind your growth programs. No one can measure a mission statement. Yet I still believe it’s a critical factor in the long term success, particularly for those who want to innovate and be #1, not just follow a trail of best practices to a median level of success

Let’s say we wanted to understand what makes people anxious, and we have two variables we want to study: a supplement intervention or their meditation practice.

The supplement intervention is easier to measure because it’s a) an addition and b) it can be objectively measured from an input level (200mg is the same across the globe). 

Whereas the meditation practice is likely highly variable person to person. One could “meditate” for 10 minutes, but those 10 minutes could be rigorous and focused, or they could be what amounts to daydreaming about the perfect grilled cheese sandwich (simple and traditional, but with sharp cheddar cheese and a pear slice). 

Now imagine we wanted to measure something even more internal and invisible, like someone’s level of optimism. We could ask them how optimistic they feel on a scale of 1-10 (because we need ordinal variables to analyze the data), but does a 6 mean the same to everyone in all cultures, and are we accurately and honestly assessing our own levels of optimism in the first place?

That’s all to say, I think that Zapier’s programmatic product-led SEO won big due to mostly invisible factors (like timing, onboarding, product, and price point) and not because they structured their URLs or page titles in a certain way. In fact, I have a natural experiment or observational study that suggests this, because I worked at Workato, who also tried to do a similar play, but did not win nearly as big.

This makes it all the more tricky to identify what actually moves the needle in SEO success. For a quick realization of this, ask 10 different marketing leaders what the relative impact of “brand” is on the outcomes of an organic growth program. 

Experimentation and the Long Game

Ultimately, and I want to emphasize this very strongly, no decision will ever benefit from 100% certainty. 

Even if you have a systematic review of randomized controlled trials, it is still not 100% certain. That’s okay. We still need to move forward, boldly and ambitiously may I add, to drive business outcomes through organic. 

While it’s hard to fully summarize epistemology and decision theory in a short essay, my key heuristics here are:

  • Rely heavily on first principles and mechanistic understanding to inform best practices and hypotheses
  • Understand the relative risk level, and most importantly, the downside risk, of any given decision. 
  • Require higher degrees of evidence for those decisions with the highest downside risk. The goal is to never go to zero, and you should cap the riskiest decisions in your portfolio however possible (the barbell model is a great way to do this).
  • With all else, move fast and boldly, investing in actions that are unlikely to have large downside, and, if they have upside, the upside is obviously worth it. 
  • Experiment! Leave open an R&D basket for tinkering and innovation. 

Stefan Thomke puts it well in Experimentation Matters:

“Good leaders have enough humility to admit what they don’t know, identify the best options in an uncertain world, and experiment.”

To make better decisions:

  • Use the hierarchy of evidence as a guide.
  • Weigh risk vs. reward for each decision. Build an intuitive sense of expected value through refinement and documentation of decisions and outcomes. 
  • Apply counterfactuals and analysis of competing hypotheses to reduce cognitive biases 
  • Favor high-certainty evidence for high-risk moves.
  • Know that the world, and SEO, is uncertain. And act anyway.

Want more insights like this? Subscribe to our Field Notes.

Alex Birkett

Alex is a co-founder of Omniscient Digital. He loves experimentation, building things, and adventurous sports (scuba diving, skiing, and jiu jitsu primarily). He lives in Austin, Texas with his dog Biscuit.