In the early twentieth century, British administrators in colonial India attempted to solve a practical problem with an apparently rational solution.
Concerned about the prevalence of venomous cobras in Delhi, they introduced a bounty: payment for every dead snake turned in. The program worked, at least according to the metric. Cobra carcasses piled up. Reports were favorable. The policy appeared successful.
Only later did the administrators realize what they had actually incentivized.
Enterprising locals began breeding cobras specifically to collect the reward. When the government discovered the scheme and abruptly ended the bounty, the breeding operations were no longer profitable. The snakes were released. The cobra population increased beyond its original level.
The episode, though I’m unsure of its actual veracity, represents the idea that metrics influence behavior (and that many metrics can be gamed, producing perverse incentives).
That’s what this essay is about: what you track is not a neutral representation of reality, but rather a strategic choice that influences incentives, behaviors, outcomes, and the feedback loop that fuels further tracking.
Ultimate Guides: Where Did They Come From (Where Did They Go)?
Until very recently, SaaS SEOs and content marketers converged on a similar tactic: publishing ultimate guides.
- The Ultimate Guide to SaaS
- The Ultimate Guide to SaaS SEO
- The Ultimate Guide to Ultimate Guides
Why?
In a phrase, the Traffic Trap.
Marketers measured traffic, and to get consistent traffic growth, they aligned their content marketing with search keywords.
Keyword research consisted of analyzing competitor content and keywords related to your head terms, and then filtering and prioritizing a list of keywords based on their search. The terms with the highest search volume tended to be 1-2 word phrases that related to a concept, things like:
- Content marketing (989K global search volume)
- Cybersecurity (425K global search volume)
- Conversion Rate Optimization (43K global search volume)
One infers a fairly broad user intent for these keywords, something informational that defines a term and gives you a conceptual map of the topic. This is where you get titles like:
- What is Content Marketing (The Ultimate Guide)
- The Ultimate Guide to Cybersecurity
- Conversion Rate Optimization: The Complete Guide
These pages followed similar structures: introduction, what is keyword, benefits of keyword, types of keyword, how to do keyword, keyword strategies.
And as competition increased within this tactical surface area, we saw Fisherian Runaway through the concept of the Skyscraper Technique. No longer was it enough to define a term. One had to write the ultimate mega complete guide, 20,000 or more, to really emphasize how comprehensive and awesome and 10X it was.
Maybe you’d post them on inbound.org or GrowthHackers, ask your voting ring to upvote and comment.
Maybe you’d send ‘em to a few bloggers using an outreach template you found on Brian Dean’s blog (“Quick Question, love your posts. Could I get some feedback on this new article I just published?”).
Of course, there comes a point where the cost outweighs the benefit, and this is certainly what happened as marketers flocked towards 101 terminology with expensively produced guides that rehashed consensus information with a few new data points, quotes, or infographics sprinkled in.
You don’t hear as much about ultimate guides anymore, though. Why is that?
Well, AI Overviews demolished click-through-rates for consensus information and definitional content.
So the traffic trap dried up and led many marketers to take a step back, dismayed, and ask “what the hell is happening? And what’s next?”
Listicles: So Hot Right Now
Though user journeys were never as linear or as simple as SEO metrics like keyword rankings, impressions, and clicks made them out to be, they were at the very least legible and agreed upon.
Not so with AI search.
What we’ve centered on as an industry are “prompts,” which are akin to keywords in that they are the invocations you type into a chatbot to get an answer back.
There are many differences here, the biggest being we don’t actually know what prompts people are typing into chatbots or how often.
So, there’s a lot of uncertainty in what marketers should be using as a gauge for success in the channel.
Side: this is one of the more important questions with regards to AI search, as everything that happens is downstream of what you track in the first place. I’ve written about our framework here many times (rooted in first party customer data, calibrated by SEO and channel research, aligned with product and market relevancy). The LinkedIn post above from Lily Ray is very concise and good advice.
What many marketers do is simply track categorical prompts (like “best [category] tools”), and because of that, listicles have risen to prominence as a tactic, replacing Ultimate Guides.
Of course, listicles are not new. Apparently, Egyptians created listicles 4,000 years ago. More recently, I talked about listicles in relation to Surround Sound SEO at Growth Marketing Stage in Kyiv (2019). I referenced Paul Graham’s essay List of N Things in this slide:
I was obviously not the only person to discover listicles as they relate to SEO. But I like this slide because it speaks to the templatized nature of listicles, their scalability, and I was speaking about them in relation to Surround Sound SEO (which, in a sentence, is trying to appear everywhere on BOFU search engine results pages like listicles, review sites, and Reddit – sound familiar?).
This was an idea that was oppositional to the traffic incentive at the time. When I brought the idea forward at HubSpot, I had to go through constant debates around the lack of search volume for highly relevant queries, and in the beginning, I wrote most of the listicles myself and built partnerships to get us listed on other companies’ pages.
Eventually, we got buy-in and resources to scale the hell out of the play through affiliate, customer marketing, SEO, content, paid acquisition, and link building teams. But it was not immediately obvious at the time.
But now? Everyone’s writing listicles. And many are doing it in a way that can only be described as spam, publishing thousands of AI slop variants of similar content to hit every single modifier of “best X solution.”
No shade here, but how many variants of “AI” “Recruiting,” “Sourcing” and “Tools” do we really need?
The reason this is happening is fairly obvious.
In absence of agreed upon prompt measurement frameworks, many opt towards categorical prompts that will likely deliver brand recommendations in the outputs. Things like:
- “Best CRMs”
- “Best CRMs for small businesses” or
- “I run a 20 person startup and am looking for a new CRM. What are the best options?”
These prompts are asking for a list of recommendations, so naturally, ChatGPT or whatever AI tool you are using will go fetch a bunch of editorial lists, Reddit threads, and category pages from review websites and then summarize them into an output.
You ask for lists, you get lists. Then, as a tactical matter, you produce lists that hopefully get cited, and from that citation, your brand hopefully is listed in the output. Voila, brand visibility and GEO success.
And it does work fairly well, for now.
But I believe there will be a drawdown. The arbitrage window will close. The platforms will develop filtering mechanisms to preserve some semblance of the commons (or at least their ability to monetize search behavior and retain users). This is the cat and mouse game theory of arbitrage marketing.
Additionally, this may not even work for the intended output: getting recommended in AI answers.
While being indexed and used as a citation source is a valuable leading indicator of influence against a prompt or set of prompts, it by no means leads to the brand being recommended. Several SEOs have pointed this out recently, including Gaetano DiNardi:
“Brands cannot “hack” their way into getting preferential LLM treatment with old school ranking techniques.
[In this example], my client needs to drive up their review profile on Gartner and G2, get mentioned on all the other competitor listings, and ultimately build a better digital footprint in relation to Salesforce Service Cloud alternatives.”
This is where, in isolation, tactics fail. They don’t consider systemic interactions or upstream influences. A single list from a brand with no web presence is a drop of water in the ocean without commensurate coverage on review sites, Reddit, YouTube, directories, and other editorial pages. That’s why we talk about brand gravity.
To be clear, I don’t think producing listicles is a bad tactic. I think it’s a good one in most cases. We do it for ourselves and for our clients very often.
I also think that tracking your categorical entry points is absolutely best practice and should certainly be done.
But what’s happening now, at least on the tactical level, is very often cargo cult SEO based on the following path dependencies:
Brands care about being recommended in AI -> they invest in prompt tracking -> they choose bottom of the funnel prompts because they are more likely to offer specific brand recommendations -> the underlying citation sources are mostly listicles, review sites and Reddit -> they produce listicles and spam Reddit.
It’s basically the streetlight effect as applied to GEO. We are very likely in the Fisherian Runaway phase of this tactic, which in practice is most often using AI workflows to publish low quality listicles at an insanely high scale (risking both brand equity and historical SEO equity in the process).
The screenshot I shared above is not the worst I’ve seen, not by a long shot. I was approached recently by another agency who wanted to put us on their lists (obviously implying we put them on ours). They shared with me a list of nearly 100 unique pages all targeting “best SEO agencies in [city],” with inaccurate details on the pages and shallow/similar content on all of them.
Don’t think that’s a great play.
Why Is Everyone Talking About Reddit?
The other GEO tactic that gets a lot of airtime is Reddit.
“If you’re not on Reddit, you’re not in the conversation!”
I’ve spoken about Reddit before in this newsletter. I believe it is more useful as a mirror or as intelligence gathering than it is as a tactical avenue for many reasons.
I also believe that the large scale studies about citation patterns skew our understanding of its N of 1 importance to our own brands, simply because these studies look at very large and broad data sets, and Reddit is a very large website with many tentacles reaching into almost every possible topic.
Imagine an expansive list of topics, each carrying an array of its top citation domains. Expanded across enough topics, Reddit, Wikipedia, and YouTube will rise to the top simply because they touch the most topics.
I don’t think that means brands should ignore Reddit. I also find this research insightful, especially when looking at multiple studies and over time.
What I think it means is that marketers should think before rushing to a channel, and first, look at their own customer research, industry data, citations for their own intelligently chosen prompts, and their brand perception.
Then, if Reddit is deemed an influential source, one worth prioritizing in relation to everything else you could possibly do to grow your business, then it warrants strategy to fix and improve brand perception on Reddit. Sometimes that is by posting on Reddit, more often it is by building a brand and product that is worth talking about.
Aside: Interestingly, while writing this piece, an Adweek piece came out with the flashy title, “EXCLUSIVE: YouTube Overtakes Reddit as Go-To Citation Source on AI Search.” Of course, this led to a ton of LinkedIn discourse, but it all roots back to similarly misleading analyses. So even if YouTube has outpaced Reddit, in aggregate, it may not matter so much for your particular business. Keep running your own race.
Clicks, Traffic, and ChatGPT’s Real Influence
Finally, I want to touch on what I wrote about last week: ChatGPT’s influence on the customer journey.
A common refrain in SEO circles is that ChatGPT sends a very small percentage of a website’s overall traffic. Something like 1-5%.
The implicit assumption here is that traffic = value, or clicks = value.
This obviously breaks down in other domains, such as LinkedIn social, where marketers, for years, have invested in “zero click content” because, even without a click to your website, influence is occurring within the feed itself.
Whether we like it or not, this is the current nature of LLMs, where much of the influence they drive is upstream of measurable behavior.
By only measuring direct clicks from AI engines, we are looking at a very small percentage of its actual influence.
Measurement as the Reduction of Uncertainty
I want to reiterate a lesson a mentor of mine taught me a long time ago with regards to experimentation (as it applies to measurement as well):
All decisions include some level of uncertainty. Properly collected and analyzed data can reduce that uncertainty, but never eliminate it.
I could give endless examples of ways in which the data we collect results in quirky, interesting, imperfect, irrational, or even perverse outcomes. Many books have been written about this.
But the main point I want to get across in this essay is not to become jaded, nor is it to suggest any specific ways to measure SEO or AI search (we’ve covered that before and will do so again many times).
The main point here is one of introspection.
It’s important, especially in times of hype and uncertainty, not only to focus on the thing (the tactics, the channels, the advice), but how to filter noise, find signal, and contextualize the thing to see if it’s actually applicable to you. Thinking about thinking about tactics, perhaps.
When you see a tactic suggested on LinkedIn, or you see all of your competitors doing something, consider why or from what data or logic this tactic is coming. When you set KPIs or consider what telemetry or attribution systems to implement, consider what behavior is encouraged (and importantly, what behavior it may prevent or constrain).
There’s no perfect answer, as centuries of statistics and econometrics literature have emphasized.
But if we’re going to use data to make decisions, we may as well spend some time thinking about what type of decisions that data will encourage.
Want more insights like this? Subscribe to Field Notes
