Does Ranking on Google Impact ChatGPT's Recommendations?

From the moment ChatGPT came out, the marketing world has been stressing about how it will affect our industry. Most of the hand-wringing has been around using ChatGPT and other AI tools to produce content and what that means for writers. Benji wrote a great essay with our thoughts on that; specifically, how AI content will make original content even more valuable.

But less-discussed is the other side of ChatGPT and SEO: What happens if people start using ChatGPT to search instead of Google?

For those of us in SEO, that’s a scary thought. We’ve built our careers by ranking clients on Google. How do we rank on ChatGPT? Can you even control that? What does “ranking” on ChatGPT even mean? How does ChatGPT decide what products to recommend when users ask for recommendations? 

That’s what this study seeks to explore. 

We looked at product recommendations from ChatGPT across 7 different B2B SaaS product categories and compared them to what products were most popular on Google’s first page  in those same categories. 

The results give us insights into the differences between how ChatGPT and Google rank products or websites and whether there is a correlation between what ChatGPT recommends and what’s already ranking on Google. 

Why We Studied ChatGPT vs. Google

Ultimately, we want to know if we can predict which products ChatGPT is going to recommend by seeing what ranks on Google. Closely tied to that, and a bit of a “stretch” question is: If I rank on Google for a product related query, does that increase my chances of having ChatGPT recommend me for similar queries?

Why focus on product recommendations?

As we’ve written about extensively and built our service on: bottom of funnel, product related queries are the highest ROI searches that a brand can show up for. As a result, they are the most valuable queries to understand with respect to ChatGPT. When someone asks ChatGPT for product recommendations, it means they are very motivated to buy. So showing up in ChatGPT’s recommendations is very valuable. 

Why compare ChatGPT to Google? 

We’re comparing ChatGPT to Google because Google is how most people search for products. We want to understand: Is the SEO work we’re doing for clients on Google going to help them get recommended by ChatGPT? Are they correlated at all?

How We Did This Study

We focused this study on B2B SaaS products because that’s what we know best. (We may do a follow up study on consumer product searches.)

We Googled 6 different phrases: 

1. Best time clock software
2. Inventory management software
3. Best crm
4. Best heatmapping tools
5. Best ecommerce platforms
6. Best accounting software for small businesses

Then, we asked the same thing of ChatGPT but in natural language:

1. “What are the best time clock software options?”
2. “What are some good inventory software options?”
3. “What are the best CRM options?
4. “What are some of the best heatmapping tools?”
5. “What are the best ecommerce platforms?”
6. “What are the best accounting software options for small businesses?”

We recorded every single product recommended by both Google and ChatGPT in a spreadsheet. 

Recording ChatGPT’s results is easy because it just lists 6–8 tools: 

Note: ChatGPT doesn’t recommend the same tools every time (even if you ask it the exact same question). It swaps in 1–3 new tools in its answer every time you ask. To keep things simple, we just used the first answer from ChatGPT.

On Google, however, there are pages and pages of results, each of which is typically a list of 10–30+ products. So we recorded every single product mentioned in each result on the first page, which was typically around 100 products.

Since the goal of this study is to see if ranking on Google can predict whether a product will be recommended by ChatGPT, we counted how many times each product mentioned by either tool was found on Google’s first page— that is, how many times it was repeated inside the list posts and software recommendation sites on the first page. We can call this the Page One Popularity number. That’s what you see in the right most column of the screenshot above (color formatted so we can see patterns visually).

So if there is some correlation with ChatGPT’s results and Google’s, we should, in theory, see high page one popularity numbers in ChatGPT’s recommendations.

ChatGPT Does Tend to Mention the Most Popular Products on Google’s First Page

Here are the average page one popularity numbers from Google (blue) and ChatGPT’s (red) product recommendations in each category.

For every query besides “best time clock software”, ChatGPT recommends products that are mentioned more often on Google’s first page than the average product listed by Google itself. In other words, ChatGPT tends to recommend the most popular products from Google’s first page results. 

So there is some correlation between the products listed in the page one results on Google and what ChatGPT recommends. Remember, of course, that correlation does not equal causation — it doesn’t mean ChatGPT is “reading” Google and listing products mentioned there, it just means the two tools have some overlap (sometimes a decent overlap) in the products they recommend when users ask for recommendations. 

Why does Google, on average, list less popular products than ChatGPT?

Why are Google’s products on average mentioned less times on its first page results than ChatGPT’s? 

The answer is not that Google doesn’t list popular products. It absolutely does. We’re literally defining popularity by how many times a product is mentioned on Page 1 of Google. 

The reason it’s page one popularity numbers are lower is because the list posts on page one of Google are so full of random obscure products that it’s bringing Google’s averages down. 

In contrast, ChatGPT doesn’t waste time listing as many lesser known products. When you ask ChatGPT for product recommendations, it only lists 7–8, and it biases those towards genuinely popular products. 

You can see this extremely clearly when we visualize the results for the query “best CRM”:

Every product is shown above as a blue line (if Google listed it) or red line (if ChatGPT listed it). The height of the line is how many times it appeared inside Google’s page one results. 

First, note that Google also lists the most popular products. The most popular products on the right side are in order of popularity: Salesforce, Pipedrive, HubSpot, Zoho CRM, Freshsales, and Zendesk Sell. They’re all there on Google’s first page. 

But the key takeaway is on the left side: The articles on Google’s first page list dozens of obscure, lesser known, products. That’s what’s bringing the average popularity of Google’s product suggestions down in the earlier graphs. 

In contrast, all of ChatGPT’s red lines are clustered on the right, meaning it’s listing the most popular CRMs, as defined by how many times they show up on Google. That’s notable. It means that ChatGPT is biased towards recommending genuinely more popular products.

ChatGPT fans could even argue that these results show how ChatGPT is more useful than Google — you don’t have to sift through pages of nonsense, you can get the heart of the answer more quickly. But that’s a debate we’re not getting into here. 

Instead, we can start to answer the main question we set out to explore.

Does this mean ranking on Google helps you rank on ChatGPT?

Let’s be honest, it would be extremely convenient if the above data meant that you could “rank” on ChatGPT by simply ranking on Google. Those of us in the SEO space know how to rank on Google already, so this would let us sort of “hack” the process of ranking on ChatGPT. 

But can we really get on ChatGPT by just ranking on Google? The data above gives a mixed answer. 

Yes, on average, ChatGPT is recommending the more popular products on Google’s first page. But by no means is the correlation perfect! We saw plenty of instances of:

1. ChatGPT not listing the popular products on Google.
   
2. ChatGPT listing products that are barely mentioned on Google.

For example, here are the 8 products recommended by ChatGPT for “best heatmapping tools” next to how many times they were mentioned on Google’s first page: 

Here is that same data, presented visually, in the same away we did for “best CRM” above: 

Indeed, ChatGPT mentions the three most popular heatmapping tools: Hotjar, Mouseflow, and Lucky Orange. After that, Crazy Egg and FullStory are also very well known tools in this space and are mentioned 6 and 7 times on Google’s first page. So those five products make sense. 

But then ChatGPT also lists Matomo and Yandex Metrica, which were only listed once out of 117 tools (42 unique tools) listed on the first page of Google. Once! These tools are not popular heatmapping tools by any means. I, personally, have been in the UX space for nearly a decade and have used multiple heatmapping tools for many clients and have never heard of Matomo or Yandex Metrica for heatmapping. Yet, ChatGPT recommends them. 

On the other hand, Smartlook, a former client of ours, is mentioned 9 times on page one of Google for “best heatmapping tools”. Some of that is due to our SEO work, sure, but we don’t own all of the sites on the first page, so it means they are genuinely popular in this space as well. But Smartlook wasn’t mentioned by ChatGPT at all when we asked it about the best heatmapping tools. 

This goes to show that being one of the most popular tools in Google’s results does not guarantee that ChatGPT will also mention you. 

This pattern of ChatGPT also throwing in a few obscure products in its recommendations wasn’t limited to heatmapping tools. When asked about “best inventory management software”, it recommended TradeGecko, among others, which is not mentioned even a single time out of 97 products listed on the first page of Google!

How can we make sense of this? Why is ChatGPT usually listing the most popular products on Google, but also throwing in some less popular products? How is ChatGPT getting its recommendations in the first place? Is the correlation with Google’s results just random chance or is it expected based on how the two products work? 

Reasons Why We Can Expect Some Overlap Between Google and ChatGPT’s Results

ChatGPT is a language model, meaning it ingests a bunch of text — such as articles on the internet, books, sites like Reddit, and more — and it learns what patterns or sequences of words most commonly appear together. It’s then trained to put words together in a sequence that closely matches those patterns. 

(See OpenAI’s original paper explaining how GPT-2 works.) 

For example if you ask it to complete the sentence “A large ecommerce platform based in Canada is ____”, its model, based on all of the text it’s “read” or been trained on, guesses the word that is most likely to come next. That word, in this case is, of course, “Shopify”. 

At a high level, that’s largely all ChatGPT does. It puts words together in ways that most closely match patterns in everything it’s read (which is terabytes worth of data). 

So, for our question of how to get ChatGPT to recommend your product, if nothing has been written about your product in books or on the web before, it’s impossible for ChatGPT to mention you. It won’t associate your product’s name with sequences of words about your product category because no one has done that already. 

That means, by its very nature, ChatGPT isn’t likely to mention brand new products in a category. 

Inversely, this means that if a bunch of text on the internet has talked about your product, you have a good shot at being mentioned by ChatGPT. In fact, the more your product is talked about on the internet, the higher your chances of being mentioned by ChatGPT. 

And, specifically, since it’s a language model, the more your product is mentioned in the context of the category-related words you want to show up for, like “CRM” or “small business accounting tools”, the higher the chances that ChatGPT’s language model will associate your product’s name with those words.  

But the details matter. Specifically, as marketers, knowing exactly what ChatGPT has been trained on would help us understand how to get it to mention our company or our products. 

Exactly what information is ChatGPT trained on? Does it “read” Google?

OpenAI is intentionally vague about exactly what material its latest model, GPT-4, is trained on — as this article in the New Yorker says — in order to not help their competition. 

They just say it’s trained on both publicly available data and private data they license:

In terms of what this “publicly available data” could be, one of the original papers from OpenAI about how GPT-2 worked said they devised their own version of a web crawler, focusing specifically on pages that were linked to by humans, on sites such as Reddit: 

Again, this paper was about GPT-2, but based on this, we can say it’s unlikely that GPT-4 was trained by literally Googling things. So we can’t explain the correlation between Google’s results and ChatGPT’s recommendations in our data above by saying “ChatGPT reads Google.” But rather, the correlation likely comes at a more general, higher level: 

Google scrapes the web and links to pages based on what it decides is most relevant to a search query. ChatGPT also scrapes the web (among other materials) and writes natural language answers based on what it thinks is the most common sequence of words on a topic. 

Both tools are scraping the web and determining, in their own ways, what’s most relevant, popular, or likely to be said by others when you ask it a question. So it’s not totally surprising that they have overlap! 

At a high level, that means the goal of marketing teams should be more or less similar for both: be mentioned as much as possible in content around the internet to increase your chances of being mentioned on Google or by ChatGPT. 

Google is Easier to Manipulate or Hack Your Way Into

Google is just an index of websites (we’re not talking about Google Bard here, just the regular Google that millions of people use daily). It ranks them based on a bunch of factors, including, notably, how many times other sites link to them. These two characteristics of Google…

1. Google just links to websites, as opposed to generating its own content 
2. Google chooses those websites in large part by how many other sites link to it 

…make it relatively easy to get onto Page 1 of Google even if you aren’t really that popular. 

Specifically, you can hack your way onto Google’s first page in two notable ways:

First, you can just pay one of the sites already ranking on page one to list your tool. This sounds nefarious but it’s actually extremely common. One of the most obvious examples in software are review sites like G2 and Capterra. G2’s pages say they are listing the “best [insert space] software” but everyone knows the top products on those lists are just paying G2 the most. And yes, you can also just pay Google to show up in one of the ad spots, but we’re just talking about ranking organically for now because ChatGPT could, in theory, also serve contextual ads if they wanted. 

Second, you can “SEO” your way onto the first page. Search engine optimization (SEO) is a long established field, with a set of best practices around getting on the first page of Google for a given search term, in particular (1) good on page content and (2) backlinks. 

What’s important here is that you can rank your site on the first page, even if you aren’t the most popular — that is, your tool hasn’t been mentioned all over the internet. Trust me, we’ve done this for many clients who didn’t come to us as the market leader in their space. You absolutely don’t need to be Salesforce to get on page one for “best CRM”, or you don’t need to be QuickBooks to get on page 1 for “accounting software”. You do have to know how to do SEO, of course, but that’s knowable (you can hire an agency like us, build an in-house team, contract freelancers, etc.). 

Why Ranking on ChatGPT Seems Harder: You Genuinely Have to Be Popular

In contrast, based on the results of this study and how ChatGPT works, getting ChatGPT to suggest your product seems harder. This is for the same two reasons that make Google easier. 

First, ChatGPT is generating its responses by itself word for word. It’s not linking to other people’s content. So its results are a lot more concise. Look at our study above, ChatGPT recommended 7–8 tools for each query, whereas there were over 100 tools listed in the 8–10 results on Page 1 of Google for the same query. For one, that’s just a much smaller list you need to get onto. Second, you can’t use the typical SEO tactic where you create a page that fulfills most of Google’s criteria, but also throw in a mention of your client’s tool. ChatGPT isn’t linking to your page where you can write whatever you want, you need it to literally write about you. 

Second, their language model doesn’t seem to have obvious factors you can control and manipulate (like backlinks for Google). As we detailed above, these GPT models read a ton of text, and then predict word for word what that text would likely say in response to a question. So it’s spitting back a sort of “average” or “most probable” list of words based on the terabytes of text it’s been trained on. How do you hack that? There isn’t some clear action you can take like “building links” to get its algorithm to mention you. You kind of genuinely have to be mentioned a lot in its training text (which we don’t even really know the identity of). This is harder. 

Nonetheless, the core takeaway from above still holds: you need to be mentioned in as many places and as many times as possible to increase the chances that ChatGPT mentions you to its users. In the end, that’s marketing. And you could even argue it’s more honest marketing than trying to “game” Google by buying a favorable domain or building a bunch of links — you have to actually get popular. 

How Ranking on Google Can Increase the Chances of ChatGPT Mentioning You

Ranking on Google is one (great) way of becoming popular. Google is still the #1 portal into the internet for the majority of the world, by a wide margin (93% vs 3% for Bing). 

If you get a bunch of SEO traffic from useful, relevant keywords in your niche, then it’ll naturally expose you to other people who may also write about you. 

In other words, SEO is just a great evergreen marketing channel (yes, we’re biased, but it’s true). Paid ads, the other dominant inbound channel, can also work, as can influencer marketing, going on podcasts, or a bunch of other marketing tactics. But SEO has the advantages of (1) showing up for the exact right terms and (2) giving you free traffic for as long as you’re ranking — you don’t have to keep paying for every click. 

Ranking on Google for relevant terms in your industry is a way to get more people on the internet to write about you, which increases the chances that ChatGPT or other AI tools will read about you when being trained. 

All of This Only Matters If ChatGPT Becomes a Popular Search Engine

Of course, all of this is only relevant if a significant percent of the public starts using ChatGPT or equivalent AI to do product searches. From the searchers’ side, there are pros and cons to ChatGPT vs. Google. 

On the pro side, the analysis above shows that ChatGPT on average, for product queries, suggests more genuinely popular products. You don’t have to sift through pages of obscure products no one has heard of to get to the actually popular products in a category. You don’t have to wonder “did they just pay ChatGPT to get listed?” like you do with G2 or Capterra lists. 

But there are also cons to using ChatGPT for product searches. If you actually do want a long list, ChatGPT is more of a pain. You have to keep asking for more. Second, you can’t verify sources or citations like you can on Google. If you want to ignore G2’s list on Google, for example, you just don’t click on it. You can also more easily search for recommendations from a particular source on Google (e.g. “best tax software pcmag” or “best blender wirecutter”). 

With ChatGPT, if it mentions an obscure product (like the heatmap examples above), you’re sort of at a loss as to why. Finally, the propensity of ChatGPT to make up random facts could just hurt people’s overall confidence in its results, causing them to use it less as a search engine replacement. 

None of us can predict the future, but in June 2023, ChatGPT saw its first decline in users — almost 10% — which is somewhat significant. 

We’ll see how Google’s AI equivalent, Bard, does in popularity once it’s released to the wider public. 

Future Studies

In this study, we only looked at the correlation of how many times ChatGPT’s recommended products were mentioned in Google’s first page results. 

But there are a lot of other proxies for popularity in society that we could also look into to see if ChatGPT’s recommendations are more strongly correlated with other factors. 

For example: 

• Domain rating or domain authority — These metrics from SEO tools are largely a measure of how many backlinks a site gets, that could also be an indication of the likelihood that ChatGPT will run into that product in its training text. 

• Reddit — We know ChatGPT uses Reddit as one of its training sources, so we can try to find a way to correlate its recommended products with popularity on Reddit. 

• Size of company — We could even look at whether ChatGPT biases towards large or public companies since it’s likely that those companies are just mentioned more in society and literature than smaller private ones. 

Any other ideas, questions, or requests for us to study? Ask us in the comments.