Thumbnail: Darwinism In Search Revisited And Extended

Originally Published on Search Engine Journal August 18, 2021 (Jason Barnard)

New insights on how Google search ranking works plus the concept of the SERP as a product and how it plays a role in SEO.

On Google’s July 29th Search Off the Record podcast, Gary Illyes explained how rich elements get their place in the search results.

In doing so, he confirmed what he had told a roomful of SEO professionals in Australia in 2018, which was the basis of my 2019 Darwinism in Search article.

Darwinism in search and how search ranking works.

I wrote that article from memory in a pub a couple of hours after hearing Gary’s explanation, and filled in the gaps – things he didn’t explain or that I had forgotten – with some guesswork.

Not all of my guesses were correct. But from what Gary just shared, it seems I was close.

That article inspired me to dig deeper and go to Seattle to interview algorithm team leads from Bing (see below).

As the Bing team is more open than Google, we gained great insight into the mechanics of how search engines work.

In that series, I suggested that since the technology, the data, the audience, and the ultimate goal are the same, what the people at Bing shared with me could broadly be (and helpfully) applied to Google.

Now, we obviously cannot assume every detail is the same.

But in the Search Off the Record podcast, Gary confirms that search engines largely function in the same way.

Search Engines Function in Much the Same Way

When explaining how Universal Search works and how search engines build the final search results page (SERP), Gary said:

“It’s not Google-specific. Other engines do it as well, and because most search engines rank results in much the same way… this is probably applicable to every Search Engine…”

This brings the Bing series into better focus and suggests (to me, at least) that reading these articles is a wise investment of your time.

  1. How Google Search Ranking Works: Gary Illyes, Google (Darwinism in Search)
  2. How Ranking Works at Bing: Frédéric Dubut, Senior Program Manager Lead, Bing
  3. Discovering, Crawling, Extracting and Indexing at Bing: Fabrice Canel Principal Program Manager, Bing
  4. How the Q&A / Featured Snippet Algorithm Works: Ali Alvi, Principal Lead Program Manager AI Products, Bing
  5. How the Image and Video Algorithm Works: Meenaz Merchant, Principal Program Manager Lead, AI and Research, Bing
  6. How the Whole Page Algorithm Works: Nathan Chalmers, Program Manager, Search Relevance Team, Bing
Darwinism & Google SERPs As Product: Revisiting How Search Works

Bing Series 2

And good news – Frédéric Dubut and Fabrice Canel at Bing have agreed, in principle, to a second Bing Series with me in September. They are both incredibly smart and delightful people, so I cannot imagine the series will be anything but wonderful from every perspective. Stay tuned.

There is a “SERP Anatomy” Algorithm at Google

Once the idea was presented to me by Nathan Chalmers, it seemed so obvious and I was pretty ashamed not to have thought about it before.

There must be an algorithm that builds the “product.” Pure Darwinism can’t function in a commercial world.

Bing and Google have to implement an algorithm to build the universal result (i.e., integrate the rich elements into the foundation list of blue links). That algorithm has the remit to build the best product possible from the candidates offered by the different verticals (blue links, videos, featured snippets, images, related searches, ads, podcasts, etc.).

That algorithm is built purely to evaluate the pros and cons of each candidate set in the context of the user query and to what extent their inclusion will add value to the SERP in its ultimate goal to serve the intent of the user’s search query.

Reminder: The fundamental goal of the SERP is to provide the user with the best solution to their problem/answer to their question, as efficiently as possible.

At Bing, they build the anatomy of the SERP with what they call “The Whole Page Algorithm.” And ironically, it has a component called Darwin.

Gary Illyes and John Mueller refer to it rather obliquely, and jokingly call it “Super Search Engine” (John) and “Universal Mixer” (Gary), but do indeed confirm its existence at Google.

I’m sticking with the Whole Page Algorithm, at least for the moment.

Importantly, Gary confirms what Nathan Chalmers from Bing told me: That although click-through rate isn’t used in the algorithms that drive the rankings, they are a major component of the Whole Page Algorithm.

Here is a quote from the Nathan Chalmers article:

On SERP user behavior is a very, very important metric for the Whole Page Algorithm.

The success of failure of any combination of blue links and rich elements is measured by how the user interacts with it.

So, click-through rate doesn’t affect rankings in the ‘traditional’ sense that we have tended to understand it.

On-SERP behavior doesn’t affect the blue link rankings, nor the bids submitted by the candidate sets aiming to replace them.

But they are a very big part of how that whole page is organized.

User behavior on the SERP is fed back into the algorithm (Darwin), which gives the machine the corrective or reinforcement signals it needs to improve its performance.

So, when Googlers tell us that click-through rate is not used in the ranking algorithm, they are telling the truth.

But the issue is more nuanced than that.

It doesn’t affect ranking, but since the Whole Page algorithm is very much user behavior-based, it massively affects whether any prospective rich element appears on page 1 or not.

At Bing, one part of the Whole Page Algorithm has the right to promote, demote, or veto any and all results the other algorithms offer.

That algorithm is called Darwin.

In the podcast, Illyes seems to attribute less veto power to the Whole Page Algorithm alternative at Google. It is possible that Google’s Whole Page Algorithm is given less power.

Because it is essentially designing the Google product and therefore has a commercial focus (see below), I believe it has a similar level of power, and Illyes is just being coy.

But the big takeaway here is that both Google and Bing have a “Super Search Engine” that designs the anatomy of the SERP based on a multitude of user signals (including click-through rate) and also human feedback (think Quality Raters who focus on SERP anatomy).

This concept of a Whole Page Algorithm is fundamentally important to us as SEO professionals. And if we are smart, it should change our approach to SEO. It certainly changed mine when I first learned about it!

Packaging Your Solutions So That Google Recommends Them

I repeat this phrase to myself every morning:

“The art of SEO is packaging your content so that it fits Google’s SERP product.”

In the context of the modern multimedia SERP, that means:

  • Providing the best and most effective solution to the user’s problem.
  • Providing that solution in the most appropriate format for that user in their current context.
  • Packaging the content so that it is well suited to Google’s product (the SERP).

And in that context, this approach makes sense:

  1. I provide the solution to a problem expressed by the subset of Google’s users that are my audience.
  2. I am asking Google to recommend my solution as the best answer for their user in that user’s current situation.
  3. What can I do to convince Google to recommend my solution rather than the solution offered by my competition?
Darwinism & Google SERPs As Product: Revisiting How Search Works

The answer to that third point is relatively simple and based on three pillars: Understanding, Credibility, and Deliverability.

Nail all three, or give up and go home.

The SERP is Google’s Product

I missed the importance of one thing Nathan Chalmers mentioned in passing a year ago. His algorithm is designed solely to build the “Bing product” (AKA, the SERP).

I had assumed it is the same at Google and from what Gary Illyes said, that seems to be the case.

So a change in perspective is helpful. Both Google and Bing (in fact, all search engines) offer a freemium product: the SERP.

And that changes my three pillars. Rather than Understanding, Credibility, and Deliverability, we are looking at Understanding, Credibility, and Suitability.

That is, the suitability of your content to be integrated into their product and offered up as a solution to the subset of their users who are your audience.

Deliverability is, of course, part of that suitability. But the concept of Deliverability is too narrow and fails to emphasize the SERP as a product.

Darwinism & Google SERPs As Product: Revisiting How Search Works

How Does Google Monetize Its “SERP Product?”

I really want to insist here that when I say product I am in no way talking about Google Ads, or your products/services, or any commercial offers Google and Bing show in their SERP.

The SERP is the search engine’s product. This is what drives their revenue.

Basically, we have three approaches, and three wins for Bing and Google. I’m probably missing a few tricks here, so please feel free to ping me on Twitter with any I have missed:

  • Search users “pay” by providing Google and Bing with monetizable behavioral data.
  • Advertisers pay for clicks on the myriad of ad elements.
  • SEO, businesses, and other creators pay through the provision of content that populates the SERP in the hope of being shown as one of the recommended solutions.

Google SERP as a Product = Ads

No. And “no” again :)

It is incredibly important to bear in mind that the product is a freemium model. Google aims to help its users to the solution to their problem, and they aim to make the process as efficient as possible whether they get any immediate financial return (ad click) or not.

The vast majority of searches do not result in an ad click. If you look at that on a search-by-search basis, then it makes no sense.

If you look at it from the perspective that each search result costs Google peanuts, then the calculation is simply that (these numbers are 100% invented) the 2% of paid clicks on the 10% of commercial queries pays for both free and paid-by-an-ad-click usage.

In the context of the SERP as a product, Google and Bing train the Whole Page Algorithm to find the ideal balance that both serves the user and makes Google money.

That’s not on the basis of individual searches, but of “some search intents will pay, most won’t.” Ensuring the user finds their solution efficiently is the key.

A User-Focused Freemium Product Driven by the Whole Page Algorithm

Google Ads competes just like any other rich element. If the ad cannot efficiently and effectively bring the user satisfaction in the context of their search (intent, geo, device…), then Google will not show it.

All of Google’s algorithms in the multiple verticals offer up their best candidates (hence the concept of “candidate sets”), but the Whole Page Algorithm designs the freemium SERP product and has absolute power of veto and promotion.

The Whole Page Algorithm is the keystone for Google and therefore should be a super-important focus for SEO pros, too.

Whether that means looking at click-through rates, perceived intent, ad suitability, format appropriateness, device compatibility, or any one of many more factors, that final “suitable-for-our-product” filter can make or break even the best optimized content you create.

Blows my mind every time  :)

Similar Posts