Crawling, Indexing and Ranking: how do Search Engines work?

Search engines have evolved into answer engines. And very sophisticated ones.

Their job is to discover, interpret, and organize content on the internet; to show users results that are relevant to them and their needs.

To be sure that your content appears to as many users as possible, you must understand how to make it visible first.

Visibility is the most critical aspect of SEO; the ‘zero moment’ when everything starts. If the search engine does not index your website, it cannot show it among its results.

But, how do search engines work? They do three things: crawling, indexing, and ranking.

Crawling: is the analysis of the webpages on the internet. The search engine scans the code and the content of every URL it finds.
Indexing: is the gathering of contents found during the crawling process. If a page is on this list, it means that the search engine indexed it, and it can appear among the search results for the related queries.
Ranking: is the order in which the indexed results appear on the result page (SERP). The list goes from the most relevant content to the least.

Crawling

Crawling is the process through which the search engine rummages the content on the world wide web: websites old and new, articles, product sheets, images, links, etc.

Search engines use crawlers (also called bots or spiders) that, through specific algorithms, decide which websites to scan and how often, distributing the so-called crawl budget.

The crawler discovers new content by identifying and recording every link it finds on the scanned pages and then puts them in a list of to-be-crawled URLs.

This action is critical for SEO because this is the moment when the search engine discovers the number and the quality of connections of a page, both inbound and outbound.

Do you wanna know more about crawling? Learn how Search Engines’ crawlers work reading our article.

Indexing

Once it has extracted the page content, the crawler will put it on the index of the visited pages and organizes all the information. The information is then used to measure the relevance of the page compared to similar ones.

Ranking

While the two previous activities are the “behind-the-scenes” of the work of search engines, ranking is the most evident one.

SERP is where an SEO consultant can show its customers the results of her work. Once a user enters a keyword on the search bar, the search engine will search on its index the pages that match the search query.

Then it gives to these pages a score calculated out of 200 different ranking factors.

If you did a good SEO job, the page, blog post, or product sheet you worked on would get a better positioning than your competitors.

Ranking factors

A page ranking process includes different aspects.

Putting technicalities aside, an ‘answer engine’ like Google continually learns how to provide information in the best way possible, putting into relation different factors:

Website typology: a simple ranking, made by the search engine to tell a search query from another;
Context: related to each search query;
Time: The importance given to a single factor varies with the query;
Layout: SERP will show different results depending on the search intent.

Website typology

As soon as the user types a query, the first thing the search engine does is to categorize it so that it can return the best typology for that query.

For example:

Your Money, Your life (YMYL) websites;
Local businesses’ websites;
Adults websites;
News websites.
Etc.

It is an-ever changing ranking, but it helps determine to which ‘place’ a query belongs.

Context

The search engine considers context, too. It extracts any relevant information form the user who types a query, understanding the social, historical, and environmental factors in play and providing the most useful answer. These factors are, for example, position, time, query type.

Time

Another aspect that you must keep into consideration is the relation between the ‘weight’ given to a query, the time it has been performed, and the time the content has been indexed.

This because every search query has a different intent, and the search engine knows it.

For this reason, results for, say, “First World War” are more influenced by the source authority, while for “movies out now” the search engine prioritizes the content freshness.