Skip to content
BENIBIX Learn

Crawling

Marketing🔍 SEOSearch Engines

Also known as Web Crawler, Search Engine Spider, Crawler, Bot

How a search engine finds pages. A program reads a page and follows its links.

Think of it like

It is like a mail carrier walking every street. They follow each road to find new houses.

Example

A new blog post is linked from the home page. Googlebot visits the home page, follows the link, and fetches the post the same day.

Why it matters

A page that is never crawled can never rank. Links and a sitemap are how crawlers learn it exists.

Definition

Crawling is how a search engine finds pages. A program called a crawler fetches a page and reads it. It then follows the links on that page to find more pages. Pages with no links pointing to them are hard to find. A crawler only reads a page, it does not decide to keep it.
Crawling is the first box. The page is read here, not yet stored.

Common mistake

Thinking crawled means listed. Crawling is only reading. Being stored for search is a later step.

Key takeaways

  1. 1
    Follow links. Crawlers move from page to page.
  2. 2
    Reads, not keeps. Storing is a separate step.
  3. 3
    Help it find. Links and sitemaps show the way.

Compare

Learn it properly