# Crawling

> How a search engine finds pages. A program reads a page and follows its links.

**Field:** Marketing & SEO  
**Category:** SEO  
**Also known as:** Web Crawler, Search Engine Spider, Crawler, Bot

## Think of it like

It is like a mail carrier walking every street. They follow each road to find new houses.

## Definition

Crawling is how a search engine finds pages. A program called a crawler fetches a page and reads it. It then follows the links on that page to find more pages. Pages with no links pointing to them are hard to find. A crawler only reads a page, it does not decide to keep it.

## Example

A new blog post is linked from the home page. Googlebot visits the home page, follows the link, and fetches the post the same day.

## Why it matters

A page that is never crawled can never rank. Links and a sitemap are how crawlers learn it exists.

## Common mistake

Thinking crawled means listed. Crawling is only reading. Being stored for search is a later step.

## Three principles

- **Follow links.** Crawlers move from page to page.
- **Reads, not keeps.** Storing is a separate step.
- **Help it find.** Links and sitemaps show the way.

## Related terms

- [Indexing](https://learn.benibix.com/term/marketing/indexing)
- [Search Engine](https://learn.benibix.com/term/marketing/search-engine)
- [XML Sitemap](https://learn.benibix.com/term/marketing/xml-sitemap)
- [robots.txt](https://learn.benibix.com/term/marketing/robots-txt)
- [Internal Link](https://learn.benibix.com/term/marketing/internal-link)

## Told apart from

- [Crawling vs Indexing](https://learn.benibix.com/compare/crawling-vs-indexing): Crawling is the engine reading your page. Indexing is the engine deciding to keep it. A page can be read and not kept.

## Taught in

- [How Search Engines Work](https://learn.benibix.com/learn/marketing-seo/how-search-engines-work): SEO, 1/15

![Three boxes in a row joined by arrows: crawling, indexing and ranking. The first box shows a program following links.](https://learn.benibix.com/diagrams/crawl-index-rank.svg)

---

Source: https://learn.benibix.com/term/marketing/crawling
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-20
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
