Skip to content
BENIBIX Learn

Crawling vs Indexing

Crawling is the engine reading your page. Indexing is the engine deciding to keep it. A page can be read and not kept.

Marketing🔍 SEO

Crawling

How a search engine finds pages. A program reads a page and follows its links.

Crawling is how a search engine finds pages. A program called a crawler fetches a page and reads it. It then follows the links on that page to find more pages. Pages with no links pointing to them are hard to find. A crawler only reads a page, it does not decide to keep it.

Example
A new blog post is linked from the home page. Googlebot visits the home page, follows the link, and fetches the post the same day.
Why it matters
A page that is never crawled can never rank. Links and a sitemap are how crawlers learn it exists.
Full entry
Marketing🔍 SEO

Indexing

Storing a crawled page in the engine's big list. Then it can show up in results.

Indexing is when a search engine stores a page it has crawled so it can show it in results. The engine works out what the page is about and files it in a giant list called the index. Not every crawled page gets in. Pages that are thin, copied or marked noindex are often left out. Only an indexed page can rank.

Example
Emma searches site:example.com and sees 120 of her 150 pages. The other 30 were crawled but never indexed.
Why it matters
A page that is not in the index can not be seen by searchers, however good it is.
Full entry

In one line

Crawling
How a search engine finds pages. A program reads a page and follows its links.
Indexing
Storing a crawled page in the engine's big list. Then it can show up in results.