🔍Crawling
Also known as Web Crawler, Search Engine Spider, Crawler, Bot
How a search engine finds pages. A program reads a page and follows its links.
Think of it like
It is like a mail carrier walking every street. They follow each road to find new houses.
Example
A new blog post is linked from the home page. Googlebot visits the home page, follows the link, and fetches the post the same day.
Why it matters
A page that is never crawled can never rank. Links and a sitemap are how crawlers learn it exists.
Definition
Common mistake
Key takeaways
- 1Follow links. Crawlers move from page to page.
- 2Reads, not keeps. Storing is a separate step.
- 3Help it find. Links and sitemaps show the way.