🔍robots.txt
Also known as Robots.txt File, Robots Exclusion Protocol
A text file that tells crawlers which parts of a site not to fetch.
Think of it like
It is like a sign on a gate that says "Staff only". Polite visitors stay out.
Example
A shop blocks its /checkout/ and /cart/ paths so crawlers spend their time on product pages instead.
Why it matters
One wrong line can stop crawlers reading a whole site. It is the first file to check when traffic vanishes.
Definition
Example code
ini
User-agent: *
Disallow: /checkout/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlCommon mistake
Key takeaways
- 1A request. Polite bots obey, others may not.
- 2Crawl, not index. It does not remove pages from results.
- 3At the root. Always at /robots.txt.