Skip to content
BENIBIX Learn

robots.txt

Marketing🔍 SEOTechnical SEO

Also known as Robots.txt File, Robots Exclusion Protocol

A text file that tells crawlers which parts of a site not to fetch.

Think of it like

It is like a sign on a gate that says "Staff only". Polite visitors stay out.

Example

A shop blocks its /checkout/ and /cart/ paths so crawlers spend their time on product pages instead.

Why it matters

One wrong line can stop crawlers reading a whole site. It is the first file to check when traffic vanishes.

Definition

robots.txt is a plain text file at the root of a site that tells crawlers which paths they may not fetch. Each rule names a crawler and a path to allow or block. Well-behaved crawlers obey it, but it is a request, not a lock. It controls crawling, not indexing. A blocked page can still show in results if other sites link to it.

Example code

ini
User-agent: *
Disallow: /checkout/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml
Every crawler is asked to skip the checkout and cart. The last line says where the sitemap is.

Common mistake

Using it to hide a page from results. Blocking stops crawlers from seeing a noindex tag, so the URL can still be listed.

Key takeaways

  1. 1
    A request. Polite bots obey, others may not.
  2. 2
    Crawl, not index. It does not remove pages from results.
  3. 3
    At the root. Always at /robots.txt.

Compare

Learn it properly