# robots.txt

> A text file that tells crawlers which parts of a site not to fetch.

**Field:** Marketing & SEO  
**Category:** SEO  
**Also known as:** Robots.txt File, Robots Exclusion Protocol

## Think of it like

It is like a sign on a gate that says "Staff only". Polite visitors stay out.

## Definition

robots.txt is a plain text file at the root of a site that tells crawlers which paths they may not fetch. Each rule names a crawler and a path to allow or block. Well-behaved crawlers obey it, but it is a request, not a lock. It controls crawling, not indexing. A blocked page can still show in results if other sites link to it.

## Example

A shop blocks its /checkout/ and /cart/ paths so crawlers spend their time on product pages instead.

## Example code

```ini
User-agent: *
Disallow: /checkout/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml
```

Every crawler is asked to skip the checkout and cart. The last line says where the sitemap is.

## Why it matters

One wrong line can stop crawlers reading a whole site. It is the first file to check when traffic vanishes.

## Common mistake

Using it to hide a page from results. Blocking stops crawlers from seeing a noindex tag, so the URL can still be listed.

## Three principles

- **A request.** Polite bots obey, others may not.
- **Crawl, not index.** It does not remove pages from results.
- **At the root.** Always at /robots.txt.

## Related terms

- [Noindex Tag](https://learn.benibix.com/term/marketing/noindex)
- [XML Sitemap](https://learn.benibix.com/term/marketing/xml-sitemap)
- [Crawling](https://learn.benibix.com/term/marketing/crawling)
- [Technical SEO](https://learn.benibix.com/term/marketing/technical-seo)
- [AI Crawler](https://learn.benibix.com/term/marketing/ai-crawler)
- [llms.txt](https://learn.benibix.com/term/marketing/llms-txt)

## Told apart from

- [robots.txt vs Noindex Tag](https://learn.benibix.com/compare/robots-txt-vs-noindex): robots.txt asks crawlers not to read a page. Noindex lets them read it and asks them not to list it in the results.
- [llms.txt vs robots.txt](https://learn.benibix.com/compare/llms-txt-vs-robots-txt): People call llms.txt the robots.txt for AI, and it is not. robots.txt tells bots which pages they may not fetch. llms.txt is a short guide that points AI tools to the pages worth reading. It blocks nothing at all. To keep an AI bot out, you still need a line in robots.txt.

## Taught in

- [Robots.txt and XML Sitemaps](https://learn.benibix.com/learn/marketing-seo/robots-txt-and-xml-sitemaps): SEO, 8/15

---

Source: https://learn.benibix.com/term/marketing/robots-txt
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-20
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
