Skip to content
BENIBIX Learn

robots.txt vs Noindex Tag

robots.txt asks crawlers not to read a page. Noindex lets them read it and asks them not to list it in the results.

Marketing🔍 SEO

robots.txt

A text file that tells crawlers which parts of a site not to fetch.

robots.txt is a plain text file at the root of a site that tells crawlers which paths they may not fetch. Each rule names a crawler and a path to allow or block. Well-behaved crawlers obey it, but it is a request, not a lock. It controls crawling, not indexing. A blocked page can still show in results if other sites link to it.

Example
A shop blocks its /checkout/ and /cart/ paths so crawlers spend their time on product pages instead.
Why it matters
One wrong line can stop crawlers reading a whole site. It is the first file to check when traffic vanishes.
Full entry
Marketing🔍 SEO

Noindex Tag

A tag that tells search engines to leave a page out of their results.

A noindex tag tells search engines not to show a page in their results. It goes in the page's HTML head as a robots meta tag, or in an HTTP header. The crawler has to fetch the page to see it. So a page with noindex must not also be blocked in robots.txt. It is the right tool for thank-you pages, internal search pages and thin pages.

Example
An online course hides its "thank you for buying" page with noindex. Strangers can no longer find it in search and skip the payment.
Why it matters
It is the sure way to keep a page out of results. People can still visit it.
Full entry

In one line

robots.txt
A text file that tells crawlers which parts of a site not to fetch.
Noindex Tag
A tag that tells search engines to leave a page out of their results.