robots.txt vs Noindex Tag
robots.txt asks crawlers not to read a page. Noindex lets them read it and asks them not to list it in the results.
robots.txt
A text file that tells crawlers which parts of a site not to fetch.
robots.txt is a plain text file at the root of a site that tells crawlers which paths they may not fetch. Each rule names a crawler and a path to allow or block. Well-behaved crawlers obey it, but it is a request, not a lock. It controls crawling, not indexing. A blocked page can still show in results if other sites link to it.
Noindex Tag
A tag that tells search engines to leave a page out of their results.
A noindex tag tells search engines not to show a page in their results. It goes in the page's HTML head as a robots meta tag, or in an HTTP header. The crawler has to fetch the page to see it. So a page with noindex must not also be blocked in robots.txt. It is the right tool for thank-you pages, internal search pages and thin pages.
In one line
- robots.txt
- A text file that tells crawlers which parts of a site not to fetch.
- Noindex Tag
- A tag that tells search engines to leave a page out of their results.