Skip to content
BENIBIX Learn

Robots.txt and XML Sitemaps

Level 4 · The technical layer

Technical SEO starts with telling crawlers where to go. What robots.txt blocks, what a sitemap lists, and why noindex is the way to hide a page.

Course contentsLesson 8 of 15

After this lesson you can

  • Read a simple robots.txt file
  • Say what an XML sitemap does
  • Pick noindex or robots.txt for a page

12 mincovers 4 glossary entries

Worth reading first: Internal Links and URL Slugs

How to tell crawlers where to go

Technical SEO is the work that makes a site easy for crawlers to read. Three tools steer them. A robots.txt file says which parts of a site they may not visit. An XML sitemap lists the pages you want found. A noindex tag tells the engine not to show one page in results, even though it may visit it.

The robots.txt file sits at the top level of a site, like example.com/robots.txt. It holds simple lines. A Disallow line names a folder crawlers should skip, such as a basket or a search page. Well-behaved crawlers obey it. Bad ones ignore it, so it is not a lock.

Blocking a page in robots.txt does not keep it out of results. If other sites link to it, the engine can still list its address, without reading it. To keep a page out of results, let the crawler read it and give it a noindex tag.

An XML sitemap is a list of the pages you want crawled, with the date each one last changed. One sitemap file can hold up to 50,000 addresses. Big sites use several. You tell the engine where it is in robots.txt and in its own tools.

A sitemap helps the engine find pages, but it does not promise they will be stored or ranked. It matters most for new sites with few links, big sites with deep pages, and sites that change often.

Noindex is a line in the page's code, or a note the site sends with the page. Use it on thank-you pages, internal search results and thin tag pages. They add nothing for a searcher.

The common mistake is putting noindex and a robots.txt block on the same page. The crawler is blocked, so it never reads the noindex, and the page can stay in results. Pick one job: block crawling, or block showing.

ini
User-agent: *
Disallow: /basket/
Disallow: /search/

Sitemap: https://example.com/sitemap.xml
A small robots.txt. Crawlers skip the basket and the internal search, and the last line says where the sitemap is.
Robots.txt says where not to go. The sitemap says where to go. Noindex says do not show, and only works if the crawler can read it.

Worked example

A shop cleans up its crawl

An online shop has 2,000 products and 30,000 filter pages, like red shoes size 6 under £50. Crawlers spend their visits on the filter pages, and new products take weeks to appear.

It blocks the filter pages in robots.txt and sends a sitemap of just the 2,000 products. New products now show up in results within three days.

The words for this lesson4

Check yourself

How do you keep a page out of search results for good?

Finished this lesson?

Marking it complete also marks its 4 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.