Robots.txt and XML Sitemaps
Level 4 · The technical layer
Technical SEO starts with telling crawlers where to go. What robots.txt blocks, what a sitemap lists, and why noindex is the way to hide a page.
Course contentsLesson 8 of 15
Level 1 · How search works
Level 2 · Keywords
Level 3 · The page itself
Level 4 · The technical layer
Level 5 · Links and local search
Level 6 · Measuring and AI search
After this lesson you can
- Read a simple robots.txt file
- Say what an XML sitemap does
- Pick noindex or robots.txt for a page
12 mincovers 4 glossary entries
Worth reading first: Internal Links and URL Slugs
How to tell crawlers where to go
Technical SEO is the work that makes a site easy for crawlers to read. Three tools steer them. A robots.txt file says which parts of a site they may not visit. An XML sitemap lists the pages you want found. A noindex tag tells the engine not to show one page in results, even though it may visit it.
The robots.txt file sits at the top level of a site, like example.com/robots.txt. It holds simple lines. A Disallow line names a folder crawlers should skip, such as a basket or a search page. Well-behaved crawlers obey it. Bad ones ignore it, so it is not a lock.
Blocking a page in robots.txt does not keep it out of results. If other sites link to it, the engine can still list its address, without reading it. To keep a page out of results, let the crawler read it and give it a noindex tag.
An XML sitemap is a list of the pages you want crawled, with the date each one last changed. One sitemap file can hold up to 50,000 addresses. Big sites use several. You tell the engine where it is in robots.txt and in its own tools.
A sitemap helps the engine find pages, but it does not promise they will be stored or ranked. It matters most for new sites with few links, big sites with deep pages, and sites that change often.
Noindex is a line in the page's code, or a note the site sends with the page. Use it on thank-you pages, internal search results and thin tag pages. They add nothing for a searcher.
The common mistake is putting noindex and a robots.txt block on the same page. The crawler is blocked, so it never reads the noindex, and the page can stay in results. Pick one job: block crawling, or block showing.
User-agent: *
Disallow: /basket/
Disallow: /search/
Sitemap: https://example.com/sitemap.xmlWorked example
A shop cleans up its crawl
An online shop has 2,000 products and 30,000 filter pages, like red shoes size 6 under £50. Crawlers spend their visits on the filter pages, and new products take weeks to appear.
It blocks the filter pages in robots.txt and sends a sitemap of just the 2,000 products. New products now show up in results within three days.
The words for this lesson4
Check yourself
How do you keep a page out of search results for good?
Finished this lesson?
Marking it complete also marks its 4 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.