# Robots.txt and XML Sitemaps

> Technical SEO starts with telling crawlers where to go. What robots.txt blocks, what a sitemap lists, and why noindex is the way to hide a page.

**Course:** [Search Engine Optimization (SEO)](https://learn.benibix.com/learn/marketing-seo)  
**Time:** 12 min

**After this lesson you can:**

- Read a simple robots.txt file
- Say what an XML sitemap does
- Pick noindex or robots.txt for a page

## How to tell crawlers where to go

Technical SEO is the work that makes a site easy for crawlers to read. Three tools steer them. A robots.txt file says which parts of a site they may not visit. An XML sitemap lists the pages you want found. A noindex tag tells the engine not to show one page in results, even though it may visit it.

The robots.txt file sits at the top level of a site, like example.com/robots.txt. It holds simple lines. A Disallow line names a folder crawlers should skip, such as a basket or a search page. Well-behaved crawlers obey it. Bad ones ignore it, so it is not a lock.

Blocking a page in robots.txt does not keep it out of results. If other sites link to it, the engine can still list its address, without reading it. To keep a page out of results, let the crawler read it and give it a noindex tag.

An XML sitemap is a list of the pages you want crawled, with the date each one last changed. One sitemap file can hold up to 50,000 addresses. Big sites use several. You tell the engine where it is in robots.txt and in its own tools.

A sitemap helps the engine find pages, but it does not promise they will be stored or ranked. It matters most for new sites with few links, big sites with deep pages, and sites that change often.

Noindex is a line in the page's code, or a note the site sends with the page. Use it on thank-you pages, internal search results and thin tag pages. They add nothing for a searcher.

The common mistake is putting noindex and a robots.txt block on the same page. The crawler is blocked, so it never reads the noindex, and the page can stay in results. Pick one job: block crawling, or block showing.

```ini
User-agent: *
Disallow: /basket/
Disallow: /search/

Sitemap: https://example.com/sitemap.xml
```
A small robots.txt. Crawlers skip the basket and the internal search, and the last line says where the sitemap is.

![A crawler arrives at a site and checks three things. The robots.txt file turns it away from the basket and search folders. The XML sitemap hands it a list of pages to visit. A thank-you page carries a noindex tag, so the crawler reads it but the page is kept off the results. A warning shows a page that is both blocked and noindexed: the crawler never reads the tag, so the page can still appear.](https://learn.benibix.com/diagrams/mkt-seo-robots-sitemap.svg)

### Example: A shop cleans up its crawl

An online shop has 2,000 products and 30,000 filter pages, like red shoes size 6 under £50. Crawlers spend their visits on the filter pages, and new products take weeks to appear.

It blocks the filter pages in robots.txt and sends a sitemap of just the 2,000 products. New products now show up in results within three days.

**Words in this lesson:** [Technical SEO](https://learn.benibix.com/term/marketing/technical-seo), [robots.txt](https://learn.benibix.com/term/marketing/robots-txt), [XML Sitemap](https://learn.benibix.com/term/marketing/xml-sitemap), [Noindex Tag](https://learn.benibix.com/term/marketing/noindex)

**Question:** How do you keep a page out of search results for good?

**Answer:** Let crawlers read it, and give it a noindex tag. Robots.txt only stops crawling, and the address can still be listed. Noindex keeps the page out, but only if the crawler is allowed to read it.

---

Source: https://learn.benibix.com/learn/marketing-seo/robots-txt-and-xml-sitemaps
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-20
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
