# AI Crawler

> A bot that fetches web pages for AI tools. Some train models, and some answer questions.

**Field:** Marketing & SEO  
**Category:** Digital Marketing in 2026  
**Also known as:** AI Bot, LLM Crawler, AI Web Crawler, AI User Agent

## Think of it like

It is like people reading the signs in a town. Some take notes for school, and some help a friend right now.

## Definition

An AI crawler is a bot that fetches web pages for an AI company. Some collect pages to train future models. Some fetch pages to answer searches inside an AI tool. Others visit a page only when a person asks about it. Each one has a name you can allow or block in robots.txt. Blocking the training bot does not have to block the search bot.

## Example

A news site blocks GPTBot, the crawler OpenAI uses for training, in robots.txt. It still lets in OAI-SearchBot, so its stories can show up in ChatGPT search.

## Example code

```ini
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
```

The training bot is asked to stay out. The search bot may read every page.

## Why it matters

Blocking every AI bot can drop you out of AI answers. Letting every bot in gives your pages away for training. Each choice has a cost.

## Common mistake

Treating all AI bots as one. A training bot, a search bot and a fetch for one user are three choices.

## Three principles

- **Three jobs.** Train, search, or fetch for one person.
- **Named one by one.** Each bot has its own name to match.
- **A request, not a lock.** Polite bots obey robots.txt; others may not.

## Related terms

- [robots.txt](https://learn.benibix.com/term/marketing/robots-txt)
- [Crawling](https://learn.benibix.com/term/marketing/crawling)
- [llms.txt](https://learn.benibix.com/term/marketing/llms-txt)
- [AI Search Engine](https://learn.benibix.com/term/ai/ai-search-engine)
- [Generative Engine Optimization (GEO)](https://learn.benibix.com/term/marketing/generative-engine-optimization)
- [AI Training Opt-Out](https://learn.benibix.com/term/ai/ai-training-opt-out)

![A website in the middle with a robots.txt gate in front of it. Three bots come to the gate. A training bot is turned away. A search bot is let in and its pages feed an AI search answer. A user fetch bot comes when a person asks about a page. Beside the site, an llms.txt file lists the main pages, but it is a map, not a gate.](https://learn.benibix.com/diagrams/m26-ai-crawlers.svg)

---

Source: https://learn.benibix.com/term/marketing/ai-crawler
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-16
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
