Skip to content
BENIBIX Learn

AI Crawler

Also known as AI Bot, LLM Crawler, AI Web Crawler, AI User Agent

A bot that fetches web pages for AI tools. Some train models, and some answer questions.

Think of it like

It is like people reading the signs in a town. Some take notes for school, and some help a friend right now.

Example

A news site blocks GPTBot, the crawler OpenAI uses for training, in robots.txt. It still lets in OAI-SearchBot, so its stories can show up in ChatGPT search.

Why it matters

Blocking every AI bot can drop you out of AI answers. Letting every bot in gives your pages away for training. Each choice has a cost.

Definition

An AI crawler is a bot that fetches web pages for an AI company. Some collect pages to train future models. Some fetch pages to answer searches inside an AI tool. Others visit a page only when a person asks about it. Each one has a name you can allow or block in robots.txt. Blocking the training bot does not have to block the search bot.

Example code

ini
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
The training bot is asked to stay out. The search bot may read every page.
Three bots, three jobs. The gate can turn one away and let another in.

Common mistake

Treating all AI bots as one. A training bot, a search bot and a fetch for one user are three choices.

Key takeaways

  1. 1
    Three jobs. Train, search, or fetch for one person.
  2. 2
    Named one by one. Each bot has its own name to match.
  3. 3
    A request, not a lock. Polite bots obey robots.txt; others may not.