📡AI Crawler
Also known as AI Bot, LLM Crawler, AI Web Crawler, AI User Agent
A bot that fetches web pages for AI tools. Some train models, and some answer questions.
Think of it like
It is like people reading the signs in a town. Some take notes for school, and some help a friend right now.
Example
A news site blocks GPTBot, the crawler OpenAI uses for training, in robots.txt. It still lets in OAI-SearchBot, so its stories can show up in ChatGPT search.
Why it matters
Blocking every AI bot can drop you out of AI answers. Letting every bot in gives your pages away for training. Each choice has a cost.
Definition
Example code
ini
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Common mistake
Key takeaways
- 1Three jobs. Train, search, or fetch for one person.
- 2Named one by one. Each bot has its own name to match.
- 3A request, not a lock. Polite bots obey robots.txt; others may not.