Should I block GPTBot in robots.txt? Training crawlers vs. AI search crawlers
Blocking GPTBot is a choice about AI training, not about being found. The crawlers that put you in AI search answers are different, and blocking those by mistake is a common reason businesses disappear from ChatGPT.
Two kinds of AI crawlers
- Training: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (a Google control token for Gemini). They collect content for models (OpenAI, Anthropic).
- Search and user fetches: OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, Bingbot and Googlebot. They fetch pages so assistants can answer and cite.
What blocking does
Blocking a training crawler keeps your content out of future model training. It does not remove you from ChatGPT search. Blocking a search crawler does remove you from the answers that depend on it.
A safe default for a local business
Allow the search crawlers. Decide on training crawlers as a matter of preference. AEO Pro never changes your training-bot choice and reports those bots separately, without scoring them.
Check where you stand
Get a free AI check: your AI-readiness score, the questions we asked and which AI crawlers your site allows. Essentials ($49/month) fixes your homepage; Complete ($199/month) covers every page. Cancel anytime, 30-day money-back guarantee.
Questions
Does blocking GPTBot hurt my visibility in ChatGPT?
Not in ChatGPT search, which uses OAI-SearchBot. GPTBot is for training.
Should I block ClaudeBot?
It is your choice. ClaudeBot is Anthropic’s training crawler; Claude-SearchBot and Claude-User are the ones that fetch pages for answers.