Is my website blocking ChatGPT? How to check robots.txt, firewalls and JavaScript
A site can look perfect in your browser and still be closed to AI assistants. There are three places to check: robots.txt, your firewall or security plugin, and whether your text is in the HTML at all.
1. robots.txt
Open yoursite.com/robots.txt. Look for “Disallow: /” under User-agent: * or under the AI search crawlers: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), and Bingbot and Googlebot. GPTBot and ClaudeBot are training crawlers; blocking them does not remove you from search answers.
2. Firewall, CDN and security plugins
Bot protection on a CDN or a WordPress security plugin can answer AI crawlers with a 403 or a challenge page. robots.txt can say “allowed” while the firewall still blocks. Check the plugin or CDN settings for rules that block unknown or AI bots.
3. Content without JavaScript
View the page source (not the inspector). If your services, bios and prices are not in it, many AI crawlers will not see them either.
Check it automatically
The free AI check fetches your site as an AI crawler and as a browser, compares the two and scores crawler access as part of the 11 checks in the AI-readiness checklist.
Check where you stand
Get a free AI check: your AI-readiness score, the questions we asked and whether AI crawlers can read your site. Essentials ($49/month) fixes your homepage; Complete ($199/month) covers every page. Cancel anytime, 30-day money-back guarantee.
Questions
Does blocking GPTBot block ChatGPT search?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages when a user asks. Block or allow them separately.
My robots.txt allows everything. Can I still be blocked?
Yes. A firewall, CDN bot protection or security plugin can block AI crawlers regardless of robots.txt.