Robots.txt Checker
Parse and validate your robots.txt against 20+ AI crawlers — GPTBot, PerplexityBot, ClaudeBot, GrokBot, Gemini, Bing/Copilot and more. Instantly see who you're blocking and exactly how to fix it.
Check any domain — results in seconds
About this tool
How do I block a specific AI bot in robots.txt?
Add a new section with the bot's user agent name and Disallow: /. For example, to block GPTBot: User-agent: GPTBot Disallow: / This prevents that specific bot from crawling any page on your site.
How do I allow AI bots while blocking others?
First create a wildcard block, then add explicit allows for the bots you want: User-agent: * Disallow: / User-agent: GPTBot Allow: / User-agent: PerplexityBot Allow: / This blocks everything by default but explicitly allows specific crawlers.
What's the difference between blocking training vs. inference crawlers?
Some bots (like GPTBot and CCBot) are used for training AI models. Others (like OAI-SearchBot and PerplexityBot) crawl for real-time retrieval. Blocking training bots means your content won't appear in future model knowledge. Blocking inference bots means AI won't cite you in real-time answers. Most sites want to block training but allow inference.
Does robots.txt affect how often Google/Gemini cites me?
Googlebot is what Google uses for its search index, which Gemini draws from. If you block Googlebot, Google can't index your pages and Gemini can't cite content it hasn't indexed. The Google-Extended token specifically controls AI training usage — you can block Google-Extended without affecting regular Google indexing.
Is robots.txt enforceable — do AI bots have to obey it?
Reputable AI companies (OpenAI, Anthropic, Google, Perplexity, xAI) all honor robots.txt. Disreputable scrapers may not. For the major AI platforms that drive actual citations, robots.txt is a reliable and respected signal.