Free tool

robots.txt Checker for AI Crawlers

Paste a site's robots.txt and a path. You'll see whether each of 13 AI crawlers from OpenAI, Anthropic, Google, Apple, Perplexity, Meta, and Common Crawl may fetch it, whether it's used for training, search or a user's request, and the exact line that decided.

Paste any site's file from https://its-domain/robots.txt. It's checked in your browser; nothing is fetched or sent.

Training
6 of 6 blocked
Search
0 of 3 blocked
User fetch
0 of 4 blocked
  • BlockedGPTBotOpenAITraining
    Line 8: Disallow: / (its own group)

    Crawls content that may be used to train OpenAI's models. Docs

  • AllowedOAI-SearchBotOpenAISearch
    Line 12: Allow: / (the * group)

    Decides whether your pages can appear in ChatGPT search. Docs

  • AllowedChatGPT-UserOpenAIUser fetch
    Line 12: Allow: / (the * group)

    Fetches pages when a ChatGPT user asks; OpenAI says robots.txt rules may not apply. Docs

  • BlockedClaudeBotAnthropicTraining
    Line 8: Disallow: / (its own group)

    Collects content that may be used for model training. Docs

  • AllowedClaude-SearchBotAnthropicSearch
    Line 12: Allow: / (the * group)

    Indexes content for Claude's search results. Docs

  • AllowedClaude-UserAnthropicUser fetch
    Line 12: Allow: / (the * group)

    Fetches pages when a Claude user asks. Docs

  • BlockedGoogle-ExtendedGoogleTraining
    Line 8: Disallow: / (its own group)

    Controls use for Gemini training and grounding; doesn't affect Google Search. Docs

  • BlockedApplebot-ExtendedAppleTraining
    Line 8: Disallow: / (its own group)

    Controls use of Applebot's crawl for Apple's generative AI training; it doesn't crawl. Docs

  • AllowedPerplexityBotPerplexitySearch
    Line 12: Allow: / (the * group)

    Surfaces and links sites in Perplexity's search results. Docs

  • AllowedPerplexity-UserPerplexityUser fetch
    Line 12: Allow: / (the * group)

    Fetches pages a user asks for; Perplexity says it generally ignores robots.txt. Docs

  • BlockedMeta-ExternalAgentMetaTraining
    Line 8: Disallow: / (its own group)

    Crawls for uses such as training AI models or indexing content. Docs

  • AllowedMeta-ExternalFetcherMetaUser fetch
    Line 12: Allow: / (the * group)

    Fetches links a user asks for; Meta says it may bypass robots.txt. Docs

  • BlockedCCBotCommon CrawlTraining
    Line 8: Disallow: / (its own group)

    Builds Common Crawl's open web corpus, widely used to train models. Docs

Questions

Does blocking GPTBot keep my site out of ChatGPT search?
No. OpenAI uses separate crawlers: GPTBot collects content that may be used for training, while OAI-SearchBot decides whether your pages can appear in ChatGPT search. Blocking one leaves the other as it was. Anthropic separates them the same way, with ClaudeBot for training and Claude-SearchBot for search.
Why doesn't a crawler follow my User-agent: * rules?
Under RFC 9309 a crawler obeys the group that names it and ignores the * group. If you give GPTBot its own group to block training, the private paths in your * group no longer apply to it, so repeat them there.
Which rule wins when Allow and Disallow both match?
The longest matching path wins, and when an Allow and a Disallow are the same length, Allow wins. The checker shows the deciding line for every crawler, so you can see which one applied.
Do user-triggered fetchers follow robots.txt?
Not always. Fetchers such as ChatGPT-User act when a person asks the assistant to open a page, and their operators say robots.txt rules may not apply to them. The checker marks those crawlers when your file blocks them.
Is my robots.txt sent anywhere?
No. The file is checked in your browser and nothing is fetched or sent. Paste it from https://the-site/robots.txt.

Read more