AI Crawlers and robots.txt: Let ChatGPT Read Your Site

robots.txt is a small text file at the root of your website (yourdomain.com/robots.txt) that tells automated visitors, called crawlers, which parts of the site they may read. AI products use their own crawlers. If your robots.txt blocks them, either on purpose or because a plugin or default setting did it for you, your site may simply not be available to ChatGPT, Claude or Perplexity, and no amount of good content will fix that.

This is the second item in the brand foundations checklist in Lumen ("AI crawlers allowed"). Here is what each crawler does, how to check your file, and how to fix it. It pairs with llms.txt, which guides models once they can reach you.

The crawlers that matter

Each company runs different bots for different jobs, and the distinction matters. A bot that builds a search index is what gets you recommended in answers; a bot that collects training data is a separate decision you may treat differently.

1. 01
   OAI-SearchBot (OpenAI): Indexes sites so they can appear in ChatGPT search. This is the most important one: if you block it, your site does not show up in ChatGPT search results.
2. 02
   GPTBot (OpenAI): Collects content that may be used to train OpenAI models. Blocking it does not remove you from ChatGPT search, which relies on OAI-SearchBot.
3. 03
   ChatGPT-User (OpenAI): Visits a page when a person asks ChatGPT to open or read it. Blocking it breaks "read this link" requests.
4. 04
   ClaudeBot (Anthropic): Anthropic's crawler for Claude. Anthropic documents separate bots for search and for user-initiated visits, so check its current documentation before writing specific rules.
5. 05
   PerplexityBot (Perplexity): Builds the index behind Perplexity answers and its citations.
6. 06
   Google-Extended (Google): A robots.txt token that controls whether Google may use your content for Gemini. It does not affect your normal Google Search ranking, which is handled by Googlebot.

Should you allow them?

If you want to be recommended when people ask AI assistants about your category, yes, at least the search crawlers. Being invisible to them is the same as not being in the index. Some publishers block training crawlers for licensing reasons, and that is a legitimate choice, but make it deliberately. The common failure is the accidental block: a blanket rule that nobody realised was shutting out every bot.

How to check your robots.txt

• Open https://yourdomain.com/robots.txt in your browser. If it says 404, there is no file, which means nothing is blocked (and Lumen counts it as passing).
• Look for a rule that blocks everything, like the one below, either under User-agent: * or under a specific AI bot.
• Search the file for the bot names above. A group with "Disallow: /" under any of them is a full block.
• Remember that a specific group overrides the generic one: allowing OAI-SearchBot by name works even if User-agent: * is restrictive.

User-agent: *
Disallow: /

That two-line snippet blocks every crawler that does not have its own rule, AI bots included. It is common on staging sites and gets copied to production by mistake.

What a healthy file looks like

Add a group for each AI bot you want to welcome and keep your existing rules for everything else. This is an example to adapt, not to paste blindly:

# Allow AI crawlers (everything else stays as it is)
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

# Rules for everyone else
User-agent: *
Disallow: /admin/

Sitemap: https://yourdomain.com/sitemap.xml

Where to edit it on common platforms

• WordPress: Settings, Reading, and make sure "Discourage search engines from indexing this site" is unchecked, since it adds a blanket block. SEO plugins such as Yoast or Rank Math have a robots.txt editor.
• Shopify: edit the robots.txt.liquid template in your theme code (Online Store, Themes, Edit code).
• Wix: SEO tools, Robots.txt editor.
• Cloudflare: check the AI crawler controls under Security. Depending on your settings, Cloudflare can block AI bots or add its own robots.txt rules even if your file looks fine.
• Custom site: edit the robots.txt file in your public folder and deploy.

Check again after the change

Reload your robots.txt to confirm the new version is live (CDNs sometimes cache the old one for hours). In Lumen, open Actions, Brand foundations and press Recheck: the item turns green when none of OAI-SearchBot, GPTBot, ClaudeBot, Google-Extended or PerplexityBot is blocked. Then follow the effect on your visibility in GEO Actions, keeping in mind that crawlers take days or weeks to revisit a site.