## The quick answer

**To configure robots.txt for AI crawlers, add a `User-agent` block for each named bot (`GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended`, `CCBot`) and set `Allow: /` to permit crawling or `Disallow: /` to block it, then place the file at your domain root (`https://example.com/robots.txt`).** Blocking an AI crawler removes your pages from that engine's training and retrieval corpus, which usually lowers your chance of being cited in ChatGPT, Claude, Perplexity or Gemini answers. As of 2026, this guide lists the canonical user-agent strings for each major AI crawler, the allow and block directives, the citation tradeoff of blocking, and a copy-paste robots.txt template.

> Note on data: AI crawler operators rename and split user agents over time (OpenAI alone runs three distinct bots). The strings below are accurate as of 2026; verify current user agents on each vendor's official crawler documentation before relying on a directive.

## What robots.txt controls for AI crawlers (and what it does not)

Robots.txt is a plain-text file at a domain root that tells crawlers which paths they may fetch, using `User-agent` lines to name a bot and `Allow` or `Disallow` lines to permit or forbid paths. As of 2026, robots.txt governs three different AI crawler jobs (training data collection, live retrieval for answers, and search indexing), and a single directive can hit one job without affecting the others.

Robots.txt is a voluntary standard (the Robots Exclusion Protocol), so it depends on each crawler choosing to obey it. OpenAI, Anthropic, Perplexity and Google all publish that their named crawlers respect robots.txt as of 2026.

What a robots.txt directive does and does not do:

- **Does control fetching**: a `Disallow: /` under `User-agent: GPTBot` instructs GPTBot to stop fetching every path on the host. OpenAI documents that GPTBot honors this directive.
- **Does not retroactively delete data**: blocking GPTBot in 2026 stops future fetches but does not remove pages already ingested into a prior training snapshot.
- **Does not block one engine by blocking another**: blocking `GPTBot` (OpenAI training) does not block `ClaudeBot` (Anthropic) or `PerplexityBot` (Perplexity), because each operator runs a separately named crawler.
- **Does not stop a determined non-compliant bot**: robots.txt is an instruction, not a firewall. A crawler that ignores the protocol still fetches the page; enforcement requires server-side blocking (WAF, IP rules).

For the underlying concept of how AI engines surface and cite your pages, see [AI search](/content/glossary/ai-search/index.html) and [AI citations](/content/glossary/ai-citations/index.html).

## The canonical AI crawler user agents to name in robots.txt

AI crawlers are identified in robots.txt by their exact user-agent token, which must be spelled and cased as the operator publishes it, because robots.txt user-agent matching is case-insensitive on the token but ignores any block whose token it cannot match. As of 2026, the table below lists the canonical user-agent strings for the major AI crawlers and the job each one performs.

| User agent (robots.txt token) | Operator | Primary job | Respects robots.txt (as of 2026) |
| --- | --- | --- | --- |
| `GPTBot` | OpenAI | Training data collection for GPT models | Yes |
| `OAI-SearchBot` | OpenAI | Indexing for ChatGPT search results | Yes |
| `ChatGPT-User` | OpenAI | Live fetch when a user asks ChatGPT to browse | Yes |
| `ClaudeBot` | Anthropic | Training and retrieval for Claude | Yes |
| `Claude-Web` | Anthropic | Live fetch for Claude browsing | Yes |
| `PerplexityBot` | Perplexity | Indexing for Perplexity answers | Yes |
| `Perplexity-User` | Perplexity | Live fetch on a user query | Yes (user-initiated) |
| `Google-Extended` | Google | Gemini training and grounding opt-out | Yes |
| `Googlebot` | Google | Classic Search + AI Overviews indexing | Yes |
| `CCBot` | Common Crawl | Open dataset many models train on | Yes |
| `Bytespider` | ByteDance | Training data collection | Inconsistent, verify |

Three distinctions decide your config. First, OpenAI splits its crawler into three tokens (`GPTBot` for training, `OAI-SearchBot` for search indexing, `ChatGPT-User` for live user-triggered fetches), so blocking `GPTBot` alone still lets ChatGPT retrieve your page live. Second, `Google-Extended` controls Gemini training and grounding but does not control `Googlebot`, so blocking `Google-Extended` keeps you in Google Search and AI Overviews while opting out of Gemini training. Third, `CCBot` (Common Crawl) feeds many third-party models indirectly, so a model you never named can still train on your pages through the Common Crawl dataset.

## Allow or block: the directive syntax for each AI crawler

A robots.txt block pairs one `User-agent` token with one or more `Allow` or `Disallow` path rules, and the most specific matching rule wins for a given path. As of 2026, allowing an AI crawler requires either no `Disallow` rule for that token or an explicit `Allow: /`, while blocking requires `Disallow: /` under the exact token.

To allow every AI crawler (default for maximum citation eligibility), name no AI bot with a `Disallow`, or be explicit:

```plaintext
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /
```

To block one AI crawler while allowing the rest, target only that token:

```plaintext
User-agent: GPTBot
Disallow: /
```

To block all AI training but keep live retrieval and search, block the training bots and leave the user-fetch bots allowed:

```plaintext
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /
```

To block a single sensitive path for an AI crawler while allowing the rest of the site, use a path-scoped `Disallow`:

```plaintext
User-agent: GPTBot
Disallow: /private/
Allow: /
```

Two syntax rules prevent silent failures. First, the `User-agent` token must match exactly (`GPTBot`, not `GptBot` spelled with a missing capital intent, not `OpenAI`); an unrecognized token means the block is ignored and the bot falls back to the wildcard `User-agent: *` rule. Second, a wildcard `User-agent: *` with `Disallow: /` blocks every crawler including Googlebot, so never use the wildcard to target only AI bots.

## The citation tradeoff: what blocking AI crawlers costs you

Blocking an AI crawler in robots.txt removes your pages from that engine's retrieval and training corpus, which directly lowers your eligibility to be cited in that engine's answers. As of 2026, an engine cannot cite a source it was instructed not to fetch, so a `Disallow: /` under `PerplexityBot` makes Perplexity citations of your domain effectively impossible.

The tradeoff splits by crawler job:

- **Blocking training bots** (`GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`) keeps your content out of future model weights but does not remove pages already in a 2026 or earlier training snapshot. The cost is reduced presence in parametric (memorized) answers over time.
- **Blocking retrieval and search bots** (`OAI-SearchBot`, `PerplexityBot`, `ChatGPT-User`) is the larger citation cost, because these bots feed the live, cited answers users see today. Blocking them removes you from the source list that ChatGPT search and Perplexity attach to answers.
- **Blocking `Googlebot`** is the most expensive directive: it removes you from Google Search, Google AI Overviews and AI Mode at once, so never use it to manage AI exposure.

The decision is rarely all-or-nothing. Publishers with paid or proprietary content often block training bots (`GPTBot`, `CCBot`) to protect the corpus while allowing retrieval bots (`OAI-SearchBot`, `PerplexityBot`) so the brand still earns cited answers. Most B2B SaaS and marketing sites that want AI visibility should allow all named crawlers, because every block narrows the set of engines that can cite the page. For why citation eligibility matters more than raw rankings, see [why your Google rankings do not translate into AI citations](/content/blog/google-rankings-vs-ai-citations/index.html) and [what GEO is](/content/blog/what-is-geo/index.html).

## A copy-paste robots.txt template for AI crawlers (2026)

This template allows all major AI crawlers (maximum citation eligibility) while keeping classic search intact, and includes commented block lines you can uncomment to restrict training only. As of 2026, place the file at `https://yourdomain.com/robots.txt` and serve it with a `200` status and `text/plain` content type.

```plaintext
# robots.txt: AI crawler configuration (updated 2026)
# Default: allow all AI crawlers for maximum citation eligibility.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

# Keep classic search open (do not block Googlebot to manage AI).
User-agent: Googlebot
Allow: /

# Wildcard fallback for all other bots.
User-agent: *
Allow: /

# --- To block AI training only, comment out the Allow blocks above
# --- for these tokens and uncomment the lines below:
# User-agent: GPTBot
# Disallow: /
# User-agent: ClaudeBot
# Disallow: /
# User-agent: CCBot
# Disallow: /
# User-agent: Google-Extended
# Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml
```

Three deployment checks prevent a dead file. First, robots.txt must sit at the host root, not a subfolder; a file at `/blog/robots.txt` is ignored. Second, each subdomain needs its own robots.txt (`blog.example.com/robots.txt` is separate from `example.com/robots.txt`). Third, a `Sitemap:` line is global, not tied to any `User-agent` block, and helps every compliant crawler discover your pages.

## How to verify AI crawlers are obeying your robots.txt

Verifying AI crawler behavior requires reading your server access logs for each named user agent and confirming the fetch pattern matches your robots.txt directive, because robots.txt is advisory and only logs prove compliance. As of 2026, a compliant crawler that you allowed appears in logs fetching pages, while a crawler you blocked should stop appearing after its next robots.txt re-read.

Verification steps in order:

1. **Grep your access logs** for each user-agent token: search for `GPTBot`, `ClaudeBot`, `PerplexityBot`, `OAI-SearchBot` and `Google-Extended` in the user-agent field of your web server or CDN logs.
2. **Confirm allowed bots fetch**: an allowed `GPTBot` should appear fetching multiple paths with `200` responses. Zero GPTBot hits over weeks can mean a misconfigured block or simply low crawl priority.
3. **Confirm blocked bots stop**: after a `Disallow: /`, a compliant crawler re-reads robots.txt (often within 24 hours) then stops fetching. Continued fetches indicate either a caching delay or a non-compliant bot.
4. **Watch for non-compliant fetchers**: a bot ignoring `Disallow` requires server-side enforcement (WAF rule, IP block, or a CDN bot-management ruleset), because robots.txt cannot stop it.

Logs answer "did the crawler fetch my page", not "did the engine cite my page". A crawler can fetch your content and still not cite it in an answer, because fetching is necessary but not sufficient for citation. To measure the second question (which engines actually cite your domain, per query and per country), you need on-engine citation data rather than crawl logs; see [how to measure AI share of voice](/content/blog/measure-ai-share-of-voice/index.html).

## From crawl access to citation: closing the loop

Allowing AI crawlers in robots.txt makes a page eligible to be cited, but eligibility is not citation, so the next question is which engines actually cite your domain and which competing sources they cite instead. As of 2026, robots.txt is the permission layer and on-engine citation tracking is the measurement layer, and the two answer different questions.

How the two layers differ:

| Question | robots.txt + logs | On-engine citation tracking |
| --- | --- | --- |
| Can GPTBot fetch my pages? | Yes (directive + logs) | No |
| Did ChatGPT actually fetch a page? | Yes (server logs) | No |
| Does Perplexity cite my domain for a query? | No | Yes |
| Which sources does AI cite per country? | No | Yes |
| Who is the contact behind a cited source? | No | Yes |

Getspotted is the citation layer, not a crawler config tool: one `/search` call returns the sources Google, ChatGPT, AI Overview, Perplexity, Claude and Gemini cite for a query, per country, plus the contacts behind each cited source. Used after you have allowed the AI crawlers above, robots.txt opens the door and Getspotted reports who walked through it. Browse [which sources AI cites](/content/sources/index.html) for live examples, or read [what generative engine optimization is](/content/glossary/generative-engine-optimization/index.html) for the strategy that turns crawl access into citations.

If you are a developer wiring citation data into your own monitor or AI agent, the [API docs](/content/docs/index.html) show the `/search` and contact endpoints. If you want a faster path to being cited once crawlers can reach you, see [how to get cited in AI answers](/content/get-cited-in-ai-answers/index.html).

## FAQ

### How do I allow GPTBot in robots.txt?

To allow GPTBot, add a `User-agent: GPTBot` block with `Allow: /` (or simply name no `Disallow` for GPTBot), then place robots.txt at your domain root. As of 2026, OpenAI documents that GPTBot honors robots.txt, so an `Allow: /` directive permits GPTBot to fetch every path for GPT training data collection.

### What is the ClaudeBot user agent string?

`ClaudeBot` is Anthropic's crawler token used for training and retrieval, and `Claude-Web` is the separate token for live browsing fetches. As of 2026, name `ClaudeBot` exactly (case-insensitive on the token) in a `User-agent: ClaudeBot` block; an unrecognized spelling makes the block fall back to the wildcard `User-agent: *` rule.

### How do I give PerplexityBot crawl access?

Give PerplexityBot crawl access with a `User-agent: PerplexityBot` block and `Allow: /`, and allow `Perplexity-User` too if you want live user-initiated fetches. As of 2026, PerplexityBot indexes pages for Perplexity answers, so allowing it is required for your domain to appear in Perplexity's cited source list.

### Should I block or allow AI crawlers?

Most B2B SaaS and marketing sites that want AI visibility should allow all named AI crawlers, because every block removes the domain from that engine's citable corpus. Block AI crawlers only to protect paid or proprietary content; in that case block training bots (`GPTBot`, `CCBot`) while allowing retrieval bots (`OAI-SearchBot`, `PerplexityBot`) to keep earning cited answers.

### What does Google-Extended do in robots.txt?

`Google-Extended` controls whether Google uses your pages for Gemini training and grounding, separately from `Googlebot` which controls Search and AI Overviews indexing. As of 2026, a `Disallow: /` under `Google-Extended` opts you out of Gemini training while keeping you in Google Search and AI Overviews, because the two tokens govern different jobs.

### Does blocking AI crawlers remove my content from ChatGPT?

Blocking AI crawlers in robots.txt stops future fetches but does not remove content already ingested into a prior training snapshot. As of 2026, a `Disallow: /` under `GPTBot` prevents new GPT training fetches, yet pages crawled before the block can remain in earlier model weights until that model is retrained.

### Where does robots.txt for AI crawlers need to be placed?

robots.txt must sit at the host root, served as `https://yourdomain.com/robots.txt` with a `200` status and `text/plain` content type. As of 2026, a file in a subfolder (`/blog/robots.txt`) is ignored, and each subdomain needs its own robots.txt, because crawlers fetch the file only from the exact host root they are crawling.

### Can robots.txt fully stop a non-compliant AI bot?

No, robots.txt is an advisory instruction, not a firewall, so a crawler that ignores the Robots Exclusion Protocol still fetches your pages. As of 2026, stopping a non-compliant AI bot requires server-side enforcement (a WAF rule, IP block, or CDN bot-management ruleset); robots.txt only governs crawlers that choose to obey it.

GEOAI visibilityrobots.txtAI crawlerstechnical SEO

A

Written by

Alexis Maresca

Cofounder, Getspotted · GEO & AI visibility expert

Alexis Maresca is a cofounder of Getspotted and a specialist in Generative Engine Optimization (GEO). He helps brands and agencies understand which sources AI engines like ChatGPT, Perplexity, Claude and Google AI Overviews cite, and how to get featured in AI-generated answers.