Back
How to block AI crawlers
What you are actually blocking when you block AI crawlers AI crawlers are automated clients used by AI companies to collect public web content for different jobs: model development, AI-search indexing, product search or a one-off page fetch requested by a user. Those jobs are not interchangeable, so blocking every “AI bot” with one blanket […]
What you are actually blocking when you block AI crawlers
AI crawlers are automated clients used by AI companies to collect public web content for different jobs: model development, AI-search indexing, product search or a one-off page fetch requested by a user. Those jobs are not interchangeable, so blocking every “AI bot” with one blanket rule can remove useful visibility as well as unwanted collection.
The practical goal is to separate training, search and user-request traffic, publish the right robots.txt rules for documented crawlers, then enforce your policy against spoofed or undeclared automation that a text file cannot control.
Quick answer
To block AI crawlers, use four steps: identify the crawlers actually visiting your site, decide a policy by purpose, publish crawler-specific robots.txt rules, then verify and enforce those rules at the traffic level. A common middle path is to block dedicated training crawlers while keeping AI-search crawlers allowed.
Step 1: See which AI crawlers actually visit your site
Start with your logs, not a copied bot list. AI crawler inventories change quickly, and not every site sees the same agents. CDN, reverse-proxy, WAF or server access logs tell you which automated clients are actually requesting your content.
Search recent requests for published crawler tokens such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, CCBot, Applebot, Amazonbot and Meta-ExternalAgent. Then record what each one is doing rather than stopping at the name.
- Frequency: is the crawler visiting hourly, daily or only occasionally?
- Coverage: is it reading a few articles, your entire blog, product pages, PDFs or API endpoints?
- Volume: how many requests does it make, and does the pattern create meaningful origin load?
- Purpose: does the operator document the token as model training, search/indexing or a user-triggered fetch?
- Business value: can the crawler send referrals, citations or discovery, or does it only collect content?
Actionable first step
Pull the last 7-30 days of request logs and group traffic by crawler token and requested path. Do not block yet. Build a short inventory of “seen”, “wanted”, “unwanted” and “uncertain”. No log access? Start with the free site check at /check-your-site/.
Important: a user-agent string is only a claim. Any scraper can call itself GPTBot, Googlebot or ClaudeBot. Use the name to build your inventory, then verify identity before you create trusted allow rules.
Step 2: Set a policy per purpose – not one blanket AI rule
The phrase AI crawler hides several different business decisions. The cleanest policy starts with purpose, not vendor name.
01.
Dedicated model-development crawlers. OpenAI documents GPTBot for content that may be used in training, while Anthropic documents ClaudeBot for model development. If you do not want future public content collected for that purpose, these are direct opt-out candidates.
02.
AI-search crawlers. OAI-SearchBot, Claude-SearchBot and PerplexityBot are used to surface or improve search results. Blocking them can reduce AI-search visibility, citations or referral opportunities. Decide based on measurable discovery value, not fear of the word “AI”.
03.
Control tokens rather than crawlers. Google-Extended and Applebot-Extended do not represent separate web-crawling clients. They tell Google or Apple how already crawled content may be used for AI-related purposes. Blocking these tokens can preserve normal search crawling.
04.
User-request fetchers. ChatGPT-User, Claude-User, Perplexity-User and Amzn-User may fetch a page because a person asked for it. Their robots.txt behaviour is not identical: OpenAI says rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores robots.txt; Amazon says Amzn-User may not follow all directives; Anthropic says its bots honour robots.txt.
05.
Open-data or mixed-purpose crawlers. CCBot builds the Common Crawl dataset, while Amazonbot and Meta- ExternalAgent have broader documented product or AI uses. These need a more deliberate decision because blocking can affect more than one downstream use.
A sensible default for many publishers
Refuse training where the operator provides a clean training opt-out, preserve search/indexing where it creates value, and treat user-request agents separately. Then review mixed-purpose crawlers against your own content, licensing and traffic goals.
For deeper bot-specific decisions, see GPTBot, ClaudeBot and PerplexityBot.
Step 3: Block AI crawlers with robots.txt using verified tokens
Once the policy is clear, publish it in /robots.txt at the root of each host or subdomain you want to control. Use exact, documented tokens. Do not paste a giant third-party blocklist you cannot maintain.
AI crawler robots.txt tokens verified on 20 August 2026:
| Operator | Token | What it does / what blocking changes |
|---|---|---|
| 01OpenAI | GPTBot | Model-development crawler. Disallowing it signals that site content should not be used to train OpenAI generative AI foundation models. |
| 02OpenAI | OAI-SearchBot | Search crawler for surfacing websites in ChatGPT search. Allow it if ChatGPT search visibility matters. |
| 03OpenAI | ChatGPT-User | User-initiated fetcher, not an automatic web crawler. OpenAI says robots.txt rules may not apply. |
| 04Anthropic | ClaudeBot | Model-development crawler. Blocking it signals that future materials should be excluded from Anthropic model-training datasets. |
| 05Anthropic | Claude-SearchBot | Search crawler used to improve Claude search result quality and visibility. |
| 06Anthropic | Claude-User | User-directed fetcher. Anthropic says its bots honour robots.txt and allows this agent to be controlled separately. |
| 07Perplexity | PerplexityBot | Search/indexing crawler designed to surface and link websites in Perplexity results. Not used to train foundation models. |
| 08Perplexity | Perplexity-User | User-initiated fetcher. Perplexity says it generally ignores robots.txt because the fetch was requested by a user. |
| 09Google | Google-Extended | Control token, not a separate crawler. Controls AI training and grounding uses; blocking it does not affect inclusion or ranking in Google Search. |
| 10Apple | Applebot-Extended | Control token, not a crawler. Opts content out of Apple foundation-model training while normal Applebot search crawling can continue. |
| 11Common Crawl | CCBot | Open-data crawler. It honours robots.txt, but blocking it also blocks collection for non-training uses of the Common Crawl dataset. |
| 12Meta | Meta-ExternalAgent | Meta crawler documented for uses including foundation-model training and direct indexing for products. |
| 13Meta | Meta-ExternalFetcher | User-request fetcher. Meta documents that it may bypass robots.txt for user-requested fetching. |
| 14Amazon | Amazonbot | Crawler used to improve Amazon products and services; Amazon says collected content may be used to train Amazon AI models. |
| 15Amazon | Amzn-SearchBot | Search crawler for Amazon search experiences such as Alexa. Amazon says it is not used for generative-AI model training. |
| 16Amazon | Amzn-User | User-initiated fetcher for live answers. Amazon says it may not follow all robots.txt directives. |
Source check: OpenAI · Anthropic · Perplexity · Google · Apple · Common Crawl · Meta · Amazon.
A “no training, keep search visibility” policy might look like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Amzn-SearchBot
Allow: /Do not copy this blindly. CCBot is an open-data crawler rather than a clean training-only control, and mixed-purpose agents such as Amazonbot or Meta-ExternalAgent need a business decision of their own. User-request fetchers are deliberately absent because operator documentation says robots.txt may not always control them.
Also remember what robots.txt cannot do: it does not delete material already collected, revoke third-party copies or prevent an undeclared scraper from requesting a public URL.
Step 4: Enforce the policy when robots.txt is not enough
robots.txt is a policy signal for clients that identify themselves and honour the Robots Exclusion Protocol. It is not authentication, authorisation or a network block. If a scraper ignores the file, spoofs a trusted crawler name or never declares itself, the request can still reach your public content.
That leaves four enforcement jobs.
- Verify crawlers you want to trust. Match declared agents against operator-published IP ranges or reverse-DNS guidance where available. OpenAI, Perplexity, Apple, Common Crawl and Amazon all publish verification data; Anthropic publishes crawler IP information too.
- Do not trust names alone. A request claiming to be GPTBot or Googlebot should not inherit allow-list privileges until its source is verified.
- Detect undeclared automation by behaviour. Evaluate behaviour, browser, network, request sequence and endpoint activity so a silent scraper cannot become trusted simply by using a normal browser user-agent.
- Apply policy at the perimeter. Allow authorised crawlers, limit automation that has conditional value, and block high-confidence harmful collection before it consumes origin resources or reaches sensitive application paths.
This is also where AI crawling overlaps with content theft. The commercial risk is not the token in the user-agent. It is unauthorised automated access to content, pricing, research, product data or other material you have decided should not be collected at machine scale.
Security rule
Treat robots.txt as the published policy. Treat verified identity and traffic-level controls as enforcement. If access must truly be restricted, public robots directives are never a substitute for authentication or authorisation.
After you publish: verify that the policy is doing what you intended
After changing robots.txt or perimeter rules, verify the result instead of assuming the edit worked. The objective is not “fewer AI bots”. It is the right automation allowed, the unwanted automation reduced, and no accidental loss of valuable search visibility.
- Confirm the file: fetch /robots.txt publicly and check the exact host, token spelling and path rules.
- Watch compliant crawler volume: compare request counts before and after the change. Some operators need time to pick up a new robots policy.
- Protect AI-search visibility: if you chose to keep search crawlers, confirm OAI-SearchBot, Claude-SearchBot, PerplexityBot or other approved agents still reach allowed pages.
- Protect normal SEO: make sure Googlebot and other search crawlers are not caught by a broad wildcard rule.
- Look for spoofing: if requests using a blocked crawler name continue from unrelated networks, verify whether they are impostors rather than concluding the documented operator ignored your file.
- Measure business impact: monitor origin load, content-access volume and referral traffic so the policy can be adjusted with evidence.
Maintenance cadence
Review AI crawler policy monthly. Operator tokens, product roles and robots.txt behaviour change faster than normal SEO documentation, so the updated date on this page should move whenever the token table is re-verified.
How ADPAL handles the enforcement layer
ADPAL adds Advanced detection at the managed reverse proxy. It evaluates behaviour, browser, network, request sequence and endpoint activity before requests reach protected application workflows. Policies can preserve verified search crawlers and other approved automation while suspicious traffic is challenged, limited or blocked according to policy.
That gives you a practical second layer for AI crawler control: robots.txt communicates what documented crawlers should do; perimeter enforcement handles spoofed, undeclared or non-compliant automation that a user-agent list cannot reliably identify.
Point your DNS at the managed reverse proxy – live in hours, then a short monitoring period before enforcing. CMS-integrated deployment is available through hosting partners.
Cookieless, no cross-site tracking profiles, EU (Frankfurt) data residency.
Use policy, then enforcement
Start in monitoring mode. Verify the crawlers you want to keep, review the content and endpoints being collected, then apply allow, limit or block policies with evidence rather than a blanket “AI = bad” rule.
Three AI crawler blocking mistakes to avoid
Blocking every AI-related token. Training, search and user-request agents do different jobs. A blanket block can throw away citation and referral opportunities you actually wanted.
Assuming a crawler name proves identity. User-agent strings are easy to spoof. Verify important allow-listed bots against operator infrastructure before you trust them.
Publishing robots.txt and stopping there. The file only controls clients that honour it. Undeclared scraping and spoofed automation need traffic-level detection and enforcement.
Frequently asked questions
Does blocking AI crawlers hurt Google rankings?
Not if you target the correct tokens. Google states that Google-Extended does not affect inclusion or ranking in Google Search. Apple likewise documents Applebot-Extended as a data-use control separate from normal Applebot search crawling. The SEO risk comes from accidentally blocking Googlebot or other genuine search crawlers with broad rules.
Does blocking GPTBot stop my site appearing in ChatGPT Search?
No, the controls are separate. OpenAI documents GPTBot for model-development crawling and OAI-SearchBot for ChatGPT search. You can disallow GPTBot while allowing OAI-SearchBot if your goal is to refuse training collection while keeping ChatGPT search visibility.
Should I block all AI bots?
Usually not by default. Separate training crawlers, search crawlers, control tokens and user-request fetchers first. Then decide based on content licensing, infrastructure cost, referral value and the purpose of each agent. Mixed-purpose crawlers deserve an explicit business decision rather than a copied blocklist.
Can robots.txt stop AI scraping completely?
No. robots.txt works for compliant crawlers. It does not authenticate the client or prevent an undeclared scraper from requesting a public URL. Some documented user-request fetchers may also ignore or not always follow robots.txt. Use verification and traffic-level controls when you need enforcement.
How do I know whether an AI crawler in my logs is real?
Treat the user-agent as a label, then verify the source. Several operators publish IP ranges or reverse-DNS guidance for their bots, including OpenAI, Perplexity, Apple, Common Crawl, Amazon and Anthropic. A crawler claiming a trusted name from unrelated infrastructure should not be automatically allow-listed.
How often should I update AI crawler rules?
Review them monthly, and immediately after a major operator change. AI companies regularly add or split agents by purpose. Keep a dated crawler-policy inventory, re-check primary operator documentation, and verify that wanted search agents still reach the pages you intend to expose.
Take control of AI crawler access without disappearing from useful search
Start with your own traffic. Identify the crawlers reaching valuable content, decide which uses you want to support, then verify whether robots.txt is enough or whether the same URLs are still being collected by spoofed or undeclared automation.
Next step
See which automated traffic is reaching your site before you add more blocklists. Use the evidence to preserve useful AI-search access while controlling collection that creates no value for your business.