AI crawlers: understand, verify and
control automated access

AI crawlers collect public web content for model development, AI search, retrieval and other automated services. Some can support discovery and referrals, while others extract content, consume resources or only claim to be trusted crawlers.

Use this hub to understand AI crawler traffic, verify who is actually accessing your site and decide which crawlers to allow, limit or block.

. Guides

Choose your starting point

Start with the practical blocking guide, or go directly to crawler verification and modern AI bot detection

. Start here

How to block AI crawlers

Identify the AI crawlers visiting your site, separate training from search and user-request traffic, set the right policy and enforce it without accidentally blocking useful discovery

Read the complete guide

Verify and detect AI traffic

AI BOT DETECTION

AI bots: new threats and how to detect them

See how AI bots are changing from simple crawlers into agents that can navigate, call APIs and act on websites

Read

CRAWLER IDENTITY

Not all crawlers are good

Learn why a crawler name proves nothing - and how to verify who is really behind the request