Home / Blog/

AI-crawler traffic to small stores tripled in the last year — here’s what our network sees

Back
RESEARCH

AI-crawler traffic to small stores tripled in the last year — here’s what our network sees

yk-adpal-prod-admin

We looked across sites protected by technology that guards 2.5M+ websites, and one pattern stood out clearly enough to write down: AI-crawler traffic to small online stores has roughly tripled over the past year. Not to news sites, not to documentation hubs — to product and catalogue pages on shops with a few hundred SKUs and no dedicated engineering team.

What we actually measured

We compared crawler identity, request patterns and target paths across the network for the twelve months ending December 2025, focusing specifically on stores with fewer than 5,000 monthly visits — the segment least likely to have any crawler policy in place at all.

3.1×

YoY increase in AI-crawler requests

3.1×

YoY increase in AI-crawler requests

3.1×

YoY increase in AI-crawler requests

What we actually measured

We compared crawler identity, request patterns and target paths across the network for the twelve months ending December 2025, focusing specifically on stores with fewer than 5,000 monthly visits — the segment least likely to have any crawler policy in place at all.

That’s not a hypothetical gap. It’s the default state of most of the small stores we looked at:

  • Why this is a small-store problem specifically
  • Why this is a small-store problem specifically
  • Why this is a small-store problem specifically

What we’d suggest checking first

You don’t need a security team to get a first read on this. Pull a week of server or CDN logs and look for requests from known AI-crawler user-agents against your product and category pages specifically — that’s where the growth we measured is concentrated. If the volume looks high relative to your actual traffic, it’s worth deciding — deliberately, rather than by default — which crawlers you want to allow.

.The gap

Where the gap appears

Price and catalogue scraping

Automated visitors fetch public pages or endpoints to copy prices, availability and product data

Price and catalogue scraping

Automated visitors fetch public pages or endpoints to copy prices, availability and product data

Price and catalogue scraping

Automated visitors fetch public pages or endpoints to copy prices, availability and product data

Price and catalogue scraping

Automated visitors fetch public pages or endpoints to copy prices, availability and product data

robots.txt can express a preference. It cannot connect request patterns, endpoint use and business outcomes

. Related materials