Home / Blog/

AI-crawler traffic to small stores tripled in the last year — here’s what our network sees4

Back
RESEARCH

AI-crawler traffic to small stores tripled in the last year — here’s what our network sees4

yk-adpal-prod-admin

What counts as a “bad” bot

Not all automated traffic is a threat. Search engine crawlers, uptime monitors and accessibility tools are bots too — and you want them on your site. A “bad” bot is automation that acts against your interests: scraping prices, testing stolen credentials, filling out forms with spam, or holding inventory hostage in a cart it never checks out.

Quick answer

To block bad bots, first confirm the traffic is actually automated using server logs (not just analytics), then apply detection at the perimeter — before requests reach your application — using behavior-based rules rather than IP blocklists alone.

Signs your site has a bot problem

01.

Traffic spikes that don’t correspond to any marketing activity

02.

Traffic spikes that don’t correspond to any marketing activity

Any one of these can have another explanation. Two or three together, on a site with public forms or a catalogue, is worth a closer look.

Step 2 — Choose the right blocking method

Once you’ve confirmed automated traffic, the method you choose depends on how sophisticated it is:

  • IP blocklists stop known bad actors quickly, but rotating proxies make them a short-term fix.
  • IP blocklists stop known bad actors quickly, but rotating proxies make them a short-term fix.
  • IP blocklists stop known bad actors quickly, but rotating proxies make them a short-term fix.

Most sites end up layering more than one method.

Signs your site has a bot problem

Blocking by IP alone. Residential proxies mean bad bots often share IP ranges with real customers.

Blocking by IP alone. Residential proxies mean bad bots often share IP ranges with real customers.

Blocking by IP alone. Residential proxies mean bad bots often share IP ranges with real customers.

. Related materials