AI bots: new threats and how to detect them

The web is moving from bots that read to agents that can act. A traditional crawler fetches pages. An AI agent can navigate a site, compare products, call APIs, fill forms, use an account or attempt a transaction on someone else’s behalf. That makes AI bot detection a different problem from spotting an old scraper with a suspicious User-Agent.

The business question has changed too. Some AI automation creates value through search, discovery or delegated customer actions. Some extracts content, consumes expensive endpoints or automates abuse. The goal is no longer “block bots”. It is to identify what kind of automation is present, verify what can be verified, understand what it is doing, and apply the right policy before it reaches valuable workflows.

For context, Thales Imperva reports that automated bot traffic exceeded 53% of global web traffic in 2025, while 27% of bot attacks in its dataset targeted APIs. That is bot traffic overall, not “AI bots” specifically, but it explains why the rise of agentic automation matters to website and API security. Source: 2026 Bad Bot Report.

AI crawler: automated software that primarily reads and indexes public content. Its job may be model development, AI search, retrieval or dataset creation.

AI agent: automation that can take multi-step actions toward a goal. It may browse, search, select, authenticate, submit, call a tool or move through a customer journey.

AI-assisted malicious automation: an attacker can use AI to make existing abuse more adaptive – for example by changing paths after errors, varying request patterns or selecting the next target dynamically. The underlying objective may still be scraping, credential abuse, fake signups or API exploitation.

This is why the phrase AI bot traffic now covers very different visitors: public crawlers, search fetchers, browser agents, commerce agents, internal automation and malicious tools. Treating them as one category creates both false positives and false trust.

Recent controlled research on AI browsing agents supports the same direction: browser automation can be difficult to separate from human browsing using one static fingerprint, while behavioural and multi-layer signals can add useful discrimination. These studies are early and controlled, so they should inform detection design rather than be treated as a universal production benchmark. FP-Agent research · behavioural detection study.

For content-heavy sites, start with content theft protection and web scraping protection. For software-heavy sites, the same trend makes the API security surface more important because agents and attack automation can bypass the visible page entirely.

This is the core change in AI bot detection: there is no single magic AI header. You build confidence by combining independent evidence and by asking whether the resulting activity makes sense for the protected workflow.

One of the most important new ideas is that bots and agents can prove who they are instead of asking sites to trust a User-Agent or IP range forever.

Web Bot Auth is an emerging approach built on HTTP Message Signatures. The agent signs the request with cryptographic key material; the website can verify that signature and bind the request to a declared operator. Google describes Web Bot Auth as experimental and says it is testing the approach with some AI agents on Google infrastructure. The IETF work is still an Internet-Draft, so this is an emerging standard, not something every AI bot supports today.

Primary references: Google Web Bot Auth documentation · IETF HTTP Message Signatures for automated traffic draft.

For an SMB, the immediate lesson is simple: keep supporting the verification methods available today, but design policy so stronger authenticated agent identities can slot in later without rebuilding your security model.

Cryptographic identity helps with declared agents. It does nothing for automation that deliberately stays anonymous, pretends to be a normal browser or uses infrastructure you have never seen before.

That traffic has to be evaluated by what it does. The strongest practical signals are usually not “AI-specific”. They are the same request-level inconsistencies that reveal automation generally – but AI agents make correlation and context more important because the traffic can look plausible for longer.

You do not need an enterprise security programme to get materially better visibility. A small team can build a useful baseline in one afternoon.

ADPAL puts the detection layer in front of protected application workflows. Advanced detection evaluates behaviour, browser, network, request sequence and endpoint context so unknown or harmful automation can be controlled without depending on a static crawler list.

Declared and approved automation can be handled separately. That matters as AI traffic becomes more mixed: the goal is to preserve useful search, partner and delegated traffic while applying tighter policy to scraping, account abuse, form abuse and high-cost API activity.

ADPAL complements, rather than replaces, your existing controls. Authentication, authorisation, MFA, WAF rules, server-side validation, API gateways and payment controls still own their jobs.

Point your DNS at the managed reverse proxy – live in hours, then a short monitoring period before enforcing. CMS-integrated deployment is available through hosting partners.

Cookieless, no cross-site tracking profiles, EU (Frankfurt) data residency.

AI agents are not a future traffic class. They are becoming part of normal web automation. The safest response is not a bigger static blocklist. It is better visibility, stronger identity where available and behaviour-aware control where identity disappears.

If you want to see how that traffic is reaching your own site, start with the protected workflows that matter commercially, then compare declared crawlers with the unknown automation around them.