How to stop web scraping

Web scraping becomes a business problem when automated visitors copy data at a scale, frequency or purpose you did not agree to: prices for repricing, inventory for competitive intelligence, product catalogues for replication, or original content for reuse elsewhere.

The hard part is not blocking “a scraper”. It is separating extraction you want – search engines, approved partners and selected AI crawlers – from automation that takes value without giving anything back.

By [Author] · Reviewed by [Security Lead], ADPAL · Updated 20 August 2026

OWASP classifies scraping as a distinct automated threat to web applications. Its automated-threat framework is useful because scraping often abuses normal, public functionality rather than exploiting a software vulnerability. Source: OWASP OAT-011 Scraping.

Most commercial scraping is a pipeline, not a person repeatedly refreshing your website. The exact tooling varies, but the business workflow is usually simple: choose a target, fetch pages or endpoints, extract structured fields, then reuse the data somewhere else.

For an eCommerce store, that may mean prices, availability, SKUs and promotions feeding a competitor-intelligence or repricing system. OWASP even uses continuous competitor-store scraping and dynamic pricing as an example of automated business activity.

Start with server or reverse-proxy logs. Analytics can show unusual traffic, but scraping detection is stronger when you can see the requests themselves: which URLs were fetched, how often, from where, and in what order.

Avoid the common shortcut of declaring every Python, cURL or datacentre request malicious. Those are useful clues, not proof. Legitimate tools use HTTP libraries and cloud infrastructure too. The strongest case comes from several signals lining up.

No useful log access? Scan my site to establish which automated traffic is reaching the website before you start adding blocking rules.

Scraping protection becomes easier to prioritise when you stop thinking in pages and start thinking in business assets. The same scraper may touch several assets during one crawl.

The objective is not to block crawling. Search engines crawl your site so it can appear in search. Approved comparison partners, monitoring services and other tools may also be commercially useful. Your policy should answer two questions: who is this crawler, and what access do you actually want to give it?

Do not trust a familiar user-agent string by itself. Google explicitly warns that Googlebot user-agents are spoofed and recommends reverse/forward DNS verification or comparison with published Google crawler IP ranges. Source: Google crawler verification.

Several low-cost controls are worth using. The mistake is treating any one of them as complete anti-scraping protection.

The robots.txt point is especially important. The Robots Exclusion Protocol lets a site publish rules that crawlers are requested to honour, but the standard explicitly says those rules are not a form of access authorisation. Source: RFC 9309.

Rate limiting remains useful. Use tighter budgets around data-rich and computationally expensive endpoints. Just design limits around normal customer and partner behaviour, and review what happens during promotions or traffic peaks before enforcing aggressively.

The difficult scrapers are the ones that do not present a useful name or obvious fixed source. They can use normal browser stacks, distributed networks and request pacing that avoids crude thresholds. At that point, detection has to move from one static identifier to the pattern of requests.

For ADPAL, the relevant signal set is behaviour, browser, network, request sequence and endpoint. Correlating those signals lets the protection layer distinguish automated extraction from the traffic that should remain available, without making a user-agent string the source of truth.

Perimeter placement matters because the decision is made before the request reaches the protected application. If a scraping request is blocked there, the origin does not need to render the page or return the protected response.

Do not judge success by a dashboard number called “bots blocked”. Measure whether the business problem changes.

ADPAL runs in front of the protected website. Advanced detection evaluates behaviour, browser, network, request sequence and endpoint so unwanted automation can be stopped before it reaches data-rich application flows.

Verified search crawlers and automation you choose to permit can remain accessible. Unknown or deceptive automation does not gain trusted access simply because it claims a familiar identity.

Cookieless, no cross-site tracking profiles, EU (Frankfurt) data residency.

Web scraping is easiest to manage when the policy is explicit: know what data matters, verify the automation you want, reduce obvious exposure, then enforce unwanted extraction before it reaches the application.

If you suspect competitors, aggregators or unknown bots are repeatedly reading your catalogue, prices or content, measure the traffic first. A clean baseline makes every later decision safer.