Not all crawlers are good

Your access log says Googlebot. That does not prove Google sent the request. The HTTP User-Agent header is information supplied by the client, not an authentication mechanism. A scraper can copy a trusted crawler name and hope your firewall, rate limits or hand-written allow rules trust the label.

That matters because search engine spiders need broad access to public content. If you trust the name alone, an impostor can inherit the same access you meant for Google, Bing or an SEO crawler. If you block the name alone, you can damage the legitimate crawler instead.

The safer order is simple: identify the claim, verify the infrastructure, then decide what access that verified crawler should have. This page stays on that identity question. For crawler blocking policy and robots.txt rules, use the dedicated AI crawler enforcement guide.

By [Author] · Reviewed by [Security Lead], ADPAL · Updated 20 August 2026

A fake crawler is automated traffic that declares a trusted crawler identity in its User-Agent while coming from infrastructure that does not verify as belonging to that operator. Google explicitly warns that Googlebot User-Agent strings are often spoofed by other crawlers and recommends verification before you block or trust a request.

This is a form of user-agent spoofing: the client presents a different software identity in the header. It is not a vulnerability you can patch. The User-Agent field is part of a normal HTTP request, and HTTP itself does not authenticate the name written there.

For search engine spiders and commercial crawlers, the practical question is therefore not “does this string look familiar?” It is “can I verify that the request came from infrastructure the named operator controls?”

Before you change a firewall rule, keep these four points separate.

Official references: RFC 9110 · Google crawler verification · Bingbot verification · Ahrefs IP ranges.

This page does not split traffic into “good bots” and “bad bots”. That is a usefulness decision. For verification, a more useful taxonomy is whether the identity claim can be proven.

User-agent spoofing is simply sending a different identity string in an HTTP request. Browsers, command-line clients and scraping libraries can all set request headers. That makes User-Agent useful for classification, troubleshooting and analytics – but weak as a security boundary.

The incentive is obvious. A site may rate-limit an unknown scraper but exempt a search crawler to protect SEO. If the exception checks only the User-Agent string, the rule can be inverted: the fake crawler gets privileged access while the honest crawler becomes the only one that can be reliably blocked by name.

This is why “block suspicious User-Agents” is only a quick filter for unsophisticated automation. It is not crawler verification. Google itself says the Googlebot User-Agent is often spoofed, which is why its documentation tells site owners to verify the request source.

For Google crawlers, use one of the two verification methods Google currently documents. For a one-off investigation, forward-confirmed reverse DNS is easy to run from a terminal. For large-scale enforcement, published IP ranges are easier to automate.

Step 1. Take the source IP from the request log and run a reverse lookup. Google’s current documentation uses this example:

The documented example resolves to crawl-66-249-66-1.googlebot.com. For Google requests, the exact expected suffix depends on the crawler or fetcher type. Common crawlers typically use googlebot.com; other Google crawler classes can use google.com or googleusercontent.com.

Step 2. Resolve the hostname forward:

Step 3. Confirm that the forward lookup returns the original source IP. Both directions must agree. A PTR hostname on its own is not enough.

Step 4. Make sure the hostname suffix is one Google documents for the crawler class you are checking. Do not accept a hostname merely because it contains the word “google” somewhere in the middle.

Google publishes machine-readable ranges for common crawlers, special-case crawlers and user-triggered fetchers. Match the source address against the relevant current list. This is the better route for a reverse proxy, WAF or automated verification pipeline.

Do not hardcode a one-time copy forever. Google updates its address lists. Bing also warns that crawler addresses can change, and Ahrefs actively publishes current crawler ranges. Re-fetch current verification data instead of treating last quarter’s list as permanent truth.

“Googlebot” is not one universal crawler with one business effect. Google publishes a family of common crawlers plus special-case and user-triggered fetchers. That distinction matters before you block anything.

The table below focuses on Google crawler names most likely to create an expensive mistake for an SMB running search, shopping, media or paid campaigns.

Important nuance: Google says rules addressed to the generic Googlebot token can affect Google Search and other Google products. Some specialised crawlers also accept the generic Googlebot token. Read the current crawler documentation before assuming a narrow rule affects only one surface.

Current source: Google common crawlers and Google special-case crawlers.

A fake crawler name is usually a way to obtain access, not the attacker’s final goal. Once trusted, the automation can move through public content at the speed and depth you intended to permit for a legitimate crawler.

These terms are easy to mix up. Keep the definitions short and operational.

A failed verification should change the trust level, not trigger panic. It tells you one useful fact: the request has not earned the privileges attached to the crawler name.

Do not automatically label every unverifiable crawler malicious. Small tools may publish poor infrastructure documentation. Some fetchers use different infrastructure classes. The right next step is to remove special trust and judge the traffic by what it actually does.

If your goal has moved from verification to enforcement, continue with How to block AI crawlers from your website or the broader bad bot blocking guide.

Manual verification is worth doing. It teaches you what is actually in the logs and exposes unsafe allow rules quickly. The limitation is scale: identity is a request-by-request question.

ADPAL moves crawler verification and bot detection to the perimeter. Declared crawlers can be checked against trusted operator infrastructure before they receive crawler-specific access. A familiar name that does not verify does not inherit the privileges of the genuine crawler.

Traffic with no reliable crawler identity is handled separately. Advanced detection evaluates behaviour, browser, network, request sequence and endpoint, so undeclared or spoofed automation does not have to appear on a static bot list before it can be controlled.

The goal is not to block every crawler. It is to preserve the traffic you want – including genuine search crawling – while removing the blind trust that spoofed names exploit.

Point your DNS at the managed reverse proxy – live in hours, then a short monitoring period before enforcing. CMS-integrated deployment is available through hosting partners.

Cookieless, no cross-site tracking profiles, EU (Frankfurt) data residency.

Crawler names are useful labels. They are not proof. The safest crawler policy starts by verifying who is actually requesting access, then giving each verified operator only the access that makes sense for your business.