AI Crawlers and AI Agent Management: Why User-Agent Strings Are No Longer Enough

The web is entering a new phase. For years, site owners mostly had to deal with human traffic, in occasions there were some malicious bots that visited the site but those can be differentiated, as mostly when it comes to automated traffic we think about search crawlers like Googlebot and Bingbot. Today, that traffic has expanded to include AI training crawlers, AI search indexers, live user-triggered fetchers, and emerging autonomous AI agents that may browse, compare, submit forms, or even transact on behalf of users.

That creates a new identity problem: how do you know whether a request claiming to be an AI crawler or AI agent is actually legitimate?

Historically, many bots identified themselves using the HTTP User-Agent header. A request might say it is from Googlebot, GPTBot, ClaudeBot, PerplexityBot, or bingbot. That made logging and robots.txt controls relatively simple.

The problem is that User-Agent strings are trivial to spoof. Google explicitly warns that HTTP user-agent strings can be spoofed and provides additional verification guidance for genuine Google crawlers. Microsoft gives similar guidance for Bingbot: reverse DNS the source IP, confirm it resolves to a search.msn.com hostname, then forward-resolve that hostname back to the original IP.

For basic logging, User-Agent may be useful. For security decisions, it is not enough.

Not all AI-related web traffic is the same. Treating every AI request as “a bot” misses important distinctions.

These crawlers collect public web content that may be used to train or improve future AI models. Examples include OpenAI’s GPTBot, Anthropic’s ClaudeBot, Mistral’s MistralAI-Training, Amazon’s Amazonbot, and Apple’s Applebot-related training controls. OpenAI documents GPTBot as a crawler used for generative AI foundation model training, while Anthropic describes ClaudeBot as a bot that collects web content that could contribute to model training.

These crawlers build indexes used by AI answer engines. For example, OpenAI’s OAI-SearchBot is used to surface sites in ChatGPT search features, and Perplexity’s PerplexityBot is used to surface and link websites in Perplexity search results.

These requests happen when a user asks an AI assistant to visit or summarize a specific page. Examples include OpenAI’s ChatGPT-User, Anthropic’s Claude-User, Perplexity’s Perplexity-User, Amazon’s Amzn-User, and Mistral’s MistralAI-User. These are especially important because blocking them may prevent users from using AI assistants to access or cite your content.

This is the emerging category that is pushing security. Agentic traffic may not just read pages. It may submit forms, make choices, initiate purchases, or act on behalf of a user. This is where traditional crawler validation is not enough.

Today, AI crawler verification usually falls into four levels of confidence.

This is the weakest option. It is useful for analytics and robots.txt matching, but it should not be trusted for allowlisting or privileged access.

Many AI providers now publish IP ranges for their bots. OpenAI publishes JSON IP range files for crawlers such as GPTBot, Perplexity publishes JSON ranges for PerplexityBot and Perplexity-User, Anthropic publishes a bots.json feed for Claude crawlers, and Apple publishes Applebot CIDR ranges.

This is better than User-Agent alone, but it requires ongoing maintenance. IP ranges change, WAF rules drift, and stale allowlists can create false positives or false negatives.

This is the classic model used by major search crawlers. Google documents that common crawlers generally use published IP ranges and reverse DNS hostnames matching Googlebot patterns. Bing recommends reverse DNS and forward DNS validation to confirm that traffic is genuinely from Bingbot. Apple also supports identifying Applebot traffic through reverse DNS in the *.applebot.apple.com domain or by matching Apple’s published CIDR JSON file.

This provides stronger crawler identity, but it still only proves infrastructure identity. It does not prove user intent, authorization, or delegation. Meaning we know what LLM is trying to fulfill the user request but we do not know if that request has good intentions.

This is where AI agent identity needs to go. HTTP Message Signatures, standardized as RFC 9421, define a way to sign selected parts of an HTTP message so the receiver can verify authenticity and integrity. Cloudflare’s Web Bot Auth uses cryptographic signatures in HTTP messages to verify that a request comes from an automated bot, using published public keys and signing keys.

This matters because agentic traffic needs more than “this came from a known IP range.” If an AI agent is submitting a form, making a purchase, or interacting with a customer account, the receiving system needs stronger proof. It should know which agent made the request, whether the message was tampered with, whether the request is fresh, and whether the user or organization actually authorized the action.

We should be able to have 2 main categories when it comes to managing AI interactions. One of them is AI crawler management and the other AI agent management.

For public read-only content, the concern of crawlers accessing the site is minimal, we should take care simply of ensuring that the sites are not DoSed. It may be acceptable to allow verified crawlers based solely on  User-Agent even though may be preferred to add IP range or reverse DNS verification to the User-Agent. This keeps content visible in AI search and answer engines while reducing spoofed bot traffic.

For dynamic endpoints, we should step up the game. Specially whe it comes to submitting forms. Require source validation, rate limits, cache protections, and bot scoring. I would argue that for sensitive application page or flows we must require cryptographic identity.

Nevertheless, as the market stands at the moment this does not seem feasible. But we should ensure that not any AI agent that can submit forms, trigger transactions, modify data, or access user-specific information. Those that we allow should use signed requests, replay protection, timestamps, and explicit delegated authorization.

AI crawler and AI agent management is becoming a core part of web security and content governance. The old model of trusting User-Agent strings is no longer sufficient. Published IP ranges and reverse DNS are useful for crawler validation, but they are not enough for agentic workflows.

The future model is clear: crawler traffic can be managed with verified network identity, but AI agents need cryptographic identity and delegated authorization.

Organizations that build this distinction into their WAF, CDN, bot management, and application security layers will be better prepared for the agentic web.

---

Sources:

https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26

https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

https://docs.perplexity.ai/docs/resources/perplexity-crawlers

https://developer.amazon.com/en/amazonbot

https://docs.mistral.ai/robots

https://support.apple.com/en-us/119829

https://www.rfc-editor.org/info/rfc9421

https://help.openai.com/en/articles/11845367-chatgpt-works-cloud-browser-allowlisting