Guide

How to actually classify a site's traffic: humans, SEO, AI agents and malicious probes

On a professional site, the access log mixes five very different populations: myself and my technical tools, real human visitors, search engine bots, generative AI agents, and malicious probes hunting for configuration files or forgotten credentials. Conflating them distorts every statistic derived from them.

The first rule, counter-intuitively, is that the requested path must be checked before the user-agent. A scanner requesting an AWS credentials file or a /.env while announcing itself as “GPTBot” is still an attacker impersonating a credible identity — not a legitimate AI agent. Classify by user-agent first, and that malicious traffic silently pollutes your “AI agent” statistics, skewing any GEO visibility analysis.

Once probes are filtered out by path, the user-agent becomes reliable again for three useful categories: classic indexing bots (search engines), AI assistant agents (training their models or answering a query), and real human browsers. That last category further breaks down by source — direct access, organic search, social media, partner referral — to know where the audience actually comes from.

The most valuable signal, and the most often ignored, is the difference between an AI agent training a model and an AI agent answering a question a human just asked, right now. That second case — a conversational assistant fetching a specific page because a user asked it about that topic — is the real GEO signal: proof an AI judged a page trustworthy enough to fetch live before answering.