# agent-traffic: Are people or AI tools reading your site?

Canonical: https://www.kartikaydhar.com/blog/agent-traffic-people-agents-site-traffic
Published: 2026-11-11T03:33:47.536+00:00
Updated: 2026-11-11T02:09:50.204109+00:00
Category: Tool Breakdowns

I tested agent-traffic to separate people, AI tools reading for someone and background crawlers. Here is what the labels can and cannot tell you.

More of Mintlify's documentation traffic comes from AI tools, the company says. Some read a page because a person asked a question right now. Others copy pages in bulk for training or AI search. If your analytics lumps them together as bots, you lose the difference between those visits.

Mintlify, which makes documentation sites, built agent-traffic to label page requests as likely people, AI tools reading for a person, AI crawlers, search crawlers or other bots. It can also spot programs pretending to be a known crawler.

The useful split is between a page fetched to answer someone now and pages copied in the background. I used sample requests I made up, so the counts describe my test, not real traffic.

## What I tested

I installed agent-traffic v0.1.0. The install added one package, with no other software packages needed alongside it.

I first checked the identifying names programs send when asking for a page, called user agents. All 135 examples supplied with the package matched their expected identity. They came from 98 entries for agents and crawlers and 35 entries for generic web-request tools. An entry can contain several examples.

That checks whether the package recognises its own examples. It does not establish how accurate it will be on unfamiliar traffic.

I then ran 14 sample page requests. The labels mean different things for a team reading its traffic report:

- agent, 3 requests: an AI tool reading your page because someone asked it to do something now. Mine included Claude Code and ChatGPT-User, plus a request asking for a plain-text page.
- ai-crawler, 2 requests: an AI company copying pages in advance for training or AI search. Mine were GPTBot and PerplexityBot. These are background collection, not evidence of someone considering your product right now.
- crawler, 1 request: a search engine collecting your page for its index. Googlebot received this label.
- bot, 6 requests: other automated activity. My examples included curl and python-requests scripts, automated browsers, LinkedIn and Slack link previews, and AhrefsBot. A link preview should not count as a person reading the page.
- human, 1 request: something that looks like a person using a browser. My plain Chrome example received this label with low confidence.
- unknown, 1 request: too little information to decide. My empty request landed here, so it should remain separate from the people count.

I also checked crawler identities using matching and mismatched network addresses, then ran a small server that chose a page format based on the label. Both worked on the supplied inputs, as the examples below show.

Image: Illustration: What I tested

Each result includes a reason for its label. The crawler identity checks appear separately below the sample traffic breakdown.

## How would a team start?

An engineer installs the package into the code that handles page requests or processes the site's traffic logs, the records of those requests. The command uses npm, a tool engineers use to install JavaScript packages:

npm i @mintlify/agent-traffic

The README also offers npx skills add mintlify/agent-traffic, which gives your team's coding assistant instructions for setting this up.

No account or API key was needed for my test. The package is free under the MIT licence, which permits use and modification, including in commercial software, subject to its terms. Your team still needs to run the code, store the results and build the report.

My suggested first step is to label one week of logs without blocking anything. An engineer and PM can then review a sample together. The PM gets a breakdown by visit type, with the confidence level and reason beside each label, rather than one combined bot count.

For older logs, an engineer can label each distinct user agent once and apply the result to matching requests. Keep the extra request details too when collecting new logs. They can reveal an AI tool that uses a browser's name.

## What else does a page request reveal?

Programs send small pieces of information with a page request, called request headers. The user agent is one of them. Another, the Accept header, says which page formats the program prefers.

Markdown is a plain-text version of a page, with simple marks for headings and lists, that AI tools can often read more easily than a fully designed web page. Clean text may help a tool describe your product accurately, but that is a possible benefit, not a result from this test.

One of my requests used Chrome's name but preferred markdown over a normal web page. agent-traffic labelled it as an agent whose product name it could not identify. It recognised the behaviour without claiming to know which tool was behind it.

Even a familiar name may be shared. The README warns that different products can send the same name, so ChatGPT-User does not always identify the application that made the request.

## Can it catch a fake crawler?

An IP address is the network address a request comes from. Some crawler operators publish lists of addresses they use, called IP ranges. agent-traffic can compare the real source address with those lists to check a crawler's claimed identity.

I loaded those lists and supplied these combinations:

- GPTBot with 73.12.4.9 failed the check and was labelled as a program pretending to be that crawler.
- GPTBot with an address chosen from OpenAI's published list was confirmed.
- Googlebot with 66.249.66.1 was confirmed.
- Googlebot with 3.5.140.2 failed the check and was labelled as a program pretending to be that crawler.

Here, confirmed means the address is on the operator's published list. It does not mean the visit is useful or welcome. A failed check means the supplied address does not match that crawler's published addresses. A third result, not checked, must stay separate from either outcome.

The engineer needs to pass the real visitor address, load the published lists and refresh them about daily. If a list cannot be downloaded, the relevant identity stays unconfirmed instead of automatically failing the check.

## How did I serve a plain-text page to AI tools?

I wrote a small server that used the label to choose between markdown and a normal web page. Requests labelled agent got markdown; everything else got the normal page. The package supplied the label, while my server supplied both versions of the content.

Image: Illustration: How did I serve a plain-text page to AI tools?

My server chose the page format after reading the label. The pricing text is dummy content, not agent-traffic pricing.

Claude Code got markdown. So did the request using Chrome's name that preferred markdown. curl and GPTBot got the normal page under my rule. GPTBot is an AI crawler, so a team wanting to serve it markdown would need to include that category too.

I also enabled an optional check for the full set of headers a real browser sends. Chrome's name without those headers was labelled a fake browser. With the full set, it was labelled human with medium confidence.

That check needs care. The README says to enable it only when browser headers reach your code unchanged over a secure web connection. Software sitting between the visitor and your code can remove those details. If that happens, a real browser could fail the check.

Keep the product facts identical in both page formats. The label should change how you present the content, not the claims a reader receives.

## What it can't tell you

- AI tools working inside a person's own Chrome can look exactly like that person, so the package misses them.
- A human label can be wrong. A program can send a browser's name, and even a fuller header check does not prove a person is reading.
- Coding tools running from someone's own connection cannot have their identity confirmed through a crawler operator's address list.

I tested v0.1.0. Keep the first integration easy to undo.

## What this means for product and GTM teams

For product and go-to-market teams, I would compare which pages AI tools fetch for people with which pages background crawlers collect. A rise in training-crawler activity should not be reported as a rise in people asking about the product.

The labels suggest what requested a page. They cannot tell you whether anyone read it, quoted it in an answer or bought anything. Those questions need separate evidence.

If AI tools regularly request your setup or pricing pages, those are candidates for checking that both the designed page and plain-text version contain clear, matching facts. The breakdown can help choose pages to review without claiming that a visit became a sale.

For related reading on automated browsers, see the agent-browser breakdown and the Lightpanda and automated browsers comparison.

## Frequently asked questions

### Does my site have to use Mintlify?

The package is open source and installed in your own code. The README lists support for Node and other web runtimes without requiring a Mintlify site.

### Does it create the plain-text page for me?

No. My demo supplied both page formats and used the label to choose between them.

### Can I use existing traffic logs?

Yes, if they contain user agents. Logs with only those names will miss the extra clue from a request preferring markdown.

### Is there a paid account to set up?

I needed no account or key to use the package. Running the code and storing the resulting traffic report remain the team's responsibility.

## Sources

- Mintlify: documentation-site platform. Checked 11 November 2026 (IST).
- Mintlify agent-traffic README: traffic changes reported by Mintlify, categories, installation, integration, licence and limits. Checked 11 November 2026 (IST).
- Agent and crawler dataset and Web-request tool dataset, which supplied the examples for my check.
- OpenAI GPTBot addresses and Google crawler addresses, used for the identity checks.
