AI bots on your site: who visits and how to control them

Sites used to get people and search engine crawlers. Now there are AI companies' bots too: some collect text to train models, others fetch an answer for ChatGPT or Perplexity right now. On many sites they already outnumber Googlebot.

People usually ask two questions: “how do I block them” and “how do I get found through ChatGPT”. The answers point in opposite directions, and that is part of the confusion. The bots are different, and each kind has to be handled on its own.

Three kinds of AI bots

The main names (as written both in robots.txt and in logs):

Two subtleties:

• You won't see Google-Extended or Applebot-Extended in your logs. They aren't separate bots, just names for rules in robots.txt. The crawling is done by Googlebot and Applebot. Blocking Google-Extended doesn't remove the site from Google Search — it only says “don't use my text for Gemini”.
• CCBot builds Common Crawl, an open archive of the web that a great many companies train their models on. If you want to opt out of training, block it too.

Finding AI bots in your logs

If the site runs on nginx, every request is written to /var/log/nginx/access.log, and the line includes the name the visitor gave itself. How many times each AI bot came during the log period:

sudo grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot" \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

On Apache the log is usually /var/log/apache2/access.log. If the web server runs in Docker, the log may live inside the container: docker logs container-name 2>&1 | grep ....

What the bot gets in response matters too. If the server refuses it, the log line has the code 403, 429 or 444:

sudo grep -i "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

200 — pages are served. Lots of 403 and 429 — the server is turning the bot away, possibly without you wanting it to. The usual culprits are a security plugin, an nginx rule left over from an old attack, or a rate limit.

One caveat: anyone can call themselves GPTBot. The name in the log is whatever the visitor wrote about itself. The big companies publish lists of their addresses you can verify against. For a sense of “who and how much”, the name is usually enough.

How much load they add

Training bots sometimes crawl an entire site in a few hours. On a small VPS you notice: CPU load goes up, the database answers slower, and real people wait.

A rough share of requests from AI bots:

total=$(sudo wc -l < /var/log/nginx/access.log)
bots=$(sudo grep -ciE "GPTBot|ClaudeBot|CCBot|PerplexityBot|OAI-SearchBot|Claude-SearchBot" /var/log/nginx/access.log)
echo "AI bots: $bots of $total requests"

If the share is large and the server feels it, that is a reason to block training bots or ask them to come less often.

Controlling them: robots.txt

robots.txt is a file at the root of the site (https://your-site.com/robots.txt) where you say which bots may go where. The major AI companies promise to respect it.

Opt out of training but stay in AI search — the most common choice:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

OAI-SearchBot, Claude-SearchBot and PerplexityBot aren't mentioned here, so, like everyone else, they may go everywhere.

Close only part of the site to AI — paid articles, for example:

User-agent: GPTBot
Disallow: /premium/

What robots.txt won't do:

• It's a request, not a lock. Honest bots follow it, others don't. To really keep a bot out, you need a refusal on the server or in Cloudflare.
• Bots acting on behalf of a user may not follow it. Their owners say plainly that since a person asked to open the page, robots.txt may not apply to that request.
• What was already collected stays collected. The rule applies to future crawls.

After editing, check that the file opens, and see in your logs that the bot really stopped visiting the closed part. If it keeps coming and the name is genuine, that is grounds for a complaint to the bot's owner.

Controlling them: Cloudflare

If the site is behind Cloudflare, there is a switch for blocking AI bots. It turns them away before they reach your server — so it works as a lock, not a request.

It's easy to get this wrong the other way round: you turned blocking on a year ago, and now you wonder why ChatGPT can't find your site. Your server's logs won't show this refusal: the request simply never reaches it. All that's left is an indirect sign — a bot used to come, and then it disappeared.

Before switching it on, see exactly which bots it blocks, and decide whether you want AI search to find you.

In short

1. Decide what you want: opt out of training, stay in AI search, or both. These are different bots.
2. Look in your logs: who comes, how often, and what they get back.
3. Close what you don't want in robots.txt, by bot name.
4. Make sure neither the server nor Cloudflare is refusing the bots you do want.
5. Check the logs again in a week.

How vpsFocus does this

vpsFocus has a “Bots & AI” screen for this. It reads the web server logs on your VPS and shows, for each site:

• which bots came: search engines, AI search, training and on-behalf-of-user requests, each separately;
• what share of the site's requests and traffic comes from bots;
• which bots the server refuses (403, 429, 444) — including when a bot can't even read robots.txt;
• who visits sections closed in robots.txt;
• which AI bots used to come and then disappeared — for example, after blocking was switched on in Cloudflare. If the site is behind Cloudflare, the app can read that switch too.

The SEO audit in the same app checks robots.txt and tells you whether the site is open to search engines, AI search and training.

Logs are read on your server and go nowhere; only counts like “requests per day” are kept, without log lines or visitor addresses.

Download vpsFocus