# Keep an AI crawler off your bill

- Page: https://nano133.com/guides/keep-ai-crawlers-off-your-bill
- Stage: Alpha · last checked 2026-10-31 · no node needed. Its test script passed on real files, and the same rule ran on nano133.com and echoisland.party on 2026-10-30.
- Start here · about 10 minutes
- Agent pack: https://nano133.com/guides/keep-ai-crawlers-off-your-bill/SPEC.md · https://nano133.com/guides/keep-ai-crawlers-off-your-bill/PROMPT.md · https://nano133.com/guides/keep-ai-crawlers-off-your-bill/test.mjs
- Use at your own risk, for education.

> Find the AI crawler that sends most of your requests, block it by name with one Cloudflare rule, and leave the helpful ones alone.

Some AI crawlers read a site all day, and the site pays for the work. On nano133.com, one crawler sent about 99% of all AI crawler requests in a week: 1.01 million requests and 12.3 GB. This guide shows you how to find a crawler like that on your own site, block it by name with one Cloudflare rule, and leave the helpful ones alone. It moves no Nano.

> **Warning:** **A wrong rule blocks real visitors.** A rule matches the name that a visitor sends, so a name that is too short can match a browser. Test the rule with real requests before you trust it (step 3), and know where to delete it (step 5).

## What goes wrong

- **The bill grows and nobody sees why.** One crawler can send most of a site's requests. A page's own statistics do not show it, because the crawler does not run your page's scripts.
- **A rule is too wide** and blocks a browser or a search engine, so real visitors see an error.
- **A rule blocks link previews.** `facebookexternalhit` makes the picture when someone shares your link on Facebook or Instagram.
- **A rule blocks the AI assistants that would have found your site.** The search and user-request crawlers are how an assistant finds and describes your game.

## What you need

- **A domain of your own on Cloudflare.** Cloudflare calls it a zone. A game on only a `workers.dev` address has no zone: use step 4.
- **Node.js 18 or newer,** to run the two small files. They need no package and make no network call.
- **About ten minutes,** and a browser that is signed in to Cloudflare.
- **Or an AI assistant** to run the two small files for you. The box at the top of this page has a prompt. You press Deploy yourself.

> **Rule:** Never block `facebookexternalhit`, a search engine that you want (`Googlebot`, `bingbot`), or an AI search or user-request crawler (`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`, `ChatGPT-User`, `Claude-User`, `Perplexity-User`). `rule.mjs` refuses these names.

> **Rule:** Block one crawler by its own name. Never block by a word that a browser also sends, such as `Mozilla` or `bot`. `rule.mjs` refuses those too.

*Picture: Three kinds of visitor to always let in: link previews, search engines, and AI assistants that search. Each has a green tick.*

## Step 1: Find the crawler

**Do:** In the Cloudflare dashboard, open your domain, then **AI Crawl Control**, then **Metrics**, and read the crawlers of the last 7 days.

*Picture: On nano133.com, one AI crawler sent 99% of all AI crawler requests in 7 days. All the others together sent 1%. Your chart may look the same, or it may show no crawler that stands out.*

Each crawler has its requests and the data that it took. Cloudflare also names the operator and the kind of crawler. A training crawler reads a site to teach a model. A search or user-request crawler reads a page so that an assistant can answer a question about it. Only the first kind is a reason to block.

**Check:** one crawler sends most of the requests, and you have its name as the page shows it (for example `Meta-ExternalAgent`). If no crawler stands out, stop here: you need no rule.

## Step 2: Make the rule text

**Do:** Run `rule.mjs` with the crawler's name. It prints the text of the rule, and it refuses a name that you must not block.

```sh
node rule.mjs meta-externalagent
```

The file keeps a short list of names that stay allowed, refuses a word that every browser sends, and refuses a name with a quote or a space (a quote could change what the rule matches).

File `rule.mjs` (https://nano133.com/guides/keep-ai-crawlers-off-your-bill/code/rule.mjs):

```js
// Makes the text of a Cloudflare custom rule that blocks AI crawlers by name, and refuses a name that you must not block.
// Usage: node rule.mjs <crawler name> [<crawler name> ...]        for example: node rule.mjs meta-externalagent
// Prints the text for "Edit expression" (exit 0). A refused name prints nothing on stdout, says why on stderr and exits with 2.
// A name matches when the user agent CONTAINS it, in any case: so a name must not be a word that a browser also sends.

/** Names that stay allowed, and why. A name is refused when it contains one of these or one of these contains it. */
const LEAVE = {
  facebookexternalhit: "it makes the picture of a link that someone shares on Facebook or Instagram",
  googlebot: "it is a search engine",
  bingbot: "it is a search engine",
  duckduckbot: "it is a search engine",
  applebot: "it is a search engine",
  "oai-searchbot": "it lets ChatGPT search find your site",
  "chatgpt-user": "it reads a page when a person asks ChatGPT about it",
  perplexitybot: "it lets Perplexity find your site",
  "perplexity-user": "it reads a page when a person asks Perplexity about it",
  "claude-searchbot": "it lets Claude search find your site",
  "claude-user": "it reads a page when a person asks Claude about it",
};
/** Words that browsers send in their user agent: a rule on one of these would block real visitors. */
const BROWSER_WORDS = ["mozilla", "chrome", "safari", "firefox", "edge", "gecko", "khtml", "applewebkit", "windows", "macintosh", "android", "iphone", "linux", "compatible", "mobile"];

const names = process.argv.slice(2).map((n) => n.trim());
const refuse = (why) => {
  console.error(why);
  process.exit(2);
};
if (!names.length) refuse("give the crawler's name, for example: node rule.mjs meta-externalagent");
for (const name of names) {
  if (!/^[A-Za-z][A-Za-z0-9._-]{3,}$/.test(name)) refuse(`"${name}" is not a crawler name: letters, digits, dots and dashes, at least 4 characters, no spaces and no quotes.`);
  const low = name.toLowerCase();
  const word = BROWSER_WORDS.find((w) => low.includes(w));
  if (word) refuse(`"${name}" holds "${word}", a word that browsers send. A rule on it would block real visitors.`);
  for (const [leave, why] of Object.entries(LEAVE)) {
    if (low.includes(leave) || leave.includes(low)) refuse(`"${name}" stays allowed: ${why}. Never block it.`);
  }
}
console.log(names.map((n) => `(http.user_agent contains "${n}")`).join(" or "));
```

**Check:** run `node rule.mjs meta-externalagent`: it prints `(http.user_agent contains "meta-externalagent")`. Run `node rule.mjs facebookexternalhit`: it prints nothing and exits with 2.

## Step 3: Add the rule in Cloudflare

**Do:** Add a custom rule in Cloudflare with the text from step 2, set the action to **Block**, and deploy it.

*Picture: The check after you deploy: a visitor and a link preview still get in (200), and the crawler is stopped (403).*

In Cloudflare, open your domain, then **Security**, **Security rules**, **Create rule**, **Custom rules**. Name the rule. Choose **Edit expression** and paste the text from step 2. Choose the action **Block** and press **Deploy**. Cloudflare asks you to confirm.

Under the text, Cloudflare shows "Total requests matched" for the last 30 days. On nano133.com the rule for `meta-externalagent` and `meta-webindexer` matched 1.19 million requests, 18% of all traffic. A number near 0 means the crawler does not visit you, so the rule changes nothing now. A number near 100% means the rule is too wide: do not deploy it.

**Check:** ten seconds after the deploy, run `curl -s -o /dev/null -w "%{http_code}\n" -A "meta-externalagent/1.1" https://your-domain/`: it prints 403. Run it with `-A "Mozilla/5.0 (Macintosh) Safari/605.1.15"`: it prints 200. Run it with `-A "facebookexternalhit/1.1"`: it prints 200.

## Step 4: A game on only a workers.dev address

**Do:** Put `guard` at the top of your Worker's `fetch`, so that a named crawler gets a 403 first.

A `workers.dev` address has no zone, so a Cloudflare rule is not possible. The request still reaches your Worker and counts as one request, but the guard ends it before any database read or other work. Your game's files are served by Cloudflare without running your Worker, so the saving is smaller than for a site on a server.

File `guard.mjs` (https://nano133.com/guides/keep-ai-crawlers-off-your-bill/code/guard.mjs):

```js
// For a Worker on a workers.dev address (no zone, so no Cloudflare rule): answers 403 to a request whose user agent contains the name of
// a crawler, before your code reads the database. Use the same names that rule.mjs accepts: it refuses the ones that must stay allowed.
//
// In your Worker:
//   import { guard } from "./guard.mjs";
//   export default { async fetch(request, env) { const no = guard(request, ["meta-externalagent"]); if (no) return no; /* the rest */ } };
//
// A check by hand:  node guard.mjs "<user agent>" <name> [<name> ...]      prints 403 or 200
import { pathToFileURL } from "node:url";

/** A 403 Response when the request's user agent contains one of `names` (in any case), else null. */
export function guard(request, names) {
  const agent = (request.headers.get("user-agent") ?? "").toLowerCase();
  return names.some((n) => n && agent.includes(n.toLowerCase())) ? new Response("Forbidden", { status: 403 }) : null;
}

if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
  const [agent, ...names] = process.argv.slice(2);
  const no = guard(new Request("https://example.test/", { headers: { "user-agent": agent ?? "" } }), names);
  console.log(no ? no.status : 200);
}
```

**Check:** run `node guard.mjs "meta-externalagent/1.1" meta-externalagent`: it prints `403`. Run `node guard.mjs "Mozilla/5.0 Safari/605.1.15" meta-externalagent`: it prints `200`.

## Step 5: Undo it, and check it later

**Do:** A week after the deploy, open **AI Crawl Control** again and compare. To undo the rule, delete it on the same page.

**Check:** the blocked crawler's requests are near zero, and the crawlers that you left allowed still show their requests.

## What breaks, and what it costs

| Failure | Loss | Check |
| --- | --- | --- |
| A rule that is too wide blocks real visitors | Visitors, and every sale that they would have made | Step 3: a browser name still gets 200, and "Total requests matched" is a small share |
| A rule blocks link previews | A shared link shows no picture | Step 2: `rule.mjs` refuses `facebookexternalhit`; step 3: the third test request gets 200 |
| A rule blocks an AI search or user-request crawler | Assistants stop finding or describing your game | Step 1: only a training crawler is a reason; step 2: `rule.mjs` refuses their names |
| A crawler sends a false name | The rule does not stop it | Step 5: its requests did not fall. Cloudflare's bot rules do more than a name |
| A game on `workers.dev` has no zone | No Cloudflare rule can exist and the bill stays | Step 4: the guard in your Worker |
| The rule stays after the crawler leaves | Nothing: a rule that matches no request costs nothing | Step 5 |

## Sources

- [Cloudflare: AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) · the page of step 1
- [Cloudflare: custom rules](https://developers.cloudflare.com/waf/custom-rules/) · the page of step 3
- [The game starter's README](https://github.com/nano133-com/nano-game-starter#optional-stop-an-ai-crawler-that-costs-you-money) · the same step for a game's creator
- [Launch my game](https://nano133.com/build/nanoterra/launch) · where a creator sends a game for review

Code: MIT license. Written by nano133.com from code that runs on this site.