# Keep an AI crawler off your bill: implementation spec

Follow these steps in order. Each step ends with a check. Stop and ask the site owner when a step says "ask".

- Guide: https://nano133.com/guides/keep-ai-crawlers-off-your-bill · as Markdown: https://nano133.com/guides/keep-ai-crawlers-off-your-bill.md
- Stage: **Alpha**. We ran its code and its checks pass. It read a real node (read only), or, if it needs no node, it ran on real signatures or real files. Where it copies code that nano133.com runs, a test shows both act the same. Last checked 2026-10-31. No node needed. Its test script passed on real files, and the same rule ran on nano133.com and echoisland.party on 2026-10-30.
- Use at your own risk, for education.
- Code files: https://nano133.com/guides/keep-ai-crawlers-off-your-bill/code/rule.mjs · https://nano133.com/guides/keep-ai-crawlers-off-your-bill/code/guard.mjs

## Rules that must hold

1. Never block `facebookexternalhit`, a search engine that you want (`Googlebot`, `bingbot`), or an AI search or user-request crawler (`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`, `ChatGPT-User`, `Claude-User`, `Perplexity-User`). `rule.mjs` refuses these names.
2. Block one crawler by its own name. Never block by a word that a browser also sends, such as `Mozilla` or `bot`. `rule.mjs` refuses those too.

## Steps

The guide has the how for each step. This list has what must be true at its end.

1. **Find the crawler.** Check: one crawler sends most of the requests, and you have its name as the page shows it (for example `Meta-ExternalAgent`). If no crawler stands out, stop here: you need no rule.
2. **Make the rule text.** Check: run `node rule.mjs meta-externalagent`: it prints `(http.user_agent contains "meta-externalagent")`. Run `node rule.mjs facebookexternalhit`: it prints nothing and exits with 2.
3. **Add the rule in Cloudflare.** Check: ten seconds after the deploy, run `curl -s -o /dev/null -w "%{http_code}\n" -A "meta-externalagent/1.1" https://your-domain/`: it prints 403. Run it with `-A "Mozilla/5.0 (Macintosh) Safari/605.1.15"`: it prints 200. Run it with `-A "facebookexternalhit/1.1"`: it prints 200.
4. **A game on only a workers.dev address.** Check: run `node guard.mjs "meta-externalagent/1.1" meta-externalagent`: it prints `403`. Run `node guard.mjs "Mozilla/5.0 Safari/605.1.15" meta-externalagent`: it prints `200`.
5. **Undo it, and check it later.** Check: the blocked crawler's requests are near zero, and the crawlers that you left allowed still show their requests.

## Done when

- [ ] Step 1 check passes.
- [ ] Step 2 check passes.
- [ ] Step 3 check passes.
- [ ] Step 4 check passes.
- [ ] Step 5 check passes.
- [ ] `node test.mjs` passes (put the code files next to it). It needs no node and sends nothing.
- [ ] You told the owner the stage of this guide and of the track you used.

## What breaks, and what it costs

| Failure | Loss | Check |
| --- | --- | --- |
| A rule that is too wide blocks real visitors | Visitors, and every sale that they would have made | Step 3: a browser name still gets 200, and "Total requests matched" is a small share |
| A rule blocks link previews | A shared link shows no picture | Step 2: `rule.mjs` refuses `facebookexternalhit`; step 3: the third test request gets 200 |
| A rule blocks an AI search or user-request crawler | Assistants stop finding or describing your game | Step 1: only a training crawler is a reason; step 2: `rule.mjs` refuses their names |
| A crawler sends a false name | The rule does not stop it | Step 5: its requests did not fall. Cloudflare's bot rules do more than a name |
| A game on `workers.dev` has no zone | No Cloudflare rule can exist and the bill stays | Step 4: the guard in your Worker |
| The rule stays after the crawler leaves | Nothing: a rule that matches no request costs nothing | Step 5 |

## Notes for agents

You are reading this guide as the agent that helps the owner of a site or a game. Ask the owner for these things before you start. Never guess them:

1. The domain, and whether it is on Cloudflare (a zone) or only a `workers.dev` address. This decides step 3 or step 4.
2. The crawler to block. Show the owner the AI Crawl Control page's crawlers (the owner reads it to you, or pastes it) and ask which one. Do not pick one yourself.
3. Whether the owner wants to deploy the rule. A Cloudflare rule changes a live site. Show the owner the exact text from `rule.mjs` first, and let the owner press Deploy, or ask the owner's permission in words before you do it with a browser tool.

Never block `facebookexternalhit`, a search engine or an AI search or user-request crawler: `rule.mjs` refuses their names, and you must not edit it to allow them. Never use a word that a browser sends. After the deploy, run the three test requests of step 3 and show the owner the three answers. A rule is not done until the browser name gets 200.