Somebody reads a headline about AI companies scraping the web, and by lunchtime the question arrives: should we block them? It is a fair question with a bad answer everywhere you look, because almost every article treats "AI crawler" as one thing.
It is three things, they want different things from you, and one of them is how a small business gets recommended by an assistant at all.
Three kinds of bot
- Search crawlers. The classic ones, indexing pages so they can appear in results. Blocking these means disappearing from search. Nobody is confused about this one.
- Training crawlers. Collecting text to train a model. Whatever they take is absorbed into something that will not link back to you. This is the one the headlines are about.
- Retrieval crawlers. Fetching a page at the moment somebody asks a question, so the assistant can answer using it and cite the source. This is the one that sends you customers.
The second and third are frequently the same company and are usually separate agents with separate names. That distinction is the entire decision, and it is what a blanket "block AI bots" instruction throws away.
The trade, in one sentence each way
Allow them, and your opening hours, your service area and your prices can be used to answer a question somebody is asking right now, with your name on the answer. Block them, and they cannot recommend what they cannot read.
For most small businesses that settles it. If your website exists so people can find you, discover what you do and get in touch, then being quoted by an assistant is the same thing as being found, and refusing it is refusing a channel. The mechanics of being quotable well are in getting quoted by AI search.
Who should think harder
The calculation genuinely inverts when the content is the product. A publisher, a course seller, a photographer whose images are the thing being sold, an archive somebody pays to read. There, being absorbed into a model that answers without sending anyone to you is a straight loss, and blocking training crawlers while allowing retrieval ones is a coherent position.
Note that this is a small minority of business websites, and it is almost never true of a plumber, a clinic, a letting agent or a shop.
Check what yours does right now
Open yoursite.com/robots.txt in a browser. It is a plain text file every site has, whether or not anyone wrote it. User-agent: names a bot, Disallow: tells it to stay out, and Allow: lets it in. If you see Disallow: / under User-agent: *, your entire site is asking every crawler to leave, which is occasionally there by accident after a build.
Our free website checker will read a page the way a crawler does and tell you if it is asking to be left out of search.
What a published Webkio site serves
Worth being specific rather than vague about our own product. A published site generates its robots.txt automatically: it allows all crawlers, blocks the transactional paths that carry personal data and one-time tokens, and points at the sitemap.
User-agent: *
Allow: /
Disallow: /order/
Disallow: /cart
Disallow: /account
Sitemap: https://yoursite.example/sitemap.xml
So AI crawlers are allowed by default, along with everything else, and the pages nobody should be indexing are excluded. There is no per-crawler switch in the product today. An unclaimed preview site is the one exception and asks every bot to stay away entirely, because nobody has agreed to publish it yet.
If you decide to block
Two things worth knowing before you do. Robots.txt is a request, honoured by the major operators and ignored by bad actors, so it is a policy statement rather than a lock. And blocking is easy to over-apply: a rule aimed at training crawlers that also catches retrieval agents removes you from assistant answers while doing nothing about the training that already happened.
If the concern is a specific page rather than the site, the better tool is not publishing it. Anything genuinely confidential does not belong on a public website with a rule asking politely that nobody read it.
Frequently asked questions
Should I block AI crawlers from my website?
Most small businesses should not. Assistants increasingly answer questions like "who repairs boilers near me" by reading and citing real pages, and a blocked site cannot be recommended. Blocking makes sense mainly when your content is itself the product, such as a publisher or a paid archive, where being answered without a visit is a direct loss.
What is the difference between an AI crawler and a search crawler?
There are three kinds. Search crawlers index pages for search results. Training crawlers collect text to train a model and will not link back. Retrieval crawlers fetch your page at the moment someone asks a question so the assistant can answer and cite you. Blocking all three in one rule removes you from the one that sends customers.
How do I check which crawlers my website allows?
Open yoursite.com/robots.txt in a browser. Every site serves one whether or not anyone wrote it. User-agent names a bot, Disallow tells it to stay out, Allow lets it in. A Disallow: / under User-agent: * means the whole site is asking every crawler to leave, which is sometimes left behind by accident after a build.
Does robots.txt actually stop anyone?
It is a request rather than a lock. The major search and AI operators honour it; bad actors ignore it entirely. Treat it as a published policy, and keep anything genuinely confidential off a public website rather than relying on a rule that asks politely.