Owners get asked it now, often by a relative: "Are you showing up in ChatGPT?" A lot of the advice that follows is about blocking. Check your robots.txt, it says, because you may be turning the AI crawlers away without knowing it.
We wanted to know how often that is actually true. So we took the 696 independent US businesses with a website from our first study and ran our own AI visibility checker on every one of them on November 16, 2026: the same tool, making the same three requests anyone gets when they paste in an address.
Almost nobody is blocking anything. Of the 680 sites whose crawler rules could be read, 660 (97.1%) turn away none of the 11 crawlers we check. The door is open. What is missing is on the other side of it: 30.9% of home pages state a price, 46.7% state their opening hours, and 17.3% carry all four of the facts a customer asks an assistant for.
The door: who turns the crawlers away
For each site the checker reads robots.txt once per crawler, by name, the way the crawler itself does. Sites refusing each one, out of 680:
| Crawler | What it is for | Sites refusing it | Share |
|---|---|---|---|
| OAI-SearchBot | Assistant search | 2 | 0.3% |
| Claude-SearchBot | Assistant search | 3 | 0.4% |
| PerplexityBot | Assistant search | 2 | 0.3% |
| Googlebot | Search engine | 2 | 0.3% |
| Bingbot | Search engine | 2 | 0.3% |
| ChatGPT-User | Opens a page on request | 2 | 0.3% |
| Claude-User | Opens a page on request | 2 | 0.3% |
| Perplexity-User | Opens a page on request | 2 | 0.3% |
| GPTBot | Training | 19 | 2.8% |
| ClaudeBot | Training | 18 | 2.6% |
| Google-Extended | Training | 17 | 2.5% |
20 sites (2.9%) refuse at least one AI crawler, and 17 of those 20 refuse the training crawlers only. That is a decision, not a mistake: OpenAI's own crawler page says of its three bots that "each setting is independent of the others", so a site can keep its text out of model training and still be found and linked in ChatGPT's search answers. Those 17 owners, or whoever built their sites, appear to have done exactly that.
The accidental kind is rare. 3 sites turn away a crawler that an assistant's answers depend on, and 2 of them refuse every crawler on the list, Google and Bing included. Those are not AI policies. They are sites closed to every search engine. We cannot see why from outside, but a site that refuses Google is not making a statement about AI.
39 sites (5.7%) have no robots.txt at all, which crawlers read as "everything is allowed". The other 641 have one, and in 20 of them it refuses an AI crawler.
The page: is the text there to read
A crawler that reaches a page still has to find words on it. The checker reads the HTML as it arrives, without running scripts, because that is the least any crawler is certain to get: text that only appears after a script has finished is text a crawler may never see. It then counts the words. Of the 683 home pages we could read:
| What arrives without running scripts | Sites | Share |
|---|---|---|
| A readable page (150 words or more) | 589 | 86.2% |
| A thin page (50 to 149 words) | 50 | 7.3% |
| Almost nothing (under 50 words) | 44 | 6.4% |
The typical home page delivers 516 words. The 44 sites in the last row are the ones to worry about: a visitor with a browser sees a full page, and a crawler sees a title and a loading screen. 8 sites (1.2%) go further and carry a noindex instruction on their own home page, which asks search engines to leave it out entirely.
Put the first two sections together. 579 of 683 sites (84.8%) have nothing in the way: no crawler refused, the text in the HTML, no instruction against indexing. For those 84.8%, "can an assistant read my website" has the answer yes.
The facts: what an assistant finds when it gets in
Being readable is not the same as having something to say. When somebody asks an assistant for "a dentist near me that is open Saturday" or "what does a deep clean cost", the answer has to come from somewhere. The checker looks on the home page for the four facts those questions turn on:
| Fact, written on the home page as text | Sites | Share of 683 |
|---|---|---|
| A street address | 514 | 75.3% |
| A phone number | 499 | 73.1% |
| Opening hours | 319 | 46.7% |
| At least one price | 211 | 30.9% |
And how many of the four each home page has:
| Facts on the page | Sites | Share |
|---|---|---|
| None of the four | 98 | 14.3% |
| One | 78 | 11.4% |
| Two | 174 | 25.5% |
| Three | 215 | 31.5% |
| All four | 118 | 17.3% |
98 home pages (14.3%) state none of the four. That is more than four times the number of sites refusing any AI crawler. If you count the businesses an assistant cannot answer a basic question about, the ones with nothing to read outnumber the ones that locked the door by a wide margin.
Structured data, the block of schema.org markup that states a business's name, address, phone and hours as fields rather than prose, is absent on 390 sites (57.1%). 203 have some of it, and 90 (13.2%) state all four of those fields. We wrote up what that markup is and how to add it in structured data for small business websites.
By trade
Share of each trade's readable home pages with each thing on it:
| Trade | Sites | Text in the HTML | Phone | Hours | Address | A price | All four facts |
|---|---|---|---|---|---|---|---|
| Restaurants, cafes and bars | 107 | 79% | 64% | 52% | 69% | 22% | 14% |
| Hair and beauty | 101 | 77% | 74% | 51% | 75% | 34% | 23% |
| Trades and workshops | 112 | 83% | 71% | 58% | 78% | 34% | 23% |
| Dentists and clinics | 128 | 95% | 91% | 61% | 80% | 25% | 20% |
| Gyms and studios | 148 | 89% | 61% | 32% | 70% | 30% | 15% |
| Hotels and guesthouses | 69 | 93% | 81% | 22% | 83% | 45% | 9% |
| Rentals | 18 | 94% | 67% | 33% | 78% | 39% | 6% |
Dentists and clinics are the most reachable by phone (91% print the number), which matches what the homepage study found for tappable phone links. Prices are the gap everywhere: the best trade for them, hotels and guesthouses, manages 45%, and restaurants, cafes and bars sit at 22%. We did not measure why, and there are innocent reasons: a menu that is a PDF, a photo of a price board and a link out to an ordering platform all read as "no price on the page" to this check, exactly as they do to a crawler. The low hours figure for hotels and guesthouses (22%) deserves the same caution, since a business with reception around the clock, or one that opens by appointment, may have no hours to print. The Rentals row is 18 sites and is shown for completeness, not as a finding.
One number we did not expect
165 of the 683 sites (24.2%) serve an llms.txt, the plain-text summary file proposed at llmstxt.org for language models to read.
We did not believe the count, so we opened 14 of them by hand. 13 were real files, and 4 of those named Wix in the text. The plainest explanation is that site builders have started writing the file for their customers, and that most of these owners do not know they have one. We cannot tell from outside who wrote the rest. For transparency: Webkio does the same for the sites built on it, as our changelog records. The checker reports the file as information and nothing more, because it is a proposal, and we know of no assistant that has promised to read it.
What to do with this
If you run one of these businesses, the order of work the numbers suggest is the reverse of the usual advice.
- Write the four facts on your home page, as text. Address, phone, hours and at least a starting price. Not in an image, not only on a contact page. This is where most sites fall down, and it costs nothing. If you are unsure about publishing prices, we covered that decision for one trade in should a dental practice show prices; the reasoning travels.
- Add the structured version. On a Webkio site it is written from your business details; on any other, the LocalBusiness schema generator produces the block to paste in.
- Then check the door, once. Paste your address into the checker. It takes less than a minute and names the exact line to change if a crawler is refused. You are probably fine: 97.1% of sites are.
- Decide about training on purpose. Refusing GPTBot, ClaudeBot or Google-Extended keeps your text out of model training and does not remove you from answers. Whether you want that is yours to decide; should you let AI crawlers read your website walks through it.
None of this is a ranking trick. An open door and four written facts make a page an assistant can quote. Whether it does is a separate question, and getting quoted by AI search is about that part.
How we did it, and what this cannot tell you
The sample. The 699 business listings from our first study, in 20 US metro areas and seven trades, all independent businesses with a working website; that article describes how they were found. 3 of the listings repeat a domain already in the list, which leaves 696 sites. It is not a random sample of American small business, and a business with no website is not in it at all.
The run. One pass on November 16, 2026, one site at a time per worker, with the checker's own user agent, which says who we are. Each site received the three requests the public tool makes: the home page, robots.txt and llms.txt. Sites that gave no answer were asked once more later. No business is named anywhere, and hosts are stored as hashes.
What happened to every site. 683 were read (98.1%). 4 answered our checker with HTTP 403, and 9 timed out or could not be reached on both attempts. Those 4 refusals tell you how those sites treat an unfamiliar checker. They are not evidence about any AI crawler and are counted nowhere above as blocking. At 7 more sites the page was readable but robots.txt was not, so the crawler table uses 680 sites. 47 home pages redirected to a different domain, and we read the page they landed on.
What counts as refusing a crawler. The rule in robots.txt that applies to that crawler's name, or the wildcard rule when the file names nobody, disallows the home page. We checked 8 of the refusing sites by reading their files ourselves, and all 8 matched the tool. A site counts as serving an llms.txt when that address returns a text file with something in it, not the site's ordinary "page not found" page.
The limits, plainly.
- robots.txt is a request. A firewall or a host-level "block AI bots" switch can turn a crawler away while robots.txt allows it, and nothing outside the site can see that. The true share of sites closed to AI crawlers is therefore at least 2.9%, not exactly that.
- Only the home page was read. Hours and prices that live on an inner page are missed, so those two figures are floors.
- The fact checks are pattern matching. A phone number inside an image, or hours drawn by a widget, are not found. That is also why a crawler does not find them.
- Readable is not recommended. Nothing here measures how often any assistant mentions any business, and nobody outside the assistant companies can.
FAQ
What share of small business websites block AI crawlers?
In this check of 680 US small-business sites on November 16, 2026, 2.9% refused at least one AI crawler in robots.txt, and 0.4% refused one that an assistant's answers depend on. 97.1% refused none.
Does blocking GPTBot take a business out of ChatGPT's answers?
Not according to OpenAI, which documents GPTBot (training), OAI-SearchBot (search results in ChatGPT) and ChatGPT-User (a page opened at a user's request) as separate settings. 17 sites in this sample refuse training and nothing else.
What is most often missing from the page?
A price. 30.9% of the 683 home pages state one, against 75.3% with a street address and 73.1% with a phone number. Opening hours are on 46.7%.
How many small business websites have an llms.txt?
24.2% of the sites we read (165 of 683). The ones we opened by hand looked generated by the site builder, and no assistant we know of has committed to using the file.
Can I cite these numbers?
Yes, with a link to this page and the date. They were measured on November 16, 2026 on the same sample as our three earlier studies, and the method is above.