AI reads websites now. Yours might be turning it away.

Ask yourself how you last chose a restaurant in a town you didn’t know. A few years ago the honest answer was Google, a map, and twenty open tabs. Increasingly, the honest answer is: you asked an AI assistant, and it just told you.

“Plan me a weekend around Dartmoor.” “Who’s a good web developer in Devon?” “Find me a secure dog field near South Brent.” Every one of those is a real kind of query people type into ChatGPT, Claude, Perplexity and Google’s AI results every day — and every answer is assembled from websites the AI was able to read. If your site is one of them, you’re in the answer. If it isn’t, you don’t exist for that customer, and you will never know it happened.

The traffic you will never see

Here’s the part that catches business owners out: none of this shows up in your analytics.

Analytics tools exist to count humans. They work hard to filter out bots — and an AI reading your website is, by definition, a bot. So the AI crawler that reads your whole site tonight won’t register as a visit. The person who asks an assistant for a recommendation tomorrow and gets your business in the answer may never click through at all — they got what they needed: your name, your number, the fact that you’re the right fit.

I recently pulled a month of raw server logs for a client — a UK publisher — and compared them against their (accurate, cookieless) analytics. The split stopped me for a moment:

AI now reads that site more than everyone else combined — over half of all page views, roughly three times the human readership. One assistant’s crawler alone made up 94% of the AI traffic. And the site served it flawlessly: over 99% of those requests returned the page successfully.

It gets starker with documents. On PDFs, the AI-to-human gap wasn’t 3× — it was around 100×. One research paper was fetched by AI roughly 700 times for every one human download. That paper is quietly informing thousands of AI-written answers; the analytics dashboard records a couple of downloads a week.

If you judge AI’s importance by your dashboard, you’ll conclude it doesn’t matter. That’s the dashboard’s blind spot, not reality.

Two different kinds of AI visit

It helps to know that “AI reading your website” is actually two separate things, done by different bots, controlled by different rules:

Training crawls. These are the slow, thorough reads that feed the models themselves. What they collect shapes what the model knows — who you are, what you do, what you’re authoritative about. It’s a long game: content crawled today informs models released later.

Live retrieval. When someone asks an assistant a question right now, many AI tools go and fetch current web pages mid-answer, then cite them. This is the fast game — it’s how your site ends up quoted, linked and recommended in an answer written three seconds ago.

On the client site above, the split between the two was not subtle:

Ninety-five per cent of the AI reading was real-time — assistants fetching pages live, mid-conversation, to answer someone’s actual question. Training sat at under one per cent, and not by choice: the host blocks training crawlers by default. Which is the point — your site can be open to one kind of AI and closed to the other without you ever having decided anything, because the two are controlled by different crawler rules. That leads to the bit I have strong feelings about.

Some hosts block AI on your behalf

A number of hosting companies now block AI crawlers by default, to save server resources. On the client site above, that’s exactly what the logs showed: live crawlers welcomed, training crawlers turned away at the door — so the material is eligible to be cited in real-time answers, but invisible to the models themselves. Nobody chose that. It was a hosting default, discovered only because we went looking.

I understand why hosts do it — crawler traffic costs them compute. But for a business website, I think blocking AI wholesale is a genuinely bad trade. You’re saving pennies of bandwidth and declining the fastest-growing channel through which new customers find businesses. It’s the 2026 equivalent of blocking Google in 2005 because the crawler was using your dial-up.

(There’s a pleasing footnote here for static sites: a pre-rendered site served from a global edge costs practically nothing to crawl, so the “save server resources” argument evaporates. Nothing strains, whether the reader is a person or a machine.)

What actually helps

The good news: most of what makes a site legible to AI is what made it legible to search engines all along, done properly.

• Real HTML. Content that exists in the page as served, not assembled by JavaScript after the fact. Static sites pass this test by construction.
• Structured data. Machine-readable statements of who you are, what you offer, and where you operate, embedded in the page.
• A consistent story. The same clear description of what you do, everywhere it appears — your site, your directory listings, your entity on Wikidata. AI models lean on corroborating sources when deciding what’s true about you.
• Answer-first pages. Pages that state the answer plainly near the top, rather than burying it under six paragraphs of preamble. Humans prefer this too, which is not a coincidence.

And some honesty about the fashionable bits: there’s an idea called llms.txt — a special file summarising your site for AI tools. This site ships one, because it costs nothing. But there’s no good evidence it makes a measurable difference today, so it’s not something I’d ever bill a client for. Beware of anyone selling “AI optimisation” by the yard.

What’s realistic

AI answers are non-deterministic. Your website is one small voice among everything a model has read, and nobody can guarantee you a slot in an answer — anyone who promises otherwise is selling something. The honest framing is the same as early SEO: you can’t force the outcome, but you can remove every barrier between your site and the systems people now use to choose, and give your material the best possible chance of being read, understood and repeated accurately.

The first step costs nothing: find out whether AI can actually read your site today. If you can’t remember making that decision, it was probably made for you — by a plugin, a firewall setting, or your host’s defaults. If you’d like a second pair of eyes on it, this is exactly the kind of small, fixed piece of work I do on the hourly clock. Email james@willcocks.uk.