This started with a skateboarder.
Our 404 page wasn't showing. Instead of the branded page we'd built — the one that says "Ironic, given what we do" — visitors to a missing URL got our hosting provider's default: a stock photo of a man on a skateboard, gliding through a purple void, apologising for an accident that was not intentional. The configuration was correct. The file existed. The page still wouldn't fire.
So we did what you do when the settings say one thing and the site does another. We stopped looking at the settings and started reading the raw server logs.
The 404 answer was in there — the host's edge layer was answering missing URLs itself, before our configuration was ever consulted. But that wasn't the interesting line. The interesting line was this one:
GPTBot — OpenAI's crawler, the thing that decides whether ChatGPT knows your site exists — had requested our homepage and been refused at the door. A 403. Forbidden. Not by our robots.txt, which welcomes it. Not by any setting we chose. By the hosting provider's bot protection, running silently at a layer above everything we control, on its own judgement of who deserves entry.
The same log showed other crawlers getting the same treatment. Our site's entire proposition is helping businesses understand what AI systems see when they look at a website. And for some unknown period, the answer for our own site was: nothing. Not because of anything we configured. Because of a default we didn't know existed, on infrastructure we don't operate, discoverable only by reading raw logs that most site owners will never open.
Meanwhile, in the opposite direction
The same week, we were preparing an AI-readability file for a client on Shopify — a document that tells AI systems what the store is, what it sells and how its content should be understood. Standard practice: write the file, upload it, done.
Except Shopify got there first, and didn't mention it. In early May 2026, Shopify quietly switched on a set of AI-facing files for stores on its platform — llms.txt, agents.md and a cluster of technical endpoints — with no announcement and no email to merchants. Open any Shopify store's /agents.md today and a generated file loads that nobody on the merchant's team created. It isn't a description. It's an instruction manual: how an AI shopping agent should discover the catalogue, build a basket and start a checkout. Shopify's own documentation now describes agents.md as the canonical agent discovery file that every store automatically serves.
Two details matter here, and both reward suspicion.
First, the merchant's traditional control file doesn't govern any of this. Shopify's documentation is admirably direct about it: robots.txt rules are advisory, and blocking AI crawlers there affects only open-web scraping — it does not stop product data being sent to AI channels through Shopify's own catalogue pipe. A merchant who "blocked AI" in robots.txt and considered the job done hasn't blocked the main road. They've closed a window while the platform holds the shop door open.
Second, if you replace Shopify's generated file with your own — which is exactly what a well-meaning SEO would do, and exactly what we were about to do — your version replaces theirs entirely. It doesn't merge. Which means a nicely written brand document can silently amputate the machinery that lets AI agents actually transact with the store. The file looks better and the store works worse, and nothing anywhere tells you.
And then the store doing both at once
Here's where it gets properly absurd. During the same stretch of work we crawled a storefront — it doesn't matter whose — whose robots.txt carried Shopify's full agentic boilerplate. Allow all. AI shopping agents formally invited. The paperwork of a business that wants every machine on the internet reading its catalogue.
The crawl came back with 146 of 172 pages blocked. Rate limited. We slowed to a single polite request at a time. Still blocked. The stated policy welcomed every crawler in existence; the network layer was turning all of them away, including — presumably — the AI shopping agents the policy had just invited in.
Nobody at that business decided this. That's the point. One layer was configured by the platform, another by the host or a security vendor, the policy file by a template. No human had checked whether the layers agree — possibly ever. The store is formally open and functionally invisible, and its owner has no idea.
The pattern
Three vendors, one week, one pattern. Our host blocks AI crawlers by default and doesn't say so. Shopify feeds AI channels by default and didn't announce it. A third business inherited both behaviours at once, pointing in opposite directions. In every case the decision was made above or below the layer the site owner can see — and in every case the visible controls (the settings panel, robots.txt, the theme) told a story the infrastructure contradicted.
And it's about to get a deadline. Cloudflare — the layer sitting in front of a vast slice of the web — now sorts crawlers into Search, Agent and Training, and from September 15 applies the strictest rule to multi-purpose bots. Googlebot crawls for both search and AI training; block Training, and on Cloudflare's network you've blocked Google — at the network level, where robots.txt can't help you. Anyone who once clicked "Block AI bots" — a visible, sensible-looking control — inherits this behaviour automatically, and free-tier sites that never touched their settings get moved to the new defaults on the same date. The setting says "protect my content from AI." The infrastructure hears "remove me from search." Fourth vendor, same disease — this time with a date on it. If your site sits behind Cloudflare, that dashboard is worth ten minutes before mid-September.
This is the actual state of AI visibility in 2026. It is not a content problem or a keywords problem. It's a layers problem. Between a website and an AI system sit an edge network, a bot-protection product, a platform's commerce pipe and a set of auto-generated files — each with its own defaults, each changing without notice, each capable of overruling everything beneath it.
What checking actually looks like
None of this requires exotic tooling. It requires asking a different question. Not "what do the settings say?" but "what actually happens when an AI crawler knocks?" In practice: request the site as GPTBot, ClaudeBot and PerplexityBot and compare what comes back against a normal browser — a 403 or a challenge page at that layer makes everything downstream irrelevant. Read the server logs for what real crawlers received, because the logs don't have opinions. Fetch the AI-facing files actually being served on the live domain — not the ones you wrote, the ones visitors get — and check the platform hasn't quietly replaced, redirected or supplemented them. And when a platform ships a default, document it before you override it, because the default may contain working machinery the shiny replacement lacks.
The live experiment
Or skip the tooling entirely and run the test anyone can run. With our host's AI Audit dashboard showing every AI crawler as Allowed and zero blocked, we asked the AI systems themselves to read the site while we watched the server logs.
ChatGPT, asked to summarise the site, suggested the domain might be offline. Given the exact URL of an article and told to fetch it, it reported the page was "blocking retrieval by my browser." Perplexity, same test: "I can't fetch that page directly right now" — and then it did something more instructive than failing. It answered the question anyway, from Wikipedia and a rival site on the same subject. When AI can't read you, it doesn't go quiet. It cites someone else.
The differential made it conclusive that something was wrong, and specific to us. The same page, the same morning: Google's crawler fetched it. Anthropic's fetcher retrieved it in full. Bing was reaching the origin happily. OpenAI and Perplexity were turned away. And robots.txt — checked, clean — welcomed every one of them by name. Whatever was deciding who got in was doing it selectively, above our configuration, while the dashboard continued to report everything as allowed.
Before pointing at the host, we ruled out the boring explanation: our sister site runs on WordPress, this one is static HTML and custom-built usually means custom problems. A control test — the same AI systems, fetching a page on the WordPress site, on the same hosting account, minutes later — came back clean. Full article, forty-one kilobytes, served in three milliseconds, logged and timestamped. So it wasn't the platform. It wasn't robots.txt. It wasn't our IP reputation, which only ever explained what a human browser saw, never why corporate AI infrastructure was failing to connect. Each of those took a support conversation, a test and a log export to eliminate — and eliminating the wrong answers turned out to be most of the work.
What it actually was
The real cause was almost embarrassingly ordinary once we found it, and it had been sitting in plain sight the entire time: the SSL certificate on this domain — not the sister site's, this one specifically — was stale. Our own dashboard showed it as "Active," which we'd read on day one and moved past, because "Active" sounds like "fine." It wasn't lying, exactly. It just wasn't measuring the thing that mattered.
Ordinary browsers tolerate a shaky certificate handshake and quietly paper over it, which is why human visitors only ever saw an occasional "checking your browser" verification screen — mildly annoying, easy to write off as a glitch, never suspicious enough to chase. AI crawlers, running lean, strict HTTP clients with none of a browser's patience for imperfect TLS, don't get that grace. Their connections simply failed before any page, challenge screen or robots.txt file was ever reached. Two completely different symptoms — a cosmetic annoyance for humans, total invisibility for machines — with one root cause neither the CDN panel, the security settings nor three rounds of support diagnosis ever named.
A fresh certificate, reissued and given time to propagate, fixed it. We tested the same way we'd tested the failure: asked ChatGPT and Perplexity to fetch the same article again. Both succeeded, in detail — accurate, specific summaries of the actual argument, not the vague paraphrase or the "might not exist" guess from before. The server logs agreed: real requests, real user agents, real content served, nothing challenged.
We now check this properly, for the slightly embarrassing reason that we got caught by it ourselves. The site preaching AI visibility was invisible to the two AI systems that increasingly decide who gets recommended, and we only found out because a log line looked odd on a Friday and we didn't let it go.
The controls you can see aren't the controls that matter. Neither, it turned out, was the setting we spent four days convinced it must be. What mattered was one level further down than any dashboard goes — and the only way to find it was to stop trusting what the panels said and test what actually happened on the wire.
How to check your own site
You don't need a hosting background for this, just the willingness to ask an AI to do something and watch what happens. Open ChatGPT or Perplexity and paste a URL from your own site with an instruction to fetch and summarise it — not search for it, fetch the exact address. A generic answer, a "might not be online" response or an outright failure to retrieve it is your first signal. If you have access to server logs, check them for the same window: a real crawler visit leaves a real user agent and a real response size behind. A certificate that shows "Active" in a dashboard is worth a second look too — active isn't the same as currently valid for every kind of visitor, and the gap between those two only shows up when something other than a browser tries to connect.
If the last two thousand words were a bit much, here is the whole thing again, as told to a seven-year-old.
You have a shop. Robots visit shops — some work for Google, some for ChatGPT — and they read everything, so that when a person asks "where's a good shop for this?", the robot can answer. You put a welcome mat out for every robot. Ours literally says "AI crawlers — welcome, read everything."
Then we read the visitor book — the boring list of everyone who came to the door — and found ChatGPT's robot being turned away. Not by us. By the landlord. Without telling us. The landlord's control panel, the buttons we can actually press, said every robot was welcome. The street said otherwise.
So we tested it. We asked the robots to visit two shops on the same street, with the same landlord, and watched both visitor books at once. At one shop the robot walked in and read a whole article — it's in the book, timestamped, forty-one kilobytes. At the other shop the robot never appeared in the book at all. It wasn't turned away at the door. It never reached the door.
Why it matters: when a robot can't read your shop, it doesn't go quiet. It sends the customer to a different shop. We watched that happen too — asked about our own article, one AI recommended somebody else's.
What you can do: don't trust the buttons. Ask the robots to visit — paste your own URL into ChatGPT and Perplexity and tell them to fetch it — then read your visitor book and see who actually arrived. Ten minutes, free, and you'll know something about your site that no dashboard will tell you.
P.S. — Somewhere in all this we also went looking for a missing 404 page and found a skateboarder instead — Hostinger's default error graphic, served from a directory one level above anywhere a customer is allowed to look. We never did evict him; he lives behind a wall we can't open, in a folder our own logs can see and our own file manager can't reach. He seems like the right mascot for a week spent discovering that the thing actually in charge is never quite where the settings page points.