Can AI Engines Read Your Site? Three Layers

I was a few paragraphs into a post about Cloudflare’s September 15 change when I stopped to check my own site, because writing "here is what your edge does" without looking at my own edge is how you end up issuing a correction in public.

Three commands.

dig +short NS amirmousavi.com
dig +short amirmousavi.com
curl -sI https://amirmousavi.com | grep -iE "^(server|cf-|x-vercel)"

The nameservers came back vern.ns.cloudflare.com. and paige.ns.cloudflare.com., which is what I expected. The A records came back 64.29.17.1 and 216.198.79.1, which is not what I expected: those are Vercel addresses, not Cloudflare addresses. The response headers came back server: Vercel and x-vercel-id: yul1::…, and nothing else. No cf-ray. No cf-cache-status. Not one Cloudflare header on the whole response.

Cloudflare runs my DNS and nothing else. Requests resolve through Cloudflare’s nameservers and then go straight to Vercel. Every AI bot control Cloudflare ships, including the categories everyone is writing about this month and the default that moves on September 15, sits at an edge my traffic never touches.

My robots.txt does not say that. It cannot: it is a text file about preferences, served by the same origin either way. The nameserver lookup does not say it either, and that is the trap, because nameservers tell you who could be in the path rather than who is. Only the second and third commands settle it.

That is the post, demonstrated on one domain: three different surfaces will each tell you a different story about who can read your site, and only one of them is in the request path.

Quick answer

Three layers decide whether an AI engine can read your site, and they do not agree with each other. robots.txt is advisory: Perplexity documents that its user-triggered fetcher "generally ignores robots.txt rules," and Google documents the same behavior for its entire user-triggered fetcher class, including the Google-Agent token added on March 20, 2026. The CDN or edge is enforced: it returns a 403 regardless of what any bot promises, and a robots.txt-parsing checker structurally cannot see it. Web Bot Auth is cryptographic and experimental: Google’s own documentation says "Not all Google user agents are using Web Bot Auth." What I do instead of trusting a checker: run dig and curl against my own hostname first, treat robots.txt as a published preference rather than a control, and keep ai-input=yes in my Content-Signal because being readable by answer engines is the point of the site.

What did three commands tell me about my own edge?

They told me which layer is real on my domain, and the answer took two of the three to reach.

dig +short NS amirmousavi.com returns Cloudflare nameservers. On its own that sentence is compatible with two completely different architectures: a site proxied through Cloudflare’s network, and a site using Cloudflare only as an authoritative DNS host. The industry shorthand is orange cloud versus grey cloud, and it is invisible from outside the dashboard unless you look at what the record resolves to.

dig +short amirmousavi.com is where it resolves. A proxied record answers with a Cloudflare anycast address from the ranges Cloudflare publishes at cloudflare.com/ips-v4. Mine answers 64.29.17.1 and 216.198.79.1, which are not in that list. www resolves through 1c2f67713588e1c5.vercel-dns-017.com. to 64.29.17.65 and 216.198.79.65. That is a Vercel edge, reached directly.

curl -sI confirms it from the other end. The apex returns HTTP/2 308 to https://www.amirmousavi.com/, so I checked the destination as well; www returns HTTP/2 200 with server: Vercel, x-vercel-cache: HIT, and x-vercel-id: yul1::…. A Cloudflare proxy adds cf-ray on essentially every response. There is no cf-ray here.

So the three layers, plus a fourth surface most people miss, and where each one lives:

Layer	What it controls	Advisory or enforced	Who honors it	Source and date
`robots.txt`	Which paths a crawler should request, by user-agent token	Advisory. A file the client chooses to read	Search crawlers largely do. User-triggered fetchers largely do not: Perplexity-User "generally ignores robots.txt rules"; Google’s user-triggered fetchers "generally ignore robots.txt rules"	Perplexity crawler docs (read August 21, 2026) · Google user-triggered fetchers (updated August 19, 2026)
CDN or edge bot rules	Whether the request gets a response at all, by category, IP, signature and user agent	Enforced. The decision happens in the request path	Nobody has to honour it. It is applied to them	Cloudflare: Block AI Bots (updated July 1, 2026) · AWS WAF Bot Control rule group (read August 21, 2026)
`NOARCHIVE` / `NOCACHE` robots meta tag	Whether Microsoft uses an already-indexed page in Bing Chat answers and in model training	Advisory, and vendor-specific. Lives in the page `<head>`, not in a site-level file	Microsoft, for its own surfaces	Bing Webmaster Blog, Fabrice Canel (September 22, 2023)
Web Bot Auth	Whether a bot can cryptographically prove it is who it claims, per request	Neither yet. It is an identity signal an edge may act on	Google is testing it; Cloudflare and AWS WAF can consume it	Google: Web Bot Auth (Experimental) (updated May 4, 2026)

Four rows for three layers: the Bing meta tag is a fourth surface rather than a fourth layer, and it is the one people are most surprised to find. Three of the four are invisible to a robots.txt audit. Google puts the edge row in its own documentation. Its page on AI features and your website, last updated December 10, 2025, lists among SEO best practices: "Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure." That clause has been sitting in Google’s documentation for at least eight months and I have never once seen it quoted in a GEO checklist.

Does robots.txt still control who reads your site?

It never controlled anything. It declares, and the client decides. What changed in 2026 is who joined that class: a client that does not just fetch, but navigates and acts.

Perplexity’s crawler documentation splits its traffic in two. PerplexityBot is the indexing crawler, and Perplexity’s guidance is straightforward: "To ensure your site appears in search results, we recommend allowing PerplexityBot in your site’s robots.txt file and permitting requests from our published IP ranges listed below." That second half is Perplexity pointing past robots.txt at the enforcement layer, in its own documentation. Perplexity-User is the fetcher that runs when someone asks a question. Of that one, Perplexity writes: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Not may not apply. Ignores.

Google documents an entire category the same way. Its user-triggered fetchers page, last updated August 19, 2026, states: "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules." Nine fetchers are listed, and Google notes the list "is not exhaustive." The newest one matters most. Google-Agent was added on March 20, 2026; the changelog entry describes it as rolling out "over the next few weeks," to "be used by Google agents hosted on Google infrastructure to navigate the web and perform actions upon user request." An agent that navigates and acts, in a class that ignores robots.txt, under a token most blocklists have never heard of.

I already published the full vendor roster for OpenAI, Anthropic, Meta and Google Search rather than reprint it here, in how to measure AI search traffic — including the exact sentences each vendor uses about robots.txt and where their tokens diverge. The short version, and it holds up on re-reading today: of the three assistant vendors in that roster, two — OpenAI and Meta — publish a robots.txt exemption for user-triggered fetches, and Anthropic publishes none. Treating them as a single "AI bots" bucket is the error underneath most blocklists in circulation.

Then there is the case where the declaration was honored and the request arrived anyway. On August 4, 2025, Cloudflare reported that "Perplexity uses not only their declared user-agent, but also a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked," and that in response "we have de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling." Whatever you conclude about that specific incident, it establishes the general point cleanly: a rule written against a user-agent string is a rule written against a value the client chooses.

AWS has quietly encoded the same skepticism in its taxonomy. Its Bot Control rule group applies a label, bot:user_triggered:verified, for "a bot that is similar to a verified bot, but that might be directly invoked by end users," and then says what it does with it: "This category of bot is treated by the Bot Control rules like an unverified bot." A vendor writing a WAF has already decided that user-triggered means untrusted.

What actually changes at the edge on September 15, 2026?

Cloudflare classifies AI traffic by behavior — Search, Agent, Training — and from September 15, 2026 the legacy "Block AI bots" preset stops excluding mixed-purpose crawlers from that classification.

That sentence is the change. Cloudflare’s Block AI Bots documentation, last updated July 1, 2026, defines the three presets: Search is "crawlers that collect or index your content to answer questions about it later"; Agent is "automated activity acting in real time on a person’s behalf, such as chat fetch bots and browser-use agents"; Training is "crawlers taking your content to train or fine-tune a model, including mixed-purpose crawlers that are used both for Training and for Search." Read that last clause twice. The reclassification is inside the definition of Training, not only inside a default that flips on a date.

Two separate things happen on September 15, and the coverage keeps welding them together.

Configuration	Behaviour before September 15, 2026	Behaviour after	Source
Legacy "Block AI bots" preset, already enabled	Blocks training crawlers. "This option excludes mixed-purpose bots that are used both for Training and for Search."	Mixed-purpose crawlers "will also be blocked by all configurations to block AI training, including the legacy 'Block AI bots' option." The preset is labeled "Block AI bots [Deprecating on September 15, 2026]"	Cloudflare docs, July 1, 2026
Any configuration that blocks Training	Single-purpose training crawlers blocked	"Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training"	Cloudflare blog, July 1, 2026
A domain onboarding to Cloudflare after the date	Not applicable	"For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default."	Cloudflare blog, July 1, 2026
Existing domain, no new action taken	Current settings	Defaults are not restated for existing domains. "Before September 15, all customers can opt out of these new defaults"	Cloudflare docs, July 1, 2026

Note which row carries which quote. The docs page does not name Googlebot, Applebot or BingBot anywhere; that sentence is on the blog. I checked because a first draft of my own notes had it on the wrong URL, and a quote attached to the wrong source is a quote a reader cannot verify.

Now the correction I think matters most. A lot of secondary coverage describes the September 15 defaults as landing on "new customers, new sites by existing customers, and all existing free customers." Cloudflare’s own post says "all new domains onboarding to Cloudflare." I could not find the broader scope stated anywhere at source, so I am not repeating it. The genuinely broad change for existing customers is the other one: the mixed-purpose rule, which reaches anyone who ever ticked the legacy toggle. Somebody who enabled "Block AI bots" in 2024 as a training opt-out made a decision whose meaning changes without them touching it. That is a better story than the one being told, and it is the one that is documented.

It is also worth knowing what stays allowed. Cloudflare’s verified bot categories page, updated July 1, 2026, still lists OAI-SearchBot under the legacy value "AI Search" and Googlebot under "Search Engine Crawler," and notes that under the taxonomy introduced on July 1, 2026 both are treated as Search behavior, the legacy value being "retained for backward compatibility with existing rules." Search-classified traffic remains allowed under the new defaults. The exposure is concentrated in crawlers that do two jobs with one token.

One structural note that surprised me: "Each blocking option will block Verified bots classified with that behavior, plus additional unverified bots that fall under these classifications." Cloudflare’s own categories page confirms the shift in plain terms: "Historically, Verified bots have been excluded in default bot configurations across all plans. Now, all customers have the option to configure AI bot policies to define their block vs. allow expectations." Verified status used to be a pass. It is now just a label the policy reads.

Why can a robots.txt checker say yes while your edge says no?

Because a checker fetches one text file and parses it, and an edge decision is made per request, against a user agent, an IP, and increasingly a signature. These are not the same measurement. A robots.txt-parsing tool cannot observe a WAF rule any more than reading a restaurant’s posted hours tells you whether the door is locked.

This is the same failure mode I wrote about in can you trust AI visibility scores: a tool reports the thing it can measure, the buyer reads it as the thing they care about, and nobody checks the gap. A green check for "AI crawlers allowed" means the file permits them. It does not mean a request arrives.

There is a third place to look that almost no audit does, and it is not a file at all. Microsoft’s opt-out for generative answers is a robots meta tag, in the page head. Fabrice Canel published it on September 22, 2023: content tagged NOARCHIVE "will not be included in Bing Chat answers, not be linked to in the answers," and "we will not use the content for training Microsoft’s generative AI foundation models." Content with NOCACHE "may be included in Bing Chat answers. We will only display URL/Snippet/Title in the answer." And the default, stated plainly: "Content without NOCACHE tag and without NOARCHIVE tag may be included in Bing Chat answers … site content may be used in training our generative AI foundation models." Two caveats the page carries and most summaries drop: content tagged either way "will still appear in our search results," and if both tags are present Microsoft treats it as NOCACHE. The post is from September 2023 and still says "Bing Chat," a name Microsoft has since retired, which is worth weighing before treating it as current.

That control lives per page, in HTML, and a site-level robots.txt audit will never see it. A CMS template, a plugin, or a staging default can set it across thousands of URLs without anyone deciding to. If you inherited a site, this is a five-second check worth making, and it is the sort of thing I keep on the SEO architecture checklist rather than in anyone’s memory.

Is Cloudflare the outlier, or is everyone doing this?

On the evidence I could verify today, Cloudflare is the outlier. It is the only one of the platforms I checked that moves a default on a calendar date.

Vercel, which is my actual edge, states it about as clearly as a doc can. On its WAF managed rulesets page, last updated July 17, 2026: "The ruleset is inactive by default. In the dashboard this is labeled Allow. Matching traffic is not evaluated and reaches your application." The available actions are Log and Deny, and you choose one. Nothing happens to AI traffic unless somebody turns it on.

Netlify ships blocking as an installable extension. Its documentation, last updated February 13, 2026, describes "The User Agent Blocker extension is built by Netlify" which "can block web requests from a preset list of common AI crawlers, SEO/Search crawlers that you choose from," implemented as "a Netlify Edge Function." Opt-in by construction: an extension you install and configure.

Fastly puts it behind a purchase and a config. ContentGuard requires that you buy Bot Management and "deploy Bot Management using pre-cache inspection" on each service, then write VCL snippets in vcl_recv against variables that "all start with fastly.bot." — including fastly.bot.category.is_ai_crawler. Powerful, and nobody arrives there by accident.

AWS WAF is the interesting one. I had it filed as unverified, so I went and read the rule group documentation myself. Bot Control is a managed rule group you add to a web ACL, and you are "charged additional fees when you use this managed rule group." Inside it, one rule stands apart from the other fifteen category rules. CategoryAI "inspects for artificial intelligence (AI) bots" and, after a note in the source, "applies the action to all matches, regardless of whether the bots are verified or unverified. Rule action: Block." Every other Category* rule applies "only to unverified bots." AWS singled out AI traffic, and it did it inside a product you have to opt into and pay for.

I did not verify Akamai, so Akamai is not in this section. I would rather leave a gap than characterise a vendor’s default from a summary.

What is Web Bot Auth, and is it ready to matter?

It is the layer that would make the other two coherent, and it is not ready.

Google’s documentation for it was last updated on May 4, 2026, on a page headed "Authenticate requests with Web Bot Auth (experimental)." The idea is simple and overdue: instead of a bot asserting an identity in a header anyone can copy, it signs the request. "Instead of relying solely on self-reported headers and IP addresses, Web Bot Auth allows agents to cryptographically sign their requests." Mechanically, participating requests carry a Signature-Agent header set to g="https://agent.bot.goog", you fetch the public keys from https://agent.bot.goog/.well-known/http-message-signatures-directory, and you verify against HTTP Message Signatures, RFC 9421.

Google is unusually direct about the state of it. "Web Bot Auth is currently a draft specification developed by the IETF WBA Working Group, and it may change over time." "Not all Google user agents are using Web Bot Auth." "We don’t sign every request of a particular agent. Be sure that you fall back to the established methods of bot verification." And the operative instruction for anyone tempted to rebuild their access rules on it: "We recommend that in addition to Web Bot Auth you continue relying on IP addresses, reverse DNS, and user-agent strings as we gradually roll out signed traffic."

The edge vendors are further along than that reads. Cloudflare has a Web Bot Auth reference page in its bot verification docs, last updated July 1, 2026, defining it as "an authentication method that leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot." AWS WAF’s Bot Control already emits labels for it: web_bot_auth:verified, web_bot_auth:invalid, web_bot_auth:expired, and web_bot_auth:unknown_bot, with a note that "When the web_bot_auth:verified label is present, the Category<category_name> and TGT_TokenAbsent rules do not match, allowing verified WBA hosts to proceed."

I cannot tell from that documentation how the WBA exemption interacts with CategoryAI, given that CategoryAI is documented as applying "to all matches, regardless of whether the bots are verified or unverified." AWS’s note uses the placeholder Category<category_name>, which on its face covers every Category rule including CategoryAI. Those two sentences are in tension and I am not going to resolve them by guessing. If you run AWS WAF and you care about the answer, test it rather than infer it.

Here is why I still think layer three is the one to watch. Layers one and two both spend most of their effort deciding whether to believe a user-agent string. Layer three ends the argument. When signing is ubiquitous, "who is allowed to read this" stops being a guess about user agents and becomes a policy about verified identities. We are not there. Google says so itself.

How to audit your own edge

Six steps, in order. Each one has a way it misleads, and the caveat is the point.

Read your nameservers. dig +short NS example.com. This tells you who runs your DNS. How it misleads: it does not tell you who is in the request path. Mine returns Cloudflare nameservers on a site with no Cloudflare proxy. If you stop here, you will spend an afternoon looking for a dashboard you do not have.
Resolve the record you actually serve. dig +short example.com and dig +short www.example.com. Compare the answers against cloudflare.com/ips-v4 or your CDN’s published ranges. A proxied record answers with the CDN’s anycast address; a DNS-only record answers with your origin. How it misleads: apex and www are frequently configured differently. Mine differ.
Read the response headers on the hostname you actually serve. curl -sI https://www.example.com. Look for server, cf-ray, x-vercel-id, x-amz-cf-id, x-served-by. How it misleads: my apex returns a 308 redirect, so headers there describe the redirector, not the application. Follow to the destination. And absence of a header is weaker evidence than presence of one; some configurations strip them.
Request as a declared AI crawler and diff the result. curl -A "OAI-SearchBot/1.4" -sI https://www.example.com, then compare status code and content-length against your browser-user-agent request. How it misleads, and this is the important caveat: you are only testing user-agent string matching. Real bot rules verify IP ranges, reverse DNS, and increasingly signatures, none of which curl can satisfy or trigger. A 200 here does not prove the real OAI-SearchBot gets a 200. It proves nobody wrote a naive user-agent rule. On my domain both requests returned HTTP/2 200 with an identical content-length: 41843.
Check for a page-level meta tag. curl -s https://www.example.com/ | grep -i 'name="robots"'. You are looking for noarchive or nocache, which is Microsoft’s generative-answer opt-out. How it misleads: one URL is not a site. Templates apply this unevenly, so check an article, a category page, and anything behind a different template before concluding.
Read robots.txt last, and read it as a declaration. curl -s https://www.example.com/robots.txt. How it misleads: it tells you what you asked for. It tells you nothing about what happened. Reading it first is how a perfectly correct file talks you out of investigating a 403 you are actually serving. Named user-agent groups fully override the * group, so a group you added for one crawler silently exempts it from every rule you wrote above it.

If you are doing this before a launch rather than as a diagnosis, steps two through four belong on the website launch checklist alongside the other things nobody remembers until the first week of traffic.

My working take

My site ships Content-Signal: search=yes, ai-input=yes, ai-train=no on every user-agent group in robots.txt, and the ai-input=yes is deliberate. It says answer engines may ground responses in this content. I get asked why I do not set it to no, usually by someone who reads it as a defensive posture. It is not one. Setting ai-input=no does not protect the content — the pages are public, the same crawlers fetch them, and the enforcement layer is not there anyway. All it does is ask to be left out of the answers. For a site whose entire llms.txt and Markdown-rendition work exists to be readable by machines, that would be a strange thing to ask for.

I hold two things at once and they are less contradictory than they sound. I want to be read, and I want to know who is reading. Those need different layers. ai-input=yes is a statement of intent at layer one. Knowing whether it is honored happens in logs, which is a measurement problem rather than an access problem. This whole post is about who can read your site; the mirror-image question, what an agent can write into your stack once you connect one, is in a separate post.

What I do not know: whether any answer engine parses Content-Signal at all. I have not found a vendor that documents doing so, and I am not going to infer it from the absence of evidence. I ship it because it is a cheap, machine-readable statement of terms, and because I would rather have said it in public than not. If that turns out to be a file nobody reads, I will have lost fifty-eight lines of text.

I also do not have a Cloudflare edge to configure, which is why there is no dashboard walkthrough in this post. My September 15 exposure is zero, and I only know that because I checked instead of assuming. That is the transferable part. I do not know how many people writing about this change have run dig against their own apex. I know that I had not, until three paragraphs before I started this post, and that I was wrong about my own answer.

The boring conclusion, and I believe it is the right one: robots.txt is where you state a preference, the edge is where a decision gets made, and until Web Bot Auth is not experimental, anything you enforce is enforced against a string. Design accordingly. Audit the layer that is in the request path, not the one that is easiest to read.

Frequently asked questions about AI crawler access

Does blocking AI training in Cloudflare block Googlebot?

After September 15, 2026, it can. Cloudflare’s July 1, 2026 blog post states that "since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training," whether through the new category options or the legacy "Block AI bots" service. Cloudflare says customers can opt out of the new default configurations in Security settings before the date. This applies only to sites actually proxied through Cloudflare, which is worth verifying before acting on it.

Do I need to change anything before September 15, 2026?

Only if your traffic passes through Cloudflare’s proxy and you have a configuration that blocks Training, including the legacy toggle. Confirm the proxy first with dig +short yourdomain.com and a check for a cf-ray response header; Cloudflare nameservers alone do not confirm it. If you are proxied and blocking Training, decide deliberately whether you want mixed-purpose search crawlers included, because after that date they are.

Can a robots.txt checker tell me whether AI crawlers are blocked?

No, and the limitation is structural rather than a quality problem. A checker fetches /robots.txt and parses directives; an edge rule is evaluated per request against user agent, IP, and potentially a cryptographic signature. Nothing about the file’s contents reveals a firewall decision. The only tools that can observe it are ones that send real requests with crawler user agents and report the status code they receive.

Which AI vendors say their fetchers ignore robots.txt?

Perplexity’s documentation says of Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Google says of its entire user-triggered fetcher class, which includes Google-Agent: "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules." Vendor policies diverge more than the roundups suggest, and I compared them token by token in how to measure AI search traffic.

Should I implement Web Bot Auth now?

Not as a control you depend on. Google’s documentation, updated May 4, 2026, calls its implementation experimental, states that "Not all Google user agents are using Web Bot Auth," and recommends continuing to rely on IP addresses, reverse DNS, and user-agent strings. If your CDN can already consume the signal, as Cloudflare and AWS WAF can, there is no reason to turn that off. Just do not build an access policy that assumes signatures are present.

Sources I used

Verified at source: August 21, 2026.