Cloudflare Starts Blocking AI Crawlers on September 15. Is Your Site Ready?

Updated 20 October 2026. The change went live on September 15, and what Cloudflare shipped differs from the July announcement this post was first written against. This version describes the final rules, from Cloudflare's September 15 announcement. The short version for a service business: your existing settings carried over, a new Disallow AI Training setting lets you refuse AI training without leaving search, and Block now stops Googlebot, Bingbot and Applebot as well.

On September 15, 2026, Cloudflare changed what happens when an AI crawler knocks on your website's door.

If you run a clinic, a firm, or a contracting business, your first instinct is probably that this is a publisher problem, something for news sites and media companies to worry about. Mostly, you're right. The strictest new defaults are for sites that earn money from ads, and your services page doesn't run banner ads.

Here's why it still matters to you: the change is a forcing function, not a threat. In auditing local service businesses across the GTA, the single most common technical failure we find isn't bad schema or thin content. It's a site that is actively blocking the AI crawlers it desperately wants to be read by, usually because somebody clicked a one-button "block AI" setting in Cloudflare, a security plugin or a hosting panel.

Those businesses didn't lose AI visibility on September 15. Many had lost it months earlier, and nobody told them.

This post covers what Cloudflare actually changed, whether it affects your site, how to check in about four minutes whether you're invisible to ChatGPT, and exactly which crawlers a service business should allow.

What did Cloudflare change on September 15, 2026?

Cloudflare sorts crawler behaviour into three categories, and since July every plan, free included, can set each one separately. The vocabulary is worth learning, because it now shows up everywhere:

• Search: a crawler collecting and indexing content to answer questions later. This is the classic Googlebot job.
• Agent: automated behaviour acting in real time on a specific person's behalf. This is what fires when someone asks an assistant about your business right now and it goes and fetches your page.
• Training: a crawler taking your content to train or fine-tune a model.

A mixed-use crawler is one bot doing several of those jobs under a single user agent, so a site owner can't allow one and refuse another. When Cloudflare announced the policy on July 1, it pointed squarely at Google, Microsoft and Apple, whose crawlers index for search and feed AI products from the same fetch.

What actually shipped on September 15:

• A new Training setting, Disallow AI Training. It publishes a no-training preference in your robots.txt. Googlebot, Bingbot and Applebot stay allowed for search, and training-only crawlers, including OpenAI's, Anthropic's, Amazon's and Meta's, are blocked. Cloudflare says blocking those training-only crawlers does not affect search. One caveat it states itself: Bing will not read the robots.txt preference until Microsoft's support lands, targeted for early 2027.
• Block now blocks everything. Block and "Block on pages with ads" now apply to mixed-use crawlers too, so choosing Block stops Googlebot, Bingbot and Applebot from reaching your site at all, search included.
• The old one-click "Block AI bots" setting is being retired. Existing settings migrated automatically. A site that had it switched on now has Search on Allow, Training on Disallow AI Training, and Agent on Block on pages with ads.
• New domains get a preset. When you add a domain, Cloudflare asks whether the site makes money from ads. If it does, Training defaults to Disallow AI Training and Agent to Block on pages with ads. If it doesn't, all three are set to Allow.

The July announcement went further than this. It said mixed-use crawlers would be blocked outright on pages with ads, and that existing free-plan sites that hadn't changed their settings would get the new defaults. The September version replaced that with the Disallow AI Training setting, after Apple, Google and Microsoft agreed to honor a robots.txt no-training preference (Microsoft by early 2027), and says existing customers' settings carry over on their own.

Does this affect my business website?

Probably not directly, and you should still check. Cloudflare says existing settings carried over on their own, and the strict presets are for new domains that earn money from ads, which most clinic, firm and contractor websites don't.

Two situations are worth a look:

If you ever switched on Cloudflare's "Block AI bots" setting, you have been migrated: Training is now Disallow AI Training, and agents are blocked on pages with ads. On a site without ads, that keeps you in search and in live AI lookups and refuses AI training crawlers. If you want future AI models to learn about your business, set Training back to Allow.

If you're tempted to tidy up and choose Block, don't. Block now stops Googlebot, Bingbot and Applebot as well, which removes you from Google and Bing search, not just from AI training.

The honest summary: this policy was not written with your dental practice in mind. It was written to make Google and Apple stop hiding AI training inside a search crawl. Your dental practice just happens to sit on the same infrastructure.

Why are so many small business sites already invisible to ChatGPT?

Because blocking AI crawlers takes one click, and the button doesn't explain what it costs you. Blocking scrapers sounds unambiguously good, so owners switch it on in a security plugin, a hosting panel or Cloudflare without asking which crawlers it stops.

Which ones matters. Training crawlers feed future models. Search crawlers build the index an assistant looks things up in, and agent requests fetch your page live while someone waits for an answer. Block the last two and you drop out of the answers being written today.

Think about what actually happens when a prospective patient asks ChatGPT "who's the best dentist in Liberty Village for a cracked tooth?" The model doesn't only rely on what it memorised during training. It looks for current pages, through ChatGPT's search index and, for some requests, a live fetch. If your site refuses those requests, you don't appear as an outdated entry. You don't appear at all, and a competitor who left their door open gets described in detail instead.

We see the same self-inflicted wound in three other places:

• WordPress security plugins with an "AI bot protection" feature enabled by default after an update.
• Managed hosting platforms that added blanket AI-bot blocking as a customer-friendly feature.
• Copy-pasted robots.txt files from a template or a forum post that disallow GPTBot and friends without comment.

None of these announce themselves. Your rankings don't drop. Your analytics look normal, because AI referrals are a small share of traffic for most sites and easy to miss. You simply stop being recommended, quietly, and there's no notification.

How do I check if my site is blocking AI crawlers?

Run these four checks in order. The whole thing takes about four minutes and needs no tools beyond a browser.

1. Read your robots.txt. Go to yourdomain.com/robots.txt. Look for any Disallow: / sitting under a User-agent: line that names OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot or Bingbot. Those are the search and live-fetch agents, so if you find one, that's your answer. A disallow for a training crawler such as GPTBot, ClaudeBot or Google-Extended keeps you out of model training rather than out of today's answers, so make sure it was a deliberate choice.

2. Check Cloudflare's AI crawler controls. In the Cloudflare dashboard, open your domain's Security Settings and find the AI bot controls for Search, Training and Agent. Cloudflare enforces these at the network level regardless of what your robots.txt says, which is why people forget they exist. Search and Agent should say Allow. Training can say Allow or Disallow AI Training. Nothing should say Block: since September 15, Block also stops Googlebot, Bingbot and Applebot.

3. Check your CMS and host. In WordPress, review any security or SEO plugin with bot-blocking options. On managed hosting, look for an "AI bot" or "scraper protection" setting in your control panel.

4. Ask the engines directly. Open ChatGPT, Perplexity and Gemini and ask each one the question a real customer would ask — "best [your service] in [your neighbourhood]" — then ask "what can you tell me about [your business name]?" If the engines can describe your competitors in specifics and go vague or wrong on you, you have a visibility problem worth diagnosing properly.

That fourth check is the one that tells you what the first three cost you. It's also the manual version of what our AI Visibility Audit does across six engines with a scored report — but you can absolutely do a rough version yourself in ten minutes, and you should.

Which AI crawlers should a service business allow?

Allow every crawler that can put you in an answer, and make a deliberate decision about the ones that only take. For a local service business trying to win AI recommendations, the calculus is simpler than it is for a publisher: you are not selling content, you're selling appointments, so being read is almost always worth more than being protected.

On Cloudflare, that translates to Search: Allow, Agent: Allow, and Training: Allow, or Disallow AI Training if you'd rather stay out of model training. Disallow AI Training keeps you in search; Block does not.

The one that trips people up is Google-Extended, in both directions. Google's crawler documentation says it controls whether your content may be used to train Gemini models and to ground answers in the Gemini app and Vertex AI, and that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode are part of Search, and Google's AI features page names Googlebot as the control for them. So blocking Google-Extended takes you out of Gemini app answers grounded in Search, not out of AI Overviews. The setting that removes you from AI Overviews is a block on Googlebot, which is exactly what Cloudflare's Block option now does.

If you want the working version, here's a robots.txt block that explicitly welcomes the engines that matter while keeping bots off your admin and API paths:

# AI answer engines, explicitly allowed
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Googlebot
User-agent: Bingbot
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /login/

Sitemap: https://yourdomain.com/sitemap.xml

That's close to what runs on this site. Pair it with an llms.txt file at your root — a short markdown summary of who you are, what you do, and your key URLs — which gives a model a clean, authoritative description to work from instead of one assembled from whatever it scraped. Our schema markup templates for AI cover the structured-data half of the same job.

What is pay per crawl, and will I ever get paid for it?

Pay per crawl is Cloudflare's mechanism for charging AI companies to access your content, and for a local service business the realistic answer is that you will never meaningfully earn from it. It works over standard HTTP: a crawler either presents payment intent in its request headers, or receives a 402 Payment Required response with the price attached. Publishers set one of three states per crawler — Allow (free access), Charge (pay the configured price), or Block (no access at any price).

Cloudflare has signalled that this is evolving from pay-per-crawl into pay-per-use, where publishers get paid when their content actually creates value — when it appears in an AI answer, or when an agent buys premium information to complete a task — rather than merely when a bot fetches a page.

For The New York Times, that's a business model. For a physiotherapy clinic in North York, it's the wrong lever entirely. Your content isn't the product; it's the marketing for the product. Charging ChatGPT a fraction of a cent to read your services page, and thereby not appearing when a patient asks who treats vertigo, is the worst trade in the building.

Set your AI crawlers to Allow and move on. The money is in the appointment, not the crawl.

What should I do now?

Work through this list once. For most businesses it's a half-hour job, and it's the difference between being readable and being absent.

1. Read your robots.txt and remove any disallow rules aimed at AI crawlers you actually want reading you.
2. Check Cloudflare's AI crawler controls under your domain's Security Settings. Search and Agent on Allow, Training on Allow or Disallow AI Training, and nothing on Block.
3. Audit plugins and hosting settings for bot-blocking features enabled without your knowledge.
4. Re-read your robots.txt after any Cloudflare change. Cloudflare's Bot Preference Sync can now write your dashboard choices into robots.txt, so check that the file says what you meant.
5. Publish an llms.txt at your domain root with a plain description of your business, services, locations and key URLs.
6. Test in the engines. Ask ChatGPT, Perplexity and Gemini about your business and write down what they say today, so you can tell whether your changes worked.
7. Re-test in 30 days. Crawler access changes don't show up instantly; give the engines a cycle to re-fetch.

The businesses that will win AI recommendations over the next year are not the ones with the cleverest content. They're the ones a machine can actually reach, read and verify. That starts with leaving the door open.

Frequently asked questions

Will Cloudflare's September 15 change break my website?

No. It changed which automated crawlers can reach your pages, not how human visitors experience your site, and existing settings carried over on their own. Nothing about your design, speed, forms or booking system changes. The one way to hurt yourself now is choosing Block: since September 15 it also stops Googlebot, Bingbot and Applebot, which takes you out of Google and Bing search as well as AI answers.

Do I need to do anything if I'm not on Cloudflare?

Yes, though not because of this policy. Cloudflare's change doesn't apply to you, but the underlying question does: your robots.txt, security plugins and host settings may still be blocking AI crawlers. Run the same four checks in this post. The most common cause of AI invisibility we find has nothing to do with Cloudflare.

Does blocking Google-Extended hurt my Google rankings?

No. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." It controls whether your content may be used to train Gemini models and to ground answers in the Gemini app and Vertex AI. AI Overviews and AI Mode are part of Search and are governed by Googlebot, so blocking Google-Extended does not remove you from them. Blocking Googlebot does.

Should I ever block AI crawlers?

Sometimes. If your business model is selling content (courses, research, subscription media), restricting training crawlers while allowing search and agent crawlers is defensible, and on Cloudflare the Disallow AI Training setting does exactly that. If you sell services and want to be recommended, blocking is self-defeating. A clinic that blocks GPTBot to protect its blog posts has protected content nobody was paying for and lost patients who were ready to book.

How long until AI engines notice I've unblocked them?

Expect 30 to 90 days for meaningful change. Crawlers re-fetch on their own schedules, and your content has to be read, associated with your business entity, and then judged worth citing. Real-time agent fetches improve immediately once access is open, but the deeper index and knowledge-graph effects build over one to two months.

Is llms.txt an actual standard that engines follow?

Not a ratified standard, and adoption is inconsistent — no engine guarantees it reads one. It's cheap insurance rather than a silver bullet: a single markdown file giving models a clean, first-party description of your business instead of one inferred from scraped fragments. We publish one and recommend clients do too, while being clear it supplements schema rather than replacing it.

About the author: Lorne Fade is the founder of Fade Digital, a Toronto Generative Engine Optimization agency. He has 20+ years in digital marketing and is also COO of VR Vision. He writes about how AI search actually decides which businesses get recommended.

Not sure whether the engines can see you? The AI Visibility Audit tests your business across ChatGPT, Claude, Perplexity, Gemini, Grok and Copilot, and hands you a prioritized fix list — $499, and it's yours to keep whether or not you continue.