Cloudflare Starts Blocking AI Crawlers on September 15. Is Your Site Ready?

On September 15, 2026, Cloudflare changes what happens by default when an AI crawler knocks on your website's door.

If you run a clinic, a firm, or a contracting business, your first instinct is probably that this is a publisher problem — something for news sites and media companies to worry about. Mostly, you're right. The headline change targets pages that carry advertising, and your services page doesn't run banner ads.

Here's why it still matters to you: the deadline is a forcing function, not a threat. In auditing local service businesses across the GTA, the single most common technical failure we find isn't bad schema or thin content. It's a site that is actively blocking the AI crawlers it desperately wants to be read by — usually because somebody clicked a one-button "block AI scrapers" toggle in Cloudflare, or installed a security plugin that did it for them.

Those businesses aren't losing AI visibility on September 15. They already lost it, months ago, and nobody told them.

This post covers what Cloudflare is actually changing, whether it hits your site, how to check in about four minutes whether you're already invisible to ChatGPT, and exactly which crawlers a service business should allow.

What is Cloudflare actually changing on September 15, 2026?

Cloudflare is splitting AI crawler traffic into three categories and changing the default rules for pages that host advertising. From September 15, 2026, ad-carrying pages will permit search crawlers but automatically block training and agent crawlers — and any "mixed-use" crawler that won't declare which job it's doing on a given request gets blocked outright on those pages.

The three categories Cloudflare now distinguishes are worth learning, because this vocabulary is going to show up everywhere over the next year:

• Search — a crawler collecting and indexing content to answer questions later. This is the classic Googlebot job.
• Agent — automated behaviour acting in real time on a specific person's behalf. This is what fires when someone asks ChatGPT about your business right now and it goes and fetches your page.
• Training — a crawler taking your content to train or fine-tune a model.

A mixed-use crawler is one bot doing several of those jobs under a single user agent, so a site owner can't allow one and refuse another. As TechCrunch and others reported when the policy landed on July 1, the change points squarely at Google, Microsoft and Apple, whose crawlers index for search and feed AI products from the same fetch. Cloudflare's stated goal is to pressure those companies into separating their crawlers so publishers can make a real choice.

The deadline is the lever. Separate your crawlers by September 15, or get blocked by default on ad-supported pages.

Does this affect my business website?

Probably not directly — and you should still check. The new defaults apply automatically to any new domain added to Cloudflare, any new site added to an existing account, and every site on the free tier. But the aggressive blocking behaviour is scoped to pages that host advertising, which most clinic, firm and contractor websites don't.

So there are two groups reading this:

If you're on a paid Cloudflare plan with no ads on your site, the September 15 defaults are close to a non-event. You keep your existing configuration, and you can override defaults from the dashboard whenever you want.

If you're on Cloudflare's free tier — which is a very large share of small business sites — your zone gets switched to the new rule set on September 15. If you don't run ads, the practical impact is still small. But your crawler configuration is about to be touched by someone other than you, and that is a good enough reason to go look at it.

Either way, you can opt out through your Cloudflare Security settings any time before September 15, and existing paying customers keep the ability to re-admit specific crawlers from the dashboard.

The honest summary: this policy was not written with your dental practice in mind. It was written to make Google and Apple stop hiding AI training inside a search crawl. Your dental practice just happens to sit on the same infrastructure.

Why are so many small business sites already invisible to ChatGPT?

Because blocking AI crawlers takes one click, and the button doesn't explain what it costs you. Cloudflare's "AI Scrapers and Crawlers" toggle sits under Security → Bots, works on every plan including free, and switches on in under a minute. It was designed for publishers protecting content from model training. Small business owners flip it on because blocking scrapers sounds unambiguously good.

It isn't. That switch doesn't only stop training crawlers — it stops the fetches that put you in AI answers.

Think about what actually happens when a prospective patient asks ChatGPT "who's the best optometrist in Liberty Village for dry eye?" The model doesn't only rely on what it memorised during training. It sends an agent request to fetch current pages. If your site refuses that request, you don't appear as an outdated entry. You don't appear at all, and a competitor who left their door open gets described in detail instead.

We see the same self-inflicted wound in three other places:

• WordPress security plugins with an "AI bot protection" feature enabled by default after an update.
• Managed hosting platforms that added blanket AI-bot blocking as a customer-friendly feature.
• Copy-pasted robots.txt files from a template or a forum post that disallow GPTBot and friends without comment.

None of these announce themselves. Your rankings don't drop. Your analytics look normal, because AI referral traffic is small and mostly misattributed anyway — ChatGPT accounts for roughly 87% of all AI referral traffic, and total LLM traffic still sits under 1% of organic sessions for most sites. You simply stop being recommended, quietly, and there's no notification.

How do I check if my site is blocking AI crawlers?

Run these four checks in order. The whole thing takes about four minutes and needs no tools beyond a browser.

1. Read your robots.txt. Go to yourdomain.com/robots.txt. Look for any Disallow: / sitting under a User-agent: line that names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, or Google-Extended. If you find one, that's your answer.

2. Check the Cloudflare toggle. In the Cloudflare dashboard, go to Security → Bots and find AI Scrapers and Crawlers. If it's on, Cloudflare is blocking AI crawlers at the network level regardless of what your robots.txt says — and this override is the one people forget exists.

3. Check your CMS and host. In WordPress, review any security or SEO plugin with bot-blocking options. On managed hosting, look for an "AI bot" or "scraper protection" setting in your control panel.

4. Ask the engines directly. Open ChatGPT, Perplexity and Gemini and ask each one the question a real customer would ask — "best [your service] in [your neighbourhood]" — then ask "what can you tell me about [your business name]?" If the engines can describe your competitors in specifics and go vague or wrong on you, you have a visibility problem worth diagnosing properly.

That fourth check is the one that tells you what the first three cost you. It's also the manual version of what our AI Visibility Audit does across six engines with a scored report — but you can absolutely do a rough version yourself in ten minutes, and you should.

Which AI crawlers should a service business allow?

Allow every crawler that can put you in an answer, and make a deliberate decision about the ones that only take. For a local service business trying to win AI recommendations, the calculus is simpler than it is for a publisher: you are not selling content, you're selling appointments, so being read is almost always worth more than being protected.

The one that trips people up is Google-Extended. It doesn't affect your traditional Google rankings at all, which makes it feel safe to block. What it does affect is whether Google may use your content in Gemini and AI Overviews — the surface reaching more than 1.5 billion people a month, per Alphabet's Q1 2025 earnings. Blocking it is a decision to sit out the largest AI answer surface on the internet.

If you want the working version, here's a robots.txt block that explicitly welcomes the engines that matter while keeping bots off your admin and API paths:

# AI answer engines — explicitly allowed
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Googlebot
User-agent: Bingbot
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /login/

Sitemap: https://yourdomain.com/sitemap.xml

That's close to what runs on this site. Pair it with an llms.txt file at your root — a short markdown summary of who you are, what you do, and your key URLs — which gives a model a clean, authoritative description to work from instead of one assembled from whatever it scraped. Our schema markup templates for AI cover the structured-data half of the same job.

What is pay per crawl, and will I ever get paid for it?

Pay per crawl is Cloudflare's mechanism for charging AI companies to access your content, and for a local service business the realistic answer is that you will never meaningfully earn from it. It works over standard HTTP: a crawler either presents payment intent in its request headers, or receives a 402 Payment Required response with the price attached. Publishers set one of three states per crawler — Allow (free access), Charge (pay the configured price), or Block (no access at any price).

Cloudflare has signalled that this is evolving from pay-per-crawl into pay-per-use, where publishers get paid when their content actually creates value — when it appears in an AI answer, or when an agent buys premium information to complete a task — rather than merely when a bot fetches a page.

For The New York Times, that's a business model. For a physiotherapy clinic in North York, it's the wrong lever entirely. Your content isn't the product; it's the marketing for the product. Charging ChatGPT a fraction of a cent to read your services page, and thereby not appearing when a patient asks who treats vertigo, is the worst trade in the building.

Set your AI crawlers to Allow and move on. The money is in the appointment, not the crawl.

What should I do before September 15?

Work through this list once. For most businesses it's a half-hour job, and it's the difference between being readable and being absent.

1. Read your robots.txt and remove any disallow rules aimed at AI crawlers you actually want reading you.
2. Check Security → Bots in Cloudflare. If "AI Scrapers and Crawlers" is on and you're a service business, turn it off.
3. Audit plugins and hosting settings for bot-blocking features enabled without your knowledge.
4. Decide deliberately about free tier. If your zone is on the free plan, you're getting the new defaults on September 15 — go set your preferences before the change is made for you.
5. Publish an llms.txt at your domain root with a plain description of your business, services, locations and key URLs.
6. Test in the engines. Ask ChatGPT, Perplexity and Gemini about your business and write down what they say today, so you can tell whether your changes worked.
7. Re-test in 30 days. Crawler access changes don't show up instantly; give the engines a cycle to re-fetch.

The businesses that will win AI recommendations over the next year are not the ones with the cleverest content. They're the ones a machine can actually reach, read and verify. That starts with leaving the door open.

Frequently asked questions

Will Cloudflare's September 15 change break my website?

No. The change alters which automated crawlers can access certain pages — it has no effect on how human visitors experience your site. Nothing about your design, speed, forms or booking system changes. The risk is invisibility to AI assistants, not downtime, and only ad-carrying pages hit the aggressive default blocking behaviour.

Do I need to do anything if I'm not on Cloudflare?

Yes, though not because of this policy. Cloudflare's deadline doesn't apply to you, but the underlying question does: your robots.txt, security plugins and host settings may still be blocking AI crawlers. Run the same four checks in this post. The most common cause of AI invisibility we find has nothing to do with Cloudflare.

Does blocking Google-Extended hurt my Google rankings?

No. Google-Extended controls only whether Google may use your content in Gemini and AI Overviews. Your traditional search rankings are unaffected either way. That's precisely why it gets blocked carelessly — it looks consequence-free, but blocking it removes you from AI Overviews, which Alphabet reported reaching more than 1.5 billion people monthly.

Should I ever block AI crawlers?

Sometimes. If your business model is selling content — courses, research, subscription media — restricting training crawlers while allowing search and agent crawlers is defensible. If you sell services and want to be recommended, blocking is self-defeating. A clinic that blocks GPTBot to protect its blog posts has protected content nobody was paying for and lost patients who were ready to book.

How long until AI engines notice I've unblocked them?

Expect 30 to 90 days for meaningful change. Crawlers re-fetch on their own schedules, and your content has to be read, associated with your business entity, and then judged worth citing. Real-time agent fetches improve immediately once access is open, but the deeper index and knowledge-graph effects build over one to two months.

Is llms.txt an actual standard that engines follow?

Not a ratified standard, and adoption is inconsistent — no engine guarantees it reads one. It's cheap insurance rather than a silver bullet: a single markdown file giving models a clean, first-party description of your business instead of one inferred from scraped fragments. We publish one and recommend clients do too, while being clear it supplements schema rather than replacing it.

About the author: Lorne Fade is the founder of Fade Digital, a Toronto Generative Engine Optimization agency. He has 22+ years in digital marketing and is also COO of VR Vision. He writes about how AI search actually decides which businesses get recommended.

Not sure whether the engines can see you? The AI Visibility Audit tests your business across ChatGPT, Claude, Perplexity, Gemini, Grok and Copilot, and hands you a prioritized fix list — $499, and it's yours to keep whether or not you continue.