Cloudflare Blocking AI Crawlers
AI Artificial Intelligence LLM, Featured Jul 03, 2025
Blocking AI Crawlers from Accessing Websites
Based on the most recent and credible sources, here is a detailed list of web hosting companies in the USA that are now blocking AI, along with granular details on the scope and nature of these restrictions:
Cloudflare
-
Default AI Bot Blocking: As of July 2025, Cloudflare now blocks AI bots (also known as AI crawlers) from crawling and scraping content on all client websites by default. This applies to all new domains that sign up with Cloudflare and is being rolled out to existing customers as well.
-
Granular Controls: Website owners can manually allow or block specific AI bots and opt into a “pay-per-crawl” system, enabling them to monetize AI bot access if desired.
-
Scope: The block targets web crawlers used by AI companies (such as those from OpenAI and Google) to harvest data for training large language models. This is not a blanket block on all bots, but specifically those identified as AI crawlers.
| Company | AI Blocking Status | Details |
|---|---|---|
| Cloudflare | Default block | All new domains blocked by default; granular opt-in/out; “pay-per-crawl” monetization |
GoDaddy (Managed WordPress Hosting)
-
No General AI Block: There is no evidence that GoDaddy is currently blocking AI bots or AI crawlers at the platform level for all hosting products.
-
Managed WordPress Hosting: GoDaddy’s Managed WordPress Hosting blocks specific plugins and restricts PHP execution in some directories for security and performance reasons. These restrictions are not explicitly aimed at AI or AI crawlers but rather at plugins and scripts that could pose security or performance risks.
-
Firewall Features: GoDaddy’s Website Security firewall enables customers to manually block or allow specific IP addresses, which can be used to block known AI bot IPs; however, this is not a default or automated AI-blocking feature.
-
Granularity: The only blocks in place are for specific plugins (see blocklisted plugins list) and general security measures, not for AI bots as a category.
| Company | AI Blocking Status | Details |
|---|---|---|
| GoDaddy | No default AI block | Managed WordPress restricts plugins and PHP execution for security, not specifically AI. |
Other Major US Web Hosts
-
No Public AI Blocking Policy: As of July 2025, there is no public evidence or reporting that other major US-based web hosting companies (such as Bluehost, HostGator, WP Engine, DreamHost, SiteGround, etc.) have implemented default blocks against AI bots or crawlers. Most continue to rely on robots.txt or manually set firewall rules by site owners.
Summary Table
| Web Host | AI Blocking Policy (as of July 2025) | Granular Details |
|---|---|---|
| Cloudflare | Default block for AI bots/crawlers | All new domains are blocked by default; opt-in/out and pay-per-crawl options are available. |
| GoDaddy | No default AI block | Managed WP blocks some plugins and PHP execution, not AI bots |
| Other Hosts | No known default AI block | No public AI-specific block; standard robots.txt/manual controls only |
Key Points:
-
Cloudflare is the only primary US-based provider currently enforcing a default block on AI bots and crawlers across its network, with granular controls for site owners.
-
GoDaddy restricts specific plugins and scripts on Managed WordPress plans for security, but does not block AI bots by default.
-
As of July 2025, other US web hosts have not announced or implemented default AI bot blocking.
Optimizing for AI and Allowing AI to Crawl Your Website.
Should you continue optimizing your website and content for AI? Or should you block AI bots from crawling your website?
Let me know what you think! Leave a comment below.