How to Block AI Crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider) on Your Server in 2026

You shouldn’t blindly block traffic without keeping a record of what your server is doing. You can instruct NGINX to log blocked bot attempts to a separate file so they don’t clutter your main website analytics.Suggested read: What is Fail2Ban with Setup & Configuration? (Detailed Guide)

• Training crawlers such as GPTBot, ClaudeBot, and Bytespider collect content for AI development.
• User-triggered agents, such as ChatGPT-User and Perplexity-User, may help your pages appear as cited sources in AI-generated answers.

Google uses the Googlebot user agents to crawl, index, and rank your pages in standard search engine results. For AI training and generative model grounding (such as feeding live search context to Gemini), Google uses a separate control token known as Google-Extended. Disallowing Google-Extended tells Google not to use your content for training and grounding in certain Gemini systems. Google states that Google-Extended doesn’t affect inclusion or ranking in Google Search.While adding a disallow directive to your robots.txt file is the traditional method for managing bots, it is ineffective against these bots because it runs on an honor system. Aggressive scrapers, poorly configured rogue bots, and many proprietary data brokers routinely ignore robots.txt entirely.

• Identify the AI crawlers you may want to block
• Set rules in robots.txt
• block requests at the NGINX server level
• Configure Cloudflare’s AI crawler controls
• Test your rules and confirm they work

If you are dealing with an incredibly aggressive bot that is hitting your server thousands of times a minute and draining your bandwidth, you should use NGINX’s special 444 response code. Unlike standard HTTP codes, 444 doesn’t send any headers or error pages to the bot; it simply closes the connection immediately. This can reduce the response work and outbound data associated with rejected requests, although NGINX must still accept and process the connection far enough to match the rule. 

Why robots.txt Alone Won’t Stop AI Crawlers

Google-Extended doesn’t control AI features that form part of Google Search itself. Access to those features is governed through Googlebot and standard Search controls such as nosnippet, data-nosnippet, max-snippet, and noindex.A crawler that doesn’t provide the required payment information receives an HTTP 402 Payment Required response. Availability and crawler participation remain limited, so this shouldn’t yet be treated as a general replacement for blocking.Reputable AI crawlers will respect robots.txt directives, but rogue scrapers or malicious bots can easily ignore them. NGINX blocks are significantly more powerful because they intercept and drop the connection at the server level based on the User-Agent or IP address before the site even loads. Custom NGINX rules provide stronger enforcement than robots.txt because they can reject matching requests before they reach the application. User-Agent matching is not foolproof, since another client can spoof a crawler’s name.
For stronger protection, combine NGINX rules with rate limiting, request logs, verified crawler IP ranges where available, and edge-level bot controls such as Cloudflare.To watch your server actively defend itself in real-time, you can filter your live server logs for specific bot names. Log in to your server via SSH and run the command:To edit your robots.txt, you can simply open your web host’s file manager, create this file, and copy-paste the code below to ask all major 2026 AI training bots to stay away from your entire site:

Which AI Bots Should You Block in 2026?

Claude-SearchBot should be considered separately, as Anthropic uses it to support search results rather than for general model training. 

Targeting High-Volume AI Training Crawlers

No, blocking AI crawlers like GPTBot or ClaudeBot won’t negatively impact your Google search rankings. These bots are completely separate from Googlebot, which is the scraper responsible for indexing your site for search engine results. Blocking AI scrapers only prevents your content from being scraped to train their large language models.

• GPTBot
• ClaudeBot
• Bytespider
• Amazonbot
• Applebot-Extended
• Google-Extended
• Meta-ExternalAgent
• CCBot

Before enforcing broad blocks across your entire domain, you can also establish governance rules using Cloudflare’s AI Audit tools. The AI Audit dashboard provides visibility into which AI services are scanning your website, allowing you to clearly see the volume of requests from specific bots and understand how they interact with your content.User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: Google-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

Why robots.txt alone is not enough (RFC 9309 is advisory)

Review Cloudflare’s displayed outcome before applying category-wide rules, particularly where a crawler also supports conventional search discovery.The easiest way for a novice to test if their server block is working is to pretend to be an AI bot using the command line. Open your computer’s terminal (or Command Prompt) and type the following command exactly:

• ChatGPT-User
• Perplexity-User
• OAI-SearchBot
• Claude-User

Another good option is to return a 410 Gone status code, which signals to the AI crawler that the resource it is looking for has been permanently deleted and won’t be returned. Use 410 Gone only when the requested resource has been permanently removed. Don’t return 410 solely as a bot-blocking technique for pages that remain available to other visitors. Use 403 or 444 when you are refusing access based on the requester rather than the state of the resource. Complex rewrite logic inside an NGINX if block can produce unexpected results, particularly when it is placed inside a location block. A simple condition that performs only a return is much less problematic.By filtering this list by “User Agent” or specifically looking at the “Block AI bots” rule metrics, you can visually confirm how many thousands of requests from scrapers like ClaudeBot and Bytespider were successfully dropped by Cloudflare’s network before they ever touched your origin server.

Will Blocking AI Crawlers Hurt Your Google Rankings?

However, there is a massive technical caveat for web admins: unlike most web scrapers, Google-Extended doesn’t have its own separate HTTP request user-agent string. The physical crawling is still executed using standard Google crawler user agents, meaning the Google-Extended token functions purely in a control capacity within your robots.txt file.By adding a simple directive like access_log /var/log/nginx/blocked_ai_bots.log; inside the block that returns your 403 or 444 code, you create an isolated, easily readable audit trail where you can safely monitor which AI companies are trying to harvest your data.After deploying your NGINX mapping rules, you shouldn’t simply assume they are functioning perfectly. Server-level configurations are powerful but unforgiving; a minor syntax error could potentially block legitimate traffic or fail to stop the unwanted bots you are targeting.You also need to distinguish between different types of AI bots.AI crawlers can consume bandwidth, increase server load, and collect your content without sending visitors back to your website. Blocking them isn’t as simple as adding a few lines to robots.txt, since not every crawler respects those instructions.Suggested read: How to Block IP Address Using WordPress .htaccess File to Stop Bad VisitorsAllowing these agents enables AI services to retrieve and cite your current content in response to user requests. It doesn’t guarantee that your page will be selected, cited, ranked prominently, or visited by the user. However, if your domain contains sensitive data or strictly paywalled content, you should weigh your goals carefully. If you prefer total privacy over AI search exposure, block these alongside the bulk training bots.

Implementing a Multi-Layered Strategy for AI Bot Management

Additionally, from September 15, 2026, Cloudflare will automatically block ‘Training’ and ‘Agent’ bots on pages with ads for all new domains, while keeping Search bots allowed. Cloudflare’s updated system will apply the most restrictive rule to multi-purpose crawlers, such as Googlebot or Applebot, that crawl for both search indexing and AI training.

Layer 1 – Block AI Bots with robots.txt

Suggested read: How To Use ModSecurity and OWASP CRS For Web App Firewall (WAF) To Secure Your WebsiteThe following control tokens and user agents are associated primarily with AI training, model development, or large-scale dataset collection:If you want to protect your infrastructure and intellectual property, you need to implement server-level blocking using NGINX or implement other firewall measures. This allows you to intercept these unauthorized requests at the server level, where you can drop the connection by returning a 403 Forbidden response before the request ever reaches your web application or consumes your server’s computing resources.tail -f /var/log/nginx/access.log | grep -i "gptbot"

Cloudflare provides two related ways to manage AI traffic:By analyzing this traffic, novice users can make data-driven security decisions, such as explicitly allowing search-focused tools that drive referral traffic, while firmly rejecting exploitative scrapers.map $http_user_agent $is_ai_bot {
default 0;
"~*GPTBot" 1;
"~*ClaudeBot" 1;
"~*Bytespider" 1;
"~*Amazonbot" 1;
"~*Meta-ExternalAgent" 1;
"~*CCBot" 1;
}

Blocking every AI-related user agent could reduce your visibility as well as your server traffic.From 15 September 2026, Cloudflare plans to evaluate multi-purpose crawlers against all their declared behaviors. A crawler that combines Search and Training may therefore be affected by your Training rule even when Search traffic is allowed.

Choosing the response code

Blocking AI crawlers won’t harm your traditional Google Search rankings if you implement your security blocks correctly and target the right bots. Google designed its ecosystem to allow web admins to opt out of generative AI training without sacrificing their organic SEO visibility.

403 Forbidden 

Cloudflare is also testing Pay Per Crawl, which is currently in closed beta. Participating site owners can set a price and choose whether to allow, block, or charge supported crawlers.

444 (Nginx-specific, drops the connection with no response)

Relying on a single method for blocking unwanted AI traffic can leave your infrastructure exposed. If you want to build a truly resilient defense, it is best to implement a layered strategy that covers public directives, server-level interception, and network-edge filtering. This ensures that if one mechanism fails or is ignored, your subsequent layers act as a fail-safe to protect your server resources.

410 Gone (signals permanent removal, deters re-crawls)

With RunCloud, you can easily manage and deploy sites on your own cloud infrastructure without needing to be a command-line expert. Its highly intuitive dashboard allows you to make quick NGINX updates and instantly reload your server to deploy those security rules across all your hosted applications in just a few clicks.

Logging Blocked Requests for Audit

Suggested read: How to Use Cloudflare Firewall Rules to Protect Your Web ApplicationIn this article, we have discussed how to manage custom NGINX configurations and protect your websites. It might sound overwhelming, but RunCloud makes the entire process effortless.Integrating with these tools can be a good marketing move, but it requires balancing visibility against your specific privacy concerns.

Layer 3 – Block AI Bots at the Edge with Cloudflare WAF

Start using RunCloud today.

• “Search” (crawlers indexing content to answer queries later and drive referrals)
• “Agent” (real-time automated bots acting on a human’s behalf, like chat fetch bots) 
• “Training” (scrapers permanently absorbing your data to train large language models).

Configure AI Bot Policies

Real-time fetchers operate under a different set of rules than bulk scrapers. Instead of indiscriminately harvesting data, these digital assistants visit your server only when a human query explicitly triggers them to fetch your content. This makes them an important component for maintaining visibility in modern, AI-driven search results.If you want to block traffic, you should use the map directive, which is a better and highly optimized alternative to if statements. When a visitor sends a request, NGINX checks their User-Agent exactly once against this map and assigns a variable (like a simple true/false flag). Because it is evaluated outside the complex location rules, it is incredibly fast and can automatically apply to every single website hosted on your entire server without repeating code.Reputable companies like Google and OpenAI currently program their bots to respect these rules, but aggressive scrapers, rogue data brokers, and malicious AI crawlers will completely ignore your robots.txt file and scrape your content anyway. Therefore, relying on this file alone leaves your server vulnerable to heavy automated traffic, so you must implement server-level blocking as well.After identifying the crawlers, you need to decide how your server should handle them when they arrive. Here is a list of appropriate HTTP status codes you can return when rejecting a web request.Anthropic uses separate user agents for different purposes. ClaudeBot crawls content for model development, Claude-SearchBot supports Claude’s search results, and Claude-User may retrieve a page in response to an individual user’s request. Because of this shared infrastructure, you must be extremely cautious when writing your NGINX rules or WAF policies. Using broad wildcards like *Google* or *bot* to stop scrapers will result in a catastrophic SEO failure, as you will accidentally block the legitimate Googlebot, Googlebot-Image, and Googlebot-Video indexers that your business relies on.You need to validate your setup to ensure your defenses are correct and not causing collateral damage to your site’s availability. Follow the steps below to verify that your block list is successfully intercepting requests.

If you search for tutorials on how to block bots using NGINX, almost every old guide will tell you to place an if statement directly inside your server block (for example, if ($http_user_agent ~* "GPTBot") { return 403; }).PerplexityBot is also separate from bulk training crawlers. Perplexity states that it uses PerplexityBot to build its search index and surface links in Perplexity results, not to train foundation models. Blocking it may prevent your pages from appearing in Perplexity search results. 

Future of AI Content Monetization

curl -I -A "GPTBot/1.0" https://example.com

In AI Crawl Control, open the Crawlers tab to review individual crawlers and set each one to Allow or Block.To set up your first layer of defense, create or edit a simple text file named robots.txt and place it in the root directory of your website (usually the public_html or htdocs folder). 

Verify that NGINX Blocking is Working

Finally, you can verify your edge-level blocks by reviewing your visual data within the Cloudflare dashboard. Navigate to Security and then click on Events to see a complete log of all web traffic that triggered your WAF rules.ClaudeBot crawls web content for Anthropic’s model development. Claude-SearchBot supports Claude’s search results, while Claude-User may retrieve a page in response to an individual user’s request.
You may choose to block ClaudeBot while allowing Claude-SearchBot and Claude-User if you want to limit training access without removing your content from Claude’s search and user-directed retrieval features.

Curl with a faked GPTBot User-Agent and expect 403

Suggested read: Linux Server Hardening: 11 Steps to Secure a Production VPS Because the phrase “Google-Extended” will never appear in your server’s incoming HTTP requests, you can’t block it at the NGINX or Cloudflare WAF level using a simple User-Agent string match. You must use the traditional robots.txt opt-out method for Google’s AI training and rely on server-level blocks for other companies’ bots.

Tail NGINX access logs and grep for the bot user-agents

It depends on your goals, as GPTBot crawls your site globally for AI training, whereas ChatGPT-User and Perplexity-User act as real-time search agents triggered by active user prompts. Blocking these “User” bots will prevent those AI tools from summarizing or linking to your live pages in their chat interfaces. If you want your site to be cited as a live source in AI-generated answers, you should leave user bots alone and block only training bots.This is exactly why you should identify the crawler and its purpose before deciding whether to block it.

Cross-check with Cloudflare bot analytics

For each type of bot, you can choose to “Block (on all pages),” “Allow (do not block),” or select “Block on pages with ads,” which uses Cloudflare’s automated detection to block bots strictly on monetized pages while leaving the rest of your site accessible.Whether you’re hosting a single high-traffic web application or managing dozens of client sites, having complete, frictionless control over your hosting environment is important. 

Final Thoughts: Take Control of Your Server Traffic

Returning a 403 Forbidden status is the most common and standard choice because it clearly communicates to the bot that the server understood the request but is actively refusing to fulfill it. If you manage your DNS through Cloudflare, you can easily intercept AI bots at the network edge before they ever reach your web server by configuring their built-in bot policies. Cloudflare has updated its approach to AI traffic by introducing new features that categorize bots into three distinct behaviors:No, blocking Google-Extended only prevents your site’s content from being used to train Google’s generative AI models, such as Gemini. It operates entirely independently from traditional search indexing and advertising systems. Disallowing Google-Extended doesn’t affect your inclusion or ranking in Google Search. It is separate from the Googlebot controls used for conventional Search crawling.
Avoid blocking broader Google user agents or IP ranges at the server or firewall level, since an overly broad rule could interfere with Google services that rely on those crawlers.AI-related crawlers serve several different purposes. Some collect content for model training, while others build search indexes or retrieve pages in response to a user request. High-volume crawling can consume bandwidth and server resources, but the impact depends on the crawler, its request rate, your caching configuration, and your application stack.

Should You Block AI Crawlers? Common Questions Answered