How to Edit robots.txt in WordPress (With & Without Plugin)

While editing the robots.txt file directly without a plugin can be more technically involved, it offers some distinct advantages for advanced users and those who prefer greater control over their website’s settings. Not having a strong web presence is like not existing at all. Your website is your storefront, your portfolio, and your voice.📖 Suggested read: 10 Best WordPress Management Tools To Easily Manage Multiple WebsitesBut that’s not all! We’re going beyond traditional search engines.If you aren’t comfortable with command line tools but still want to edit your configuration files manually, then you should note that RunCloud users can manage and edit their robots.txt through a clean, intuitive interface, offering a safer alternative to direct FTP or SSH access while maintaining control.OpenAI provides clear documentation on its crawlers, allowing you to make informed decisions about how you want them to interact with your site.Controlling your site’s interaction with crawlers is essential for a healthy web presence. You want search engines to index your valuable content efficiently, avoid duplicate content issues, and protect sensitive areas. With the advent of AI crawlers, you also need to consider how your content might be used for training AI models, and robots.txt gives you a voice in that process.

What is robots.txt?

Controlling how search engine crawlers discover and interact with it is crucial for its success.Start your free trial today and see how easy WordPress hosting can be.

Purpose of robots.txt

📖 Suggested read: How Google Interprets the robots.txt Specification

• Preventing Crawling of Duplicate Content: Websites often have multiple URLs that lead to the same content (e.g., with and without “www” or with different URL parameters). robots.txt can help you indicate which version you prefer search engines to crawl and avoid duplicate content penalties.
• Blocking Access to Sensitive Directories: You might have areas of your website that are not intended for public consumption, such as your admin panel (/wp-admin/ on WordPress), staging environments, or directories containing internal files. robots.txt can tell crawlers to steer clear of these. However, remember that this is a suggestion, not an enforced restriction; for true security, use proper authentication and authorization methods.
• Conserving Crawl Budget: Large websites, especially e-commerce sites, can have thousands or even millions of pages. Search engines have a limited “crawl budget” – the amount of time and resources they’ll dedicate to crawling a particular site. You can ensure that crawlers prioritize indexing your most valuable content by strategically disallowing less important or dynamically generated pages (like certain search result filters).
• Specifying the Location of Sitemaps: robots.txt can include a directive pointing search engines to your XML sitemap(s). Sitemaps are lists of all the important URLs on your site, helping search engines discover and index your content more efficiently.

The primary purpose of robots.txt is to manage crawler traffic, not to hide pages from search results entirely. While it strongly suggests which areas to avoid, it doesn’t guarantee exclusion from search indexes. Here’s a breakdown of its key functions:

How Search Engines Use robots.txt

In your robots.txt file, you can use the Disallow: directive followed by the specific page’s URL path. For example, Disallow: /private-page/ will block that URL.

1. User-agent: This line specifies which crawler the following rules apply to. User-agent: * means the rules apply to all crawlers. You can also target specific crawlers, like User-agent: Googlebot.
2. Comments: Lines starting with # are considered comments and will be ignored by web crawlers.
3. Disallow: This is the core directive. It tells the crawler not to access the specified URL path. For example, Disallow: /private/ would instruct crawlers to avoid the /private/ directory and everything within it.
4. Allow: This directive is used to override a Disallow rule. It’s helpful if you want to block an entire directory but allow access to a specific file or subfolder within it. For instance:

📖 Suggested read: 15 Best Performance Testing Tools to Improve Your Site Editing robots.txt can impact your site’s search visibility, so proceed cautiously; mistakes can block important content. Platforms such as RunCloud make managing and editing your robots.txt easier and safer through a user-friendly interface.User-agent: *
Disallow: /wp-admin/
Disallow: /private/
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /

By the end of this guide, you’ll be a robots.txt expert! 

Why is robots.txt More Important Now?

• Data Usage Control: AI models require massive datasets for training. Your website’s content, if crawled, could potentially be used to train these models. robots.txt allows you to express your preference regarding this usage. You can choose to allow AI crawlers to access your content, disallow them entirely, or selectively control access to specific parts of your site.
• Performance and Bandwidth: AI crawlers can be very aggressive and potentially overload your server. robots.txt can help you manage this by limiting their access or specifying crawl delays (though not all crawlers respect crawl-delay directives).
• Ethical Considerations: There are ongoing discussions and debates about the ethics of using publicly available data to train commercial AI models. robots.txt allows website owners to participate in this discussion by making their preferences known.
• Search Indexing: Some AI crawlers enhance search experiences within specific AI products. Controlling access to these can directly impact whether your content is discoverable through these AI-powered search interfaces.

How to Edit the robots.txt in WordPress

Therefore, robots.txt should not be relied upon for security; use proper authentication and .htaccess rules for sensitive areas. Also, a disallowed page may still appear in search results if it is linked from other pages.

Method 1: Editing robots.txt with a Plugin

But managing files directly on your server, especially for tasks like editing robots.txt, can be intimidating. Traditional methods such as FTP clients and cPanel’s File Manager can feel clunky and risk errors if you’re not experienced.

1. Install and Activate an SEO Plugin: You can use popular plugins such as Yoast SEO, Rank Math, All in One SEO Pack, SEOPress, etc. These plugins offer a wide range of SEO features, including robots.txt management.
2. Locate the robots.txt Editor: The exact location of the editor varies depending on the plugin. Refer to the official documentation of your WordPress plugin to follow along:
3. Edit the File: The plugin will typically provide a text area where you can view and edit the contents of your robots.txt file. Make your desired changes and exclude any unnecessary paths from crawling.
4. Save Changes: Click the “Save Changes” or similar button to apply your modifications.
5. Test your Changes: Use the Google search console’s robots.txt tester to ensure your edits work correctly.

📖 Suggested read: How to Fix WordPress Revisions Not Showing [SOLVED]

• User-Friendly Interface: Simplifies and streamlines the editing process.
• Syntax Validation: Many plugins will check for basic syntax errors, reducing the risk of accidentally blocking important parts of your site.
• Integrated with SEO Tools: Often provides seamless integration with other SEO features within the plugin.
• Reversion Options: Some plugins offer version history or backup options, allowing you to revert to previous versions if needed.

Method 2: Editing robots.txt without a Plugin

In the age of AI, we’ll show you how to manage your website’s presence in tools like ChatGPT and Claude using specific robots.txt user-agent configurations.

1. Access Your Server: You’ll need to connect to your server using one of the following methods:
   • RunCloud File Manager: RunCloud provides a built-in file manager for creating and modifying text files on your server.
   • FTP (File Transfer Protocol): Connect to your server using an FTP client such as FileZilla. Your web hosting provider usually provides your FTP credentials (hostname, username, and password).
   • cPanel File Manager: Most web hosting providers offer cPanel, which includes a web-based File Manager. Log in to your cPanel account and navigate to the File Manager.
   • SSH (Secure Shell): Advanced users can use SSH to obtain command-line access to the server. This requires SSH credentials and familiarity with command-line tools.

2. Locate the robots.txt File: Navigate to the root directory of your WordPress installation. This is typically the directory where your wp-config.php file is located. If it doesn’t exist, you can create a new plain text file named robots.txt.
3. Edit the File:
   • RunCloud File Manager: Locate the file and click on it; this will open a text editor in a new browser window.
   • FTP: Download the robots.txt file to your computer and edit it using a plain text editor such as Notepad++, Sublime Text, or VS Code. You can’t use a word processor like Microsoft Word. After editing the file, save it and upload the modified file back to the server, overwriting the existing file.
   • cPanel File Manager: Right-click on the robots.txt file and select “Edit” or “Code Edit” (depending on your cPanel version). Make your changes directly in the web-based editor.
   • SSH: Use the nano or vim command-line text editors to edit your robot.txt file.

4. Save Changes: Ensure you save the changes you’ve made to the file. If using FTP, ensure the updated file is uploaded and overwrites the old one.
5. Test Your Changes: After editing your file directly, it is essential to test it. Use Google Search Console’s robots.txt Tester tool to verify that your changes are working as intended and that you haven’t accidentally blocked important content.

While robots.txt has always been important for managing traditional search engine crawlers, its importance has grown significantly with the rise of AI-powered crawlers. These AI crawlers, used by companies like OpenAI, are designed to gather vast amounts of data from the web, often to train large language models (LLMs) and other AI systems. This introduces new considerations for website owners regarding data usage, privacy, and control.📖 Suggested read: How To Install WordPress With RunCloud | Step-By-Step GuideUser-agent: ChatGPT-User
Allow: /

Update your robots.txt file whenever you make significant changes to your website’s structure or want to adjust which sections are accessible to search engines. Regular reviews are recommended, but not frequent updates.

• Higher Risk of Errors: Requires careful attention to syntax; mistakes can have significant negative consequences.
• Less User-Friendly: Requires technical knowledge of FTP, cPanel, or SSH.
• No versioning: If you make a mistake, it will be difficult to revert.

Use Google Search Console’s robots.txt Tester tool. This tool validates your syntax and shows if any URLs you intend to block are accessible.This is where RunCloud shines.Using User-agent: * and Disallow: / in your robots.txt file will instruct all search engines not to crawl any part of your website, effectively removing it from search results.User-agent: GPTBot
Disallow: /

Yes, some WordPress SEO plugins can create or modify your robots.txt file. Review plugin settings carefully to avoid conflicts or unintended disallow rules.Robots.txt is a site-wide file instructing search engine crawlers which directories to avoid. At the same time, meta tags are placed within individual page HTML and control how that specific page is indexed and displayed. Meta tags offer finer-grained control.

• GPTBot: This is OpenAI’s primary crawler for training its generative AI foundation models (like those powering GPT-3, GPT-4, etc.). Disallowing GPTBot signals that you do not want your website’s content used for training these models.

📖 Suggested read: How to Easily Create a WordPress Staging Site in RunCloud

• OAI-SearchBot: This crawler specifically displays websites in search results within ChatGPT’s search features. Allowing OAI-SearchBot can help your site appear in these results.

Here’s why RunCloud is the best choice for your WordPress site:

• ChatGPT-User: It is not used to crawl the web automatically. It is a proxy through which a user’s request is sent. It is not used for general training purposes.

You will also learn to control whether your content is used to train these powerful AI models, giving you unprecedented control over your digital footprint. This feature has become increasingly important with the advancement of AI, so understanding your robots.txt file and web robots in general is more powerful than ever.Throughout this guide, we’ve shown you why robots.txt is a small but mighty file that manages how search engines and, increasingly, AI crawlers interact with your website. We discussed its core purposes: managing crawl traffic, conserving crawl budget, and specifying sitemap locations. We also explained that robots.txt is a suggestion, not a security mechanism.

1. Blocks all crawlers (including traditional search engines) from /wp-admin/ and /private/.
2. Blocks GPTBot from the entire site (preventing training data collection).
3. Allows OAI-SearchBot to crawl the entire site (enabling appearance in ChatGPT search).

The basic syntax uses User-agent to specify the crawler (e.g., User-agent: * for all) and Disallow to specify paths to block (e.g., Disallow: /wp-admin/). Allow can override Disallow for specific files or folders within a disallowed directory.RunCloud’s intuitive dashboard provides a built-in File Manager that simplifies server-side file management. No more juggling FTP clients or navigating complex cPanel interfaces!

After Action Report: Mastering Your robots.txt File

However, it does have a few disadvantages:This approach ensures that all the necessary features and tools are built directly into the core system. You have complete autonomy over the file’s content and structure without depending on external plugins or add-ons. This eliminates compatibility issues, potential security risks, and the need for additional software installations. The robots.txt file is a plain text file in your website’s root directory (e.g., yourwebsite.com/robots.txt). It is a set of instructions for search engine crawlers (also known as bots or spiders) that tells them which parts of your site they shouldn’t access. It’s a tool for controlling how search engines interact with your website, but it’s not a security mechanism. It is a public file that any human or bot can see.If you don’t want to use an additional plugin, you can directly access and modify the robots.txt file on your server. However, this requires more technical expertise and, if not done carefully, carries a higher risk of errors.In this guide, we will discuss the robots.txt file, a critical tool for managing your WordPress robots.txt file and optimizing your site for SEO. We’ll cover everything from the basics of robots.txt rules and directives (like Disallow and Allow) to editing your file using popular SEO plugins or even directly via FTP or cPanel.

• Direct Access: Access your website’s root directory and locate your robots.txt file.
• Built-in Text Editor: RunCloud’s File Manager includes a powerful yet easy-to-use text editor directly within the dashboard. You can make changes to your robots.txt file, save them, and then be done – all without leaving your browser.
• Secure and Streamlined: RunCloud’s interface provides a secure and streamlined way to manage server files, minimizing the risk of accidental errors.
• One-Click Staging: Create a staging environment to experiment with before pushing changes to the live server.

Ready for Effortless WordPress Management? Choose RunCloud.

User-agent: OAI-SearchBot
Allow: /

This section will focus on managing AI crawlers using robots.txt, specifically for OpenAI crawlers. By understanding these crawlers and their user agents, you can control their access to your website’s content.

• Blazing Fast Performance: Optimized server configurations and caching mechanisms ensure your site loads incredibly quickly, improving user experience and SEO.
• Robust Security: Built-in security features, including firewalls and regular security updates, protect your site from threats.
• Effortless Management: An intuitive dashboard makes server management a breeze, even for non-technical users.
• Scalability: Easily scale your server resources as your website grows.
• Expert Support: RunCloud’s support team can assist you with any questions or issues.

While we provided an overview of OpenAI’s crawlers and user agents, we recommend reading AI / LLM User Agents: Blocking Guide to learn more.

FAQs on robots.txt in WordPress