How to Edit robots.txt in WordPress (and Check It Works)

You change one line in robots.txt, hit publish, and everything looks fine—until you realise Googlebot (or an AI crawler) can no longer fetch the pages that matter. Robots.txt mistakes are quiet: your site still works for humans, but crawlers follow the rules you shipped.

Before you edit anything, you need to know what you’re actually editing on WordPress. Many sites don’t have a real file on the server at all—WordPress can generate a virtual robots.txt response, and a physical robots.txt in the site root overrides it. Both Yoast and SiteGround describe this virtual-vs-physical behaviour, and why creating a real file takes priority (Yoast robots.txt help, SiteGround WordPress robots.txt guide).

This guide shows you how to check what your site is serving at /robots.txt, edit it safely via an SEO plugin or FTP, and verify the live result with a robots.txt checker. You’ll also see a safe baseline example (including the common Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php rules), plus simple rules for allowing or blocking specific AI crawlers. One last reminder before you start: robots.txt controls crawling, not indexing—use meta robots tags or HTTP headers for noindex; the robots meta tag generator writes both.

How to Find Your Current Robots.txt (And Spot Virtual vs Physical)

The fastest way to see your WordPress robots.txt is to fetch what search bots fetch. Open a browser and visit https://yourdomain.com/robots.txt (replace with your domain). SiteGround documents this exact check for viewing the file WordPress outputs at https://www.siteground.com/kb/wordpress-robots-txt.

If the page loads, you have a robots.txt response, even when no file exists on disk. If you see a 404 or a redirect loop, fix that first because crawlers will hit the same URL.

How to Tell Whether Robots.txt Is Virtual or Physical

WordPress can generate a virtual robots.txt when your site root does not contain a physical file. Yoast spells this out: WordPress generates a virtual robots.txt by default, and you override it by creating a physical robots.txt in the site root (Yoast).

To confirm “virtual vs physical,” check the server, not just the browser output.

If you can locate a real file named robots.txt in your site’s root directory (same level as wp-config.php), you have a physical robots.txt.
If no file exists in the root, WordPress is generating the output dynamically (a virtual robots.txt), as SiteGround notes in its WordPress robots.txt guide.

This distinction decides what “edit robots.txt WordPress” means on your site. A physical file uploaded by FTP or your host’s file manager overrides the virtual output, even if an SEO plugin shows an editor. SiteGround states that a physical robots.txt in the root folder overrides the virtual one WordPress generates by default (SiteGround).

How to Edit Robots.txt in WordPress: Plugin Editor vs FTP Upload

When you edit robots.txt WordPress, you have two safe options: change the virtual output through an SEO plugin, or upload a physical robots.txt file to your site root by FTP (or your host’s file manager). The choice matters because a physical file in the root overrides WordPress’s virtual response and anything a plugin shows in an editor.

Edit WordPress Robots.txt With an SEO Plugin (Virtual Output)

If your site currently serves a virtual WordPress robots.txt, the simplest workflow is an SEO plugin editor. Yoast SEO documents that WordPress generates a virtual file by default, and that the easiest way to create or edit it is through Yoast SEO in the WordPress Dashboard (Yoast robots.txt help).

In practice, you open your plugin’s settings (Yoast SEO, Rank Math, or All in One SEO), find the robots.txt editor, paste your rules, and save. Then reload /robots.txt in a browser to confirm the output changed.

If your plugin does not show a file editor, your WordPress install may block file editing. Yoast notes its “File Editor” menu will not appear if file editing is disabled (Yoast).

Upload a Physical robots.txt by FTP (Physical File Wins)

If you need full control, upload a real robots.txt file to the WordPress root directory (the same level as wp-admin and wp-content). Use an FTP client like FileZilla, or your host’s file manager. Name the file exactly robots.txt.

Download your current rules by copying what you see at /robots.txt.
Edit the text file locally.
Upload it to the site root, replacing any existing robots.txt.
Re-open /robots.txt to confirm the server is serving the physical file.

Yoast is explicit here: to override the virtual file, create a physical robots.txt in the root folder (Yoast). SiteGround says the same: a physical robots.txt in the root overrides the dynamic WordPress output (SiteGround).

A Safe Robots.txt Example for Most WordPress Sites

A physical file in your site root overrides WordPress’s virtual output, so treat your WordPress robots.txt as production configuration. Start with a baseline that blocks low-value admin crawling while keeping the one endpoint many sites need for front-end features.

Copy-paste this as a safe default for most WordPress sites (then replace the sitemap URL with yours):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap_index.xml

Yoast documents that WordPress generates the first three lines by default (Yoast), and that adding a sitemap line is part of what many SEO plugins output when you create robots.txt (Yoast sitemap line example).

What Each Line Changes in Crawling Behavior

User-agent: * applies the rules to all crawlers that follow the robots exclusion protocol, including Googlebot and Bingbot.

Disallow: /wp-admin/ asks crawlers to stay out of the WordPress admin area. This usually reduces wasted crawl activity on login, settings, and admin screens that should never rank.

Allow: /wp-admin/admin-ajax.php creates a specific exception. Many themes and plugins use admin-ajax.php for front-end requests, so allowing it avoids accidental breakage for bots that fetch resources while rendering pages.

Sitemap: points crawlers to your XML sitemap. Use the exact sitemap URL your SEO plugin outputs (often /sitemap_index.xml for Yoast SEO). This line helps discovery, but remember it still does not force indexing. Robots.txt controls crawling. Indexing decisions happen elsewhere (for example, via meta robots tags).

How Do You Allow or Block AI Crawlers in Robots.txt?

Robots.txt already guides search bots, and the same file can guide AI crawlers. If you want your WordPress robots.txt to allow or block specific AI user agents, add a dedicated group for each bot and keep the rules simple. These directives affect crawling access; they do not force indexing or removal.

Use one-line rules like these (copy-paste, then adjust paths if needed):

GPTBot (OpenAI training crawler):
User-agent: GPTBotDisallow: /
OAI-SearchBot (OpenAI search crawler):
User-agent: OAI-SearchBotAllow: /
ClaudeBot (Anthropic crawler):
User-agent: ClaudeBotDisallow: /
PerplexityBot (Perplexity crawler):
User-agent: PerplexityBotDisallow: /

If you prefer the opposite behavior, flip the rule. For example, to allow GPTBot sitewide, use Allow: /. To block only a section (say, /members/), use Disallow: /members/ instead of Disallow: /.

Choosing What to Block: Training vs Search Discovery

Blocking training bots is a business choice, not an SEO requirement. Some sites block GPTBot or ClaudeBot to reduce content reuse in model training. Other sites allow them because they want their content to appear in AI answers and citations. Some teams split the difference: they block training bots and allow search-focused crawlers (for example, OAI-SearchBot) so pages can still be discovered for AI search experiences.

After you edit robots.txt WordPress, re-open /robots.txt and confirm your AI bot groups appear exactly as written. A typo in a user agent name means the rule never matches.

How to Verify Your Robots.txt Works (Robots.txt Checker + AI Crawler Checker)

A robots.txt change is only real when bots can fetch it and parse it. After you edit robots.txt WordPress, verify the live response at /robots.txt, then run a robots.txt checker to catch mistakes you’ll miss by eye (wrong sitemap URL, a blocked directory you meant to allow, or an AI crawler rule that never matches).

Open the live file: visit https://yourdomain.com/robots.txt in an incognito window. Confirm you see the exact lines you saved or uploaded.
Hard-refresh and recheck: caching can keep old output around. WP VIP notes that /robots.txt can be cached for long periods by page cache (WP VIP robots.txt), so verify again after a purge if your host/CDN supports it.
Validate rules and sitemap: paste the live contents into Balzac’s free robots.txt and sitemap checker. Use it to confirm your directives parse cleanly and your Sitemap: line points to a real XML sitemap.
Confirm AI bot matching: run Balzac’s free AI crawler checker to make sure your GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot groups match the user-agent strings you wrote.

If the checkers show rules you did not expect, look for two common causes: (1) you uploaded a physical robots.txt that overrides a plugin editor, or (2) your environment serves a different robots.txt than your custom domain. WP Engine notes that only environments using a custom domain will see modifications made to robots.txt (WP Engine robots.txt guide).

If you want to rebuild from scratch, use Balzac’s robots.txt generator once to produce a clean baseline, then re-run the same verification steps on the final output.

Conclusion: The Checklist Before You Hit Publish

Before you ship changes to WordPress robots.txt, run a short checklist. Robots.txt mistakes tend to be silent: your pages keep working for humans while crawlers stop fetching what matters.

Confirm you are editing the right source. If a physical robots.txt exists in the site root, it overrides any virtual output from WordPress or an SEO plugin.
Re-check your baseline crawl rules. For most sites, blocking /wp-admin/ and allowing /wp-admin/admin-ajax.php is a safe default (per Yoast’s documented default output).
Keep a valid sitemap line. Use the exact URL your SEO plugin publishes (often /sitemap_index.xml for Yoast SEO). A wrong sitemap URL wastes discovery.
Review AI crawler groups line by line. Make sure the user agent names are spelled exactly (for example, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot). One typo makes the rule irrelevant.
Do not treat robots.txt as deindexing. If you need a page out of search results, use noindex via meta robots or an HTTP header. Robots.txt controls crawling access.
Retest after every change. Open /robots.txt in a browser, then run a robots.txt checker. If you changed AI bot rules, run an AI crawler checker too. Retest again after cache clears or migrations, since hosts and CDNs can keep serving an older version.

If you want one practical next step: open /robots.txt in an incognito window right now and confirm the output matches what you think you published. That single check catches most “edit robots.txt WordPress” failures.

Balzac publishes SEO articles directly to WordPress.