robots.txt Validator and Generator
Parse a robots.txt the way Google does, test whether a URL is blocked for a given crawler, and generate a clean WordPress file with AI crawler controls.
How to use the robots.txt validator
- Open yoursite.com/robots.txt, copy the whole file and paste it into the validator. Parsing happens as you type, so you can edit the file here and watch the issues clear.
- Check the group list at the bottom. Each User-agent block is shown with its rules the way a crawler reads it, including empty groups that allow everything.
- Type a path into the URL tester and pick a crawler. The tool applies the same rules Google uses: the most specific matching user-agent group wins, then the longest matching path, and Allow beats Disallow on a tie.
- Switch to the Generate tab to build a WordPress-friendly file. Tick the crawlers you want to keep out, add your sitemap URL, then copy, download or send the result to the validator.
How matching works
A crawler looks for the group whose User-agent line best matches its own name. Googlebot uses the “Googlebot” group if there is one, otherwise the “*” group, and it ignores the others. Within that group it compares the URL path against every Allow and Disallow rule. The rule with the longest path wins; if an Allow and a Disallow tie on length, Allow wins. Paths are prefixes, so Disallow: /blog blocks /blog, /blog/ and /blog-archive. Use * for a wildcard and $ to anchor the end of a URL.
robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it, just without a description. To keep a page out of the index, let it be crawled and add a noindex meta tag. Blocking a page and noindexing it at the same time means Google never sees the noindex.
Rules that cause real damage
- Disallow: / for * or Googlebot. Removes the entire site from search. Usually a staging setting that was carried to production.
- Blocking /wp-content/ or CSS and JavaScript files. Google renders pages like a browser. If it cannot load the styles and scripts, it may see a broken layout and rank accordingly.
- Blocking /wp-admin/ without allowing admin-ajax.php. Some themes load content through admin-ajax; blocking it can hide that content from Google.
- Relying on Crawl-delay. Google ignores it. Use Search Console crawl settings or fix the server.
- Using noindex in robots.txt. Google stopped supporting it in 2019.
About blocking AI crawlers
GPTBot, CCBot, ClaudeBot and Google-Extended collect content for model training and are separate from the crawlers that power search results. Blocking them does not affect Google Search rankings. Whether to block them is a business decision: some sites prefer to keep their content out of training data, others want to be cited by AI assistants. The generator lists them separately so the choice is explicit.
Frequently asked questions
Where does robots.txt live and how do I edit it in WordPress?
It must be at the root of the domain: yoursite.com/robots.txt. WordPress generates a virtual one if no file exists. Rank Math and Yoast both have an editor for it, or you can upload a physical file by FTP, which then takes precedence.
Should I block tag and author archives?
Only if they add nothing for searchers. Blocking them in robots.txt stops crawling but not indexing; a noindex tag on those archives is usually the better tool. The generator includes the options for sites that have decided to block them.
How long until changes take effect?
Google caches robots.txt for up to 24 hours. You can ask for a recrawl in Search Console under Settings, robots.txt report.
Related tools
Background reading: where robots.txt sits in a full audit, in the technical SEO checklist.