AI Crawler Access Checker
Checks whether sixteen AI crawlers are allowed to read a page. It reads your robots.txt the way a crawler does, then requests the page as GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and Bingbot to catch blocks that robots.txt does not show.
How to use the crawler access checker
- Enter the address of a page you want assistants to be able to cite and press Check access. The run takes about twenty seconds because each crawler is tested in turn.
- Read the two columns separately. The robots.txt column is the instruction you have published. The live request column is what your server actually did when a request arrived carrying that crawler’s name.
- If robots.txt says allowed and the live request says blocked, the block is in a firewall, CDN or security plugin. That is the case site owners never find on their own.
- Decide deliberately. Blocking training crawlers while allowing search and live-fetch crawlers is a reasonable position. Blocking everything removes you from the answers.
The three kinds of AI crawler
They are routinely lumped together, and the difference decides what blocking costs you.
Training crawlers
GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider, and the Google-Extended and Applebot-Extended tokens. They collect text to train models. Blocking them keeps your content out of future training sets and has no effect on whether you appear in search results or in live answers.
Search index crawlers
OAI-SearchBot, PerplexityBot, Claude-SearchBot, and also Googlebot and Bingbot, whose indexes feed AI Overviews, AI Mode, Copilot and ChatGPT search. Blocking these removes your pages from the pool that answers are built from.
User-triggered fetchers
ChatGPT-User, Perplexity-User and Claude-User. They fetch a page at the moment a person asks the assistant about it or pastes its link. Blocking them means the assistant tells that person it could not read your site.
Where hidden blocks come from
- CDN bot settings. Cloudflare offers a one-click option to block AI crawlers, and on some plans it is switched on by default for new sites.
- Hosting firewalls. Shared hosts, including several popular with Pakistani websites, rate-limit or challenge unfamiliar user agents at the server level.
- Security plugins. Wordfence, Sucuri and similar tools carry bot lists that are updated without the site owner being told.
- Old robots.txt templates. Files copied from blog posts in 2023 often contain a blanket Disallow for every AI user agent.
- JavaScript rendering. Not a block, but the same result. These crawlers do not run JavaScript, so a page that builds its content in the browser looks empty to them.
To change the robots.txt side, the robots.txt validator and generator has each AI crawler as a separate option.
What this tool cannot tell you
- Requests come from this website’s server, not from the crawlers’ own networks. A firewall that verifies the crawler’s IP address may treat the genuine crawler differently. Your server logs are the final word.
- Google-Extended and Applebot-Extended are robots.txt tokens with no crawler of their own, so they have no live test.
- A crawler being allowed does not mean it has visited. Check your access logs for the user agent names to see who actually comes.
Frequently asked questions
Should I block AI crawlers?
It is a business decision. If your content is the product, as with paid research or a news archive, blocking training crawlers is reasonable. If you sell a service and want to be recommended, being readable matters more than being protected. Most businesses should allow search and user-triggered crawlers at minimum.
Does blocking Google-Extended affect my Google rankings?
No. Google states it has no effect on Search. It controls whether your content is used for Gemini. AI Overviews are part of Search and are governed by Googlebot and the snippet controls, not by Google-Extended.
robots.txt allows everything but the live test shows 403. What now?
Look at the server header in the result. If it names Cloudflare, check Security, Bots in the dashboard. Otherwise ask your host whether a firewall rule blocks the user agent, and check any security plugin’s blocked list.
Why is the run limited?
Each check makes nine requests to the site you enter. The limit of ten runs an hour keeps the tool from being used to load other people’s servers.
Do I need an llms.txt file?
Not today. No major engine has confirmed that it reads the file. It does no harm, but it will not fix an access problem.
Related tools and services
Access is the first step of the GEO service, followed by the rewrites and the off-site work that get a readable site cited.