CiteSiteFree tool
AI crawler checker
Paste a domain. This reads its robots.txt and reports, agent by agent, which AI and search crawlers are allowed in — and, for the ones that are not, which line of your file did it and what the block actually costs you.
Reads one file. Nothing is stored, and no account is needed.
Why agent-by-agent, and not a score
Almost every AI-crawler check answers the question are you blocking AI bots? That question has no useful answer, because the bots in a robots.txt do three unrelated jobs.
One group fetches pages so an assistant can read and quote them in an answer it is composing right now. Block one of those and no amount of on-page work will produce a citation from that engine — it cannot read the page it would be citing. A second group is the classic search crawlers, two of which also feed AI answers: Bing’s index is what Copilot reads, and Googlebot’s is what AI Overviews draw on. The third group collects text to train future models, and has no bearing at all on whether an assistant names you this afternoon.
Reported as one number, a site that blocks CCBot and a site that blocks PerplexityBot look equally cautious. One of them has made a licensing decision. The other has removed itself from an answer engine. So this tool never averages them.
The mistake this exists to catch
The single most common finding is a Disallow: / under a named AI user-agent that somebody added during a general tightening of the file, months ago, alongside a dozen scrapers — and Google-Extended going in with them. Nothing breaks. The site keeps ranking, traffic is unchanged, Search Console is clean. It simply stops appearing in Gemini and AI Overviews, and there is no report anywhere that says so.
That is why the output names the group that decided each verdict. A rule written at an agent by name is a line to delete; a Disallow: / inherited from User-agent: * is a site-wide setting that happens to catch it, and the fix is somewhere else in the file entirely.
What it does not tell you
Whether you are actually cited. Crawler access is a precondition, not an outcome: a site every engine can read may still go unnamed because nothing on it answers the question being asked. This tool checks the door is open. Whether anyone walks through it is what a full analysis measures, by putting real questions to ChatGPT, Gemini and Perplexity and recording what comes back.
It also reads only /robots.txt. A page can be excluded by a noindex meta tag, an X-Robots-Tag header, or a login wall, none of which appear in that file.
Questions
- Does blocking GPTBot stop ChatGPT citing my site?
- No. GPTBot collects training data. The crawler that decides whether ChatGPT search can read and link to your pages is OAI-SearchBot, and it obeys a separate rule. Blocking GPTBot while allowing OAI-SearchBot is a coherent position: no training use, still citable.
- What is Google-Extended, and why is it the one people get wrong?
- Google-Extended is not a crawler. It is a switch that tells Google whether your content may be used by Gemini and its AI answers, and Googlebot does the actual fetching. Because it appears in robots.txt alongside real bots, it is often disallowed during a general clean-up of AI crawlers — which quietly removes the site from Google's AI surfaces while leaving it in the search index, so nothing looks broken.
- Is robots.txt actually enforced?
- It is a request, not a lock. The major operators publish their user-agent names and honour the file; there is nothing in the protocol that compels anyone to. If you need a page not to be read, robots.txt is the wrong tool — it is a public list of what you would rather people did not fetch.
- Should I block AI crawlers at all?
- It depends entirely on which one. Blocking an answer engine's crawler removes you from that engine's answers, which is a marketing decision. Blocking a training crawler is a licensing decision with no effect on whether you are cited today. They are frequently made as one decision, and they are not one decision.
- Where does this tool get its verdicts?
- It fetches your robots.txt once and applies the same most-specific-wins matching the CiteSite crawler obeys: an agent with its own group ignores the catch-all entirely. Nothing is stored, and no account is required.
Related
Which AI crawlers matter, and which ones you can safely block — the same three categories, at length, with what each operator has actually published about its bots.
An open robots.txt is the floor. CiteSite asks ChatGPT, Gemini and Perplexity the questions your customers ask and records whether you are named, recommended and cited — and whether the answer came from live search or from memory.