What is robots.txt?
robots.txt is a plain text file at the root of a website — for example example.com/robots.txt — that tells crawlers which parts of the site they may visit. Search engines and well-behaved AI crawlers read it before they crawl. It is a request, not a lock: it does not hide pages from people, and badly behaved bots can ignore it.
Blocking AI crawlers
AI companies send their own crawlers. Some collect pages to train models (such as GPTBot, ClaudeBot and CCBot); others fetch pages to answer a user’s question or to show your site in AI search results (such as OAI-SearchBot, Claude-SearchBot and PerplexityBot). “Block AI training only” keeps your content out of training data while still letting AI search tools find and link to you. Google-Extended and Applebot-Extended only control AI use; blocking them does not remove you from Google Search or Siri.
How to use the robots.txt generator
- Choose the default for all crawlers and list any folders to keep out, such as /admin/ or /cart/.
- Add your sitemap so search engines find every page.
- Pick the AI crawlers to block, or use one of the presets.
- Download robots.txt and upload it to the root folder of your site, so it opens at yourdomain.com/robots.txt.
- Test addresses against your rules before you publish.
How the tester decides
- A crawler follows the group that names it; if none does, it follows the “User-agent: *” group.
- Within that group, the longest matching rule wins. If an Allow and a Disallow rule are equally long, Allow wins.
- * matches any characters and $ marks the end of the address, so Disallow: /*.pdf$ blocks every PDF file.
Frequently asked questions
Will robots.txt remove a page from Google?
No. It stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out of search results, let it be crawled and add a noindex meta tag.
Does blocking GPTBot stop ChatGPT from showing my site?
GPTBot is only used for training. ChatGPT’s search results use OAI-SearchBot, and visits a user asks for come from ChatGPT-User. Block those too only if you do not want to appear in ChatGPT at all.
Where do I put the robots.txt file?
In the root folder of your domain, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file.