What is robots.txt?
robots.txt is a plain text file at the root of a website (yoursite.com/robots.txt) that tells crawlers which parts of the site they may or may not fetch. Rules are grouped by user agent, the name a crawler identifies itself with, and use simple directives such as Disallow and Allow. The file can also point to your XML sitemap.
The Robots Exclusion Protocol dates from 1994 and was formalized by the IETF as RFC 9309 in 2022. It works on trust: reputable crawlers follow it, but it is a request, not a security measure. It also controls crawling, not indexing, so a blocked URL can still show up in search results if other pages link to it. Do not confuse it with llms.txt, which points AI systems to content instead of setting access rules.
How robots.txt affects AI crawlers
Major AI providers document the user agents of their AI crawlers and how those bots handle robots.txt, which makes the file the main place to decide which AI systems may read your content. The key is that different bots do different jobs. OpenAI, for example, uses GPTBot to collect training data and OAI-SearchBot to find pages for ChatGPT search. Blocking GPTBot signals that your content should not be used to train OpenAI's models, while OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers.
Some tokens only control usage, not crawling. Google-Extended decides whether Google may use your content for Gemini, without affecting Google Search. And OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person triggers those visits.
How to set up robots.txt for AI crawlers
Decide per type of bot: training crawlers, search crawlers and user-triggered fetchers serve different goals.
If you want to be found in AI answers, do not block search crawlers such as OAI-SearchBot and PerplexityBot.
Add a separate User-agent group for each bot you want to treat differently, and test the file after every change.
Make sure no broad rule, such as Disallow: / for all user agents, blocks AI crawlers by accident, and that your CDN or firewall does not block them either.
Review the file regularly, because new AI crawlers keep appearing and providers update their user agents.
robots.txt only states your intentions. To see which bots actually visit and what they read, check your server logs or use Trackbase's Crawler Analytics.
Examples of robots.txt
A news publisher blocks GPTBot and CCBot to keep its articles out of training data, but allows OAI-SearchBot so its stories can still be cited in ChatGPT search.
A Dutch bicycle webshop disallows its cart, checkout and filter URLs for all crawlers but keeps product and category pages open.
A B2B software vendor finds that a leftover "Disallow: /" from its staging site blocked every crawler after a relaunch, and fixes it the same day.
A SaaS company allows all major AI crawlers and adds a Sitemap line, so crawlers can find new documentation pages.
Frequently asked questions
Should I block AI crawlers in robots.txt?
Only if you have a clear reason. Blocking training crawlers such as GPTBot keeps your content out of future model training, but blocking search crawlers such as OAI-SearchBot or PerplexityBot can remove your pages from AI answers. Brands that want AI visibility usually allow search crawlers and make a deliberate choice about training crawlers.
Does robots.txt keep my content out of AI answers?
Not completely. It only works for crawlers that respect it, and only for future crawling. AI models may already know about you from earlier data, and they also learn from other sites that mention you. Some providers also say robots.txt may not apply to fetches a user triggers directly. So robots.txt limits access to your own pages, not what AI models say about you.
How do I check whether my robots.txt blocks AI crawlers?
Open yoursite.com/robots.txt and look for User-agent groups that name AI bots, such as GPTBot, ClaudeBot or PerplexityBot, and for broad Disallow rules that apply to all agents. Use a robots.txt testing tool to check specific URLs. Then compare with your server logs, because firewalls and CDNs can block bots even when robots.txt allows them.