2026-11-09

robots.txt explained with ready examples (including AI crawlers)

What robots.txt does and does not do, and how to allow or block GPTBot, PerplexityBot and Google-Extended without hurting visibility.

By Khalid Namra7 min read

Quick answer

robots.txt is a text file at your site root telling crawlers which paths they may visit. It blocks crawling, not indexing: a blocked page can still appear without a description if found via links. To keep a page out use noindex, and never block CSS and JS Google needs to render the page.

A basic example

User-agent: * / Disallow: /cart / Disallow: /account / Sitemap: https://example.com/sitemap.xml

Each directive goes on its own line in the real file.

AI crawlers

User-agentOwner and purpose
GPTBotOpenAI — training data collection
OAI-SearchBotOpenAI — appearing in ChatGPT search results
PerplexityBotPerplexity — indexing for answers
Google-ExtendedGoogle — use in Gemini; does not affect Search ranking
ClaudeBotAnthropic — data collection

Allow or block?

If you want AI engines to mention you, do not block search crawlers like OAI-SearchBot and PerplexityBot. You may block training crawlers only if you have a business reason; that is a separate decision from search visibility.

Frequently asked questions

Where does robots.txt go?

At the domain root: example.com/robots.txt; each subdomain has its own.

Does Disallow remove a page from Google?

No. Use noindex and allow crawling so Google sees the tag.

Related SADARA services

  • Sadara SEOSadara SEO is the Sadara platform (getsadara.com) for Arabic SEO: it audits your site, uncovers what keeps it out of Google's top results, and turns findings into weekly, practical steps in your market's language.

Leadership is a decision

Send your site link and get an initial analysis of your visibility across search and AI engines.