By Khalid Namra7 min read
Quick answer
robots.txt is a text file at your site root telling crawlers which paths they may visit. It blocks crawling, not indexing: a blocked page can still appear without a description if found via links. To keep a page out use noindex, and never block CSS and JS Google needs to render the page.
A basic example
User-agent: * / Disallow: /cart / Disallow: /account / Sitemap: https://example.com/sitemap.xml
Each directive goes on its own line in the real file.
AI crawlers
| User-agent | Owner and purpose |
|---|---|
| GPTBot | OpenAI — training data collection |
| OAI-SearchBot | OpenAI — appearing in ChatGPT search results |
| PerplexityBot | Perplexity — indexing for answers |
| Google-Extended | Google — use in Gemini; does not affect Search ranking |
| ClaudeBot | Anthropic — data collection |
Allow or block?
If you want AI engines to mention you, do not block search crawlers like OAI-SearchBot and PerplexityBot. You may block training crawlers only if you have a business reason; that is a separate decision from search visibility.
Frequently asked questions
Where does robots.txt go?
At the domain root: example.com/robots.txt; each subdomain has its own.
Does Disallow remove a page from Google?
No. Use noindex and allow crawling so Google sees the tag.
Related SADARA services
- Sadara SEOSadara SEO is the Sadara platform (getsadara.com) for Arabic SEO: it audits your site, uncovers what keeps it out of Google's top results, and turns findings into weekly, practical steps in your market's language.
