Free tool

robots.txt Generator

Build a correct robots.txt controlling Google, Bing and AI crawlers such as GPTBot.

Search engine crawlers

AI crawlers

GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Applebot-Extended.

Disallowed paths (one per line)Such as admin panels, login pages and internal search results.

Crawl delay (optional)Google ignores this directive; Bing and Yandex respect it.

Sitemap URL (optional)Crawlers find /sitemap.xml automatically; add this only for a custom path.

robots.txt file

User-agent: *
Allow: /
Disallow: /admin
Disallow: /app
Disallow: /login

Why this matters

robots.txt is the first file any crawler requests. One wrong line can hide your entire site from Google — one of the most expensive and most common technical mistakes after a launch.

Since 2023 a new crawler class exists: AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Blocking them protects content from training but reduces the chance your brand is cited in their answers.

How to use it

1Choose the global rule: allow all or block all.
2Add paths to hide, such as an admin panel or internal search pages.
3Decide your stance on AI crawlers.
4Copy the file and upload it at your site root as /robots.txt.

Frequently asked questions

Does robots.txt stop a page appearing in Google?

Not necessarily. Blocking crawling stops Google reading the page, but it can still appear without a description if other pages link to it. To truly hide a page use a noindex tag and leave crawling allowed so Google can see it.

Should I add a Sitemap line?

Helpful but not required — crawlers find /sitemap.xml automatically. Add it when your sitemap lives at an unusual path.

Next step

Does your brand appear in ChatGPT and Gemini answers?

These tools fix one page at a time. SADARA scans your whole site, asks AI engines your customers' real questions, and tells you whether you or a competitor appears — with a visibility score and an action plan.

Start the free scan