CRAWL

Robots.txt Generator

Build a robots.txt file with allow and disallow rules, crawl-delay and your sitemap link.

Rule group 1

User-agent

Allow (one path per line)

Disallow (one path per line)

Sitemap URL

Crawl-delay in seconds (optional, Google ignores it)

Block common AI training crawlers

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/

FAQ

Robots.txt Generator: questions

Where do I upload robots.txt?

To the root of your domain, so it opens at yourdomain.com/robots.txt.

Does Disallow remove a page from Google?

No. It stops crawling, but a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of the index.