Robots.txt Guide: How to Write One Without Blocking Your Own Site
Robots.txt is a plain text file with a few lines in it, and it has taken more sites out of Google than any penalty I've dealt with. The usual story: a developer blocks everything on the staging server, the site goes live, and the file comes along for the ride. Traffic drops to near zero over a couple of weeks and nobody connects it to a five-line text file.
So it is worth understanding exactly what the file does and, just as importantly, what it doesn't do.
What robots.txt is for
The file sits at the root of your domain, at yoursite.com/robots.txt. Well-behaved crawlers such as Googlebot and Bingbot read it before crawling and follow its rules about which paths they may request.
It controls crawling, not indexing. That distinction trips people up constantly. If you block a URL in robots.txt, Google won't fetch it, but it can still index the bare URL if other pages link to it. You'll sometimes see these in search results with a note that no description is available. If you want a page out of the index, it needs a noindex meta tag or header, and Google must be allowed to crawl the page to see that tag.
Robots.txt is also not security. Anyone can read the file, and badly behaved bots ignore it. Never use it to "hide" an admin path or private files; listing them actually advertises where they are.
The basic syntax
A robots.txt file is made of groups. Each group starts with one or more User-agent lines followed by rules.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml
User-agent: *means the rules apply to every crawler that doesn't have its own group.Disallow: /path/blocks crawling of anything starting with that path.Allow:carves an exception out of a broader Disallow. Google picks the most specific (longest) matching rule.Sitemap:points crawlers at your XML sitemap. You can list several.*matches any sequence of characters and$marks the end of a URL.Disallow: /*.pdf$blocks all PDFs.
An empty Disallow: line means nothing is blocked. Disallow: / means everything is blocked. One character separates a healthy site from an invisible one.
Google ignores the crawl-delay directive and the old noindex rule inside robots.txt. Bing does respect crawl-delay. The Sitemap: line is worth keeping even on small sites; the XML sitemaps guide covers what that file should contain.
Sensible examples
A typical WordPress site:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://example.com/sitemap_index.xml
Blocking internal search results is reasonable because they generate endless thin URLs.
An ecommerce store with filter parameters:
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /*?sort=
Disallow: /*?*filter=
Be careful with parameter rules. If some filtered pages are ones you want ranking (say, "cotton kurtas under 1000" on a clothing store in Surat), make sure your pattern doesn't catch them. Shopify generates a default robots.txt and lets you edit it through a robots.txt.liquid template; its default rules are reasonable for most stores, so change them only with a specific reason.
You can create a starting file with the free robots.txt generator, then adjust it by hand.
Mistakes that cause real damage
Blocking CSS and JavaScript. Old advice said to block /wp-includes/ and theme folders. Google renders pages much like a browser, and if it can't load your CSS and JS it may see a broken layout or miss content. Leave those resources crawlable.
Using robots.txt to remove indexed pages. If a page is already indexed and you block it, Google can no longer see the noindex tag you added, so it may stay in results. Add noindex first, wait for it to drop out, and only then block crawling if you still want to.
Blocking pages that have backlinks. Links pointing to a blocked URL can't pass much value through to the rest of your site because Google can't see the page's own links. If a page has good links, let it be crawled and redirect it properly if it's obsolete.
Case and trailing slash errors. Paths are case-sensitive. Disallow: /Blog does not block /blog. Disallow: /shop blocks /shop, /shop/ and also /shopping-guide, which you may not have intended.
Forgetting subdomains. Each subdomain and protocol has its own robots.txt. blog.example.com doesn't follow the rules at example.com/robots.txt.
Should you block AI crawlers?
Many site owners now add groups for AI crawlers such as GPTBot or Google-Extended. Google-Extended controls whether your content is used for some of Google's AI model training; it doesn't affect normal Google Search crawling or ranking. Whether to block these is a business decision rather than an SEO one. A news publisher might block them; a SaaS company that wants to be mentioned in AI answers might not.
Testing your file
Search Console has a robots.txt report that shows the version Google last fetched and any parse errors. For individual URLs, the URL Inspection tool tells you whether a page is blocked. Test after every change, and keep a copy of the previous version so you can roll back quickly.
If you have a crawl budget concern on a large site, robots.txt is one of the main tools for it; crawl budget explains when that's worth your time. For a small site, the right robots.txt is usually short and boring.
Open your live robots.txt right now and check for a lone Disallow: / under User-agent: *. If it's there and you didn't mean it, that's the most valuable fix you'll make this week.