Robots.txt File: What It Is, How to Create One & Common Mistakes (With Examples)
Learn what a robots.txt file does, how to create one step by step, copy ready-to-use examples and avoid mistakes that block pages by accident.
A single small text file can decide which parts of your website search engines are allowed to crawl. That file is robots.txt, and it is easy to get wrong in ways that quietly hurt your visibility. One misplaced line can block an entire site, while a missing file can leave crawlers wasting time on pages you never wanted them to visit.
This guide explains what a robots.txt file is, how to create one step by step, gives ready-to-use examples, and shows the mistakes to avoid. You will also find free tools to generate and test your own file.
Key Takeaways
- A robots.txt file tells crawlers which URLs they may crawl. It lives at the root of your domain.
- It controls crawling, not indexing. A blocked URL can still appear in search if other pages link to it.
- It is not a security tool. Never rely on it to hide private content.
- Always test the file after changing it and include a link to your XML sitemap.
What Is a Robots.txt File?
A robots.txt file is a plain text file that follows the Robots Exclusion Protocol. It gives instructions to crawlers, the automated programs that visit websites, about which paths they should or should not request. Well-behaved crawlers, including Googlebot, check this file before crawling a site.
The file must sit at the root of the host it applies to, for example https://example.com/robots.txt. A file placed in a subfolder is ignored, and a subdomain needs its own file.
What Does Robots.txt Do for SEO?
- Manages crawl activity. You can keep crawlers away from low-value areas such as internal search results, cart pages or admin folders.
- Points to your sitemap. A sitemap line helps crawlers discover your important URLs.
- Reduces server load. Limiting crawling of heavy or endless URL patterns can help on large sites.
- Sets rules per crawler. You can give different instructions to different user agents.
What it does not do is remove pages from search results. For that, use a noindex directive on the page itself or Google’s removal tools. If you block a page in robots.txt, Google cannot crawl it to see a noindex tag, so the two should not be combined on the same URL.
Robots.txt Syntax Explained
The file is made of groups of rules. Each group starts with a user agent and is followed by allow or disallow lines.
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Names the crawler the rules apply to | User-agent: Googlebot |
| Disallow | Blocks a path from being crawled | Disallow: /private/ |
| Allow | Permits a path, often inside a blocked folder | Allow: /private/public-page.html |
| Sitemap | Gives the full URL of your sitemap | Sitemap: https://example.com/sitemap.xml |
Two wildcard characters are supported by major search engines: * matches any sequence of characters and $ marks the end of a URL. Paths are case sensitive, so /Photos/ and /photos/ are different. Google also ignores some directives you may see elsewhere, such as crawl-delay, so do not depend on it for Googlebot.
Robots.txt Examples You Can Copy
Allow All Crawlers
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
An empty Disallow value means nothing is blocked.
Block a Folder
User-agent: *
Disallow: /internal-search/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml
Typical WordPress File
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml
This keeps crawlers out of the admin area while still allowing the AJAX file that many themes and plugins need.
Block One Crawler Only
User-agent: ExampleBot
Disallow: /
User-agent: *
Disallow:
Replace ExampleBot with the user agent you want to restrict.
How to Create a Robots.txt File Step by Step
- Decide what to block. List areas that offer no search value, such as admin screens, internal search results, cart and checkout pages, and endless filter combinations.
- Write the rules. Start with
User-agent: *, add your Disallow lines, and finish with your sitemap URL. The free Robots.txt Generator builds the file for you from a few choices. - Create the XML sitemap. If you do not have one yet, the XML Sitemap Generator can produce it from your list of URLs.
- Save it as robots.txt. Use a plain text file with UTF-8 encoding and the exact lowercase name.
- Upload it to your site root. It must be reachable at yourdomain.com/robots.txt.
- Test it. Open the URL in a browser and run it through the Robots.txt Checker to confirm the rules say what you intend.
Robots.txt and AI Crawlers
More site owners now decide separately whether AI-related crawlers may access their content. Each vendor publishes its own user agent names, so check the official documentation before adding a rule, and review them periodically because names can change. Remember that robots.txt is a request that responsible crawlers honour, not a technical barrier. The AI Robots Checker and AI Crawler Checker can help you see how your current rules apply to these bots.
Whether to allow or block them is a business decision. Allowing access may support visibility in AI-generated answers, while blocking gives you more control over how your content is used.
Common Robots.txt Mistakes
- Blocking the whole site by accident.
Disallow: /underUser-agent: *stops all compliant crawlers. This often happens when a staging file is copied to a live site. - Blocking CSS and JavaScript. Google needs these files to render pages properly, so blocking them can harm how your pages are understood.
- Using robots.txt to hide private pages. The file is public and anyone can read it. Protect sensitive content with authentication.
- Combining Disallow with noindex. A blocked page cannot be crawled, so its noindex tag is never seen.
- Putting the file in the wrong place. It must be at the root of the exact host and protocol you want to control.
- Forgetting the sitemap line. It is optional but helpful for discovery.
- Ignoring case sensitivity. A rule for
/Blog/will not match/blog/.
How to Test and Monitor Your Robots.txt
- Open the live file in your browser and read every line.
- Run the Crawlability Checker on key pages to confirm they are not blocked.
- Use the Indexability Checker to see whether noindex tags or canonicals are affecting those pages.
- Review the robots.txt report in Google Search Console to see how Google fetched the file and whether it found problems.
- Recheck after every site migration, redesign or plugin change.
Robots.txt Checklist
- File is at the root of the correct domain and subdomain
- Important pages, CSS and JavaScript are not blocked
- Low-value areas are disallowed where appropriate
- Sitemap URL is included and works
- No sensitive content relies on the file for protection
- File tested after every change
Frequently Asked Questions
Where do I put a robots.txt file?
Place it in the root directory of your site so it loads at yourdomain.com/robots.txt. It cannot work from a subfolder.
Does robots.txt stop a page from appearing on Google?
Not reliably. It stops crawling, but a blocked URL can still be listed if other pages link to it. Use noindex or removal tools to keep a page out of results.
Is a robots.txt file required?
No. Without one, crawlers assume they may crawl everything. It is still useful for managing crawling and pointing to your sitemap.
Can I block only certain search engines?
Yes. Create a group for the specific user agent and give it its own rules.
How do I check if my robots.txt is working?
Open the file in a browser, test it with a robots.txt checker, and review Google Search Console to see how Google reads it.
Final Thoughts
A good robots.txt file is short, intentional and tested. Block only what you genuinely do not need crawled, keep important resources open, link your sitemap, and re-test after every major change. Done well, it keeps crawlers focused on the pages that matter.
Ready to create yours? Use the Robots.txt Generator to build a clean file in minutes, then verify it with the Robots.txt Checker.