Robots.txt Generator
Give compliant crawlers a clear access policy. Create a small robots.txt file from the paths you actually want crawled, with readable rules you can review before publishing at the root of your site.
A clear way in. A deliberate boundary.
IN YOUR BROWSER / PRIVATE INPUTGood input. Useful output.
Define crawler access with a downloadable robots.txt.
Try the example, or make it your own.How to use the
Robots.txt Generator
- 01
Pick the crawler group
Use * for a general group or enter a specific user-agent token. A token is a crawler name, not a full browser user-agent string.
- 02
List the intended paths
Put each allow or disallow path on its own line. Paths start with /. Check their spelling and case against real URLs; a small typo can change which pages match.
- 03
Add the sitemap and publish
Optionally supply an absolute sitemap URL. Download robots.txt, inspect the text and publish it at the relevant host's root. Then request /robots.txt directly to confirm the server returns your file.
Read a rule as an access instruction
The generator writes a single selected crawler group. It does not merge the result with your existing groups, inspect your server configuration or test every affected URL. Keep a copy of the current file and compare the complete policy before replacing it.
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Names the group of crawlers a rule applies to. | User-agent: * |
| Disallow | Requests that a matching path not be crawled. | Disallow: /admin/ |
| Allow | Creates an allowed path, including an exception within a broader block. | Allow: /assets/public/ |
| Sitemap | Points crawlers toward a sitemap location. | Sitemap: https://example.com/sitemap.xml |
Keep an admin area out of routine crawling
This example permits ordinary pages while requesting that compliant crawlers avoid the admin path. The folder must still have real authentication. The sitemap URL is a discovery hint, not an instruction to index every URL inside it.
User agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xmlUser-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlCrawling, indexing and privacy are different jobs
A robots rule asks a crawler not to request a URL. It is not a password and it does not prevent people from visiting that URL. Do not list sensitive locations and assume the file protects them. Use authentication and authorization for private content.
Blocking crawling also does not reliably remove a URL from search. A crawler may know a URL exists through links while being unable to retrieve its content. If a page should carry a noindex directive, the relevant crawler normally needs access to read that directive. Choose an indexing policy deliberately instead of combining contradictory controls.
Review the policy from the outside
Check the exact host
A file on one subdomain does not automatically establish the policy for every other subdomain. Inspect the public host and protocol you intend to manage.
Avoid accidental site-wide blocks
Disallow: / is a broad instruction. It can be appropriate for a deliberately closed crawler group, but should not slip out of staging into a public site's general group.
Confirm the delivered response
A CMS route, redirect or CDN may return an HTML page instead of the text file. View the HTTP response, not just the file in your repository.
Robots.txt Generator:
questions, answered.
Does Allow: / override every disallow rule?
No. Do not read a robots file as a simple last-line-wins list. Matching specificity matters. Test important paths against the crawler's documented interpretation before publishing.
Can robots.txt block AI training but allow AI search?
Some providers use distinct tokens for different purposes. Set policy per documented token and verify it with the crawler readiness checker; a blanket assumption about all AI bots is unreliable.
Why does my blocked page still appear in search?
A crawl restriction and an indexing decision are different. Review the URL's indexing controls and search-engine guidance rather than repeatedly adding more disallow lines.
Will this generator change my live website?
No. It produces a downloadable file. You decide where and when to publish it, and you should preserve any existing crawler groups that still serve a purpose.
The calculations and checks above describe this tool's implementation. Use these primary references for the underlying formats and publishing guidance.
Create a robots.txt file ↗ (opens in a new tab)Robots Exclusion Protocol ↗ (opens in a new tab)Make access an intentional choice.
Generate the policy, review critical paths, then verify the file your public host serves.
Use the Robots.txt Generator↑