robots.txt Generator
Create a valid robots.txt file with allow/disallow rules, crawl-delay, and sitemaps.
Rules
robots.txt
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Sitemap: https://thetoolstoday.com/sitemap.xml
How to use robots.txt Generator
Set user-agent and paths
Choose the user-agent and list Allow/Disallow paths (one per line).
Add sitemaps and options
Include sitemap URLs and an optional crawl-delay if needed.
Copy or download
Copy the robots.txt content or download it for your site root.
Frequently asked questions
Does robots.txt prevent indexing?
Not reliably. It controls crawling; pages may still be indexed if discovered via links. Use noindex meta/headers for indexing control.
Where should I put robots.txt?
At the root of your host, e.g. https://example.com/robots.txt.
Is crawl-delay supported by Google?
No, Googlebot does not follow Crawl-delay. Manage Google crawl rate in Search Console instead.
Can I block sensitive folders here?
You can disallow crawling, but do not treat robots.txt as security. Protect sensitive content with authentication.
Is this generated locally?
Yes. Your rules are composed in the browser and not uploaded.
About robots.txt Generator
The robots.txt Generator creates a properly formatted robots.txt file to guide search engine crawlers. Define user-agents, allow or disallow paths, add crawl-delay hints, and include sitemap locations, then copy or download the result. Generating it locally lets you review rules carefully before deploying, reducing the risk of accidentally blocking important pages.
How robots.txt works
robots.txt lives at the root of your site and provides crawling directives to well-behaved bots. Rules are matched by user-agent and path prefixes. Disallow prevents crawling of matching paths (not guaranteed indexing control), while Allow can be used to permit exceptions within a disallowed tree. Because syntax mistakes can hide large parts of a site, it is important to validate rules and understand that robots.txt is public.
Sitemaps and crawl-delay
Including Sitemap directives helps crawlers discover your sitemap URLs quickly. Crawl-delay is a non-standard hint supported by some bots (notably not Googlebot) to slow request rates; use it cautiously. This generator formats common directives clearly so you can produce a baseline file for most sites and adjust per your hosting and crawl budget needs.
Safe usage
Never rely on robots.txt to protect sensitive or private content — blocked URLs can still be indexed if linked elsewhere, and the file itself reveals paths. Use proper authentication and noindex where appropriate. Generate and review locally, then test in search console tools after deployment to confirm the intended behavior.
Use robots.txt Generator on your site
This tool runs free and private in the browser. Paste it anywhere — blog posts, documentation, team wikis — and your visitors will use it without leaving your site. The embed includes a small "Made with The Tools Today" credit link.
Related tools
Sitemap Generator
Generate an XML sitemap from a list of URLs with priority and change frequency.
Open toolMeta Tag Generator
Generate SEO meta tags, canonical links, and robots directives with a live SERP preview.
Open toolOpen Graph & Twitter Card Preview
Create and preview Open Graph and Twitter card meta tags for rich social sharing.
Open toolhtaccess Redirect Generator
Generate Apache .htaccess rules for HTTPS, www, trailing slash, and redirects.
Open tool