Skip to main content
100% private — runs in your browser

robots.txt Generator

Create a valid robots.txt file with allow/disallow rules, crawl-delay, and sitemaps.

Rules

robots.txt

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

Sitemap: https://thetoolstoday.com/sitemap.xml

How to use robots.txt Generator

  1. Set user-agent and paths

    Choose the user-agent and list Allow/Disallow paths (one per line).

  2. Add sitemaps and options

    Include sitemap URLs and an optional crawl-delay if needed.

  3. Copy or download

    Copy the robots.txt content or download it for your site root.

Frequently asked questions

Does robots.txt prevent indexing?

Not reliably. It controls crawling; pages may still be indexed if discovered via links. Use noindex meta/headers for indexing control.

Where should I put robots.txt?

At the root of your host, e.g. https://example.com/robots.txt.

Is crawl-delay supported by Google?

No, Googlebot does not follow Crawl-delay. Manage Google crawl rate in Search Console instead.

Can I block sensitive folders here?

You can disallow crawling, but do not treat robots.txt as security. Protect sensitive content with authentication.

Is this generated locally?

Yes. Your rules are composed in the browser and not uploaded.

About robots.txt Generator

The robots.txt Generator creates a properly formatted robots.txt file to guide search engine crawlers. Define user-agents, allow or disallow paths, add crawl-delay hints, and include sitemap locations, then copy or download the result. Generating it locally lets you review rules carefully before deploying, reducing the risk of accidentally blocking important pages.

How robots.txt works

robots.txt lives at the root of your site and provides crawling directives to well-behaved bots. Rules are matched by user-agent and path prefixes. Disallow prevents crawling of matching paths (not guaranteed indexing control), while Allow can be used to permit exceptions within a disallowed tree. Because syntax mistakes can hide large parts of a site, it is important to validate rules and understand that robots.txt is public.

Sitemaps and crawl-delay

Including Sitemap directives helps crawlers discover your sitemap URLs quickly. Crawl-delay is a non-standard hint supported by some bots (notably not Googlebot) to slow request rates; use it cautiously. This generator formats common directives clearly so you can produce a baseline file for most sites and adjust per your hosting and crawl budget needs.

Safe usage

Never rely on robots.txt to protect sensitive or private content — blocked URLs can still be indexed if linked elsewhere, and the file itself reveals paths. Use proper authentication and noindex where appropriate. Generate and review locally, then test in search console tools after deployment to confirm the intended behavior.

Use robots.txt Generator on your site

This tool runs free and private in the browser. Paste it anywhere — blog posts, documentation, team wikis — and your visitors will use it without leaving your site. The embed includes a small "Made with The Tools Today" credit link.

Open the tool page →

Your privacy, your choice

We use cookies to understand how the site is used and to show relevant ads. Every tool on this site runs entirely in your browser, so the files and text you process are never uploaded. Read our privacy policy.