HIFreeTools
ENES

Robots.txt Generator

Build a valid robots.txt file in seconds — allow or block crawlers, add per-path rules and a crawl-delay, and include your sitemap line, ready to copy or save.

FreeNo sign-upPrivate — runs in your browser, nothing is uploaded

robots.txt
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Save this as robots.txt in your site's root (e.g. example.com/robots.txt). Remember: robots.txt is a crawl hint, not a security control — it does not hide or protect a page. Generated in your browser.

Quick answer

A robots.txt file tells web crawlers which parts of your site they may or may not request. This generator builds a valid file from simple choices: allow all crawlers, block all crawlers, or set custom Allow/Disallow rules for a specific user-agent, plus an optional crawl-delay and your sitemap URL. Save the output as robots.txt in your site's root, e.g. example.com/robots.txt.

Formula & method

The tool writes standard robots.txt directives. 'Allow all' emits User-agent: * with an empty Disallow; 'Block all' emits Disallow: /. Custom mode lets you set the user-agent and list Allow and Disallow paths (one per line), plus an optional Crawl-delay. A Sitemap: line is appended when you provide a sitemap URL. The file must live at the domain root to take effect.

Examples

Example 1: Allow everything
Input
Policy = Allow all
Result
User-agent: *\nDisallow:
Why
An empty Disallow permits crawlers to access the whole site.
Example 2: Block a folder
Input
Disallow: /admin/
Result
User-agent: *\nDisallow: /admin/
Why
Stops compliant crawlers from requesting anything under /admin/.
Example 3: Add your sitemap
Input
Sitemap = https://example.com/sitemap.xml
Result
Sitemap: https://example.com/sitemap.xml
Why
Helps search engines discover all your URLs.

When to use this tool

  • Keeping crawlers out of admin, cart, search, or staging paths.
  • Pointing search engines to your sitemap.
  • Setting a crawl-delay for bots that hammer your server.

Common mistakes

  • Treating robots.txt as security — it only asks well-behaved crawlers to stay out; it does not hide or protect a page.
  • Disallowing a page you also want de-indexed: if it's blocked, Google can't see a noindex tag, so use noindex instead.
  • Placing the file anywhere but the root — only example.com/robots.txt is read, not example.com/folder/robots.txt.

Frequently asked questions

Where does robots.txt go?

In the very root of your domain, reachable at example.com/robots.txt. Crawlers ignore it anywhere else.

Does robots.txt remove a page from Google?

No. It blocks crawling, but a blocked URL can still be indexed if linked elsewhere. To remove a page, allow crawling and add a noindex meta tag, or use the removal tools.

What does Disallow: with nothing after it mean?

An empty Disallow means 'disallow nothing' — i.e. allow the crawler to access everything. Disallow: / blocks the entire site.

Can I set rules for just Googlebot?

Yes. In custom mode, set the user-agent to Googlebot (or any bot) and its Allow/Disallow rules; repeat for other bots as needed.

Is robots.txt case-sensitive?

The paths are. /Admin/ and /admin/ are different, so match the exact casing of your URLs.

Is anything uploaded?

No — the file is generated in your browser.

Sources & references

External references open in a new tab. We are independent and not affiliated with these organizations.

  • ✓ Free to use
  • ✓ No sign-up required
  • ✓ Runs entirely in your browser — nothing is uploaded.
  • ✓ Formula and method shown above
Last updated: How we calculateReport a correctionDisclaimer

Provided “as is” for general information only — results may be inaccurate, so verify before you rely on them. No warranty; use at your own risk.

Built and reviewed by HIFreeTools against the formula shown above and any authoritative references cited on this page. See our methodology and editorial standards.

Related tools

Embed this tool on your site

Free to embed, no sign-up. Paste this code where you want the robots.txt generator to appear: