🧰Locly

Generate a robots.txt File

Build User-agent, Disallow, Allow, and Sitemap rules as copyable text

About This Tool

robots.txt Generator helps you draft a robots.txt file with User-agent, Disallow, Allow, and Sitemap lines. Output is built locally in your browser so you can copy it to your site root.

How to Use

  1. Set a User-agent (often *)
  2. Add Disallow and Allow paths
  3. Optionally add Sitemap URLs
  4. Add more User-agent blocks if needed
  5. Copy the generated file

Common Use Cases

  • Block staging paths
  • Allow a public folder while disallowing /admin
  • Add sitemap references
  • Draft rules before deploy
  • Teach crawl directives

FAQ

Q1: What does User-agent mean?

It names which crawler the following rules apply to. * means all crawlers.

Q2: What does Disallow do?

It asks matching crawlers not to fetch that path. It is not a security lock.

Q3: What does Allow do?

It can reopen a path under a broader Disallow for crawlers that honor Allow.

Q4: What is Sitemap?

A URL pointing to your XML sitemap so crawlers can discover pages faster.

Q5: Is the file uploaded?

No. Generation is processed locally in your browser.

Related Tools

URL EncodeUTM Builder