ToolBrivo

Press slash to focus search.

SEO

Robots.txt generator

Write Allow, Disallow, and sitemap lines for one or more user agents.

robots.txt tells cooperating crawlers which paths they may fetch. It does not guarantee that a page will be indexed, and it does not hide a file from someone who already knows the address. This generator writes User-agent groups, Allow and Disallow lines, and Sitemap lines. It does not add rules for CSS, JavaScript, or images unless you enter those paths.

User-agent group 1

Add an Allow or Disallow path, or a sitemap URL.

How to use it

  1. Enter a user agent. Use * for every crawler that does not match a more specific group.
  2. Add Allow or Disallow paths that start with /. Add one group per user agent.
  3. Add absolute http or https sitemap URLs. Copy the file or download robots.txt.

Examples

A public site with one private path

User-agent * with Allow: / and Disallow: /private, plus Sitemap: https://example.com/sitemap.xml, is a complete file. A second group can disallow / for one named crawler.

Limits

  • A robots.txt rule controls crawling for agents that choose to obey it. It is not access control, and it does not protect a private file.
  • Blocking a page in robots.txt can prevent a crawler from seeing it. That is not the same as removing a URL from an index.
  • Paths must start with / and cannot contain spaces, #, or line breaks. Sitemap addresses must be absolute http or https URLs.
  • The file uses the common User-agent, Allow, Disallow, and Sitemap records. It does not emit a Content-Signal or other extension.

Privacy

The user agents, paths, and sitemap URLs stay in this browser. They are not sent to a server and they are not written to local storage.

Questions