A public site with one private path
User-agent * with Allow: / and Disallow: /private, plus Sitemap: https://example.com/sitemap.xml, is a complete file. A second group can disallow / for one named crawler.
Press slash to focus search.
Write Allow, Disallow, and sitemap lines for one or more user agents.
robots.txt tells cooperating crawlers which paths they may fetch. It does not guarantee that a page will be indexed, and it does not hide a file from someone who already knows the address. This generator writes User-agent groups, Allow and Disallow lines, and Sitemap lines. It does not add rules for CSS, JavaScript, or images unless you enter those paths.
Add an Allow or Disallow path, or a sitemap URL.
User-agent * with Allow: / and Disallow: /private, plus Sitemap: https://example.com/sitemap.xml, is a complete file. A second group can disallow / for one named crawler.
The user agents, paths, and sitemap URLs stay in this browser. They are not sent to a server and they are not written to local storage.