Robots.txt Generator

robots.txt is a plain text file at the root of a site that tells crawlers which paths they may fetch. It is a request, not a lock: well-behaved crawlers such as Googlebot, Bingbot and our own EchoRankBot honour it; a page you need to keep private needs authentication, not a Disallow line. The EchoPilot audit reads robots.txt first on every run and reports, under rule S5, whether the file exists and whether it blocks the whole site from search engines by mistake.

This generator builds the file from groups. Each group starts with one or more User-agent lines, followed by the Allow and Disallow rules for those agents and an optional Crawl-delay. A single group with User-agent * covers every crawler; add a second group when one crawler needs different rules, for example to keep an AI crawler out of a section while search engines may read it. The Sitemap line is independent of groups and takes the full URL of your sitemap.xml. The preview updates as you type, and the Copy and Download buttons give you the file to upload as /robots.txt.

Two habits keep the file safe. Do not disallow / for User-agent * unless the site is meant to be invisible to search engines; the audit flags that as a fail. And do not list private paths hoping to hide them: the file is public, and a Disallow line is a pointer, not a shield. Once uploaded, run the free score on your domain; the S4 and S5 rules read the live file and the sitemap it names.