Robots.txt Generator

Create a correct robots.txt file with crawler rules and sitemap reference.

robots.txt
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

Website Management Tools

About the Robots.txt Generator

The robots.txt Generator builds the small file that tells search engine crawlers which parts of your site they may visit. You choose whether to allow crawlers, list any paths to block, add an optional crawl delay and enter your sitemap address, and the tool writes the file.

A sensible robots.txt keeps crawlers out of areas such as admin folders and internal search pages, and points them to your sitemap. Save it as robots.txt in the main folder of your site, so that it opens at yoursite.com/robots.txt.

Take great care: one wrong line can block your entire site from search engines. robots.txt also does not hide or secure a page. A blocked page can still appear in Google if others link to it, so use a noindex tag to keep a page out of results, and a password for private files.

How to use this tool

  1. 1Choose whether to allow crawlers.
  2. 2List any paths to block, with one per line.
  3. 3Add your sitemap address.
  4. 4Save the output as robots.txt and upload it to your site's main folder.

Best-practice tips

  • Never block your CSS or JavaScript files, as Google needs them to read your pages.
  • Google ignores crawl delay, so it mainly matters for other crawlers.
  • Check the result in Google Search Console after uploading it.

Answers

Robots.txt Generator FAQs

Not reliably. It stops crawling, not indexing. Use a noindex tag on a page that Google can crawl.

Related tools

Continue with these related website management tools.