Robots.txt Generator
Create a correct robots.txt file with crawler rules and sitemap reference.
User-agent: * Allow: / Disallow: /wp-admin/ Disallow: /cart/ Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml
Website Management Tools
About the Robots.txt Generator
The robots.txt Generator builds the small file that tells search engine crawlers which parts of your site they may visit. You choose whether to allow crawlers, list any paths to block, add an optional crawl delay and enter your sitemap address, and the tool writes the file.
A sensible robots.txt keeps crawlers out of areas such as admin folders and internal search pages, and points them to your sitemap. Save it as robots.txt in the main folder of your site, so that it opens at yoursite.com/robots.txt.
Take great care: one wrong line can block your entire site from search engines. robots.txt also does not hide or secure a page. A blocked page can still appear in Google if others link to it, so use a noindex tag to keep a page out of results, and a password for private files.
How to use this tool
- 1Choose whether to allow crawlers.
- 2List any paths to block, with one per line.
- 3Add your sitemap address.
- 4Save the output as robots.txt and upload it to your site's main folder.
Best-practice tips
- Never block your CSS or JavaScript files, as Google needs them to read your pages.
- Google ignores crawl delay, so it mainly matters for other crawlers.
- Check the result in Google Search Console after uploading it.
Answers
Robots.txt Generator FAQs
Not reliably. It stops crawling, not indexing. Use a noindex tag on a page that Google can crawl.
In the top folder of your website, so that it opens at yoursite.com/robots.txt.
It blocks crawlers from your whole site. Only use it on a site that you want kept out of search engines.
Related tools
Continue with these related website management tools.
Website SEO Score Checker
Get an instant SEO score out of 100 with a detailed breakdown of title, meta, headings, images, links, social tags and technical signals from the live page.
Website Ping Test
Ping any website: measure real response times from your browser and check whether the server is online, with its HTTP status and redirects.
Website Page Speed Checker
Measure real load timing for any page: response time, HTML size, code-to-text ratio, resource counts and Core Web Vitals-style recommendations.