robots.txt is a file in your website's root that tells search engine crawlers which pages they can and cannot access. It's a key part of technical SEO.

Can I have multiple rules?

Yes. Add separate rules for different user-agents (e.g., Googlebot, Bingbot) with different allow/disallow paths.

Robots.txt Generator

Generate robots.txt files for your website. Control search engine crawling easily.

How to use: Configure crawl rules and generate a robots.txt file — copy or download.

Rule 1

Allow (one per line)

Disallow (one per line)

robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Robots.txt Generator

Generate properly formatted robots.txt files for your website. Configure user-agent rules, allow/disallow paths, sitemap URL, and crawl delay. Add multiple rule sets for different crawlers.

Frequently Asked Questions

What is robots.txt?: robots.txt is a file in your website's root that tells search engine crawlers which pages they can and cannot access. It's a key part of technical SEO.
Can I have multiple rules?: Yes. Add separate rules for different user-agents (e.g., Googlebot, Bingbot) with different allow/disallow paths.