Crawler rules with Sitemap and separate sections.
Free and ad-free. If it saved you time — support the project ♥
robots.txt is the first file a crawler reads: it saves crawl budget by closing utility sections and filters.
Part of the SEO & Analytics section: Everything a crawler sees: the snippet, structured data, crawl rules and the sitemap.
Together with XML Sitemap Generator, Meta Tag Generator and HTML to BBCode the whole job is covered: prepare, verify, publish.
robots.txt does not protect data: it is a hint. Real closing needs noindex or authentication.
All computation happens in the browser: text is never uploaded and never logged. No external libraries, so the page opens fast even on mobile data.
No. Use noindex, authentication or a password on the section.
Paste data or load a text file. Choose the mode and settings. Press Run or hit Ctrl+Enter. Copy the result or save it as a file.
Yes. All 100 tools are open with no subscriptions, limits or sign-up.
Processing runs in your browser: text and files are never uploaded. Network checks call public APIs only.
Yes. Code, markup, images and text belong to you with no restrictions.