robotstxt.tools
robotstxt.tools is a live English-language website hosted on Amazon.com, Inc. (AS16509) in the United States, served by Vercel. cl0q first saw it on 2026-09-15 and last crawled it on 2026-09-15. It resolves to 76.76.21.21. Its favicon is fingerprinted by SHA-256, so you can find every other site that serves the same icon.
How the site describes itself:
Robots.txt Generator – Free Online Robots.txt Tool
Generate a valid robots.txt file for free. Add crawler rules, Allow and Disallow paths, sitemaps and presets, validate errors, then copy or download your robots.txt.
| Status | 200 OK — live & indexed by cl0q |
|---|---|
| First seen | 2026-09-15 |
| Last crawled | 2026-09-15 |
| IPv4 | 76.76.21.21 |
| Network | AS16509 Amazon.com, Inc. · California |
| Language | en |
| Server | Vercel |