cl0q-crawler

About the cl0q web crawler

cl0q-crawler is the indexing bot for cl0q.com, an open-web search engine.

How to identify it

Our crawler sends this User-Agent:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 cl0q-crawler/1.0 (+https://cl0q.com/bot)

The UA is browser-shaped for compatibility, but always carries the cl0q-crawler/1.0 token and a link back to this page — so it is honestly identifiable, not impersonating a person.

What it does (and doesn't)

• Fetches mainly your home page to read the title, description and visible text for our search index.
• Respects robots.txt and honors 429/503 Retry-After back-off.
• Paces itself per host to avoid hammering your server.
• Does not log in, submit forms, run your JavaScript, solve CAPTCHAs/challenges, or rotate IPs to hide. If we hit a bot-challenge page, we stop and move on.

Control or block it via robots.txt

Slow us down (one request per N seconds):

User-agent: cl0q-crawler
Crawl-delay: 10

Limit which paths we may fetch:

User-agent: cl0q-crawler
Disallow: /private/

Block us entirely:

User-agent: cl0q-crawler
Disallow: /

Changes take effect on our next visit.

Contact

Questions, removal requests, or want your site crawled/allow-listed? Reach us via the cl0q community on Discord, or through cl0q.com.