cl0q bots

About the cl0q web crawlers

cl0q.com is an independent search engine. Three small bots build its index. All of them identify themselves, link back to this page, and run only from 143.110.230.182 (DigitalOcean, AS14061). The same list is published as /bot/ips.json for firewall and WAF allow-lists.

cl0q-crawler: the indexer

Mozilla/5.0 (compatible; cl0q-crawler/1.0; +https://cl0q.com/bot)

Fetches your robots.txt and your home page to read its title, description and visible text for the search index. Each site is visited rarely (about once every few months).

cl0q-discovery: finds new sites

Mozilla/5.0 (compatible; cl0q-discovery/1.0; +https://cl0q.com/bot)

Follows a limited number of ordinary public links (at most 25 pages per site per pass) only to learn which other websites exist. It never follows links with query strings, and skips login, account, admin, cart and search pages.

cl0q-favicon: site icons

Mozilla/5.0 (compatible; cl0q-favicon/1.0; +https://cl0q.com/bot)

Fetches /favicon.ico, or the icon your home page declares, so search results can show your logo. Usually just two requests: robots.txt and the icon.

How they behave

• At most one request per second to any server (one every two seconds per site for cl0q-discovery), even when many sites share that server.
• Respect robots.txt, including Crawl-delay (up to 30 s). Rules for our own name take precedence over User-agent: *.
• Back off immediately on 403, 429 or 503, a dropped connection, or a timeout, and don't retry that site in the same pass.
• Never log in, submit forms, run your JavaScript, solve CAPTCHAs or bot challenges, or rotate IP addresses to hide.

Control or block them via robots.txt

User-agent: cl0q applies to all three bots; use cl0q-crawler, cl0q-discovery or cl0q-favicon to address one.

Slow us down (one request per N seconds):

User-agent: cl0q
Crawl-delay: 10

Keep us out of some paths:

User-agent: cl0q
Disallow: /private/

Block us entirely:

User-agent: cl0q
Disallow: /

Changes take effect on our next visit.

Open source

Our crawler is open source. The node software — the same crawler, packaged so anyone can run a community node — lives at codeberg.org/cl0qsearch/crawler-node. Read the code, check how it behaves, file an issue, or run your own node. (The search engine itself is going open source soon.)

Contact

Questions, removal requests, a report about our traffic, or want your site crawled or allow-listed? Email cl0qsearch@gmail.com (include the time and your server’s IP if you saw something odd), or reach us via the cl0q community on Discord.