cl0q Developer Documentation

cl0q crawls the open web, builds its own inverted index, and serves it from a small read-only HTTP API. It is not a front-end for anybody else's search results. This page documents the public API, the query language, how the index is built, and how cl0q's crawler behaves if it visits your site.

Privacy

cl0q is built so that there is no search history to leak, subpoena, sell, or breach. The guarantees below are architectural — they follow from how the system is built, not from a promise to behave.

What we do keep

cl0q keeps no per-request access log. For a normal request — including every search — no IP address, requested URL, query string, or user agent is written to disk. There is no per-request record to analyse, profile, share, or sell, because none is kept.

The only things logged are application errors and operational events (a failed or slow database query, the server starting or stopping), for a short retention window used strictly for uptime monitoring and debugging. These entries contain no IP address and no search terms — a rejected query, for instance, records only its length, never its content.

Things you explicitly opt into

Two features necessarily store data, because they cannot work otherwise. Both are opt-in and neither is required to use cl0q:

• Saved-search alerts (/api/watch) store the query you asked to be alerted about, plus the browser push endpoint to notify. That is the entire point of the feature. Deleting the subscription removes it.
• API key signup (/api/signup) optionally accepts an email address so we can contact you about the key. It is optional — leave it blank and no contact information is stored at all. The key itself is always stored hashed.

See also the privacy policy.

Quick start

Search is public and needs no key:

curl 'https://cl0q.com/search?q=open+source+search&limit=5'

For the API-key-gated endpoints, mint a free key and send it as a header:

curl -X POST https://cl0q.com/api/signup
# -> {"api_key":"cl0q_...","tier":"free","per_min":30,"per_day":1000, ...}

curl -H 'X-API-Key: cl0q_...' 'https://cl0q.com/api/favicon?domain=github.com'

Search API

GET /search no key required

Parameters

Example

curl 'https://cl0q.com/search?q=intitle:handbook+rust&limit=10&apex_only=1&lang=en'

Response

JSON with a result array and a real total (not an estimate), so you can paginate honestly. Each result carries the indexed page metadata: domain, page title, h1, description, a match-highlighted snippet, HTTP status, and the crawl timestamp. include_org=1 adds network/ASN and DNS fields; include_favicon=1 adds icon bytes.

Query operators

Operators are parsed out of q; whatever remains is full-text matched. Prefix any operator with - to negate it. Operators can be combined.

Notes and limits

• org: requires at least 3 characters. Shorter values are rejected with a 400 rather than silently running an unindexed scan — a deliberate choice to keep one query from degrading the service for everyone.
• The infrastructure operators (ip:, cname:, asn:, org:, favicon:) are feature-detected at startup. If one isn't available on this instance it returns a clear 400 saying so, instead of pretending to work.
• A query consisting only of operators is valid; free text is optional.

Suggest API

GET /suggest no key required

Built for typeahead: it's tuned for latency and rate-limited separately from /search. cl0q also ships an OpenSearch descriptor, so browsers can add cl0q as a search engine and use this endpoint for suggestions.

Favicon-hash API

GET /api/favicon API key required

Favicons are content-addressed by SHA-256, which makes them a pivot: sites sharing an icon are usually related — same operator, same framework default, same phishing kit, same rebrand. This is the same class of pivot Shodan/FOFA-style tooling uses, over cl0q's open-web index.

# forward: domain -> hash + who else uses it
curl -H 'X-API-Key: cl0q_...' 'https://cl0q.com/api/favicon?domain=github.com'

# reverse: hash -> domains
curl -H 'X-API-Key: cl0q_...' 'https://cl0q.com/api/favicon?hash=<hex-sha256>&limit=50'

Pass exactly one of domain or hash. A malformed hash returns 400.

Subdomains API

GET /api/subdomains API key required

Returns subdomains cl0q discovered by crawling the open web. This is passive — it reflects what was found by following public links and DNS, and is not a live brute-force of your zone.

API keys & rate limits

Getting a key

curl -X POST https://cl0q.com/api/signup
curl -X POST https://cl0q.com/api/signup -d 'email=you@example.com'   # optional

Returns the key once, in the response body. It is shown only once — cl0q stores only a hash and cannot show it to you again. The email is optional and used solely to contact you about the key; omit it and nothing identifying is stored.

Sending a key

Header X-API-Key: cl0q_... (preferred) or query param ?key=cl0q_....

Limits

Paid tiers raise per_min/per_day on the same key. During beta, everyone gets a free key immediately.

Other endpoints

How the index is built

Four stages, each feeding the next. Every stage is incremental and restartable — nothing depends on a full rebuild. The important thing to notice is what crosses each boundary: a layer only ever hands the next one a small, well-defined artefact.

1. Discovery

A breadth-first crawler walks outward from a curated seed list of link-rich, bot-friendly pages, recording every new registrable domain it sees. Because a fixed seed list plateaus quickly — it re-walks the same neighbourhood and stops finding anything — discovery also continuously re-samples domains already known to the index and mines their homepages for outbound links. The known web becomes a rotating frontier rather than a finished list.

2. Page crawl

Discovered domains are fetched and parsed into the index: title, h1, meta description, keywords, canonical, Open Graph and Twitter card metadata, charset, detected language, robots directives, outbound link count, image count, word count, HTTP status, response latency, redirect count, and server header.

Two details matter for quality. Content-type gating means only text/html is parsed, so binary responses never become garbage "pages". And anti-bot interstitial detection means when a challenge page is served instead of real content, cl0q recognises it, refuses to index the challenge text, and records why — rather than indexing thousands of identical "Checking your browser…" pages.

Most fetches do not end up in the index. Every gate below exists to keep something out:

3. Enrichment

Separate passes resolve DNS (A/AAAA/MX/CNAME) for domains discovered link-only, and map IPs to their network/ASN and country. These feed the ip:, cname:, asn: and org: operators and the region facets. Favicons are fetched and stored content-addressed by SHA-256, so an icon shared by 100,000 sites is stored once — which is also what makes the favicon pivot cheap.

4. Serving

A read-only API serves the index. Nothing in the read path writes to the crawl tables, so search load and crawl load stay independent.

Federation

cl0q can accept crawl results from trusted independent nodes, so contributors can crawl on their own hardware and contribute back without ever receiving database credentials. Submissions are authenticated, re-validated server-side against central policy, and tagged with their source so any node's contribution can be audited or removed wholesale. Interested in running one? Use /api/feedback.

Ranking & matching

Matching is PostgreSQL full-text search over a weighted vector, not a bag of words. Fields are weighted by trustworthiness:

Ranking then layers on an apex boost (an apex domain outranks a subdomain, all else equal) and a network popularity signal derived from how much of the index sits on a given network. Substring operators (site:, intitle:, org:) are trigram-indexed, which is why org: needs 3 characters — a shorter pattern has no trigram to look up.

Search runs in two stages: a candidate-and-rank pass, then a single batched detail fetch for the page you asked for. Hot queries are briefly cached. Result total is a true count, not an estimate.

For site owners: the cl0q crawler

If cl0q's crawler visits you, here's exactly what it does and how to control it.

Identifying it

The crawler sends a user agent containing cl0q-crawler and a link to cl0q.com/bot. It is browser-shaped so that naive filters don't reject it, but it always self-identifies and is always contactable — it will never pretend to be an ordinary browser to evade you.

Controlling it

• robots.txt is respected. Disallow the crawler and it stops.
• Rate limits are per-host and per-IP. Concurrency against any single site or address is capped, and one busy network can't monopolise a crawl batch.
• 429 and 503 are honoured, including Retry-After. Throttling is treated as "come back later", not as a dead page.
• Repeated failures back off automatically via a per-group circuit breaker, so a struggling host isn't hammered.
• Removal: use /api/report to request removal of a domain.

Being crawled efficiently

The crawler sends If-None-Match / If-Modified-Since when it has previously stored an ETag or Last-Modified for you. Serve validators and a re-crawl costs you a 304 and a few bytes instead of a full page render. cl0q fetches your homepage — it does not spider your whole site.

Errors