API Documentation

Search, read, map and crawl the web from one API. Every example below is a request that works against the live API — most of them without a key. Try any of them in the playground.

# Overview

One API for four jobs: search the web, read one page, list a site's addresses, and crawl a site. Search, extract, map and answer run without an API key at a lower rate limit; crawl and batch need one. Every response carries X-Request-Id, and X-RateLimit-Limit / Remaining / Reset report where you stand.

Example

Base URL   http://searchx.dev
Auth       Authorization: Bearer sk-sx-YOUR_KEY   (optional on most endpoints)

Read one page
  GET  /api/v1/extract        one URL, any combination of output formats
  POST /api/v1/extract        the same, with schemas and scripted fetches

Understand a site
  GET  /api/v1/map            a site's URLs, from our index and its sitemap
  POST /api/v1/crawl          crawl a site as a job          (API key)
  POST /api/v1/batch          scrape a list of URLs as a job (API key)
  GET  /api/v1/crawl/{id}     job state and one page of documents
  DELETE /api/v1/crawl/{id}   stop a job                     (API key)

Search
  GET  /api/v1/search         hybrid keyword + semantic
  GET  /api/v1/answer         a synthesized answer with sources
  GET  /api/v1/images/search  image search
  GET  /api/v1/suggest        autocomplete                   (API key)
  GET  /api/v1/web_lookup     search, fetch and find, in one call

Protocols
  REST        every endpoint above
  MCP         https://mcp.searchx.dev/mcp
  gRPC-Web    /searchx.SearchService/

# Playground

Send a real request from this page and read the real answer, headers included. Endpoints that run anonymously need nothing from you. Crawl and batch need a key — paste your own, it is used for that request and nothing else.

Fetch one page and return every requested shape of it in a single call.

Runs without a key. A key raises the rate limit and adds a daily quota.

url

formats

Every format you tick becomes a key in the response — null with a reason in warnings when it could not be produced.

only_main_content

Default true. false keeps navigation, sidebars and footers.

The key stays in this tab: it is sent to the API with your request and is never stored.

curl -X POST "http://searchx.dev/api/v1/extract" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown","links"]}'

Pick an endpoint, fill it in, press Send.

# Authentication

Pass an API key as a Bearer token. Most read endpoints accept requests without one and answer at the anonymous rate limit, so you can try everything before you sign up. Jobs that spend real crawling capacity — crawl, batch, cancel — and the autocomplete endpoint require a key.

Example

# Anonymous: works, lower limits
curl "http://searchx.dev/api/v1/search?q=kubernetes&per_page=3"

# With a key
export SEARCHX_API_KEY="sk-sx-..."
curl "http://searchx.dev/api/v1/search?q=kubernetes&per_page=3" \
  -H "Authorization: Bearer $SEARCHX_API_KEY"

# Without a key, an endpoint that needs one answers 401:
curl -X POST "http://searchx.dev/api/v1/crawl" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","limit":2}'

{"error":"missing_token","message":"An API key is required for this endpoint. Create one at http://searchx.dev/dashboard"}

# Errors and warnings

Two different things can go wrong, and they are reported differently. A request that failed returns a typed error envelope with a real HTTP status. A request that succeeded but could not honour part of what you asked returns 200 with the part missing and a warnings entry naming it — a format is never dropped silently, and a fetch option we could not apply is never pretended.

Name	Type	Default	Description
`invalid_request`	400	—	A parameter is missing, malformed or out of range.
`url_blocked`	403	—	The address is not one this API will fetch.
`robots_denied`	403	—	The site's robots.txt disallows it.
`blocked_by_site`	403	—	The site refused us.
`not_found`	404	—	The page, or the job id, is not there.
`unsupported_content_type`	415	—	The response is not something this endpoint can extract, or the converter could not read the document it was given.
`converter_unavailable`	503	—	The document format is supported but the converter is not running. This is our side, not the site — retry later.
`internal`	500	—	Our fault.
`upstream_unavailable`	502	—	A service this request depends on did not answer.
`fetch_timeout`	504	—	The page did not finish inside the budget.

Example

curl -G "http://searchx.dev/api/v1/extract" --data-urlencode "url=ftp://example.com"
# HTTP 400
{
  "error": {
    "code": "invalid_request",
    "message": "only http and https URLs can be extracted",
    "url": "ftp://example.com"
  }
}

# 'message' is a fixed, caller-safe sentence per code — upstream error text is
# never echoed back. 'http_status' carries what the target site answered, when known.

# A warning is not an error. This request succeeded:
curl -G "http://searchx.dev/api/v1/extract" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "formats=markdown,summary"
# HTTP 200
{
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
  "summary": null,
  "warnings": [
    "summary: the language model is unavailable right now"
  ]
}

# A warning names its subject: a format, or "fetch" or "answer" for the
# optional passes those endpoints run.

# Extract a page

Fetch one URL and get it back clean. Static-friendly pages answer in a few hundred milliseconds; JS-gated and bot-protected pages fall back to a headless browser automatically. GET takes the scalar parameters; POST takes the same plus everything structured — schemas, headers, cookies, browser actions.

Name	Type	Default	Description
`url`	string	—	Required. A bare host is read as https.
`format`	rich \| markdown \| text	rich	The original single-shape response. Ignored when formats is given.
`formats`	string[]	—	The multi-format response. See below.
`max_length`	int	50000	Character cap on returned content. 0 returns the whole document.
`refresh`	bool	false	Bypass the cache.
`llm`	bool	false	Refine the markdown with a language model.

Example

curl -G "http://searchx.dev/api/v1/extract" \
  --data-urlencode "url=https://example.com"

{
  "url": "https://example.com",
  "title": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
  "word_count": 28,
  "engine": "static",
  "fetched_at": "2026-08-17T22:30:42Z",
  "took_ms": 118,
  "fetch": { "engine": "static", "browser_used": false, "proxy_tier": "direct", "proxy_used": "direct" }
}

# Many formats, one fetch

Ask for several shapes of the same page and pay for one fetch. Every format you request is a key in the response — and it stays a key even when it could not be produced, set to null with the reason in warnings. Nothing you asked for ever disappears without being accounted for, and nothing you did not ask for is returned.

Name	Type	Default	Description
`markdown`	string	—	Cleaned main content as markdown.
`html`	string	—	Cleaned main-content HTML, scoped the same way the markdown is.
`raw_html`	string	—	The page's HTML as delivered.
`text`	string	—	Plain text.
`links`	string[]	—	Links inside the extracted subtree. A page with none returns [], not null.
`images`	string[]	—	Images inside the extracted subtree.
`screenshot`	string	—	PNG as a data: URI. { "type": "screenshot", "full_page": true } captures the whole scroll.
`json`	object	—	Schema-validated structured data. Needs a schema or a prompt.
`summary`	string	—	A short summary of the page.
`tables`	object[]	—	Every table on the page as data: headers and rows, parsed from the same tables the markdown shows. Empty layout columns and rows are dropped; cells keep their links. The markdown itself still renders tables as tables.
`change_tracking`	object	—	How the page compares to the copy stored under your tag.

Example

# GET form: comma-separated names
curl -G "http://searchx.dev/api/v1/extract" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "formats=markdown,links,html,summary"

{
  "url": "https://example.com",
  "title": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
  "links": ["https://iana.org/domains/example"],
  "html": "<div><h1>Example Domain</h1>...",
  "summary": null,
  "fetch": { "engine": "static", "browser_used": false, "proxy_tier": "direct", "proxy_used": "direct" },
  "warnings": [
    "summary: the language model is unavailable right now"
  ]
}

# summary is null and says why. It is not missing, and the request is still a 200.

# POST form: a format can be an object carrying its own options
curl -X POST "http://searchx.dev/api/v1/extract" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": [
      "markdown",
      { "type": "screenshot", "full_page": true }
    ]
  }'

# Scoping and freshness

Say which part of the document you mean before it is converted, so markdown, text, html, links and images all describe the same subtree. Any scoping parameter forces a live fetch rather than a stored crawl copy. max_age is a freshness bound of your own: serve a cached extraction only while it is younger than this.

Name	Type	Default	Description
`only_main_content`	bool	true	Keep only the detected main content. false keeps navigation, sidebars and footers.
`include_tags`	string[]	—	CSS selectors; only these subtrees are extracted. Comma-separated on GET, an array on POST.
`exclude_tags`	string[]	—	CSS selectors dropped from the document.
`max_age`	int	—	Seconds. 0 always re-fetches. Omitted means any cached copy is fine.

Example

curl -G "http://searchx.dev/api/v1/extract" \
  --data-urlencode "url=https://vuejs.org/guide/introduction" \
  --data-urlencode "formats=markdown" \
  --data-urlencode "include_tags=h1"

{ "engine": "scoped-static", "markdown": "Introduction", "warnings": [] }

# Drop the furniture instead of naming what to keep:
curl -X POST "http://searchx.dev/api/v1/extract" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "formats": ["markdown", "links"],
    "exclude_tags": [".related", "#comments", "nav"],
    "max_age": 3600
  }'

# Schema extraction

Hand the json format a JSON Schema and get validated structured data instead of prose. The schema is normalized before use: every property becomes required and additional properties are refused, so a partial answer fails rather than passing quietly. A prompt can stand in for a schema, or sharpen one. When the model cannot produce a conforming answer the key is null and warnings says so — you are never handed a guess.

Example

curl -X POST "http://searchx.dev/api/v1/extract" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product",
    "formats": [
      "markdown",
      {
        "type": "json",
        "schema": {
          "type": "object",
          "properties": {
            "name":  { "type": "string" },
            "price": { "type": "number" },
            "in_stock": { "type": "boolean" }
          }
        },
        "prompt": "Extract the product name, its price as a number, and whether it is in stock."
      }
    ]
  }'

{
  "markdown": "...",
  "json": { "name": "...", "price": 49, "in_stock": true },
  "warnings": []
}

# When the model is unreachable or its answer does not fit the schema:
{ "json": null, "warnings": ["json: the extraction model did not answer"] }

# Change tracking

Ask what changed since you last looked. The change_tracking format compares the page against the copy stored under your tag and answers new, same, changed or removed — with an optional git-style diff and an optional per-field diff of an extracted schema. Detection is a normalized content hash, whitespace- and case-insensitive but order-significant, so a page whose sections were merely reordered is reported as changed and flagged reordered_only rather than hidden. A page that now 404s is answered 200 with removed when a baseline exists, and stays a 404 when none does.

Name	Type	Default	Description
`modes`	git-diff \| json	—	Reports on top of the status. git-diff adds the unified diff and its hunks; json adds the per-field before/after and requires a schema. Omitted means status only.
`tag`	string	—	Which of this URL's independent baselines to compare against. Letters, digits and - _ . : only, 64 characters at most.
`retain`	bool	true	false compares without keeping a copy. A caller that never retains can only ever be answered new.

Example

curl -X POST "http://searchx.dev/api/v1/extract" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": [
      "markdown",
      { "type": "change_tracking", "modes": ["git-diff"], "tag": "homepage-watch" }
    ]
  }'

{
  "change_tracking": {
    "status": "new",
    "tag": "homepage-watch",
    "modes": ["git-diff"],
    "version": 1,
    "content_hash": "3799ffb91992507b3b4fbfe9188e2822c52f0c5245dea017342a2e9e57b25784",
    "previous_scraped_at": null,
    "retained": true,
    "diff": {
      "text": "--- previous\n+++ current\n@@ -0,0 +1,5 @@\n+# Example Domain\n...",
      "hunks": [{ "header": "@@ -0,0 +1,5 @@", "old_start": 0, "new_start": 1, "lines": [...] }]
    },
    "stats": { "previous_lines": 0, "current_lines": 5, "added_lines": 5, "removed_lines": 0 },
    "current_scraped_at": "2026-08-17T22:31:43Z"
  },
  "warnings": []
}

# Run it again later and status becomes same or changed, with previous_scraped_at
# and previous_first_seen_at telling you how long the page had been stable.

# Watch one field rather than the whole page:
{
  "formats": [{
    "type": "change_tracking",
    "modes": ["json"],
    "tag": "price-watch",
    "schema": { "type": "object", "properties": { "price": { "type": "number" } } }
  }]
}
# -> change_tracking.json.changed_fields: ["price"], with previous and current values.

# Fetch control, and an honest report

These say how the page is fetched rather than what is extracted from it. All are optional; a request that sets none of them fetches exactly as it always did. Setting any of them makes the request one specific fetch, so it is neither served from nor written to the shared cache. Every response carries a fetch object saying which engine ran, whether a browser was spent, which egress tier actually applied and where the request left from — and anything we could not honour is a warning rather than a silent omission.

Name	Type	Default	Description
`wait_for`	string	—	CSS selector to wait for. Only the browser engine can honour it, and it does not by itself force one — if the static engine answers first, fetch.warnings says wait_for was not applied.
`timeout_ms`	int	25000	Budget for the whole fetch, actions included. 1000 to 30000.
`mobile`	bool	false	Emulate a phone: viewport, touch and user agent together.
`country`	string	—	ISO 3166-1 alpha-2. Leave through an egress proxy in that country.
`proxy`	auto \| basic \| stealth	auto	Egress tier. stealth also renders every page in the evasion browser.
`headers`	object	—	POST only. At most 32. Connection headers are refused and reported.
`cookies`	object[]	—	POST only. At most 64. Sent on the static fetch and injected into the browser alike. domain defaults to the host.
`actions`	object[]	—	POST only. At most 20 browser steps driven after the page settles. Asking for any forces the browser engine.

Example

# Ask for an egress tier this deployment cannot serve:
curl -G "http://searchx.dev/api/v1/extract" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "formats=markdown" \
  --data-urlencode "proxy=basic" --data-urlencode "country=JP"

{
  "fetch": {
    "engine": "static",
    "browser_used": false,
    "proxy_tier": "direct",
    "proxy_used": "direct",
    "warnings": [
      "no egress proxy is configured for JP; the request left from the server's own address",
      "the basic proxy tier needs a configured egress proxy; the request left from the server's own address"
    ]
  },
  "warnings": [
    "fetch: no egress proxy is configured for JP; the request left from the server's own address",
    "fetch: the basic proxy tier needs a configured egress proxy; the request left from the server's own address"
  ]
}

# The request still succeeded. It simply told you what it actually did.

# Scripted interaction, then read the page it produced:
curl -X POST "http://searchx.dev/api/v1/extract" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/search",
    "formats": ["markdown", "screenshot"],
    "wait_for": "#results",
    "timeout_ms": 20000,
    "mobile": true,
    "headers": { "Accept-Language": "de-DE" },
    "cookies": [{ "name": "consent", "value": "accepted" }],
    "actions": [
      { "type": "fill", "selector": "#q", "value": "hydraulic press" },
      { "type": "press", "value": "enter" },
      { "type": "wait", "timeout": 2000 },
      { "type": "screenshot" }
    ]
  }'

# Action types: click, click_xy, reload, fill, type, input, write, select, wait,
# scroll, screenshot, press, execute_js, evaluate.
# A step this deployment cannot perform is skipped and reported in warnings.

# Documents and images

A document URL is extracted as a document rather than a page. PDFs go through their text layer where there is one and OCR where there is not. Office and OpenDocument formats — .docx, .xlsx, .pptx, .odt, .ods, .odp, .rtf, .epub, .csv and the legacy .doc/.xls/.ppt — are converted the same way, and content_type names the format that was actually read. The format is resolved from the response rather than the URL, so a download link with no extension still converts. Images go through EXIF, OCR and a vision model. Anything else is answered 415 unsupported_content_type — including a document served as application/octet-stream from a path with no extension, where nothing in the response identifies it as one.

Example

curl -G "http://searchx.dev/api/v1/extract" \
  --data-urlencode "url=https://arxiv.org/pdf/1706.03762" \
  --data-urlencode "formats=markdown" \
  --data-urlencode "max_length=400"

{
  "content_type": "pdf",
  "engine": "pdf",
  "markdown": "#### Provided proper attribution is provided, Google hereby grants permission to...",
  "word_count": 56,
  "full_length": 40820,
  "truncated": true,
  "took_ms": 207,
  "fetch": { "engine": "pdf", "browser_used": false }
}

# full_length and truncated appear whenever max_length cut the document.
# Send max_length=0 for the whole thing.

# A live office document, converted the same way:
curl -G "http://searchx.dev/api/v1/extract" --data-urlencode "url=https://example.com/report.docx"
{
  "content_type": "docx",
  "title": "Quarterly Report",
  "markdown": "# Quarterly Report\n\nRevenue rose across every region...",
  "word_count": 1840
}

# Extract a document you upload

When you hold the file rather than a link to it, POST it to /api/v1/extract/file as multipart/form-data. The same converters run, so an upload and a URL to the same document return the same shape — the only difference is where the bytes came from. Up to 32MB; scanned pages are read with OCR unless you send ocr=false. The file itself is never stored. It is hashed, and only that hash and the extracted text are kept, so uploading the same document again answers from the earlier extraction instead of converting it twice — the response says so with cached: true, and content_hash identifies the document. Uploads are kept apart from crawled content: nothing you upload is indexed, made searchable, or mixed into the crawl store.

Example

curl -F "file=@report.pdf" \
  -F "max_length=0" \
  "http://searchx.dev/api/v1/extract/file"

{
  "title": "Quarterly Report",
  "content_type": "document",
  "page_type": "pdf",
  "markdown": "# Quarterly Report\n\nRevenue rose across every region...",
  "word_count": 1840,
  "content_hash": "9f2b...c41e",
  "engine": "pdf-inspector"
}

# The same file again — answered from the previous extraction:
{ "content_hash": "9f2b...c41e", "cached": true, "markdown": "# Quarterly Report..." }

# Options match the URL endpoint: llm=true to clean the text with the model,
# ocr=false to skip OCR, max_length=0 for the whole document.
# find=... searches inside the document: only the matching chunks come back,
# ranked best-first — also on GET /api/v1/extract/file/{hash}?find=..., so a
# document sent earlier is searched by hash without resending it.

# Reading a long document without holding it all at once: send it once with a
# chunk_size, then page by content_hash — the file is never sent again.
curl -F "file=@book.pdf" -F "chunk_size=6000" "http://searchx.dev/api/v1/extract/file"
{
  "content_hash": "9f2b...c41e",
  "total_chunks": 84,
  "chunk_index": -1,
  "next_chunk_index": 0,
  "chunks": [{ "index": 0, "title": "Chapter I", "word_count": 940 }, ...]
}

curl "http://searchx.dev/api/v1/extract/file/9f2b...c41e?chunk_index=0&chunk_size=6000"
{ "chunk_index": 0, "has_more": true, "next_chunk_index": 1,
  "chunk": { "title": "Chapter I", "markdown": "..." } }

# Chunks break on headings, and a table that spans a boundary keeps its header
# row on the far side. Without chunk_size the whole document comes back as before.

# Map a site

List a site's URLs without crawling it. Two sources are merged in parallel: the pages we have already crawled, which carry a title and description, and the site's own sitemaps, discovered from robots.txt or the conventional locations and followed through sitemap indexes and .gz files. URLs are canonicalised and deduplicated across both, and sources tells you who contributed what. The whole call is bounded by timeout_ms — a slow sitemap does not hold up the answer.

Name	Type	Default	Description
`url`	string	—	Required. The site to map.
`search`	string	—	Ranks the returned URLs by how often the words appear in the address itself. Lexical only — nothing is filtered out.
`limit`	int	1000	Up to 5000.
`include_subdomains`	bool	true	A site's pages are routinely on one.
`ignore_sitemap`	bool	false	Answer from the crawled index alone, without fetching the site.
`timeout_ms`	int	10000	Up to 30000.

Example

curl -G "http://searchx.dev/api/v1/map" \
  --data-urlencode "url=https://vuejs.org" \
  --data-urlencode "limit=5"

{
  "urls": [
    { "url": "https://vuejs.org/guide/quick-start", "title": "Quick Start | Vue.js", "description": "Vue.js - The Progressive JavaScript Framework" },
    { "url": "https://vuejs.org/api/ssr.html", "title": "Server-Side Rendering API | Vue.js", "description": "..." }
  ],
  "total": 5,
  "took_ms": 240,
  "sources": { "index": 5, "sitemap": 0 }
}

# Crawl a site

Crawling is a job. POST answers immediately with an id, the crawl runs against a global concurrency bound so it cannot starve live search, and you poll for documents. It seeds itself from the sitemap, follows links inside the scope you set, obeys robots.txt including Crawl-delay, and paces itself per host. Pass "async": false for the original single-response behaviour, bounded by a server-side deadline. Requires an API key.

Name	Type	Default	Description
`url`	string	—	Required. Where the crawl starts.
`limit`	int	100	Pages to fetch, failures included. Operator ceiling, 5000 by default. max_pages is accepted as the original name.
`max_discovery_depth`	int	3	Links from a seed the crawl may follow. Seeds are 0. max_depth is the original name.
`include_paths`	string[]	—	Go regexps against the URL path. Unanchored — blog matches /en/blog/x; use ^ and $ to anchor. The URLs you named yourself are always crawled.
`exclude_paths`	string[]	—	Same matching, and it wins over include_paths.
`sitemap`	include \| skip \| only	include	include seeds from the sitemap and still follows links; only fetches exactly what the sitemap lists; skip uses your URL alone.
`allow_subdomains`	bool	false
`allow_external_links`	bool	false	Follow links off the site. Still bounded by limit.
`crawl_entire_domain`	bool	true	false keeps the crawl inside the seed URL's directory.
`ignore_query_parameters`	bool	false	Drop the query string, so ?page=1 and ?page=2 are one page.
`deduplicate_similar_urls`	bool	true	Treat addresses differing only by scheme, www, case, a trailing slash or an index file as one page.
`ignore_robots_txt`	bool	false	robots.txt is obeyed by default, Crawl-delay included.
`max_concurrency`	int	4	Pages fetched at once by this crawl.
`delay_ms`	int	—	Minimum spacing between requests to one host. A larger Crawl-delay in robots.txt wins.
`webhook`	string \| object	—	Where to report progress. A bare string is read as the url.
`async`	bool	true	false runs the crawl inline and returns every page in one response.

Example

curl -X POST "http://searchx.dev/api/v1/crawl" \
  -H "Authorization: Bearer $SEARCHX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "limit": 500,
    "max_discovery_depth": 4,
    "include_paths": ["^/docs/"],
    "exclude_paths": ["\\.pdf$"],
    "sitemap": "include",
    "webhook": {
      "url": "https://hooks.example.com/searchx",
      "secret": "whsec_...",
      "events": ["crawl.page", "crawl.completed"]
    }
  }'

{
  "id": "crawl_9f3c1a7b2e4d5068",
  "job_id": "crawl_9f3c1a7b2e4d5068",
  "url": "https://docs.example.com",
  "status": "scraping",
  "status_url": "/api/v1/crawl/crawl_9f3c1a7b2e4d5068"
}

# A webhook with a secret is signed: X-SearchX-Signature: sha256=<hex HMAC of the raw body>.
# Events: crawl.started, crawl.page, crawl.completed, crawl.failed.

# Crawl diff mode

Re-crawl a site you have crawled before and be told only what moved. Every page carries the same change_tracking report /extract returns, the job carries a change tally of new, changed, unchanged and removed, and only_changed drops the pages that did not move from the result while the tally still counts every page the crawl saw — so an empty result is distinguishable from a crawl that fetched nothing. Omit change_tag and the crawl derives its identity from what decides which pages it can reach: the seed, the path filters, the subdomain and external-link switches, crawl_entire_domain, the URL-normalisation switches, sitemap and depth. The budgets are deliberately left out, so raising a limit keeps the history rather than starting a new one. Narrowing the scope starts a separate history instead of reporting every excluded page as removed.

Name	Type	Default	Description
`track_changes`	bool	false	Compare against the last crawl of this tag and store this one as the new baseline. Any other change parameter turns it on by itself.
`change_tag`	string	—	Which baseline to measure against. Pass your own to keep one history across a scope change, or to have two crawls share a baseline.
`change_modes`	git-diff \| json	—	git-diff is the unified diff of the page's markdown. json diffs the values of change_schema and costs a model call per page. Omitting both is the cheapest crawl: an unchanged page is recognised from what the last crawl recorded, without reading the stored copies back.
`change_schema`	object	—	JSON Schema whose values the json mode compares. That mode requires it.
`change_prompt`	string	—	Instruction that goes with change_schema.
`only_changed`	bool	false	Return only pages whose status is not same.

Example

curl -X POST "http://searchx.dev/api/v1/crawl" \
  -H "Authorization: Bearer $SEARCHX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "limit": 500,
    "track_changes": true,
    "change_modes": ["git-diff"],
    "only_changed": true
  }'

# Then poll the job:
{
  "status": "completed",
  "total": 3,
  "change": {
    "tag": "docs.example.com:default",
    "pages_crawled": 482,
    "new": 1,
    "changed": 2,
    "unchanged": 479,
    "removed": 0,
    "only_changed": true,
    "previous_crawl_at": "2026-08-16T04:00:11Z"
  },
  "data": [
    {
      "url": "https://docs.example.com/guide/auth",
      "markdown": "...",
      "change_tracking": { "status": "changed", "diff": { "text": "..." }, "stats": { "added_lines": 12, "removed_lines": 3 } }
    }
  ]
}

# total is 3 while pages_crawled is 482: the tally counts what the crawl saw,
# the result carries what moved.
#
# removed counts pages the previous crawl of this tag reached that this one could
# not, plus pages it still links that now answer 404. A crawl that was cancelled,
# ran out of time or hit its page budget reports none — pages it never reached
# cannot be told apart from deleted ones.

# Near-duplicate collapsing

A site that renders the same article under several paths, or paginates the same list, otherwise hands you the same text several times and charges you tokens for each copy. Turn on collapse_near_duplicates and a page that repeats one already in this crawl's results is folded into it. The job reports collapsed, a count, and duplicates, naming what each collapsed page was folded into and how far apart the two were — because a count alone would leave you unable to tell a deduplicated crawl from one that simply found less.

Name	Type	Default	Description
`collapse_near_duplicates`	bool	false	Fold near-copies into the page they repeat. The first page of a group is always the one kept.
`collapse_distance`	int	4	Hamming radius over a 64-bit content fingerprint, clamped to 8. Setting it implies the option. A page too thin to fingerprint is never a duplicate.

Example

curl -X POST "http://searchx.dev/api/v1/crawl" \
  -H "Authorization: Bearer $SEARCHX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://shop.example.com", "limit": 300, "collapse_near_duplicates": true }'

# In the job status:
{
  "total": 214,
  "collapsed": 86,
  "duplicates": [
    { "url": "https://shop.example.com/p/42?ref=nav", "collapsed_into": "https://shop.example.com/p/42", "distance": 0 },
    { "url": "https://shop.example.com/list?page=3", "collapsed_into": "https://shop.example.com/list?page=2", "distance": 3 }
  ],
  "data": [...]
}

# Job status, paging and cancelling

Poll GET /api/v1/crawl/{id} — the same endpoint serves batches at /api/v1/batch/{id} — and follow next until it is absent. The response body has a hard size cap, so a page of large documents can stop before your limit; next then points at the document it stopped on, which means nothing is skipped and no response is unbounded. A page that could not be fetched is still a document and carries error instead of content. DELETE stops a running job and returns its final state; the documents it already produced stay readable until the job expires.

Name	Type	Default	Description
`skip`	int	0	Cursor: how many documents to pass over.
`limit`	int	50	Documents per page, before the size cap applies. Up to 500.

Example

curl "http://searchx.dev/api/v1/crawl/crawl_9f3c1a7b2e4d5068?limit=20"

{
  "id": "crawl_9f3c1a7b2e4d5068",
  "kind": "crawl",
  "status": "scraping",
  "total": 137,
  "completed": 134,
  "failed": 3,
  "limit": 500,
  "expires_at": "2026-08-19T04:12:00Z",
  "next": "/api/v1/crawl/crawl_9f3c1a7b2e4d5068?limit=20&skip=20",
  "data": [
    { "url": "https://docs.example.com/", "title": "Docs", "markdown": "...", "depth": 0, "status_code": 200,
      "metadata": { "site_name": "Example Docs", "engine": "static" } },
    { "url": "https://docs.example.com/gone", "error": { "code": "not_found", "message": "the page was not found", "http_status": 404 } }
  ]
}

# Stop it:
curl -X DELETE "http://searchx.dev/api/v1/crawl/crawl_9f3c1a7b2e4d5068" \
  -H "Authorization: Bearer $SEARCHX_API_KEY"

# An id that expired or never existed:
curl "http://searchx.dev/api/v1/crawl/crawl_deadbeef"
# HTTP 404
{ "error": { "code": "not_found", "message": "no job with that id is being kept; jobs expire after their retention window", "url": "crawl_deadbeef" } }

# Batch scrape

The same job machinery with the addresses supplied instead of discovered: no link following, no sitemap, one document per address. The URL cap is operator-configured, 1000 by default. robots.txt, per-host pacing, webhooks, change tracking and near-duplicate collapsing all work exactly as they do for a crawl — and a batch's identity is the list itself, so re-running the same list compares it against the last run, while changing the list starts its own history rather than reporting the addresses you dropped as removed. Requires an API key.

Example

curl -X POST "http://searchx.dev/api/v1/batch" \
  -H "Authorization: Bearer $SEARCHX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://a.example.com/pricing",
      "https://b.example.com/pricing",
      "https://c.example.com/pricing"
    ],
    "track_changes": true,
    "change_tag": "competitor-pricing",
    "change_modes": ["json"],
    "change_schema": { "type": "object", "properties": { "starting_price": { "type": "number" } } },
    "only_changed": true,
    "webhook": "https://hooks.example.com/searchx"
  }'

{ "id": "batch_5c2a...", "job_id": "batch_5c2a...", "status": "scraping", "status_url": "/api/v1/batch/batch_5c2a..." }

# Poll GET /api/v1/batch/{id}, cancel with DELETE /api/v1/batch/{id}.
# "async": false instead returns { results, total, took_ms } in one response.

# Web search

Hybrid search across our index — BM25 plus vector embeddings, fused and reranked — and the live web. Runs without an API key.

Name	Type	Default	Description
`q`	string	—	Required. Supports a site: operator anywhere in the query.
`mode`	hybrid \| semantic \| keyword	hybrid
`per_page`	int	10	Up to 50.
`page`	int	1
`country`	string	—	ISO country code for geo ranking. Inferred from the caller when omitted.
`lang`	string	—	en, uz, ru, de, fr, es, it, …
`site`	string	—	Restrict to one domain. Equivalent to the site: operator.
`freshness`	string	—	day, week, month, year, or an ISO range 2026-01-01..2026-01-31, open ended with 2026-01-01.. or ..2026-01-31. Pages with no known date rank lower rather than being removed.
`safe_search`	bool	true	Pass false to disable.
`category`	string	—	news, videos, images, jobs, hotels, flights, shopping.

Example

curl -G "http://searchx.dev/api/v1/search" \
  --data-urlencode "q=kubernetes horizontal pod autoscaler" \
  --data-urlencode "mode=hybrid" \
  --data-urlencode "per_page=2"

{
  "query": "kubernetes horizontal pod autoscaler",
  "mode": "hybrid",
  "total": 40,
  "took_ms": 849,
  "results": [
    {
      "url": "https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/",
      "title": "Horizontal Pod Autoscaling",
      "snippet": "In Kubernetes, a HorizontalPodAutoscaler automatically updates a workload resource...",
      "score": 1268.06,
      "domain": "kubernetes.io",
      "trust_score": 0.96,
      "citation": "[Horizontal Pod Autoscaling](...) — kubernetes.io"
    }
  ]
}

# Restrict to one site:
curl -G "http://searchx.dev/api/v1/search" --data-urlencode "q=site:kubernetes.io hpa"

# Answer

A synthesized answer with the sources it came from. When the model is unconfigured, unreachable or too slow the status stays 200 — the sources are still valid — answer is empty and warnings carries the reason. A response with an answer carries no warnings key at all.

Name	Type	Default	Description
`q`	string	—	Required. query is accepted as the original name.
`max_results`	int	3	Sources to read, 1 to 5.
`include_answer`	bool	true

Example

curl -G "http://searchx.dev/api/v1/answer" \
  --data-urlencode "q=what is a container runtime"

{
  "query": "what is a container runtime",
  "answer": "",
  "sources": [
    { "title": "...", "url": "https://...", "domain": "...", "content": "..." }
  ],
  "took_ms": 4120,
  "warnings": ["answer: the language model is unavailable right now"]
}

# The sources came back and are usable. The synthesis did not, and said so.

# Web lookup

Search, fetch the top results in parallel, chunk each page and return only the chunks that contain your term — one call instead of four. The match is word-boundary precise, so 4-modda does not match 14-modda.

Name	Type	Default	Description
`query`	string	—	Required.
`find`	string	—	Term to find in the fetched pages. Defaults to the query.
`top_k`	int	3	URLs to fetch in parallel, 1 to 5.
`per_page`	int	5	Search results to consider.

Example

curl -G "http://searchx.dev/api/v1/web_lookup" \
  --data-urlencode "query=kubernetes HPA autoscaling" \
  --data-urlencode "find=HPA" \
  --data-urlencode "top_k=3"

{
  "query": "kubernetes HPA autoscaling",
  "find": "HPA",
  "results": [
    {
      "url": "https://kubernetes.io/docs/...",
      "title": "Horizontal Pod Autoscaling",
      "matched_chunks": [{ "index": 1, "title": "How does a HorizontalPodAutoscaler work?", "markdown": "...", "word_count": 350 }],
      "total_chunks": 49
    }
  ]
}

# Images and autocomplete

Image search runs anonymously. Autocomplete requires an API key.

Name	Type	Default	Description
`q`	string	—	Required on both.
`per_page`	int	—	images/search. Results per page.
`safe`	strict \| moderate \| off	strict	images/search.
`min_width`	int	—	images/search. Minimum image width.
`min_height`	int	—	images/search. Minimum image height.
`limit`	int	8	suggest. Max suggestions.

Example

curl -G "http://searchx.dev/api/v1/images/search" \
  --data-urlencode "q=red ferrari" --data-urlencode "per_page=2"

curl -G "http://searchx.dev/api/v1/suggest" \
  -H "Authorization: Bearer $SEARCHX_API_KEY" \
  --data-urlencode "q=kuber" --data-urlencode "limit=5"

{ "suggestions": ["Kubernetes Monitoring", "Kubernetes Volumes: PV, PVC, and StorageClass", "..."] }

# SDKs

Official clients for Python and Node.

Example

# ─── Python ──────────────────────────────
pip install searchx

from searchx import SearchX

sx = SearchX(api_key="sk-sx-YOUR_KEY")

results = sx.search("kubernetes deployment", mode="hybrid")
page    = sx.extract("https://example.com")
answer  = sx.answer("what is docker", include_answer=True)
images  = sx.images("sunset mountain", per_page=10)


# ─── Node / TypeScript ───────────────────
npm install searchx

import { SearchX } from 'searchx'

const sx = new SearchX({ apiKey: 'sk-sx-YOUR_KEY' })

const results = await sx.search('kubernetes deployment')
const page    = await sx.extract('https://example.com')
const answer  = await sx.answer('what is docker')
const images  = await sx.images('sunset mountain', { perPage: 10 })

# MCP

A Model Context Protocol server, so Claude Desktop, Cursor or any MCP-capable agent can search and read the web directly. JSON-RPC 2.0 over streamable HTTP. Argument names follow MCP convention — query where REST uses q.

Example

Endpoint   POST https://mcp.searchx.dev/mcp
Health     GET  https://mcp.searchx.dev/mcp/health
Tools      search | answer | extract | suggest

curl -X POST "https://mcp.searchx.dev/mcp" \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0", "id": 1,
    "method": "tools/call",
    "params": {
      "name": "extract",
      "arguments": { "url": "https://kubernetes.io/docs", "max_length": 10000 }
    }
  }'

# ~/Library/Application Support/Claude/claude_desktop_config.json
{
  "mcpServers": {
    "searchx": {
      "url": "https://mcp.searchx.dev/mcp",
      "transport": "http"
    }
  }
}

# Rate limits and tiers

Anonymous requests are rate limited per minute. A key raises that limit and adds a daily quota; the headers on every response tell you where you stand. Crawl and batch spend real fetching capacity and are key-only.

Example

X-RateLimit-Limit       requests allowed in the current window
X-RateLimit-Remaining   requests left
X-RateLimit-Reset       seconds until the window rolls over
X-Request-Id            quote this when reporting a problem

Tier         Price    Daily      Rate/min
──────────────────────────────────────────
Anonymous    $0       —          30
Free         $0       3,000      60
Starter      $9       10,000     120
Pro          $49      100,000    300
Enterprise   $199     unlimited  1,000

429 is returned when either the minute or the daily quota is hit.
Manage plans at http://searchx.dev/pricing