Every request is authenticated with your secret API key, sent as a Bearer token in the Authorization header. Create and reveal your key from the dashboard.

Keep it secret

Treat the key like a password — never ship it in client-side code. If it leaks, rotate it from the API Keys page; the old secret is invalidated immediately.

There is no per-minute request limit. Credits are the only meter — while your key has credits, you can spend them as fast as you like, so you don't need to build backoff for a rate limit that never fires.

Credits

Successful requests consume credits: 1 credit per page scraped (requesting multiple formats for one page is still 1 credit, and a cached page costs the same as a fresh one), and 1 credit per 10 requested search results (based on num). Failed calls are free.

Errors return a JSON envelope with a stable machine code, an HTTP status, whether the request is safe to retry, and a human remediation hint.

Error codes

Switch on the stable `code` — statuses and messages may change, codes won't. Retryable errors are safe to retry with backoff.

The whole public API is published as an OpenAPI 3.1 document. Point Swagger UI, Stoplight, an SDK generator, or an agent at it — it describes every live endpoint, request body, response shape and error code.

What it covers

Only endpoints that are live today. There is no separate extract endpoint — structured extraction is this scrape endpoint with jsonOptions, below — and search is /v1/search, with the engine as a request field. Map and crawl are not shipped yet.

Give any MCP-compatible client — Claude Desktop, Claude Code, Cursor — first-class tools to read the web: scrape a page to clean Markdown, search Google, Bing or DuckDuckGo, batch-fetch, and pull structured fields out of HTML.

It is a faithful adapter over the same public CrawlFox REST API documented here — the same bespoke engine does the work, so a protected page comes back as content rather than as a block. There is nothing new to learn about the underlying behaviour, and tool calls spend credits at the same rate as the equivalent REST call.

What you need

A CrawlFox account. The server requires an OAuth token — the sign-in your client opens is the same one you use for the dashboard. No API key is ever pasted into, or stored by, the client.

Point your client at this URL:

OAuth 2.1

Your client registers itself dynamically (DCR) with PKCE and opens a browser for you to sign in and approve access. There is no client ID to request from us and no key to paste.

Discovery

A spec-compliant client discovers everything it needs from these two documents. An unauthenticated call to /mcp returns 401 with a WWW-Authenticate header pointing at them — that response starts the flow.

GET /.well-known/oauth-protected-resource/mcp   # RFC 9728
GET /.well-known/oauth-authorization-server     # RFC 8414

These shapes genuinely differ between clients: Claude Desktop adds remote servers through Connectors (its config file only launches local stdio servers), Cursor takes a bare url, and Claude Code requires an explicit type.

Claude Desktop · Connectors

Remote servers are added through Connectors, not claude_desktop_config.json — that file only launches local stdio servers.

https://mcp.crawlfox.io/mcp

1. Open Settings → Connectors → Add custom connector.
2. Paste the URL above, save, then click Connect to run the OAuth sign-in.

Cursor · mcp.json

Add to ~/.cursor/mcp.json for every project, or .cursor/mcp.json for just one.

{
  "mcpServers": {
    "crawlfox": {
      "url": "https://mcp.crawlfox.io/mcp"
    }
  }
}

1. Open Settings → MCP and sign in when Cursor prompts.

Claude Code · CLI

Add it from the terminal, or write the entry by hand. The type field is required — an entry with a url and no type is read as stdio and fails.

claude mcp add --transport http crawlfox https://mcp.crawlfox.io/mcp

{
  "mcpServers": {
    "crawlfox": {
      "type": "http",
      "url": "https://mcp.crawlfox.io/mcp"
    }
  }
}

1. Run /mcp, pick crawlfox, then Authenticate.
2. A browser opens for sign-in; the session reconnects authenticated.

Any MCP host · Spec

A spec-compliant client discovers everything it needs from these two documents. Registration is dynamic (DCR) with PKCE, so there is no client ID to request from us.

GET /.well-known/oauth-protected-resource/mcp   # RFC 9728
GET /.well-known/oauth-authorization-server     # RFC 8414

1. An unauthenticated call to /mcp returns 401 with a WWW-Authenticate header pointing at the metadata above — that response starts the flow.

Choose a tool

Every tool the server registers, in registration order. Eight of them are single-purpose shortcuts over a scrape restricted to one output format — they exist so a model can ask for exactly what it needs without reasoning about the format list.

Arguments

Every argument each tool accepts. Your client also shows these from the server's own schema; this table is here so you can read them before connecting. Only url-style arguments are required unless marked otherwise.

crawlfox_scrape

Fetch one URL as clean, LLM-ready content — Markdown, HTML, text, links, images or emails. Handles JavaScript-rendered and protected pages. Maps to POST /v1/scrape.

crawlfox_search

Web search via Google, Bing, or DuckDuckGo, returning ranked organic results. Maps to POST /v1/search.

crawlfox_batch_scrape

Scrape several URLs in one call, returning one result per page. Maps to POST /v1/batch.

crawlfox_extract

Pull named fields out of a page into JSON using CSS selectors — deterministic, no AI in the loop. Maps to POST /v1/scrape.

crawlfox_metadata

Return only page metadata: title, description, and status. Maps to POST /v1/scrape.

crawlfox_links

Extract every link on a page. Maps to POST /v1/scrape.

crawlfox_images

Extract every image URL on a page. Maps to POST /v1/scrape.

crawlfox_emails

Extract every email address on a page. Maps to POST /v1/scrape.

crawlfox_text

Return a page as plain text. Maps to POST /v1/scrape.

crawlfox_html

Return a page as clean HTML. Maps to POST /v1/scrape.

crawlfox_scrape_get

Lightweight GET scrape of a URL, for quick one-off fetches. Maps to GET /v1/scrape/:url.

crawlfox_get_log

Look up the status and timing of a past request by its id. Maps to GET /v1/logs/:id.

crawlfox_log_result

Retrieve the full stored result of a past scrape by id. Maps to GET /v1/logs/:id/result.

Once the server is connected you do not call these tools by hand — you ask in plain language and the client picks the tool. These are the shapes that work well, with the call each one produces.

Read one page

The most common case. The model picks crawlfox_scrape and gets Markdown back.

You: Summarise https://example.com/pricing for me.

→ crawlfox_scrape { "url": "https://example.com/pricing" }

Research a topic across sources

Search first, then read the results. Ask for the search explicitly when you want breadth before depth, otherwise a client may scrape the first URL it can guess.

You: Find three recent write-ups on Rust async runtimes and
     summarise what they disagree about.

→ crawlfox_search        { "query": "rust async runtime comparison", "count": 10 }
→ crawlfox_scrape        { "url": "<first result>" }
→ crawlfox_scrape        { "url": "<second result>" }

Pull specific fields out of a page

When you want data rather than prose, name the fields. Selectors are CSS, matched against the fetched DOM, and the result comes back as JSON — the same deterministic extraction the REST API does, with no model in the loop guessing values.

You: From https://example.com/product/42 get me the title, price
     and stock status.

→ crawlfox_extract {
    "url": "https://example.com/product/42",
    "selectors": { "title": "h1", "price": ".price", "stock": "[data-stock]" }
  }

Read many pages at once

Batch when you already have the list. One call, one result per page, in the order you gave them.

You: Scrape these five docs pages and tell me which mention rate limits.

→ crawlfox_batch_scrape { "urls": ["https://…/a", "https://…/b", …] }

When a page is slow

A page that renders slowly can exceed the default time budget. Ask for more time and the model passes timeout through. It defaults to 30000 ms; below 1000 the call is rejected, and above 90000 it is capped at 90000 rather than refused.

You: Scrape https://example.com/heavy — give it more time than usual.

→ crawlfox_scrape { "url": "https://example.com/heavy", "timeout": 60000 }

Costs

Tool calls spend credits at exactly the same rate as the REST call behind them — 1 credit per page scraped, 1 per 10 requested search results. Looking up a past request or its stored result is free. Failed calls are free.