# Scrape API

> Turns one web page into LLM-ready content: the page as Markdown, plus its metadata, headings, links, images, structured data, language and AI-use signals. We read the raw HTML, which is what AI crawlers such as GPTBot and ClaudeBot see, and render the page as a browser shows it when that HTML is an empty JavaScript app shell.

Only the links? [Link Extraction](https://rankfield.dev/endpoints/html-links).

- Page: https://rankfield.dev/endpoints/scrape
- Full reference: https://rankfield.dev/docs/scrape/scrape-a-page

## Request

`GET https://rankfield.dev/v1/scrape?url={url}`

Send `Authorization: Bearer YOUR_API_KEY`. Get a free key at https://rankfield.dev/?tab=key#playground (no account).

```bash
curl "https://rankfield.dev/v1/scrape?url=https://rankfield.dev/blog/what-is-site-strength" \
  -H "Authorization: Bearer $RANKFIELD_API_KEY"
```

Response:

```json
{
  "url": "https://rankfield.dev/blog/what-is-site-strength",
  "finalUrl": "https://rankfield.dev/blog/what-is-site-strength",
  "status": 200,
  "redirects": [],
  "contentType": "text/html; charset=utf-8",
  "bytes": 27716,
  "truncated": false,
  "fetchedAt": "2026-10-26T18:52:46Z",
  "timing": {
    "dnsMs": 0.0,
    "connectMs": 0.0,
    "tlsMs": 0.0,
    "firstByteMs": 42.9,
    "totalMs": 43.2
  },
  "language": {
    "code": "en",
    "script": "Latn",
    "confidence": 1,
    "source": "content",
    "declared": {
      "code": "en",
      "region": null,
      "script": null,
      "htmlLang": "en",
      "contentLanguage": null,
      "metaContentLanguage": null,
      "ogLocale": null
    },
    "detected": {
      "code": "en",
      "iso639_3": "eng",
      "name": "English",
      "script": "Latn",
      "confidence": 1,
      "reliable": true
    },
    "agree": true,
    "textLetters": 837
  },
  "metadata": {
    "title": "What is Site Strength? — Rankfield",
    "description": "Another SEO score? Here is what ours tells you, the evidence behind it, and how 20 real domains compare.",
    "canonical": null,
    "robots": null,
    "xRobotsTag": null,
    "hreflang": [],
    "siteName": null,
    "author": null,
    "publishedTime": null,
    "favicon": null,
    "ogImage": null,
    "charset": "utf-8"
  },
  "headings": [
    {
      "level": 1,
      "text": "What is Site Strength?"
    },
    {
      "level": 2,
      "text": "How we build Site Strength"
    },
    {
      "level": 2,
      "text": "Okay, enough science. How do the numbers compare?"
    },
    "…"
  ],
  "content": {
    "mode": "page",
    "extraction": "document",
    "words": 854,
    "truncated": false,
    "markdown": "[← All articles](https://rankfield.dev/blog)  \nTHE SCORE\n\n# What is Site Strength?\n\nAnother SEO score? Here is what ours tells you, the evidence behind it, and how 20 real domains compare.\n\nSite Strength is our 0–100 estimate of a website's link strength. It looks at how many different websites link to a domain and how much weight those links carry. Use it to decide which competitors or link prosp …"
  },
  "links": [
    {
      "url": "https://rankfield.dev/",
      "text": "Rankfield",
      "rel": [],
      "internal": true,
      "nofollow": false,
      "sponsored": false,
      "ugc": false,
      "occurrences": 3
    },
    {
      "url": "https://rankfield.dev/endpoints/",
      "text": "Endpoints",
      "rel": [],
      "internal": true,
      "nofollow": false,
      "sponsored": false,
      "ugc": false,
      "occurrences": 1
    },
    {
      "url": "https://rankfield.dev/free-seo-tools/site-strength",
      "text": "Free tools",
      "rel": [],
      "internal": true,
      "nofollow": false,
      "sponsored": false,
      "ugc": false,
      "occurrences": 1
    },
    "…"
  ],
  "images": [
    {
      "src": "https://rankfield.dev/images/site-strength-links.webp",
      "alt": "Illustrated websites sending links toward one domain",
      "width": 1672,
      "height": 941,
      "lazy": false,
      "srcset": []
    }
  ],
  "structuredData": {
    "jsonLd": [],
    "microdata": [],
    "openGraph": {},
    "twitter": {}
  },
  "signals": {
    "noai": false,
    "noimageai": false,
    "tdmReserved": null,
    "tdmPolicy": null,
    "tdmSource": null
  },
  "blocked": null,
  "cached": false,
  "ageSeconds": 0
}
```

## Parameters

| Name | In | Type | Required | Default | Description |
|---|---|---|---|---|---|
| `url` | query | string | yes |  | A public `http` or `https` URL. A missing scheme gets `https://`. |
| `scope` | query | string | no | `page` | How much of the page the Markdown covers. `page`: the whole page in reading order, each section once, collapsed content included, without the site's menus and link lists (navigation, header and footer links, sidebars, category and archive lists, language and currency pickers, share and follow buttons), skip links and cookie banners; contact details such as an e-mail address or a phone number stay. `main`: the main article only, or the page's content when the article found is very short next to the rest of the page. `all`: everything, menus and footers included, each section still once. |
| `include` | query | string | no | `all parts` | Comma-separated: `content`, `links`, `images`, `headings`, `metadata`, `structuredData`, `language`, `signals`, `highlights`. Parts left out come back `null`. `highlights` alone returns just the passages, and needs `query`. |
| `query` | query | string | no |  | Words to look for, 1–300 characters. Adds `highlights`: the page's passages that best match them, best first. Close forms count ("price" finds "prices"), synonyms don't; a phrase in double quotes matches as a phrase. |
| `maxPassages` | query | integer | no | `5` | 1–20: the most passages returned. Only with `query`. |
| `passageChars` | query | integer | no | `600` | 200–2,000: about how long each passage is, in characters. Only with `query`. |
| `render` | query | string | no | `auto` | `auto` runs the page's JavaScript only when its raw HTML is an empty app shell; `never` returns the raw HTML only, as AI crawlers see it; `always` runs it every time. |
| `screenshot` | query | string | no |  | `viewport` or `fullPage`: also return a WebP image of the page as a browser shows it, the first screen or the whole page up to 12,000 px tall. |
| `viewport` | query | string | no | `mobile` | `mobile` (412×915, like Google's smartphone crawler) or `desktop` (1366×768): the screen for rendering and screenshots. |
| `maxAge` | query | integer | no | `600` | Seconds a stored answer may be old (default 10 minutes), up to 2,592,000. `0` reads the site fresh. |
| `timeout` | query | integer | no | `20` | Seconds the call may take, 1 to 60. |

## Response

| Field | Type | Description |
|---|---|---|
| `url, finalUrl, status, redirects` | string, integer, array | What you asked, the page that answered, the website's HTTP status (a 404 is data, not an error) and each redirect. |
| `contentType, bytes, truncated, fetchedAt, timing` | mixed | The page's type and size, whether it was cut (over 10 MB, or Markdown over 1 MB), when we fetched it, and DNS/connect/TLS/first-byte/total milliseconds. |
| `content` | object | `markdown`, `words`, `mode` (the scope used: `page`, `main` or `all`), `extraction` (`article` when `main` found an article, `document` for the whole page, `text` for a plain-text or Markdown body passed through, or `pdf` for a PDF's text) and `truncated`. A product page whose text doesn't name the product starts with the product's name, description and price from the page's structured data. |
| `metadata` | object | Title, description, canonical, meta robots, X-Robots-Tag, hreflang, site name, author, published time, favicon, og:image, charset. |
| `headings` | array | `{level, text}`, H1 to H6 in page order. |
| `links, images` | array | Each once, in page order: links with anchor text and rel flags; images with alt, size, srcset and `lazy`. |
| `structuredData` | object | Parsed JSON-LD, microdata, OpenGraph and Twitter card tags. |
| `language, signals` | object | The page's language; `noai`, `noimageai` and the TDM reservation. |
| `highlights` | array \| null | Only with `query`: the best-matching passages, best first, each `{text, heading, offset, score, matched}`. `[]` when nothing matches. |
| `blocked` | object \| null | `{by, kind}` when a firewall answered instead of the page, such as a "verify you're human" challenge (reported, never rendered or solved); the page parts are then `null`. |
| `render` | object | Whether the page was rendered and why (`rendered`, `reason`), the time taken, and the files robots.txt kept out (`robotsBlocked`); or `unavailable`, and the raw-HTML answer. |
| `screenshot` | object | With `screenshot`: a WebP image, `{format, kind, viewport, bytes, data}` (base64). |
| `cached, ageSeconds` | boolean, integer | Whether the answer came from our store, and its age. |

## Errors

Errors come as `{"error":{"code":"…","message":"…","details":{…}}}` with the HTTP status.

| Status | Code | When |
|---|---|---|
| 400 | `invalid_input` | A parameter is missing, unknown or malformed. |
| 401 | `unauthorized` | The key is missing, unknown or revoked. |
| 403 | `robots_disallowed` | robots.txt disallows Rankfield-User for the page or a redirect target; `details` has the rule and line. |
| 403 | `blocked` | The website's bot protection refused us. Reported at once and never worked around; `details` says which protection. Not counted as usage. |
| 403 | `site_excluded` | The site is on the exclusion list: Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us. |
| 415 | `unsupported_content_type` | A page that answered 2xx isn't HTML, plain text, markdown or a readable PDF (an image, JSON, a scanned or password-protected PDF…). `details.reason` says why for PDFs. A website's error page in another type comes back as an answer. |
| 422 | `address_not_public` | The host resolves to a private or internal address. |
| 429 | `busy` | Our fetch capacity is busy: the call was refused at once, waited 4 seconds without a turn, or its time limit ran out while it waited (then `details.reason` is `capacity`). A new job also gets it when the shared job queue is full (`Retry-After` 60). Not counted as usage. Retry after `Retry-After` seconds. |
| 500 | `internal` | Our bug. Retry. |
| 502 | `fetch_failed` | The website could not be reached (DNS, connection, TLS, more than 5 redirects or a bad one, a malformed answer); `details.reason` says which. |
| 503 | `unavailable` | The service is restarting or briefly unavailable. Retry after `Retry-After`. (A full fetch capacity answers `429` `busy`.) |
| 504 | `timeout` | The website did not answer within `timeout`. With `details.reason: "sitePace"`, the call waited its turn for this website instead, which we fetch gently, shared by all customers. Retry after `Retry-After`, or allow a longer `timeout`. |

## Price and limits

- Free; no per-call charge.
- No per-key limits. When the server is busy you get `429` `busy` with `Retry-After`.
- Only answered calls count as usage (plus the website errors listed below); our own errors (`500`, `503`), `429` `busy` and a `504` with `details.reason` `sitePace` don't.

## Up to 10,000 URLs in one job

`POST /v1/scrape/jobs` with an `Idempotency-Key` header and `{"items":[{"url":"https://example.com/"}, …]}`. Each item takes the same parameters as one call, except `timeout`: items use the default, 20 seconds. Every item is checked first: if any is invalid, the whole request is a `400` that lists the first 20. A new job answers `202`; the same `Idempotency-Key` with a byte-identical body returns the same job (`200`, `Idempotency-Replayed: true`), and with any other body is a `409`.

Read `GET /v1/scrape/jobs/{id}/results?cursor=0&limit=100`: finished items as they complete, each `{index, ok, result}` or `{index, ok: false, error}`. Pass `cursor` back until `hasMore` is false. `POST /v1/scrape/jobs/{id}/cancel` stops a job. A job runs 16 items at a time; an item that meets `busy` waits and retries by itself. Results are kept 24 hours.

Send as many jobs as you like. Jobs from all keys share one queue: when 200,000 items are already waiting, a new job answers `429` `busy` with `Retry-After` (60 seconds).

Reference: https://rankfield.dev/docs/get-started/batch-jobs

## Notes

- A page that fills itself in with JavaScript is rendered when its raw HTML is an empty app shell; `render=never` shows what crawlers that run no JavaScript get.
- Repeat calls to the same website within about a minute wait their turn for that site's polite slot (we fetch each site gently for all customers). If the wait runs past the call's time limit (20 s by default), the call ends with `504` `timeout` and `details.reason: "sitePace"`. It's not charged; retry after `Retry-After`, or allow a longer `timeout`.
- We fetch as `Rankfield-User`, obey robots.txt on every request and redirect, render a page only when its raw HTML is an empty JavaScript app shell (or every page with `render=always`), use no cookies or proxies, and never work around a firewall. We fetch each site gently, shared by all customers. More: [rankfield.dev/bot](https://rankfield.dev/bot).

## MCP

Tools on https://rankfield.dev/mcp: `scrape_page`, `start_scrape_job`, `get_scrape_results`.