> ## Documentation index
> Fetch the complete documentation index at: https://rankfield.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape a page

> Turn a web page into clean Markdown for LLMs, with its metadata, headings, links, images, structured data, language and AI-use signals.

`GET https://rankfield.dev/v1/scrape?url={url}`

```bash cURL
curl 'https://rankfield.dev/v1/scrape?url=https://rankfield.dev/blog/what-is-site-strength' \
  -H 'Authorization: Bearer YOUR_API_KEY'
```

```python Python
import requests

response = requests.get(
    "https://rankfield.dev/v1/scrape?url=https://rankfield.dev/blog/what-is-site-strength",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
)
print(response.json())
```

```javascript JavaScript
const response = await fetch("https://rankfield.dev/v1/scrape?url=https://rankfield.dev/blog/what-is-site-strength", {
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
console.log(await response.json());
```

```json 200
{
  "url": "https://rankfield.dev/blog/what-is-site-strength",
  "finalUrl": "https://rankfield.dev/blog/what-is-site-strength",
  "status": 200,
  "redirects": [],
  "contentType": "text/html; charset=utf-8",
  "bytes": 27716,
  "truncated": false,
  "fetchedAt": "2026-10-26T18:52:46Z",
  "timing": {
    "dnsMs": 0.0,
    "connectMs": 0.0,
    "tlsMs": 0.0,
    "firstByteMs": 42.9,
    "totalMs": 43.2
  },
  "language": {
    "code": "en",
    "script": "Latn",
    "confidence": 1,
    "source": "content",
    "declared": {
      "code": "en",
      "region": null,
      "script": null,
      "htmlLang": "en",
      "contentLanguage": null,
      "metaContentLanguage": null,
      "ogLocale": null
    },
    "detected": {
      "code": "en",
      "iso639_3": "eng",
      "name": "English",
      "script": "Latn",
      "confidence": 1,
      "reliable": true
    },
    "agree": true,
    "textLetters": 837
  },
  "metadata": {
    "title": "What is Site Strength? — Rankfield",
    "description": "Another SEO score? Here is what ours tells you, the evidence behind it, and how 20 real domains compare.",
    "canonical": null,
    "robots": null,
    "xRobotsTag": null,
    "hreflang": [],
    "siteName": null,
    "author": null,
    "publishedTime": null,
    "favicon": null,
    "ogImage": null,
    "charset": "utf-8"
  },
  "headings": [
    {
      "level": 1,
      "text": "What is Site Strength?"
    },
    {
      "level": 2,
      "text": "How we build Site Strength"
    },
    {
      "level": 2,
      "text": "Okay, enough science. How do the numbers compare?"
    }
  ],
  "content": {
    "mode": "page",
    "extraction": "document",
    "words": 854,
    "truncated": false,
    "markdown": "[← All articles](https://rankfield.dev/blog)  \nTHE SCORE\n\n# What is Site Strength?\n\nAnother SEO score? Here is what ours tells you, the evidence behind it, and how 20 real domains compare.\n\nSite Strength is our 0–100 estimate of a website's link strength. It looks at how many different websites link to a domain and how much weight those links carry. Use it to decide which competitors or link prosp …"
  },
  "links": [
    {
      "url": "https://rankfield.dev/",
      "text": "Rankfield",
      "rel": [],
      "internal": true,
      "nofollow": false,
      "sponsored": false,
      "ugc": false,
      "occurrences": 3
    },
    {
      "url": "https://rankfield.dev/endpoints/",
      "text": "Endpoints",
      "rel": [],
      "internal": true,
      "nofollow": false,
      "sponsored": false,
      "ugc": false,
      "occurrences": 1
    },
    {
      "url": "https://rankfield.dev/free-seo-tools/site-strength",
      "text": "Free tools",
      "rel": [],
      "internal": true,
      "nofollow": false,
      "sponsored": false,
      "ugc": false,
      "occurrences": 1
    }
  ],
  "images": [
    {
      "src": "https://rankfield.dev/images/site-strength-links.webp",
      "alt": "Illustrated websites sending links toward one domain",
      "width": 1672,
      "height": 941,
      "lazy": false,
      "srcset": []
    }
  ],
  "structuredData": {
    "jsonLd": [],
    "microdata": [],
    "openGraph": {},
    "twitter": {}
  },
  "signals": {
    "noai": false,
    "noimageai": false,
    "tdmReserved": null,
    "tdmPolicy": null,
    "tdmSource": null
  },
  "blocked": null,
  "cached": false,
  "ageSeconds": 0
}
```

```json 401
{
  "error": {
    "code": "unauthorized",
    "message": "A valid API key is required"
  }
}
```

We fetch the page once, as plain HTML: what AI crawlers such as GPTBot and ClaudeBot see. When that HTML is an empty JavaScript app shell, we render the page and read it as a browser shows it; `render=never` keeps the raw HTML only, and `render=always` renders every page.

Send the parameters as a query string with `GET`, or as a JSON object (up to 1 MB) with `POST` to the same path. For many URLs, [start a job](/docs/scrape/start-a-job).

## Parameters

### Headers

| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| `Authorization` | string | Yes | – | `Bearer` followed by your API key. See [Authentication](/docs/get-started/authentication). |

### Query parameters

| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| `url` | string | Yes | – | A public `http` or `https` URL. Without a scheme, `https://` is added. |
| `scope` | string | No | `page` | How much of the page the Markdown covers. `page`: the whole page in reading order, each section once, collapsed content included, without the site's menus and link lists (navigation, header and footer links, sidebars, category and archive lists, language and currency pickers, share and follow buttons), skip links and cookie banners; contact details such as an e-mail address or a phone number stay. `main`: the main article only, or the page's content when the article found is very short next to the rest of the page. `all`: everything, menus and footers included, each section still once. |
| `include` | string | No | `all parts` | Only these parts, comma-separated: `content`, `links`, `images`, `headings`, `metadata`, `structuredData`, `language`, `signals`, `highlights`. Parts left out come back as `null`. `highlights` alone returns just the passages, and needs `query`. |
| `query` | string | No | – | Words to look for, 1–300 characters. Adds `highlights`: the page's passages that best match them, best first. Close forms count ("price" finds "prices"), synonyms don't; put a phrase in double quotes to match it as a phrase. The same page and query always give the same passages. |
| `maxPassages` | integer | No | `5` | 1–20: the most passages `highlights` holds. Only with `query`: without it, `400`. |
| `passageChars` | integer | No | `600` | 200–2,000: about how long each passage is, in characters. Only with `query`: without it, `400`. |
| `render` | string | No | `auto` | `auto` runs the page's JavaScript only when its raw HTML is an empty app shell; `never` returns the raw HTML only, as AI crawlers see it; `always` runs it every time. See [Rendered pages](#rendered-pages). |
| `screenshot` | string | No | – | `viewport` or `fullPage`: also return a WebP image of the page as a browser shows it, the first screen or the whole page up to 12,000 px tall. |
| `viewport` | string | No | `mobile` | `mobile` or `desktop`: the screen size for rendering and screenshots. Mobile is 412×915, like Google's smartphone crawler; desktop is 1366×768. |
| `maxAge` | integer | No | `600` | How old a stored answer may be, in seconds, up to 2,592,000 (30 days). `0` fetches fresh. |
| `timeout` | integer | No | `20` | How long the call may take, in seconds, 1–60. |

## Response

| Field | Type | Description |
|---|---|---|
| `url` | string | The page you asked for. |
| `finalUrl` | string | The page that answered, after redirects. |
| `status` | integer | The website's HTTP status. A 404 or 500 from the website is reported here, not as an error. |
| `redirects` | array | Each redirect, `{url, status}`. |
| `contentType`, `bytes` | string, integer | The page's Content-Type and size. |
| `truncated` | boolean | `true` when the page was over 10 MB or its Markdown over 1 MB, and the first part was kept. |
| `fetchedAt` | string | When the page was fetched (UTC). |
| `timing` | object | Milliseconds for DNS, connecting, TLS, the first byte and the whole fetch. |
| `language` | object | `code` (such as `en`; `null` when unknown), `script`, `confidence` (0–1), `source` (`content`: read from the text, `declared`: from the markup, or `none`), `declared` (what the markup says: `<html lang>`, `Content-Language`, the meta tag, `og:locale`) and `detected` (the reading of the text alone). |
| `metadata` | object | Title, description, canonical, meta robots, X-Robots-Tag, hreflang, site name, author, published time, favicon, og:image and charset. |
| `headings` | array | The H1–H6 outline in page order, `{level, text}`. |
| `content` | object | `markdown`, `words`, `mode` (the scope used: `page`, `main` or `all`), `extraction` (`article` when `main` found an article, `document` for the whole page, `text` for a plain-text or Markdown body passed through, or `pdf` for a PDF's text) and `truncated`. A product page whose text doesn't name the product starts with the product's name, description and price from the page's structured data. |
| `highlights` | array \| null | Only with `query`: the passages that best match it, best first, each `{text, heading, offset, score, matched}`: the passage as plain text, the headings above it (such as `Acme widgets › Pricing`, or `null`), where it starts in `content.markdown` (in characters), its score, and the query words found in it. `[]` when nothing matches; `null` when the page is blocked or isn't text. |
| `links` | array | Every link once, in page order: `url`, `text`, `rel`, `internal`, `nofollow`, `sponsored`, `ugc`, `occurrences`. |
| `images` | array | Every image once: `src`, `alt`, `width`, `height`, `srcset`, and `lazy` when the URL only sits in a lazy-loading attribute. |
| `structuredData` | object | `jsonLd` (parsed), `microdata`, `openGraph` and `twitter`. |
| `signals` | object | AI-use signals: `noai`, `noimageai`, and the TDM reservation (`tdmReserved`, `tdmPolicy`, `tdmSource`). |
| `render` | object | `{mode, rendered, reason, tier, ms, timedOut, requests, robotsBlocked}`, or `unavailable` saying why the page wasn't rendered (the answer is then the raw-HTML one). |
| `screenshot` | object | With `screenshot`: `{format: "webp", kind, viewport, bytes, data}`, the image in base64. |
| `blocked` | object \| null | `{by, kind}` when a firewall answered instead of the page, such as a "verify you're human" challenge. It is reported as it came, never rendered or solved. |
| `cached`, `ageSeconds` | boolean, integer | Whether this answer came from our store, and how old it is. |

## Rendered pages

When Scrape renders a page (an empty app shell under the default `render=auto`, or every page with `render=always`), it loads the page in Chrome, runs its JavaScript and builds every part of the answer from the page as a browser shows it: content, links, metadata, highlights. `render.rendered` says whether it ran, and `render.reason` says why or why not. When the rendering lane is busy, the page can't be rendered in time, or the site's robots.txt asks for a Crawl-delay of a second or more (`crawlDelay`), you get the raw-HTML answer with `render.unavailable` saying why.

Rendering follows the same rules as every Rankfield fetch:

- robots.txt for every script and file the page loads (the ones it keeps out are listed in `render.robotsBlocked`);
- no cookies and no form posts;
- images, media, fonts and trackers are not loaded unless you ask for a screenshot.

A page behind a bot challenge is reported as blocked, never worked around.

## PDF documents

A PDF (`application/pdf`, or an untyped file that starts as a PDF) is read as a document:

- **Content:** the document's text as `content.markdown` (`extraction: "pdf"`), the same for every `scope`, plus `content.pdf`: `pages`, `pagesRead` and `stopped`.
- **Length:** up to the first 300 pages and 10 MB are read. When the text ends early, `stopped` says why (`sizeLimit`, `pageLimit`, `timeLimit`, `textLimit`) and `content.truncated` is true.
- **Metadata:** the document's title (or its first line), author, subject as `description`, creation date as `publishedTime`.
- Links, images, headings and structured data are empty.

A PDF we can't read is a `415` `unsupported_content_type` with `details.reason`: `noTextLayer` (scanned pages, no text), `encrypted` (needs a password), `damaged` or `tooComplex`. Any other type (images, JSON, XML…) on a 2xx is a `415` with `details.contentType`.

## How we fetch

We fetch as `Rankfield-User` (`Mozilla/5.0 (compatible; Rankfield-User/1.0; +https://rankfield.dev/bot)`).

Scrape reads the raw HTML first. When that HTML is an empty JavaScript app shell, it renders the page in Chrome and answers from what a browser shows (`render.rendered` true). Pages with their content in the HTML are answered from the HTML, as before. Ask `render=never` for the raw HTML only.

- **robots.txt:** obeyed on every request and redirect, following the rules for `Rankfield-User`, or for `RankfieldBot` when the file has none for it, or `*` when it has neither. A page it disallows is refused and never fetched.
- **Firewalls:** a challenge, CAPTCHA or block page is reported in the answer, never worked around and never retried.
- **Never:** cookies, logins, proxies or other bots' names.
- **Excluded sites:** Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us are never fetched.
- **Pace:** each website is fetched gently, shared by all customers, slower when its robots.txt sets a `Crawl-delay` or after a slow answer. Many URLs on one site go at that site's pace.

Repeat calls to the same website within about a minute can wait their turn for that site's slot. If the wait runs past the call's time limit (20 seconds by default), the call ends with `504` `timeout` and `details.reason` `sitePace`. It doesn't count as usage: retry after `Retry-After`, or allow a longer `timeout`.

More: [Rankfield crawlers](/bot).

## Errors

Errors are `{"error": {"code", "message", "details"}}`. Branch on `code`; `message` is for people.

| Status | Code | When |
|---|---|---|
| 400 | `invalid_input` | A parameter is missing, unknown or malformed. |
| 401 | `unauthorized` | No valid API key. |
| 403 | `robots_disallowed` | robots.txt disallows Rankfield-User for the page or a redirect; `details` has the rule and line. |
| 403 | `site_excluded` | The site is on the exclusion list. |
| 403 | `blocked` | The website's bot protection refused us. Reported at once and never worked around; `details` says which protection. Not counted as usage. |
| 415 | `unsupported_content_type` | A page that answered 2xx isn't HTML, plain text, markdown or a readable PDF (an image, JSON, a scanned or password-protected PDF…). `details.reason` says why for PDFs. A website's error page in another type is an answer, not a 415. |
| 422 | `address_not_public` | The host resolves to a private or internal address. |
| 429 | `busy` | Our fetch capacity is busy: the call was refused at once, waited 4 seconds without a turn, or its time limit ran out while it waited (then `details.reason` is `capacity`). Not counted as usage. Retry after `Retry-After` seconds. |
| 500 | `internal` | Our bug. Retry. |
| 502 | `fetch_failed` | The website could not be reached (DNS, connection, TLS, more than 5 redirects); `details.reason` says which. |
| 503 | `unavailable` | The service is restarting or briefly unavailable. Retry after `Retry-After`. (A full fetch capacity answers `429` `busy`.) |
| 504 | `timeout` | The website didn't answer within the call's time limit. When `details.reason` is `sitePace`, the call waited its turn for this website instead, which we fetch gently, shared by all customers: retry after `Retry-After`, or allow a longer `timeout`. |

Only answered pages and the `415`, `502` and website-side `504` errors count as usage; refusals, our own errors (`500`, `503`), `429` `busy` and a `504` with `details.reason` `sitePace` don't.

## Limits

No per-key limits. When the server is busy you get `429` `busy` with `Retry-After`. Waiting calls are served in turn across keys, so a key with a few calls is never stuck behind one with thousands.

## MCP

On the [MCP server](/docs/agents/mcp-server), the tool for this is `scrape_page`.