Scrape API

Turns one web page into LLM-ready content: the page as Markdown, plus its metadata, headings, links, images, structured data, language and AI-use signals. We read the raw HTML, which is what AI crawlers such as GPTBot and ClaudeBot see, and render the page as a browser shows it when that HTML is an empty JavaScript app shell.

Only the links? Link Extraction.

GEThttps://rankfield.dev/v1/scrape?url={url}

Send Authorization: Bearer YOUR_API_KEY. Get a free key; no account needed.

curl "https://rankfield.dev/v1/scrape?url=https://rankfield.dev/blog/what-is-site-strength" \
  -H "Authorization: Bearer $RANKFIELD_API_KEY"

import os
import requests

response = requests.get(
    "https://rankfield.dev/v1/scrape?url=https://rankfield.dev/blog/what-is-site-strength",
    headers={"Authorization": f"Bearer {os.environ['RANKFIELD_API_KEY']}"},
)
print(response.json())

const response = await fetch("https://rankfield.dev/v1/scrape?url=https://rankfield.dev/blog/what-is-site-strength", {
  headers: {
    Authorization: `Bearer ${process.env.RANKFIELD_API_KEY}`,
  },
});
console.log(await response.json());

Response

{
  "url": "https://rankfield.dev/blog/what-is-site-strength",
  "finalUrl": "https://rankfield.dev/blog/what-is-site-strength",
  "status": 200,
  "redirects": [],
  "contentType": "text/html; charset=utf-8",
  "bytes": 27716,
  "truncated": false,
  "fetchedAt": "2026-10-26T18:52:46Z",
  "timing": {
    "dnsMs": 0.0,
    "connectMs": 0.0,
    "tlsMs": 0.0,
    "firstByteMs": 42.9,
    "totalMs": 43.2
  },
  "language": {
    "code": "en",
    "script": "Latn",
    "confidence": 1,
    "source": "content",
    "declared": {
      "code": "en",
      "region": null,
      "script": null,
      "htmlLang": "en",
      "contentLanguage": null,
      "metaContentLanguage": null,
      "ogLocale": null
    },
    "detected": {
      "code": "en",
      "iso639_3": "eng",
      "name": "English",
      "script": "Latn",
      "confidence": 1,
      "reliable": true
    },
    "agree": true,
    "textLetters": 837
  },
  "metadata": {
    "title": "What is Site Strength? — Rankfield",
    "description": "Another SEO score? Here is what ours tells you, the evidence behind it, and how 20 real domains compare.",
    "canonical": null,
    "robots": null,
    "xRobotsTag": null,
    "hreflang": [],
    "siteName": null,
    "author": null,
    "publishedTime": null,
    "favicon": null,
    "ogImage": null,
    "charset": "utf-8"
  },
  "headings": [
    {
      "level": 1,
      "text": "What is Site Strength?"
    },
    {
      "level": 2,
      "text": "How we build Site Strength"
    },
    {
      "level": 2,
      "text": "Okay, enough science. How do the numbers compare?"
    },
    "…"
  ],
  "content": {
    "mode": "page",
    "extraction": "document",
    "words": 854,
    "truncated": false,
    "markdown": "[← All articles](https://rankfield.dev/blog)  \nTHE SCORE\n\n# What is Site Strength?\n\nAnother SEO score? Here is what ours tells you, the evidence behind it, and how 20 real domains compare.\n\nSite Strength is our 0–100 estimate of a website's link strength. It looks at how many different websites link to a domain and how much weight those links carry. Use it to decide which competitors or link prosp …"
  },
  "links": [
    {
      "url": "https://rankfield.dev/",
      "text": "Rankfield",
      "rel": [],
      "internal": true,
      "nofollow": false,
      "sponsored": false,
      "ugc": false,
      "occurrences": 3
    },
    {
      "url": "https://rankfield.dev/endpoints/",
      "text": "Endpoints",
      "rel": [],
      "internal": true,
      "nofollow": false,
      "sponsored": false,
      "ugc": false,
      "occurrences": 1
    },
    {
      "url": "https://rankfield.dev/free-seo-tools/site-strength",
      "text": "Free tools",
      "rel": [],
      "internal": true,
      "nofollow": false,
      "sponsored": false,
      "ugc": false,
      "occurrences": 1
    },
    "…"
  ],
  "images": [
    {
      "src": "https://rankfield.dev/images/site-strength-links.webp",
      "alt": "Illustrated websites sending links toward one domain",
      "width": 1672,
      "height": 941,
      "lazy": false,
      "srcset": []
    }
  ],
  "structuredData": {
    "jsonLd": [],
    "microdata": [],
    "openGraph": {},
    "twitter": {}
  },
  "signals": {
    "noai": false,
    "noimageai": false,
    "tdmReserved": null,
    "tdmPolicy": null,
    "tdmSource": null
  },
  "blocked": null,
  "cached": false,
  "ageSeconds": 0
}

Parameters#

Name	Type	Description
`url`query, required	string	A public `http` or `https` URL. A missing scheme gets `https://`.
`scope`query, optional	stringdefault `page`	How much of the page the Markdown covers. `page`: the whole page in reading order, each section once, collapsed content included, without the site's menus and link lists (navigation, header and footer links, sidebars, category and archive lists, language and currency pickers, share and follow buttons), skip links and cookie banners; contact details such as an e-mail address or a phone number stay. `main`: the main article only, or the page's content when the article found is very short next to the rest of the page. `all`: everything, menus and footers included, each section still once.
`include`query, optional	stringdefault `all parts`	Comma-separated: `content`, `links`, `images`, `headings`, `metadata`, `structuredData`, `language`, `signals`, `highlights`. Parts left out come back `null`. `highlights` alone returns just the passages, and needs `query`.
`query`query, optional	string	Words to look for, 1–300 characters. Adds `highlights`: the page's passages that best match them, best first. Close forms count ("price" finds "prices"), synonyms don't; a phrase in double quotes matches as a phrase.
`maxPassages`query, optional	integerdefault `5`	1–20: the most passages returned. Only with `query`.
`passageChars`query, optional	integerdefault `600`	200–2,000: about how long each passage is, in characters. Only with `query`.
`render`query, optional	stringdefault `auto`	`auto` runs the page's JavaScript only when its raw HTML is an empty app shell; `never` returns the raw HTML only, as AI crawlers see it; `always` runs it every time.
`screenshot`query, optional	string	`viewport` or `fullPage`: also return a WebP image of the page as a browser shows it, the first screen or the whole page up to 12,000 px tall.
`viewport`query, optional	stringdefault `mobile`	`mobile` (412×915, like Google's smartphone crawler) or `desktop` (1366×768): the screen for rendering and screenshots.
`maxAge`query, optional	integerdefault `600`	Seconds a stored answer may be old (default 10 minutes), up to 2,592,000. `0` reads the site fresh.
`timeout`query, optional	integerdefault `20`	Seconds the call may take, 1 to 60.

Response#

Field	Type	Description
`url` `finalUrl` `status` `redirects`	string, integer, array	What you asked, the page that answered, the website's HTTP status (a 404 is data, not an error) and each redirect.
`contentType` `bytes` `truncated` `fetchedAt` `timing`	mixed	The page's type and size, whether it was cut (over 10 MB, or Markdown over 1 MB), when we fetched it, and DNS/connect/TLS/first-byte/total milliseconds.
`content`	object	`markdown`, `words`, `mode` (the scope used: `page`, `main` or `all`), `extraction` (`article` when `main` found an article, `document` for the whole page, `text` for a plain-text or Markdown body passed through, or `pdf` for a PDF's text) and `truncated`. A product page whose text doesn't name the product starts with the product's name, description and price from the page's structured data.
`metadata`	object	Title, description, canonical, meta robots, X-Robots-Tag, hreflang, site name, author, published time, favicon, og:image, charset.
`headings`	array	`{level, text}`, H1 to H6 in page order.
`links` `images`	array	Each once, in page order: links with anchor text and rel flags; images with alt, size, srcset and `lazy`.
`structuredData`	object	Parsed JSON-LD, microdata, OpenGraph and Twitter card tags.
`language` `signals`	object	The page's language; `noai`, `noimageai` and the TDM reservation.
`highlights`	array \| null	Only with `query`: the best-matching passages, best first, each `{text, heading, offset, score, matched}`. `[]` when nothing matches.
`blocked`	object \| null	`{by, kind}` when a firewall answered instead of the page, such as a "verify you're human" challenge (reported, never rendered or solved); the page parts are then `null`.
`render`	object	Whether the page was rendered and why (`rendered`, `reason`), the time taken, and the files robots.txt kept out (`robotsBlocked`); or `unavailable`, and the raw-HTML answer.
`screenshot`	object	With `screenshot`: a WebP image, `{format, kind, viewport, bytes, data}` (base64).
`cached` `ageSeconds`	boolean, integer	Whether the answer came from our store, and its age.

Errors#

Errors come as {"error":{"code":"…","message":"…","details":{…}}} with the HTTP status.

Status	Code	When
`400`	`invalid_input`	A parameter is missing, unknown or malformed.
`401`	`unauthorized`	The key is missing, unknown or revoked.
`403`	`robots_disallowed`	robots.txt disallows Rankfield-User for the page or a redirect target; `details` has the rule and line.
`403`	`blocked`	The website's bot protection refused us. Reported at once and never worked around; `details` says which protection. Not counted as usage.
`403`	`site_excluded`	The site is on the exclusion list: Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us.
`415`	`unsupported_content_type`	A page that answered 2xx isn't HTML, plain text, markdown or a readable PDF (an image, JSON, a scanned or password-protected PDF…). `details.reason` says why for PDFs. A website's error page in another type comes back as an answer.
`422`	`address_not_public`	The host resolves to a private or internal address.
`429`	`busy`	Our fetch capacity is busy: the call was refused at once, waited 4 seconds without a turn, or its time limit ran out while it waited (then `details.reason` is `capacity`). A new job also gets it when the shared job queue is full (`Retry-After` 60). Not counted as usage. Retry after `Retry-After` seconds.
`500`	`internal`	Our bug. Retry.
`502`	`fetch_failed`	The website could not be reached (DNS, connection, TLS, more than 5 redirects or a bad one, a malformed answer); `details.reason` says which.
`503`	`unavailable`	The service is restarting or briefly unavailable. Retry after `Retry-After`. (A full fetch capacity answers `429` `busy`.)
`504`	`timeout`	The website did not answer within `timeout`. With `details.reason: "sitePace"`, the call waited its turn for this website instead, which we fetch gently, shared by all customers. Retry after `Retry-After`, or allow a longer `timeout`.

Price and limits#

Free; no per-call charge.
No per-key limits. When the server is busy you get 429 busy with Retry-After.
Only answered calls count as usage (plus the website errors listed below); our own errors (500, 503), 429 busy and a 504 with details.reason sitePace don't.

Up to 10,000 URLs in one job#

POST /v1/scrape/jobs with an Idempotency-Key header and {"items":[{"url":"https://example.com/"}, …]}. Each item takes the same parameters as one call, except timeout: items use the default, 20 seconds. Every item is checked first: if any is invalid, the whole request is a 400 that lists the first 20. A new job answers 202; the same Idempotency-Key with a byte-identical body returns the same job (200, Idempotency-Replayed: true), and with any other body is a 409.

Read GET /v1/scrape/jobs/{id}/results?cursor=0&limit=100: finished items as they complete, each {index, ok, result} or {index, ok: false, error}. Pass cursor back until hasMore is false. POST /v1/scrape/jobs/{id}/cancel stops a job. A job runs 16 items at a time; an item that meets busy waits and retries by itself. Results are kept 24 hours.

Send as many jobs as you like. Jobs from all keys share one queue: when 200,000 items are already waiting, a new job answers 429 busy with Retry-After (60 seconds).

Reference: https://rankfield.dev/docs/get-started/batch-jobs

Notes#

A page that fills itself in with JavaScript is rendered when its raw HTML is an empty app shell; render=never shows what crawlers that run no JavaScript get.
Repeat calls to the same website within about a minute wait their turn for that site's polite slot (we fetch each site gently for all customers). If the wait runs past the call's time limit (20 s by default), the call ends with 504 timeout and details.reason: "sitePace". It's not charged; retry after Retry-After, or allow a longer timeout.
We fetch as Rankfield-User, obey robots.txt on every request and redirect, render a page only when its raw HTML is an empty JavaScript app shell (or every page with render=always), use no cookies or proxies, and never work around a firewall. We fetch each site gently, shared by all customers. More: rankfield.dev/bot.