Scrape a page
Turn a web page into clean Markdown for LLMs, with its metadata, headings, links, images, structured data, language and AI-use signals.
We fetch the page once, as plain HTML: what AI crawlers such as GPTBot and ClaudeBot see. When that HTML is an empty JavaScript app shell, we render the page and read it as a browser shows it; render=never keeps the raw HTML only, and render=always renders every page.
Send the parameters as a query string with GET, or as a JSON object (up to 1 MB) with POST to the same path. For many URLs, start a job.
Parameters
Headers
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
Authorization | string | Yes | – | Bearer followed by your API key. See Authentication. |
Query parameters
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
url | string | Yes | – | A public http or https URL. Without a scheme, https:// is added. |
scope | string | No | page | How much of the page the Markdown covers. page: the whole page in reading order, each section once, collapsed content included, without the site's menus and link lists (navigation, header and footer links, sidebars, category and archive lists, language and currency pickers, share and follow buttons), skip links and cookie banners; contact details such as an e-mail address or a phone number stay. main: the main article only, or the page's content when the article found is very short next to the rest of the page. all: everything, menus and footers included, each section still once. |
include | string | No | all parts | Only these parts, comma-separated: content, links, images, headings, metadata, structuredData, language, signals, highlights. Parts left out come back as null. highlights alone returns just the passages, and needs query. |
query | string | No | – | Words to look for, 1–300 characters. Adds highlights: the page's passages that best match them, best first. Close forms count ("price" finds "prices"), synonyms don't; put a phrase in double quotes to match it as a phrase. The same page and query always give the same passages. |
maxPassages | integer | No | 5 | 1–20: the most passages highlights holds. Only with query: without it, 400. |
passageChars | integer | No | 600 | 200–2,000: about how long each passage is, in characters. Only with query: without it, 400. |
render | string | No | auto | auto runs the page's JavaScript only when its raw HTML is an empty app shell; never returns the raw HTML only, as AI crawlers see it; always runs it every time. See Rendered pages. |
screenshot | string | No | – | viewport or fullPage: also return a WebP image of the page as a browser shows it, the first screen or the whole page up to 12,000 px tall. |
viewport | string | No | mobile | mobile or desktop: the screen size for rendering and screenshots. Mobile is 412×915, like Google's smartphone crawler; desktop is 1366×768. |
maxAge | integer | No | 600 | How old a stored answer may be, in seconds, up to 2,592,000 (30 days). 0 fetches fresh. |
timeout | integer | No | 20 | How long the call may take, in seconds, 1–60. |
Response
| Field | Type | Description |
|---|---|---|
url | string | The page you asked for. |
finalUrl | string | The page that answered, after redirects. |
status | integer | The website's HTTP status. A 404 or 500 from the website is reported here, not as an error. |
redirects | array | Each redirect, {url, status}. |
contentType, bytes | string, integer | The page's Content-Type and size. |
truncated | boolean | true when the page was over 10 MB or its Markdown over 1 MB, and the first part was kept. |
fetchedAt | string | When the page was fetched (UTC). |
timing | object | Milliseconds for DNS, connecting, TLS, the first byte and the whole fetch. |
language | object | code (such as en; null when unknown), script, confidence (0–1), source (content: read from the text, declared: from the markup, or none), declared (what the markup says: <html lang>, Content-Language, the meta tag, og:locale) and detected (the reading of the text alone). |
metadata | object | Title, description, canonical, meta robots, X-Robots-Tag, hreflang, site name, author, published time, favicon, og:image and charset. |
headings | array | The H1–H6 outline in page order, {level, text}. |
content | object | markdown, words, mode (the scope used: page, main or all), extraction (article when main found an article, document for the whole page, text for a plain-text or Markdown body passed through, or pdf for a PDF's text) and truncated. A product page whose text doesn't name the product starts with the product's name, description and price from the page's structured data. |
highlights | array | null | Only with query: the passages that best match it, best first, each {text, heading, offset, score, matched}: the passage as plain text, the headings above it (such as Acme widgets › Pricing, or null), where it starts in content.markdown (in characters), its score, and the query words found in it. [] when nothing matches; null when the page is blocked or isn't text. |
links | array | Every link once, in page order: url, text, rel, internal, nofollow, sponsored, ugc, occurrences. |
images | array | Every image once: src, alt, width, height, srcset, and lazy when the URL only sits in a lazy-loading attribute. |
structuredData | object | jsonLd (parsed), microdata, openGraph and twitter. |
signals | object | AI-use signals: noai, noimageai, and the TDM reservation (tdmReserved, tdmPolicy, tdmSource). |
render | object | {mode, rendered, reason, tier, ms, timedOut, requests, robotsBlocked}, or unavailable saying why the page wasn't rendered (the answer is then the raw-HTML one). |
screenshot | object | With screenshot: {format: "webp", kind, viewport, bytes, data}, the image in base64. |
blocked | object | null | {by, kind} when a firewall answered instead of the page, such as a "verify you're human" challenge. It is reported as it came, never rendered or solved. |
cached, ageSeconds | boolean, integer | Whether this answer came from our store, and how old it is. |
Rendered pages
When Scrape renders a page (an empty app shell under the default render=auto, or every page with render=always), it loads the page in Chrome, runs its JavaScript and builds every part of the answer from the page as a browser shows it: content, links, metadata, highlights. render.rendered says whether it ran, and render.reason says why or why not. When the rendering lane is busy, the page can't be rendered in time, or the site's robots.txt asks for a Crawl-delay of a second or more (crawlDelay), you get the raw-HTML answer with render.unavailable saying why.
Rendering follows the same rules as every Rankfield fetch:
- robots.txt for every script and file the page loads (the ones it keeps out are listed in
render.robotsBlocked); - no cookies and no form posts;
- images, media, fonts and trackers are not loaded unless you ask for a screenshot.
A page behind a bot challenge is reported as blocked, never worked around.
PDF documents
A PDF (application/pdf, or an untyped file that starts as a PDF) is read as a document:
- Content: the document's text as
content.markdown(extraction: "pdf"), the same for everyscope, pluscontent.pdf:pages,pagesReadandstopped. - Length: up to the first 300 pages and 10 MB are read. When the text ends early,
stoppedsays why (sizeLimit,pageLimit,timeLimit,textLimit) andcontent.truncatedis true. - Metadata: the document's title (or its first line), author, subject as
description, creation date aspublishedTime. - Links, images, headings and structured data are empty.
A PDF we can't read is a 415 unsupported_content_type with details.reason: noTextLayer (scanned pages, no text), encrypted (needs a password), damaged or tooComplex. Any other type (images, JSON, XML…) on a 2xx is a 415 with details.contentType.
How we fetch
We fetch as Rankfield-User (Mozilla/5.0 (compatible; Rankfield-User/1.0; +https://rankfield.dev/bot)).
Scrape reads the raw HTML first. When that HTML is an empty JavaScript app shell, it renders the page in Chrome and answers from what a browser shows (render.rendered true). Pages with their content in the HTML are answered from the HTML, as before. Ask render=never for the raw HTML only.
- robots.txt: obeyed on every request and redirect, following the rules for
Rankfield-User, or forRankfieldBotwhen the file has none for it, or*when it has neither. A page it disallows is refused and never fetched. - Firewalls: a challenge, CAPTCHA or block page is reported in the answer, never worked around and never retried.
- Never: cookies, logins, proxies or other bots' names.
- Excluded sites: Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us are never fetched.
- Pace: each website is fetched gently, shared by all customers, slower when its robots.txt sets a
Crawl-delayor after a slow answer. Many URLs on one site go at that site's pace.
Repeat calls to the same website within about a minute can wait their turn for that site's slot. If the wait runs past the call's time limit (20 seconds by default), the call ends with 504 timeout and details.reason sitePace. It doesn't count as usage: retry after Retry-After, or allow a longer timeout.
More: Rankfield crawlers.
Errors
Errors are {"error": {"code", "message", "details"}}. Branch on code; message is for people.
| Status | Code | When |
|---|---|---|
| 400 | invalid_input | A parameter is missing, unknown or malformed. |
| 401 | unauthorized | No valid API key. |
| 403 | robots_disallowed | robots.txt disallows Rankfield-User for the page or a redirect; details has the rule and line. |
| 403 | site_excluded | The site is on the exclusion list. |
| 403 | blocked | The website's bot protection refused us. Reported at once and never worked around; details says which protection. Not counted as usage. |
| 415 | unsupported_content_type | A page that answered 2xx isn't HTML, plain text, markdown or a readable PDF (an image, JSON, a scanned or password-protected PDF…). details.reason says why for PDFs. A website's error page in another type is an answer, not a 415. |
| 422 | address_not_public | The host resolves to a private or internal address. |
| 429 | busy | Our fetch capacity is busy: the call was refused at once, waited 4 seconds without a turn, or its time limit ran out while it waited (then details.reason is capacity). Not counted as usage. Retry after Retry-After seconds. |
| 500 | internal | Our bug. Retry. |
| 502 | fetch_failed | The website could not be reached (DNS, connection, TLS, more than 5 redirects); details.reason says which. |
| 503 | unavailable | The service is restarting or briefly unavailable. Retry after Retry-After. (A full fetch capacity answers 429 busy.) |
| 504 | timeout | The website didn't answer within the call's time limit. When details.reason is sitePace, the call waited its turn for this website instead, which we fetch gently, shared by all customers: retry after Retry-After, or allow a longer timeout. |
Only answered pages and the 415, 502 and website-side 504 errors count as usage; refusals, our own errors (500, 503), 429 busy and a 504 with details.reason sitePace don't.
Limits
No per-key limits. When the server is busy you get 429 busy with Retry-After. Waiting calls are served in turn across keys, so a key with a few calls is never stuck behind one with thousands.
MCP
On the MCP server, the tool for this is scrape_page.