Audit a website

Start a technical SEO and AEO audit of a website: we crawl its pages and check each one against 213 rules.

POSThttps://rankfield.dev/v1/site-audit

curl -X POST 'https://rankfield.dev/v1/site-audit' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"domain":"shop.example"}'

import requests

response = requests.post(
    "https://rankfield.dev/v1/site-audit",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"domain": "shop.example"},
)
print(response.json())

const response = await fetch("https://rankfield.dev/v1/site-audit", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({"domain":"shop.example"}),
});
console.log(await response.json());

{
  "audit": {
    "id": "1414774bd14c096e08ec5b95fc183bcf",
    "domain": "https://shop.example",
    "status": "queued",
    "createdAt": "2026-10-28T20:45:39Z",
    "startedAt": null,
    "finishedAt": null,
    "expiresAt": null,
    "stoppedBy": [],
    "error": null,
    "options": {
      "domain": "https://shop.example",
      "maxPages": 10000,
      "subdomains": false,
      "sources": [
        "links",
        "sitemaps"
      ],
      "urls": [],
      "mode": "full",
      "sample": false,
      "samplePerTemplate": 500,
      "checkExternal": true,
      "checkResources": true,
      "previous": null,
      "speed": null
    },
    "progress": {
      "phase": "queued"
    },
    "counts": {}
  }
}

{
  "error": {
    "code": "unauthorized",
    "message": "A valid API key is required"
  }
}

The call answers at once (200, status: "queued") with the audit's id; the crawl runs in the background. Follow it, then read the results, during the crawl or after. Big sites take time: a site that answers fast is crawled at up to 50 pages a second, a slow or struggling one at 2.

Parameters

Send a JSON object.

Headers

Name	Type	Required	Default	Description
`Authorization`	string	Yes	–	`Bearer` followed by your API key. See Authentication.

Body

Name	Type	Required	Default	Description
`domain`	string	Yes, or `url`	–	The site: a host such as `example.com`, or a URL whose host is used. `https` unless written `http://`.
`maxPages`	integer	No	`10000`	HTML pages to fetch at most, 1 to 5,000,000.
`subdomains`	boolean	No	`false`	Also crawl subdomains such as `blog.example.com`.
`sources`	string[]	No	`["links","sitemaps"]`	Where URLs come from: `links` (a crawl from the homepage), `sitemaps` (robots.txt's sitemaps, else `/sitemap.xml`), `urls` (your `urls` list).
`urls`	string[]	No	–	Up to 10,000 URLs of the site you know exist, from Search Console, analytics or logs: fetched, and used to find orphan pages.
`mode`	string	No	`full`	`full`: content and links. `status`: only the status of every URL in the sitemaps and `urls` (no content): broken pages and redirect chains in minutes.
`sample`	boolean	No	`false`	Audit at most `samplePerTemplate` pages per URL template and estimate the rest, for very large sites.
`samplePerTemplate`	integer	No	`500`	Pages per template when `sample` is true, 10 to 10,000.
`checkExternal`	boolean	No	`true`	Check each distinct external link target once, at most 2,000 per audit.
`checkResources`	boolean	No	`true`	Check each distinct image, script and stylesheet of the site once, at most 2,000 per audit.
`previous`	string	No	–	The `id` of an earlier finished audit of the same site: unchanged pages reuse its records, and diff compares the two.
`speed`	integer	No	–	The most fetches a second, 2 to 50. By default the speed adapts to the site, up to 50.

Response

200 with the audit, the same object Follow an audit reads:

Field	Type	Description
`audit.id`	string	The audit's id: keep it to read the audit.
`audit.domain`	string	The site, as we crawl it (`https://example.com`).
`audit.status`	string	`queued`, `running`, `postPass` (the site-wide checks), `done`, `canceled` or `failed`.
`audit.options`	object	The parameters as we read them, defaults filled in.
`audit.progress`, `audit.counts`	object	Pages fetched so far, the speed and the issues found: see Follow an audit.
`audit.createdAt`, `audit.startedAt`, `audit.finishedAt`, `audit.expiresAt`	string \| null	When it was created, started and finished, and when its results are deleted (30 days after it ends), in UTC.
`audit.stoppedBy`	string[]	Why it ended early: `maxPages`, `siteErrors`, `rateLimited` (the site kept rate-limiting us for most of ten minutes), `canceled` or `time`; empty when it crawled everything it found.
`audit.queuePosition`	integer	Its place in the queue, while `queued`.

How we crawl

We fetch as Rankfield-User (Mozilla/5.0 (compatible; Rankfield-User/1.0; +https://rankfield.dev/bot)), as plain HTML without running JavaScript.

Speed: each audit adapts to its site. It starts at 5 fetches a second and climbs, up to 50, while the site answers quickly and without errors. At the first sign of strain (a 429, a 503, another 5xx, a connection error or a slower answer) it halves at once and honours Retry-After, never going below 2 a second, and pauses for a minute after three 5xx in a row. A site that keeps failing (more than half of its last 200 answers) stops the audit with stoppedBy: ["siteErrors"]; what was found is kept. The limits are the same for every site.
A site that rate-limits us: each 429 or 503 halves the speed, and we leave the site alone for as long as its Retry-After asks (30 seconds when it gives none, 5 minutes at most). While we wait, progress shows siteHold and gives no time estimate. A site that holds us for most of ten minutes stops the audit with stoppedBy: ["rateLimited"]; what was found is kept, the site-wide checks run on it, and the summary's notes say so.
robots.txt and Crawl-delay: obeyed on every request, for Rankfield-User, or for RankfieldBot when the file has none for it, or * when it has neither.
Redirects: not followed blindly. Each hop is fetched and recorded on its own, so chains and loops are exact, and robots.txt is checked on every hop.
Other sites: a link, redirect or resource that leaves the site gets one status check (checkExternal) and is never crawled. Cloudflare's own /cdn-cgi/ paths are never fetched.
Not checked is never broken: an external URL we did not check is never counted as a broken link: a site we never fetch (below), one robots.txt disallows, or one whose bot protection refused our check (a 401, 403 or 429, or a challenge page). They are listed with status=unchecked and counted in counts.inventory.externalUnchecked.
Never: cookies, logins, JavaScript, proxies or other bots' names. A firewall page is reported, never worked around. Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages and sites whose owners asked us are never fetched.
What we keep: each page's record (status, headers of interest, title, description and h1 cut to 200 characters, counts, hashes) and its links; never the page's text. Results are kept 30 days after the audit ends.

More: Rankfield crawlers.

Errors

Errors are {"error": {"code", "message", "details"}}. Branch on code; message is for people.

Status	Code	When
400	`invalid_input`	A parameter is missing, unknown or malformed (unknown parameters are refused), or the body is over 1 MB.
401	`unauthorized`	No valid API key.
403	`site_excluded`	The site is on the exclusion list.
404	`not_found`	No audit with this id for this key (only the key that started an audit can read it), an unknown issue, or a page that is not in the audit.
422	`address_not_public`	The host resolves to a private or internal address.
429	`busy`	The audit queue is full: 64 audits already wait (`details.reason` `capacity`). Not counted as usage. Retry after `Retry-After` seconds.
500	`internal`	Our bug. Retry.
503	`unavailable`	New audits are paused while the service is short of storage, or it is restarting. Retry after `Retry-After`.

Limits

No per-key limits. Up to 8 audits crawl at once and the rest wait in order; when 64 already wait, a new audit answers 429 busy with Retry-After. At most one audit over 1,000,000 pages runs at a time and 2 more wait; another one answers 429 busy with Retry-After, and smaller audits can start ahead of a waiting one. An audit fetches at most 5,000,000 pages.

MCP

On the MCP server, the tool for this is site_audit_start.

Audit a website

#Parameters

#Headers

#Body

#Response

#How we crawl

#Errors

#Limits

#MCP