Site Audit API
Crawls a website from its raw HTML as Rankfield-User and checks every page against 213 rules: broken pages and links, redirects, indexability, canonicals, sitemaps, orphan pages, titles, headings, images, duplicates, structured data, hreflang, URL hygiene, security headers, mobile, social cards and AI-crawler access. Then it joins the pages into one link graph for click depth, inlinks, internal link score, orphans, redirect chains and duplicates. The audit runs in the background: start it, then read its progress and results while it crawls.
Need only the URLs? Use Site Map. How crawlers see one page? Use Bot View.
https://rankfield.dev/v1/site-auditSend Authorization: Bearer YOUR_API_KEY. Get a free key; no account needed.
Parameters#
| Name | Type | Description |
|---|---|---|
domain | string | The website, e.g. example.com; a URL works too (its host is used). Write http://example.com for a site without HTTPS. |
maxPages | integer | HTML pages to fetch at most, 1 to 5,000,000. |
subdomains | boolean | true also crawls subdomains such as blog.example.com. |
sources | array | Where URLs come from: links (a crawl from the homepage), sitemaps, urls (your list). |
urls | array | Up to 10,000 URLs of the site you know exist (Search Console, analytics, logs): fetched, and used to find orphan pages. |
mode | string | full reads content and links; status only checks the status of every URL in the sitemaps and urls, in minutes. |
sample | boolean | true audits at most samplePerTemplate pages (default 500) per URL template and estimates the rest. |
checkExternal, checkResources | boolean | Check external link targets, and the site's images, scripts and stylesheets, once each (2,000 at most per audit). |
previous | string | An earlier audit of the same site: unchanged pages reuse its records, and …/diff compares the two. |
speed | integer | The most fetches a second, 2 to 50. By default the speed adapts to the site, up to 50. |
Response#
| Field | Type | Description |
|---|---|---|
audit.id audit.status | string | The audit's id, and queued, running, postPass (the site-wide checks), done, canceled or failed. |
audit.progress | object | Pages fetched, queued and known, the speed now (rate), the site's response time (ttfbP50Ms), errors, an estimate of the time left (null while the site makes us wait, with siteHold saying why and until when), how often the site rate-limited us (rateLimited) and the issues found so far by severity. |
audit.counts | object | health: the share of crawled internal URLs with no error-level issue, 0 to 100, with its parts. inventory: crawled URLs, HTML pages, documents, redirects, errors, resources and status classes, apart from issues, and the external URLs checked, broken and not checked (never counted as broken). issues: per severity, issueTypes (kinds) and affectedUrls (URL–issue pairs). |
audit.options | object | The parameters as we read them, defaults filled in. |
audit.stoppedBy | array | Why it ended early: maxPages, siteErrors, rateLimited, canceled or time. |
audit.createdAt audit.finishedAt audit.expiresAt | string | null | When it was created and finished, and when its results are deleted (30 days after it ends). |
Errors#
Errors come as {"error":{"code":"…","message":"…","details":{…}}} with the HTTP status.
| Status | Code | When |
|---|---|---|
400 | invalid_input | A parameter is missing, unknown or malformed, or the body is over 1 MB. |
401 | unauthorized | The key is missing, unknown or revoked. |
403 | site_excluded | The site is on the exclusion list: Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us. |
404 | not_found | No audit with this id for this key: only the key that started an audit can read it. |
422 | address_not_public | The host resolves to a private or internal address. |
429 | busy | The audit queue is full: 64 audits already wait. Not counted as usage. Retry after Retry-After seconds. |
503 | unavailable | New audits are paused while the service is short of storage, or it is restarting. Retry after Retry-After. |
Price and limits#
- Free; no per-call charge.
- No per-key limits. Up to 8 audits crawl at once and the rest wait in order; when 64 wait, a new audit answers
429busywithRetry-After. At most one audit over 1,000,000 pages runs at a time and 2 more wait; another one answers429busywithRetry-After, and smaller audits can start ahead of a waiting one. - An audit fetches at most 5,000,000 pages. Results are kept 30 days after it ends.
Read an audit#
GET /v1/site-audit/{id} gives the status and live progress; poll it every few seconds. Issue counts are live, and every result can be read during the crawl.
…/{id}/summary gives notes on how complete the audit is and the ten most important problems in plain words. Then …/issues (every issue found, by group such as Internal pages, Indexability, Links and Content, each with a short name and the URLs affected out of the set it reads, such as 3 of 103 HTML pages), …/issues/{issue} (its URLs, with that issue's own columns), …/pages and …/links (records, with ready-made lists and their counts at …/pages/filters and …/links/filters), …/structure (the folder tree), …/log (the crawl log), …/history (every audit of the site), …/redirects, …/duplicates, …/orphans, …/templates, …/sitemaps, …/diff/{previousId} and …/export. Lists page with cursor and limit. POST …/{id}/cancel stops a crawl and keeps what it found.
Reference: https://rankfield.dev/docs/site-audit/read-the-results
curl "https://rankfield.dev/v1/site-audit/169c615ff6c5057c8ca5384a6e2df47a/summary" \
-H "Authorization: Bearer $RANKFIELD_API_KEY"import os
import requests
response = requests.get(
"https://rankfield.dev/v1/site-audit/169c615ff6c5057c8ca5384a6e2df47a/summary",
headers={"Authorization": f"Bearer {os.environ['RANKFIELD_API_KEY']}"},
)
print(response.json())const response = await fetch("https://rankfield.dev/v1/site-audit/169c615ff6c5057c8ca5384a6e2df47a/summary", {
headers: {
Authorization: `Bearer ${process.env.RANKFIELD_API_KEY}`,
},
});
console.log(await response.json());{
"id": "169c615ff6c5057c8ca5384a6e2df47a",
"domain": "https://rankfield.dev",
"status": "done",
"health": {
"score": 100,
"urlsWithErrors": 0,
"urlsWithoutErrors": 202,
"internalUrls": 202,
"note": "The share of crawled internal URLs with no error-level issue; site-wide findings are not counted per URL."
},
"inventory": {
"internal": 202,
"htmlPages": 103,
"indexable": 103,
"documents": 96,
"redirects": 0,
"errors": 0,
"resources": 3,
"disallowed": 0,
"notFetched": 0,
"known": 199,
"sitemapUrls": 99,
"externalChecked": 0,
"externalBroken": 0,
"externalUnchecked": {
"total": 0,
"excluded": 0,
"robotsDisallowed": 0,
"botBlocked": 0
},
"byStatus": {
"2xx": 202,
"3xx": 0,
"4xx": 0,
"5xx": 0,
"failed": 0
}
},
"notes": [],
"headline": "Health 100 of 100: 202 of 202 crawled URLs have no errors. 103 HTML pages, 96 other documents; 0 kinds of error, 22 issue types in all.",
"top": [
{
"issue": "schema-none",
"name": "No structured data",
"group": "ai",
"severity": "notice",
"site": false,
"urls": 103,
"of": {
"set": "indexable",
"urls": 103,
"label": "indexable pages"
},
"share": 100.0,
"text": "No structured data: 103 of 103 indexable pages. The indexable page has no structured data. Add JSON-LD for the page's type (Article, Product, Organization, BreadcrumbList…).",
"example": "https://rankfield.dev/"
},
{
"issue": "no-hsts",
"name": "HSTS missing",
"group": "other",
"severity": "notice",
"site": false,
"urls": 103,
"of": {
"set": "htmlPages",
"urls": 103,
"label": "HTML pages"
},
"share": 100.0,
"text": "HSTS missing: 103 of 103 HTML pages. The HTTPS page sends no Strict-Transport-Security header. Send HSTS (max-age of at least six months) once the whole site is HTTPS.",
"example": "https://rankfield.dev/"
},
{
"issue": "no-preferred-image",
"name": "No preferred image",
"group": "social",
"severity": "notice",
"site": false,
"urls": 103,
"of": {
"set": "htmlPages",
"urls": 103,
"label": "HTML pages"
},
"share": 100.0,
"text": "No preferred image: 103 of 103 HTML pages. The page names no preferred image (og:image, or an image in its structured data). Name the page's main image, so search and AI results can show it.",
"example": "https://rankfield.dev/"
},
"…"
]
}Notes#
- We crawl as
Rankfield-User, obey robots.txt andCrawl-delayon every request, run no JavaScript and never work around a firewall. Each audit starts at 5 fetches a second and climbs to 50 while the site answers quickly and cleanly; at the first429,5xxor slower answer it halves at once and honoursRetry-After, never below 2 a second, for every site alike. A site that keeps rate-limiting us for most of ten minutes ends the audit early (stoppedBy: ["rateLimited"]) with what was found. An external link we could not check (a site we never fetch, robots.txt, bot protection) is never counted as broken. More: rankfield.dev/bot.
See also#
- Full reference: https://rankfield.dev/docs/site-audit/audit-a-website
- MCP tools on
https://rankfield.dev/mcp:site_audit_start,site_audit_status,site_audit_summary,site_audit_issues,site_audit_issue_pages,site_audit_pages,site_audit_links,site_audit_redirects,site_audit_duplicates,site_audit_orphans,site_audit_templates,site_audit_sitemaps,site_audit_diff,site_audit_export,site_audit_page_filters,site_audit_link_filters,site_audit_structure,site_audit_log,site_audit_history,site_audit_list,site_audit_cancel - This page as Markdown: /endpoints/site-audit.md