Site Audit API

Crawls a website from its raw HTML as Rankfield-User and checks every page against 213 rules: broken pages and links, redirects, indexability, canonicals, sitemaps, orphan pages, titles, headings, images, duplicates, structured data, hreflang, URL hygiene, security headers, mobile, social cards and AI-crawler access. Then it joins the pages into one link graph for click depth, inlinks, internal link score, orphans, redirect chains and duplicates. The audit runs in the background: start it, then read its progress and results while it crawls.

Need only the URLs? Use Site Map. How crawlers see one page? Use Bot View.

POSThttps://rankfield.dev/v1/site-audit

Send Authorization: Bearer YOUR_API_KEY. Get a free key; no account needed.

Parameters#

NameTypeDescription
domainbody, required unless urlstringThe website, e.g. example.com; a URL works too (its host is used). Write http://example.com for a site without HTTPS.
maxPagesbody, optionalintegerdefault 10000HTML pages to fetch at most, 1 to 5,000,000.
subdomainsbody, optionalbooleandefault falsetrue also crawls subdomains such as blog.example.com.
sourcesbody, optionalarraydefault links, sitemapsWhere URLs come from: links (a crawl from the homepage), sitemaps, urls (your list).
urlsbody, optionalarrayUp to 10,000 URLs of the site you know exist (Search Console, analytics, logs): fetched, and used to find orphan pages.
modebody, optionalstringdefault fullfull reads content and links; status only checks the status of every URL in the sitemaps and urls, in minutes.
samplebody, optionalbooleandefault falsetrue audits at most samplePerTemplate pages (default 500) per URL template and estimates the rest.
checkExternal, checkResourcesbody, optionalbooleandefault trueCheck external link targets, and the site's images, scripts and stylesheets, once each (2,000 at most per audit).
previousbody, optionalstringAn earlier audit of the same site: unchanged pages reuse its records, and …/diff compares the two.
speedbody, optionalintegerThe most fetches a second, 2 to 50. By default the speed adapts to the site, up to 50.

Response#

FieldTypeDescription
audit.id audit.statusstringThe audit's id, and queued, running, postPass (the site-wide checks), done, canceled or failed.
audit.progressobjectPages fetched, queued and known, the speed now (rate), the site's response time (ttfbP50Ms), errors, an estimate of the time left (null while the site makes us wait, with siteHold saying why and until when), how often the site rate-limited us (rateLimited) and the issues found so far by severity.
audit.countsobjecthealth: the share of crawled internal URLs with no error-level issue, 0 to 100, with its parts. inventory: crawled URLs, HTML pages, documents, redirects, errors, resources and status classes, apart from issues, and the external URLs checked, broken and not checked (never counted as broken). issues: per severity, issueTypes (kinds) and affectedUrls (URL–issue pairs).
audit.optionsobjectThe parameters as we read them, defaults filled in.
audit.stoppedByarrayWhy it ended early: maxPages, siteErrors, rateLimited, canceled or time.
audit.createdAt audit.finishedAt audit.expiresAtstring | nullWhen it was created and finished, and when its results are deleted (30 days after it ends).

Errors#

Errors come as {"error":{"code":"…","message":"…","details":{…}}} with the HTTP status.

StatusCodeWhen
400invalid_inputA parameter is missing, unknown or malformed, or the body is over 1 MB.
401unauthorizedThe key is missing, unknown or revoked.
403site_excludedThe site is on the exclusion list: Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us.
404not_foundNo audit with this id for this key: only the key that started an audit can read it.
422address_not_publicThe host resolves to a private or internal address.
429busyThe audit queue is full: 64 audits already wait. Not counted as usage. Retry after Retry-After seconds.
503unavailableNew audits are paused while the service is short of storage, or it is restarting. Retry after Retry-After.

Price and limits#

  • Free; no per-call charge.
  • No per-key limits. Up to 8 audits crawl at once and the rest wait in order; when 64 wait, a new audit answers 429 busy with Retry-After. At most one audit over 1,000,000 pages runs at a time and 2 more wait; another one answers 429 busy with Retry-After, and smaller audits can start ahead of a waiting one.
  • An audit fetches at most 5,000,000 pages. Results are kept 30 days after it ends.

Read an audit#

GET /v1/site-audit/{id} gives the status and live progress; poll it every few seconds. Issue counts are live, and every result can be read during the crawl.

…/{id}/summary gives notes on how complete the audit is and the ten most important problems in plain words. Then …/issues (every issue found, by group such as Internal pages, Indexability, Links and Content, each with a short name and the URLs affected out of the set it reads, such as 3 of 103 HTML pages), …/issues/{issue} (its URLs, with that issue's own columns), …/pages and …/links (records, with ready-made lists and their counts at …/pages/filters and …/links/filters), …/structure (the folder tree), …/log (the crawl log), …/history (every audit of the site), …/redirects, …/duplicates, …/orphans, …/templates, …/sitemaps, …/diff/{previousId} and …/export. Lists page with cursor and limit. POST …/{id}/cancel stops a crawl and keeps what it found.

Reference: https://rankfield.dev/docs/site-audit/read-the-results

curl "https://rankfield.dev/v1/site-audit/169c615ff6c5057c8ca5384a6e2df47a/summary" \
  -H "Authorization: Bearer $RANKFIELD_API_KEY"
Response
{
  "id": "169c615ff6c5057c8ca5384a6e2df47a",
  "domain": "https://rankfield.dev",
  "status": "done",
  "health": {
    "score": 100,
    "urlsWithErrors": 0,
    "urlsWithoutErrors": 202,
    "internalUrls": 202,
    "note": "The share of crawled internal URLs with no error-level issue; site-wide findings are not counted per URL."
  },
  "inventory": {
    "internal": 202,
    "htmlPages": 103,
    "indexable": 103,
    "documents": 96,
    "redirects": 0,
    "errors": 0,
    "resources": 3,
    "disallowed": 0,
    "notFetched": 0,
    "known": 199,
    "sitemapUrls": 99,
    "externalChecked": 0,
    "externalBroken": 0,
    "externalUnchecked": {
      "total": 0,
      "excluded": 0,
      "robotsDisallowed": 0,
      "botBlocked": 0
    },
    "byStatus": {
      "2xx": 202,
      "3xx": 0,
      "4xx": 0,
      "5xx": 0,
      "failed": 0
    }
  },
  "notes": [],
  "headline": "Health 100 of 100: 202 of 202 crawled URLs have no errors. 103 HTML pages, 96 other documents; 0 kinds of error, 22 issue types in all.",
  "top": [
    {
      "issue": "schema-none",
      "name": "No structured data",
      "group": "ai",
      "severity": "notice",
      "site": false,
      "urls": 103,
      "of": {
        "set": "indexable",
        "urls": 103,
        "label": "indexable pages"
      },
      "share": 100.0,
      "text": "No structured data: 103 of 103 indexable pages. The indexable page has no structured data. Add JSON-LD for the page's type (Article, Product, Organization, BreadcrumbList…).",
      "example": "https://rankfield.dev/"
    },
    {
      "issue": "no-hsts",
      "name": "HSTS missing",
      "group": "other",
      "severity": "notice",
      "site": false,
      "urls": 103,
      "of": {
        "set": "htmlPages",
        "urls": 103,
        "label": "HTML pages"
      },
      "share": 100.0,
      "text": "HSTS missing: 103 of 103 HTML pages. The HTTPS page sends no Strict-Transport-Security header. Send HSTS (max-age of at least six months) once the whole site is HTTPS.",
      "example": "https://rankfield.dev/"
    },
    {
      "issue": "no-preferred-image",
      "name": "No preferred image",
      "group": "social",
      "severity": "notice",
      "site": false,
      "urls": 103,
      "of": {
        "set": "htmlPages",
        "urls": 103,
        "label": "HTML pages"
      },
      "share": 100.0,
      "text": "No preferred image: 103 of 103 HTML pages. The page names no preferred image (og:image, or an image in its structured data). Name the page's main image, so search and AI results can show it.",
      "example": "https://rankfield.dev/"
    },
    "…"
  ]
}

Notes#

  • We crawl as Rankfield-User, obey robots.txt and Crawl-delay on every request, run no JavaScript and never work around a firewall. Each audit starts at 5 fetches a second and climbs to 50 while the site answers quickly and cleanly; at the first 429, 5xx or slower answer it halves at once and honours Retry-After, never below 2 a second, for every site alike. A site that keeps rate-limiting us for most of ten minutes ends the audit early (stoppedBy: ["rateLimited"]) with what was found. An external link we could not check (a site we never fetch, robots.txt, bot protection) is never counted as broken. More: rankfield.dev/bot.

See also#

  • Full reference: https://rankfield.dev/docs/site-audit/audit-a-website
  • MCP tools on https://rankfield.dev/mcp: site_audit_start, site_audit_status, site_audit_summary, site_audit_issues, site_audit_issue_pages, site_audit_pages, site_audit_links, site_audit_redirects, site_audit_duplicates, site_audit_orphans, site_audit_templates, site_audit_sitemaps, site_audit_diff, site_audit_export, site_audit_page_filters, site_audit_link_filters, site_audit_structure, site_audit_log, site_audit_history, site_audit_list, site_audit_cancel
  • This page as Markdown: /endpoints/site-audit.md