> ## Documentation index
> Fetch the complete documentation index at: https://rankfield.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Audit a website

> Start a technical SEO and AEO audit of a website: we crawl its pages and check each one against 213 rules.

`POST https://rankfield.dev/v1/site-audit`

```bash cURL
curl -X POST 'https://rankfield.dev/v1/site-audit' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"domain":"shop.example"}'
```

```python Python
import requests

response = requests.post(
    "https://rankfield.dev/v1/site-audit",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"domain": "shop.example"},
)
print(response.json())
```

```javascript JavaScript
const response = await fetch("https://rankfield.dev/v1/site-audit", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({"domain":"shop.example"}),
});
console.log(await response.json());
```

```json 200
{
  "audit": {
    "id": "1414774bd14c096e08ec5b95fc183bcf",
    "domain": "https://shop.example",
    "status": "queued",
    "createdAt": "2026-10-28T20:45:39Z",
    "startedAt": null,
    "finishedAt": null,
    "expiresAt": null,
    "stoppedBy": [],
    "error": null,
    "options": {
      "domain": "https://shop.example",
      "maxPages": 10000,
      "subdomains": false,
      "sources": [
        "links",
        "sitemaps"
      ],
      "urls": [],
      "mode": "full",
      "sample": false,
      "samplePerTemplate": 500,
      "checkExternal": true,
      "checkResources": true,
      "previous": null,
      "speed": null
    },
    "progress": {
      "phase": "queued"
    },
    "counts": {}
  }
}
```

```json 401
{
  "error": {
    "code": "unauthorized",
    "message": "A valid API key is required"
  }
}
```

The call answers at once (`200`, `status: "queued"`) with the audit's `id`; the crawl runs in the background. [Follow it](/docs/site-audit/follow-an-audit), then [read the results](/docs/site-audit/read-the-results), during the crawl or after. Big sites take time: a site that answers fast is crawled at up to 50 pages a second, a slow or struggling one at 2.

## Parameters

Send a JSON object.

### Headers

| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| `Authorization` | string | Yes | – | `Bearer` followed by your API key. See [Authentication](/docs/get-started/authentication). |

### Body

| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| `domain` | string | Yes, or `url` | – | The site: a host such as `example.com`, or a URL whose host is used. `https` unless written `http://`. |
| `maxPages` | integer | No | `10000` | HTML pages to fetch at most, 1 to 5,000,000. |
| `subdomains` | boolean | No | `false` | Also crawl subdomains such as `blog.example.com`. |
| `sources` | string[] | No | `["links","sitemaps"]` | Where URLs come from: `links` (a crawl from the homepage), `sitemaps` (robots.txt's sitemaps, else `/sitemap.xml`), `urls` (your `urls` list). |
| `urls` | string[] | No | – | Up to 10,000 URLs of the site you know exist, from Search Console, analytics or logs: fetched, and used to find orphan pages. |
| `mode` | string | No | `full` | `full`: content and links. `status`: only the status of every URL in the sitemaps and `urls` (no content): broken pages and redirect chains in minutes. |
| `sample` | boolean | No | `false` | Audit at most `samplePerTemplate` pages per URL template and estimate the rest, for very large sites. |
| `samplePerTemplate` | integer | No | `500` | Pages per template when `sample` is true, 10 to 10,000. |
| `checkExternal` | boolean | No | `true` | Check each distinct external link target once, at most 2,000 per audit. |
| `checkResources` | boolean | No | `true` | Check each distinct image, script and stylesheet of the site once, at most 2,000 per audit. |
| `previous` | string | No | – | The `id` of an earlier finished audit of the same site: unchanged pages reuse its records, and [diff](/docs/site-audit/read-the-results) compares the two. |
| `speed` | integer | No | – | The most fetches a second, 2 to 50. By default the speed adapts to the site, up to 50. |

## Response

`200` with the audit, the same object [Follow an audit](/docs/site-audit/follow-an-audit) reads:

| Field | Type | Description |
|---|---|---|
| `audit.id` | string | The audit's id: keep it to read the audit. |
| `audit.domain` | string | The site, as we crawl it (`https://example.com`). |
| `audit.status` | string | `queued`, `running`, `postPass` (the site-wide checks), `done`, `canceled` or `failed`. |
| `audit.options` | object | The parameters as we read them, defaults filled in. |
| `audit.progress`, `audit.counts` | object | Pages fetched so far, the speed and the issues found: see [Follow an audit](/docs/site-audit/follow-an-audit). |
| `audit.createdAt`, `audit.startedAt`, `audit.finishedAt`, `audit.expiresAt` | string \| null | When it was created, started and finished, and when its results are deleted (30 days after it ends), in UTC. |
| `audit.stoppedBy` | string[] | Why it ended early: `maxPages`, `siteErrors`, `rateLimited` (the site kept rate-limiting us for most of ten minutes), `canceled` or `time`; empty when it crawled everything it found. |
| `audit.queuePosition` | integer | Its place in the queue, while `queued`. |

## How we crawl

We fetch as `Rankfield-User` (`Mozilla/5.0 (compatible; Rankfield-User/1.0; +https://rankfield.dev/bot)`), as plain HTML without running JavaScript.

- **Speed:** each audit adapts to its site. It starts at 5 fetches a second and climbs, up to 50, while the site answers quickly and without errors. At the first sign of strain (a `429`, a `503`, another `5xx`, a connection error or a slower answer) it halves at once and honours `Retry-After`, never going below 2 a second, and pauses for a minute after three `5xx` in a row. A site that keeps failing (more than half of its last 200 answers) stops the audit with `stoppedBy: ["siteErrors"]`; what was found is kept. The limits are the same for every site.
- **A site that rate-limits us:** each `429` or `503` halves the speed, and we leave the site alone for as long as its `Retry-After` asks (30 seconds when it gives none, 5 minutes at most). While we wait, [progress](/docs/site-audit/follow-an-audit) shows `siteHold` and gives no time estimate. A site that holds us for most of ten minutes stops the audit with `stoppedBy: ["rateLimited"]`; what was found is kept, the site-wide checks run on it, and the summary's `notes` say so.
- **robots.txt and `Crawl-delay`:** obeyed on every request, for `Rankfield-User`, or for `RankfieldBot` when the file has none for it, or `*` when it has neither.
- **Redirects:** not followed blindly. Each hop is fetched and recorded on its own, so chains and loops are exact, and robots.txt is checked on every hop.
- **Other sites:** a link, redirect or resource that leaves the site gets one status check (`checkExternal`) and is never crawled. Cloudflare's own `/cdn-cgi/` paths are never fetched.
- **Not checked is never broken:** an external URL we did not check is never counted as a broken link: a site we never fetch (below), one robots.txt disallows, or one whose bot protection refused our check (a `401`, `403` or `429`, or a challenge page). They are listed with `status=unchecked` and counted in `counts.inventory.externalUnchecked`.
- **Never:** cookies, logins, JavaScript, proxies or other bots' names. A firewall page is reported, never worked around. Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages and sites whose owners asked us are never fetched.
- **What we keep:** each page's record (status, headers of interest, title, description and h1 cut to 200 characters, counts, hashes) and its links; never the page's text. Results are kept 30 days after the audit ends.

More: [Rankfield crawlers](/bot).

## Errors

Errors are `{"error": {"code", "message", "details"}}`. Branch on `code`; `message` is for people.

| Status | Code | When |
|---|---|---|
| 400 | `invalid_input` | A parameter is missing, unknown or malformed (unknown parameters are refused), or the body is over 1 MB. |
| 401 | `unauthorized` | No valid API key. |
| 403 | `site_excluded` | The site is on the exclusion list. |
| 404 | `not_found` | No audit with this id for this key (only the key that started an audit can read it), an unknown issue, or a page that is not in the audit. |
| 422 | `address_not_public` | The host resolves to a private or internal address. |
| 429 | `busy` | The audit queue is full: 64 audits already wait (`details.reason` `capacity`). Not counted as usage. Retry after `Retry-After` seconds. |
| 500 | `internal` | Our bug. Retry. |
| 503 | `unavailable` | New audits are paused while the service is short of storage, or it is restarting. Retry after `Retry-After`. |

## Limits

No per-key limits. Up to 8 audits crawl at once and the rest wait in order; when 64 already wait, a new audit answers `429` `busy` with `Retry-After`. At most one audit over 1,000,000 pages runs at a time and 2 more wait; another one answers `429` `busy` with `Retry-After`, and smaller audits can start ahead of a waiting one. An audit fetches at most 5,000,000 pages.

## MCP

On the [MCP server](/docs/agents/mcp-server), the tool for this is `site_audit_start`.