# Site Map API

> Lists a website's URLs from what the site publishes and links: the sitemaps its robots.txt names, its RSS, Atom and JSON feeds, the links on its homepage and a short crawl of its pages. Every URL comes once, with the sources that listed it and its last-modified date where the site gives one.

Need the links of one page only? Use [Link Extraction](https://rankfield.dev/endpoints/html-links).

- Page: https://rankfield.dev/endpoints/map
- Full reference: https://rankfield.dev/docs/map/map-a-site

## Request

`GET https://rankfield.dev/v1/map?domain={domain}`

Send `Authorization: Bearer YOUR_API_KEY`. Get a free key at https://rankfield.dev/?tab=key#playground (no account).

```bash
curl "https://rankfield.dev/v1/map?domain=rankfield.dev&limit=200" \
  -H "Authorization: Bearer $RANKFIELD_API_KEY"
```

Response:

```json
{
  "domain": "rankfield.dev",
  "startUrl": "https://rankfield.dev/",
  "finalUrl": "https://rankfield.dev/",
  "status": 200,
  "complete": true,
  "stoppedBy": [],
  "counts": {
    "urls": 43,
    "returned": 43,
    "fromSitemaps": 8,
    "fromFeeds": 0,
    "fromLinks": 43,
    "withLastmod": 0,
    "duplicatesRemoved": 218,
    "outsideDomain": 25,
    "disallowedByRobots": 0,
    "invalid": 0,
    "notMatching": 0,
    "overCap": 0,
    "sitemapsNotRead": 0,
    "fetches": 12,
    "bytes": 528654
  },
  "sitemaps": [
    {
      "url": "https://rankfield.dev/sitemap.xml",
      "from": "robots.txt",
      "status": 200,
      "kind": "urlset",
      "urls": 8,
      "children": 0,
      "parent": null,
      "lastmod": null,
      "gzip": false,
      "bytes": 647,
      "truncated": false,
      "error": null
    }
  ],
  "feeds": [],
  "crawl": {
    "pagesFetched": 10,
    "maxPages": 10,
    "maxDepth": 1,
    "crawlDelay": null,
    "ok": 9,
    "failed": 0,
    "disallowed": 0,
    "homepage": {
      "error": null,
      "blocked": null,
      "links": 28,
      "nofollow": false
    },
    "robotsTxt": {
      "url": "https://rankfield.dev/robots.txt",
      "status": 200,
      "found": true,
      "sitemaps": 1
    }
  },
  "urls": [
    {
      "url": "https://rankfield.dev/",
      "sources": [
        "homepage",
        "sitemap",
        "crawl"
      ],
      "lastmod": null
    },
    {
      "url": "https://rankfield.dev/endpoints/",
      "sources": [
        "homepage",
        "crawl"
      ],
      "lastmod": null
    },
    {
      "url": "https://rankfield.dev/free-seo-tools/site-strength",
      "sources": [
        "homepage",
        "sitemap",
        "crawl"
      ],
      "lastmod": null
    },
    "…"
  ],
  "truncated": false,
  "timing": {
    "totalMs": 4049.8,
    "waitMs": 3812.9
  },
  "cached": false,
  "ageSeconds": 0
}
```

## Parameters

| Name | In | Type | Required | Default | Description |
|---|---|---|---|---|---|
| `domain` | query | string | required unless `url` |  | The website, e.g. `example.com`. Write `http://example.com` for a site without HTTPS. Send `domain` or `url`, not both. |
| `url` | query | string | no |  | Instead of `domain`: a full URL on the site. We map the whole site from its homepage. |
| `limit` | query | integer | no | `5000` | URLs to return, 1 to 50,000. We stop reading once we have this many. |
| `crawl` | query | integer | no | `10` | Pages to visit for more links besides the homepage, 0 to 100. |
| `subdomains` | query | boolean | no | `false` | `true` also keeps URLs on subdomains such as `blog.example.com`. |
| `search` | query | string | no |  | Only URLs containing these words, e.g. `pricing` or `blog 2026`. Matching pages are crawled first. |
| `sources` | query | string | no | `all three` | Comma-separated: `sitemaps`, `feeds`, `links`. |
| `maxAge` | query | integer | no | `3600` | Seconds a stored answer may be old (default 1 hour), up to 2,592,000. `0` reads the site fresh. |
| `timeout` | query | integer | no | `30` | Seconds the call may take, 1 to 60. |

## Response

| Field | Type | Description |
|---|---|---|
| `domain, startUrl, finalUrl, status` | string, integer | The site, the homepage we started from, the one that answered and its HTTP status. |
| `complete, stoppedBy` | boolean, array | `complete: false` when a limit stopped us: `limit`, `time`, `files`, `bytes`, `fetches` or `busy`. What we found still comes back. |
| `counts` | object | URLs found and returned, per source, with `lastmod`, duplicates removed, outside the domain, disallowed by robots.txt, and fetches and bytes. |
| `sitemaps, feeds` | array | Each file read: `url`, where we found it, `status`, `kind`, how many `urls`, `bytes`, `truncated`, `error`. |
| `crawl` | object | Pages fetched, the site's `crawlDelay`, failures, and the robots.txt we read. |
| `urls` | array | `{url, sources, lastmod}`. `sources` lists `homepage`, `sitemap`, `feed`, `crawl`; `lastmod` is the site's own date or `null`. |
| `truncated, timing` | boolean, object | `true` when the site has more URLs than `limit`; `totalMs` and `waitMs`. |
| `cached, ageSeconds` | boolean, integer | Whether the answer came from our store, and its age. |

## Errors

Errors come as `{"error":{"code":"…","message":"…","details":{…}}}` with the HTTP status.

| Status | Code | When |
|---|---|---|
| 400 | `invalid_input` | A parameter is missing, unknown or malformed. |
| 401 | `unauthorized` | The key is missing, unknown or revoked. |
| 403 | `robots_disallowed` | robots.txt disallows the homepage and nothing else on the site may be read (e.g. `Disallow: /`, or a robots.txt that answers 5xx). |
| 403 | `blocked` | The website's bot protection refused us. Reported at once and never worked around; `details` says which protection. Not counted as usage. |
| 403 | `site_excluded` | The site is on the exclusion list: Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us. |
| 422 | `address_not_public` | The host resolves to a private or internal address. |
| 429 | `busy` | Our fetch capacity is busy: the call was refused at once, waited 4 seconds without a turn, or its time limit ran out while it waited (then `details.reason` is `capacity`). A new job also gets it when the shared job queue is full (`Retry-After` 60). Not counted as usage. Retry after `Retry-After` seconds. |
| 500 | `internal` | Our bug. Retry. |
| 502 | `fetch_failed` | The website could not be reached (DNS, connection, TLS, more than 5 redirects or a bad one, a malformed answer); `details.reason` says which. |
| 503 | `unavailable` | The service is restarting or briefly unavailable. Retry after `Retry-After`. (A full fetch capacity answers `429` `busy`.) |
| 504 | `timeout` | The website did not answer within `timeout`. With `details.reason: "sitePace"`, the call waited its turn for this website instead, which we fetch gently, shared by all customers. Retry after `Retry-After`, or allow a longer `timeout`. |

## Price and limits

- Free; no per-call charge.
- No per-key limits. When the server is busy you get `429` `busy` with `Retry-After`.
- Only answered calls count as usage (plus the website errors listed below); our own errors (`500`, `503`), `429` `busy` and a `504` with `details.reason` `sitePace` don't.

## Up to 1,000 sites in one job

`POST /v1/map/jobs` with an `Idempotency-Key` header and `{"items":[{"domain":"example.com"}, …]}`. Each item takes the same parameters as one call, except `timeout`: items use the default, 30 seconds. Every item is checked first: if any is invalid, the whole request is a `400` that lists the first 20. A new job answers `202`; the same `Idempotency-Key` with a byte-identical body returns the same job (`200`, `Idempotency-Replayed: true`), and with any other body is a `409`.

Read `GET /v1/map/jobs/{id}/results?cursor=0&limit=100`: finished items as they complete, each `{index, ok, result}` or `{index, ok: false, error}`. Pass `cursor` back until `hasMore` is false. `POST /v1/map/jobs/{id}/cancel` stops a job. A job runs 16 items at a time; an item that meets `busy` waits and retries by itself. Results are kept 24 hours.

Send as many jobs as you like. Jobs from all keys share one queue: when 20,000 items are already waiting, a new job answers `429` `busy` with `Retry-After` (60 seconds).

Reference: https://rankfield.dev/docs/get-started/batch-jobs

## Notes

- URLs come in the order the site presents them, one spelling per page: tracking parameters and session ids removed, `rel=canonical` followed on crawled pages. We read up to 100 sitemap files, 5 feeds and 50 MB per call. An answer cut short by the time limit or a busy lane isn't reused: call again with a longer `timeout` and it starts over.
- We fetch as `Rankfield-User`, obey robots.txt on every request and redirect, run no JavaScript, use no cookies or proxies, and never work around a firewall. We fetch each site gently, shared by all customers. More: [rankfield.dev/bot](https://rankfield.dev/bot).

## MCP

Tools on https://rankfield.dev/mcp: `map_site`, `start_site_map_job`, `get_site_map_results`.