> ## Documentation index
> Fetch the complete documentation index at: https://rankfield.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Map a website

> List a website's URLs from its sitemaps, feeds, homepage links and a short crawl, each with its sources and last-modified date.

`GET https://rankfield.dev/v1/map?domain={domain}`

```bash cURL
curl 'https://rankfield.dev/v1/map?domain=rankfield.dev&limit=200' \
  -H 'Authorization: Bearer YOUR_API_KEY'
```

```python Python
import requests

response = requests.get(
    "https://rankfield.dev/v1/map?domain=rankfield.dev&limit=200",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
)
print(response.json())
```

```javascript JavaScript
const response = await fetch("https://rankfield.dev/v1/map?domain=rankfield.dev&limit=200", {
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
console.log(await response.json());
```

```json 200
{
  "domain": "rankfield.dev",
  "startUrl": "https://rankfield.dev/",
  "finalUrl": "https://rankfield.dev/",
  "status": 200,
  "complete": true,
  "stoppedBy": [],
  "counts": {
    "urls": 43,
    "returned": 43,
    "fromSitemaps": 8,
    "fromFeeds": 0,
    "fromLinks": 43,
    "withLastmod": 0,
    "duplicatesRemoved": 218,
    "outsideDomain": 25,
    "disallowedByRobots": 0,
    "invalid": 0,
    "notMatching": 0,
    "overCap": 0,
    "sitemapsNotRead": 0,
    "fetches": 12,
    "bytes": 528654
  },
  "sitemaps": [
    {
      "url": "https://rankfield.dev/sitemap.xml",
      "from": "robots.txt",
      "status": 200,
      "kind": "urlset",
      "urls": 8,
      "children": 0,
      "parent": null,
      "lastmod": null,
      "gzip": false,
      "bytes": 647,
      "truncated": false,
      "error": null
    }
  ],
  "feeds": [],
  "crawl": {
    "pagesFetched": 10,
    "maxPages": 10,
    "maxDepth": 1,
    "crawlDelay": null,
    "ok": 9,
    "failed": 0,
    "disallowed": 0,
    "homepage": {
      "error": null,
      "blocked": null,
      "links": 28,
      "nofollow": false
    },
    "robotsTxt": {
      "url": "https://rankfield.dev/robots.txt",
      "status": 200,
      "found": true,
      "sitemaps": 1
    }
  },
  "urls": [
    {
      "url": "https://rankfield.dev/",
      "sources": [
        "homepage",
        "sitemap",
        "crawl"
      ],
      "lastmod": null
    },
    {
      "url": "https://rankfield.dev/endpoints/",
      "sources": [
        "homepage",
        "crawl"
      ],
      "lastmod": null
    },
    {
      "url": "https://rankfield.dev/free-seo-tools/site-strength",
      "sources": [
        "homepage",
        "sitemap",
        "crawl"
      ],
      "lastmod": null
    },
    "…"
  ],
  "truncated": false,
  "timing": {
    "totalMs": 4049.8,
    "waitMs": 3812.9
  },
  "cached": false,
  "ageSeconds": 0
}
```

```json 401
{
  "error": {
    "code": "unauthorized",
    "message": "A valid API key is required"
  }
}
```

Every URL comes once, in one clean spelling, and says which sources listed it and when the site last changed it, where the site says so. URLs come in the order the site presents them: the homepage, its links, each sitemap in file order, the feeds, then pages found by the crawl. The same site gives the same order every time.

Send the parameters as a query string with `GET`, or as a JSON object (up to 1 MB) with `POST` to the same path. For many domains, [start a job](/docs/map/start-a-job).

## Parameters

### Headers

| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| `Authorization` | string | Yes | – | `Bearer` followed by your API key. See [Authentication](/docs/get-started/authentication). |

### Query parameters

| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| `domain` | string | Yes, or `url` | – | The website, such as `example.com`. `www.example.com` is read as given. Write `http://example.com` for a site without HTTPS. Send `domain` or `url`, not both. |
| `url` | string | Instead of `domain` | – | A full URL on the site; the whole site is mapped from its homepage. |
| `limit` | integer | No | `5000` | How many URLs to return, 1–50,000. Reading stops once there are this many. |
| `crawl` | integer | No | `10` | How many of the site's pages to visit for more links, besides the homepage, 0–100. |
| `subdomains` | boolean | No | `false` | `true` also keeps URLs on subdomains such as `blog.example.com`. |
| `search` | string | No | – | Only URLs containing these words, such as `pricing` or `blog 2026`. Matching pages are crawled first. |
| `sources` | string | No | `sitemaps,feeds,links` | Where to look, comma-separated: `sitemaps`, `feeds`, `links` (homepage links and the crawl). |
| `maxAge` | integer | No | `3600` | How old a stored answer may be, in seconds, up to 2,592,000. `0` maps fresh. An answer cut short by time or a busy lane is never reused. |
| `timeout` | integer | No | `30` | How long the call may take, in seconds, 1–60. |

## Response

| Field | Type | Description |
|---|---|---|
| `domain`, `startUrl` | string | The site, and the homepage address we started from. |
| `finalUrl`, `status` | string, integer \| null | The homepage after redirects, and its HTTP status. |
| `complete` | boolean | `true` when every source was read to the end. `false` when a limit stopped us; everything found is still returned. |
| `stoppedBy` | string[] | What stopped us: `limit`, `time`, `files` (100 sitemap files), `bytes` (50 MB), `fetches` or `busy`. |
| `counts` | object | `urls`, `returned`, `fromSitemaps`, `fromFeeds`, `fromLinks`, `withLastmod`, `duplicatesRemoved`, `outsideDomain`, `disallowedByRobots`, `invalid`, `notMatching`, `overCap`, `sitemapsNotRead`, `fetches`, `bytes`. |
| `sitemaps` | array | Each sitemap file: `url`, `from` (`robots.txt`, `homepage`, `guess`, `index`), `status`, `kind`, `urls`, `children`, `parent`, `lastmod`, `gzip`, `bytes`, `truncated`, `error`. |
| `feeds` | array | Each feed: `url`, `from`, `status`, `kind` (`rss`, `rdf`, `atom`, `jsonFeed`), `urls`, `bytes`, `truncated`, `error`. |
| `crawl` | object | `pagesFetched`, `maxPages`, `maxDepth`, the site's `crawlDelay`, `ok`, `failed`, `disallowed`, the `homepage` and the `robotsTxt` read. |
| `urls` | array | `{url, sources, lastmod}`. `sources`: `homepage`, `sitemap`, `feed`, `crawl`. `lastmod` is the site's own date (UTC) or `null`. |
| `truncated` | boolean | `true` when the site has more URLs than `limit`. |
| `timing` | object | `totalMs`, and `waitMs`: time spent waiting for this site's pace and for our fetch capacity. |
| `cached`, `ageSeconds` | boolean, integer | Whether this answer came from our store, and how old it is. |

## One spelling per page

Scheme and host are lowercased; default ports, `#fragments` and tracking parameters (`utm_*`, `gclid`, `fbclid`, `msclkid`, Mailchimp, HubSpot, Matomo and similar) are removed; `./` and `../` are resolved. `www.example.com` and `example.com` count as one site, written the way its homepage writes itself, and `http://` links on an HTTPS site become `https://`. When a crawled page names a `rel=canonical`, that URL is listed.

## Large sites

We read up to 100 sitemap files, 5 feeds and 50 MB per call, follow sitemap indexes 3 levels deep, and unzip `.gz` sitemaps. A site with many sitemap files can take longer than one call: the answer then has `complete: false` and `stoppedBy: ["time"]`. Call again with a longer `timeout`: the call starts over.

## How we fetch

We fetch as `Rankfield-User` (`Mozilla/5.0 (compatible; Rankfield-User/1.0; +https://rankfield.dev/bot)`), as plain HTML without running JavaScript.

- **robots.txt:** obeyed on every request and redirect, following the rules for `Rankfield-User`, or for `RankfieldBot` when the file has none for it, or `*` when it has neither. Sitemaps and pages it disallows are counted in `disallowedByRobots` and never fetched.
- **Firewalls:** a challenge, CAPTCHA or block page is reported in the answer, never worked around and never retried.
- **Never:** cookies, logins, JavaScript, proxies or other bots' names.
- **Excluded sites:** Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us are never fetched.
- **Pace:** each website is fetched gently, shared by all customers, slower when its robots.txt sets a `Crawl-delay` or after a slow answer. Many URLs on one site go at that site's pace.

Repeat calls to the same website within about a minute can wait their turn for that site's slot. If the wait runs past the call's time limit (30 seconds by default), the call ends with `504` `timeout` and `details.reason` `sitePace`. It doesn't count as usage: retry after `Retry-After`, or allow a longer `timeout`.

More: [Rankfield crawlers](/bot).

## Errors

Errors are `{"error": {"code", "message", "details"}}`. Branch on `code`; `message` is for people.

| Status | Code | When |
|---|---|---|
| 400 | `invalid_input` | A parameter is missing, unknown or malformed. |
| 401 | `unauthorized` | No valid API key. |
| 403 | `robots_disallowed` | robots.txt disallows the homepage and nothing else on the site may be read (for example `Disallow: /`, or a robots.txt that answers with a server error). |
| 403 | `site_excluded` | The site is on the exclusion list. |
| 403 | `blocked` | The website's bot protection refused us. Reported at once and never worked around; `details` says which protection. Not counted as usage. |
| 422 | `address_not_public` | The host resolves to a private or internal address. |
| 429 | `busy` | Our fetch capacity is busy: the call was refused at once, waited 4 seconds without a turn, or its time limit ran out while it waited (then `details.reason` is `capacity`). Not counted as usage. Retry after `Retry-After` seconds. |
| 500 | `internal` | Our bug. Retry. |
| 502 | `fetch_failed` | The website could not be reached at all (DNS, connection, TLS). |
| 503 | `unavailable` | The service is restarting or briefly unavailable. Retry after `Retry-After`. (A full fetch capacity answers `429` `busy`.) |
| 504 | `timeout` | The website didn't answer within the call's time limit. When `details.reason` is `sitePace`, the call waited its turn for this website instead, which we fetch gently, shared by all customers: retry after `Retry-After`, or allow a longer `timeout`. |

A sitemap or page that fails is reported in its own entry, not as an error. Only answered calls and the `502` and website-side `504` errors count as usage; refusals, our own errors (`500`, `503`), `429` `busy` and a `504` with `details.reason` `sitePace` don't.

## Limits

No per-key limits. When the server is busy you get `429` `busy` with `Retry-After`. Waiting calls are served in turn across keys, so a key with a few calls is never stuck behind one with thousands.

## MCP

On the [MCP server](/docs/agents/mcp-server), the tool for this is `map_site`.