Site Map API

Lists a website's URLs from what the site publishes and links: the sitemaps its robots.txt names, its RSS, Atom and JSON feeds, the links on its homepage and a short crawl of its pages. Every URL comes once, with the sources that listed it and its last-modified date where the site gives one.

Need the links of one page only? Use Link Extraction.

GEThttps://rankfield.dev/v1/map?domain={domain}

Send Authorization: Bearer YOUR_API_KEY. Get a free key; no account needed.

curl "https://rankfield.dev/v1/map?domain=rankfield.dev&limit=200" \
  -H "Authorization: Bearer $RANKFIELD_API_KEY"

import os
import requests

response = requests.get(
    "https://rankfield.dev/v1/map?domain=rankfield.dev&limit=200",
    headers={"Authorization": f"Bearer {os.environ['RANKFIELD_API_KEY']}"},
)
print(response.json())

const response = await fetch("https://rankfield.dev/v1/map?domain=rankfield.dev&limit=200", {
  headers: {
    Authorization: `Bearer ${process.env.RANKFIELD_API_KEY}`,
  },
});
console.log(await response.json());

Response

{
  "domain": "rankfield.dev",
  "startUrl": "https://rankfield.dev/",
  "finalUrl": "https://rankfield.dev/",
  "status": 200,
  "complete": true,
  "stoppedBy": [],
  "counts": {
    "urls": 43,
    "returned": 43,
    "fromSitemaps": 8,
    "fromFeeds": 0,
    "fromLinks": 43,
    "withLastmod": 0,
    "duplicatesRemoved": 218,
    "outsideDomain": 25,
    "disallowedByRobots": 0,
    "invalid": 0,
    "notMatching": 0,
    "overCap": 0,
    "sitemapsNotRead": 0,
    "fetches": 12,
    "bytes": 528654
  },
  "sitemaps": [
    {
      "url": "https://rankfield.dev/sitemap.xml",
      "from": "robots.txt",
      "status": 200,
      "kind": "urlset",
      "urls": 8,
      "children": 0,
      "parent": null,
      "lastmod": null,
      "gzip": false,
      "bytes": 647,
      "truncated": false,
      "error": null
    }
  ],
  "feeds": [],
  "crawl": {
    "pagesFetched": 10,
    "maxPages": 10,
    "maxDepth": 1,
    "crawlDelay": null,
    "ok": 9,
    "failed": 0,
    "disallowed": 0,
    "homepage": {
      "error": null,
      "blocked": null,
      "links": 28,
      "nofollow": false
    },
    "robotsTxt": {
      "url": "https://rankfield.dev/robots.txt",
      "status": 200,
      "found": true,
      "sitemaps": 1
    }
  },
  "urls": [
    {
      "url": "https://rankfield.dev/",
      "sources": [
        "homepage",
        "sitemap",
        "crawl"
      ],
      "lastmod": null
    },
    {
      "url": "https://rankfield.dev/endpoints/",
      "sources": [
        "homepage",
        "crawl"
      ],
      "lastmod": null
    },
    {
      "url": "https://rankfield.dev/free-seo-tools/site-strength",
      "sources": [
        "homepage",
        "sitemap",
        "crawl"
      ],
      "lastmod": null
    },
    "…"
  ],
  "truncated": false,
  "timing": {
    "totalMs": 4049.8,
    "waitMs": 3812.9
  },
  "cached": false,
  "ageSeconds": 0
}

Parameters#

Name	Type	Description
`domain`query, required unless `url`	string	The website, e.g. `example.com`. Write `http://example.com` for a site without HTTPS. Send `domain` or `url`, not both.
`url`query, optional	string	Instead of `domain`: a full URL on the site. We map the whole site from its homepage.
`limit`query, optional	integerdefault `5000`	URLs to return, 1 to 50,000. We stop reading once we have this many.
`crawl`query, optional	integerdefault `10`	Pages to visit for more links besides the homepage, 0 to 100.
`subdomains`query, optional	booleandefault `false`	`true` also keeps URLs on subdomains such as `blog.example.com`.
`search`query, optional	string	Only URLs containing these words, e.g. `pricing` or `blog 2026`. Matching pages are crawled first.
`sources`query, optional	stringdefault `all three`	Comma-separated: `sitemaps`, `feeds`, `links`.
`maxAge`query, optional	integerdefault `3600`	Seconds a stored answer may be old (default 1 hour), up to 2,592,000. `0` reads the site fresh.
`timeout`query, optional	integerdefault `30`	Seconds the call may take, 1 to 60.

Response#

Field	Type	Description
`domain` `startUrl` `finalUrl` `status`	string, integer	The site, the homepage we started from, the one that answered and its HTTP status.
`complete` `stoppedBy`	boolean, array	`complete: false` when a limit stopped us: `limit`, `time`, `files`, `bytes`, `fetches` or `busy`. What we found still comes back.
`counts`	object	URLs found and returned, per source, with `lastmod`, duplicates removed, outside the domain, disallowed by robots.txt, and fetches and bytes.
`sitemaps` `feeds`	array	Each file read: `url`, where we found it, `status`, `kind`, how many `urls`, `bytes`, `truncated`, `error`.
`crawl`	object	Pages fetched, the site's `crawlDelay`, failures, and the robots.txt we read.
`urls`	array	`{url, sources, lastmod}`. `sources` lists `homepage`, `sitemap`, `feed`, `crawl`; `lastmod` is the site's own date or `null`.
`truncated` `timing`	boolean, object	`true` when the site has more URLs than `limit`; `totalMs` and `waitMs`.
`cached` `ageSeconds`	boolean, integer	Whether the answer came from our store, and its age.

Errors#

Errors come as {"error":{"code":"…","message":"…","details":{…}}} with the HTTP status.

Status	Code	When
`400`	`invalid_input`	A parameter is missing, unknown or malformed.
`401`	`unauthorized`	The key is missing, unknown or revoked.
`403`	`robots_disallowed`	robots.txt disallows the homepage and nothing else on the site may be read (e.g. `Disallow: /`, or a robots.txt that answers 5xx).
`403`	`blocked`	The website's bot protection refused us. Reported at once and never worked around; `details` says which protection. Not counted as usage.
`403`	`site_excluded`	The site is on the exclusion list: Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us.
`422`	`address_not_public`	The host resolves to a private or internal address.
`429`	`busy`	Our fetch capacity is busy: the call was refused at once, waited 4 seconds without a turn, or its time limit ran out while it waited (then `details.reason` is `capacity`). A new job also gets it when the shared job queue is full (`Retry-After` 60). Not counted as usage. Retry after `Retry-After` seconds.
`500`	`internal`	Our bug. Retry.
`502`	`fetch_failed`	The website could not be reached (DNS, connection, TLS, more than 5 redirects or a bad one, a malformed answer); `details.reason` says which.
`503`	`unavailable`	The service is restarting or briefly unavailable. Retry after `Retry-After`. (A full fetch capacity answers `429` `busy`.)
`504`	`timeout`	The website did not answer within `timeout`. With `details.reason: "sitePace"`, the call waited its turn for this website instead, which we fetch gently, shared by all customers. Retry after `Retry-After`, or allow a longer `timeout`.

Price and limits#

Free; no per-call charge.
No per-key limits. When the server is busy you get 429 busy with Retry-After.
Only answered calls count as usage (plus the website errors listed below); our own errors (500, 503), 429 busy and a 504 with details.reason sitePace don't.

Up to 1,000 sites in one job#

POST /v1/map/jobs with an Idempotency-Key header and {"items":[{"domain":"example.com"}, …]}. Each item takes the same parameters as one call, except timeout: items use the default, 30 seconds. Every item is checked first: if any is invalid, the whole request is a 400 that lists the first 20. A new job answers 202; the same Idempotency-Key with a byte-identical body returns the same job (200, Idempotency-Replayed: true), and with any other body is a 409.

Read GET /v1/map/jobs/{id}/results?cursor=0&limit=100: finished items as they complete, each {index, ok, result} or {index, ok: false, error}. Pass cursor back until hasMore is false. POST /v1/map/jobs/{id}/cancel stops a job. A job runs 16 items at a time; an item that meets busy waits and retries by itself. Results are kept 24 hours.

Send as many jobs as you like. Jobs from all keys share one queue: when 20,000 items are already waiting, a new job answers 429 busy with Retry-After (60 seconds).

Reference: https://rankfield.dev/docs/get-started/batch-jobs

Notes#

URLs come in the order the site presents them, one spelling per page: tracking parameters and session ids removed, rel=canonical followed on crawled pages. We read up to 100 sitemap files, 5 feeds and 50 MB per call. An answer cut short by the time limit or a busy lane isn't reused: call again with a longer timeout and it starts over.
We fetch as Rankfield-User, obey robots.txt on every request and redirect, run no JavaScript, use no cookies or proxies, and never work around a firewall. We fetch each site gently, shared by all customers. More: rankfield.dev/bot.