Map a website

List a website's URLs from its sitemaps, feeds, homepage links and a short crawl, each with its sources and last-modified date.

GEThttps://rankfield.dev/v1/map?domain={domain}

curl 'https://rankfield.dev/v1/map?domain=rankfield.dev&limit=200' \
  -H 'Authorization: Bearer YOUR_API_KEY'

import requests

response = requests.get(
    "https://rankfield.dev/v1/map?domain=rankfield.dev&limit=200",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
)
print(response.json())

const response = await fetch("https://rankfield.dev/v1/map?domain=rankfield.dev&limit=200", {
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
console.log(await response.json());

{
  "domain": "rankfield.dev",
  "startUrl": "https://rankfield.dev/",
  "finalUrl": "https://rankfield.dev/",
  "status": 200,
  "complete": true,
  "stoppedBy": [],
  "counts": {
    "urls": 43,
    "returned": 43,
    "fromSitemaps": 8,
    "fromFeeds": 0,
    "fromLinks": 43,
    "withLastmod": 0,
    "duplicatesRemoved": 218,
    "outsideDomain": 25,
    "disallowedByRobots": 0,
    "invalid": 0,
    "notMatching": 0,
    "overCap": 0,
    "sitemapsNotRead": 0,
    "fetches": 12,
    "bytes": 528654
  },
  "sitemaps": [
    {
      "url": "https://rankfield.dev/sitemap.xml",
      "from": "robots.txt",
      "status": 200,
      "kind": "urlset",
      "urls": 8,
      "children": 0,
      "parent": null,
      "lastmod": null,
      "gzip": false,
      "bytes": 647,
      "truncated": false,
      "error": null
    }
  ],
  "feeds": [],
  "crawl": {
    "pagesFetched": 10,
    "maxPages": 10,
    "maxDepth": 1,
    "crawlDelay": null,
    "ok": 9,
    "failed": 0,
    "disallowed": 0,
    "homepage": {
      "error": null,
      "blocked": null,
      "links": 28,
      "nofollow": false
    },
    "robotsTxt": {
      "url": "https://rankfield.dev/robots.txt",
      "status": 200,
      "found": true,
      "sitemaps": 1
    }
  },
  "urls": [
    {
      "url": "https://rankfield.dev/",
      "sources": [
        "homepage",
        "sitemap",
        "crawl"
      ],
      "lastmod": null
    },
    {
      "url": "https://rankfield.dev/endpoints/",
      "sources": [
        "homepage",
        "crawl"
      ],
      "lastmod": null
    },
    {
      "url": "https://rankfield.dev/free-seo-tools/site-strength",
      "sources": [
        "homepage",
        "sitemap",
        "crawl"
      ],
      "lastmod": null
    },
    "…"
  ],
  "truncated": false,
  "timing": {
    "totalMs": 4049.8,
    "waitMs": 3812.9
  },
  "cached": false,
  "ageSeconds": 0
}

{
  "error": {
    "code": "unauthorized",
    "message": "A valid API key is required"
  }
}

Every URL comes once, in one clean spelling, and says which sources listed it and when the site last changed it, where the site says so. URLs come in the order the site presents them: the homepage, its links, each sitemap in file order, the feeds, then pages found by the crawl. The same site gives the same order every time.

Send the parameters as a query string with GET, or as a JSON object (up to 1 MB) with POST to the same path. For many domains, start a job.

Parameters

Headers

Name	Type	Required	Default	Description
`Authorization`	string	Yes	–	`Bearer` followed by your API key. See Authentication.

Query parameters

Name	Type	Required	Default	Description
`domain`	string	Yes, or `url`	–	The website, such as `example.com`. `www.example.com` is read as given. Write `http://example.com` for a site without HTTPS. Send `domain` or `url`, not both.
`url`	string	Instead of `domain`	–	A full URL on the site; the whole site is mapped from its homepage.
`limit`	integer	No	`5000`	How many URLs to return, 1–50,000. Reading stops once there are this many.
`crawl`	integer	No	`10`	How many of the site's pages to visit for more links, besides the homepage, 0–100.
`subdomains`	boolean	No	`false`	`true` also keeps URLs on subdomains such as `blog.example.com`.
`search`	string	No	–	Only URLs containing these words, such as `pricing` or `blog 2026`. Matching pages are crawled first.
`sources`	string	No	`sitemaps,feeds,links`	Where to look, comma-separated: `sitemaps`, `feeds`, `links` (homepage links and the crawl).
`maxAge`	integer	No	`3600`	How old a stored answer may be, in seconds, up to 2,592,000. `0` maps fresh. An answer cut short by time or a busy lane is never reused.
`timeout`	integer	No	`30`	How long the call may take, in seconds, 1–60.

Response

Field	Type	Description
`domain`, `startUrl`	string	The site, and the homepage address we started from.
`finalUrl`, `status`	string, integer \| null	The homepage after redirects, and its HTTP status.
`complete`	boolean	`true` when every source was read to the end. `false` when a limit stopped us; everything found is still returned.
`stoppedBy`	string[]	What stopped us: `limit`, `time`, `files` (100 sitemap files), `bytes` (50 MB), `fetches` or `busy`.
`counts`	object	`urls`, `returned`, `fromSitemaps`, `fromFeeds`, `fromLinks`, `withLastmod`, `duplicatesRemoved`, `outsideDomain`, `disallowedByRobots`, `invalid`, `notMatching`, `overCap`, `sitemapsNotRead`, `fetches`, `bytes`.
`sitemaps`	array	Each sitemap file: `url`, `from` (`robots.txt`, `homepage`, `guess`, `index`), `status`, `kind`, `urls`, `children`, `parent`, `lastmod`, `gzip`, `bytes`, `truncated`, `error`.
`feeds`	array	Each feed: `url`, `from`, `status`, `kind` (`rss`, `rdf`, `atom`, `jsonFeed`), `urls`, `bytes`, `truncated`, `error`.
`crawl`	object	`pagesFetched`, `maxPages`, `maxDepth`, the site's `crawlDelay`, `ok`, `failed`, `disallowed`, the `homepage` and the `robotsTxt` read.
`urls`	array	`{url, sources, lastmod}`. `sources`: `homepage`, `sitemap`, `feed`, `crawl`. `lastmod` is the site's own date (UTC) or `null`.
`truncated`	boolean	`true` when the site has more URLs than `limit`.
`timing`	object	`totalMs`, and `waitMs`: time spent waiting for this site's pace and for our fetch capacity.
`cached`, `ageSeconds`	boolean, integer	Whether this answer came from our store, and how old it is.

One spelling per page

Scheme and host are lowercased; default ports, #fragments and tracking parameters (utm_*, gclid, fbclid, msclkid, Mailchimp, HubSpot, Matomo and similar) are removed; ./ and ../ are resolved. www.example.com and example.com count as one site, written the way its homepage writes itself, and http:// links on an HTTPS site become https://. When a crawled page names a rel=canonical, that URL is listed.

Large sites

We read up to 100 sitemap files, 5 feeds and 50 MB per call, follow sitemap indexes 3 levels deep, and unzip .gz sitemaps. A site with many sitemap files can take longer than one call: the answer then has complete: false and stoppedBy: ["time"]. Call again with a longer timeout: the call starts over.

How we fetch

We fetch as Rankfield-User (Mozilla/5.0 (compatible; Rankfield-User/1.0; +https://rankfield.dev/bot)), as plain HTML without running JavaScript.

robots.txt: obeyed on every request and redirect, following the rules for Rankfield-User, or for RankfieldBot when the file has none for it, or * when it has neither. Sitemaps and pages it disallows are counted in disallowedByRobots and never fetched.
Firewalls: a challenge, CAPTCHA or block page is reported in the answer, never worked around and never retried.
Never: cookies, logins, JavaScript, proxies or other bots' names.
Excluded sites: Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us are never fetched.
Pace: each website is fetched gently, shared by all customers, slower when its robots.txt sets a Crawl-delay or after a slow answer. Many URLs on one site go at that site's pace.

Repeat calls to the same website within about a minute can wait their turn for that site's slot. If the wait runs past the call's time limit (30 seconds by default), the call ends with 504 timeout and details.reason sitePace. It doesn't count as usage: retry after Retry-After, or allow a longer timeout.

More: Rankfield crawlers.

Errors

Errors are {"error": {"code", "message", "details"}}. Branch on code; message is for people.

Status	Code	When
400	`invalid_input`	A parameter is missing, unknown or malformed.
401	`unauthorized`	No valid API key.
403	`robots_disallowed`	robots.txt disallows the homepage and nothing else on the site may be read (for example `Disallow: /`, or a robots.txt that answers with a server error).
403	`site_excluded`	The site is on the exclusion list.
403	`blocked`	The website's bot protection refused us. Reported at once and never worked around; `details` says which protection. Not counted as usage.
422	`address_not_public`	The host resolves to a private or internal address.
429	`busy`	Our fetch capacity is busy: the call was refused at once, waited 4 seconds without a turn, or its time limit ran out while it waited (then `details.reason` is `capacity`). Not counted as usage. Retry after `Retry-After` seconds.
500	`internal`	Our bug. Retry.
502	`fetch_failed`	The website could not be reached at all (DNS, connection, TLS).
503	`unavailable`	The service is restarting or briefly unavailable. Retry after `Retry-After`. (A full fetch capacity answers `429` `busy`.)
504	`timeout`	The website didn't answer within the call's time limit. When `details.reason` is `sitePace`, the call waited its turn for this website instead, which we fetch gently, shared by all customers: retry after `Retry-After`, or allow a longer `timeout`.

A sitemap or page that fails is reported in its own entry, not as an error. Only answered calls and the 502 and website-side 504 errors count as usage; refusals, our own errors (500, 503), 429 busy and a 504 with details.reason sitePace don't.

Limits

No per-key limits. When the server is busy you get 429 busy with Retry-After. Waiting calls are served in turn across keys, so a key with a few calls is never stuck behind one with thousands.

MCP

On the MCP server, the tool for this is map_site.

Map a website

#Parameters

#Headers

#Query parameters

#Response

#One spelling per page

#Large sites

#How we fetch

#Errors

#Limits

#MCP