Site Map
List every URL of a website from its sitemaps, feeds, links and a short crawl, deduplicated, with sources and last-modified dates.
Map lists a website's URLs from what the site itself publishes and links: the sitemaps its robots.txt names, its RSS, Atom and JSON feeds, the links on its homepage, and a short crawl of its own pages.
Pages
Map a websiteList a website's URLs from its sitemaps, feeds, homepage links and a short crawl, each with its sources and last-modified date.Start a Site Map jobMap up to 1,000 websites in one background job; each item takes the same parameters as a single call.Get a Site Map jobCheck a Site Map job: its status, how many items are done and failed, and when its results expire.Get Site Map job resultsRead a Site Map job's items as each one finishes, with its full answer or error, and page through them with a cursor.Cancel a Site Map jobStop a Site Map job. Items already finished stay readable for 24 hours; items not started are dropped.