Check how crawlers see a page
See which of 274 search, AI and SEO crawlers a page's robots.txt allows, with the deciding rule, plus llms.txt, meta robots and TDM signals.
For one URL, the robots.txt verdict for every known crawler (search engines, AI training and AI search crawlers, assistants that fetch for a user, AI agents, SEO tools, link previews, archives and ad checkers), with the rule and line that decided it, grouped by company and summed up per purpose. Around it: robots.txt itself, llms.txt and llms-full.txt, the page's robots meta tags and X-Robots-Tag headers, TDMRep reservations, whether a firewall answered, and what the raw HTML gives a crawler that runs no JavaScript.
Send the parameters as a query string with GET, or as a JSON object (up to 1 MB) with POST to the same path. For many URLs, start a job.
Parameters
Headers
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
Authorization | string | Yes | – | Bearer followed by your API key. See Authentication. |
Query parameters
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
url | string | Yes | – | The page, a full http(s) URL. |
agents | string | No | all | Which crawlers to report, comma-separated: all; ai (every agent on the ai.robots.txt list); a purpose (aiTraining, aiSearch, userFetch, aiAgent, dataProvider, search, ads, seo, linkPreview, archive, undocumented); a company (google, openai…); or crawler names (GPTBot, bingbot…). Other robots.txt names are checked as given with known: false, up to 50. Case is ignored. |
rendered | boolean | No | false | true also renders the page and compares it with the raw HTML. It's slower. See Before and after JavaScript. |
maxAge | integer | No | 600 | How old a stored answer may be, in seconds, up to 2,592,000 (30 days). 0 fetches fresh. |
timeout | integer | No | 20 | How long the call may take, in seconds, 1–60. |
Response
| Field | Type | Description |
|---|---|---|
url, agents | string, string[] | What you asked (url without its #fragment). |
finalUrl, status, redirects | The page that answered. null and [] when the page was not fetched. | |
summary | object | agents, allowed, blocked (this page), blockedSite (on /), the same per purpose in purposes, and robotsTxt: found, missing or unreachable. |
companies | array | {company, agents}. Each agent: token, matchedAs, purpose, respectsRobots (what the vendor says), known, allowed, the page and site verdicts, crawlDelay, directives, docs and note. |
page, site, finalUrl (verdicts) | object | {allowed, group, groupKind, rule, line, note}. groupKind: named, fallback (such as Googlebot for Googlebot-Image), wildcard or none. |
robotsTxt | object | url, status, outcome, bytes, how many groups and rules, sitemaps, contentSignals (such as ai-train: no) and problems by line. |
llmsTxt, llmsFullTxt | object | Whether each file is there and valid by llmstxt.org: present, valid, servedHtml (the path answers a web page), title, sections, links or words, problems. |
pageDirectives | object | metaRobots, botMetas (metas named after a crawler), xRobotsTag, and all: what applies to every crawler (noindex, nofollow, noai, maxSnippet…). |
tdm | object | TDMRep: reserved (true, false or null), source, policy, and what each place said. |
firewall | object | null | {vendor, challenged, kind, signals} for the page we fetched. |
rawHtml | object | visibleWords and mainTextWords without JavaScript, jsShell (an empty app shell) with jsShellSignals, frameworks and generator. |
registry | object | version (date) and agents: the crawler list in use. |
seenAs | object | Our own user agent and verdict, and whether we fetched the page. |
rendered | object | With rendered=true: headline, diff and robotsBlockedResources, as in Before and after JavaScript. |
cached, ageSeconds | boolean, integer | Whether this answer came from our store, and how old it is. |
What the verdicts can and cannot tell you
The verdicts are what the site's robots.txt tells each crawler. We fetch only as ourselves, so we cannot see how a firewall or CDN treats each crawler: a site can block GPTBot at its firewall while its robots.txt allows it. firewall shows only how the site answered us.
robots.txt names may only hold letters, - and _, and parsers (Google's too) read User-agent: MJ12bot as MJ. Crawlers with digits, dots or spaces in their names are matched by that prefix; matchedAs shows it.
The crawler list is built from ai.robots.txt (MIT) and the vendors' own crawler pages; each agent links its vendor's docs.
Before and after JavaScript
Googlebot runs a page's JavaScript; most AI crawlers don't. With rendered=true:
rendered.headlinesays in one sentence what changes. For example: "Before rendering a crawler sees 0 words and 0 links; after rendering, 1,240 words and 38 links."rendered.diffhas the details: words, links, title, canonical, description, meta robots, hreflang, structured data and H1.rendered.robotsBlockedResourceslists the scripts and files robots.txt keeps from crawlers.
How we fetch
We fetch as Rankfield-User (Mozilla/5.0 (compatible; Rankfield-User/1.0; +https://rankfield.dev/bot)), as plain HTML without running JavaScript, unless you ask for a rendered page or a screenshot.
- robots.txt: obeyed on every request and redirect, following the rules for
Rankfield-User, or forRankfieldBotwhen the file has none for it, or*when it has neither. Requests: robots.txt, the page, then/llms.txt,/llms-full.txtand/.well-known/tdmrep.json: 5, or 6 with a redirect's new robots.txt; redirects add their hops. A robots.txt that answers with a server error counts as closed to us, as the standard says. - Firewalls: a challenge, CAPTCHA or block page is reported in the answer, never worked around and never retried.
- Never: cookies, logins, proxies or other bots' names. JavaScript runs only when you ask for it.
- Excluded sites: Reddit, LinkedIn, Facebook, Instagram, X/Twitter and TikTok (subdomains included), Google search result pages, and sites whose owners asked us are never fetched.
- Pace: each website is fetched gently, shared by all customers, slower when its robots.txt sets a
Crawl-delayor after a slow answer. Many URLs on one site go at that site's pace.
Repeat calls to the same website within about a minute can wait their turn for that site's slot. If the wait runs past the call's time limit (20 seconds by default), the call ends with 504 timeout and details.reason sitePace. It doesn't count as usage: retry after Retry-After, or allow a longer timeout.
More: Rankfield crawlers.
Errors
Errors are {"error": {"code", "message", "details"}}. Branch on code; message is for people.
| Status | Code | When |
|---|---|---|
| 400 | invalid_input | A parameter is missing, unknown or malformed. |
| 401 | unauthorized | No valid API key. |
| 403 | site_excluded | The site is on the exclusion list. |
| 403 | blocked | The website's bot protection refused us. Reported at once and never worked around; details says which protection. Not counted as usage. |
| 422 | address_not_public | The host resolves to a private or internal address. |
| 429 | busy | Our fetch capacity is busy: the call was refused at once, waited 4 seconds without a turn, or its time limit ran out while it waited (then details.reason is capacity). Not counted as usage. Retry after Retry-After seconds. |
| 500 | internal | Our bug. Retry. |
| 502 | fetch_failed | The site could not be reached at all (its robots.txt did not answer). A page that fails after robots.txt answered still gives the verdicts. |
| 503 | unavailable | The service is restarting or briefly unavailable. Retry after Retry-After. (A full fetch capacity answers 429 busy.) |
| 504 | timeout | The website didn't answer within the call's time limit. When details.reason is sitePace, the call waited its turn for this website instead, which we fetch gently, shared by all customers: retry after Retry-After, or allow a longer timeout. |
A robots.txt that disallows us is not an error here: the verdicts still come back, and the page checks say they were not checked. Only answered calls and the 502 and website-side 504 errors count as usage; refusals, our own errors (500, 503), 429 busy and a 504 with details.reason sitePace don't.
Limits
No per-key limits. When the server is busy you get 429 busy with Retry-After. Waiting calls are served in turn across keys, so a key with a few calls is never stuck behind one with thousands.
MCP
On the MCP server, the tool for this is bot_view.