Rankfield crawlers

Rankfield runs two crawlers.

RankfieldBot

Crawls public web content on its own to build Rankfield's datasets.

User agent: Mozilla/5.0 (compatible; RankfieldBot/2.0; +https://rankfield.dev/bot)

Rankfield-User

Fetches a public page when a Rankfield customer asks for it.

When a customer asks for a rendered page, that page's own files (scripts, styles, data) are fetched as a browser would: GET only, honouring robots.txt and Crawl-delay; images, media, fonts and trackers are not loaded.

User agent: Mozilla/5.0 (compatible; Rankfield-User/1.0; +https://rankfield.dev/bot)

robots.txt

Both crawlers obey robots.txt, including Crawl-delay. Rules written for our former crawler names (RankfieldBot/1.0, RankfieldNewsBot, RankfieldPodcastBot, RankfieldNewsletterBot, RankfieldHTMLLinkEngine, RankfieldHTMLTextEngine) are still honoured.

Block RankfieldBot:

User-agent: RankfieldBot
Disallow: /

Block Rankfield-User:

User-agent: Rankfield-User
Disallow: /

Slow either one down:

User-agent: RankfieldBot
Crawl-delay: 10

Verify a request

Our sending addresses are published at https://rankfield.dev/bot/ips.json. A request using these names from any other address is not ours.

Both crawlers sign every request with Web Bot Auth. Keys: https://rankfield.dev/.well-known/http-message-signatures-directory (Rankfield-User) and https://crawl.rankfield.dev/.well-known/http-message-signatures-directory (RankfieldBot).

Contact

[email protected] Write to us ↗

Tell us if a Rankfield crawler causes you trouble or you'd like your site excluded.

Operator: Analyze AI, Inc. (Delaware, United States)