Batch jobs
Send many URLs, domains or keywords in one Rankfield batch job, follow its progress, and read results as they finish.
Use a single request for one URL or keyword. Use a batch job when you have many: one request starts the job, and you read results as each item finishes, without holding a connection open.
How a job works
- Start it.
POSTthe list to the endpoint's/jobspath. The answer is202with the job'sid. - Follow it.
GET /jobs/{id}reports its status and how many items are done. - Read results.
GET /jobs/{id}/resultsreturns finished items in the order they finish, each with its position in your list. Pass the returned cursor back to read the next ones. - Download (where offered). When the job is complete,
?format=jsonlor?format=csvstreams everything at once.
Send an Idempotency-Key header when you start a job. If your request times out and you send it again with the same key and body, you get the same job back instead of a second one.
The three job styles
Rankfield's endpoint families each keep their own job format:
| Link Extraction | Suggest | Scrape, Bot View, Brand and Site Map | |
|---|---|---|---|
| Body | {"urls": [...]}, up to 600 | {"queries": [...]}, up to 10,000 | {"items": [{...}]}, up to 10,000 (Map: 1,000) |
Idempotency-Key | Optional | Required | Required |
| Read results | nextCursor until it is null | cursor while hasMore is true | cursor while hasMore is true |
| Deadline | delivery: standard (330 s) or priority (90 s) | delivery: standard (300 s), priority (60 s) or live (6 s) | None: 16 items run at a time |
| Cancel | Not available | POST /jobs/{id}/cancel | POST /jobs/{id}/cancel |
| Kept for | 24 hours | 24 hours | 24 hours after the job finishes |
Link Extraction fetches up to 5 pages a second from one website, so about 400 pages of one site fit a priority job and about 1,500 a standard one. For many pages of one site, use standard.
Each endpoint's pages show its exact requests and answers, for example Start a Link Extraction job.
Example: read results as they finish
import os, time, requests
BASE = "https://rankfield.dev/v1/html/link-extraction"
HEADERS = {"Authorization": f"Bearer {os.environ['RANKFIELD_API_KEY']}"}
job = requests.post(f"{BASE}/jobs", headers={**HEADERS, "Idempotency-Key": "crawl-2026-10-25"},
json={"urls": ["https://example.com/", "https://www.iana.org/help/example-domains"]}).json()
cursor = 0
while cursor is not None:
page = requests.get(f"{BASE}/jobs/{job['id']}/results", headers=HEADERS,
params={"cursor": cursor, "wait": 20}).json()
for result in page["pages"]:
print(result["url"], result["counts"]["links"] if result["ok"] else result["error"]["message"])
cursor = page["nextCursor"]const BASE = "https://rankfield.dev/v1/html/link-extraction";
const headers = { Authorization: `Bearer ${process.env.RANKFIELD_API_KEY}` };
const job = await (await fetch(`${BASE}/jobs`, {
method: "POST",
headers: { ...headers, "Content-Type": "application/json", "Idempotency-Key": "crawl-2026-10-25" },
body: JSON.stringify({ urls: ["https://example.com/", "https://www.iana.org/help/example-domains"] }),
})).json();
for (let cursor = 0; cursor !== null; ) {
const page = await (await fetch(`${BASE}/jobs/${job.id}/results?cursor=${cursor}&wait=20`, { headers })).json();
for (const result of page.pages) console.log(result.url, result.ok ? result.counts.links : result.error.message);
cursor = page.nextCursor;
}