> ## Documentation index
> Fetch the complete documentation index at: https://rankfield.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Batch jobs

> Send many URLs, domains or keywords in one Rankfield batch job, follow its progress, and read results as they finish.

Use a single request for one URL or keyword. Use a batch job when you have many: one request starts the job, and you read results as each item finishes, without holding a connection open.

## How a job works

1. **Start it.** `POST` the list to the endpoint's `/jobs` path. The answer is `202` with the job's `id`.
2. **Follow it.** `GET /jobs/{id}` reports its status and how many items are done.
3. **Read results.** `GET /jobs/{id}/results` returns finished items in the order they finish, each with its position in your list. Pass the returned cursor back to read the next ones.
4. **Download** (where offered). When the job is complete, `?format=jsonl` or `?format=csv` streams everything at once.

Send an `Idempotency-Key` header when you start a job. If your request times out and you send it again with the same key and body, you get the same job back instead of a second one.

## The three job styles

Rankfield's endpoint families each keep their own job format:

| | Link Extraction | Suggest | Scrape, Bot View, Brand and Site Map |
|---|---|---|---|
| Body | `{"urls": [...]}`, up to 600 | `{"queries": [...]}`, up to 10,000 | `{"items": [{...}]}`, up to 10,000 (Map: 1,000) |
| `Idempotency-Key` | Optional | Required | Required |
| Read results | `nextCursor` until it is `null` | `cursor` while `hasMore` is true | `cursor` while `hasMore` is true |
| Deadline | `delivery`: `standard` (330 s) or `priority` (90 s) | `delivery`: `standard` (300 s), `priority` (60 s) or `live` (6 s) | None: 16 items run at a time |
| Cancel | Not available | `POST /jobs/{id}/cancel` | `POST /jobs/{id}/cancel` |
| Kept for | 24 hours | 24 hours | 24 hours after the job finishes |

Link Extraction fetches up to 5 pages a second from one website, so about 400 pages of one site fit a `priority` job and about 1,500 a `standard` one. For many pages of one site, use `standard`.

Each endpoint's pages show its exact requests and answers, for example [Start a Link Extraction job](/docs/link-extraction/start-a-job).

## Example: read results as they finish

```python Python
import os, time, requests

BASE = "https://rankfield.dev/v1/html/link-extraction"
HEADERS = {"Authorization": f"Bearer {os.environ['RANKFIELD_API_KEY']}"}

job = requests.post(f"{BASE}/jobs", headers={**HEADERS, "Idempotency-Key": "crawl-2026-10-25"},
                    json={"urls": ["https://example.com/", "https://www.iana.org/help/example-domains"]}).json()

cursor = 0
while cursor is not None:
    page = requests.get(f"{BASE}/jobs/{job['id']}/results", headers=HEADERS,
                        params={"cursor": cursor, "wait": 20}).json()
    for result in page["pages"]:
        print(result["url"], result["counts"]["links"] if result["ok"] else result["error"]["message"])
    cursor = page["nextCursor"]
```

```javascript JavaScript
const BASE = "https://rankfield.dev/v1/html/link-extraction";
const headers = { Authorization: `Bearer ${process.env.RANKFIELD_API_KEY}` };

const job = await (await fetch(`${BASE}/jobs`, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json", "Idempotency-Key": "crawl-2026-10-25" },
  body: JSON.stringify({ urls: ["https://example.com/", "https://www.iana.org/help/example-domains"] }),
})).json();

for (let cursor = 0; cursor !== null; ) {
  const page = await (await fetch(`${BASE}/jobs/${job.id}/results?cursor=${cursor}&wait=20`, { headers })).json();
  for (const result of page.pages) console.log(result.url, result.ok ? result.counts.links : result.error.message);
  cursor = page.nextCursor;
}
```