Batches
Send a file of requests to run later with OpenAI's batch API, and collect the answers as a file.
The batch API takes a file of requests, runs them over the following hours, and gives back the answers as a file. It is OpenAI's batch API, so the OpenAI SDKs' files and batches calls work once the base URL points at ai.ml.
The input file
A JSON lines file, one request per line. Each line has a custom_id that is unique in the file (it is how you match an answer to its request), method set to POST, the url to run and the request's body. Every line uses the same url, one of /v1/chat/completions, /v1/embeddings or /v1/responses. A file is at most 100 MB and 10,000 requests. A request cannot stream.
Upload it to POST /v1/files with purpose set to batch. The file is checked as it is uploaded: a line that is not valid is refused with its line number, and nothing is stored.
Run it
POST /v1/batches with the file's id, the same url as endpoint and completion_window set to 24h starts the batch. GET /v1/batches/{id} shows its progress: how many requests have finished and failed, and what they have cost so far.
Every request runs as if your key had sent it at that moment: the key's budgets, rate limits, data policy and model lists apply, each request is a row in the request log, and it is billed at the price then in force. Batches are not discounted. A request that is rate limited waits and is tried again.
The answers
When the batch is done its output_file_id names a file of the requests answered with a 2xx: each line has your custom_id and the response, with its status code, request id and body. error_file_id names the others, with the response the route gave or an error. Read either from GET /v1/files/{id}/content. Output files are kept 30 days.
Stopping early
POST /v1/batches/{id}/cancel stops a batch: no new request starts, and the batch finishes as cancelled with what had run in its files. A batch that has not finished within 24 hours finishes as expired the same way, with the requests it did not reach in the error file.