API Reference

Datasets

Copy Markdown

Open in Claude

Open in ChatGPT

Open in Cursor

---

Copy Markdown

View as Markdown

Custom Evals

Start a custom-rubric evaluation for an adapted dataset

POST/api/v1/datasets/{dataset_id}/custom-evals

List custom-rubric evaluations for a dataset, newest first

GET/api/v1/datasets/{dataset_id}/custom-evals

Get a single custom-rubric evaluation

GET/api/v1/datasets/{dataset_id}/custom-evals/{custom_eval_id}

ModelsExpand Collapse

CustomEval object { custom_eval_id, dataset_id, status, 9 more }

custom_eval_id: string

dataset_id: string

status: string

pending | running | succeeded | failed

judge_prompt: string

name: string

score_before: number

Mean rubric score on the original pairs. Null until the eval succeeds.

score_after: number

Mean rubric score on the adapted pairs. Null until the eval succeeds.

improvement_percent: number

Percentage change from score_before to score_after. Null until the eval succeeds.

scored_rows: number

Rows the judge scored. A rubric score is an average over this sample, not the whole dataset.

error_message: string

created_at: string

formatdate-time

completed_at: string

formatdate-time

CustomEvalListResponse object { items }

items: array of CustomEval { custom_eval_id, dataset_id, status, 9 more }

custom_eval_id: string

dataset_id: string

status: string

pending | running | succeeded | failed

judge_prompt: string

name: string

score_before: number

Mean rubric score on the original pairs. Null until the eval succeeds.

score_after: number

Mean rubric score on the adapted pairs. Null until the eval succeeds.

improvement_percent: number

Percentage change from score_before to score_after. Null until the eval succeeds.

scored_rows: number

Rows the judge scored. A rubric score is an average over this sample, not the whole dataset.

error_message: string

created_at: string

formatdate-time

completed_at: string

formatdate-time