Custom Evals

Start a custom-rubric evaluation for an adapted dataset

POST/api/v1/datasets/{dataset_id}/custom-evals

List custom-rubric evaluations for a dataset, newest first

GET/api/v1/datasets/{dataset_id}/custom-evals

Get a single custom-rubric evaluation

GET/api/v1/datasets/{dataset_id}/custom-evals/{custom_eval_id}

Draft a judge rubric tailored to an adapted dataset

POST/api/v1/datasets/{dataset_id}/custom-evals/prepare

ModelsExpand Collapse

CustomEval object { custom_eval_id, dataset_id, status, 9 more }

custom_eval_id: string

dataset_id: string

status: string

pending | running | succeeded | failed

judge_prompt: string

name: string

score_before: number

Mean rubric score on the original pairs. Null until the eval succeeds.

score_after: number

Mean rubric score on the adapted pairs. Null until the eval succeeds.

improvement_percent: number

Percentage change from score_before to score_after. Null until the eval succeeds.

scored_rows: number

Rows the judge scored. A rubric score is an average over this sample, not the whole dataset.

error_message: string

created_at: string

formatdate-time

completed_at: string

formatdate-time

CustomEvalListResponse object { items }

items: array of CustomEval { custom_eval_id, dataset_id, status, 9 more }

custom_eval_id: string

dataset_id: string

status: string

pending | running | succeeded | failed

judge_prompt: string

name: string

score_before: number

Mean rubric score on the original pairs. Null until the eval succeeds.

score_after: number

Mean rubric score on the adapted pairs. Null until the eval succeeds.

improvement_percent: number

Percentage change from score_before to score_after. Null until the eval succeeds.

scored_rows: number

Rows the judge scored. A rubric score is an average over this sample, not the whole dataset.

error_message: string

created_at: string

formatdate-time

completed_at: string

formatdate-time

CustomEvalPrepareResponse object { judge_prompt, adapted, sampled_rows }

judge_prompt: string

A ready-to-edit rubric, accepted as-is by judge_prompt on POST /datasets/{dataset_id}/custom-evals.

adapted: boolean

False when the stock rubric came back untouched — the sampled values did not fill both halves of the before/after contrast, or the adaptation model was unreachable. The prompt is still valid, just not tailored to this dataset.

sampled_rows: number

How many of the dataset’s own examples the rubric was written against — half original, half adapted. Lower than the usual target on a thin dataset; a dataset too thin to write from at all is rejected with a 400.