Draft a judge rubric tailored to an adapted dataset

POST/api/v1/datasets/{dataset_id}/custom-evals/prepare

Takes no body: samples the dataset’s own before/after pairs and rewrites the stock rubric around them, sizing the sample to what the dataset yields. Nothing is persisted and no judge tokens are spent — the returned judge_prompt is a suggestion to review and then POST to /datasets/{dataset_id}/custom-evals. Blocks for the full model round-trip; set a client timeout of several minutes.

Path ParametersExpand Collapse

dataset_id: string

ReturnsExpand Collapse

judge_prompt: string

A ready-to-edit rubric, accepted as-is by judge_prompt on POST /datasets/{dataset_id}/custom-evals.

adapted: boolean

False when the stock rubric came back untouched — the sampled values did not fill both halves of the before/after contrast, or the adaptation model was unreachable. The prompt is still valid, just not tailored to this dataset.

sampled_rows: number

How many of the dataset’s own examples the rubric was written against — half original, half adapted. Lower than the usual target on a thin dataset; a dataset too thin to write from at all is rejected with a 400.

Draft a judge rubric tailored to an adapted dataset

curl https://api.prod.adaptionlabs.ai/api/v1/datasets/$DATASET_ID/custom-evals/prepare \
    -X POST \
    -H "Authorization: Bearer $ADAPTION_API_KEY"

{
  "judge_prompt": "judge_prompt",
  "adapted": true,
  "sampled_rows": 0
}

Returns Examples

{
  "judge_prompt": "judge_prompt",
  "adapted": true,
  "sampled_rows": 0
}