Draft a judge rubric tailored to an adapted dataset

datasets.custom_evals.prepare() -> CustomEvalPrepareResponse

POST/api/v1/datasets/{dataset_id}/custom-evals/prepare

Takes no body: samples the dataset’s own before/after pairs and rewrites the stock rubric around them, sizing the sample to what the dataset yields. Nothing is persisted and no judge tokens are spent — the returned judge_prompt is a suggestion to review and then POST to /datasets/{dataset_id}/custom-evals. Blocks for the full model round-trip; set a client timeout of several minutes.

ParametersExpand Collapse

dataset_id: str

ReturnsExpand Collapse

class CustomEvalPrepareResponse: …

judge_prompt: str

A ready-to-edit rubric, accepted as-is by judge_prompt on POST /datasets/{dataset_id}/custom-evals.

adapted: bool

False when the stock rubric came back untouched — the sampled values did not fill both halves of the before/after contrast, or the adaptation model was unreachable. The prompt is still valid, just not tailored to this dataset.

sampled_rows: int

How many of the dataset’s own examples the rubric was written against — half original, half adapted. Lower than the usual target on a thin dataset; a dataset too thin to write from at all is rejected with a 400.

Draft a judge rubric tailored to an adapted dataset

import os
from adaption import Adaption

client = Adaption(
    api_key=os.environ.get("ADAPTION_API_KEY"),  # This is the default and can be omitted
)
response = client.datasets.custom_evals.prepare(
    "dataset_id",
)
print(response.judge_prompt)

{
  "judge_prompt": "judge_prompt",
  "adapted": true,
  "sampled_rows": 0
}

Returns Examples

{
  "judge_prompt": "judge_prompt",
  "adapted": true,
  "sampled_rows": 0
}