# Custom Evals ## Start a custom-rubric evaluation for an adapted dataset `datasets.custom_evals.create(strdataset_id, CustomEvalCreateParams**kwargs) -> CustomEval` **post** `/api/v1/datasets/{dataset_id}/custom-evals` Start a custom-rubric evaluation for an adapted dataset ### Parameters - `dataset_id: str` - `judge_prompt: str` The judge rubric. Scored per sample against the original and adapted pairs. - `name: Optional[str]` Optional label so a history of rubrics stays legible. ### Returns - `class CustomEval: …` - `custom_eval_id: str` - `dataset_id: str` - `status: str` pending | running | succeeded | failed - `judge_prompt: str` - `name: Optional[str]` - `score_before: Optional[float]` Mean rubric score on the original pairs. Null until the eval succeeds. - `score_after: Optional[float]` Mean rubric score on the adapted pairs. Null until the eval succeeds. - `improvement_percent: Optional[float]` Percentage change from score_before to score_after. Null until the eval succeeds. - `scored_rows: Optional[int]` Rows the judge scored. A rubric score is an average over this sample, not the whole dataset. - `error_message: Optional[str]` - `created_at: datetime` - `completed_at: Optional[datetime]` ### Example ```python import os from adaption import Adaption client = Adaption( api_key=os.environ.get("ADAPTION_API_KEY"), # This is the default and can be omitted ) custom_eval = client.datasets.custom_evals.create( dataset_id="dataset_id", judge_prompt="Rate how factually accurate the response is, from 0 (fabricated) to 10 (fully supported).", ) print(custom_eval.custom_eval_id) ``` #### Response ```json { "custom_eval_id": "custom_eval_id", "dataset_id": "dataset_id", "status": "running", "judge_prompt": "judge_prompt", "name": "name", "score_before": 0, "score_after": 0, "improvement_percent": 0, "scored_rows": 0, "error_message": "error_message", "created_at": "2019-12-27T18:11:19.117Z", "completed_at": "2019-12-27T18:11:19.117Z" } ``` ## List custom-rubric evaluations for a dataset, newest first `datasets.custom_evals.list(strdataset_id) -> CustomEvalListResponse` **get** `/api/v1/datasets/{dataset_id}/custom-evals` List custom-rubric evaluations for a dataset, newest first ### Parameters - `dataset_id: str` ### Returns - `class CustomEvalListResponse: …` - `items: List[CustomEval]` - `custom_eval_id: str` - `dataset_id: str` - `status: str` pending | running | succeeded | failed - `judge_prompt: str` - `name: Optional[str]` - `score_before: Optional[float]` Mean rubric score on the original pairs. Null until the eval succeeds. - `score_after: Optional[float]` Mean rubric score on the adapted pairs. Null until the eval succeeds. - `improvement_percent: Optional[float]` Percentage change from score_before to score_after. Null until the eval succeeds. - `scored_rows: Optional[int]` Rows the judge scored. A rubric score is an average over this sample, not the whole dataset. - `error_message: Optional[str]` - `created_at: datetime` - `completed_at: Optional[datetime]` ### Example ```python import os from adaption import Adaption client = Adaption( api_key=os.environ.get("ADAPTION_API_KEY"), # This is the default and can be omitted ) custom_evals = client.datasets.custom_evals.list( "dataset_id", ) print(custom_evals.items) ``` #### Response ```json { "items": [ { "custom_eval_id": "custom_eval_id", "dataset_id": "dataset_id", "status": "running", "judge_prompt": "judge_prompt", "name": "name", "score_before": 0, "score_after": 0, "improvement_percent": 0, "scored_rows": 0, "error_message": "error_message", "created_at": "2019-12-27T18:11:19.117Z", "completed_at": "2019-12-27T18:11:19.117Z" } ] } ``` ## Get a single custom-rubric evaluation `datasets.custom_evals.get(strcustom_eval_id, CustomEvalGetParams**kwargs) -> CustomEval` **get** `/api/v1/datasets/{dataset_id}/custom-evals/{custom_eval_id}` Get a single custom-rubric evaluation ### Parameters - `dataset_id: str` - `custom_eval_id: str` ### Returns - `class CustomEval: …` - `custom_eval_id: str` - `dataset_id: str` - `status: str` pending | running | succeeded | failed - `judge_prompt: str` - `name: Optional[str]` - `score_before: Optional[float]` Mean rubric score on the original pairs. Null until the eval succeeds. - `score_after: Optional[float]` Mean rubric score on the adapted pairs. Null until the eval succeeds. - `improvement_percent: Optional[float]` Percentage change from score_before to score_after. Null until the eval succeeds. - `scored_rows: Optional[int]` Rows the judge scored. A rubric score is an average over this sample, not the whole dataset. - `error_message: Optional[str]` - `created_at: datetime` - `completed_at: Optional[datetime]` ### Example ```python import os from adaption import Adaption client = Adaption( api_key=os.environ.get("ADAPTION_API_KEY"), # This is the default and can be omitted ) custom_eval = client.datasets.custom_evals.get( custom_eval_id="custom_eval_id", dataset_id="dataset_id", ) print(custom_eval.custom_eval_id) ``` #### Response ```json { "custom_eval_id": "custom_eval_id", "dataset_id": "dataset_id", "status": "running", "judge_prompt": "judge_prompt", "name": "name", "score_before": 0, "score_after": 0, "improvement_percent": 0, "scored_rows": 0, "error_message": "error_message", "created_at": "2019-12-27T18:11:19.117Z", "completed_at": "2019-12-27T18:11:19.117Z" } ``` ## Draft a judge rubric tailored to an adapted dataset `datasets.custom_evals.prepare(strdataset_id) -> CustomEvalPrepareResponse` **post** `/api/v1/datasets/{dataset_id}/custom-evals/prepare` Takes no body: samples the dataset's own before/after pairs and rewrites the stock rubric around them, sizing the sample to what the dataset yields. Nothing is persisted and no judge tokens are spent — the returned `judge_prompt` is a suggestion to review and then POST to `/datasets/{dataset_id}/custom-evals`. Blocks for the full model round-trip; set a client timeout of several minutes. ### Parameters - `dataset_id: str` ### Returns - `class CustomEvalPrepareResponse: …` - `judge_prompt: str` A ready-to-edit rubric, accepted as-is by `judge_prompt` on POST /datasets/{dataset_id}/custom-evals. - `adapted: bool` False when the stock rubric came back untouched — the sampled values did not fill both halves of the before/after contrast, or the adaptation model was unreachable. The prompt is still valid, just not tailored to this dataset. - `sampled_rows: int` How many of the dataset's own examples the rubric was written against — half original, half adapted. Lower than the usual target on a thin dataset; a dataset too thin to write from at all is rejected with a 400. ### Example ```python import os from adaption import Adaption client = Adaption( api_key=os.environ.get("ADAPTION_API_KEY"), # This is the default and can be omitted ) response = client.datasets.custom_evals.prepare( "dataset_id", ) print(response.judge_prompt) ``` #### Response ```json { "judge_prompt": "judge_prompt", "adapted": true, "sampled_rows": 0 } ``` ## Domain Types ### Custom Eval - `class CustomEval: …` - `custom_eval_id: str` - `dataset_id: str` - `status: str` pending | running | succeeded | failed - `judge_prompt: str` - `name: Optional[str]` - `score_before: Optional[float]` Mean rubric score on the original pairs. Null until the eval succeeds. - `score_after: Optional[float]` Mean rubric score on the adapted pairs. Null until the eval succeeds. - `improvement_percent: Optional[float]` Percentage change from score_before to score_after. Null until the eval succeeds. - `scored_rows: Optional[int]` Rows the judge scored. A rubric score is an average over this sample, not the whole dataset. - `error_message: Optional[str]` - `created_at: datetime` - `completed_at: Optional[datetime]` ### Custom Eval List Response - `class CustomEvalListResponse: …` - `items: List[CustomEval]` - `custom_eval_id: str` - `dataset_id: str` - `status: str` pending | running | succeeded | failed - `judge_prompt: str` - `name: Optional[str]` - `score_before: Optional[float]` Mean rubric score on the original pairs. Null until the eval succeeds. - `score_after: Optional[float]` Mean rubric score on the adapted pairs. Null until the eval succeeds. - `improvement_percent: Optional[float]` Percentage change from score_before to score_after. Null until the eval succeeds. - `scored_rows: Optional[int]` Rows the judge scored. A rubric score is an average over this sample, not the whole dataset. - `error_message: Optional[str]` - `created_at: datetime` - `completed_at: Optional[datetime]` ### Custom Eval Prepare Response - `class CustomEvalPrepareResponse: …` - `judge_prompt: str` A ready-to-edit rubric, accepted as-is by `judge_prompt` on POST /datasets/{dataset_id}/custom-evals. - `adapted: bool` False when the stock rubric came back untouched — the sampled values did not fill both halves of the before/after contrast, or the adaptation model was unreachable. The prompt is still valid, just not tailored to this dataset. - `sampled_rows: int` How many of the dataset's own examples the rubric was written against — half original, half adapted. Lower than the usual target on a thin dataset; a dataset too thin to write from at all is rejected with a 400.