Skip to content
Adaption Adaption Adaption
SupportLogin
API Reference
Datasets
Custom Evals

Start a custom-rubric evaluation for an adapted dataset

datasets.custom_evals.create(strdataset_id, CustomEvalCreateParams**kwargs) -> CustomEval
POST/api/v1/datasets/{dataset_id}/custom-evals

Start a custom-rubric evaluation for an adapted dataset

ParametersExpand Collapse
dataset_id: str
judge_prompt: str

The judge rubric. Scored per sample against the original and adapted pairs.

minLength20
maxLength20000
name: Optional[str]

Optional label so a history of rubrics stays legible.

maxLength120
ReturnsExpand Collapse
class CustomEval: …
custom_eval_id: str
dataset_id: str
status: str

pending | running | succeeded | failed

judge_prompt: str
name: Optional[str]
score_before: Optional[float]

Mean rubric score on the original pairs. Null until the eval succeeds.

score_after: Optional[float]

Mean rubric score on the adapted pairs. Null until the eval succeeds.

improvement_percent: Optional[float]

Percentage change from score_before to score_after. Null until the eval succeeds.

scored_rows: Optional[int]

Rows the judge scored. A rubric score is an average over this sample, not the whole dataset.

error_message: Optional[str]
created_at: datetime
formatdate-time
completed_at: Optional[datetime]
formatdate-time

Start a custom-rubric evaluation for an adapted dataset

import os
from adaption import Adaption

client = Adaption(
    api_key=os.environ.get("ADAPTION_API_KEY"),  # This is the default and can be omitted
)
custom_eval = client.datasets.custom_evals.create(
    dataset_id="dataset_id",
    judge_prompt="Rate how factually accurate the response is, from 0 (fabricated) to 10 (fully supported).",
)
print(custom_eval.custom_eval_id)
{
  "custom_eval_id": "custom_eval_id",
  "dataset_id": "dataset_id",
  "status": "running",
  "judge_prompt": "judge_prompt",
  "name": "name",
  "score_before": 0,
  "score_after": 0,
  "improvement_percent": 0,
  "scored_rows": 0,
  "error_message": "error_message",
  "created_at": "2019-12-27T18:11:19.117Z",
  "completed_at": "2019-12-27T18:11:19.117Z"
}
Returns Examples
{
  "custom_eval_id": "custom_eval_id",
  "dataset_id": "dataset_id",
  "status": "running",
  "judge_prompt": "judge_prompt",
  "name": "name",
  "score_before": 0,
  "score_after": 0,
  "improvement_percent": 0,
  "scored_rows": 0,
  "error_message": "error_message",
  "created_at": "2019-12-27T18:11:19.117Z",
  "completed_at": "2019-12-27T18:11:19.117Z"
}