Get a single custom-rubric evaluation

datasets.custom_evals.get(, ) -> CustomEval

GET/api/v1/datasets/{dataset_id}/custom-evals/{custom_eval_id}

ParametersExpand Collapse

dataset_id: str

custom_eval_id: str

ReturnsExpand Collapse

class CustomEval: …

custom_eval_id: str

dataset_id: str

status: str

pending | running | succeeded | failed

judge_prompt: str

name: Optional[str]

score_before: Optional[float]

Mean rubric score on the original pairs. Null until the eval succeeds.

score_after: Optional[float]

Mean rubric score on the adapted pairs. Null until the eval succeeds.

improvement_percent: Optional[float]

Percentage change from score_before to score_after. Null until the eval succeeds.

scored_rows: Optional[int]

Rows the judge scored. A rubric score is an average over this sample, not the whole dataset.

error_message: Optional[str]

created_at: datetime

formatdate-time

completed_at: Optional[datetime]

formatdate-time

Get a single custom-rubric evaluation

import os
from adaption import Adaption

client = Adaption(
    api_key=os.environ.get("ADAPTION_API_KEY"),  # This is the default and can be omitted
)
custom_eval = client.datasets.custom_evals.get(
    custom_eval_id="custom_eval_id",
    dataset_id="dataset_id",
)
print(custom_eval.custom_eval_id)

{
  "custom_eval_id": "custom_eval_id",
  "dataset_id": "dataset_id",
  "status": "running",
  "judge_prompt": "judge_prompt",
  "name": "name",
  "score_before": 0,
  "score_after": 0,
  "improvement_percent": 0,
  "scored_rows": 0,
  "error_message": "error_message",
  "created_at": "2019-12-27T18:11:19.117Z",
  "completed_at": "2019-12-27T18:11:19.117Z"
}

Returns Examples

{
  "custom_eval_id": "custom_eval_id",
  "dataset_id": "dataset_id",
  "status": "running",
  "judge_prompt": "judge_prompt",
  "name": "name",
  "score_before": 0,
  "score_after": 0,
  "improvement_percent": 0,
  "scored_rows": 0,
  "error_message": "error_message",
  "created_at": "2019-12-27T18:11:19.117Z",
  "completed_at": "2019-12-27T18:11:19.117Z"
}