AutoScientist quickstart

Introduction

Create a training-ready dataset, run AutoScientist, and download the best model checkpoint.

This guide requires adaption >= 0.7.0. It uses raw processing for a short, standalone example. For best results, first improve your dataset with Adaptive Data, then pass that adapted dataset’s ID to AutoScientist. See the AutoScientist overview for how the research loop works and how to interpret its results.

Install the SDK

pip install "adaption>=0.7.0"

Create an API key

Sign in to Adaption.
Open Settings, then open API keys.
Create a key and copy it. You will not be able to view the key again.

Use the key in Python

For local testing, you can pass the key directly:

from adaption import Adaption

client = Adaption(api_key="pt_live_...")

For applications and CI, set ADAPTION_API_KEY:

export ADAPTION_API_KEY="pt_live_..."

The SDK reads the environment variable automatically:

from adaption import Adaption

client = Adaption()

Create a training-ready dataset

Raw dataset creation is appropriate only when a local file already contains prompt and completion columns. This example uses training_data.csv with instruction and response columns:

import hashlib
from pathlib import Path

import httpx

path = Path("training_data.csv")
data = path.read_bytes()

dataset = client.datasets.create(
    source={
        "name": path.name,
        "file_format": "csv",
        "processing_mode": "raw",
        "column_mapping": {
            "prompt": "instruction",
            "completion": "response",
        },
    },
)

response = httpx.put(dataset.upload_instructions.url, content=data)
response.raise_for_status()

client.datasets.upload.complete_by_id(
    dataset.dataset_id,
    file_size_bytes=len(data),
    sha256=hashlib.sha256(data).hexdigest(),
)
dataset_id = dataset.dataset_id
print(dataset_id)

The upload is processed asynchronously after it is completed. Poll the dataset status until the row count is available before using the dataset:

import time

while True:
    status = client.datasets.get_status(dataset_id)
    if status.status == "failed":
        err = status.error_data
        msg = (err and err.message) or "unknown error"
        raise RuntimeError(f"Ingestion failed: {msg}")
    if status.row_count is not None:
        print(f"Imported {status.row_count} rows")
        break
    time.sleep(5)

Augment the dataset with additional rows

Augmentation is optional. This example targets at least 20,000 domain rows, then adds enough general-purpose rows for them to make up 40% of the augmented dataset.

import math

dataset = client.datasets.get(dataset_id=dataset_id)

domain_rows_to_add = max(0, 20_000 - (dataset.row_count or 0))
total_domain_rows = (dataset.row_count or 0) + domain_rows_to_add

target_general_ratio = 0.4
general_rows_to_add = math.ceil(
    target_general_ratio
    * total_domain_rows
    / (1 - target_general_ratio)
)

print(f"Domain rows to add: {domain_rows_to_add}")
print(f"General rows to add: {general_rows_to_add}")

Run AutoScientist

Create an AutoScientist run from the prepared dataset:

run = client.autoscientist.create(
    dataset_id=dataset_id,
    augmentation_domain_rows=domain_rows_to_add,
    augmentation_general_rows=general_rows_to_add,
)
print(run.id, run.status)  # "pending"

Wait for the run to complete

Wait for the run, then inspect its status and results:

run = client.autoscientist.wait_for_completion(run.id)

if run.status != "succeeded":
    raise RuntimeError(f"{run.status}: {run.error}")

print(f"Best win rate: {run.best_win_rate}")

Download the model

download_available confirms that AutoScientist produced a checkpoint. The download endpoint streams the archive without loading it into memory:

if not run.download_available:
    raise RuntimeError("The checkpoint is not available for download")

with client.autoscientist.with_streaming_response.download(run.id) as response:
    response.stream_to_file("best-checkpoint.tgz")

The archive contains the checkpoint from the best iteration, not necessarily the last iteration. See the create reference for all run parameters and the get reference for response fields.