Create a dataset

Import data from a local file, Hugging Face, or Kaggle without writing a conversion script. Each method returns a dataset_id that you pass to datasets.run.

The examples assume you have installed the Python SDK and set ADAPTION_API_KEY:

from adaption import Adaption

client = Adaption()

1. Upload a local file

Create the dataset, PUT the file bytes to the returned presigned URL, then confirm the upload.

Supported extensions: .csv, .json, .jsonl, .parquet, .pdf, .docx, .pptx, .xlsx, .html, .zip, and .txt.

import hashlib
from pathlib import Path

import httpx

path = Path("training_data.csv")
data = path.read_bytes()

dataset = client.datasets.create(
    source={
        "name": path.name,
        "file_format": "csv",
    },
)
response = httpx.put(dataset.upload_instructions.url, content=data)
response.raise_for_status()

client.datasets.upload.complete_by_id(
    dataset.dataset_id,
    file_size_bytes=len(data),
    sha256=hashlib.sha256(data).hexdigest(),
)
dataset_id = dataset.dataset_id
print(dataset_id)

2. Import from Hugging Face

Point at a Hugging Face dataset URL and the file(s) to import.

dataset = client.datasets.create(
    source={
        "url": "https://huggingface.co/datasets/tatsu-lab/alpaca",
        "files": [
            "data/train-00000-of-00001-a09b74b3ef9c3b56.parquet"
        ],
    },
)
dataset_id = dataset.dataset_id
print(dataset_id)

3. Import from Kaggle

Use the Kaggle dataset page URL and the files to pull.

dataset = client.datasets.create(
    source={
        "url": "https://www.kaggle.com/datasets/uciml/sms-spam-collection-dataset",
        "files": ["spam.csv"],
    },
)
dataset_id = dataset.dataset_id
print(dataset_id)

Kaggle credentials must be registered in Adaption: open API keys settings (sign in if prompted) and add your Kaggle API credentials there before importing.

Raw (passthrough) processing is currently supported only for local file uploads. Do not combine processing_mode="raw" with a Hugging Face or Kaggle URL. See Run on non-adapted data for a local-file example.

Wait for the import

Imports run asynchronously. Poll the dataset status until ingestion has populated the dataset before starting an adaptation run:

import time

while True:
    status = client.datasets.get_status(dataset_id)
    if status.status == "failed":
        error = status.error_data
        message = (error and error.message) or "unknown error"
        raise RuntimeError(f"Dataset import failed: {message}")
    if status.row_count is not None:
        print(f"Imported {status.row_count} rows")
        break
    time.sleep(5)

Next, map the imported columns, then configure the adaptation run. For endpoint details, see the create and get_status references. Local files must also be finalized with datasets.upload.complete_by_id after the presigned PUT.