Create a dataset from file upload, HuggingFace, Kaggle, or Google Sheets

datasets.create() -> DatasetCreateResponse

POST/api/v1/datasets

Unified ingest endpoint. Pass source.url to start an async import — the provider is inferred from the URL host (HuggingFace, Kaggle, or Google Sheets). Omit it to get upload instructions for a presigned S3 PUT.

ParametersExpand Collapse

source: Source

Dataset source. Pass url to import from HuggingFace, Kaggle, or Google Sheets, or name + file_format to upload a file. For Sheets, optional sheet_names selects tabs (omit or [] for all).

url: Optional[str]

HuggingFace, Kaggle, or Google Sheets URL. Provide it to import from that provider; omit it to upload a local file instead.

files: Optional[Sequence[str]]

File paths to download from the source. Required with HuggingFace/Kaggle url. Not used for Google Sheets — use sheet_names (or sheet_name) instead.

sheet_names: Optional[Sequence[str]]

Google Sheets tab titles to import. Only for Google Sheets url. Omit or pass [] to import every tab; pass one or more titles to import only those tabs.

sheet_name: Optional[str]

Legacy single Google Sheets tab title. Ignored when sheet_names is present (including []).

access: Optional[Literal["oauth", "public"]]

Google Sheets only. public imports a link-shared spreadsheet via GOOGLE_SHEETS_API_KEY (no user Google connect). oauth (default) uses the connected account.

One of the following:

"oauth"

"public"

name: Optional[str]

Name for the dataset. Required for file uploads, where it may double as a filename (“sales-data.csv”) to supply the format; derived from the URL for provider imports.

file_format: Optional[Literal["csv", "json", "jsonl", 8 more]]

Format of the file being uploaded. Optional when name ends in a supported extension — it is inferred from there, and an explicit value overrides it.

One of the following:

"csv"

"json"

"jsonl"

"parquet"

"pdf"

"docx"

"pptx"

"xlsx"

"html"

"zip"

"txt"

processing_mode: Optional[Literal["adapt", "raw"]]

How the data is ingested, for both uploads and provider imports. adapt (default) runs it through the adaptation pipeline. raw materializes the data straight to a trainable dataset with no augmentation, and needs a column_mapping plus tabular input — file_format for an upload, or files for an import. Imported files are read as one dataset, so they must share a single format and schema. Either can then be fine-tuned or run through AutoScientist.

One of the following:

"adapt"

"raw"

column_mapping: Optional[SourceColumnMapping]

Required when processing_mode is raw: which columns to canonicalize as prompt and completion. Ignored when the dataset is adapted.

prompt: str

Name of the raw column holding the prompt / input text

completion: str

Name of the raw column holding the completion / target text

context: Optional[Sequence[str]]

Names of raw columns to fold in as context alongside the prompt. Each is validated against the uploaded file at ingestion; an unknown name fails the request listing the available columns. The columns themselves are kept in the dataset as well as folded.

defer_adaption: Optional[bool]

When true, the upload is stored but Adaptive Data does not start. The dataset stays in awaiting_preprocessing until POST /datasets/:id/start-adaption.

ReturnsExpand Collapse

class DatasetCreateResponse: …

dataset_id: str

ID of the newly created dataset

status: str

Current dataset status

upload_instructions: Optional[UploadInstructions]

Upload instructions for file sources. PUT your file to the provided URL.

url: str

Pre-signed URL for uploading the file

method: str

HTTP method to use

s3_key: str

S3 object key — pass this back in the complete request if needed for verification

Create a dataset from file upload, HuggingFace, Kaggle, or Google Sheets

import os
from adaption import Adaption

client = Adaption(
    api_key=os.environ.get("ADAPTION_API_KEY"),  # This is the default and can be omitted
)
dataset = client.datasets.create(
    source={},
)
print(dataset.dataset_id)

{
  "dataset_id": "dataset_id",
  "status": "status",
  "upload_instructions": {
    "url": "https://s3.amazonaws.com/bucket/key?X-Amz-Signature=...",
    "method": "PUT",
    "s3_key": "s3_key"
  }
}

Returns Examples

{
  "dataset_id": "dataset_id",
  "status": "status",
  "upload_instructions": {
    "url": "https://s3.amazonaws.com/bucket/key?X-Amz-Signature=...",
    "method": "PUT",
    "s3_key": "s3_key"
  }
}