Create a dataset from file upload, HuggingFace, or Kaggle

POST/api/v1/datasets

Unified ingest endpoint. Pass source.url to start an async import — the provider is inferred from the URL host. Omit it to get upload instructions for a presigned S3 PUT.

Body ParametersJSONExpand Collapse

source: object { url, files, name, 3 more }

Dataset source. Pass url to import from HuggingFace or Kaggle, or name + file_format to upload a file.

url: optional string

HuggingFace or Kaggle dataset URL. Provide it to import from that provider; omit it to upload a local file instead.

files: optional array of string

File paths to download from the source. Required with url.

name: optional string

Name for the dataset. Required for file uploads, where it may double as a filename ("sales-data.csv") to supply the format; derived from the URL for provider imports.

file_format: optional "csv" or "json" or "jsonl" or 8 more

Format of the file being uploaded. Optional when name ends in a supported extension — it is inferred from there, and an explicit value overrides it.

One of the following:

"csv"

"json"

"jsonl"

"parquet"

"pdf"

"docx"

"pptx"

"xlsx"

"html"

"zip"

"txt"

processing_mode: optional "adapt" or "raw"

How the data is ingested, for both uploads and provider imports. adapt (default) runs it through the adaptation pipeline. raw materializes the data straight to a trainable dataset with no augmentation, and needs a column_mapping plus tabular input — file_format for an upload, or files for an import. Imported files are read as one dataset, so they must share a single format and schema. Either can then be fine-tuned or run through AutoScientist.

One of the following:

"adapt"

"raw"

column_mapping: optional object { prompt, completion }

Required when processing_mode is raw: which columns to canonicalize as prompt and completion. Ignored when the dataset is adapted.

prompt: string

Name of the raw column holding the prompt / input text

completion: string

Name of the raw column holding the completion / target text

ReturnsExpand Collapse

dataset_id: string

ID of the newly created dataset

status: string

Current dataset status

upload_instructions: optional object { url, method, s3_key }

Upload instructions for file sources. PUT your file to the provided URL.

url: string

Pre-signed URL for uploading the file

method: string

HTTP method to use

s3_key: string

S3 object key — pass this back in the complete request if needed for verification

Create a dataset from file upload, HuggingFace, or Kaggle

curl https://api.prod.adaptionlabs.ai/api/v1/datasets \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $ADAPTION_API_KEY" \
    -d '{
          "source": {}
        }'

{
  "dataset_id": "dataset_id",
  "status": "status",
  "upload_instructions": {
    "url": "https://s3.amazonaws.com/bucket/key?X-Amz-Signature=...",
    "method": "PUT",
    "s3_key": "s3_key"
  }
}

Returns Examples

{
  "dataset_id": "dataset_id",
  "status": "status",
  "upload_instructions": {
    "url": "https://s3.amazonaws.com/bucket/key?X-Amz-Signature=...",
    "method": "PUT",
    "s3_key": "s3_key"
  }
}