Initiate a batch upload

datasets.upload.initiate_batch() -> UploadInitiateBatchResponse

POST/api/v1/datasets/upload/initiate-batch

Initiate a batch upload. A batch must be homogeneous: either all document formats (pdf, docx, pptx, xlsx, html, zip) or all tabular formats (csv, json, jsonl, parquet). Mixed batches are rejected. TXT files are permissive and may be included in either kind of batch (or alone, in which case the dataset is tabular); their contents are converted during complete-batch.

ParametersExpand Collapse

name: str

Human-readable name for the dataset

files: Iterable[File]

List of files to upload. A single batch must be homogeneous: either all document formats (pdf, docx, pptx, xlsx, html, zip) or all tabular formats (csv, json, jsonl, parquet). Mixing the two is rejected because the two pipelines use different credit models and preprocessing stages. TXT files are permissive: they may be included in either kind of batch (or alone, producing a tabular dataset). complete-batch converts TXT contents to csv/jsonl and merges them into the resulting dataset.

file_name: str

Original file name

file_format: Literal["csv", "json", "jsonl", 8 more]

Format of the file

One of the following:

"csv"

"json"

"jsonl"

"parquet"

"pdf"

"docx"

"pptx"

"xlsx"

"html"

"zip"

"txt"

file_size_bytes: float

File size in bytes

defer_adaption: Optional[bool]

When true, the upload is stored but Adaptive Data does not start. The dataset stays in awaiting_preprocessing until POST /datasets/:id/start-adaption.

ReturnsExpand Collapse

class UploadInitiateBatchResponse: …

dataset_id: str

Dataset ID

uploads: List[Upload]

file_name: str

Original file name

upload_url: str

Presigned S3 upload URL

s3_key: str

S3 key for the uploaded file

Initiate a batch upload

import os
from adaption import Adaption

client = Adaption(
    api_key=os.environ.get("ADAPTION_API_KEY"),  # This is the default and can be omitted
)
response = client.datasets.upload.initiate_batch(
    name="my-document-dataset",
    files=[{
        "file_name": "report.pdf",
        "file_format": "pdf",
        "file_size_bytes": 1048576,
    }],
)
print(response.dataset_id)

{
  "dataset_id": "dataset_id",
  "uploads": [
    {
      "file_name": "file_name",
      "upload_url": "upload_url",
      "s3_key": "s3_key"
    }
  ]
}

Returns Examples

{
  "dataset_id": "dataset_id",
  "uploads": [
    {
      "file_name": "file_name",
      "upload_url": "upload_url",
      "s3_key": "s3_key"
    }
  ]
}