Expand your data

After you import a dataset and optionally run Adaptive Data, you can increase row count without re-running the full adaptation pipeline. Each method below returns a new dataset that includes your source rows plus the added ones.

The SDK examples assume you have configured the Python SDK and set ADAPTION_DATASET_ID to a dataset that is ready to expand:

import os

from adaption import Adaption

client = Adaption()
dataset_id = os.environ["ADAPTION_DATASET_ID"]

Data augmentation

datasets.augment retrieves additional rows from the curated pool and returns them as a new dataset. Use it when you want more examples in the same topic mix or broader coverage across topics:

• Domain rows match topics in your dataset.
• General rows come from other topics to broaden diversity.

estimate = client.datasets.augment(
    dataset_id,
    domain_rows=3_000,
    general_rows=1_000,
    estimate=True,
)
print(f"Estimated credits: {estimate.estimated_credits_consumed}")

response = client.datasets.augment(
    dataset_id,
    domain_rows=3_000,
    general_rows=1_000,
)
print(response.dataset_id, response.status)

The response carries the new dataset’s id and its current status; both are None for an estimate. Wait for the new dataset before you download or train on it:

from adaption import DatasetTimeout

try:
    final = client.datasets.wait_for_completion(response.dataset_id)
except DatasetTimeout as exc:
    print(exc.resource_id, exc.last_status)
    raise

print(final.status, final.row_count)

wait_for_completion backs off from 2 to 30 seconds and has a one-hour default timeout. A timeout stops waiting, not the expansion job. Call it again with the same dataset ID to resume tracking. The same wait applies after similar rows, translate and localize.

Omit either row parameter to skip that type of addition. Augmentation is billed on the rows retrieved. See the augment reference for current limits and request fields.

Domain and general rows work on text datasets whose topic mix can be determined. datasets.augment does not support image datasets. Domain and general rows are the same curated-pool operation as augmentation_domain_rows and augmentation_general_rows on autoscientist.create. The AutoScientist path applies those rows at training time instead of writing a new dataset.

Generate similar rows

Pass similar_rows to datasets.augment to generate new rows in the style of your dataset instead of retrieving them from the pool. Adaptive Data reads a sample of your rows, works out their topics, language, and prompt and completion conventions, and generates new rows to match. Use it when the pool has little that looks like your data, or when you want more rows that follow your own format closely:

estimate = client.datasets.augment(
    dataset_id,
    similar_rows=500,
    estimate=True,
)
print(f"Estimated credits: {estimate.estimated_credits_consumed}")

response = client.datasets.augment(dataset_id, similar_rows=500)
print(response.dataset_id, response.status)

The new dataset holds your source rows plus the generated ones.

similar_rows combines with domain_rows and general_rows in one call, and the new dataset holds all three kinds of row:

response = client.datasets.augment(
    dataset_id,
    domain_rows=3_000,
    general_rows=1_000,
    similar_rows=500,
)

Similar rows are billed at the same rate as pool rows, on the rows actually generated, up to similar_rows. A run that generates fewer rows than requested is charged only for those it delivers. Your plan may cap similar_rows per request; a request above the cap returns 400 with the limit in the body.

Translate rows

datasets.translate converts a sample of existing rows into other languages. Adaptive Data supports up to 242 languages. Provide ISO 639-1 language codes. sample_rate is the fraction of source rows expanded for each target language:

estimate = client.datasets.translate(
    dataset_id,
    sample_rate=0.25,
    languages=["es", "fr"],
    estimate=True,
)
print(f"Estimated credits: {estimate.estimated_credits_consumed}")

response = client.datasets.translate(
    dataset_id,
    sample_rate=0.25,
    languages=["es", "fr"],
)
print(response.dataset_id, response.status)

With two target languages and a 0.25 sample rate, the output contains approximately 1 + (0.25 × 2) = 1.5 times as many rows as the input.

Localize rows

datasets.localize adapts rows for both a language and a country, including locale-specific wording rather than direct translation alone. Provide country/language pairs using ISO 3166-1 alpha-2 country codes and ISO 639-1 language codes:

response = client.datasets.localize(
    dataset_id,
    sample_rate=1.0,
    pairs=[
        {"country": "CA", "language": "fr"},
        {"country": "MX", "language": "es"},
    ],
)
print(response.dataset_id, response.status)

Each pair creates a country- and language-specific variant rather than only translating the text.

Translate and localize are billed on output rows. Set estimate=True on any of these methods to price a request before it runs. Pass idempotency_key on any of them to make a retried request return the dataset the first request created instead of starting and charging for a second run. See the translate and localize references for the current schema and supported values.

Match translations to their source rows

A dataset created by translate or localize holds each sampled source row alongside its variants. When you download it, three extra columns tell you which rows belong together:

Column	Value
target_language	Name of the language the row was translated or localized into: Spanish after translate, Spanish (Mexico) after localize. original for a source row.
target_language_code	Code for that language: es after translate, or a full locale tag such as es-Latn-MX after localize. original for a source row.
translation_join_id	ID shared by a source row and every variant produced from it.

For example, here are four rows from the localize call above, in file order. Each source row shares its ID with the Spanish variant made from it:

original_prompt	target_language	target_language_code	translation_join_id
¿Qué hacen las abejas?	Spanish (Mexico)	es-Latn-MX	847c6b29-0287-4f8a-9f6b-5b7ed555d21b
What color is the sky on a clear day?	original	original	48b7feb6-0c31-4402-9078-81932756debb
What do bees make?	original	original	847c6b29-0287-4f8a-9f6b-5b7ed555d21b
¿De qué color es el cielo en un día despejado?	Spanish (Mexico)	es-Latn-MX	48b7feb6-0c31-4402-9078-81932756debb

Rows in a group are not guaranteed to be adjacent in the file. Pivot on translation_join_id to put each source row next to its variants, for example to build parallel text or to keep a row and its translations in the same train or evaluation split:

import pandas as pd

client.datasets.download(response.dataset_id, file_format="csv").write_to_file(
    "localized.csv"
)

rows = pd.read_csv("localized.csv")

parallel = rows.pivot(
    index="translation_join_id",
    columns="target_language_code",
    values="original_prompt",
)
print(parallel)

The result has one row per source row and one column per language or locale, with the source text under original:

target_language_code                                                      es-Latn-MX                                         fr-Latn-CA                               original
translation_join_id
48b7feb6-0c31-4402-9078-81932756debb  ¿De qué color es el cielo en un día despejado?  De quelle couleur est le ciel par une journée ...  What color is the sky on a clear day?
847c6b29-0287-4f8a-9f6b-5b7ed555d21b                          ¿Qué hacen las abejas?            Qu'est-ce que les abeilles produisent ?                     What do bees make?
96df9709-b201-4203-913c-3a201e6d4d98                 ¿Cuántos días tiene una semana?       Combien de jours y a-t-il dans une semaine ?           How many days are in a week?
ae0e0cd4-672f-4fda-afd4-52597412ca94                 ¿Cuál es la capital de Francia?              Quelle est la capitale de la France ?         What is the capital of France?

translation_join_id carries through datasets.combine: rows that came from a translated or localized dataset keep their value, and rows from other sources have none.

For recipes, preference pairs, row limits, and brand controls on an adaptation run, see Configure Adaptive Data.

The methods on this page each write a new dataset and leave the source unchanged. datasets.adapt also accepts translate, localize and expand, which add rows inside its own adaptation run without creating a new dataset. See Add rows and languages to a run.