Skip to content
SupportLogin
Edit on GitHub

Invent a Dataset

Generate a 100-row medical Q&A dataset from a single prompt, download it, and score its diversity with DCScore.

This runbook generates a training-ready dataset from scratch with Invent a Dataset. You write one dataset_prompt and need no source data. The example builds a 100-row medical question-answer dataset, then measures how varied the rows are.

Read more on the blog: Invent a Dataset: Data Generation for Custom Models and Invent a Dataset: Measuring Dataset Generation Abilities with Zero Seed Data.

You need an Adaption API key. See Create API keys. In Colab, add it to the secrets panel as ADAPTION_API_KEY. Locally, set it as an environment variable.

Terminal window
pip install adaption
import os
from adaption import Adaption
if "ADAPTION_API_KEY" not in os.environ:
try:
from google.colab import userdata # Colab secrets
os.environ["ADAPTION_API_KEY"] = userdata.get("ADAPTION_API_KEY")
except ImportError:
from getpass import getpass
os.environ["ADAPTION_API_KEY"] = getpass("ADAPTION_API_KEY: ")
client = Adaption()

Ask for an estimate first. Passing estimate=True returns the credit cost without starting a run.

estimate = client.datasets.invent(
dataset_prompt=(
"A dataset of medical question-answer pairs covering diagnoses, treatments, "
"drug explanations, and healthcare roles."
),
rows=100,
estimate=True,
)
print(f"Estimated credits: {estimate.estimated_credits} | Available credits: {estimate.available_credits}")

Drop estimate=True and give the dataset a name. training_type="instruction_dataset" produces prompt and completion pairs. To invent preference pairs for alignment, pass training_type="preference_pairs" instead.

dataset = client.datasets.invent(
name="Clinical Q&A",
dataset_prompt=(
"A dataset of medical question-answer pairs covering diagnoses, treatments, "
"drug explanations, and healthcare roles."
),
rows=100,
training_type="instruction_dataset",
)
dataset_id = dataset.id
print(dataset_id, dataset.status) # running

Invent can also produce the dataset in other languages and locales. See Expand across languages and locales.

The run is asynchronous. wait_for_completion polls until the dataset is ready.

record = client.datasets.wait_for_completion(dataset_id)
print(f"Generated {record.row_count} rows")
import pandas as pd
client.datasets.download(
dataset_id,
file_format="jsonl",
).write_to_file("clinical-qa.jsonl")
df = pd.read_json("clinical-qa.jsonl", lines=True)
df.head()

In Colab, google.colab.data_table.DataTable(df) gives a sortable, searchable table.

DCScore measures dataset diversity by treating it as a classification task over embeddings. The score runs from 1 (every row identical) to N (every row distinct), so the example also reports it divided by N.

Fetch the reference code and the embedding library. Rows are embedded with all-mpnet-base-v2, the default text model in the DCScore paper.

Terminal window
git clone -q https://github.com/BlueWhaleLab/DCScore
pip install -q sentence-transformers
import json
import sys
sys.path.append("DCScore")
from dcscore_function import DCScore
def evaluate_diversity(texts, model="sentence-transformers/all-mpnet-base-v2", batch_size=128, tau=0.1):
"""Return DCScore for a list of texts. 1 = all identical, len(texts) = all distinct."""
evaluator = DCScore(model)
embeddings, n, _ = evaluator.get_embedding(texts, batch_size=batch_size)
score = evaluator.calculate_dcscore_by_embedding(embeddings, kernel_type="cs", tau=tau)
return {"dcscore": score, "normalized": score / n, "n": n}
rows = []
with open("clinical-qa.jsonl", encoding="utf-8") as file:
for line in file:
if line.strip():
rows.append(json.loads(line))
data = [f"{r['enhanced_prompt']}\n\n{r['enhanced_completion']}" for r in rows]
result = evaluate_diversity(data)
print(f"Diversity score: {result['dcscore']:.1f} / {result['n']} rows (normalized {result['normalized']:.3f})")