lancedb/docs/web/curation/index.mdx

---
title: "Data curation"
sidebarTitle: "Overview"
description: "Prepare datasets for search, evaluation and training by profiling, selecting and combining records."
icon: filter
keywords: ["curation", "dataset", "training set", "sampling", "deduplication", "class balance"]
---

Data curation turns a collection of records into a dataset suited to a task.
You might select labeled examples for training, prepare a fixed evaluation
set, or remove duplicate documents from a search corpus.

Start with the result you need. Then inspect the source data, choose which
records belong, and save the selected data or the rules used to produce it.

## Choose a curation task {#questions}

| Task | What it helps you decide |
| --- | --- |
| [Create a Function](/curation/create-function) | How to run your own Python -- on CPU, on GPU, or against a remote service -- as part of SQL. |
| [Vend a Secret to a Function](/curation/vend-secret) | How a bound Secret's value actually reaches a running Function. |
| [Inspect Datasets](/curation/inspect-datasets) | How to query a table, a HuggingFace dataset, or an object-storage dataset the same way, and import without a separate download step. |
| [Chunk records](/curation/chunking) | How to split one record into several -- passages, clips, or tiles. |
| [Scope to what changed](/curation/changes) | How to process only the rows added since your last run. |
| [Profile a dataset](/curation/aggregate) | Whether coverage, missing values and class balance suit the task. |
| [Statistical Sampling](/curation/statistical-sampling) | Which random fraction to inspect or use for an experiment. |
| [Selective Sampling](/curation/selective-sampling) | How to re-fetch specific rows by ID, address or offset. |
| [Join datasets](/curation/join) | How to attach labels, metadata or scores stored elsewhere. |
| [Rank and deduplicate](/curation/windowing) | Which records to keep when several share an explicit key. |
| [Cluster by index partition](/curation/clustering) | How to group records by vector proximity without training a new clustering model. |
| [Deduplicate near-duplicates](/curation/dedupe) | Which records to collapse when they only look alike, with no shared key. |

The analytical examples in this section use **Enterprise SQL**. Connect with
the [SQL guide](/basics/sql). For a random subset through the Python SDK in
LanceDB Embedded or LanceDB Enterprise, see [Permutation](/training/permutation).

## Prepare a labeled document dataset {#shape}

Suppose you need one labeled record per document for a training experiment.
Inspect the source records, choose the latest version of each document, then
join its label. Finally, check the class balance of the selected records.

### Example data {#example-data}

Connect with an [Enterprise SQL client](/basics/sql). The examples use a small
document table and a separate label table. Expand the setup below to create them.

<Accordion title="Create the example tables">

`curation_documents` has six records describing four documents.
`curation_labels` has one label for each of three documents. `row_id` identifies
a record; `doc_id` identifies the document it describes.

Run each statement separately in a scratch database or namespace. Create these
tables once for the whole section, using fresh names if they already exist.

<CodeGroup>
```sql SQL icon="database"
CREATE TABLE curation_documents (
    row_id BIGINT, doc_id BIGINT, version BIGINT,
    source VARCHAR, caption VARCHAR, score DOUBLE
);
INSERT INTO curation_documents VALUES
    (1, 10, 1, 'web', NULL, 0.80),
    (2, 10, 2, 'web', 'A cat on a chair', 0.95),
    (3, 20, 1, 'books', 'A dog in a field', 0.90),
    (4, 30, 1, 'web', NULL, 0.70),
    (5, 30, 1, 'web', 'A cat by a window', 0.85),
    (6, 40, 1, 'books', 'A bird in a tree', 0.60);

CREATE TABLE curation_labels (doc_id BIGINT, label VARCHAR);
INSERT INTO curation_labels VALUES
    (10, 'cat'), (20, 'dog'), (30, 'cat');
```
</CodeGroup>

`score` is a sample quality score, not a search result score. Document 10 has
two versions. Document 30 has two records at the same version. Document 40
has no label. These cases let you check the effect of each operation.

</Accordion>

The following query chooses the latest record for each document, then keeps
only documents with a label:

<CodeGroup>
```sql SQL icon="database"
WITH ranked AS (
    SELECT *, row_number() OVER (
        PARTITION BY doc_id ORDER BY version DESC, row_id DESC
    ) AS rn
    FROM curation_documents
)
SELECT d.row_id, d.doc_id, d.caption, l.label
FROM ranked AS d
JOIN curation_labels AS l ON l.doc_id = d.doc_id
WHERE d.rn = 1
ORDER BY d.doc_id;
```
</CodeGroup>

The result contains documents **10, 20 and 30**, with labels **cat, dog and
cat**. Document 40 is excluded because this particular task requires a label.
The `row_id` tie-breaker makes the choice for document 30 deterministic;
choose a tie-breaker that fits your own data.

The example assumes one label per document. Duplicate keys in the label table
would multiply the result rows. The [joining guide](/curation/join#labels)
shows how to check that assumption.

## Check the selected dataset {#check-result}

Compare the result with the task's requirements: are the expected documents
present, are required fields populated, and is the class distribution suitable?
This example has two cats and one dog. That distribution may be appropriate
for exploration, but it does not establish a representative evaluation set.

[Profiling](/curation/aggregate) provides the counts to inspect.
[Statistical Sampling](/curation/statistical-sampling) and [ranking](/curation/windowing) help you
adjust the selection without changing the original records.

## Save and reuse the result {#next}

Choose the output according to how you will use it:

- **A fixed dataset:** save the query result in a new table with
  [`CREATE TABLE AS SELECT`](/sql/ddl/table#create). Keep that output unchanged
  when comparing experiments.
- **A reusable selection:** save the query as a [view](/management/view).
  A view stores the query definition, so it is not a frozen evaluation set.
- **Stored results that you refresh:** use a
  [materialized view](/management/materialized-view) when its supported query
  and refresh behavior fit the workflow.

Record the source versions and selection rules alongside the output. Continue
with [data loading](/training/data-loading) to use the curated data in a
training pipeline.