mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
Jack Ye
c220e63828
SQL now leads with why you'd reach for it beyond analytics: it's served over Flight SQL, so it's the way in for any language without a native LanceDB SDK, and broadly every public SDK method has a corresponding SQL statement. Closes by pointing at the SQL Reference as the spec that defines the precise behavior behind both the SQL surface and the SDK operations it mirrors. Moving both into Getting Started (before Agentic Skills) means a reader knows what the API surface even is before being told how to hand it to an agent.
55 lines
2.1 KiB
Plaintext
55 lines
2.1 KiB
Plaintext
---
|
|
title: "Data curation"
|
|
sidebarTitle: "Overview"
|
|
description: "Shaping a dataset before you train on it: profiling, sampling, joining, and ranking."
|
|
icon: filter
|
|
keywords: ["curation", "dataset", "training set", "sampling", "deduplication", "class balance"]
|
|
---
|
|
|
|
Curation is the work between having data and having a *dataset*: finding out
|
|
what you actually have, drawing a subset you can afford to train on, attaching
|
|
labels that live elsewhere, and picking one row per thing when you have several.
|
|
|
|
These pages are task-oriented. Each shows the operation in the context you would
|
|
reach for it, and links to the [SQL reference](/sql) for the complete syntax.
|
|
|
|
<Note>
|
|
Most curation is analytical -- grouping, joining, ranking -- which the query
|
|
builders do not do. It is therefore SQL, and SQL needs a deployment. See
|
|
[SQL](/basics/sql#sql-is-remote).
|
|
</Note>
|
|
|
|
## The four questions {#questions}
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| [What do I have?](/curation/aggregate) | counts, class balance, null coverage, distinct values |
|
|
| [What can I afford?](/curation/sampling) | a random subset, reproducible when you need it fixed |
|
|
| [What else do I know?](/curation/join) | labels, metadata and scores that live in another table |
|
|
| [Which one do I keep?](/curation/windowing) | one row per key, ranking within a group, splits |
|
|
|
|
## A worked shape {#shape}
|
|
|
|
The four compose into the same pipeline most curation ends up being:
|
|
|
|
```sql
|
|
WITH ranked AS (
|
|
SELECT d.*, c.label,
|
|
row_number() OVER (PARTITION BY d.doc_id ORDER BY d.version DESC) AS rn
|
|
FROM documents AS d
|
|
JOIN labels AS c ON c.doc_id = d.doc_id
|
|
)
|
|
SELECT * FROM ranked WHERE rn = 1
|
|
```
|
|
|
|
Join to bring in what you know, window to pick one row per document, filter to
|
|
keep it. Add [`TABLESAMPLE`](/curation/sampling) when the result is still larger
|
|
than you want, and [aggregate](/curation/aggregate) to check the balance of what
|
|
you ended up with.
|
|
|
|
## Then what {#next}
|
|
|
|
A curated result you want to keep is a
|
|
[materialized view](/management/materialized-view) -- stored, refreshed, and
|
|
readable by a [data loader](/training/data-loading) like any other table.
|