mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-14 07:42:28 +00:00
Jack Ye
fd177e0aea
The tab now runs write, read, curate, derive, train before it reaches the management side: Ingestion, Semantic Search, Data Curation, Feature Engineering, Training, then Management and Operations. Indexing joins Operations, where the index maintenance page already was, and gains it as Build & Repair. Ingestion gains a Create Table page. It was a section inside the overview, so the one thing a new reader needs first was the one thing the sidebar did not show. Data Curation is new: Aggregate, Sampling, Join, Windowing. The SQL reference answers what the syntax is; these answer why you would reach for it, framed as the questions curation actually asks -- what do I have, what can I afford, what else do I know, which one do I keep. Each links to the reference for the complete grammar rather than repeating it. They carry the things a reference page would not: that count(*) minus count(col) is your coverage, that sampling both sides of a join gives a percent rather than ten, that a window ordered by a value you recompute moves rows between train and test, and that a one percent sample tells you about the head and nothing about the tail.
55 lines
2.1 KiB
Plaintext
55 lines
2.1 KiB
Plaintext
---
|
|
title: "Data curation"
|
|
sidebarTitle: "Overview"
|
|
description: "Shaping a dataset before you train on it: profiling, sampling, joining, and ranking."
|
|
icon: filter
|
|
keywords: ["curation", "dataset", "training set", "sampling", "deduplication", "class balance"]
|
|
---
|
|
|
|
Curation is the work between having data and having a *dataset*: finding out
|
|
what you actually have, drawing a subset you can afford to train on, attaching
|
|
labels that live elsewhere, and picking one row per thing when you have several.
|
|
|
|
These pages are task-oriented. Each shows the operation in the context you would
|
|
reach for it, and links to the [SQL reference](/sql) for the complete syntax.
|
|
|
|
<Note>
|
|
Most curation is analytical -- grouping, joining, ranking -- which the query
|
|
builders do not do. It is therefore SQL, and SQL needs a deployment. See
|
|
[SDKs and SQL](/basics/apis).
|
|
</Note>
|
|
|
|
## The four questions {#questions}
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| [What do I have?](/curation/aggregate) | counts, class balance, null coverage, distinct values |
|
|
| [What can I afford?](/curation/sampling) | a random subset, reproducible when you need it fixed |
|
|
| [What else do I know?](/curation/join) | labels, metadata and scores that live in another table |
|
|
| [Which one do I keep?](/curation/windowing) | one row per key, ranking within a group, splits |
|
|
|
|
## A worked shape {#shape}
|
|
|
|
The four compose into the same pipeline most curation ends up being:
|
|
|
|
```sql
|
|
WITH ranked AS (
|
|
SELECT d.*, c.label,
|
|
row_number() OVER (PARTITION BY d.doc_id ORDER BY d.version DESC) AS rn
|
|
FROM documents AS d
|
|
JOIN labels AS c ON c.doc_id = d.doc_id
|
|
)
|
|
SELECT * FROM ranked WHERE rn = 1
|
|
```
|
|
|
|
Join to bring in what you know, window to pick one row per document, filter to
|
|
keep it. Add [`TABLESAMPLE`](/curation/sampling) when the result is still larger
|
|
than you want, and [aggregate](/curation/aggregate) to check the balance of what
|
|
you ended up with.
|
|
|
|
## Then what {#next}
|
|
|
|
A curated result you want to keep is a
|
|
[materialized view](/management/materialized-view) -- stored, refreshed, and
|
|
readable by a [data loader](/training) like any other table.
|