mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 23:32:27 +00:00
Jack Ye
70c10e4aa4
lancedb now holds a page at every path the site publishes, including the twelve Enterprise ones. Those are real open-source pages -- what the capability is, and what the embedded form does instead -- rather than blank placeholders, so the open-source site is complete on its own. sophon becomes an overlay whose page replaces the one at the same path, which means navigation lives in one place and the two repositories only have to agree on a path. That removes the whole class of problem where a page could only be reached from one of the two trees. The open-source site went from twenty-four dangling Enterprise links to one, and that one points at the dataset pages the build repository owns. Ports four changes that landed on the docs repository after this branch left it: #344 indexing and quantization rewritten #354 LeRobot native reader #361 training/index renamed to training/data-loading and rewritten #362 distance metric mismatch falls back to a brute-force scan None applied as patches, because these files were merged or split here -- quantization into the vector index page, multi-vector into vector search, performance into four pages. Each was ported by taking the upstream page and re-applying the restructure on top, which also picked up a correction that would otherwise have been lost: the stricter brute-force threshold for multi-vector columns is a Cloud limit, not an Enterprise one. The upstream LeRobot page linked to a Geneva page that no longer exists; it now points at the feature-engineering pages that replaced it.
55 lines
2.1 KiB
Plaintext
55 lines
2.1 KiB
Plaintext
---
|
|
title: "Data curation"
|
|
sidebarTitle: "Overview"
|
|
description: "Shaping a dataset before you train on it: profiling, sampling, joining, and ranking."
|
|
icon: filter
|
|
keywords: ["curation", "dataset", "training set", "sampling", "deduplication", "class balance"]
|
|
---
|
|
|
|
Curation is the work between having data and having a *dataset*: finding out
|
|
what you actually have, drawing a subset you can afford to train on, attaching
|
|
labels that live elsewhere, and picking one row per thing when you have several.
|
|
|
|
These pages are task-oriented. Each shows the operation in the context you would
|
|
reach for it, and links to the [SQL reference](/sql) for the complete syntax.
|
|
|
|
<Note>
|
|
Most curation is analytical -- grouping, joining, ranking -- which the query
|
|
builders do not do. It is therefore SQL, and SQL needs a deployment. See
|
|
[SDKs and SQL](/basics/apis).
|
|
</Note>
|
|
|
|
## The four questions {#questions}
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| [What do I have?](/curation/aggregate) | counts, class balance, null coverage, distinct values |
|
|
| [What can I afford?](/curation/sampling) | a random subset, reproducible when you need it fixed |
|
|
| [What else do I know?](/curation/join) | labels, metadata and scores that live in another table |
|
|
| [Which one do I keep?](/curation/windowing) | one row per key, ranking within a group, splits |
|
|
|
|
## A worked shape {#shape}
|
|
|
|
The four compose into the same pipeline most curation ends up being:
|
|
|
|
```sql
|
|
WITH ranked AS (
|
|
SELECT d.*, c.label,
|
|
row_number() OVER (PARTITION BY d.doc_id ORDER BY d.version DESC) AS rn
|
|
FROM documents AS d
|
|
JOIN labels AS c ON c.doc_id = d.doc_id
|
|
)
|
|
SELECT * FROM ranked WHERE rn = 1
|
|
```
|
|
|
|
Join to bring in what you know, window to pick one row per document, filter to
|
|
keep it. Add [`TABLESAMPLE`](/curation/sampling) when the result is still larger
|
|
than you want, and [aggregate](/curation/aggregate) to check the balance of what
|
|
you ended up with.
|
|
|
|
## Then what {#next}
|
|
|
|
A curated result you want to keep is a
|
|
[materialized view](/management/materialized-view) -- stored, refreshed, and
|
|
readable by a [data loader](/training/data-loading) like any other table.
|