mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
Brendan Clement
6398ff3629
Databases, namespaces, jobs and data evolution work in both offerings. Materialized views, computed columns and column refresh work in the embedded library and vary on a deployment, which has to enable them. Views and functions exist only on a deployment that enables them, and aggregation, joins, windowing and sampling are SQL, so they need one with SQL enabled. Sections now say where they differ: views have SDK methods on a deployment, dropping a function needs a further setting, a local job handle runs in-process and has no history, create_index and refresh_column behave differently against a deployment, and the overviews no longer call computed columns and materialized views available everywhere.
58 lines
2.3 KiB
Plaintext
58 lines
2.3 KiB
Plaintext
---
|
|
title: "Data curation"
|
|
sidebarTitle: "Overview"
|
|
description: "Shaping a dataset before you train on it: profiling, sampling, joining, and ranking."
|
|
icon: filter
|
|
keywords: ["curation", "dataset", "training set", "sampling", "deduplication", "class balance"]
|
|
---
|
|
|
|
Curation is the work between having data and having a *dataset*: finding out
|
|
what you actually have, drawing a subset you can afford to train on, attaching
|
|
labels that live elsewhere, and picking one row per thing when you have several.
|
|
|
|
These pages are task-oriented. Each shows the operation in the context you would
|
|
reach for it, and links to the [SQL reference](/sql) for the complete syntax.
|
|
|
|
<Note>
|
|
Most curation is analytical -- grouping, joining, ranking -- which the query
|
|
builders do not do. It is therefore SQL, and SQL needs a deployment with its
|
|
SQL endpoint enabled. See [SQL](/basics/sql#sql-is-remote).
|
|
</Note>
|
|
|
|
## The four questions {#questions}
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| [What do I have?](/curation/aggregate) | counts, class balance, null coverage, distinct values |
|
|
| [What can I afford?](/curation/sampling) | a random subset, reproducible when you need it fixed |
|
|
| [What else do I know?](/curation/join) | labels, metadata and scores that live in another table |
|
|
| [Which one do I keep?](/curation/windowing) | one row per key, ranking within a group, splits |
|
|
|
|
## A worked shape {#shape}
|
|
|
|
The four compose into the same pipeline most curation ends up being:
|
|
|
|
```sql
|
|
WITH ranked AS (
|
|
SELECT d.*, c.label,
|
|
row_number() OVER (PARTITION BY d.doc_id ORDER BY d.version DESC) AS rn
|
|
FROM documents AS d
|
|
JOIN labels AS c ON c.doc_id = d.doc_id
|
|
)
|
|
SELECT * FROM ranked WHERE rn = 1
|
|
```
|
|
|
|
Join to bring in what you know, window to pick one row per document, filter to
|
|
keep it. Add [`TABLESAMPLE`](/curation/sampling) when the result is still larger
|
|
than you want, and [aggregate](/curation/aggregate) to check the balance of what
|
|
you ended up with.
|
|
|
|
## Then what {#next}
|
|
|
|
A curated result you want to keep is a
|
|
[materialized view](/management/materialized-view) -- stored, refreshed, and
|
|
readable by a [data loader](/training/data-loading) like any other table.
|
|
Keeping a joined or windowed result this way takes SQL's
|
|
`CREATE MATERIALIZED VIEW` on a deployment that enables materialized views; the
|
|
SDK's `create_materialized_view` reads a single table.
|