mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
Jack Ye
70c10e4aa4
lancedb now holds a page at every path the site publishes, including the twelve Enterprise ones. Those are real open-source pages -- what the capability is, and what the embedded form does instead -- rather than blank placeholders, so the open-source site is complete on its own. sophon becomes an overlay whose page replaces the one at the same path, which means navigation lives in one place and the two repositories only have to agree on a path. That removes the whole class of problem where a page could only be reached from one of the two trees. The open-source site went from twenty-four dangling Enterprise links to one, and that one points at the dataset pages the build repository owns. Ports four changes that landed on the docs repository after this branch left it: #344 indexing and quantization rewritten #354 LeRobot native reader #361 training/index renamed to training/data-loading and rewritten #362 distance metric mismatch falls back to a brute-force scan None applied as patches, because these files were merged or split here -- quantization into the vector index page, multi-vector into vector search, performance into four pages. Each was ported by taking the upstream page and re-applying the restructure on top, which also picked up a correction that would otherwise have been lost: the stricter brute-force threshold for multi-vector columns is a Cloud limit, not an Enterprise one. The upstream LeRobot page linked to a Geneva page that no longer exists; it now points at the feature-engineering pages that replaced it.
99 lines
3.3 KiB
Plaintext
99 lines
3.3 KiB
Plaintext
---
|
|
title: "Sampling a dataset"
|
|
sidebarTitle: "Sampling"
|
|
description: "Drawing a subset you can afford to work with, and fixing it when it has to be reproducible."
|
|
icon: dice
|
|
keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"]
|
|
---
|
|
|
|
Most curation starts with a subset. A question you can answer in ten seconds on
|
|
1% of the data is worth asking twenty times; the same question on the whole
|
|
table is worth asking once.
|
|
|
|
## A quick look {#quick}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
```sql
|
|
SELECT * FROM images TABLESAMPLE (1) -- 1 percent
|
|
SELECT * FROM images TABLESAMPLE (500 ROWS) -- 500 rows
|
|
```
|
|
|
|
A bare number is a percentage; add `ROWS` for a count. Use it while you are
|
|
exploring — the cost of the query drops with the sample, and so does the cost
|
|
of being wrong about what you wanted to ask.
|
|
|
|
## Rows or fragments {#methods}
|
|
|
|
```sql
|
|
SELECT * FROM images TABLESAMPLE BERNOULLI (10) -- rows
|
|
SELECT * FROM images TABLESAMPLE SYSTEM (10) -- fragments
|
|
```
|
|
|
|
`BERNOULLI` considers each row independently and reads across the whole table.
|
|
`SYSTEM` picks whole Lance fragments and takes the rows in them — far less I/O,
|
|
because it skips files rather than rows.
|
|
|
|
The trade is uniformity. A fragment sample is only as representative as your
|
|
write order. If rows arrived grouped by source, by date, or by label — which is
|
|
usual — then a fragment sample inherits that grouping.
|
|
|
|
<Note>
|
|
Reach for `SYSTEM` when you want a cheap look, and `BERNOULLI` when the number
|
|
you compute has to be defensible. Omitting the method gives `BERNOULLI`.
|
|
</Note>
|
|
|
|
## A fixed evaluation set {#fixed}
|
|
|
|
Without a seed, every execution resamples. That is fine for exploring and wrong
|
|
for evaluation — a metric computed on different rows each run is not comparable
|
|
with itself.
|
|
|
|
```sql
|
|
SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42)
|
|
```
|
|
|
|
The same seed against the same data gives the same rows, which is what makes a
|
|
sample usable as a held-out set. If the underlying data changes, so does the
|
|
draw — when you need the set frozen against that too, materialize it:
|
|
|
|
```sql
|
|
CREATE TABLE eval_set AS
|
|
SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42)
|
|
```
|
|
|
|
A [materialized view](/management/materialized-view) is the other option when
|
|
you want it to track the source on a refresh rather than stay frozen.
|
|
|
|
## Sampling both sides of a join {#joins}
|
|
|
|
```sql
|
|
SELECT * FROM a TABLESAMPLE (10) JOIN b TABLESAMPLE (10) ON a.id = b.id
|
|
```
|
|
|
|
Each side samples independently, so a row survives only if both of its sides
|
|
were drawn. Sampling 10% of each side does **not** give 10% of the join — it
|
|
gives roughly 1%. Sample once, after the join, unless you specifically want the
|
|
smaller result.
|
|
|
|
## Shuffling for training {#training}
|
|
|
|
For a training loop rather than a query, the SDKs read a table in a shuffled
|
|
order without SQL at all:
|
|
|
|
```python
|
|
from lancedb.permutation import Permutation
|
|
|
|
permutation = Permutation.identity(table)
|
|
dataloader = torch.utils.data.DataLoader(permutation)
|
|
```
|
|
|
|
This is the right tool for epoch-level shuffling, where you want every row
|
|
eventually and only the order varies. `TABLESAMPLE` is for when you want fewer
|
|
rows. See [Training](/training/data-loading).
|
|
|
|
## Reference {#reference}
|
|
|
|
The full clause, its options and what it refuses are in
|
|
[Sampling](/sql/sampling).
|