mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
Brendan Clement
6398ff3629
Databases, namespaces, jobs and data evolution work in both offerings. Materialized views, computed columns and column refresh work in the embedded library and vary on a deployment, which has to enable them. Views and functions exist only on a deployment that enables them, and aggregation, joins, windowing and sampling are SQL, so they need one with SQL enabled. Sections now say where they differ: views have SDK methods on a deployment, dropping a function needs a further setting, a local job handle runs in-process and has no history, create_index and refresh_column behave differently against a deployment, and the overviews no longer call computed columns and materialized views available everywhere.
106 lines
3.6 KiB
Plaintext
106 lines
3.6 KiB
Plaintext
---
|
|
title: "Sampling a dataset"
|
|
sidebarTitle: "Sampling"
|
|
description: "Drawing a subset you can afford to work with, and fixing it when it has to be reproducible."
|
|
icon: dice
|
|
keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"]
|
|
availability:
|
|
oss: unavailable
|
|
enterprise: varies
|
|
summary: >-
|
|
`TABLESAMPLE` is SQL, which runs on an Enterprise deployment with SQL
|
|
enabled. Shuffled reads for training work in both through the SDK; see
|
|
[Shuffling for training](#training).
|
|
---
|
|
|
|
Most curation starts with a subset. A question you can answer in ten seconds on
|
|
1% of the data is worth asking twenty times; the same question on the whole
|
|
table is worth asking once.
|
|
|
|
## A quick look {#quick}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
```sql
|
|
SELECT * FROM images TABLESAMPLE (1) -- 1 percent
|
|
SELECT * FROM images TABLESAMPLE (500 ROWS) -- 500 rows
|
|
```
|
|
|
|
A bare number is a percentage; add `ROWS` for a count. Use it while you are
|
|
exploring — the cost of the query drops with the sample, and so does the cost
|
|
of being wrong about what you wanted to ask.
|
|
|
|
## Rows or fragments {#methods}
|
|
|
|
```sql
|
|
SELECT * FROM images TABLESAMPLE BERNOULLI (10) -- rows
|
|
SELECT * FROM images TABLESAMPLE SYSTEM (10) -- fragments
|
|
```
|
|
|
|
`BERNOULLI` considers each row independently and reads across the whole table.
|
|
`SYSTEM` picks whole Lance fragments and takes the rows in them — far less I/O,
|
|
because it skips files rather than rows.
|
|
|
|
The trade is uniformity. A fragment sample is only as representative as your
|
|
write order. If rows arrived grouped by source, by date, or by label — which is
|
|
usual — then a fragment sample inherits that grouping.
|
|
|
|
<Note>
|
|
Reach for `SYSTEM` when you want a cheap look, and `BERNOULLI` when the number
|
|
you compute has to be defensible. Omitting the method gives `BERNOULLI`.
|
|
</Note>
|
|
|
|
## A fixed evaluation set {#fixed}
|
|
|
|
Without a seed, every execution resamples. That is fine for exploring and wrong
|
|
for evaluation — a metric computed on different rows each run is not comparable
|
|
with itself.
|
|
|
|
```sql
|
|
SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42)
|
|
```
|
|
|
|
The same seed against the same data gives the same rows, which is what makes a
|
|
sample usable as a held-out set. If the underlying data changes, so does the
|
|
draw — when you need the set frozen against that too, materialize it:
|
|
|
|
```sql
|
|
CREATE TABLE eval_set AS
|
|
SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42)
|
|
```
|
|
|
|
A [materialized view](/management/materialized-view) is the other option when
|
|
you want it to track the source on a refresh rather than stay frozen.
|
|
|
|
## Sampling both sides of a join {#joins}
|
|
|
|
```sql
|
|
SELECT * FROM a TABLESAMPLE (10) JOIN b TABLESAMPLE (10) ON a.id = b.id
|
|
```
|
|
|
|
Each side samples independently, so a row survives only if both of its sides
|
|
were drawn. Sampling 10% of each side does **not** give 10% of the join — it
|
|
gives roughly 1%. Sample once, after the join, unless you specifically want the
|
|
smaller result.
|
|
|
|
## Shuffling for training {#training}
|
|
|
|
For a training loop rather than a query, the SDKs read a table in a shuffled
|
|
order without SQL at all, in the embedded library and on a deployment alike:
|
|
|
|
```python
|
|
from lancedb.permutation import Permutation
|
|
|
|
permutation = Permutation.identity(table)
|
|
dataloader = torch.utils.data.DataLoader(permutation)
|
|
```
|
|
|
|
This is the right tool for epoch-level shuffling, where you want every row
|
|
eventually and only the order varies. `TABLESAMPLE` is for when you want fewer
|
|
rows. See [Training](/training/data-loading).
|
|
|
|
## Reference {#reference}
|
|
|
|
The full clause, its options and what it refuses are in
|
|
[Sampling](/sql/dql/sampling).
|