mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 14:52:26 +00:00
Brendan Clement
bc9248162d
The page declared OSS unavailable, so the comparison read "Sampling | No / Varies" and the sidebar tagged the whole page Enterprise, while its last section described SDK sampling that works in both offerings. It is now titled SQL sampling, so the row and tag describe TABLESAMPLE, and the SDK alternative is a section of its own, marked for both offerings, under its existing anchor. That section's example passed Permutation.identity(table) to a DataLoader, which reads rows in stored order. It now builds a seeded shuffle and a random split with permutation_builder, checked locally against the baseline SDK, including through a DataLoader.
122 lines
4.3 KiB
Plaintext
122 lines
4.3 KiB
Plaintext
---
|
|
title: "SQL sampling"
|
|
sidebarTitle: "SQL Sampling"
|
|
description: "Drawing a subset with SQL's TABLESAMPLE, and fixing it when it has to be reproducible."
|
|
icon: dice
|
|
keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"]
|
|
availability:
|
|
oss: unavailable
|
|
enterprise: varies
|
|
summary: >-
|
|
`TABLESAMPLE` is SQL, which runs on an Enterprise deployment with SQL
|
|
enabled. Without SQL, the Python SDK samples and shuffles rows in both
|
|
offerings; see [Without SQL](#training).
|
|
---
|
|
|
|
Most curation starts with a subset. A question you can answer in ten seconds on
|
|
1% of the data is worth asking twenty times; the same question on the whole
|
|
table is worth asking once.
|
|
|
|
This page draws that subset with SQL's `TABLESAMPLE`. Without SQL, in either
|
|
offering, the Python SDK can do it too -- see [Without SQL](#training).
|
|
|
|
## A quick look {#quick}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
```sql
|
|
SELECT * FROM images TABLESAMPLE (1) -- 1 percent
|
|
SELECT * FROM images TABLESAMPLE (500 ROWS) -- 500 rows
|
|
```
|
|
|
|
A bare number is a percentage; add `ROWS` for a count. Use it while you are
|
|
exploring — the cost of the query drops with the sample, and so does the cost
|
|
of being wrong about what you wanted to ask.
|
|
|
|
## Rows or fragments {#methods}
|
|
|
|
```sql
|
|
SELECT * FROM images TABLESAMPLE BERNOULLI (10) -- rows
|
|
SELECT * FROM images TABLESAMPLE SYSTEM (10) -- fragments
|
|
```
|
|
|
|
`BERNOULLI` considers each row independently and reads across the whole table.
|
|
`SYSTEM` picks whole Lance fragments and takes the rows in them — far less I/O,
|
|
because it skips files rather than rows.
|
|
|
|
The trade is uniformity. A fragment sample is only as representative as your
|
|
write order. If rows arrived grouped by source, by date, or by label — which is
|
|
usual — then a fragment sample inherits that grouping.
|
|
|
|
<Note>
|
|
Reach for `SYSTEM` when you want a cheap look, and `BERNOULLI` when the number
|
|
you compute has to be defensible. Omitting the method gives `BERNOULLI`.
|
|
</Note>
|
|
|
|
## A fixed evaluation set {#fixed}
|
|
|
|
Without a seed, every execution resamples. That is fine for exploring and wrong
|
|
for evaluation — a metric computed on different rows each run is not comparable
|
|
with itself.
|
|
|
|
```sql
|
|
SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42)
|
|
```
|
|
|
|
The same seed against the same data gives the same rows, which is what makes a
|
|
sample usable as a held-out set. If the underlying data changes, so does the
|
|
draw — when you need the set frozen against that too, materialize it:
|
|
|
|
```sql
|
|
CREATE TABLE eval_set AS
|
|
SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42)
|
|
```
|
|
|
|
A [materialized view](/management/materialized-view) is the other option when
|
|
you want it to track the source on a refresh rather than stay frozen.
|
|
|
|
## Sampling both sides of a join {#joins}
|
|
|
|
```sql
|
|
SELECT * FROM a TABLESAMPLE (10) JOIN b TABLESAMPLE (10) ON a.id = b.id
|
|
```
|
|
|
|
Each side samples independently, so a row survives only if both of its sides
|
|
were drawn. Sampling 10% of each side does **not** give 10% of the join — it
|
|
gives roughly 1%. Sample once, after the join, unless you specifically want the
|
|
smaller result.
|
|
|
|
## Without SQL {#training}
|
|
|
|
<Badge color="green">OSS</Badge> <Badge color="red">Enterprise</Badge>
|
|
|
|
Everything above needs SQL. The Python SDK's permutation builder samples and
|
|
shuffles without it, in the embedded library and on a deployment alike:
|
|
|
|
```python
|
|
from lancedb.permutation import Permutation, permutation_builder
|
|
|
|
# Every row, in a seeded shuffled order
|
|
shuffled = permutation_builder(table).shuffle(seed=42).execute()
|
|
permutation = Permutation.from_tables(table, shuffled)
|
|
|
|
# A reproducible 10% sample: the first of two random splits
|
|
splits = permutation_builder(table).split_random(ratios=[0.1, 0.9], seed=42).execute()
|
|
sample = Permutation.from_tables(table, splits, split=0)
|
|
|
|
for batch in sample.iter(batch_size=256):
|
|
...
|
|
```
|
|
|
|
`Permutation.identity(table)` reads rows in their stored order and does not
|
|
shuffle. A `Permutation` is a map-style PyTorch dataset, so either of the above
|
|
can be passed to `torch.utils.data.DataLoader`, which reads it in the
|
|
permutation's order. A shuffled order is the right tool for epoch-level
|
|
shuffling, where you want every row eventually and only the order varies; a
|
|
split is for when you want fewer rows. See [Training](/training/data-loading).
|
|
|
|
## Reference {#reference}
|
|
|
|
The full clause, its options and what it refuses are in
|
|
[Sampling](/sql/dql/sampling).
|