mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-16 16:52:34 +00:00
Brendan Clement
06aa526ae1
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The curation, SQL and operations pages contain outdated claims and examples that do not reliably work as written. This corrects those pages against the reviewed SDK and local server evidence, including prepared statements, background jobs, cleanup and monitoring. Integration guides are grouped with their examples in #4533. Function and Secret-object documentation stays unpublished until its release. The two Function pages are listed in `.mintignore` and left out of the navigation, and the sections about them on other pages are MDX comments. Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after #4533 is squash-merged. Local assembly, Mint validation and link checks pass, and the assembled site shows no Function or Secret-object content. A local release rehearsal that uncomments the drafts and restores the navigation also passes. Hold publication until this PR and lancedb/sophon#8688 are on `main`.
103 lines
3.6 KiB
Plaintext
103 lines
3.6 KiB
Plaintext
---
|
|
title: "Random sampling"
|
|
sidebarTitle: "Sampling"
|
|
description: "TABLESAMPLE: draw a random subset of a table, by rows or by fragments."
|
|
icon: dice
|
|
keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"]
|
|
---
|
|
|
|
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
|
|
|
Use `TABLESAMPLE` to explore a subset of a table or estimate aggregate
|
|
statistics. The sampling method determines how much data is read and whether
|
|
the table's physical layout influences the sample.
|
|
|
|
```sql
|
|
SELECT * FROM documents TABLESAMPLE (10) -- 10 percent
|
|
SELECT * FROM documents TABLESAMPLE (500 ROWS) -- 500 rows
|
|
```
|
|
|
|
A bare number is a **percentage**. Add `ROWS` for a count instead.
|
|
|
|
## Rows or fragments {#methods}
|
|
|
|
The standard calls these two methods of one clause. They are different
|
|
operations, and the difference matters on Lance:
|
|
|
|
```sql
|
|
SELECT * FROM documents TABLESAMPLE BERNOULLI (10)
|
|
SELECT * FROM documents TABLESAMPLE SYSTEM (10)
|
|
```
|
|
|
|
| Method | Aliases | What it picks |
|
|
| --- | --- | --- |
|
|
| `BERNOULLI` | `ROW` | rows, each considered independently |
|
|
| `SYSTEM` | `BLOCK` | **Lance fragments**, and takes whatever rows they hold |
|
|
|
|
`BERNOULLI` gives a genuinely uniform sample and reads across the whole table to
|
|
get it. `SYSTEM` reads far less -- it skips entire fragments -- but the sample is
|
|
only as uniform as the data's layout. If rows were written in an order that
|
|
correlates with what you are measuring, a fragment sample inherits that
|
|
correlation.
|
|
|
|
Use `SYSTEM` for a quick look at the data and `BERNOULLI` when you need
|
|
independent row sampling. Omitting the method gives
|
|
`BERNOULLI`.
|
|
|
|
## Repeatability {#repeatable}
|
|
|
|
Without a seed, every execution resamples -- running the same query twice gives
|
|
different rows. `REPEATABLE` fixes the draw:
|
|
|
|
```sql
|
|
SELECT * FROM documents TABLESAMPLE (10) REPEATABLE (42)
|
|
```
|
|
|
|
Use the same seed and table snapshot to reproduce a sampling run. For a fixed
|
|
evaluation set, save the selected row IDs or materialize the sample and retain
|
|
its version. A seed alone does not preserve membership after source changes
|
|
or changes to the physical layout.
|
|
|
|
## Where the clause can go {#placement}
|
|
|
|
It attaches to a table in `FROM`, including inside a subquery, a CTE, or either
|
|
side of a join:
|
|
|
|
```sql
|
|
SELECT * FROM (SELECT * FROM documents TABLESAMPLE (10)) d
|
|
|
|
SELECT * FROM a TABLESAMPLE (5 ROWS) JOIN b ON a.id = b.id
|
|
|
|
SELECT * FROM t a TABLESAMPLE (10) JOIN t b TABLESAMPLE (10) ON a.id = b.id
|
|
```
|
|
|
|
Each side of a join samples independently. Sampling both sides of a join
|
|
shrinks the *result* far more than either fraction suggests, because a row only
|
|
survives if both its sides were drawn.
|
|
|
|
## Unsupported sampling forms {#refused}
|
|
|
|
Sampling is semantics, not a hint. A clause that were read and then not applied
|
|
would be a wrong answer rather than a missing feature, so anything that cannot
|
|
be honoured exactly is an error:
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| `TABLESAMPLE (101)` | a percentage outside `0..=100` is refused, not clamped |
|
|
| `TABLESAMPLE (1.5 ROWS)` | a row count must be whole |
|
|
| `TABLESAMPLE (BUCKET 1 OUT OF 4)` | bucket sampling is not supported |
|
|
| `(SELECT ...) d TABLESAMPLE (10)` | a subquery's result cannot be sampled; sample the table inside it |
|
|
| the `SAMPLE` spelling | use `TABLESAMPLE` |
|
|
| the `OFFSET` modifier | not supported |
|
|
|
|
<Note>
|
|
`TABLESAMPLE` is also refused on a WAL-backed table, where the rows a sample
|
|
would draw from are not all in the fragments yet.
|
|
</Note>
|
|
|
|
## Sampling outside SQL {#sdk}
|
|
|
|
For a training loop rather than a query, the SDKs read a table in a shuffled
|
|
order without going through SQL at all -- see [Training](/training/data-loading), where a
|
|
permutation drives the data loader.
|