mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
Jack Ye
70c10e4aa4
lancedb now holds a page at every path the site publishes, including the twelve Enterprise ones. Those are real open-source pages -- what the capability is, and what the embedded form does instead -- rather than blank placeholders, so the open-source site is complete on its own. sophon becomes an overlay whose page replaces the one at the same path, which means navigation lives in one place and the two repositories only have to agree on a path. That removes the whole class of problem where a page could only be reached from one of the two trees. The open-source site went from twenty-four dangling Enterprise links to one, and that one points at the dataset pages the build repository owns. Ports four changes that landed on the docs repository after this branch left it: #344 indexing and quantization rewritten #354 LeRobot native reader #361 training/index renamed to training/data-loading and rewritten #362 distance metric mismatch falls back to a brute-force scan None applied as patches, because these files were merged or split here -- quantization into the vector index page, multi-vector into vector search, performance into four pages. Each was ported by taking the upstream page and re-applying the restructure on top, which also picked up a correction that would otherwise have been lost: the stricter brute-force threshold for multi-vector columns is a Cloud limit, not an Enterprise one. The upstream LeRobot page linked to a Geneva page that no longer exists; it now points at the feature-engineering pages that replaced it.
102 lines
3.5 KiB
Plaintext
102 lines
3.5 KiB
Plaintext
---
|
|
title: "Random sampling"
|
|
sidebarTitle: "Sampling"
|
|
description: "TABLESAMPLE: draw a random subset of a table, by rows or by fragments."
|
|
icon: dice
|
|
keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"]
|
|
---
|
|
|
|
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
|
|
|
`TABLESAMPLE` draws a random subset of a table. On a lakehouse table that is
|
|
often the difference between a question you can afford to ask and one you
|
|
cannot -- an aggregate over 1% of forty billion rows answers in a fraction of
|
|
the time, and for most estimates that is enough.
|
|
|
|
```sql
|
|
SELECT * FROM documents TABLESAMPLE (10) -- 10 percent
|
|
SELECT * FROM documents TABLESAMPLE (500 ROWS) -- 500 rows
|
|
```
|
|
|
|
A bare number is a **percentage**. Add `ROWS` for a count instead.
|
|
|
|
## Rows or fragments {#methods}
|
|
|
|
The standard calls these two methods of one clause. They are different
|
|
operations, and the difference matters on Lance:
|
|
|
|
```sql
|
|
SELECT * FROM documents TABLESAMPLE BERNOULLI (10)
|
|
SELECT * FROM documents TABLESAMPLE SYSTEM (10)
|
|
```
|
|
|
|
| Method | Aliases | What it picks |
|
|
| --- | --- | --- |
|
|
| `BERNOULLI` | `ROW` | rows, each considered independently |
|
|
| `SYSTEM` | `BLOCK` | **Lance fragments**, and takes whatever rows they hold |
|
|
|
|
`BERNOULLI` gives a genuinely uniform sample and reads across the whole table to
|
|
get it. `SYSTEM` reads far less -- it skips entire fragments -- but the sample is
|
|
only as uniform as the data's layout. If rows were written in an order that
|
|
correlates with what you are measuring, a fragment sample inherits that
|
|
correlation.
|
|
|
|
Reach for `SYSTEM` when you want a cheap look at the data and `BERNOULLI` when
|
|
the number you compute has to be defensible. Omitting the method gives
|
|
`BERNOULLI`.
|
|
|
|
## Repeatability {#repeatable}
|
|
|
|
Without a seed, every execution resamples -- running the same query twice gives
|
|
different rows. `REPEATABLE` fixes the draw:
|
|
|
|
```sql
|
|
SELECT * FROM documents TABLESAMPLE (10) REPEATABLE (42)
|
|
```
|
|
|
|
This is what makes a sample usable as a **fixed evaluation set**: the same seed
|
|
against the same data gives the same rows, so a metric computed today is
|
|
comparable with one computed last week.
|
|
|
|
## Where the clause can go {#placement}
|
|
|
|
It attaches to a table in `FROM`, including inside a subquery, a CTE, or either
|
|
side of a join:
|
|
|
|
```sql
|
|
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
|
|
|
|
SELECT * FROM a TABLESAMPLE (5 ROWS) JOIN b ON a.id = b.id
|
|
|
|
SELECT * FROM t a TABLESAMPLE (10) JOIN t b TABLESAMPLE (10) ON a.id = b.id
|
|
```
|
|
|
|
Each side of a join samples independently. Sampling both sides of a join
|
|
shrinks the *result* far more than either fraction suggests, because a row only
|
|
survives if both its sides were drawn.
|
|
|
|
## What is refused {#refused}
|
|
|
|
Sampling is semantics, not a hint. A clause that were read and then not applied
|
|
would be a wrong answer rather than a missing feature, so anything that cannot
|
|
be honoured exactly is an error:
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| `TABLESAMPLE (101)` | a percentage outside `0..=100` is refused, not clamped |
|
|
| `TABLESAMPLE (1.5 ROWS)` | a row count must be whole |
|
|
| `TABLESAMPLE (BUCKET 1 OUT OF 4)` | bucket sampling is not supported |
|
|
| the `SAMPLE` spelling | use `TABLESAMPLE` |
|
|
| the `OFFSET` modifier | not supported |
|
|
|
|
<Note>
|
|
`TABLESAMPLE` is also refused on a WAL-backed table, where the rows a sample
|
|
would draw from are not all in the fragments yet.
|
|
</Note>
|
|
|
|
## Sampling outside SQL {#sdk}
|
|
|
|
For a training loop rather than a query, the SDKs read a table in a shuffled
|
|
order without going through SQL at all -- see [Training](/training/data-loading), where a
|
|
permutation drives the data loader.
|