docs: scope the sampling label to SQL and fix its shuffle example

The page declared OSS unavailable, so the comparison read "Sampling |
No / Varies" and the sidebar tagged the whole page Enterprise, while its
last section described SDK sampling that works in both offerings. It is
now titled SQL sampling, so the row and tag describe TABLESAMPLE, and
the SDK alternative is a section of its own, marked for both offerings,
under its existing anchor.

That section's example passed Permutation.identity(table) to a
DataLoader, which reads rows in stored order. It now builds a seeded
shuffle and a random split with permutation_builder, checked locally
against the baseline SDK, including through a DataLoader.
This commit is contained in:
Brendan Clement
2026-10-09 09:13:45 -07:00
parent 6b67c9135a
commit bc9248162d
+30 -14
View File
@@ -1,7 +1,7 @@
---
title: "Sampling a dataset"
sidebarTitle: "Sampling"
description: "Drawing a subset you can afford to work with, and fixing it when it has to be reproducible."
title: "SQL sampling"
sidebarTitle: "SQL Sampling"
description: "Drawing a subset with SQL's TABLESAMPLE, and fixing it when it has to be reproducible."
icon: dice
keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"]
availability:
@@ -9,14 +9,17 @@ availability:
enterprise: varies
summary: >-
`TABLESAMPLE` is SQL, which runs on an Enterprise deployment with SQL
enabled. Shuffled reads for training work in both through the SDK; see
[Shuffling for training](#training).
enabled. Without SQL, the Python SDK samples and shuffles rows in both
offerings; see [Without SQL](#training).
---
Most curation starts with a subset. A question you can answer in ten seconds on
1% of the data is worth asking twenty times; the same question on the whole
table is worth asking once.
This page draws that subset with SQL's `TABLESAMPLE`. Without SQL, in either
offering, the Python SDK can do it too -- see [Without SQL](#training).
## A quick look {#quick}
<Badge color="red">Enterprise</Badge>
@@ -83,21 +86,34 @@ were drawn. Sampling 10% of each side does **not** give 10% of the join — it
gives roughly 1%. Sample once, after the join, unless you specifically want the
smaller result.
## Shuffling for training {#training}
## Without SQL {#training}
For a training loop rather than a query, the SDKs read a table in a shuffled
order without SQL at all, in the embedded library and on a deployment alike:
<Badge color="green">OSS</Badge> <Badge color="red">Enterprise</Badge>
Everything above needs SQL. The Python SDK's permutation builder samples and
shuffles without it, in the embedded library and on a deployment alike:
```python
from lancedb.permutation import Permutation
from lancedb.permutation import Permutation, permutation_builder
permutation = Permutation.identity(table)
dataloader = torch.utils.data.DataLoader(permutation)
# Every row, in a seeded shuffled order
shuffled = permutation_builder(table).shuffle(seed=42).execute()
permutation = Permutation.from_tables(table, shuffled)
# A reproducible 10% sample: the first of two random splits
splits = permutation_builder(table).split_random(ratios=[0.1, 0.9], seed=42).execute()
sample = Permutation.from_tables(table, splits, split=0)
for batch in sample.iter(batch_size=256):
...
```
This is the right tool for epoch-level shuffling, where you want every row
eventually and only the order varies. `TABLESAMPLE` is for when you want fewer
rows. See [Training](/training/data-loading).
`Permutation.identity(table)` reads rows in their stored order and does not
shuffle. A `Permutation` is a map-style PyTorch dataset, so either of the above
can be passed to `torch.utils.data.DataLoader`, which reads it in the
permutation's order. A shuffled order is the right tool for epoch-level
shuffling, where you want every row eventually and only the order varies; a
split is for when you want fewer rows. See [Training](/training/data-loading).
## Reference {#reference}