---
title: "SQL sampling"
sidebarTitle: "SQL Sampling"
description: "Drawing a subset with SQL's TABLESAMPLE, and fixing it when it has to be reproducible."
icon: dice
keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"]
availability:
  oss: unavailable
  enterprise: varies
  summary: >-
    `TABLESAMPLE` is SQL, which runs on an Enterprise deployment with SQL
    enabled. Without SQL, the Python SDK samples and shuffles rows in both
    offerings; see [Without SQL](#training).
---

Most curation starts with a subset. A question you can answer in ten seconds on
1% of the data is worth asking twenty times; the same question on the whole
table is worth asking once.

This page draws that subset with SQL's `TABLESAMPLE`. Without SQL, in either
offering, the Python SDK can do it too -- see [Without SQL](#training).

## A quick look {#quick}

<Badge color="red">Enterprise</Badge>

```sql
SELECT * FROM images TABLESAMPLE (1)       -- 1 percent
SELECT * FROM images TABLESAMPLE (500 ROWS) -- 500 rows
```

A bare number is a percentage; add `ROWS` for a count. Use it while you are
exploring — the cost of the query drops with the sample, and so does the cost
of being wrong about what you wanted to ask.

## Rows or fragments {#methods}

```sql
SELECT * FROM images TABLESAMPLE BERNOULLI (10)   -- rows
SELECT * FROM images TABLESAMPLE SYSTEM (10)      -- fragments
```

`BERNOULLI` considers each row independently and reads across the whole table.
`SYSTEM` picks whole Lance fragments and takes the rows in them — far less I/O,
because it skips files rather than rows.

The trade is uniformity. A fragment sample is only as representative as your
write order. If rows arrived grouped by source, by date, or by label — which is
usual — then a fragment sample inherits that grouping.

<Note>
Reach for `SYSTEM` when you want a cheap look, and `BERNOULLI` when the number
you compute has to be defensible. Omitting the method gives `BERNOULLI`.
</Note>

## A fixed evaluation set {#fixed}

Without a seed, every execution resamples. That is fine for exploring and wrong
for evaluation — a metric computed on different rows each run is not comparable
with itself.

```sql
SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42)
```

The same seed against the same data gives the same rows, which is what makes a
sample usable as a held-out set. If the underlying data changes, so does the
draw — when you need the set frozen against that too, materialize it:

```sql
CREATE TABLE eval_set AS
SELECT * FROM images TABLESAMPLE (10) REPEATABLE (42)
```

A [materialized view](/management/materialized-view) is the other option when
you want it to track the source on a refresh rather than stay frozen.

## Sampling both sides of a join {#joins}

```sql
SELECT * FROM a TABLESAMPLE (10) JOIN b TABLESAMPLE (10) ON a.id = b.id
```

Each side samples independently, so a row survives only if both of its sides
were drawn. Sampling 10% of each side does **not** give 10% of the join — it
gives roughly 1%. Sample once, after the join, unless you specifically want the
smaller result.

## Without SQL {#training}

<Badge color="green">OSS</Badge> <Badge color="red">Enterprise</Badge>

Everything above needs SQL. The Python SDK's permutation builder samples and
shuffles without it, in the embedded library and on a deployment alike:

```python
from lancedb.permutation import Permutation, permutation_builder

# Every row, in a seeded shuffled order
shuffled = permutation_builder(table).shuffle(seed=42).execute()
permutation = Permutation.from_tables(table, shuffled)

# A reproducible 10% sample: the first of two random splits
splits = permutation_builder(table).split_random(ratios=[0.1, 0.9], seed=42).execute()
sample = Permutation.from_tables(table, splits, split=0)

for batch in sample.iter(batch_size=256):
    ...
```

`Permutation.identity(table)` reads rows in their stored order and does not
shuffle. A `Permutation` is a map-style PyTorch dataset, so either of the above
can be passed to `torch.utils.data.DataLoader`, which reads it in the
permutation's order. A shuffled order is the right tool for epoch-level
shuffling, where you want every row eventually and only the order varies; a
split is for when you want fewer rows. See [Training](/training/data-loading).

## Reference {#reference}

The full clause, its options and what it refuses are in
[Sampling](/sql/dql/sampling).