Files
lancedb/docs/web/curation/selective-sampling.mdx
T
Jack YeandClaude Sonnet 5 de95ed3f6d docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:

**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.

**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.

**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.

**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.

**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.

Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-03 16:17:24 -08:00

108 lines
4.5 KiB
Plaintext

---
title: "Selective Sampling"
sidebarTitle: "Selective Sampling"
description: "Fetch specific rows by row ID, row address or offset instead of drawing a random fraction."
icon: crosshairs
keywords: ["sampling", "row id", "row address", "offset", "take", "rowid", "rowaddr", "rowoffset"]
availability:
embedded: available
enterprise: available
---
Selective sampling fetches specific, already-known rows -- by row ID, row
address or offset -- rather than drawing a random fraction of the table.
Use it to re-fetch rows a previous query identified, resume from a saved
position, or pull a batch by index for a data loader. For a random subset
instead, see [Statistical Sampling](/curation/statistical-sampling).
LanceDB exposes three ways to address a row:
| Identifier | Stability | Typical source |
| --- | --- | --- |
| Row ID (`_rowid`) | Stable across compaction, with [stable row IDs](/basics/guarantees) enabled | A previous query's result, or `function_errors()` |
| Row address (`_rowaddr`) | Changes on compaction; encodes `(fragment_id << 32) \| row_offset` | A previous query's result, when you need the physical location |
| Offset | Table-wide position; **not** stable -- a row's offset shifts when an earlier row is deleted | `range(len(table))`, since every valid offset is known in advance |
## From Python {#python}
The SDK takes a list of row IDs or offsets directly, without going through
SQL:
<CodeGroup>
```python Python icon="python"
# By row ID -- from a previous search or scan that selected "_rowid"
rows = tbl.take_row_ids([1001, 1002, 1005]).to_arrow()
# By offset -- the full range is always [0, len(tbl))
rows = tbl.take_offsets([0, 1, 2]).to_arrow()
```
```typescript TypeScript icon="square-js"
// By row ID -- from a previous search or scan that selected "_rowid"
const rowsById = await tbl.takeRowIds([1001n, 1002n, 1005n]).toArray();
// By offset -- the full range is always [0, tbl.countRows())
const rowsByOffset = await tbl.takeOffsets([0, 1, 2]).toArray();
```
```rust
use lancedb::query::ExecutableQuery;
// By row ID -- from a previous search or scan that selected "_rowid"
let rows_by_id = tbl.take_row_ids(vec![1001, 1002, 1005]).execute().await?;
// By offset -- the full range is always [0, tbl.count_rows().await?)
let rows_by_offset = tbl.take_offsets(vec![0, 1, 2]).execute().await?;
```
</CodeGroup>
Both methods return a query builder; call `.to_arrow()`, `.to_list()` or
similar to execute it. Repeated *offsets* produce repeated rows, so
`take_offsets` also supports sampling with replacement -- `take_row_ids`
does not: a row ID repeated in the input list still returns that row only
once, since the lookup is a set membership test. Neither method guarantees
result order matches the order of the input list; for `take_row_ids`, pass
`with_row_id=True` to add `_rowid` to the output and align the results
yourself if you need to. `take_offsets` tracks `_rowoffset` internally to
restore repeated rows, but drops that column from the output again
afterward unless you explicitly `select` it -- add `_rowoffset` to your
selection when you need it to align a `take_offsets` result yourself.
Offsets are mostly useful for sampling: the valid range is always
`[0, len(table))`, so you can draw a random sample of offsets without
reading the table first. Row IDs require you to already have them, from an
earlier query's `_rowid` column -- but unlike offsets, a row ID keeps
addressing the same logical row even after compaction, as long as the
table has stable row IDs enabled.
## From SQL {#sql}
The same three identifiers are available as
[pseudo-columns](/sql/data-types#pseudo-columns) -- select or filter on
them like any other column:
<CodeGroup>
```sql SQL icon="database"
-- By row ID
SELECT * FROM documents WHERE _rowid IN (1001, 1002, 1005);
-- By row address
SELECT * FROM documents WHERE _rowaddr IN (4294967296, 4294967297);
-- Including the identifier in the result, to save for a later take
SELECT _rowid, * FROM documents WHERE score > 0.9;
```
</CodeGroup>
There is no dedicated Python method for taking by row address -- use
`.where("_rowaddr IN (...)")` on the SDK query builder, or the SQL form
above, since the expression language is the same either way.
## Related guides {#reference}
[Query plans](/basics/sql#query-plans) show whether a row-ID or offset
filter reads the whole table or seeks directly to the requested rows. See
[Statistical Sampling](/curation/statistical-sampling) for a random
fraction instead of specific rows, and [Permutation](/training/permutation)
for shuffled, batched reads through the Python SDK.