mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
108 lines
4.5 KiB
Plaintext
108 lines
4.5 KiB
Plaintext
---
|
|
title: "Selective Sampling"
|
|
sidebarTitle: "Selective Sampling"
|
|
description: "Fetch specific rows by row ID, row address or offset instead of drawing a random fraction."
|
|
icon: crosshairs
|
|
keywords: ["sampling", "row id", "row address", "offset", "take", "rowid", "rowaddr", "rowoffset"]
|
|
availability:
|
|
embedded: available
|
|
enterprise: available
|
|
---
|
|
|
|
Selective sampling fetches specific, already-known rows -- by row ID, row
|
|
address or offset -- rather than drawing a random fraction of the table.
|
|
Use it to re-fetch rows a previous query identified, resume from a saved
|
|
position, or pull a batch by index for a data loader. For a random subset
|
|
instead, see [Statistical Sampling](/curation/statistical-sampling).
|
|
|
|
LanceDB exposes three ways to address a row:
|
|
|
|
| Identifier | Stability | Typical source |
|
|
| --- | --- | --- |
|
|
| Row ID (`_rowid`) | Stable across compaction, with [stable row IDs](/basics/guarantees) enabled | A previous query's result, or `function_errors()` |
|
|
| Row address (`_rowaddr`) | Changes on compaction; encodes `(fragment_id << 32) \| row_offset` | A previous query's result, when you need the physical location |
|
|
| Offset | Table-wide position; **not** stable -- a row's offset shifts when an earlier row is deleted | `range(len(table))`, since every valid offset is known in advance |
|
|
|
|
## From Python {#python}
|
|
|
|
The SDK takes a list of row IDs or offsets directly, without going through
|
|
SQL:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
# By row ID -- from a previous search or scan that selected "_rowid"
|
|
rows = tbl.take_row_ids([1001, 1002, 1005]).to_arrow()
|
|
|
|
# By offset -- the full range is always [0, len(tbl))
|
|
rows = tbl.take_offsets([0, 1, 2]).to_arrow()
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
// By row ID -- from a previous search or scan that selected "_rowid"
|
|
const rowsById = await tbl.takeRowIds([1001n, 1002n, 1005n]).toArray();
|
|
|
|
// By offset -- the full range is always [0, tbl.countRows())
|
|
const rowsByOffset = await tbl.takeOffsets([0, 1, 2]).toArray();
|
|
```
|
|
|
|
```rust
|
|
use lancedb::query::ExecutableQuery;
|
|
|
|
// By row ID -- from a previous search or scan that selected "_rowid"
|
|
let rows_by_id = tbl.take_row_ids(vec![1001, 1002, 1005]).execute().await?;
|
|
|
|
// By offset -- the full range is always [0, tbl.count_rows().await?)
|
|
let rows_by_offset = tbl.take_offsets(vec![0, 1, 2]).execute().await?;
|
|
```
|
|
</CodeGroup>
|
|
|
|
Both methods return a query builder; call `.to_arrow()`, `.to_list()` or
|
|
similar to execute it. Repeated *offsets* produce repeated rows, so
|
|
`take_offsets` also supports sampling with replacement -- `take_row_ids`
|
|
does not: a row ID repeated in the input list still returns that row only
|
|
once, since the lookup is a set membership test. Neither method guarantees
|
|
result order matches the order of the input list; for `take_row_ids`, pass
|
|
`with_row_id=True` to add `_rowid` to the output and align the results
|
|
yourself if you need to. `take_offsets` tracks `_rowoffset` internally to
|
|
restore repeated rows, but drops that column from the output again
|
|
afterward unless you explicitly `select` it -- add `_rowoffset` to your
|
|
selection when you need it to align a `take_offsets` result yourself.
|
|
|
|
Offsets are mostly useful for sampling: the valid range is always
|
|
`[0, len(table))`, so you can draw a random sample of offsets without
|
|
reading the table first. Row IDs require you to already have them, from an
|
|
earlier query's `_rowid` column -- but unlike offsets, a row ID keeps
|
|
addressing the same logical row even after compaction, as long as the
|
|
table has stable row IDs enabled.
|
|
|
|
## From SQL {#sql}
|
|
|
|
The same three identifiers are available as
|
|
[pseudo-columns](/sql/data-types#pseudo-columns) -- select or filter on
|
|
them like any other column:
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
-- By row ID
|
|
SELECT * FROM documents WHERE _rowid IN (1001, 1002, 1005);
|
|
|
|
-- By row address
|
|
SELECT * FROM documents WHERE _rowaddr IN (4294967296, 4294967297);
|
|
|
|
-- Including the identifier in the result, to save for a later take
|
|
SELECT _rowid, * FROM documents WHERE score > 0.9;
|
|
```
|
|
</CodeGroup>
|
|
|
|
There is no dedicated Python method for taking by row address -- use
|
|
`.where("_rowaddr IN (...)")` on the SDK query builder, or the SQL form
|
|
above, since the expression language is the same either way.
|
|
|
|
## Related guides {#reference}
|
|
|
|
[Query plans](/basics/sql#query-plans) show whether a row-ID or offset
|
|
filter reads the whole table or seeks directly to the requested rows. See
|
|
[Statistical Sampling](/curation/statistical-sampling) for a random
|
|
fraction instead of specific rows, and [Permutation](/training/permutation)
|
|
for shuffled, batched reads through the Python SDK.
|