mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-03 09:16:52 +00:00
docs: complete SQL reference, enrich Functions drafts, rename OSS to Embedded (#4732)
## Summary - Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause, and `dedupe()`/`duplicate_pairs()` (verified against current sophon source; these were recently renamed from `vector_dedup`/`vector_duplicate_pairs`). Fix the blob-column default-projection claim and add the previously undocumented `blob_descriptor()`. - Enrich the gated Functions draft (still `.mintignore`'d, pre-release): chained Functions over blob outputs, GPU-produced blobs, Secrets, multi-input/output binding, `code=[module]` packaging. - Upgrade the materialized-view and computed-column auto-refresh drafts with precise, verified catch-up semantics, and add the missing `ALTER MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the automatic-refresh scheduler is still dev/staging-only). - Thread one running example (video frames → GPU extract/caption → dedup → auto-refreshing materialized view) through Functions, materialized views, and the LeRobotDataset integration guide instead of a separate tutorial. - Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout (52 files); activate the existing availability-comparison mechanism on `basics/offerings.mdx` for a full feature-by-feature comparison table.
This commit is contained in:
Jack Ye
@@ -4,10 +4,11 @@ sidebarTitle: "Aggregation"
|
||||
description: "GROUP BY, HAVING, and the aggregate function set."
|
||||
icon: layer-group
|
||||
keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximate"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Use SQL aggregates to summarize records, either across a whole table or
|
||||
within groups. Common examples include counts by category, totals and averages.
|
||||
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: "Changes since a version"
|
||||
sidebarTitle: "Changes"
|
||||
description: "CHANGES: read the rows appended to a table since an earlier version."
|
||||
icon: clock-rotate-left
|
||||
keywords: ["changes", "append", "incremental", "cdc", "version"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
`CHANGES` reads the rows appended to a table since an earlier
|
||||
[version](/management/table#versioning), without rescanning rows that were
|
||||
already there:
|
||||
|
||||
```sql
|
||||
SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 42)
|
||||
```
|
||||
|
||||
This returns the rows appended after version 42, up to the table's current
|
||||
version at the time the query runs.
|
||||
|
||||
## Strictly append-only {#strict}
|
||||
|
||||
The clause is named for what it requires, not just what it does: every
|
||||
change to the table between the starting version and the current one must
|
||||
be an append. If any of them inserted with changes other than a pure append
|
||||
-- an update, a delete, a merge -- the query is refused rather than silently
|
||||
returning an incomplete or wrong result. Re-run it with a later starting
|
||||
version that is itself after the non-append write, or read the table in full
|
||||
instead of incrementally.
|
||||
|
||||
`INFORMATION => APPEND_ONLY` is the only supported value: it names what kind
|
||||
of history the clause is willing to read, not a choice among several. `AT
|
||||
(VERSION => ...)` takes a positive integer; there is no `AT (TIMESTAMP =>
|
||||
...)` form for this clause, unlike other version qualifiers.
|
||||
|
||||
## Where the clause can go {#placement}
|
||||
|
||||
It attaches to a table in `FROM`, including inside a subquery, a CTE, a join,
|
||||
a set operation, or `EXPLAIN`:
|
||||
|
||||
```sql
|
||||
WITH recent AS (
|
||||
SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 42)
|
||||
)
|
||||
SELECT count(*) FROM recent
|
||||
|
||||
SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 10) a
|
||||
JOIN events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 20) b
|
||||
ON a.id = b.id
|
||||
```
|
||||
|
||||
Each occurrence of a table keeps its own starting version, even when the same
|
||||
table is referenced more than once in one statement.
|
||||
|
||||
## Unsupported forms {#refused}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `CHANGES (INFORMATION => DEFAULT)` | only `APPEND_ONLY` is supported |
|
||||
| `AT (TIMESTAMP => ...)` | only `AT (VERSION => ...)` is supported |
|
||||
| `AT (VERSION => 0)` | the starting version must be positive |
|
||||
| `... END (VERSION => ...)` | not supported; the current version is always the end |
|
||||
| `CHANGES ... TABLESAMPLE (...)` | cannot be combined on the same table reference |
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Table" icon="table" href="/sql/ddl/table">
|
||||
CREATE, ALTER, CLONE and versioning basics.
|
||||
</Card>
|
||||
<Card title="Select" icon="magnifying-glass" href="/sql/dql/select">
|
||||
The rest of the query grammar this clause attaches to.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: "DEDUPE"
|
||||
sidebarTitle: "DEDUPE"
|
||||
description: "The dedupe() table function: one representative row per near-duplicate group, scanned over the index."
|
||||
icon: layer-group
|
||||
keywords: ["dedupe", "vector_dedup", "sql", "deduplication", "near-duplicate", "table function"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
`dedupe` scans a vector column's [index](/indexing/vector-index) partitions
|
||||
for rows closer than a distance threshold, resolves the overlapping pairs it
|
||||
finds into duplicate groups, and returns one representative row per group --
|
||||
without a full pairwise scan or a shuffle. See
|
||||
[`duplicate_pairs`](/sql/dql/duplicate-pairs) to inspect the pairs themselves
|
||||
before committing to this resolution.
|
||||
|
||||
```sql
|
||||
SELECT * FROM dedupe(<table>, <vector_column>,
|
||||
distance_threshold => <float>
|
||||
[, resolution => 'connected_components' | 'direct_representative']
|
||||
[, selection => 'all' | 'first' | 'random', count => <int>, seed => <int>]
|
||||
)
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02)
|
||||
```
|
||||
|
||||
<Note>
|
||||
Earlier releases spelled this `vector_dedup`; that name is rejected on
|
||||
current deployments.
|
||||
</Note>
|
||||
|
||||
## Input {#input}
|
||||
|
||||
| Argument | Type | Required | Meaning |
|
||||
| --- | --- | --- | --- |
|
||||
| table name | string literal | yes | the table to scan |
|
||||
| vector column | string literal | yes | the indexed vector column to compare |
|
||||
| `distance_threshold` | float | yes | maximum distance for a pair to count as close |
|
||||
| `resolution` | `'connected_components'` (default) \| `'direct_representative'` | no | how overlapping pairs group together -- see [Resolution](#resolution) |
|
||||
| `selection` | `'all'` (default) \| `'first'` \| `'random'` | no | which index partitions to scan |
|
||||
| `count` | int | no | number of partitions to sample when `selection => 'random'` |
|
||||
| `seed` | int | no | random seed when `selection => 'random'` |
|
||||
|
||||
`id_column` is refused on `dedupe`: it has no separate pair-identifier output
|
||||
to project onto, only [`duplicate_pairs`](/sql/dql/duplicate-pairs#input)
|
||||
accepts it.
|
||||
|
||||
## Output {#output}
|
||||
|
||||
One row per duplicate group, **in the table's own schema** -- every column of
|
||||
`<table>`, including a BlobV2 column, comes through unchanged on the
|
||||
representative row Lance selected for that group. Unlike
|
||||
[`duplicate_pairs`](/sql/dql/duplicate-pairs#output), there is no fixed
|
||||
`dedupe`-specific schema to document: the output schema is the source
|
||||
table's schema.
|
||||
|
||||
Because the result is already shaped like the table, it composes directly
|
||||
into a new table or a materialized view:
|
||||
|
||||
```sql
|
||||
CREATE MATERIALIZED VIEW frames_deduped AS
|
||||
SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02)
|
||||
```
|
||||
|
||||
## Resolution {#resolution}
|
||||
|
||||
`resolution` chooses how overlapping pairs group together:
|
||||
|
||||
| `resolution` | |
|
||||
| --- | --- |
|
||||
| `connected_components` (default) | groups transitively: if a~b and b~c are both close, all three resolve together |
|
||||
| `direct_representative` | only pairs within the threshold of each other resolve together |
|
||||
|
||||
## Sampling partitions {#sampling}
|
||||
|
||||
`selection` scopes the scan to a sample of partitions instead of all of them,
|
||||
the same `all` (default) / `first` / `random` choice as
|
||||
[`ivf_partition`](/sql/scalar-functions#ivf-partition). Use it to estimate a
|
||||
duplicate rate on a large table before scanning it in full:
|
||||
|
||||
```sql
|
||||
SELECT * FROM dedupe('frames', 'vec',
|
||||
distance_threshold => 0.02, selection => 'random', count => 100, seed => 42)
|
||||
```
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="DUPLICATE_PAIRS" icon="clone" href="/sql/dql/duplicate-pairs">
|
||||
Inspect the close pairs themselves instead of the resolved groups.
|
||||
</Card>
|
||||
<Card title="Deduplicate near-duplicates" icon="copy" href="/curation/dedupe">
|
||||
When to reach for this instead of exact-key deduplication.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: "DUPLICATE_PAIRS"
|
||||
sidebarTitle: "DUPLICATE_PAIRS"
|
||||
description: "The duplicate_pairs() table function: every close pair in a vector column, scanned over the index."
|
||||
icon: clone
|
||||
keywords: ["duplicate_pairs", "vector_duplicate_pairs", "sql", "deduplication", "near-duplicate", "table function"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
`duplicate_pairs` scans a vector column's [index](/indexing/vector-index)
|
||||
partitions for every pair of rows closer than a distance threshold, without a
|
||||
full pairwise scan or a shuffle. It returns one row per close pair, not one
|
||||
row per source row -- use it to inspect what would be collapsed before
|
||||
committing to [`dedupe`](/sql/dql/dedupe), which resolves these same pairs
|
||||
into groups.
|
||||
|
||||
```sql
|
||||
SELECT * FROM duplicate_pairs(<table>, <vector_column>,
|
||||
distance_threshold => <float>
|
||||
[, id_column => <column>]
|
||||
[, selection => 'all' | 'first' | 'random', count => <int>, seed => <int>]
|
||||
)
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02)
|
||||
```
|
||||
|
||||
<Note>
|
||||
Earlier releases spelled this `vector_duplicate_pairs`; that name is rejected
|
||||
on current deployments.
|
||||
</Note>
|
||||
|
||||
## Input {#input}
|
||||
|
||||
| Argument | Type | Required | Meaning |
|
||||
| --- | --- | --- | --- |
|
||||
| table name | string literal | yes | the table to scan |
|
||||
| vector column | string literal | yes | the indexed vector column to compare |
|
||||
| `distance_threshold` | float | yes | maximum distance for a pair to count as close |
|
||||
| `id_column` | column name | no | project this column's value instead of the internal row ID -- see [Application-level identifiers](#id-column) |
|
||||
| `selection` | `'all'` (default) \| `'first'` \| `'random'` | no | which index partitions to scan |
|
||||
| `count` | int | no | number of partitions to sample when `selection => 'random'` |
|
||||
| `seed` | int | no | random seed when `selection => 'random'` |
|
||||
|
||||
`resolution` is refused on `duplicate_pairs`: a pair has no notion of a group,
|
||||
only [`dedupe`](/sql/dql/dedupe#input) accepts it.
|
||||
|
||||
## Output {#output}
|
||||
|
||||
One row per close pair found, in a fixed schema independent of the source
|
||||
table:
|
||||
|
||||
| Column | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `row_id_a` | uint64 | internal row ID of the first row in the pair, the same identifier as the [`_rowid`](/sql/data-types#pseudo-columns) pseudo-column |
|
||||
| `row_id_b` | uint64 | internal row ID of the second row in the pair |
|
||||
| `distance` | float32 | distance between the pair's vectors, at or below `distance_threshold` |
|
||||
| `segment_id` | int64 | which index segment the pair was found in |
|
||||
| `partition_id` | int64 | which index partition the pair was found in |
|
||||
| `dataset_version` | int64 | the dataset version the scan ran against |
|
||||
|
||||
### Application-level identifiers {#id-column}
|
||||
|
||||
A row ID is internal and not stable across compaction. Pass `id_column`, a
|
||||
top-level scalar column such as a primary key, to project its value instead:
|
||||
`row_id_a`/`row_id_b` are replaced by `id_a`/`id_b`, typed like `id_column`.
|
||||
|
||||
```sql
|
||||
SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02, id_column => 'frame_id')
|
||||
```
|
||||
|
||||
## Sampling partitions {#sampling}
|
||||
|
||||
`selection` scopes the scan to a sample of partitions instead of all of them,
|
||||
the same `all` (default) / `first` / `random` choice as
|
||||
[`ivf_partition`](/sql/scalar-functions#ivf-partition). Use it to estimate a
|
||||
distance threshold or duplicate rate on a large table before scanning it in
|
||||
full:
|
||||
|
||||
```sql
|
||||
SELECT * FROM duplicate_pairs('frames', 'vec',
|
||||
distance_threshold => 0.02, selection => 'random', count => 100, seed => 42)
|
||||
```
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="DEDUPE" icon="layer-group" href="/sql/dql/dedupe">
|
||||
Resolve these same pairs into one representative row per group.
|
||||
</Card>
|
||||
<Card title="Deduplicate near-duplicates" icon="copy" href="/curation/dedupe">
|
||||
When to reach for this instead of exact-key deduplication.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Explain"
|
||||
description: "EXPLAIN, EXPLAIN ANALYZE and EXPLAIN VERBOSE: reading and measuring a plan from SQL."
|
||||
icon: map
|
||||
keywords: ["explain", "explain analyze", "explain verbose", "query plan"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
```sql
|
||||
EXPLAIN SELECT * FROM documents WHERE identifier < 1000
|
||||
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Full-Text Search"
|
||||
description: "The fts() table function: match, fuzzy, phrase, boolean, boost and multi-match queries from SQL."
|
||||
icon: magnifying-glass
|
||||
keywords: ["fts", "sql", "full text search", "fuzzy", "phrase", "boolean", "boost", "multi-match"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Full-text search is a **table function**, `fts(table, query)`, rather than a
|
||||
predicate: it produces a relevance score alongside the matching rows, and a
|
||||
`WHERE` clause has nowhere to put one. The two string-literal arguments are
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Joins"
|
||||
description: "Combining rows from more than one table, including across namespaces."
|
||||
icon: link
|
||||
keywords: ["join", "inner", "left", "outer", "cross", "namespace"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Use SQL joins to combine rows from two or more tables.
|
||||
|
||||
## The join types {#types}
|
||||
@@ -70,3 +71,29 @@ Avoid selecting blob or vector columns you do not need on either side of a join.
|
||||
The join multiplies rows before projection is applied in some plans, so a wide
|
||||
column can be materialized many more times than the result has rows.
|
||||
</Warning>
|
||||
|
||||
## Late materialization for wide columns {#late-materialization}
|
||||
|
||||
The planner can avoid the multiplication above: when a join narrows down to
|
||||
a key column (such as `_rowid` or another narrow, selective column) before a
|
||||
wide one is needed, it can defer reading a BlobV2 or vector column until
|
||||
after the join has already settled which rows survive, instead of reading it
|
||||
for every candidate row first. In a plan where this applies, the wide column
|
||||
is read once per surviving row, where the worker runs, and never pulled into
|
||||
the query node to be joined directly.
|
||||
|
||||
```sql
|
||||
SELECT d.title, b.caption
|
||||
FROM documents AS d
|
||||
JOIN (SELECT _rowid, caption FROM captions WHERE source = 'model-v2') AS b
|
||||
ON b._rowid = d._rowid
|
||||
```
|
||||
|
||||
This is query planning, not a syntax you opt into: a join written this way
|
||||
is a **candidate** for the planner to defer the wide read, not a guarantee.
|
||||
Confirm it for a specific query with [`EXPLAIN`](/sql/dql/explain) --
|
||||
`RemoteTake` in the plan means the deferral happened; its absence means the
|
||||
wide column was read eagerly, even if the SQL above matches exactly. Neither
|
||||
a passing query nor an `EXPLAIN` that omits `RemoteTake` tells you how much
|
||||
I/O a plan did or did not save -- read the plan's own row and byte counts for
|
||||
that.
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Sampling"
|
||||
description: "TABLESAMPLE: draw a random subset of a table, by rows or by fragments."
|
||||
icon: dice
|
||||
keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Use `TABLESAMPLE` to explore a subset of a table or estimate aggregate
|
||||
statistics. The sampling method determines how much data is read and whether
|
||||
the table's physical layout influences the sample.
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "SELECT"
|
||||
description: "Projection, filtering, ordering, and composing queries out of other queries."
|
||||
icon: table-columns
|
||||
keywords: ["select", "where", "order by", "limit", "cte", "union", "distinct"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
```sql
|
||||
[WITH cte AS (...)]
|
||||
SELECT [ALL | DISTINCT] select_expr [, ...]
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Subqueries"
|
||||
description: "Queries inside queries: scalar, IN, EXISTS, correlated, and derived tables."
|
||||
icon: code-branch
|
||||
keywords: ["subquery", "exists", "in", "correlated", "derived table", "lateral"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A subquery is allowed anywhere an expression or a relation is.
|
||||
|
||||
## As a relation {#derived}
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Window Functions"
|
||||
description: "Compute over a set of rows related to the current one, without collapsing them."
|
||||
icon: chart-simple
|
||||
keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A window function computes across a set of rows related to the current one and
|
||||
returns a value **per row**. That is the difference from
|
||||
[aggregation](/sql/dql/aggregation), which collapses the rows it covers.
|
||||
|
||||
Reference in New Issue
Block a user