mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-01 00:06:52 +00:00
docs: complete SQL reference, enrich Functions drafts, rename OSS to Embedded (#4732)
## Summary - Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause, and `dedupe()`/`duplicate_pairs()` (verified against current sophon source; these were recently renamed from `vector_dedup`/`vector_duplicate_pairs`). Fix the blob-column default-projection claim and add the previously undocumented `blob_descriptor()`. - Enrich the gated Functions draft (still `.mintignore`'d, pre-release): chained Functions over blob outputs, GPU-produced blobs, Secrets, multi-input/output binding, `code=[module]` packaging. - Upgrade the materialized-view and computed-column auto-refresh drafts with precise, verified catch-up semantics, and add the missing `ALTER MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the automatic-refresh scheduler is still dev/staging-only). - Thread one running example (video frames → GPU extract/caption → dedup → auto-refreshing materialized view) through Functions, materialized views, and the LeRobotDataset integration guide instead of a separate tutorial. - Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout (52 files); activate the existing availability-comparison mechanism on `basics/offerings.mdx` for a full feature-by-feature comparison table.
This commit is contained in:
Jack Ye
@@ -4,10 +4,11 @@ sidebarTitle: "Data Types"
|
||||
description: "How Lance columns appear to SQL, including the types SQL has no name for."
|
||||
icon: shapes
|
||||
keywords: ["sql types", "arrow", "cast", "pseudo-columns", "vector", "json"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
There is no SQL type layer over Lance storage. The table provider hands
|
||||
DataFusion the dataset's Arrow schema directly, so a column is the same object
|
||||
to the query engine as it is to the format.
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
---
|
||||
title: "Database"
|
||||
sidebarTitle: "Database"
|
||||
description: "ATTACH DATABASE: bind a database to a storage location."
|
||||
icon: database
|
||||
keywords: ["attach database", "database", "sql", "ddl"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
A database is the catalog's top-level container: it holds
|
||||
[namespaces](/sql/ddl/namespace), which hold tables, views and other objects.
|
||||
`ATTACH DATABASE` binds a name to a storage location so its objects become
|
||||
reachable.
|
||||
|
||||
## ATTACH DATABASE {#attach}
|
||||
|
||||
```sql
|
||||
ATTACH DATABASE analytics WITH (location = 'analytics')
|
||||
ATTACH DATABASE IF NOT EXISTS analytics WITH (location = 'analytics')
|
||||
```
|
||||
|
||||
`location` is the only supported option, and the only way to say where a
|
||||
database lives: a relative subdirectory, never an absolute path or a URI.
|
||||
|
||||
<Warning>
|
||||
An absolute path (`/tmp/db`), an `s3://` or `file://` URI, a Windows path
|
||||
(`C:/db`), a `..` segment, an empty string and a doubled slash (`sub//db`)
|
||||
are all refused.
|
||||
</Warning>
|
||||
|
||||
`IF NOT EXISTS` makes the statement a no-op rather than an error when the
|
||||
database is already attached.
|
||||
|
||||
<Note>
|
||||
A quoted name reaches characters a bare identifier cannot, such as
|
||||
`` ATTACH DATABASE "team/search" WITH (location = 'team-search') ``.
|
||||
</Note>
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Namespace" icon="folder-tree" href="/sql/ddl/namespace">
|
||||
CREATE SCHEMA inside an attached database.
|
||||
</Card>
|
||||
<Card title="Table" icon="table" href="/sql/ddl/table">
|
||||
Creating tables once a database and namespace exist.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -17,6 +17,15 @@ AS 'def f(x): return x * 2'
|
||||
WITH (pip = 'numpy==1.26.4', python_version = '3.11')
|
||||
```
|
||||
|
||||
A function takes more than one parameter; each one binds a separate column
|
||||
or literal where the function is called:
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION ratio(numerator float64, denominator float64) RETURNS float64
|
||||
LANGUAGE python
|
||||
AS 'def ratio(numerator, denominator): return numerator / denominator'
|
||||
```
|
||||
|
||||
Parameter and result types are Arrow type names, not SQL ones: `int64`,
|
||||
`float64`, `utf8`, `bool`, `date32`, and lists such as `list<float32>` or
|
||||
`fixed_size_list<float32, 768>`. `BIGINT` is refused. A parameter accepts nulls
|
||||
@@ -51,6 +60,20 @@ version is unaffected.
|
||||
the function is baked into an image before the version becomes usable.
|
||||
</Note>
|
||||
|
||||
## Call it {#call}
|
||||
|
||||
Once registered, a function is callable by name in any query, the same as a
|
||||
built-in scalar function:
|
||||
|
||||
```sql
|
||||
SELECT id, f(x) AS doubled FROM my_table
|
||||
```
|
||||
|
||||
This is also the fastest way to check a new registration before binding it to
|
||||
a column: call it over a few rows, or over literal arguments, before
|
||||
committing to [a computed column](/sql/ddl/table#computed-column) that fills
|
||||
from it.
|
||||
|
||||
## SHOW FUNCTIONS {#show}
|
||||
|
||||
```sql
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
title: "Grant"
|
||||
sidebarTitle: "Grant"
|
||||
description: "GRANT and REVOKE: privileges on Secrets and Functions."
|
||||
icon: user-lock
|
||||
keywords: ["grant", "revoke", "privileges", "authorization", "sql", "ownership"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
```sql
|
||||
GRANT <privilege>[, ...] ON { SECRET | FUNCTION } <name> TO <grantee>[, ...]
|
||||
GRANT <privilege>[, ...] ON SCHEMA <namespace> TO <grantee>[, ...]
|
||||
REVOKE <privilege>[, ...] ON ... FROM <grantee>[, ...]
|
||||
```
|
||||
|
||||
```sql
|
||||
GRANT USAGE, UPDATE ON SECRET api_key TO ROLE readers, writers
|
||||
GRANT CREATE FUNCTION ON SCHEMA analytics TO ROLE data_science
|
||||
REVOKE USAGE ON FUNCTION embed FROM ROLE readers
|
||||
```
|
||||
|
||||
## Privileges {#privileges}
|
||||
|
||||
| Privilege | Allows |
|
||||
| --- | --- |
|
||||
| `OWNERSHIP` | everything on the object |
|
||||
| `USAGE` | using it: binding a Secret's value, calling a Function |
|
||||
| `UPDATE` | changing it |
|
||||
| `CREATE SECRET` | creating a Secret inside a `SCHEMA` |
|
||||
| `CREATE FUNCTION` | creating a Function inside a `SCHEMA` |
|
||||
|
||||
<Warning>
|
||||
`OWNERSHIP` cannot be revoked, only granted to someone else, which transfers
|
||||
it: an object always has exactly one owner. `GRANT OWNERSHIP` is refused with
|
||||
more than one grantee for the same reason.
|
||||
</Warning>
|
||||
|
||||
These are a subset of the full privilege model; see
|
||||
[Authorization](/governance/authorization#privileges) for privileges on
|
||||
tables, views and namespaces, which this statement does not grant.
|
||||
|
||||
## Grantees {#grantees}
|
||||
|
||||
```sql
|
||||
<grantee> ::= [ ROLE ] <role> | [ { USER | GROUP } ] { <name> | ID '<id>' }
|
||||
```
|
||||
|
||||
```sql
|
||||
GRANT USAGE ON SECRET api_key TO ROLE readers, USER alice, GROUP ID 'g1'
|
||||
```
|
||||
|
||||
A grantee is a role, or a user or group named by its identity-directory name
|
||||
or its directory ID. A grantee with no leading `ROLE`/`USER`/`GROUP` keyword
|
||||
takes the kind of the one before it, so `TO ROLE a, b` names two roles --
|
||||
but the first grantee in the list always needs its keyword. An unquoted name
|
||||
is folded to lowercase before it is looked up; a quoted name is looked up
|
||||
exactly as written.
|
||||
|
||||
## Naming the target {#targets}
|
||||
|
||||
A `SECRET` or `FUNCTION` target can be qualified with its namespace and
|
||||
database, the same as anywhere else: `SECRET analytics.api_key`,
|
||||
`SECRET db1.analytics.api_key`. A `SCHEMA` target is the namespace itself.
|
||||
There is no `TABLE` target: table privileges are granted through
|
||||
[Authorization](/governance/authorization), not this statement.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Authorization" icon="user-shield" href="/governance/authorization">
|
||||
The full privilege model, and how creation privileges resolve upward.
|
||||
</Card>
|
||||
<Card title="Namespace" icon="folder-tree" href="/sql/ddl/namespace">
|
||||
Creating the schemas these privileges apply to.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Index"
|
||||
description: "CREATE INDEX and DROP INDEX: one column, with the algorithm chosen by USING."
|
||||
icon: bolt
|
||||
keywords: ["create index", "drop index", "sql"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
## CREATE INDEX {#create}
|
||||
|
||||
```sql
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Job"
|
||||
description: "SHOW JOBS, SHOW ERRORS, and controlling a running job with PAUSE, RESUME and CANCEL."
|
||||
icon: hourglass-half
|
||||
keywords: ["show jobs", "show errors", "pause job", "resume job", "cancel job", "sql"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A [job](/basics/jobs) is not a catalog object (it has no schema and is not
|
||||
namespaced), but it has its own SQL statements, grouped here the same way the
|
||||
catalog objects are.
|
||||
@@ -32,7 +33,7 @@ SHOW ERRORS ON documents
|
||||
SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100
|
||||
```
|
||||
|
||||
Why rows failed. Reach for this when a [refresh](/sql/dml/refresh) reports
|
||||
Why rows failed. Reach for this when a [refresh](/sql/dml/refresh-column) reports
|
||||
fewer filled rows than you expected.
|
||||
|
||||
**Errors and limits**
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Materialized View"
|
||||
description: "CREATE MATERIALIZED VIEW and SHOW MATERIALIZED VIEWS; refreshing is a REFRESH statement."
|
||||
icon: table
|
||||
keywords: ["materialized view", "create materialized view", "show materialized views", "sql"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A materialized view stores a query's result as rows, kept current by a refresh.
|
||||
It is the counterpart to a [view](/sql/ddl/view), which stores only the query.
|
||||
|
||||
@@ -28,6 +29,29 @@ Once created, a materialized view can be queried like a table. Use the
|
||||
statements below to list, refresh and drop it. For SDK methods, see
|
||||
[Materialized views](/management/materialized-view#create).
|
||||
|
||||
{/*
|
||||
Draft for automatic materialized-view refresh. Enable this section, and
|
||||
"ALTER MATERIALIZED VIEW" below, when the server feature is released.
|
||||
|
||||
Set scheduling options at creation, continuing the join and chained
|
||||
Functions from [Functions](/management/function#chain):
|
||||
|
||||
```sql
|
||||
CREATE MATERIALIZED VIEW frames_captioned AS
|
||||
SELECT f.frame_id, extract_frame(v.video, f.ts) AS jpeg,
|
||||
caption(extract_frame(v.video, f.ts)) AS caption
|
||||
FROM videos v JOIN frames f ON v.id = f.video_id
|
||||
WHERE f.selected = true
|
||||
WITH (auto_refresh_enabled = false, auto_refresh_interval_seconds = 5)
|
||||
```
|
||||
|
||||
`auto_refresh_enabled` and `auto_refresh_interval_seconds` work the same way
|
||||
here as for a [computed column](/sql/ddl/table#automatic-refresh-options):
|
||||
the interval is a minimum pacing after the last successful refresh, and
|
||||
automatic execution needs the same server-side switch enabled. Omitting
|
||||
`WITH` leaves automatic refresh at the server's default.
|
||||
*/}
|
||||
|
||||
## SHOW MATERIALIZED VIEWS {#show}
|
||||
|
||||
```sql
|
||||
@@ -45,9 +69,37 @@ on a `db://` connection.
|
||||
REFRESH MATERIALIZED VIEW chunks
|
||||
```
|
||||
|
||||
See [REFRESH](/sql/dml/refresh#refresh-materialized-view) for the full syntax,
|
||||
See [REFRESH MATERIALIZED VIEW](/sql/dml/refresh-materialized-view) for the full syntax,
|
||||
and compatibility notes for older server versions.
|
||||
|
||||
{/*
|
||||
Draft for automatic materialized-view refresh. Enable this section when the
|
||||
server feature is released; see the note on "CREATE MATERIALIZED VIEW" above.
|
||||
|
||||
## ALTER MATERIALIZED VIEW {#alter}
|
||||
|
||||
```sql
|
||||
ALTER MATERIALIZED VIEW frames_captioned SET auto_refresh_enabled = true;
|
||||
ALTER MATERIALIZED VIEW frames_captioned SET auto_refresh_interval_seconds = 300;
|
||||
ALTER MATERIALIZED VIEW frames_captioned SET auto_refresh_enabled = false;
|
||||
```
|
||||
|
||||
The whole view toggles together; there is no per-output granularity the way
|
||||
a multi-output Function column has one setting per output. SET takes effect
|
||||
immediately and does not itself run a refresh:
|
||||
|
||||
- **Disabling** prevents new automatic refreshes. An inflight one may still
|
||||
finish, and `REFRESH MATERIALIZED VIEW` keeps working manually.
|
||||
- **Re-enabling catches up the entire backlog in one pass**, the same as a
|
||||
computed column's [automatic refresh](/sql/ddl/table#automatic-refresh-options):
|
||||
the next scheduler tick picks up every source change accumulated since the
|
||||
view was disabled, immediately, with no further write needed to trigger it.
|
||||
|
||||
A materialized view's *initial* population, at `CREATE` or when the server
|
||||
runs it, happens regardless of `auto_refresh_enabled`; only refreshes after
|
||||
that are gated by the setting.
|
||||
*/}
|
||||
|
||||
## Dropping one {#drop}
|
||||
|
||||
A materialized view **is** a table, so it is dropped with
|
||||
@@ -55,9 +107,8 @@ A materialized view **is** a table, so it is dropped with
|
||||
stored rows, unlike dropping a [view](/sql/ddl/view#drop).
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `mat_view_refresh` for a refresh, or `sql` on a
|
||||
deployment that rebuilds views through SQL. Dropping one runs the same job as
|
||||
dropping any table, `drop_table` or `drop_object`.
|
||||
**Enterprise job type for a refresh:** `sql`. Dropping, by contrast, runs the
|
||||
same job as dropping any table, `drop_table` or `drop_object`.
|
||||
</Note>
|
||||
|
||||
## What to read next {#next}
|
||||
@@ -66,7 +117,7 @@ dropping any table, `drop_table` or `drop_object`.
|
||||
<Card title="Materialized View" icon="table" href="/management/materialized-view">
|
||||
Creating one from the SDK.
|
||||
</Card>
|
||||
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/materialized-views">
|
||||
How a refresh works, locally and in LanceDB Enterprise.
|
||||
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/refresh-materialized-view">
|
||||
How a refresh works, in LanceDB Enterprise.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Namespace"
|
||||
description: "CREATE SCHEMA and DROP SCHEMA: the SQL spelling of namespace DDL."
|
||||
icon: folder-tree
|
||||
keywords: ["namespace", "schema", "create schema", "drop schema", "sql"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A [namespace](/management/namespace) is addressed in SQL as a `SCHEMA`, the
|
||||
standard's name for the same concept.
|
||||
|
||||
@@ -46,6 +47,9 @@ default) states that requirement rather than changing it.
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Database" icon="database" href="/sql/ddl/database">
|
||||
Attaching the database a namespace lives in.
|
||||
</Card>
|
||||
<Card title="Namespace" icon="folder-tree" href="/management/namespace">
|
||||
The SDK equivalent, and how a namespace fits the catalog hierarchy.
|
||||
</Card>
|
||||
|
||||
+27
-10
@@ -4,10 +4,11 @@ sidebarTitle: "Table"
|
||||
description: "CREATE, DESCRIBE, ALTER, CLONE and DROP TABLE."
|
||||
icon: table
|
||||
keywords: ["create table", "alter table", "drop table", "describe", "clone", "ddl"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
## CREATE TABLE {#create}
|
||||
|
||||
```sql
|
||||
@@ -50,7 +51,7 @@ ALTER TABLE t ADD COLUMN doubled AS value * 2
|
||||
`AS <expression>` declares a [computed column](/feature-engineering/add-computed-column).
|
||||
The column is committed empty and the expression is stored, not evaluated:
|
||||
declaring one on a billion-row table costs what declaring one on an empty table
|
||||
costs. [`REFRESH COLUMN`](/sql/dml/refresh) fills it.
|
||||
costs. [`REFRESH COLUMN`](/sql/dml/refresh-column) fills it.
|
||||
|
||||
{/*
|
||||
Draft for automatic computed-column refresh. Enable this section when the
|
||||
@@ -74,19 +75,35 @@ and is minimum pacing after a successful refresh of that output; unchanged
|
||||
inputs do not force work. A new column is immediately eligible for its first
|
||||
refresh. The server's separate automatic-refresh switch must also be enabled.
|
||||
|
||||
These options also work beside a Function binding's `version` and `branch`:
|
||||
These options also work beside a Function binding's `version` and `branch`,
|
||||
continuing the [`caption` Function](/management/function#chain) from Functions:
|
||||
|
||||
```sql
|
||||
ALTER TABLE t ADD COLUMN (label, score) AS classify(text)
|
||||
ALTER TABLE frames ADD COLUMN caption AS caption(extract_frame(video, ts))
|
||||
WITH (version = 'fv_1', auto_refresh_enabled = true,
|
||||
auto_refresh_interval_seconds = 300);
|
||||
ALTER TABLE t ALTER COLUMN score SET auto_refresh_enabled = false;
|
||||
ALTER TABLE frames ALTER COLUMN caption SET auto_refresh_enabled = false;
|
||||
```
|
||||
|
||||
Declaration settings initialize every output in the group; later SET commands
|
||||
change one output. Disabling automatic refresh prevents new submissions while
|
||||
allowing inflight jobs to finish and explicit REFRESH COLUMN to run. Column
|
||||
SUSPEND/RESUME syntax is not supported.
|
||||
change one output. SET takes effect immediately and does not itself read or
|
||||
write any row:
|
||||
|
||||
- **Disabling** prevents new automatic submissions. Inflight jobs may still
|
||||
finish, explicit `REFRESH COLUMN` keeps working, and rows written while
|
||||
disabled commit with no computed value -- they do not queue anywhere
|
||||
separate.
|
||||
- **Re-enabling catches up the entire backlog in one pass.** The next
|
||||
scheduler tick processes every row left unfilled since the column was
|
||||
disabled, immediately, with no further write required to trigger it: a
|
||||
column disabled for an hour and re-enabled fills everything that
|
||||
accumulated during that hour, not just rows written after the moment it
|
||||
is re-enabled.
|
||||
|
||||
Column SUSPEND/RESUME syntax is not supported; `SET auto_refresh_enabled` is
|
||||
the only toggle. The current setting is in the column's own field metadata,
|
||||
`computed_column.auto_refresh_enabled`, readable from `DESCRIBE` or the
|
||||
SDK's `table.schema` without submitting a job to ask.
|
||||
*/}
|
||||
|
||||
{/*
|
||||
@@ -148,7 +165,7 @@ ALTER TABLE t SET RTE WITH (maintained_indexes = '')
|
||||
ALTER TABLE t UNSET RTE
|
||||
```
|
||||
|
||||
The [realtime engine](/tables/realtime-engine) puts a write-ahead log in front
|
||||
The [real time engine](/tables/realtime-engine) puts a write-ahead log in front
|
||||
of the table, making small frequent writes cheap by batching them out of the
|
||||
commit path. `sharding` and `buckets` spread the log so concurrent writers do
|
||||
not contend on one place; `bucket` sharding names its `column`, which must be
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "View"
|
||||
description: "CREATE VIEW and DROP VIEW: a saved query, evaluated when read."
|
||||
icon: eye
|
||||
keywords: ["create view", "drop view", "sql", "catalog"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A view stores a **query**, not rows. Reading it runs the query against current
|
||||
data, so a view is never stale, and never faster than the query it wraps. It
|
||||
is the counterpart to a [materialized view](/sql/ddl/materialized-view), which
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Delete"
|
||||
description: "DELETE FROM ... WHERE: rows are marked, not erased."
|
||||
icon: trash
|
||||
keywords: ["delete", "sql"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
```sql
|
||||
DELETE FROM documents WHERE created_at < TIMESTAMP '2025-01-01 00:00:00'
|
||||
```
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Insert"
|
||||
description: "INSERT INTO, including INSERT ... SELECT to move rows server-side."
|
||||
icon: plus
|
||||
keywords: ["insert", "insert into", "sql"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
```sql
|
||||
INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur')
|
||||
|
||||
|
||||
@@ -1,19 +1,18 @@
|
||||
---
|
||||
title: "REFRESH"
|
||||
sidebarTitle: "REFRESH"
|
||||
description: "Filling computed columns and bringing materialized views up to date."
|
||||
title: "REFRESH COLUMN"
|
||||
sidebarTitle: "REFRESH COLUMN"
|
||||
description: "Filling a computed column: idempotent, scoped, and resumable."
|
||||
icon: arrows-rotate
|
||||
keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"]
|
||||
keywords: ["refresh column", "backfill", "computed column"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
LanceDB extensions with no DataFusion equivalent. They are what makes
|
||||
A LanceDB extension with no DataFusion equivalent. It is what makes
|
||||
[feature engineering](/feature-engineering) reachable from SQL: a column
|
||||
declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it.
|
||||
|
||||
## REFRESH COLUMN {#refresh-column}
|
||||
|
||||
```sql
|
||||
REFRESH COLUMN <column> ON [TABLE] <table>
|
||||
[WHERE <predicate>]
|
||||
@@ -33,7 +32,7 @@ can recompute values the failed run had already written.
|
||||
`WHERE` scopes the refresh, which is how you fill a slice and check it before
|
||||
committing to the whole table.
|
||||
|
||||
### Options {#refresh-column-options}
|
||||
## Options {#options}
|
||||
|
||||
| Option | Meaning |
|
||||
| --- | --- |
|
||||
@@ -62,39 +61,22 @@ memory and GPU options there.
|
||||
</Note>
|
||||
*/}
|
||||
|
||||
## REFRESH MATERIALIZED VIEW {#refresh-materialized-view}
|
||||
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW <name>
|
||||
```
|
||||
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW chunks
|
||||
```
|
||||
|
||||
A refresh updates the view's stored result. Refresh views in dependency order,
|
||||
starting with their source views.
|
||||
|
||||
<Note>
|
||||
Some older server versions accepted `WITH (full = true)` and `CASCADE`.
|
||||
Those options are not accepted by the SQL implementation described here.
|
||||
Use the basic statement above; do not assume that the options accepted by a
|
||||
local SDK's `refresh()` method are also SQL options.
|
||||
</Note>
|
||||
|
||||
## Watching a refresh {#watching}
|
||||
|
||||
Both statements submit work that outlives the call. Track it with
|
||||
A refresh submits work that outlives the call. Track it with
|
||||
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
|
||||
[`SHOW ERRORS`](/sql/ddl/job#errors).
|
||||
|
||||
## See also {#see-also}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
||||
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
||||
The same operation from the SDKs, and what a refresh guarantees.
|
||||
</Card>
|
||||
<Card title="Table" icon="table" href="/sql/ddl/table#alter">
|
||||
Declaring the columns this fills.
|
||||
</Card>
|
||||
<Card title="REFRESH MATERIALIZED VIEW" icon="arrows-rotate" href="/sql/dml/refresh-materialized-view">
|
||||
Bringing a view's stored result up to date, instead of a column.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,50 @@
|
||||
---
|
||||
title: "REFRESH MATERIALIZED VIEW"
|
||||
sidebarTitle: "REFRESH MATERIALIZED VIEW"
|
||||
description: "Bringing a materialized view's stored result up to date."
|
||||
icon: arrows-rotate
|
||||
keywords: ["refresh materialized view", "cascade"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
A LanceDB extension with no DataFusion equivalent.
|
||||
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW <name>
|
||||
```
|
||||
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW chunks
|
||||
```
|
||||
|
||||
A refresh updates the view's stored result. Refresh views in dependency order,
|
||||
starting with their source views.
|
||||
|
||||
<Note>
|
||||
Some older server versions accepted `WITH (full = true)` and `CASCADE`.
|
||||
Those options are not accepted by the SQL implementation described here.
|
||||
Use the basic statement above; do not assume that the options accepted by a
|
||||
local SDK's `refresh()` method are also SQL options.
|
||||
</Note>
|
||||
|
||||
## Watching a refresh {#watching}
|
||||
|
||||
A refresh submits work that outlives the call. Track it with
|
||||
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
|
||||
[`SHOW ERRORS`](/sql/ddl/job#errors).
|
||||
|
||||
## See also {#see-also}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Materialized View" icon="table" href="/sql/ddl/materialized-view">
|
||||
Creating the view this refreshes.
|
||||
</Card>
|
||||
<Card title="Refresh a materialized view" icon="layer-group" href="/feature-engineering/refresh-materialized-view">
|
||||
The same operation from the SDKs, and automatic refresh.
|
||||
</Card>
|
||||
<Card title="REFRESH COLUMN" icon="arrows-rotate" href="/sql/dml/refresh-column">
|
||||
Filling a computed column, instead of a view.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Update"
|
||||
description: "UPDATE ... SET ... WHERE, and why it does not recompute derived columns."
|
||||
icon: pen
|
||||
keywords: ["update", "sql"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
```sql
|
||||
UPDATE documents SET title = upper(title) WHERE category = 'film'
|
||||
```
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Aggregation"
|
||||
description: "GROUP BY, HAVING, and the aggregate function set."
|
||||
icon: layer-group
|
||||
keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximate"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Use SQL aggregates to summarize records, either across a whole table or
|
||||
within groups. Common examples include counts by category, totals and averages.
|
||||
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: "Changes since a version"
|
||||
sidebarTitle: "Changes"
|
||||
description: "CHANGES: read the rows appended to a table since an earlier version."
|
||||
icon: clock-rotate-left
|
||||
keywords: ["changes", "append", "incremental", "cdc", "version"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
`CHANGES` reads the rows appended to a table since an earlier
|
||||
[version](/management/table#versioning), without rescanning rows that were
|
||||
already there:
|
||||
|
||||
```sql
|
||||
SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 42)
|
||||
```
|
||||
|
||||
This returns the rows appended after version 42, up to the table's current
|
||||
version at the time the query runs.
|
||||
|
||||
## Strictly append-only {#strict}
|
||||
|
||||
The clause is named for what it requires, not just what it does: every
|
||||
change to the table between the starting version and the current one must
|
||||
be an append. If any of them inserted with changes other than a pure append
|
||||
-- an update, a delete, a merge -- the query is refused rather than silently
|
||||
returning an incomplete or wrong result. Re-run it with a later starting
|
||||
version that is itself after the non-append write, or read the table in full
|
||||
instead of incrementally.
|
||||
|
||||
`INFORMATION => APPEND_ONLY` is the only supported value: it names what kind
|
||||
of history the clause is willing to read, not a choice among several. `AT
|
||||
(VERSION => ...)` takes a positive integer; there is no `AT (TIMESTAMP =>
|
||||
...)` form for this clause, unlike other version qualifiers.
|
||||
|
||||
## Where the clause can go {#placement}
|
||||
|
||||
It attaches to a table in `FROM`, including inside a subquery, a CTE, a join,
|
||||
a set operation, or `EXPLAIN`:
|
||||
|
||||
```sql
|
||||
WITH recent AS (
|
||||
SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 42)
|
||||
)
|
||||
SELECT count(*) FROM recent
|
||||
|
||||
SELECT * FROM events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 10) a
|
||||
JOIN events CHANGES (INFORMATION => APPEND_ONLY) AT (VERSION => 20) b
|
||||
ON a.id = b.id
|
||||
```
|
||||
|
||||
Each occurrence of a table keeps its own starting version, even when the same
|
||||
table is referenced more than once in one statement.
|
||||
|
||||
## Unsupported forms {#refused}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `CHANGES (INFORMATION => DEFAULT)` | only `APPEND_ONLY` is supported |
|
||||
| `AT (TIMESTAMP => ...)` | only `AT (VERSION => ...)` is supported |
|
||||
| `AT (VERSION => 0)` | the starting version must be positive |
|
||||
| `... END (VERSION => ...)` | not supported; the current version is always the end |
|
||||
| `CHANGES ... TABLESAMPLE (...)` | cannot be combined on the same table reference |
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Table" icon="table" href="/sql/ddl/table">
|
||||
CREATE, ALTER, CLONE and versioning basics.
|
||||
</Card>
|
||||
<Card title="Select" icon="magnifying-glass" href="/sql/dql/select">
|
||||
The rest of the query grammar this clause attaches to.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: "DEDUPE"
|
||||
sidebarTitle: "DEDUPE"
|
||||
description: "The dedupe() table function: one representative row per near-duplicate group, scanned over the index."
|
||||
icon: layer-group
|
||||
keywords: ["dedupe", "vector_dedup", "sql", "deduplication", "near-duplicate", "table function"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
`dedupe` scans a vector column's [index](/indexing/vector-index) partitions
|
||||
for rows closer than a distance threshold, resolves the overlapping pairs it
|
||||
finds into duplicate groups, and returns one representative row per group --
|
||||
without a full pairwise scan or a shuffle. See
|
||||
[`duplicate_pairs`](/sql/dql/duplicate-pairs) to inspect the pairs themselves
|
||||
before committing to this resolution.
|
||||
|
||||
```sql
|
||||
SELECT * FROM dedupe(<table>, <vector_column>,
|
||||
distance_threshold => <float>
|
||||
[, resolution => 'connected_components' | 'direct_representative']
|
||||
[, selection => 'all' | 'first' | 'random', count => <int>, seed => <int>]
|
||||
)
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02)
|
||||
```
|
||||
|
||||
<Note>
|
||||
Earlier releases spelled this `vector_dedup`; that name is rejected on
|
||||
current deployments.
|
||||
</Note>
|
||||
|
||||
## Input {#input}
|
||||
|
||||
| Argument | Type | Required | Meaning |
|
||||
| --- | --- | --- | --- |
|
||||
| table name | string literal | yes | the table to scan |
|
||||
| vector column | string literal | yes | the indexed vector column to compare |
|
||||
| `distance_threshold` | float | yes | maximum distance for a pair to count as close |
|
||||
| `resolution` | `'connected_components'` (default) \| `'direct_representative'` | no | how overlapping pairs group together -- see [Resolution](#resolution) |
|
||||
| `selection` | `'all'` (default) \| `'first'` \| `'random'` | no | which index partitions to scan |
|
||||
| `count` | int | no | number of partitions to sample when `selection => 'random'` |
|
||||
| `seed` | int | no | random seed when `selection => 'random'` |
|
||||
|
||||
`id_column` is refused on `dedupe`: it has no separate pair-identifier output
|
||||
to project onto, only [`duplicate_pairs`](/sql/dql/duplicate-pairs#input)
|
||||
accepts it.
|
||||
|
||||
## Output {#output}
|
||||
|
||||
One row per duplicate group, **in the table's own schema** -- every column of
|
||||
`<table>`, including a BlobV2 column, comes through unchanged on the
|
||||
representative row Lance selected for that group. Unlike
|
||||
[`duplicate_pairs`](/sql/dql/duplicate-pairs#output), there is no fixed
|
||||
`dedupe`-specific schema to document: the output schema is the source
|
||||
table's schema.
|
||||
|
||||
Because the result is already shaped like the table, it composes directly
|
||||
into a new table or a materialized view:
|
||||
|
||||
```sql
|
||||
CREATE MATERIALIZED VIEW frames_deduped AS
|
||||
SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02)
|
||||
```
|
||||
|
||||
## Resolution {#resolution}
|
||||
|
||||
`resolution` chooses how overlapping pairs group together:
|
||||
|
||||
| `resolution` | |
|
||||
| --- | --- |
|
||||
| `connected_components` (default) | groups transitively: if a~b and b~c are both close, all three resolve together |
|
||||
| `direct_representative` | only pairs within the threshold of each other resolve together |
|
||||
|
||||
## Sampling partitions {#sampling}
|
||||
|
||||
`selection` scopes the scan to a sample of partitions instead of all of them,
|
||||
the same `all` (default) / `first` / `random` choice as
|
||||
[`ivf_partition`](/sql/scalar-functions#ivf-partition). Use it to estimate a
|
||||
duplicate rate on a large table before scanning it in full:
|
||||
|
||||
```sql
|
||||
SELECT * FROM dedupe('frames', 'vec',
|
||||
distance_threshold => 0.02, selection => 'random', count => 100, seed => 42)
|
||||
```
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="DUPLICATE_PAIRS" icon="clone" href="/sql/dql/duplicate-pairs">
|
||||
Inspect the close pairs themselves instead of the resolved groups.
|
||||
</Card>
|
||||
<Card title="Deduplicate near-duplicates" icon="copy" href="/curation/dedupe">
|
||||
When to reach for this instead of exact-key deduplication.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: "DUPLICATE_PAIRS"
|
||||
sidebarTitle: "DUPLICATE_PAIRS"
|
||||
description: "The duplicate_pairs() table function: every close pair in a vector column, scanned over the index."
|
||||
icon: clone
|
||||
keywords: ["duplicate_pairs", "vector_duplicate_pairs", "sql", "deduplication", "near-duplicate", "table function"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
`duplicate_pairs` scans a vector column's [index](/indexing/vector-index)
|
||||
partitions for every pair of rows closer than a distance threshold, without a
|
||||
full pairwise scan or a shuffle. It returns one row per close pair, not one
|
||||
row per source row -- use it to inspect what would be collapsed before
|
||||
committing to [`dedupe`](/sql/dql/dedupe), which resolves these same pairs
|
||||
into groups.
|
||||
|
||||
```sql
|
||||
SELECT * FROM duplicate_pairs(<table>, <vector_column>,
|
||||
distance_threshold => <float>
|
||||
[, id_column => <column>]
|
||||
[, selection => 'all' | 'first' | 'random', count => <int>, seed => <int>]
|
||||
)
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02)
|
||||
```
|
||||
|
||||
<Note>
|
||||
Earlier releases spelled this `vector_duplicate_pairs`; that name is rejected
|
||||
on current deployments.
|
||||
</Note>
|
||||
|
||||
## Input {#input}
|
||||
|
||||
| Argument | Type | Required | Meaning |
|
||||
| --- | --- | --- | --- |
|
||||
| table name | string literal | yes | the table to scan |
|
||||
| vector column | string literal | yes | the indexed vector column to compare |
|
||||
| `distance_threshold` | float | yes | maximum distance for a pair to count as close |
|
||||
| `id_column` | column name | no | project this column's value instead of the internal row ID -- see [Application-level identifiers](#id-column) |
|
||||
| `selection` | `'all'` (default) \| `'first'` \| `'random'` | no | which index partitions to scan |
|
||||
| `count` | int | no | number of partitions to sample when `selection => 'random'` |
|
||||
| `seed` | int | no | random seed when `selection => 'random'` |
|
||||
|
||||
`resolution` is refused on `duplicate_pairs`: a pair has no notion of a group,
|
||||
only [`dedupe`](/sql/dql/dedupe#input) accepts it.
|
||||
|
||||
## Output {#output}
|
||||
|
||||
One row per close pair found, in a fixed schema independent of the source
|
||||
table:
|
||||
|
||||
| Column | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `row_id_a` | uint64 | internal row ID of the first row in the pair, the same identifier as the [`_rowid`](/sql/data-types#pseudo-columns) pseudo-column |
|
||||
| `row_id_b` | uint64 | internal row ID of the second row in the pair |
|
||||
| `distance` | float32 | distance between the pair's vectors, at or below `distance_threshold` |
|
||||
| `segment_id` | int64 | which index segment the pair was found in |
|
||||
| `partition_id` | int64 | which index partition the pair was found in |
|
||||
| `dataset_version` | int64 | the dataset version the scan ran against |
|
||||
|
||||
### Application-level identifiers {#id-column}
|
||||
|
||||
A row ID is internal and not stable across compaction. Pass `id_column`, a
|
||||
top-level scalar column such as a primary key, to project its value instead:
|
||||
`row_id_a`/`row_id_b` are replaced by `id_a`/`id_b`, typed like `id_column`.
|
||||
|
||||
```sql
|
||||
SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02, id_column => 'frame_id')
|
||||
```
|
||||
|
||||
## Sampling partitions {#sampling}
|
||||
|
||||
`selection` scopes the scan to a sample of partitions instead of all of them,
|
||||
the same `all` (default) / `first` / `random` choice as
|
||||
[`ivf_partition`](/sql/scalar-functions#ivf-partition). Use it to estimate a
|
||||
distance threshold or duplicate rate on a large table before scanning it in
|
||||
full:
|
||||
|
||||
```sql
|
||||
SELECT * FROM duplicate_pairs('frames', 'vec',
|
||||
distance_threshold => 0.02, selection => 'random', count => 100, seed => 42)
|
||||
```
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="DEDUPE" icon="layer-group" href="/sql/dql/dedupe">
|
||||
Resolve these same pairs into one representative row per group.
|
||||
</Card>
|
||||
<Card title="Deduplicate near-duplicates" icon="copy" href="/curation/dedupe">
|
||||
When to reach for this instead of exact-key deduplication.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Explain"
|
||||
description: "EXPLAIN, EXPLAIN ANALYZE and EXPLAIN VERBOSE: reading and measuring a plan from SQL."
|
||||
icon: map
|
||||
keywords: ["explain", "explain analyze", "explain verbose", "query plan"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
```sql
|
||||
EXPLAIN SELECT * FROM documents WHERE identifier < 1000
|
||||
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Full-Text Search"
|
||||
description: "The fts() table function: match, fuzzy, phrase, boolean, boost and multi-match queries from SQL."
|
||||
icon: magnifying-glass
|
||||
keywords: ["fts", "sql", "full text search", "fuzzy", "phrase", "boolean", "boost", "multi-match"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Full-text search is a **table function**, `fts(table, query)`, rather than a
|
||||
predicate: it produces a relevance score alongside the matching rows, and a
|
||||
`WHERE` clause has nowhere to put one. The two string-literal arguments are
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Joins"
|
||||
description: "Combining rows from more than one table, including across namespaces."
|
||||
icon: link
|
||||
keywords: ["join", "inner", "left", "outer", "cross", "namespace"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Use SQL joins to combine rows from two or more tables.
|
||||
|
||||
## The join types {#types}
|
||||
@@ -70,3 +71,29 @@ Avoid selecting blob or vector columns you do not need on either side of a join.
|
||||
The join multiplies rows before projection is applied in some plans, so a wide
|
||||
column can be materialized many more times than the result has rows.
|
||||
</Warning>
|
||||
|
||||
## Late materialization for wide columns {#late-materialization}
|
||||
|
||||
The planner can avoid the multiplication above: when a join narrows down to
|
||||
a key column (such as `_rowid` or another narrow, selective column) before a
|
||||
wide one is needed, it can defer reading a BlobV2 or vector column until
|
||||
after the join has already settled which rows survive, instead of reading it
|
||||
for every candidate row first. In a plan where this applies, the wide column
|
||||
is read once per surviving row, where the worker runs, and never pulled into
|
||||
the query node to be joined directly.
|
||||
|
||||
```sql
|
||||
SELECT d.title, b.caption
|
||||
FROM documents AS d
|
||||
JOIN (SELECT _rowid, caption FROM captions WHERE source = 'model-v2') AS b
|
||||
ON b._rowid = d._rowid
|
||||
```
|
||||
|
||||
This is query planning, not a syntax you opt into: a join written this way
|
||||
is a **candidate** for the planner to defer the wide read, not a guarantee.
|
||||
Confirm it for a specific query with [`EXPLAIN`](/sql/dql/explain) --
|
||||
`RemoteTake` in the plan means the deferral happened; its absence means the
|
||||
wide column was read eagerly, even if the SQL above matches exactly. Neither
|
||||
a passing query nor an `EXPLAIN` that omits `RemoteTake` tells you how much
|
||||
I/O a plan did or did not save -- read the plan's own row and byte counts for
|
||||
that.
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Sampling"
|
||||
description: "TABLESAMPLE: draw a random subset of a table, by rows or by fragments."
|
||||
icon: dice
|
||||
keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Use `TABLESAMPLE` to explore a subset of a table or estimate aggregate
|
||||
statistics. The sampling method determines how much data is read and whether
|
||||
the table's physical layout influences the sample.
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "SELECT"
|
||||
description: "Projection, filtering, ordering, and composing queries out of other queries."
|
||||
icon: table-columns
|
||||
keywords: ["select", "where", "order by", "limit", "cte", "union", "distinct"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
```sql
|
||||
[WITH cte AS (...)]
|
||||
SELECT [ALL | DISTINCT] select_expr [, ...]
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Subqueries"
|
||||
description: "Queries inside queries: scalar, IN, EXISTS, correlated, and derived tables."
|
||||
icon: code-branch
|
||||
keywords: ["subquery", "exists", "in", "correlated", "derived table", "lateral"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A subquery is allowed anywhere an expression or a relation is.
|
||||
|
||||
## As a relation {#derived}
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Window Functions"
|
||||
description: "Compute over a set of rows related to the current one, without collapsing them."
|
||||
icon: chart-simple
|
||||
keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A window function computes across a set of rows related to the current one and
|
||||
returns a value **per row**. That is the difference from
|
||||
[aggregation](/sql/dql/aggregation), which collapses the rows it covers.
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Functions"
|
||||
description: "Where to find each kind of function, and the dialect version they come from."
|
||||
icon: function
|
||||
keywords: ["functions", "datafusion", "scalar", "aggregate", "window", "fts", "table function"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org),
|
||||
currently **version 54**. The function set is DataFusion's default registration
|
||||
plus what Lance adds.
|
||||
@@ -77,6 +78,29 @@ like any other column (order by it, filter on it, project it) and exists only
|
||||
in a relation that came from `fts()`. See
|
||||
[Data types](/sql/data-types#pseudo-columns).
|
||||
|
||||
### `dedupe` and `duplicate_pairs` {#dedupe}
|
||||
|
||||
Scan a vector column's [index](/indexing/vector-index) partitions for rows
|
||||
closer than a distance threshold, without a full pairwise scan or a shuffle:
|
||||
|
||||
```sql
|
||||
SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02)
|
||||
SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02)
|
||||
```
|
||||
|
||||
<Note>
|
||||
Earlier releases spelled these `vector_duplicate_pairs` and `vector_dedup`;
|
||||
those names are rejected on current deployments.
|
||||
</Note>
|
||||
|
||||
`duplicate_pairs` returns every close pair it finds, in a fixed schema.
|
||||
`dedupe` instead resolves overlapping pairs into duplicate groups and returns
|
||||
one representative row **in the table's own schema** per group, so its result
|
||||
composes directly into `CREATE TABLE ... AS` or
|
||||
`CREATE MATERIALIZED VIEW ... AS`. See [`DEDUPE`](/sql/dql/dedupe) and
|
||||
[`DUPLICATE_PAIRS`](/sql/dql/duplicate-pairs) for the full input and output
|
||||
schema, resolution grouping, and partition sampling.
|
||||
|
||||
### `generate_series` {#generate-series}
|
||||
|
||||
```sql
|
||||
|
||||
@@ -32,7 +32,7 @@ for batch in reader:
|
||||
|
||||
SQL results arrive as Arrow record batches. Iterating over the reader avoids
|
||||
collecting the entire result in application memory. See
|
||||
[Queries](/basics/query) for result handling.
|
||||
[Read and manage results](/basics/sql#results) for more.
|
||||
|
||||
## Usage {#usage}
|
||||
|
||||
@@ -47,6 +47,7 @@ operations that have no SQL form, such as a merge insert. See
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| [Database](/sql/ddl/database) | `ATTACH DATABASE` |
|
||||
| [Namespace](/sql/ddl/namespace) | `CREATE SCHEMA`, `DROP SCHEMA` |
|
||||
| [Table](/sql/ddl/table) | `CREATE`, `DESCRIBE`, `ALTER`, `CLONE`, `DROP TABLE` |
|
||||
| [View](/sql/ddl/view) | `CREATE VIEW`, `DROP VIEW` |
|
||||
@@ -67,7 +68,8 @@ Draft for the Functions release. Add this row to the table above, after Index:
|
||||
| [Update](/sql/dml/update) | `UPDATE ... SET` |
|
||||
| [Delete](/sql/dml/delete) | `DELETE FROM` |
|
||||
| [Copy](/sql/dml/copy) | `COPY` (refused by recent deployment versions) |
|
||||
| [Refresh](/sql/dml/refresh) | filling computed columns and materialized views |
|
||||
| [Refresh column](/sql/dml/refresh-column) | filling a computed column |
|
||||
| [Refresh materialized view](/sql/dml/refresh-materialized-view) | bringing a view's stored result up to date |
|
||||
|
||||
### DQL {#dql}
|
||||
|
||||
@@ -80,6 +82,7 @@ Draft for the Functions release. Add this row to the table above, after Index:
|
||||
| [Aggregation](/sql/dql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set |
|
||||
| [Window functions](/sql/dql/window) | `OVER`, framing, ranking and offset functions |
|
||||
| [Sampling](/sql/dql/sampling) | `TABLESAMPLE`, by rows or by fragments |
|
||||
| [Changes](/sql/dql/changes) | `CHANGES`: rows appended since an earlier version |
|
||||
| [Full-Text Search](/sql/dql/fts) | the `fts()` table function, and its query forms |
|
||||
| [Explain](/sql/dql/explain) | `EXPLAIN`, `EXPLAIN ANALYZE`, `EXPLAIN VERBOSE` |
|
||||
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Operators"
|
||||
description: "Comparison, logical and arithmetic operators, literals, and precedence."
|
||||
icon: plus-minus
|
||||
keywords: ["operators", "literals", "precedence", "null", "cast", "interval"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
LanceDB uses DataFusion's SQL operators. This page describes comparison,
|
||||
logical and arithmetic operators, literal values, type conversions and
|
||||
operator precedence.
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Prepared Statements"
|
||||
description: "What a deployment does when a Flight SQL client prepares a statement or binds parameter values."
|
||||
icon: bolt
|
||||
keywords: ["prepared statement", "flight sql", "parameters", "bind"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A Flight SQL client can prepare a statement before it runs it. A deployment
|
||||
accepts that, but it keeps no plan between runs and does not accept parameter
|
||||
values.
|
||||
@@ -50,6 +51,13 @@ Do not put untrusted input, such as a value from a user or a request, directly
|
||||
into the statement text. Quoting it does not protect the statement the way
|
||||
binding a parameter would.
|
||||
|
||||
<Note>
|
||||
This limitation is specific to the Flight SQL protocol's `PREPARE`/bind step.
|
||||
The Python SDK's `execute_query(sql, parameters=...)` binds parameter values
|
||||
through a separate, LanceDB-specific mechanism -- see
|
||||
[Parameters in SQL queries](/basics/sql#parameters).
|
||||
</Note>
|
||||
|
||||
## When to use prepared statements {#when}
|
||||
|
||||
Prepare a statement when your client requires it, or to get a statement's
|
||||
|
||||
@@ -4,10 +4,11 @@ sidebarTitle: "Scalar Functions"
|
||||
description: "The per-row function set: math, string, regex, date/time, array, struct, and what Lance adds."
|
||||
icon: calculator
|
||||
keywords: ["scalar functions", "string functions", "math functions", "regex", "array functions", "json"]
|
||||
availability:
|
||||
embedded: unavailable
|
||||
enterprise: available
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Functions that take a value and return a value, one row at a time. The set is
|
||||
based on DataFusion's registration plus Lance and deployment-specific
|
||||
functions. The names below are selected examples, not an exhaustive catalog
|
||||
@@ -98,6 +99,21 @@ functions supported by your deployment and inspect SQL
|
||||
index-backed plan. See [SQL vector search](/basics/sql#no-vector-sql).
|
||||
</Warning>
|
||||
|
||||
A third, optional argument is a comma-separated options string. `metric`
|
||||
selects `l2` (default, alias `euclidean`), `cosine`, or `dot`; the
|
||||
index-search rewrite also reads `nprobes` and `use_index`:
|
||||
|
||||
```sql
|
||||
SELECT id FROM documents
|
||||
ORDER BY distance(embedding, ARRAY[0.1, 0.2, 0.3], 'metric=cosine,nprobes=20')
|
||||
LIMIT 10
|
||||
|
||||
-- Force a flat scan, bypassing the index:
|
||||
SELECT id FROM documents
|
||||
ORDER BY exact_distance(embedding, ARRAY[0.1, 0.2, 0.3], 'use_index=false')
|
||||
LIMIT 10
|
||||
```
|
||||
|
||||
## Structs and fields {#structs}
|
||||
|
||||
```text
|
||||
@@ -146,6 +162,44 @@ semantics and the simple tokenizer.
|
||||
query, the planner uses it instead of scanning, which is the reason to prefer
|
||||
it over `LIKE` for keyword tests.
|
||||
|
||||
### `ivf_partition(vector, ...)` {#ivf-partition}
|
||||
|
||||
```sql
|
||||
SELECT ivf_partition(embedding, selection => 'random', count => 100, seed => 42) AS partition,
|
||||
array_agg(id) AS ids
|
||||
FROM documents
|
||||
GROUP BY partition
|
||||
```
|
||||
|
||||
Assigns each row to one of `count` IVF-style partitions over `vector`, using
|
||||
the named arguments `selection` (partition-center strategy; `'random'` is
|
||||
sampled with `seed`) and `seed`. Grouping by its result buckets rows by
|
||||
vector proximity without building or querying a vector index — useful for
|
||||
blocking a large table into comparable chunks, such as before a pairwise
|
||||
similarity pass.
|
||||
|
||||
### `blob_descriptor(column)` {#blob-descriptor}
|
||||
|
||||
```sql
|
||||
SELECT id, blob_descriptor(video) AS descriptor FROM videos
|
||||
```
|
||||
|
||||
A [BlobV2](/management/table#managed-blobs) column's bytes are not fetched by
|
||||
a plain `SELECT`: that already returns `large_binary`, the same as any other
|
||||
binary column. `blob_descriptor` instead returns a small struct describing
|
||||
where the payload lives, without reading it. Pass one row's descriptor to
|
||||
the SDK's blob file reader to open that payload directly:
|
||||
|
||||
```python
|
||||
rows = db.execute_query("SELECT id, blob_descriptor(video) AS descriptor FROM videos").read_all().to_pylist()
|
||||
with table.open_blob_file(rows[0]["descriptor"]) as reader:
|
||||
payload = reader.read()
|
||||
```
|
||||
|
||||
A descriptor is pinned to the table version it was read at: it keeps
|
||||
resolving correctly even after the table is appended to and checked out to a
|
||||
newer version.
|
||||
|
||||
### JSON accessors {#json}
|
||||
|
||||
| Function | Returns |
|
||||
|
||||
Reference in New Issue
Block a user