rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-16 16:52:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: correct data workflows, SQL and operations (#4534)

Browse Source 
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.

The curation, SQL and operations pages contain outdated claims and
examples that do not reliably work as written.

This corrects those pages against the reviewed SDK and local server
evidence, including prepared statements, background jobs, cleanup and
monitoring. Integration guides are grouped with their examples in #4533.

Function and Secret-object documentation stays unpublished until its
release. The two Function pages are listed in `.mintignore` and left out
of the navigation, and the sections about them on other pages are MDX
comments.

Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after
#4533 is squash-merged.

Local assembly, Mint validation and link checks pass, and the assembled
site shows no Function or Secret-object content. A local release
rehearsal that uncomments the drafts and restores the navigation also
passes. Hold publication until this PR and lancedb/sophon#8688 are on
`main`.
This commit is contained in:
Brendan Clement
2026-10-15 14:57:20 -07:00
committed by GitHub
parent 758f738b8c
commit 06aa526ae1
72 changed files with 1997 additions and 1254 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/sql/ddl/function.mdx
+8 -3
View File
@@ -6,17 +6,22 @@ icon: function
keywords: ["create function", "drop function", "show functions", "sql", "udf"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
## CREATE FUNCTION {#create}
```sql
CREATE FUNCTION f(x BIGINT) RETURNS BIGINT
CREATE FUNCTION f(x int64) RETURNS int64
LANGUAGE python
AS 'def f(x): return x * 2'
WITH (pip = 'numpy==1.26.4', python_version = '3.11')
```
Parameter and result types are Arrow type names, not SQL ones: `int64`,
`float64`, `utf8`, `bool`, `date32`, and lists such as `list<float32>` or
`fixed_size_list<float32, 768>`. `BIGINT` is refused. A parameter accepts nulls
unless it is declared `NOT NULL`.
The body is Python source. Its dependencies are declared with it and are part
of the **immutable function version** -- changing one produces a new version
rather than altering the registered one, so a column filled by an earlier
@@ -27,7 +32,7 @@ version is unaffected.
| `pip` | pip requirements for the worker environment |
| `conda` | conda packages instead of pip |
| `conda_channels` | channels in priority order; requires `conda` |
| `python_version` | worker Python version, defaulting to yours |
| `python_version` | worker Python version; `3.12` when omitted, whatever Python you send the statement from |
| `gpu` | `'true'` when every execution needs a GPU |
**Errors and limits**
docs/web/sql/ddl/index.mdx
+21 -11
View File
@@ -1,30 +1,38 @@
---
title: "Index"
sidebarTitle: "Index"
description: "CREATE INDEX and DROP INDEX — one column, server-chosen name."
description: "CREATE INDEX and DROP INDEX — one column, with the algorithm chosen by USING."
icon: bolt
keywords: ["create index", "drop index", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
## CREATE INDEX {#create}
```sql
CREATE INDEX ON documents (title)
CREATE INDEX ON documents USING btree (title)
CREATE INDEX text_search ON documents USING fts (text)
```
`USING` is required: it names the algorithm. The scalar algorithms are `btree`,
`bitmap` and `label_list`; `fts` builds a full-text index; and the vector
algorithms are `ivf_pq`, `ivf_rq`, `ivf_flat`, `ivf_hnsw_sq` and `ivf_hnsw_pq`,
where the deployment supports them. Index options go in a `WITH (...)` clause.
<Note>
SQL `CREATE INDEX` covers **one column** and takes **no user-chosen name** --
the server names it. For a multi-column or explicitly-named index, or to
choose the algorithm, use the SDK's
[`create_index`](/indexing).
In LanceDB Enterprise, SQL `CREATE INDEX` covers **one column**. A name between
`INDEX` and `ON` names the index, where the deployment's configuration allows
it; without one, the server names it after the column, as in `title_idx`. For a
multi-column index, use the SDK's [`create_index`](/indexing).
</Note>
<Note>
**Enterprise job type:** `create_ivf_pq`, `create_btree`, `create_fts`, and one
more per algorithm -- the type names which algorithm is building, not a
different feature. See [Build & Repair](/indexing/reindexing).
**Enterprise job type:** `reindex_` followed by the algorithm, such as
`reindex_btree` or `reindex_ivf_pq`; the name says which algorithm is building,
not a different feature. An SDK build that must not replace an existing index
runs as `create_` followed by the algorithm instead. See
[Build & Repair](/indexing/reindexing).
</Note>
## DROP INDEX {#drop}
@@ -35,7 +43,9 @@ DROP INDEX documents.title_idx
Takes the qualified `table.index` name. Find it with the SDK's
[`list_indices`](/indexing) if you created the index without
naming it yourself.
naming it yourself. A deployment builds the index in the background after
`CREATE INDEX` returns; until that build finishes, the index is not listed and
`DROP INDEX` reports it as not found.
## What to read next {#next}
docs/web/sql/ddl/job.mdx
+2 -2
View File
@@ -6,7 +6,7 @@ icon: hourglass-half
keywords: ["show jobs", "show errors", "pause job", "resume job", "cancel job", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
A [job](/basics/jobs) is not a catalog object -- it has no schema and is not
namespaced -- but it has its own SQL statements, grouped here the same way the
@@ -73,7 +73,7 @@ being ignored.
Cancellation is a request, matching the SDK's
[`cancel()`](/basics/jobs#managing): it does not guarantee the job stops before
doing more work, and asking twice or cancelling a finished job does nothing.
`CANCEL JOB` itself can be unavailable on a deployment with no job registry
`CANCEL JOB` itself can be unavailable on an Enterprise cluster with no job registry
configured, which is a deployment condition rather than something you can work
around from SQL.
</Note>
docs/web/sql/ddl/materialized-view.mdx
+11 -12
View File
@@ -6,13 +6,13 @@ icon: table
keywords: ["materialized view", "create materialized view", "show materialized views", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
A materialized view stores a query's result as rows, kept current by a refresh
-- the counterpart to a [view](/sql/ddl/view), which stores only the query.
<Note>
Materialized views are off by default on a deployment and have to be enabled
Materialized views are off by default in LanceDB Enterprise and have to be enabled
there. Which statements and query shapes a deployment accepts, whether it needs
the source table to have stable row ids, and how it refreshes a view depend on
its LanceDB Enterprise version and configuration.
@@ -24,10 +24,9 @@ its LanceDB Enterprise version and configuration.
CREATE MATERIALIZED VIEW <name> AS SELECT ...
```
On a `db://` connection, the SDK's
[`create_materialized_view`](/management/materialized-view#create) sends this
statement. Once created, a view is queried like a table, refreshed with SQL and
listed with SQL.
Once created, a materialized view can be queried like a table. Use the
statements below to list, refresh and drop it. For SDK methods, see
[Materialized views](/management/materialized-view#create).
## SHOW MATERIALIZED VIEWS {#show}
@@ -36,8 +35,8 @@ SHOW MATERIALIZED VIEWS
```
<Note>
Cheaper than the SDK equivalent. `list_materialized_views` finds views by
reading every table's schema, costing one open per table; this asks the server.
This asks the server for its views, as the SDK's `list_materialized_views` does
on a `db://` connection.
</Note>
## Refreshing one {#refresh}
@@ -47,7 +46,7 @@ REFRESH MATERIALIZED VIEW chunks
```
See [REFRESH](/sql/dml/refresh#refresh-materialized-view) for the full syntax,
including `CASCADE` and `full = true`.
and compatibility notes for older server versions.
## Dropping one {#drop}
@@ -57,8 +56,8 @@ stored rows, unlike dropping a [view](/sql/ddl/view#drop).
<Note>
**Enterprise job type:** `mat_view_refresh` for a refresh, or `sql` on a
deployment that rebuilds views through SQL; `drop_table` for dropping one, same
as any table.
deployment that rebuilds views through SQL. Dropping one runs the same job as
dropping any table, `drop_table` or `drop_object`.
</Note>
## What to read next {#next}
@@ -68,6 +67,6 @@ as any table.
Creating one from the SDK.
</Card>
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/materialized-views">
How a refresh works, locally and on a deployment.
How a refresh works, locally and in LanceDB Enterprise.
</Card>
</CardGroup>
docs/web/sql/ddl/namespace.mdx
+10 -8
View File
@@ -6,7 +6,7 @@ icon: folder-tree
keywords: ["namespace", "schema", "create schema", "drop schema", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
A [namespace](/management/namespace) is addressed in SQL as a `SCHEMA` -- the
standard's name for the same concept.
@@ -14,14 +14,16 @@ standard's name for the same concept.
## CREATE SCHEMA {#create}
```sql
CREATE SCHEMA public
CREATE SCHEMA db1.ns1.ns2
CREATE SCHEMA analytics
CREATE SCHEMA db1.`ns1$ns2`
CREATE SCHEMA IF NOT EXISTS `ns1$ns2`
```
A dotted path creates the whole chain: `db1.ns1.ns2` creates `ns2` inside
`ns1` inside `db1`, provided the parents already exist. A name containing the
namespace path delimiter itself is backtick-quoted, as in `` `ns1$ns2` `` above.
A nested namespace is one backtick-quoted name with `$` between its
components: `` db1.`ns1$ns2` `` creates `ns2` inside `ns1` in database `db1`,
provided `ns1` already exists. The part before the dot is the database, so a
three-part path such as `db1.ns1.ns2` is rejected. `public` is the database's
root namespace, which is never created or dropped.
`IF NOT EXISTS` makes the statement a no-op rather than an error when the
namespace is already there.
@@ -29,14 +31,14 @@ namespace is already there.
## DROP SCHEMA {#drop}
```sql
DROP SCHEMA db1.public
DROP SCHEMA db1.analytics
DROP SCHEMA IF EXISTS `a$b`
DROP SCHEMA `a$b` RESTRICT
```
<Warning>
`CASCADE` is rejected: **"DROP SCHEMA CASCADE is not supported; drop the
objects inside first."** A namespace's tables, views and functions must be
objects inside first."** A namespace's tables, views and other objects must be
dropped explicitly before the namespace itself can go. `RESTRICT` -- the
default -- states that requirement rather than changing it.
</Warning>
docs/web/sql/ddl/table.mdx
+21 -13
View File
@@ -6,7 +6,7 @@ icon: table
keywords: ["create table", "alter table", "drop table", "describe", "clone", "ddl"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
## CREATE TABLE {#create}
@@ -30,16 +30,15 @@ The table's columns and types, without a plan -- the SQL counterpart of
## ALTER TABLE {#alter}
`ALTER TABLE ... ADD COLUMN` covers three different things, distinguished by
what follows the column name.
What `ALTER TABLE ... ADD COLUMN` does depends on what follows the column
name.
### A plain column {#plain-column}
```sql
ALTER TABLE t ADD COLUMN c INT
```
Added empty and nullable. Nothing is computed.
SQL `ADD COLUMN` requires an `AS` binding, so a plain column such as
`ALTER TABLE t ADD COLUMN c INT` is rejected. Add an empty, nullable column
with the SDK instead -- see
[Add nullable columns](/management/table#schema-add-nullable-columns).
### A computed column {#computed-column}
@@ -52,6 +51,9 @@ The column is committed empty and the expression is stored, not evaluated --
declaring one on a billion-row table costs what declaring one on an empty table
costs. [`REFRESH COLUMN`](/sql/dml/refresh) fills it.
{/*
Draft for the Functions release.
### A function column {#function-column}
```sql
@@ -86,6 +88,7 @@ ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2')
`WITH` on a plain computed column accepts only `version`. Resource options such
as `num_workers` apply to function columns, where there are workers to size.
</Note>
*/}
### Table properties {#properties}
@@ -95,19 +98,23 @@ ALTER TABLE t SET UNENFORCED PRIMARY KEY (id)
Records which column identifies a row. **Unenforced** is not a hedge in the
naming: uniqueness is not checked on write, and declaring it does not make
duplicates impossible. It tells the planner what to assume.
duplicates impossible. It tells the planner what to assume. The column must be
declared `NOT NULL`.
```sql
ALTER TABLE t SET WAL
ALTER TABLE t SET WAL WITH (sharding = 'bucket', buckets = 4)
ALTER TABLE t SET WAL WITH (sharding = 'bucket', column = 'id', buckets = 4)
ALTER TABLE t SET WAL WITH (maintained_indexes = '')
ALTER TABLE t UNSET WAL
```
A write-ahead log makes small frequent writes cheap by batching them out of the
commit path. `sharding` and `buckets` spread the log so concurrent writers do
not contend on one place; `maintained_indexes` chooses which indexes are kept
current against WAL rows. See
not contend on one place; `bucket` sharding names its `column`, which must be
the table's unenforced primary key, and inserts into the table then need a
scalar index on that key. `maintained_indexes` chooses which indexes are kept
current against WAL rows. A table with a computed column cannot take a
write-ahead log. See
[Background jobs](/maintenance/background-jobs#wal) for how the log gets
folded back into the table.
@@ -162,7 +169,8 @@ Removes the table and every version of it. There is no undrop.
</Warning>
<Note>
**Enterprise job type:** `drop_table`. See
**Enterprise job type:** `drop_table` or `drop_object`, depending on how the
deployment stores its catalog. See
[Background jobs](/maintenance/background-jobs#teardown).
</Note>
docs/web/sql/ddl/view.mdx
+1 -1
View File
@@ -6,7 +6,7 @@ icon: eye
keywords: ["create view", "drop view", "sql", "catalog"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
A view stores a **query**, not rows. Reading it runs the query against current
data, so a view is never stale -- and never faster than the query it wraps. It