rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-13 07:12:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: restructure Documentation around one axis, promote the SQL reference

Browse Source 
Every group now answers exactly one question. Get started and Concepts cover
what the system is; the verb groups cover doing something with it; Operate,
Secure and Deploy cover running it; SQL reference is lookup. Maintenance becomes
Operate, Governance becomes Secure, and the rest take verbs to match.

Three new pages in Concepts and Get started:

Guarantees states what a write promises when it returns, which version a reader
sees, what survives a failure, and -- the half usually missing -- what is not
guaranteed: no multi-table transactions, no read-your-writes across processes
without an interval, unenforced primary keys that do not enforce, no background
compaction.

Limits collects bounds that were stated in passing across a dozen pages, with
the numbers verified against the code rather than inferred: name lengths, job
event caps, SHOW ERRORS bounds, the sampling range that is refused rather than
clamped.

Architecture illustrates embedded against distributed as the one deployment
decision, and is written to cover both rather than only Enterprise.

Indexing splits by question. Choosing an index is a Query decision and moves
there; building, repairing and GPU-building it stay in Operate.

SQL becomes a top-level reference of fifteen pages instead of a sub-group of
Query. Auditing sophon-sql turned up a large undocumented surface: ALTER TABLE
ADD COLUMN in its plain, computed and function forms with FILTER, IGNORE NULLS
and OVER modifiers; SET UNENFORCED PRIMARY KEY; SET and UNSET WAL with sharding;
CREATE FUNCTION with its dependency options; REFRESH COLUMN and REFRESH
MATERIALIZED VIEW; and the SHOW family. None of it was written down.
This commit is contained in:
Jack Ye
2026-09-13 23:32:15 -07:00
parent 7b112debca
commit 5b5ed930ac
13 changed files with 1257 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/sql/data-types.mdx
+108
View File
@@ -0,0 +1,108 @@
---
title: "SQL data types"
sidebarTitle: "Data Types"
description: "How Lance columns appear to SQL, including the types SQL has no name for."
icon: shapes
keywords: ["sql types", "arrow", "cast", "pseudo-columns", "vector", "blob", "json"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
There is no SQL type layer over Lance storage. The table provider hands
DataFusion the dataset's Arrow schema directly, so a column is the same object
to the query engine as it is to the format.
The consequence is that SQL type names here are **spellings of Arrow types**,
not a system that maps onto them. [Data types](/basics/data-types) documents the
storage side; this page covers what changes when you reach a column from SQL.
## SQL names for Arrow types {#names}
| SQL | Arrow |
| --- | --- |
| `BOOLEAN` | `Boolean` |
| `TINYINT` `SMALLINT` `INT` `BIGINT` | `Int8` `Int16` `Int32` `Int64` |
| `UTINYINT` … `UBIGINT` | the unsigned counterparts |
| `REAL` / `FLOAT` | `Float32` |
| `DOUBLE` | `Float64` |
| `VARCHAR` / `TEXT` / `STRING` | `Utf8` |
| `BYTEA` / `BINARY` | `Binary` |
| `DATE` | `Date32` |
| `TIME` | `Time64` |
| `TIMESTAMP` | `Timestamp`, with or without a timezone |
| `INTERVAL` | `Interval` |
| `STRUCT` / `LIST` / `MAP` | the nested equivalents |
DataFusion's [type reference](https://datafusion.apache.org/user-guide/sql/data_types.html)
describes the full mapping.
## Types SQL has no name for {#no-name}
Three column kinds you will meet that no SQL type name covers.
### Vectors {#vectors}
A vector column is `FixedSizeList<Float32, dim>`. It is selectable, testable for
nullness, and joinable like any other column:
```sql
SELECT id FROM documents WHERE vector IS NOT NULL
```
<Warning>
Arithmetic over it is **list** arithmetic. There is no SQL similarity operator
and no SQL form for vector search — run the search in an SDK and use SQL over
its result. See [SDKs and SQL](/basics/apis#no-vector-sql).
</Warning>
### JSON {#json}
Stored as JSONB in `large_binary` with the `lance.json` extension name. It reads
through accessor functions rather than a `->` operator:
```sql
SELECT json_get_string(data, 'name') FROM events
WHERE json_get_int(data, 'status') = 200
```
`json_get`, its typed variants, and `json_extract` are **index-accelerated** by a
JSON scalar index. See [Functions](/sql/functions#json).
### Blobs {#blobs}
A blob column is read either as its binary contents or as a **description** — a
pointer plus metadata, without fetching the bytes. That choice changes the
column's Arrow type, and therefore the schema your query sees.
<Warning>
The default reads blob columns as descriptions and other binary columns as
binary. A query selecting a blob column and expecting bytes gets a struct.
</Warning>
## Pseudo-columns {#pseudo-columns}
Row metadata, available only when named — so it stays out of `SELECT *`:
| Column | Type | |
| --- | --- | --- |
| `_rowid` | `UInt64` | stable row identifier |
| `_rowaddr` | `UInt64` | physical row address |
| `_rowoffset` | `UInt64` | offset within the fragment |
| `_score` | `Float32` | relevance, from [`fts()`](/sql/functions#fts) |
`_rowid` and `_rowaddr` are declared nullable, but not because a row lacks one —
their validity bitmap is reused internally as a selection vector. Do not read a
null there as meaningful.
`_rowid` is what [materialized views](/management/materialized-view) depend on,
which is why a source table needs stable row ids enabled at creation.
## Casting {#casting}
```sql
SELECT CAST(identifier AS VARCHAR) FROM documents
SELECT TRY_CAST(maybe_number AS BIGINT) FROM raw
```
`TRY_CAST` yields null rather than failing the query on a value that will not
convert. See [Operators](/sql/operators#casting).
docs/web/sql/ddl.mdx
+192
View File
@@ -0,0 +1,192 @@
---
title: "DDL"
sidebarTitle: "DDL"
description: "Creating and altering catalog objects: tables, views, indexes, functions, and columns."
icon: pen-ruler
keywords: ["ddl", "create table", "alter table", "add column", "create function", "create index", "wal"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
Statements that change what exists rather than what it holds. For the values in
a table, see [DML](/sql/dml).
## Tables {#tables}
```sql
CREATE TABLE documents (id BIGINT, title VARCHAR, created_at TIMESTAMP)
CREATE TABLE recent AS SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
DROP TABLE documents
```
Column types are Arrow types under SQL names -- see
[Data types](/sql/data-types). A table in a [namespace](/namespaces) is written
as a dotted path: `prod.events.clicks`.
## Views {#views}
```sql
CREATE VIEW recent_matches AS
SELECT t1.floats FROM documents AS t1
INNER JOIN categories AS t2 ON t1.ints = t2.ints
DROP VIEW recent_matches
```
`CREATE VIEW` returns the view's schema -- column name, type, nullability --
resolved from the query at creation time. A view stores only the query; see
[Views](/management/view).
## Columns {#columns}
`ALTER TABLE ... ADD COLUMN` covers three different things, distinguished by
what follows the column name.
### A plain column {#plain-column}
```sql
ALTER TABLE t ADD COLUMN c INT
```
Added empty and nullable. Nothing is computed.
### A computed column {#computed-column}
```sql
ALTER TABLE t ADD COLUMN doubled AS value * 2
```
`AS <expression>` declares a [computed column](/feature-engineering/add-computed-column).
The column is committed empty and the expression is stored, not evaluated --
declaring one on a billion-row table costs what declaring one on an empty table
costs. [`REFRESH COLUMN`](/sql/refresh) fills it.
### A function column {#function-column}
```sql
ALTER TABLE t ADD COLUMN embedded AS embed(name) WITH (version = 'fv_1')
```
When the expression calls a [registered function](#functions), the column is
filled by running that function on workers. `version` pins which immutable
function version fills it, so a later registration does not silently change what
the column means.
Several modifiers narrow what gets computed:
| | |
| --- | --- |
| `FILTER (WHERE pred)` | only rows matching `pred` are computed |
| `IGNORE NULLS` | rows with null inputs are skipped |
| `OVER (PARTITION BY col)` | the function sees rows grouped by `col` |
A function returning several values fills several columns at once:
```sql
ALTER TABLE t ADD COLUMN (label, score) AS classify(text) WITH (version = 'fv_1')
```
```sql
ALTER TABLE t ADD COLUMN c AS f(x) FILTER (WHERE keep) WITH (version = 'fv_1')
ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2')
```
<Note>
`WITH` on a plain computed column accepts only `version`. Resource options such
as `num_workers` apply to function columns, where there are workers to size.
</Note>
## Functions {#functions}
```sql
CREATE FUNCTION f(x BIGINT) RETURNS BIGINT
LANGUAGE python
AS 'def f(x): return x * 2'
WITH (pip = 'numpy==1.26.4', python_version = '3.11')
```
The body is Python source. Its dependencies are declared with it and are part of
the **immutable function version** -- changing one produces a new version rather
than altering the registered one, so a column filled by an earlier version is
unaffected.
| Option | |
| --- | --- |
| `pip` | pip requirements for the worker environment |
| `conda` | conda packages instead of pip |
| `conda_channels` | channels in priority order; requires `conda` |
| `python_version` | worker Python version, defaulting to yours |
| `gpu` | `'true'` when every execution needs a GPU |
**Errors and limits**
| | |
| --- | --- |
| `LANGUAGE` other than `python` | refused |
| `pip` and `conda` together | refused -- pick one |
| `conda_channels` without `conda` | refused |
| `gpu` set to anything but `'true'` | refused |
| An empty body | refused |
| Any other option | refused, and the message lists what is supported |
See [Functions](/management/function) for the SDK equivalent and for diagnosing
environment mismatches.
## Indexes {#indexes}
```sql
CREATE INDEX ON documents (title)
DROP INDEX documents.title_idx
```
<Note>
SQL `CREATE INDEX` covers **one column** and takes **no user-chosen name** --
the server names it. For a multi-column or named index, use the SDK. `DROP
INDEX` takes the qualified `table.index` name; find it with `list_indices`.
</Note>
Index type and parameters are chosen in the SDK; see [Indexing](/indexing).
## Table properties {#properties}
```sql
ALTER TABLE t SET UNENFORCED PRIMARY KEY (id)
```
Records which column identifies a row. **Unenforced** is not a hedge in the
naming: uniqueness is not checked on write, and declaring it does not make
duplicates impossible. It tells the planner what to assume.
```sql
ALTER TABLE t SET WAL
ALTER TABLE t SET WAL WITH (sharding = 'bucket', buckets = 4)
ALTER TABLE t SET WAL WITH (maintained_indexes = '')
ALTER TABLE t UNSET WAL
```
A write-ahead log makes small frequent writes cheap by batching them out of the
commit path. `sharding` and `buckets` spread the log so concurrent writers do
not contend on one place; `maintained_indexes` chooses which indexes are kept
current against WAL rows.
<Warning>
[`TABLESAMPLE`](/sql/sampling) is refused on a WAL-backed table, because rows a
sample would draw from are not all in the fragments yet.
</Warning>
## See also {#see-also}
<CardGroup cols={2}>
<Card title="DML" icon="pen-to-square" href="/sql/dml">
Changing rows rather than structure.
</Card>
<Card title="Refresh" icon="arrows-rotate" href="/sql/refresh">
Filling the columns declared here.
</Card>
<Card title="Management" icon="folder-tree" href="/management">
The same operations from the SDKs.
</Card>
<Card title="SHOW" icon="list" href="/sql/show">
Listing what exists.
</Card>
</CardGroup>
docs/web/sql/dml.mdx
+81
View File
@@ -0,0 +1,81 @@
---
title: "DML"
sidebarTitle: "DML"
description: "INSERT, UPDATE, DELETE and COPY — changing the rows a table holds."
icon: pen-to-square
keywords: ["insert", "update", "delete", "copy", "dml"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
Statements that change rows. For the shape of the table, see [DDL](/sql/ddl).
Each of these commits a new table version, so
[versioning](/tables/versioning) and the
[write guarantees](/basics/guarantees#writes) apply -- including that the
version advances even when a statement affects zero rows.
## INSERT {#insert}
```sql
INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur')
INSERT INTO archive SELECT * FROM documents WHERE created_at < '2026-01-01'
```
`INSERT ... SELECT` is the form worth knowing: it moves rows between tables
server-side, without a round trip through your process.
## UPDATE {#update}
```sql
UPDATE documents SET title = upper(title) WHERE category = 'film'
```
Assignments take any expression the [dialect](/sql/functions) supports,
including references to other columns of the same row. Without a `WHERE` clause
every row is updated.
<Warning>
An update does not recompute [computed columns](/feature-engineering/add-computed-column)
that derive from the column you changed. A filled value keeps what it was
computed from -- see [Guarantees](/basics/guarantees#jobs).
</Warning>
## DELETE {#delete}
```sql
DELETE FROM documents WHERE created_at < '2025-01-01'
```
Deletion marks rows rather than erasing them, so the data remains recoverable
for as long as the version holding it survives
[retention](/maintenance/compaction).
## COPY {#copy}
```sql
COPY (SELECT id, title FROM documents) TO 's3://bucket/export/' STORED AS PARQUET
```
Writes a query result to object storage. This is the export path when the
consumer is another engine rather than an application -- results land as files
rather than streaming to a client.
## Merge insert {#merge-insert}
Upsert -- matching incoming rows against existing keys -- has no SQL form. It is
an SDK operation, because it needs a source relation and a per-match policy that
a single statement cannot express. See
[Merge insert](/ingestion/merge-insert).
## See also {#see-also}
<CardGroup cols={2}>
<Card title="Ingestion" icon="inbox" href="/ingestion">
The same operations from the SDKs.
</Card>
<Card title="DDL" icon="pen-ruler" href="/sql/ddl">
Changing structure rather than rows.
</Card>
</CardGroup>
docs/web/sql/explain.mdx
+51
View File
@@ -0,0 +1,51 @@
---
title: "EXPLAIN"
sidebarTitle: "Explain"
description: "Reading the plan for a statement, and measuring it."
icon: map
keywords: ["explain", "explain analyze", "query plan", "describe"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
```sql
EXPLAIN SELECT * FROM documents WHERE identifier < 1000
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
```
`EXPLAIN` resolves the plan without running the query. `EXPLAIN ANALYZE`
executes it and annotates every operator with runtime metrics — elapsed time,
rows produced, bytes read, index comparisons.
The operators are the same ones the SDK methods produce, so the reasoning about
index coverage and filter pushdown carries across unchanged. The two pages that
explain how to read them:
<CardGroup cols={2}>
<Card title="Explain" icon="map" href="/search/explain">
Reading a plan: operators, pushdown, index usage.
</Card>
<Card title="Explain Analyze" icon="gauge" href="/search/explain-analyze">
Reading the metrics, and a worked before/after.
</Card>
</CardGroup>
## What to look for {#what-to-look-for}
| Sign | Means |
| --- | --- |
| `ScalarIndexQuery` | the predicate used a scalar index |
| `LanceScan` with high `bytes_read` | no index, or too many columns selected |
| `FilterExec` above a search | the filter ran after search rather than pushing into it |
| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used |
A `LanceScan` feeding a join is the common one worth acting on — see
[Joins](/sql/join#performance).
## DESCRIBE {#describe}
```sql
DESCRIBE documents
```
The table's columns and types, without a plan.
docs/web/sql/index.mdx
+64 -42
View File
@@ -1,63 +1,85 @@
---
title: "SQL"
title: "SQL reference"
sidebarTitle: "Overview"
description: "Analytical SQL over the lakehouse: the dialect, where it runs, and what it can reach."
description: "The SQL surface: dialect, where a query runs, and a page per construct."
icon: table-list
keywords: ["sql", "datafusion", "flight sql", "analytics", "dialect"]
keywords: ["sql", "datafusion", "flight sql", "reference", "dialect"]
---
<Badge color="red">Enterprise</Badge> SQL runs on a deployment. A local database
has no engine to plan a statement, so `execute_query` raises there -- see
[SDKs and SQL](/basics/apis).
The dialect is [Apache DataFusion's](/sql/functions), so anything its reference
documents for that version works here unchanged. What these pages cover is the
parts that are LanceDB's: how a query reaches the engine, how Lance columns
appear to it, and the extensions a lakehouse over multimodal data needs.
This is a reference rather than a guide: one page per construct, syntax first.
The guides elsewhere in these docs show the same operations from the SDKs.
## The dialect {#dialect}
LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org),
currently **version 54**. The grammar, built-in functions and type system are
DataFusion's, so anything its reference documents for that version works here
unchanged and is not reproduced.
What these pages add is the part that is LanceDB's: how a query reaches the
engine, how Lance columns appear to it, and the statements a lakehouse needs
that DataFusion has no concept of.
## Where a query runs {#execution}
Queries execute on the server. The client sends a statement and receives Arrow
record batches -- nothing is converted to pandas or Python objects unless you
ask for it. A query feeding training or downstream processing therefore pays no
serialization cost to get there.
Queries execute on the server and results come back as Arrow record batches --
nothing is converted to pandas or Python objects unless you ask. `execute_query`
hands back a reader as soon as the first batch is ready rather than when the
query finishes, so a large result set does not have to fit in memory before you
start reading it. See [Query and job](/basics/jobs#queries).
Results stream. `execute_query` hands back a reader as soon as the first batch
is ready rather than when the query finishes, so a large result set does not
have to fit in memory before you can start reading it. See
[Query and job](/basics/jobs#queries).
Served over an [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint, so any Flight SQL client reaches it.
## What differs from a warehouse {#differences}
## Reference {#reference}
Two things, and both come from the storage being Lance rather than something
SQL-shaped:
- **Columns are Arrow types**, not a SQL type system mapped onto storage. A
vector column is a `FixedSizeList<Float32>` and behaves like one. See
[Data types](/basics/data-types).
- **Row metadata is available as columns** -- `_rowid`, `_rowaddr`,
`_rowoffset` -- but only when named, so they stay out of `SELECT *`.
## The statements {#statements}
**Reading data**
| | |
| --- | --- |
| [Data types](/sql/data-types) | how Lance columns appear to SQL, and pseudo-columns |
| [SELECT](/sql/select) | projection, filtering, ordering, set operations, CTEs |
| [Aggregation](/sql/aggregation) | `GROUP BY`, `HAVING`, aggregate functions |
| [Joins](/sql/join) | inner, outer, cross, and joining across namespaces |
| [Window functions](/sql/window) | `OVER (...)`, ranking, running totals |
| [Sampling](/sql/sampling) | `TABLESAMPLE`, for a random subset |
| [Functions and types](/sql/functions) | the function set, and what LanceDB adds |
| [Subqueries](/sql/subqueries) | scalar, `IN`, `EXISTS`, correlated, derived tables |
| [Joins](/sql/join) | inner, outer, cross, and namespace-qualified paths |
| [Aggregation](/sql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set |
| [Window functions](/sql/window) | `OVER`, framing, ranking and offset functions |
| [Sampling](/sql/sampling) | `TABLESAMPLE`, by rows or by fragments |
Beyond reading, SQL also writes (`INSERT`, `UPDATE`, `DELETE`, `COPY`), manages
the catalog (`CREATE TABLE`, `CREATE VIEW`, `DROP`, `SHOW`) and drives feature
engineering (`CREATE FUNCTION`, `REFRESH COLUMN`, `REFRESH MATERIALIZED VIEW`).
Those are documented with the operation rather than here -- see
[Management](/management) and [Feature engineering](/feature-engineering).
**Changing things**
<Note>
Vector search has no SQL form. It is a query-builder operation -- see
[Vector search](/search/vector-search). Full-text search does: the
[`fts()`](/sql/functions#fts) table function and the
[`contains_tokens`](/sql/functions#contains-tokens) predicate.
</Note>
| | |
| --- | --- |
| [DDL](/sql/ddl) | tables, views, indexes, functions, columns, WAL |
| [DML](/sql/dml) | `INSERT`, `UPDATE`, `DELETE`, `COPY` |
| [REFRESH](/sql/refresh) | filling computed columns and materialized views |
**Introspection**
| | |
| --- | --- |
| [SHOW](/sql/show) | jobs, functions, materialized views, errors |
| [EXPLAIN](/sql/explain) | plans, and measuring them |
**Expressions**
| | |
| --- | --- |
| [Operators](/sql/operators) | comparison, logical, arithmetic, literals, precedence |
| [Functions](/sql/functions) | the function set, and what Lance accelerates |
## What SQL cannot do {#cannot}
| | |
| --- | --- |
| Vector search | no SQL form -- a query-builder operation |
| Merge insert | no SQL form -- needs a source relation and a per-match policy |
| Run on a local database | no engine to plan the statement |
Full-text search **is** available, as the [`fts()`](/sql/functions#fts) table
function and the [`contains_tokens`](/sql/functions#contains-tokens) predicate.
See [Limits](/basics/limits#sql) for the bounds these pages enforce.
docs/web/sql/operators.mdx
+81
View File
@@ -0,0 +1,81 @@
---
title: "Operators and literals"
sidebarTitle: "Operators"
description: "Comparison, logical and arithmetic operators, literals, and precedence."
icon: plus-minus
keywords: ["operators", "literals", "precedence", "null", "cast", "interval"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
The operator set is DataFusion's. This page covers what is worth knowing when
the operands are Lance columns.
## Comparison {#comparison}
| | |
| --- | --- |
| `=` `!=` `<>` `<` `<=` `>` `>=` | ordinary comparison |
| `IS NULL` / `IS NOT NULL` | null test — `= NULL` is never true |
| `IS DISTINCT FROM` | comparison that treats null as a value |
| `BETWEEN a AND b` | inclusive range |
| `IN (...)` | set membership |
| `LIKE` / `ILIKE` | pattern match, `%` and `_` |
| `~` `~*` `!~` | regular expression match |
<Note>
For keyword matching on a text column, [`contains_tokens`](/sql/functions#contains-tokens)
is index-accelerated where `LIKE` is not. Reach for `LIKE` only when you need
its wildcard semantics.
</Note>
## Logical {#logical}
`AND`, `OR`, `NOT`. Three-valued: a comparison against null is **unknown**, and
`NOT unknown` is still unknown. A `WHERE` clause keeps only rows where the
predicate is true, so unknown behaves like false there — but not inside `NOT
IN`, which is the usual place this bites. See
[Subqueries](/sql/subqueries#in-exists).
## Arithmetic {#arithmetic}
`+` `-` `*` `/` `%`, with the usual precedence. String concatenation is `||`.
<Warning>
Arithmetic over a vector column is **list** arithmetic, not scalar arithmetic —
a vector is a `FixedSizeList<Float32>`. When the operation you want is a
similarity comparison, use [vector search](/search/vector-search); there is no
SQL form for it.
</Warning>
## Literals {#literals}
| | |
| --- | --- |
| String | `'text'` — doubled quote to escape: `'it''s'` |
| Number | `42`, `3.14` |
| Boolean | `TRUE`, `FALSE` |
| Null | `NULL` |
| Date / timestamp | `DATE '2026-01-01'`, `TIMESTAMP '2026-01-01 12:00:00'` |
| Interval | `INTERVAL '7 days'` |
| Array | `[1, 2, 3]` |
Identifiers are quoted with double quotes when they need to be — a reserved word
or a name with mixed case: `SELECT "order" FROM "MyTable"`.
## Casting {#casting}
```sql
SELECT CAST(identifier AS VARCHAR), identifier::VARCHAR FROM documents
```
Both forms work. `TRY_CAST` returns null instead of erroring on a value that
will not convert, which is the one to use over a column you do not control.
## Precedence {#precedence}
Highest to lowest: `::` → unary `-` → `* / %` → `+ -` → `||` → comparison →
`IS` → `NOT` → `AND` → `OR`.
Parenthesize anything mixing `AND` and `OR`. `a OR b AND c` is `a OR (b AND c)`,
which is rarely what was meant.
docs/web/sql/refresh.mdx
+95
View File
@@ -0,0 +1,95 @@
---
title: "REFRESH"
sidebarTitle: "Refresh"
description: "Filling computed columns and bringing materialized views up to date."
icon: arrows-rotate
keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
LanceDB extensions with no DataFusion equivalent. They are what makes
[feature engineering](/feature-engineering) reachable from SQL: a column
declared by [DDL](/sql/ddl#computed-column) starts empty, and a refresh fills it.
## REFRESH COLUMN {#refresh-column}
```sql
REFRESH COLUMN <column> ON [TABLE] <table>
[WHERE <predicate>]
[WITH (<option> = <value> [, ...])]
```
```sql
REFRESH COLUMN doubled ON my_table
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
```
A refresh gives values to rows that lack them and **leaves filled rows alone**.
That makes it idempotent, cheap to repeat as data arrives, and resumable after a
failure -- re-running costs only the rows that did not complete.
`WHERE` scopes the refresh, which is how you fill a slice and check it before
committing to the whole table.
### Options {#refresh-column-options}
| Option | Meaning |
| --- | --- |
| `num_workers` | how many workers to use |
| `cpu_millis` | CPU budget per worker |
| `memory_bytes` | memory budget per worker |
| `gpu_count` | GPUs per worker |
| `batch_size` | rows a worker takes at a time |
| `error_policy` | `'skip'` continues past failing rows instead of failing the job |
```sql
REFRESH COLUMN embedding ON documents
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
```
`num_workers` sets a count; the budget options size each worker. They are
alternatives rather than a pair.
<Note>
`error_policy` and `num_workers` are refused for a column filled by a registered
[function](/management/function) with its own resource budget -- use the CPU,
memory and GPU options there.
</Note>
## REFRESH MATERIALIZED VIEW {#refresh-materialized-view}
```sql
REFRESH MATERIALIZED VIEW <name> [WITH (<option> = <value>)]
```
```sql
REFRESH MATERIALIZED VIEW chunks
REFRESH MATERIALIZED VIEW chunks WITH (full = true)
```
By default a refresh is **incremental** -- it processes what the source has
added since the last one. `full = true` recomputes every row, which is what you
want after changing something the view's rows depend on.
<Warning>
`CASCADE` parses but is **not implemented**. A view built on another view is
refreshed by refreshing each in dependency order yourself.
</Warning>
## Watching a refresh {#watching}
Both statements submit work that outlives the call. Track it with
[`SHOW JOBS`](/sql/show#jobs), and diagnose failures with
[`SHOW ERRORS`](/sql/show#errors).
## See also {#see-also}
<CardGroup cols={2}>
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
The same operation from the SDKs, and what a refresh guarantees.
</Card>
<Card title="DDL" icon="pen-ruler" href="/sql/ddl#columns">
Declaring the columns this fills.
</Card>
</CardGroup>
docs/web/sql/show.mdx
+87
View File
@@ -0,0 +1,87 @@
---
title: "SHOW"
sidebarTitle: "Show"
description: "Listing jobs, functions, materialized views, and the errors a job produced."
icon: list
keywords: ["show jobs", "show functions", "show materialized views", "show errors", "show tables"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
Introspection. `SHOW TABLES` is DataFusion's; the rest are LanceDB extensions
covering objects DataFusion has no concept of.
## SHOW TABLES {#tables}
```sql
SHOW TABLES
```
Lists the tables the session can see.
## SHOW MATERIALIZED VIEWS {#materialized-views}
```sql
SHOW MATERIALIZED VIEWS
```
<Note>
Cheaper than the SDK equivalent. `list_materialized_views` finds views by
reading every table's schema, costing one open per table; this asks the server.
</Note>
## SHOW FUNCTIONS {#functions}
```sql
SHOW FUNCTIONS
```
Lists registered [functions](/management/function) and their versions. Since a
version is immutable and a column pins the version that fills it, this is how
you find which versions are still in use before dropping one.
## SHOW JOBS {#jobs}
```sql
SHOW JOBS
```
Every job the server knows about — id, type, state, creation time. The SQL
counterpart of `list_jobs`; see [Query and job](/basics/jobs).
## SHOW ERRORS {#errors}
```sql
SHOW ERRORS ON <table> [FOR JOB '<job-id>'] [LIMIT <n>]
```
```sql
SHOW ERRORS ON documents
SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100
```
Why rows failed. This is the statement to reach for when a
[refresh](/sql/refresh) reports fewer filled rows than you expected.
**Errors and limits**
| | |
| --- | --- |
| `LIMIT` | must be between 1 and 100,000 |
| Failed jobs scanned without `FOR JOB` | 32 most recent |
| Tables the server does not open directly | not supported |
Without `FOR JOB` this surveys recent failures and stops at 32 jobs. When the
listing stops at your `LIMIT`, the message says so — narrow with `FOR JOB`
rather than raising the limit blindly.
## See also {#see-also}
<CardGroup cols={2}>
<Card title="Management" icon="folder-tree" href="/management">
The SDK listing calls, per object.
</Card>
<Card title="Query and job" icon="hourglass-half" href="/basics/jobs">
What a job handle exposes beyond these listings.
</Card>
</CardGroup>
docs/web/sql/subqueries.mdx
+76
View File
@@ -0,0 +1,76 @@
---
title: "Subqueries"
sidebarTitle: "Subqueries"
description: "Queries inside queries: scalar, IN, EXISTS, correlated, and derived tables."
icon: code-branch
keywords: ["subquery", "exists", "in", "correlated", "derived table", "lateral"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
A subquery is allowed anywhere an expression or a relation is.
## As a relation {#derived}
```sql
SELECT * FROM (
SELECT category, count(*) AS n FROM documents GROUP BY category
) AS counts
WHERE n > 100
```
A derived table needs an alias. It can be [sampled](/sql/sampling) like a base
table:
```sql
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
```
## As a scalar {#scalar}
```sql
SELECT title, (SELECT count(*) FROM revisions r WHERE r.doc_id = d.id) AS revisions
FROM documents AS d
```
A scalar subquery must return one row and one column. Returning more is an
error rather than a silently-taken first row.
## IN and EXISTS {#in-exists}
```sql
SELECT title FROM documents
WHERE category_id IN (SELECT id FROM categories WHERE active)
SELECT title FROM documents AS d
WHERE EXISTS (SELECT 1 FROM revisions r WHERE r.doc_id = d.id)
```
`EXISTS` stops at the first matching row, so it is the cheaper test when you
only care whether anything matches. `IN` builds the full set.
`NOT IN` and `NOT EXISTS` differ on nulls: `NOT IN` against a set containing a
null yields no rows, because the comparison is unknown rather than false. Prefer
`NOT EXISTS` unless you have ruled nulls out.
## Correlated subqueries {#correlated}
A subquery referencing a column from the enclosing query — `d.id` above — is
correlated, and is conceptually evaluated per outer row. The planner usually
rewrites it into a join.
<Note>
Check with [`EXPLAIN`](/sql/explain) when a correlated subquery is slow. A plan
showing a join has been optimized; one showing a repeated scan has not, and
rewriting it as an explicit [join](/sql/join) is the fix.
</Note>
## CTEs as an alternative {#ctes}
A [CTE](/sql/select#ctes) is a named subquery, and is usually more readable when
the same subquery appears twice or when nesting gets deep.
```sql
WITH active AS (SELECT id FROM categories WHERE active)
SELECT title FROM documents WHERE category_id IN (SELECT id FROM active)
```