diff --git a/docs/web/architecture.mdx b/docs/web/architecture.mdx
new file mode 100644
index 000000000..40b305786
--- /dev/null
+++ b/docs/web/architecture.mdx
@@ -0,0 +1,150 @@
+---
+title: "Architecture"
+sidebarTitle: "Architecture"
+description: "How LanceDB is put together, and what changes between running it embedded and running it distributed."
+icon: cubes
+keywords: ["architecture", "embedded", "distributed", "object storage", "compute", "deployment"]
+---
+
+LanceDB has one storage layer and two ways to put compute on top of it. Which
+one you run decides where queries execute, what runs on your behalf, and what
+you have to operate — and almost nothing else. The data is the same
+[Lance](/basics/lance) tables either way.
+
+## One storage layer {#storage}
+
+Everything durable lives in object storage: table data, manifests, indexes. No
+machine owns a dataset, and no server has to be running for the data to exist.
+
+```
+ ┌──────────────────────────────────────────┐
+ │ object storage │
+ │ table data · manifests · indexes │
+ └──────────────────────────────────────────┘
+```
+
+That is the whole persistence story, and it is why the two deployment shapes can
+differ so much above it while staying compatible below it. A table written
+embedded is opened by a deployment, and the reverse, without conversion.
+
+## Embedded {#embedded}
+
+```
+ your process
+ ┌────────────────────────────────┐
+ │ application │
+ │ LanceDB library │ ← plans, executes, indexes, compacts
+ └────────────────────────────────┘
+ │
+ ┌─────────▼─────────┐
+ │ object storage │
+ └───────────────────┘
+```
+
+`lancedb.connect("./data")` or `connect("s3://bucket/path")` gives you a
+**library**, not a client. Your process opens the tables, plans the queries,
+runs the scans, builds the indexes, and writes the new versions. There is no
+server in the picture at all.
+
+**What follows from that.** Nothing runs unless you call it — no background
+compaction, no server-side [jobs](/basics/jobs), no
+[SQL endpoint](/basics/apis). Scale is your process's scale: a query gets the
+cores and memory your process has. Access control is whatever your object store
+enforces, since there is no other gatekeeper.
+
+**What it is good at.** Development, notebooks, single-node pipelines, embedding
+LanceDB inside an application, and anything where "no service to operate" is
+worth more than elasticity.
+
+## Distributed {#distributed}
+
+Enterprise
+
+```
+ your process
+ ┌──────────────┐
+ │ application │
+ │ LanceDB SDK │ ← a thin client
+ └──────┬───────┘
+ │ db:// (REST / Flight SQL)
+ ┌──────▼──────────────────────────────────────┐
+ │ control plane config · identity · policy │
+ ├─────────────────────────────────────────────┤
+ │ data plane query serving · writes │
+ │ indexers indexing · compaction │
+ └──────┬──────────────────────────────────────┘
+ │
+ ┌──────▼────────────┐
+ │ object storage │
+ └───────────────────┘
+```
+
+`connect("db://...")` gives you a **client**. Planning and execution move to a
+cluster, and the work splits across tiers that scale independently: a control
+plane governing configuration, identity and policy; a data plane serving reads
+and writes; and indexers building indexes and compacting outside the request
+path.
+
+**What follows from that.** Compute and storage are separated, so a query fleet
+scales for interactive traffic without also scaling indexing capacity, and heavy
+background work does not steal resources from user queries. Because a server
+exists, so do the things that need one: [SQL](/sql),
+[functions](/management/function), server-side jobs, and
+[authorization](/governance/authorization) per object rather than per bucket.
+
+**What it is good at.** Many concurrent queries, continuous ingestion and
+indexing, multi-team access with real permissions, and workloads too large for
+one machine.
+
+## What changes, precisely {#differences}
+
+| | Embedded | Distributed |
+| --- | --- | --- |
+| What you connect to | a library in your process | a cluster |
+| Where queries plan and execute | your process | the data plane |
+| Table handle | `LanceTable` | `RemoteTable` |
+| Indexing and compaction | when you call it | dedicated indexers |
+| [Jobs](/basics/jobs) | run in-process, no server record | real jobs: id, status, events, cancel |
+| [SQL](/sql) | unavailable | Flight SQL endpoint |
+| [Functions](/management/function) | unavailable | run on workers |
+| Access control | object-store permissions | [per-object grants](/governance/authorization) |
+| Scale bound | your process | the cluster |
+| What you operate | nothing | a deployment |
+
+Both expose the same table API — create, open, mutate, query — so application
+code moves between them by changing the URI. What does not move is code that
+depends on a row in the right-hand column.
+
+## Choosing, and changing your mind {#choosing}
+
+Start embedded. It has no operational cost and the API is the same one you will
+keep using.
+
+Move when you hit something in the right-hand column — usually concurrent
+serving, ingestion that outgrows one machine, or a permissions requirement.
+Because the storage layer is identical, moving is a connection-string change
+plus a data migration, not a rewrite.
+
+
+The one thing to decide early is **stable row ids**, which cannot be enabled
+after a table exists and are required for
+[materialized views](/management/materialized-view). See
+[Limits](/basics/limits#schema).
+
+
+## Where to go next {#next}
+
+
+
+ What a write promises, and what a reader sees.
+
+
+ The open format underneath both shapes.
+
+
+ The distributed tiers in detail.
+
+
+ The bounds you can actually hit.
+
+
diff --git a/docs/web/basics/guarantees.mdx b/docs/web/basics/guarantees.mdx
new file mode 100644
index 000000000..c5e2aeea5
--- /dev/null
+++ b/docs/web/basics/guarantees.mdx
@@ -0,0 +1,130 @@
+---
+title: "Guarantees"
+sidebarTitle: "Guarantees"
+description: "What a write promises when it returns, what a reader sees, and what survives a failure."
+icon: shield-check
+keywords: ["guarantees", "consistency", "atomicity", "durability", "concurrency", "acid"]
+---
+
+What the system promises, and — just as usefully — what it does not. Most
+surprises in practice come from assuming one of the guarantees below is
+stronger or weaker than it is.
+
+## Writes {#writes}
+
+**A write is atomic.** Every write commits a new table version. Either the
+version exists and holds all of the change, or it does not exist and holds none
+of it. There is no state in which a reader sees half an `add()`.
+
+**A write returns only after the commit.** When the call returns, the version
+is durable in object storage. Nothing is buffered in your process waiting to be
+flushed.
+
+**Every write advances the version**, including one that affects zero rows. A
+delete whose predicate matches nothing still commits. Version numbers count
+commits, not changes.
+
+**Concurrent writes are resolved optimistically.** A transaction records the
+change and the version it was based on. At commit time, a conflicting
+concurrent operation is detected rather than silently overwritten, and the
+commit is retried against the newer version. Operations that do not collide —
+appending while another process appends — both succeed.
+
+
+`merge_insert` reports its retry attempts in the result. A high count means
+contention on the same keys, not an error.
+
+
+## Reads {#reads}
+
+**A read sees a consistent snapshot.** A query reads one table version
+throughout; it never sees a mixture of two.
+
+**Which version, by default, is the one open at the time the table was opened.**
+This is the guarantee most often assumed to be stronger than it is:
+
+
+A long-lived connection **does not observe other processes' writes** until it
+reopens the table or its `read_consistency_interval` elapses. The default is
+right for a batch job reading a stable snapshot and wrong for a service
+expected to see fresh data.
+
+
+Set `read_consistency_interval` when several processes write the same table and
+readers must observe those writes. The cost is a metadata check on that
+interval. See [Consistency](/tables/consistency).
+
+**Your own writes are always visible to you.** The interval governs other
+writers, not yourself.
+
+## Durability and history {#durability}
+
+**Old versions are retained, not overwritten.** A new version does not destroy
+the one before it, which is what makes [time travel](/tables/versioning) work
+and what lets a reader finish a scan that started on an older version.
+
+**Deletion is a mark, not an erase.** A deleted row is recorded in a deletion
+file. The data is recoverable for as long as the version that contains it
+exists.
+
+
+Retention is finite. [Compaction](/maintenance/compaction) prunes versions
+older than the retention window — **7 days by default** — and a tag or branch
+pointing at a pruned version no longer resolves. If you rely on reading old
+versions, widen the window before running it, not after.
+
+
+**Dropping a table ends its history.** Versioning does not survive
+`drop_table`; there is no undrop.
+
+## Jobs {#jobs}
+
+**A refresh never revisits a filled row.** It gives values to rows that lack
+them and leaves the rest alone. Two consequences, both relied on elsewhere:
+
+- **Retrying is safe and cheap.** A failed backfill is resumed, not restarted —
+ re-running costs only the rows that did not complete.
+- **A filled value does not track its input.** If a column derives from `price`
+ and a row's `price` later changes, the derived value stays as computed.
+ Recomputing means dropping the column and declaring it again.
+
+**A failure is not a rollback.** Rows filled before a failure keep their values,
+because each commits as it completes.
+
+**Cancellation is a request.** `cancel()` asks a job to stop; it does not
+guarantee it stops before doing more work, and cancelling a finished job does
+nothing.
+
+**A job id is opaque.** It correlates with server logs and the jobs API.
+Parsing it, or storing it to resume the job later, is not supported.
+
+See [Query and job](/basics/jobs).
+
+## Schema {#schema}
+
+**Schema changes are versioned like data changes.** Adding, renaming, retyping
+or dropping a column commits a version, so a schema change can be travelled back
+through.
+
+**A computed column's inputs are protected.** While a declaration references a
+column, that column cannot be renamed, retyped, or dropped — the declaration is
+part of the schema and will not be invalidated out from under itself.
+
+**Most schema changes do not rewrite data.** Adding a derived column writes only
+the new column. Type changes are the exception and can be substantial.
+
+## What is not guaranteed {#not-guaranteed}
+
+Worth stating plainly, because each is occasionally assumed:
+
+| | |
+| --- | --- |
+| Multi-table transactions | A commit covers one table. There is no cross-table atomic write. |
+| Read-your-writes across processes | Governed by `read_consistency_interval`, not automatic. |
+| Unenforced primary keys | `SET UNENFORCED PRIMARY KEY` records intent; it does not enforce uniqueness. |
+| Ordering of concurrent appends | Two appends both succeed; their relative row order is not defined. |
+| Index freshness | An index does not cover rows added after it was built until [optimized](/maintenance/compaction). |
+| Automatic compaction | Nothing runs in the background. `optimize()` runs when you call it. |
+
+The last two are the most common source of "why is this query slow" — see
+[Limits](/basics/limits) for the bounds that pair with them.
diff --git a/docs/web/basics/limits.mdx b/docs/web/basics/limits.mdx
new file mode 100644
index 000000000..6727dc94b
--- /dev/null
+++ b/docs/web/basics/limits.mdx
@@ -0,0 +1,103 @@
+---
+title: "Limits"
+sidebarTitle: "Limits"
+description: "The bounds you can actually hit, in one place."
+icon: ruler
+keywords: ["limits", "maximum", "quota", "constraints", "bounds"]
+---
+
+Limits worth designing around, collected so you do not have to find them by
+hitting them. Each one is enforced -- where a value is refused rather than
+clamped, that is stated, because the two fail very differently.
+
+This page covers bounds that are the same for everyone. Deployment-specific
+quotas -- storage, throughput, concurrency -- come from your deployment.
+
+## Names {#names}
+
+| Object | Maximum | |
+| --- | --- | --- |
+| Database | 64 characters | |
+| Namespace | 64 characters | |
+| Table | 128 characters | |
+| Branch | 64 characters | |
+
+A name must not be empty, and must not contain `$` or `:` -- both are reserved
+as path delimiters, so a name holding one could not be addressed unambiguously.
+Length is counted in characters, not bytes.
+
+## Jobs {#jobs}
+
+| | |
+| --- | --- |
+| `job.events()` default | 1,000 rows |
+| `job.events()` maximum | 10,000 rows |
+| `SHOW ERRORS ... LIMIT` | between 1 and 100,000 |
+| Failed jobs scanned by one `SHOW ERRORS` | 32 |
+
+
+`job.events()` **truncates silently**. A job that emits an event per fragment
+will lose its tail unless you pass `limit` explicitly. This is the limit on this
+page most likely to mislead you, because nothing in the result says it was cut.
+
+
+`SHOW ERRORS` without `FOR JOB` scans the most recent failed jobs and stops at
+32 of them. Narrow with `FOR JOB` when you need a specific one rather than a
+survey.
+
+See [Query and job](/basics/jobs).
+
+## Schema and types {#schema}
+
+| | |
+| --- | --- |
+| `map` with `keys_sorted = true` | rejected |
+| `IVF_RQ` vector dimensions | must be divisible by 8 |
+| FTS `max_token_length` | 40 by default; longer tokens are dropped |
+
+Stable row ids **cannot be enabled after a table exists**. A table created
+without them cannot become a
+[materialized view](/management/materialized-view) source without being
+recreated -- the one schema decision here that is genuinely irreversible.
+
+See [Data types](/basics/data-types).
+
+## SQL {#sql}
+
+Enterprise
+
+| | |
+| --- | --- |
+| `TABLESAMPLE` percentage | 0 to 100, **refused** outside -- not clamped |
+| `TABLESAMPLE (n ROWS)` | whole numbers only |
+| `TABLESAMPLE BUCKET` | not supported |
+| `TABLESAMPLE` on a WAL-backed table | not supported |
+| `CREATE INDEX` | one column, server-chosen name |
+| `REFRESH MATERIALIZED VIEW ... CASCADE` | parsed, not yet implemented |
+
+A sampling percentage over 100 is an error rather than a full scan, because a
+query asking for 150 percent has a mistake in it and reading it as 100 would
+hide the mistake. See [Sampling](/sql/sampling).
+
+## What is unavailable where {#availability}
+
+Not quotas, but the same class of surprise -- things that exist in one place and
+not another.
+
+| | |
+| --- | --- |
+| SQL on a local database | unavailable; `execute_query` raises |
+| SQL methods in the TypeScript SDK | none; use a Flight SQL client |
+| Function methods in the TypeScript SDK | none; use Python or SQL |
+| Vector search in SQL | no SQL form; use a query builder |
+| Geospatial `st_*` functions | behind a build feature LanceDB does not ship |
+| `tableExists` / `namespaceExists` in TypeScript | not present |
+| Server-side jobs on a local database | run in-process; `events()` raises |
+
+See [SDKs and SQL](/basics/apis) for why the first two hold.
+
+## Nothing here is a quota {#not-quotas}
+
+None of the above is a billing or capacity limit. Rows per table, table count,
+storage, and request rate are bounded by your deployment rather than by the
+format -- ask your deployment, not this page.
diff --git a/docs/web/docs.json b/docs/web/docs.json
index 45b193a41..f2bd300fd 100644
--- a/docs/web/docs.json
+++ b/docs/web/docs.json
@@ -98,26 +98,29 @@
"tab": "Documentation",
"groups": [
{
- "group": "Introduction",
+ "group": "Get started",
"pages": [
"index",
- "quickstart"
+ "quickstart",
+ "architecture"
]
},
{
- "group": "Basics",
+ "group": "Concepts",
"pages": [
"basics/lance",
"basics/catalog",
"basics/jobs",
"basics/data-types",
"tables/consistency",
+ "basics/guarantees",
+ "basics/limits",
"basics/apis",
"basics/agentic-skills"
]
},
{
- "group": "Management",
+ "group": "Manage",
"pages": [
"management/index",
"management/database",
@@ -142,7 +145,7 @@
]
},
{
- "group": "Ingestion",
+ "group": "Ingest",
"pages": [
"ingestion/index",
"ingestion/insert",
@@ -172,20 +175,18 @@
"reranking/eval"
]
},
- "search/explain",
- "search/explain-analyze",
{
- "group": "SQL",
+ "group": "Indexing",
"pages": [
- "sql/index",
- "sql/select",
- "sql/aggregation",
- "sql/join",
- "sql/window",
- "sql/sampling",
- "sql/functions"
+ "indexing/index",
+ "indexing/vector-index",
+ "indexing/fts-index",
+ "indexing/scalar-index",
+ "indexing/quantization"
]
- }
+ },
+ "search/explain",
+ "search/explain-analyze"
]
},
{
@@ -199,7 +200,7 @@
]
},
{
- "group": "Training",
+ "group": "Train",
"pages": [
"training/why-lancedb",
"training/index",
@@ -207,13 +208,8 @@
]
},
{
- "group": "Maintenance",
+ "group": "Operate",
"pages": [
- "indexing/index",
- "indexing/vector-index",
- "indexing/fts-index",
- "indexing/scalar-index",
- "indexing/quantization",
"indexing/reindexing",
"maintenance/compaction",
{
@@ -227,6 +223,26 @@
"performance"
]
},
+ {
+ "group": "SQL reference",
+ "pages": [
+ "sql/index",
+ "sql/data-types",
+ "sql/select",
+ "sql/subqueries",
+ "sql/join",
+ "sql/aggregation",
+ "sql/window",
+ "sql/sampling",
+ "sql/ddl",
+ "sql/dml",
+ "sql/refresh",
+ "sql/show",
+ "sql/explain",
+ "sql/operators",
+ "sql/functions"
+ ]
+ },
{
"group": "Support",
"pages": [
diff --git a/docs/web/sql/data-types.mdx b/docs/web/sql/data-types.mdx
new file mode 100644
index 000000000..5392641ab
--- /dev/null
+++ b/docs/web/sql/data-types.mdx
@@ -0,0 +1,108 @@
+---
+title: "SQL data types"
+sidebarTitle: "Data Types"
+description: "How Lance columns appear to SQL, including the types SQL has no name for."
+icon: shapes
+keywords: ["sql types", "arrow", "cast", "pseudo-columns", "vector", "blob", "json"]
+---
+
+Enterprise [SQL runs on a deployment.](/sql)
+
+There is no SQL type layer over Lance storage. The table provider hands
+DataFusion the dataset's Arrow schema directly, so a column is the same object
+to the query engine as it is to the format.
+
+The consequence is that SQL type names here are **spellings of Arrow types**,
+not a system that maps onto them. [Data types](/basics/data-types) documents the
+storage side; this page covers what changes when you reach a column from SQL.
+
+## SQL names for Arrow types {#names}
+
+| SQL | Arrow |
+| --- | --- |
+| `BOOLEAN` | `Boolean` |
+| `TINYINT` `SMALLINT` `INT` `BIGINT` | `Int8` `Int16` `Int32` `Int64` |
+| `UTINYINT` … `UBIGINT` | the unsigned counterparts |
+| `REAL` / `FLOAT` | `Float32` |
+| `DOUBLE` | `Float64` |
+| `VARCHAR` / `TEXT` / `STRING` | `Utf8` |
+| `BYTEA` / `BINARY` | `Binary` |
+| `DATE` | `Date32` |
+| `TIME` | `Time64` |
+| `TIMESTAMP` | `Timestamp`, with or without a timezone |
+| `INTERVAL` | `Interval` |
+| `STRUCT` / `LIST` / `MAP` | the nested equivalents |
+
+DataFusion's [type reference](https://datafusion.apache.org/user-guide/sql/data_types.html)
+describes the full mapping.
+
+## Types SQL has no name for {#no-name}
+
+Three column kinds you will meet that no SQL type name covers.
+
+### Vectors {#vectors}
+
+A vector column is `FixedSizeList`. It is selectable, testable for
+nullness, and joinable like any other column:
+
+```sql
+SELECT id FROM documents WHERE vector IS NOT NULL
+```
+
+
+Arithmetic over it is **list** arithmetic. There is no SQL similarity operator
+and no SQL form for vector search — run the search in an SDK and use SQL over
+its result. See [SDKs and SQL](/basics/apis#no-vector-sql).
+
+
+### JSON {#json}
+
+Stored as JSONB in `large_binary` with the `lance.json` extension name. It reads
+through accessor functions rather than a `->` operator:
+
+```sql
+SELECT json_get_string(data, 'name') FROM events
+WHERE json_get_int(data, 'status') = 200
+```
+
+`json_get`, its typed variants, and `json_extract` are **index-accelerated** by a
+JSON scalar index. See [Functions](/sql/functions#json).
+
+### Blobs {#blobs}
+
+A blob column is read either as its binary contents or as a **description** — a
+pointer plus metadata, without fetching the bytes. That choice changes the
+column's Arrow type, and therefore the schema your query sees.
+
+
+The default reads blob columns as descriptions and other binary columns as
+binary. A query selecting a blob column and expecting bytes gets a struct.
+
+
+## Pseudo-columns {#pseudo-columns}
+
+Row metadata, available only when named — so it stays out of `SELECT *`:
+
+| Column | Type | |
+| --- | --- | --- |
+| `_rowid` | `UInt64` | stable row identifier |
+| `_rowaddr` | `UInt64` | physical row address |
+| `_rowoffset` | `UInt64` | offset within the fragment |
+| `_score` | `Float32` | relevance, from [`fts()`](/sql/functions#fts) |
+
+`_rowid` and `_rowaddr` are declared nullable, but not because a row lacks one —
+their validity bitmap is reused internally as a selection vector. Do not read a
+null there as meaningful.
+
+`_rowid` is what [materialized views](/management/materialized-view) depend on,
+which is why a source table needs stable row ids enabled at creation.
+
+## Casting {#casting}
+
+```sql
+SELECT CAST(identifier AS VARCHAR) FROM documents
+SELECT TRY_CAST(maybe_number AS BIGINT) FROM raw
+```
+
+`TRY_CAST` yields null rather than failing the query on a value that will not
+convert. See [Operators](/sql/operators#casting).
diff --git a/docs/web/sql/ddl.mdx b/docs/web/sql/ddl.mdx
new file mode 100644
index 000000000..2629f263d
--- /dev/null
+++ b/docs/web/sql/ddl.mdx
@@ -0,0 +1,192 @@
+---
+title: "DDL"
+sidebarTitle: "DDL"
+description: "Creating and altering catalog objects: tables, views, indexes, functions, and columns."
+icon: pen-ruler
+keywords: ["ddl", "create table", "alter table", "add column", "create function", "create index", "wal"]
+---
+
+Enterprise [SQL runs on a deployment.](/sql)
+
+Statements that change what exists rather than what it holds. For the values in
+a table, see [DML](/sql/dml).
+
+## Tables {#tables}
+
+```sql
+CREATE TABLE documents (id BIGINT, title VARCHAR, created_at TIMESTAMP)
+CREATE TABLE recent AS SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
+DROP TABLE documents
+```
+
+Column types are Arrow types under SQL names -- see
+[Data types](/sql/data-types). A table in a [namespace](/namespaces) is written
+as a dotted path: `prod.events.clicks`.
+
+## Views {#views}
+
+```sql
+CREATE VIEW recent_matches AS
+SELECT t1.floats FROM documents AS t1
+INNER JOIN categories AS t2 ON t1.ints = t2.ints
+
+DROP VIEW recent_matches
+```
+
+`CREATE VIEW` returns the view's schema -- column name, type, nullability --
+resolved from the query at creation time. A view stores only the query; see
+[Views](/management/view).
+
+## Columns {#columns}
+
+`ALTER TABLE ... ADD COLUMN` covers three different things, distinguished by
+what follows the column name.
+
+### A plain column {#plain-column}
+
+```sql
+ALTER TABLE t ADD COLUMN c INT
+```
+
+Added empty and nullable. Nothing is computed.
+
+### A computed column {#computed-column}
+
+```sql
+ALTER TABLE t ADD COLUMN doubled AS value * 2
+```
+
+`AS ` declares a [computed column](/feature-engineering/add-computed-column).
+The column is committed empty and the expression is stored, not evaluated --
+declaring one on a billion-row table costs what declaring one on an empty table
+costs. [`REFRESH COLUMN`](/sql/refresh) fills it.
+
+### A function column {#function-column}
+
+```sql
+ALTER TABLE t ADD COLUMN embedded AS embed(name) WITH (version = 'fv_1')
+```
+
+When the expression calls a [registered function](#functions), the column is
+filled by running that function on workers. `version` pins which immutable
+function version fills it, so a later registration does not silently change what
+the column means.
+
+Several modifiers narrow what gets computed:
+
+| | |
+| --- | --- |
+| `FILTER (WHERE pred)` | only rows matching `pred` are computed |
+| `IGNORE NULLS` | rows with null inputs are skipped |
+| `OVER (PARTITION BY col)` | the function sees rows grouped by `col` |
+
+A function returning several values fills several columns at once:
+
+```sql
+ALTER TABLE t ADD COLUMN (label, score) AS classify(text) WITH (version = 'fv_1')
+```
+
+```sql
+ALTER TABLE t ADD COLUMN c AS f(x) FILTER (WHERE keep) WITH (version = 'fv_1')
+ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2')
+```
+
+
+`WITH` on a plain computed column accepts only `version`. Resource options such
+as `num_workers` apply to function columns, where there are workers to size.
+
+
+## Functions {#functions}
+
+```sql
+CREATE FUNCTION f(x BIGINT) RETURNS BIGINT
+LANGUAGE python
+AS 'def f(x): return x * 2'
+WITH (pip = 'numpy==1.26.4', python_version = '3.11')
+```
+
+The body is Python source. Its dependencies are declared with it and are part of
+the **immutable function version** -- changing one produces a new version rather
+than altering the registered one, so a column filled by an earlier version is
+unaffected.
+
+| Option | |
+| --- | --- |
+| `pip` | pip requirements for the worker environment |
+| `conda` | conda packages instead of pip |
+| `conda_channels` | channels in priority order; requires `conda` |
+| `python_version` | worker Python version, defaulting to yours |
+| `gpu` | `'true'` when every execution needs a GPU |
+
+**Errors and limits**
+
+| | |
+| --- | --- |
+| `LANGUAGE` other than `python` | refused |
+| `pip` and `conda` together | refused -- pick one |
+| `conda_channels` without `conda` | refused |
+| `gpu` set to anything but `'true'` | refused |
+| An empty body | refused |
+| Any other option | refused, and the message lists what is supported |
+
+See [Functions](/management/function) for the SDK equivalent and for diagnosing
+environment mismatches.
+
+## Indexes {#indexes}
+
+```sql
+CREATE INDEX ON documents (title)
+DROP INDEX documents.title_idx
+```
+
+
+SQL `CREATE INDEX` covers **one column** and takes **no user-chosen name** --
+the server names it. For a multi-column or named index, use the SDK. `DROP
+INDEX` takes the qualified `table.index` name; find it with `list_indices`.
+
+
+Index type and parameters are chosen in the SDK; see [Indexing](/indexing).
+
+## Table properties {#properties}
+
+```sql
+ALTER TABLE t SET UNENFORCED PRIMARY KEY (id)
+```
+
+Records which column identifies a row. **Unenforced** is not a hedge in the
+naming: uniqueness is not checked on write, and declaring it does not make
+duplicates impossible. It tells the planner what to assume.
+
+```sql
+ALTER TABLE t SET WAL
+ALTER TABLE t SET WAL WITH (sharding = 'bucket', buckets = 4)
+ALTER TABLE t SET WAL WITH (maintained_indexes = '')
+ALTER TABLE t UNSET WAL
+```
+
+A write-ahead log makes small frequent writes cheap by batching them out of the
+commit path. `sharding` and `buckets` spread the log so concurrent writers do
+not contend on one place; `maintained_indexes` chooses which indexes are kept
+current against WAL rows.
+
+
+[`TABLESAMPLE`](/sql/sampling) is refused on a WAL-backed table, because rows a
+sample would draw from are not all in the fragments yet.
+
+
+## See also {#see-also}
+
+
+
+ Changing rows rather than structure.
+
+
+ Filling the columns declared here.
+
+
+ The same operations from the SDKs.
+
+
+ Listing what exists.
+
+
diff --git a/docs/web/sql/dml.mdx b/docs/web/sql/dml.mdx
new file mode 100644
index 000000000..74d7f4966
--- /dev/null
+++ b/docs/web/sql/dml.mdx
@@ -0,0 +1,81 @@
+---
+title: "DML"
+sidebarTitle: "DML"
+description: "INSERT, UPDATE, DELETE and COPY — changing the rows a table holds."
+icon: pen-to-square
+keywords: ["insert", "update", "delete", "copy", "dml"]
+---
+
+Enterprise [SQL runs on a deployment.](/sql)
+
+Statements that change rows. For the shape of the table, see [DDL](/sql/ddl).
+
+Each of these commits a new table version, so
+[versioning](/tables/versioning) and the
+[write guarantees](/basics/guarantees#writes) apply -- including that the
+version advances even when a statement affects zero rows.
+
+## INSERT {#insert}
+
+```sql
+INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur')
+
+INSERT INTO archive SELECT * FROM documents WHERE created_at < '2026-01-01'
+```
+
+`INSERT ... SELECT` is the form worth knowing: it moves rows between tables
+server-side, without a round trip through your process.
+
+## UPDATE {#update}
+
+```sql
+UPDATE documents SET title = upper(title) WHERE category = 'film'
+```
+
+Assignments take any expression the [dialect](/sql/functions) supports,
+including references to other columns of the same row. Without a `WHERE` clause
+every row is updated.
+
+
+An update does not recompute [computed columns](/feature-engineering/add-computed-column)
+that derive from the column you changed. A filled value keeps what it was
+computed from -- see [Guarantees](/basics/guarantees#jobs).
+
+
+## DELETE {#delete}
+
+```sql
+DELETE FROM documents WHERE created_at < '2025-01-01'
+```
+
+Deletion marks rows rather than erasing them, so the data remains recoverable
+for as long as the version holding it survives
+[retention](/maintenance/compaction).
+
+## COPY {#copy}
+
+```sql
+COPY (SELECT id, title FROM documents) TO 's3://bucket/export/' STORED AS PARQUET
+```
+
+Writes a query result to object storage. This is the export path when the
+consumer is another engine rather than an application -- results land as files
+rather than streaming to a client.
+
+## Merge insert {#merge-insert}
+
+Upsert -- matching incoming rows against existing keys -- has no SQL form. It is
+an SDK operation, because it needs a source relation and a per-match policy that
+a single statement cannot express. See
+[Merge insert](/ingestion/merge-insert).
+
+## See also {#see-also}
+
+
+
+ The same operations from the SDKs.
+
+
+ Changing structure rather than rows.
+
+
diff --git a/docs/web/sql/explain.mdx b/docs/web/sql/explain.mdx
new file mode 100644
index 000000000..fa6f258d1
--- /dev/null
+++ b/docs/web/sql/explain.mdx
@@ -0,0 +1,51 @@
+---
+title: "EXPLAIN"
+sidebarTitle: "Explain"
+description: "Reading the plan for a statement, and measuring it."
+icon: map
+keywords: ["explain", "explain analyze", "query plan", "describe"]
+---
+
+Enterprise [SQL runs on a deployment.](/sql)
+
+```sql
+EXPLAIN SELECT * FROM documents WHERE identifier < 1000
+EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
+```
+
+`EXPLAIN` resolves the plan without running the query. `EXPLAIN ANALYZE`
+executes it and annotates every operator with runtime metrics — elapsed time,
+rows produced, bytes read, index comparisons.
+
+The operators are the same ones the SDK methods produce, so the reasoning about
+index coverage and filter pushdown carries across unchanged. The two pages that
+explain how to read them:
+
+
+
+ Reading a plan: operators, pushdown, index usage.
+
+
+ Reading the metrics, and a worked before/after.
+
+
+
+## What to look for {#what-to-look-for}
+
+| Sign | Means |
+| --- | --- |
+| `ScalarIndexQuery` | the predicate used a scalar index |
+| `LanceScan` with high `bytes_read` | no index, or too many columns selected |
+| `FilterExec` above a search | the filter ran after search rather than pushing into it |
+| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used |
+
+A `LanceScan` feeding a join is the common one worth acting on — see
+[Joins](/sql/join#performance).
+
+## DESCRIBE {#describe}
+
+```sql
+DESCRIBE documents
+```
+
+The table's columns and types, without a plan.
diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx
index 91bed36da..3e4056308 100644
--- a/docs/web/sql/index.mdx
+++ b/docs/web/sql/index.mdx
@@ -1,63 +1,85 @@
---
-title: "SQL"
+title: "SQL reference"
sidebarTitle: "Overview"
-description: "Analytical SQL over the lakehouse: the dialect, where it runs, and what it can reach."
+description: "The SQL surface: dialect, where a query runs, and a page per construct."
icon: table-list
-keywords: ["sql", "datafusion", "flight sql", "analytics", "dialect"]
+keywords: ["sql", "datafusion", "flight sql", "reference", "dialect"]
---
Enterprise SQL runs on a deployment. A local database
has no engine to plan a statement, so `execute_query` raises there -- see
[SDKs and SQL](/basics/apis).
-The dialect is [Apache DataFusion's](/sql/functions), so anything its reference
-documents for that version works here unchanged. What these pages cover is the
-parts that are LanceDB's: how a query reaches the engine, how Lance columns
-appear to it, and the extensions a lakehouse over multimodal data needs.
+This is a reference rather than a guide: one page per construct, syntax first.
+The guides elsewhere in these docs show the same operations from the SDKs.
+
+## The dialect {#dialect}
+
+LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org),
+currently **version 54**. The grammar, built-in functions and type system are
+DataFusion's, so anything its reference documents for that version works here
+unchanged and is not reproduced.
+
+What these pages add is the part that is LanceDB's: how a query reaches the
+engine, how Lance columns appear to it, and the statements a lakehouse needs
+that DataFusion has no concept of.
## Where a query runs {#execution}
-Queries execute on the server. The client sends a statement and receives Arrow
-record batches -- nothing is converted to pandas or Python objects unless you
-ask for it. A query feeding training or downstream processing therefore pays no
-serialization cost to get there.
+Queries execute on the server and results come back as Arrow record batches --
+nothing is converted to pandas or Python objects unless you ask. `execute_query`
+hands back a reader as soon as the first batch is ready rather than when the
+query finishes, so a large result set does not have to fit in memory before you
+start reading it. See [Query and job](/basics/jobs#queries).
-Results stream. `execute_query` hands back a reader as soon as the first batch
-is ready rather than when the query finishes, so a large result set does not
-have to fit in memory before you can start reading it. See
-[Query and job](/basics/jobs#queries).
+Served over an [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
+endpoint, so any Flight SQL client reaches it.
-## What differs from a warehouse {#differences}
+## Reference {#reference}
-Two things, and both come from the storage being Lance rather than something
-SQL-shaped:
-
-- **Columns are Arrow types**, not a SQL type system mapped onto storage. A
- vector column is a `FixedSizeList` and behaves like one. See
- [Data types](/basics/data-types).
-- **Row metadata is available as columns** -- `_rowid`, `_rowaddr`,
- `_rowoffset` -- but only when named, so they stay out of `SELECT *`.
-
-## The statements {#statements}
+**Reading data**
| | |
| --- | --- |
+| [Data types](/sql/data-types) | how Lance columns appear to SQL, and pseudo-columns |
| [SELECT](/sql/select) | projection, filtering, ordering, set operations, CTEs |
-| [Aggregation](/sql/aggregation) | `GROUP BY`, `HAVING`, aggregate functions |
-| [Joins](/sql/join) | inner, outer, cross, and joining across namespaces |
-| [Window functions](/sql/window) | `OVER (...)`, ranking, running totals |
-| [Sampling](/sql/sampling) | `TABLESAMPLE`, for a random subset |
-| [Functions and types](/sql/functions) | the function set, and what LanceDB adds |
+| [Subqueries](/sql/subqueries) | scalar, `IN`, `EXISTS`, correlated, derived tables |
+| [Joins](/sql/join) | inner, outer, cross, and namespace-qualified paths |
+| [Aggregation](/sql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set |
+| [Window functions](/sql/window) | `OVER`, framing, ranking and offset functions |
+| [Sampling](/sql/sampling) | `TABLESAMPLE`, by rows or by fragments |
-Beyond reading, SQL also writes (`INSERT`, `UPDATE`, `DELETE`, `COPY`), manages
-the catalog (`CREATE TABLE`, `CREATE VIEW`, `DROP`, `SHOW`) and drives feature
-engineering (`CREATE FUNCTION`, `REFRESH COLUMN`, `REFRESH MATERIALIZED VIEW`).
-Those are documented with the operation rather than here -- see
-[Management](/management) and [Feature engineering](/feature-engineering).
+**Changing things**
-
-Vector search has no SQL form. It is a query-builder operation -- see
-[Vector search](/search/vector-search). Full-text search does: the
-[`fts()`](/sql/functions#fts) table function and the
-[`contains_tokens`](/sql/functions#contains-tokens) predicate.
-
+| | |
+| --- | --- |
+| [DDL](/sql/ddl) | tables, views, indexes, functions, columns, WAL |
+| [DML](/sql/dml) | `INSERT`, `UPDATE`, `DELETE`, `COPY` |
+| [REFRESH](/sql/refresh) | filling computed columns and materialized views |
+
+**Introspection**
+
+| | |
+| --- | --- |
+| [SHOW](/sql/show) | jobs, functions, materialized views, errors |
+| [EXPLAIN](/sql/explain) | plans, and measuring them |
+
+**Expressions**
+
+| | |
+| --- | --- |
+| [Operators](/sql/operators) | comparison, logical, arithmetic, literals, precedence |
+| [Functions](/sql/functions) | the function set, and what Lance accelerates |
+
+## What SQL cannot do {#cannot}
+
+| | |
+| --- | --- |
+| Vector search | no SQL form -- a query-builder operation |
+| Merge insert | no SQL form -- needs a source relation and a per-match policy |
+| Run on a local database | no engine to plan the statement |
+
+Full-text search **is** available, as the [`fts()`](/sql/functions#fts) table
+function and the [`contains_tokens`](/sql/functions#contains-tokens) predicate.
+
+See [Limits](/basics/limits#sql) for the bounds these pages enforce.
diff --git a/docs/web/sql/operators.mdx b/docs/web/sql/operators.mdx
new file mode 100644
index 000000000..804ed2ea7
--- /dev/null
+++ b/docs/web/sql/operators.mdx
@@ -0,0 +1,81 @@
+---
+title: "Operators and literals"
+sidebarTitle: "Operators"
+description: "Comparison, logical and arithmetic operators, literals, and precedence."
+icon: plus-minus
+keywords: ["operators", "literals", "precedence", "null", "cast", "interval"]
+---
+
+Enterprise [SQL runs on a deployment.](/sql)
+
+The operator set is DataFusion's. This page covers what is worth knowing when
+the operands are Lance columns.
+
+## Comparison {#comparison}
+
+| | |
+| --- | --- |
+| `=` `!=` `<>` `<` `<=` `>` `>=` | ordinary comparison |
+| `IS NULL` / `IS NOT NULL` | null test — `= NULL` is never true |
+| `IS DISTINCT FROM` | comparison that treats null as a value |
+| `BETWEEN a AND b` | inclusive range |
+| `IN (...)` | set membership |
+| `LIKE` / `ILIKE` | pattern match, `%` and `_` |
+| `~` `~*` `!~` | regular expression match |
+
+
+For keyword matching on a text column, [`contains_tokens`](/sql/functions#contains-tokens)
+is index-accelerated where `LIKE` is not. Reach for `LIKE` only when you need
+its wildcard semantics.
+
+
+## Logical {#logical}
+
+`AND`, `OR`, `NOT`. Three-valued: a comparison against null is **unknown**, and
+`NOT unknown` is still unknown. A `WHERE` clause keeps only rows where the
+predicate is true, so unknown behaves like false there — but not inside `NOT
+IN`, which is the usual place this bites. See
+[Subqueries](/sql/subqueries#in-exists).
+
+## Arithmetic {#arithmetic}
+
+`+` `-` `*` `/` `%`, with the usual precedence. String concatenation is `||`.
+
+
+Arithmetic over a vector column is **list** arithmetic, not scalar arithmetic —
+a vector is a `FixedSizeList`. When the operation you want is a
+similarity comparison, use [vector search](/search/vector-search); there is no
+SQL form for it.
+
+
+## Literals {#literals}
+
+| | |
+| --- | --- |
+| String | `'text'` — doubled quote to escape: `'it''s'` |
+| Number | `42`, `3.14` |
+| Boolean | `TRUE`, `FALSE` |
+| Null | `NULL` |
+| Date / timestamp | `DATE '2026-01-01'`, `TIMESTAMP '2026-01-01 12:00:00'` |
+| Interval | `INTERVAL '7 days'` |
+| Array | `[1, 2, 3]` |
+
+Identifiers are quoted with double quotes when they need to be — a reserved word
+or a name with mixed case: `SELECT "order" FROM "MyTable"`.
+
+## Casting {#casting}
+
+```sql
+SELECT CAST(identifier AS VARCHAR), identifier::VARCHAR FROM documents
+```
+
+Both forms work. `TRY_CAST` returns null instead of erroring on a value that
+will not convert, which is the one to use over a column you do not control.
+
+## Precedence {#precedence}
+
+Highest to lowest: `::` → unary `-` → `* / %` → `+ -` → `||` → comparison →
+`IS` → `NOT` → `AND` → `OR`.
+
+Parenthesize anything mixing `AND` and `OR`. `a OR b AND c` is `a OR (b AND c)`,
+which is rarely what was meant.
diff --git a/docs/web/sql/refresh.mdx b/docs/web/sql/refresh.mdx
new file mode 100644
index 000000000..3208261eb
--- /dev/null
+++ b/docs/web/sql/refresh.mdx
@@ -0,0 +1,95 @@
+---
+title: "REFRESH"
+sidebarTitle: "Refresh"
+description: "Filling computed columns and bringing materialized views up to date."
+icon: arrows-rotate
+keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"]
+---
+
+Enterprise [SQL runs on a deployment.](/sql)
+
+LanceDB extensions with no DataFusion equivalent. They are what makes
+[feature engineering](/feature-engineering) reachable from SQL: a column
+declared by [DDL](/sql/ddl#computed-column) starts empty, and a refresh fills it.
+
+## REFRESH COLUMN {#refresh-column}
+
+```sql
+REFRESH COLUMN ON [TABLE]