From 5b5ed930acd8daf607f857346fbff0556db37a16 Mon Sep 17 00:00:00 2001 From: Jack Ye Date: Sun, 13 Sep 2026 23:32:15 -0700 Subject: [PATCH] docs: restructure Documentation around one axis, promote the SQL reference Every group now answers exactly one question. Get started and Concepts cover what the system is; the verb groups cover doing something with it; Operate, Secure and Deploy cover running it; SQL reference is lookup. Maintenance becomes Operate, Governance becomes Secure, and the rest take verbs to match. Three new pages in Concepts and Get started: Guarantees states what a write promises when it returns, which version a reader sees, what survives a failure, and -- the half usually missing -- what is not guaranteed: no multi-table transactions, no read-your-writes across processes without an interval, unenforced primary keys that do not enforce, no background compaction. Limits collects bounds that were stated in passing across a dozen pages, with the numbers verified against the code rather than inferred: name lengths, job event caps, SHOW ERRORS bounds, the sampling range that is refused rather than clamped. Architecture illustrates embedded against distributed as the one deployment decision, and is written to cover both rather than only Enterprise. Indexing splits by question. Choosing an index is a Query decision and moves there; building, repairing and GPU-building it stay in Operate. SQL becomes a top-level reference of fifteen pages instead of a sub-group of Query. Auditing sophon-sql turned up a large undocumented surface: ALTER TABLE ADD COLUMN in its plain, computed and function forms with FILTER, IGNORE NULLS and OVER modifiers; SET UNENFORCED PRIMARY KEY; SET and UNSET WAL with sharding; CREATE FUNCTION with its dependency options; REFRESH COLUMN and REFRESH MATERIALIZED VIEW; and the SHOW family. None of it was written down. --- docs/web/architecture.mdx | 150 ++++++++++++++++++++++++++ docs/web/basics/guarantees.mdx | 130 ++++++++++++++++++++++ docs/web/basics/limits.mdx | 103 ++++++++++++++++++ docs/web/docs.json | 62 +++++++---- docs/web/sql/data-types.mdx | 108 +++++++++++++++++++ docs/web/sql/ddl.mdx | 192 +++++++++++++++++++++++++++++++++ docs/web/sql/dml.mdx | 81 ++++++++++++++ docs/web/sql/explain.mdx | 51 +++++++++ docs/web/sql/index.mdx | 106 ++++++++++-------- docs/web/sql/operators.mdx | 81 ++++++++++++++ docs/web/sql/refresh.mdx | 95 ++++++++++++++++ docs/web/sql/show.mdx | 87 +++++++++++++++ docs/web/sql/subqueries.mdx | 76 +++++++++++++ 13 files changed, 1257 insertions(+), 65 deletions(-) create mode 100644 docs/web/architecture.mdx create mode 100644 docs/web/basics/guarantees.mdx create mode 100644 docs/web/basics/limits.mdx create mode 100644 docs/web/sql/data-types.mdx create mode 100644 docs/web/sql/ddl.mdx create mode 100644 docs/web/sql/dml.mdx create mode 100644 docs/web/sql/explain.mdx create mode 100644 docs/web/sql/operators.mdx create mode 100644 docs/web/sql/refresh.mdx create mode 100644 docs/web/sql/show.mdx create mode 100644 docs/web/sql/subqueries.mdx diff --git a/docs/web/architecture.mdx b/docs/web/architecture.mdx new file mode 100644 index 000000000..40b305786 --- /dev/null +++ b/docs/web/architecture.mdx @@ -0,0 +1,150 @@ +--- +title: "Architecture" +sidebarTitle: "Architecture" +description: "How LanceDB is put together, and what changes between running it embedded and running it distributed." +icon: cubes +keywords: ["architecture", "embedded", "distributed", "object storage", "compute", "deployment"] +--- + +LanceDB has one storage layer and two ways to put compute on top of it. Which +one you run decides where queries execute, what runs on your behalf, and what +you have to operate — and almost nothing else. The data is the same +[Lance](/basics/lance) tables either way. + +## One storage layer {#storage} + +Everything durable lives in object storage: table data, manifests, indexes. No +machine owns a dataset, and no server has to be running for the data to exist. + +``` + ┌──────────────────────────────────────────┐ + │ object storage │ + │ table data · manifests · indexes │ + └──────────────────────────────────────────┘ +``` + +That is the whole persistence story, and it is why the two deployment shapes can +differ so much above it while staying compatible below it. A table written +embedded is opened by a deployment, and the reverse, without conversion. + +## Embedded {#embedded} + +``` + your process + ┌────────────────────────────────┐ + │ application │ + │ LanceDB library │ ← plans, executes, indexes, compacts + └────────────────────────────────┘ + │ + ┌─────────▼─────────┐ + │ object storage │ + └───────────────────┘ +``` + +`lancedb.connect("./data")` or `connect("s3://bucket/path")` gives you a +**library**, not a client. Your process opens the tables, plans the queries, +runs the scans, builds the indexes, and writes the new versions. There is no +server in the picture at all. + +**What follows from that.** Nothing runs unless you call it — no background +compaction, no server-side [jobs](/basics/jobs), no +[SQL endpoint](/basics/apis). Scale is your process's scale: a query gets the +cores and memory your process has. Access control is whatever your object store +enforces, since there is no other gatekeeper. + +**What it is good at.** Development, notebooks, single-node pipelines, embedding +LanceDB inside an application, and anything where "no service to operate" is +worth more than elasticity. + +## Distributed {#distributed} + +Enterprise + +``` + your process + ┌──────────────┐ + │ application │ + │ LanceDB SDK │ ← a thin client + └──────┬───────┘ + │ db:// (REST / Flight SQL) + ┌──────▼──────────────────────────────────────┐ + │ control plane config · identity · policy │ + ├─────────────────────────────────────────────┤ + │ data plane query serving · writes │ + │ indexers indexing · compaction │ + └──────┬──────────────────────────────────────┘ + │ + ┌──────▼────────────┐ + │ object storage │ + └───────────────────┘ +``` + +`connect("db://...")` gives you a **client**. Planning and execution move to a +cluster, and the work splits across tiers that scale independently: a control +plane governing configuration, identity and policy; a data plane serving reads +and writes; and indexers building indexes and compacting outside the request +path. + +**What follows from that.** Compute and storage are separated, so a query fleet +scales for interactive traffic without also scaling indexing capacity, and heavy +background work does not steal resources from user queries. Because a server +exists, so do the things that need one: [SQL](/sql), +[functions](/management/function), server-side jobs, and +[authorization](/governance/authorization) per object rather than per bucket. + +**What it is good at.** Many concurrent queries, continuous ingestion and +indexing, multi-team access with real permissions, and workloads too large for +one machine. + +## What changes, precisely {#differences} + +| | Embedded | Distributed | +| --- | --- | --- | +| What you connect to | a library in your process | a cluster | +| Where queries plan and execute | your process | the data plane | +| Table handle | `LanceTable` | `RemoteTable` | +| Indexing and compaction | when you call it | dedicated indexers | +| [Jobs](/basics/jobs) | run in-process, no server record | real jobs: id, status, events, cancel | +| [SQL](/sql) | unavailable | Flight SQL endpoint | +| [Functions](/management/function) | unavailable | run on workers | +| Access control | object-store permissions | [per-object grants](/governance/authorization) | +| Scale bound | your process | the cluster | +| What you operate | nothing | a deployment | + +Both expose the same table API — create, open, mutate, query — so application +code moves between them by changing the URI. What does not move is code that +depends on a row in the right-hand column. + +## Choosing, and changing your mind {#choosing} + +Start embedded. It has no operational cost and the API is the same one you will +keep using. + +Move when you hit something in the right-hand column — usually concurrent +serving, ingestion that outgrows one machine, or a permissions requirement. +Because the storage layer is identical, moving is a connection-string change +plus a data migration, not a rewrite. + + +The one thing to decide early is **stable row ids**, which cannot be enabled +after a table exists and are required for +[materialized views](/management/materialized-view). See +[Limits](/basics/limits#schema). + + +## Where to go next {#next} + + + + What a write promises, and what a reader sees. + + + The open format underneath both shapes. + + + The distributed tiers in detail. + + + The bounds you can actually hit. + + diff --git a/docs/web/basics/guarantees.mdx b/docs/web/basics/guarantees.mdx new file mode 100644 index 000000000..c5e2aeea5 --- /dev/null +++ b/docs/web/basics/guarantees.mdx @@ -0,0 +1,130 @@ +--- +title: "Guarantees" +sidebarTitle: "Guarantees" +description: "What a write promises when it returns, what a reader sees, and what survives a failure." +icon: shield-check +keywords: ["guarantees", "consistency", "atomicity", "durability", "concurrency", "acid"] +--- + +What the system promises, and — just as usefully — what it does not. Most +surprises in practice come from assuming one of the guarantees below is +stronger or weaker than it is. + +## Writes {#writes} + +**A write is atomic.** Every write commits a new table version. Either the +version exists and holds all of the change, or it does not exist and holds none +of it. There is no state in which a reader sees half an `add()`. + +**A write returns only after the commit.** When the call returns, the version +is durable in object storage. Nothing is buffered in your process waiting to be +flushed. + +**Every write advances the version**, including one that affects zero rows. A +delete whose predicate matches nothing still commits. Version numbers count +commits, not changes. + +**Concurrent writes are resolved optimistically.** A transaction records the +change and the version it was based on. At commit time, a conflicting +concurrent operation is detected rather than silently overwritten, and the +commit is retried against the newer version. Operations that do not collide — +appending while another process appends — both succeed. + + +`merge_insert` reports its retry attempts in the result. A high count means +contention on the same keys, not an error. + + +## Reads {#reads} + +**A read sees a consistent snapshot.** A query reads one table version +throughout; it never sees a mixture of two. + +**Which version, by default, is the one open at the time the table was opened.** +This is the guarantee most often assumed to be stronger than it is: + + +A long-lived connection **does not observe other processes' writes** until it +reopens the table or its `read_consistency_interval` elapses. The default is +right for a batch job reading a stable snapshot and wrong for a service +expected to see fresh data. + + +Set `read_consistency_interval` when several processes write the same table and +readers must observe those writes. The cost is a metadata check on that +interval. See [Consistency](/tables/consistency). + +**Your own writes are always visible to you.** The interval governs other +writers, not yourself. + +## Durability and history {#durability} + +**Old versions are retained, not overwritten.** A new version does not destroy +the one before it, which is what makes [time travel](/tables/versioning) work +and what lets a reader finish a scan that started on an older version. + +**Deletion is a mark, not an erase.** A deleted row is recorded in a deletion +file. The data is recoverable for as long as the version that contains it +exists. + + +Retention is finite. [Compaction](/maintenance/compaction) prunes versions +older than the retention window — **7 days by default** — and a tag or branch +pointing at a pruned version no longer resolves. If you rely on reading old +versions, widen the window before running it, not after. + + +**Dropping a table ends its history.** Versioning does not survive +`drop_table`; there is no undrop. + +## Jobs {#jobs} + +**A refresh never revisits a filled row.** It gives values to rows that lack +them and leaves the rest alone. Two consequences, both relied on elsewhere: + +- **Retrying is safe and cheap.** A failed backfill is resumed, not restarted — + re-running costs only the rows that did not complete. +- **A filled value does not track its input.** If a column derives from `price` + and a row's `price` later changes, the derived value stays as computed. + Recomputing means dropping the column and declaring it again. + +**A failure is not a rollback.** Rows filled before a failure keep their values, +because each commits as it completes. + +**Cancellation is a request.** `cancel()` asks a job to stop; it does not +guarantee it stops before doing more work, and cancelling a finished job does +nothing. + +**A job id is opaque.** It correlates with server logs and the jobs API. +Parsing it, or storing it to resume the job later, is not supported. + +See [Query and job](/basics/jobs). + +## Schema {#schema} + +**Schema changes are versioned like data changes.** Adding, renaming, retyping +or dropping a column commits a version, so a schema change can be travelled back +through. + +**A computed column's inputs are protected.** While a declaration references a +column, that column cannot be renamed, retyped, or dropped — the declaration is +part of the schema and will not be invalidated out from under itself. + +**Most schema changes do not rewrite data.** Adding a derived column writes only +the new column. Type changes are the exception and can be substantial. + +## What is not guaranteed {#not-guaranteed} + +Worth stating plainly, because each is occasionally assumed: + +| | | +| --- | --- | +| Multi-table transactions | A commit covers one table. There is no cross-table atomic write. | +| Read-your-writes across processes | Governed by `read_consistency_interval`, not automatic. | +| Unenforced primary keys | `SET UNENFORCED PRIMARY KEY` records intent; it does not enforce uniqueness. | +| Ordering of concurrent appends | Two appends both succeed; their relative row order is not defined. | +| Index freshness | An index does not cover rows added after it was built until [optimized](/maintenance/compaction). | +| Automatic compaction | Nothing runs in the background. `optimize()` runs when you call it. | + +The last two are the most common source of "why is this query slow" — see +[Limits](/basics/limits) for the bounds that pair with them. diff --git a/docs/web/basics/limits.mdx b/docs/web/basics/limits.mdx new file mode 100644 index 000000000..6727dc94b --- /dev/null +++ b/docs/web/basics/limits.mdx @@ -0,0 +1,103 @@ +--- +title: "Limits" +sidebarTitle: "Limits" +description: "The bounds you can actually hit, in one place." +icon: ruler +keywords: ["limits", "maximum", "quota", "constraints", "bounds"] +--- + +Limits worth designing around, collected so you do not have to find them by +hitting them. Each one is enforced -- where a value is refused rather than +clamped, that is stated, because the two fail very differently. + +This page covers bounds that are the same for everyone. Deployment-specific +quotas -- storage, throughput, concurrency -- come from your deployment. + +## Names {#names} + +| Object | Maximum | | +| --- | --- | --- | +| Database | 64 characters | | +| Namespace | 64 characters | | +| Table | 128 characters | | +| Branch | 64 characters | | + +A name must not be empty, and must not contain `$` or `:` -- both are reserved +as path delimiters, so a name holding one could not be addressed unambiguously. +Length is counted in characters, not bytes. + +## Jobs {#jobs} + +| | | +| --- | --- | +| `job.events()` default | 1,000 rows | +| `job.events()` maximum | 10,000 rows | +| `SHOW ERRORS ... LIMIT` | between 1 and 100,000 | +| Failed jobs scanned by one `SHOW ERRORS` | 32 | + + +`job.events()` **truncates silently**. A job that emits an event per fragment +will lose its tail unless you pass `limit` explicitly. This is the limit on this +page most likely to mislead you, because nothing in the result says it was cut. + + +`SHOW ERRORS` without `FOR JOB` scans the most recent failed jobs and stops at +32 of them. Narrow with `FOR JOB` when you need a specific one rather than a +survey. + +See [Query and job](/basics/jobs). + +## Schema and types {#schema} + +| | | +| --- | --- | +| `map` with `keys_sorted = true` | rejected | +| `IVF_RQ` vector dimensions | must be divisible by 8 | +| FTS `max_token_length` | 40 by default; longer tokens are dropped | + +Stable row ids **cannot be enabled after a table exists**. A table created +without them cannot become a +[materialized view](/management/materialized-view) source without being +recreated -- the one schema decision here that is genuinely irreversible. + +See [Data types](/basics/data-types). + +## SQL {#sql} + +Enterprise + +| | | +| --- | --- | +| `TABLESAMPLE` percentage | 0 to 100, **refused** outside -- not clamped | +| `TABLESAMPLE (n ROWS)` | whole numbers only | +| `TABLESAMPLE BUCKET` | not supported | +| `TABLESAMPLE` on a WAL-backed table | not supported | +| `CREATE INDEX` | one column, server-chosen name | +| `REFRESH MATERIALIZED VIEW ... CASCADE` | parsed, not yet implemented | + +A sampling percentage over 100 is an error rather than a full scan, because a +query asking for 150 percent has a mistake in it and reading it as 100 would +hide the mistake. See [Sampling](/sql/sampling). + +## What is unavailable where {#availability} + +Not quotas, but the same class of surprise -- things that exist in one place and +not another. + +| | | +| --- | --- | +| SQL on a local database | unavailable; `execute_query` raises | +| SQL methods in the TypeScript SDK | none; use a Flight SQL client | +| Function methods in the TypeScript SDK | none; use Python or SQL | +| Vector search in SQL | no SQL form; use a query builder | +| Geospatial `st_*` functions | behind a build feature LanceDB does not ship | +| `tableExists` / `namespaceExists` in TypeScript | not present | +| Server-side jobs on a local database | run in-process; `events()` raises | + +See [SDKs and SQL](/basics/apis) for why the first two hold. + +## Nothing here is a quota {#not-quotas} + +None of the above is a billing or capacity limit. Rows per table, table count, +storage, and request rate are bounded by your deployment rather than by the +format -- ask your deployment, not this page. diff --git a/docs/web/docs.json b/docs/web/docs.json index 45b193a41..f2bd300fd 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -98,26 +98,29 @@ "tab": "Documentation", "groups": [ { - "group": "Introduction", + "group": "Get started", "pages": [ "index", - "quickstart" + "quickstart", + "architecture" ] }, { - "group": "Basics", + "group": "Concepts", "pages": [ "basics/lance", "basics/catalog", "basics/jobs", "basics/data-types", "tables/consistency", + "basics/guarantees", + "basics/limits", "basics/apis", "basics/agentic-skills" ] }, { - "group": "Management", + "group": "Manage", "pages": [ "management/index", "management/database", @@ -142,7 +145,7 @@ ] }, { - "group": "Ingestion", + "group": "Ingest", "pages": [ "ingestion/index", "ingestion/insert", @@ -172,20 +175,18 @@ "reranking/eval" ] }, - "search/explain", - "search/explain-analyze", { - "group": "SQL", + "group": "Indexing", "pages": [ - "sql/index", - "sql/select", - "sql/aggregation", - "sql/join", - "sql/window", - "sql/sampling", - "sql/functions" + "indexing/index", + "indexing/vector-index", + "indexing/fts-index", + "indexing/scalar-index", + "indexing/quantization" ] - } + }, + "search/explain", + "search/explain-analyze" ] }, { @@ -199,7 +200,7 @@ ] }, { - "group": "Training", + "group": "Train", "pages": [ "training/why-lancedb", "training/index", @@ -207,13 +208,8 @@ ] }, { - "group": "Maintenance", + "group": "Operate", "pages": [ - "indexing/index", - "indexing/vector-index", - "indexing/fts-index", - "indexing/scalar-index", - "indexing/quantization", "indexing/reindexing", "maintenance/compaction", { @@ -227,6 +223,26 @@ "performance" ] }, + { + "group": "SQL reference", + "pages": [ + "sql/index", + "sql/data-types", + "sql/select", + "sql/subqueries", + "sql/join", + "sql/aggregation", + "sql/window", + "sql/sampling", + "sql/ddl", + "sql/dml", + "sql/refresh", + "sql/show", + "sql/explain", + "sql/operators", + "sql/functions" + ] + }, { "group": "Support", "pages": [ diff --git a/docs/web/sql/data-types.mdx b/docs/web/sql/data-types.mdx new file mode 100644 index 000000000..5392641ab --- /dev/null +++ b/docs/web/sql/data-types.mdx @@ -0,0 +1,108 @@ +--- +title: "SQL data types" +sidebarTitle: "Data Types" +description: "How Lance columns appear to SQL, including the types SQL has no name for." +icon: shapes +keywords: ["sql types", "arrow", "cast", "pseudo-columns", "vector", "blob", "json"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +There is no SQL type layer over Lance storage. The table provider hands +DataFusion the dataset's Arrow schema directly, so a column is the same object +to the query engine as it is to the format. + +The consequence is that SQL type names here are **spellings of Arrow types**, +not a system that maps onto them. [Data types](/basics/data-types) documents the +storage side; this page covers what changes when you reach a column from SQL. + +## SQL names for Arrow types {#names} + +| SQL | Arrow | +| --- | --- | +| `BOOLEAN` | `Boolean` | +| `TINYINT` `SMALLINT` `INT` `BIGINT` | `Int8` `Int16` `Int32` `Int64` | +| `UTINYINT` … `UBIGINT` | the unsigned counterparts | +| `REAL` / `FLOAT` | `Float32` | +| `DOUBLE` | `Float64` | +| `VARCHAR` / `TEXT` / `STRING` | `Utf8` | +| `BYTEA` / `BINARY` | `Binary` | +| `DATE` | `Date32` | +| `TIME` | `Time64` | +| `TIMESTAMP` | `Timestamp`, with or without a timezone | +| `INTERVAL` | `Interval` | +| `STRUCT` / `LIST` / `MAP` | the nested equivalents | + +DataFusion's [type reference](https://datafusion.apache.org/user-guide/sql/data_types.html) +describes the full mapping. + +## Types SQL has no name for {#no-name} + +Three column kinds you will meet that no SQL type name covers. + +### Vectors {#vectors} + +A vector column is `FixedSizeList`. It is selectable, testable for +nullness, and joinable like any other column: + +```sql +SELECT id FROM documents WHERE vector IS NOT NULL +``` + + +Arithmetic over it is **list** arithmetic. There is no SQL similarity operator +and no SQL form for vector search — run the search in an SDK and use SQL over +its result. See [SDKs and SQL](/basics/apis#no-vector-sql). + + +### JSON {#json} + +Stored as JSONB in `large_binary` with the `lance.json` extension name. It reads +through accessor functions rather than a `->` operator: + +```sql +SELECT json_get_string(data, 'name') FROM events +WHERE json_get_int(data, 'status') = 200 +``` + +`json_get`, its typed variants, and `json_extract` are **index-accelerated** by a +JSON scalar index. See [Functions](/sql/functions#json). + +### Blobs {#blobs} + +A blob column is read either as its binary contents or as a **description** — a +pointer plus metadata, without fetching the bytes. That choice changes the +column's Arrow type, and therefore the schema your query sees. + + +The default reads blob columns as descriptions and other binary columns as +binary. A query selecting a blob column and expecting bytes gets a struct. + + +## Pseudo-columns {#pseudo-columns} + +Row metadata, available only when named — so it stays out of `SELECT *`: + +| Column | Type | | +| --- | --- | --- | +| `_rowid` | `UInt64` | stable row identifier | +| `_rowaddr` | `UInt64` | physical row address | +| `_rowoffset` | `UInt64` | offset within the fragment | +| `_score` | `Float32` | relevance, from [`fts()`](/sql/functions#fts) | + +`_rowid` and `_rowaddr` are declared nullable, but not because a row lacks one — +their validity bitmap is reused internally as a selection vector. Do not read a +null there as meaningful. + +`_rowid` is what [materialized views](/management/materialized-view) depend on, +which is why a source table needs stable row ids enabled at creation. + +## Casting {#casting} + +```sql +SELECT CAST(identifier AS VARCHAR) FROM documents +SELECT TRY_CAST(maybe_number AS BIGINT) FROM raw +``` + +`TRY_CAST` yields null rather than failing the query on a value that will not +convert. See [Operators](/sql/operators#casting). diff --git a/docs/web/sql/ddl.mdx b/docs/web/sql/ddl.mdx new file mode 100644 index 000000000..2629f263d --- /dev/null +++ b/docs/web/sql/ddl.mdx @@ -0,0 +1,192 @@ +--- +title: "DDL" +sidebarTitle: "DDL" +description: "Creating and altering catalog objects: tables, views, indexes, functions, and columns." +icon: pen-ruler +keywords: ["ddl", "create table", "alter table", "add column", "create function", "create index", "wal"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +Statements that change what exists rather than what it holds. For the values in +a table, see [DML](/sql/dml). + +## Tables {#tables} + +```sql +CREATE TABLE documents (id BIGINT, title VARCHAR, created_at TIMESTAMP) +CREATE TABLE recent AS SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days' +DROP TABLE documents +``` + +Column types are Arrow types under SQL names -- see +[Data types](/sql/data-types). A table in a [namespace](/namespaces) is written +as a dotted path: `prod.events.clicks`. + +## Views {#views} + +```sql +CREATE VIEW recent_matches AS +SELECT t1.floats FROM documents AS t1 +INNER JOIN categories AS t2 ON t1.ints = t2.ints + +DROP VIEW recent_matches +``` + +`CREATE VIEW` returns the view's schema -- column name, type, nullability -- +resolved from the query at creation time. A view stores only the query; see +[Views](/management/view). + +## Columns {#columns} + +`ALTER TABLE ... ADD COLUMN` covers three different things, distinguished by +what follows the column name. + +### A plain column {#plain-column} + +```sql +ALTER TABLE t ADD COLUMN c INT +``` + +Added empty and nullable. Nothing is computed. + +### A computed column {#computed-column} + +```sql +ALTER TABLE t ADD COLUMN doubled AS value * 2 +``` + +`AS ` declares a [computed column](/feature-engineering/add-computed-column). +The column is committed empty and the expression is stored, not evaluated -- +declaring one on a billion-row table costs what declaring one on an empty table +costs. [`REFRESH COLUMN`](/sql/refresh) fills it. + +### A function column {#function-column} + +```sql +ALTER TABLE t ADD COLUMN embedded AS embed(name) WITH (version = 'fv_1') +``` + +When the expression calls a [registered function](#functions), the column is +filled by running that function on workers. `version` pins which immutable +function version fills it, so a later registration does not silently change what +the column means. + +Several modifiers narrow what gets computed: + +| | | +| --- | --- | +| `FILTER (WHERE pred)` | only rows matching `pred` are computed | +| `IGNORE NULLS` | rows with null inputs are skipped | +| `OVER (PARTITION BY col)` | the function sees rows grouped by `col` | + +A function returning several values fills several columns at once: + +```sql +ALTER TABLE t ADD COLUMN (label, score) AS classify(text) WITH (version = 'fv_1') +``` + +```sql +ALTER TABLE t ADD COLUMN c AS f(x) FILTER (WHERE keep) WITH (version = 'fv_1') +ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2') +``` + + +`WITH` on a plain computed column accepts only `version`. Resource options such +as `num_workers` apply to function columns, where there are workers to size. + + +## Functions {#functions} + +```sql +CREATE FUNCTION f(x BIGINT) RETURNS BIGINT +LANGUAGE python +AS 'def f(x): return x * 2' +WITH (pip = 'numpy==1.26.4', python_version = '3.11') +``` + +The body is Python source. Its dependencies are declared with it and are part of +the **immutable function version** -- changing one produces a new version rather +than altering the registered one, so a column filled by an earlier version is +unaffected. + +| Option | | +| --- | --- | +| `pip` | pip requirements for the worker environment | +| `conda` | conda packages instead of pip | +| `conda_channels` | channels in priority order; requires `conda` | +| `python_version` | worker Python version, defaulting to yours | +| `gpu` | `'true'` when every execution needs a GPU | + +**Errors and limits** + +| | | +| --- | --- | +| `LANGUAGE` other than `python` | refused | +| `pip` and `conda` together | refused -- pick one | +| `conda_channels` without `conda` | refused | +| `gpu` set to anything but `'true'` | refused | +| An empty body | refused | +| Any other option | refused, and the message lists what is supported | + +See [Functions](/management/function) for the SDK equivalent and for diagnosing +environment mismatches. + +## Indexes {#indexes} + +```sql +CREATE INDEX ON documents (title) +DROP INDEX documents.title_idx +``` + + +SQL `CREATE INDEX` covers **one column** and takes **no user-chosen name** -- +the server names it. For a multi-column or named index, use the SDK. `DROP +INDEX` takes the qualified `table.index` name; find it with `list_indices`. + + +Index type and parameters are chosen in the SDK; see [Indexing](/indexing). + +## Table properties {#properties} + +```sql +ALTER TABLE t SET UNENFORCED PRIMARY KEY (id) +``` + +Records which column identifies a row. **Unenforced** is not a hedge in the +naming: uniqueness is not checked on write, and declaring it does not make +duplicates impossible. It tells the planner what to assume. + +```sql +ALTER TABLE t SET WAL +ALTER TABLE t SET WAL WITH (sharding = 'bucket', buckets = 4) +ALTER TABLE t SET WAL WITH (maintained_indexes = '') +ALTER TABLE t UNSET WAL +``` + +A write-ahead log makes small frequent writes cheap by batching them out of the +commit path. `sharding` and `buckets` spread the log so concurrent writers do +not contend on one place; `maintained_indexes` chooses which indexes are kept +current against WAL rows. + + +[`TABLESAMPLE`](/sql/sampling) is refused on a WAL-backed table, because rows a +sample would draw from are not all in the fragments yet. + + +## See also {#see-also} + + + + Changing rows rather than structure. + + + Filling the columns declared here. + + + The same operations from the SDKs. + + + Listing what exists. + + diff --git a/docs/web/sql/dml.mdx b/docs/web/sql/dml.mdx new file mode 100644 index 000000000..74d7f4966 --- /dev/null +++ b/docs/web/sql/dml.mdx @@ -0,0 +1,81 @@ +--- +title: "DML" +sidebarTitle: "DML" +description: "INSERT, UPDATE, DELETE and COPY — changing the rows a table holds." +icon: pen-to-square +keywords: ["insert", "update", "delete", "copy", "dml"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +Statements that change rows. For the shape of the table, see [DDL](/sql/ddl). + +Each of these commits a new table version, so +[versioning](/tables/versioning) and the +[write guarantees](/basics/guarantees#writes) apply -- including that the +version advances even when a statement affects zero rows. + +## INSERT {#insert} + +```sql +INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur') + +INSERT INTO archive SELECT * FROM documents WHERE created_at < '2026-01-01' +``` + +`INSERT ... SELECT` is the form worth knowing: it moves rows between tables +server-side, without a round trip through your process. + +## UPDATE {#update} + +```sql +UPDATE documents SET title = upper(title) WHERE category = 'film' +``` + +Assignments take any expression the [dialect](/sql/functions) supports, +including references to other columns of the same row. Without a `WHERE` clause +every row is updated. + + +An update does not recompute [computed columns](/feature-engineering/add-computed-column) +that derive from the column you changed. A filled value keeps what it was +computed from -- see [Guarantees](/basics/guarantees#jobs). + + +## DELETE {#delete} + +```sql +DELETE FROM documents WHERE created_at < '2025-01-01' +``` + +Deletion marks rows rather than erasing them, so the data remains recoverable +for as long as the version holding it survives +[retention](/maintenance/compaction). + +## COPY {#copy} + +```sql +COPY (SELECT id, title FROM documents) TO 's3://bucket/export/' STORED AS PARQUET +``` + +Writes a query result to object storage. This is the export path when the +consumer is another engine rather than an application -- results land as files +rather than streaming to a client. + +## Merge insert {#merge-insert} + +Upsert -- matching incoming rows against existing keys -- has no SQL form. It is +an SDK operation, because it needs a source relation and a per-match policy that +a single statement cannot express. See +[Merge insert](/ingestion/merge-insert). + +## See also {#see-also} + + + + The same operations from the SDKs. + + + Changing structure rather than rows. + + diff --git a/docs/web/sql/explain.mdx b/docs/web/sql/explain.mdx new file mode 100644 index 000000000..fa6f258d1 --- /dev/null +++ b/docs/web/sql/explain.mdx @@ -0,0 +1,51 @@ +--- +title: "EXPLAIN" +sidebarTitle: "Explain" +description: "Reading the plan for a statement, and measuring it." +icon: map +keywords: ["explain", "explain analyze", "query plan", "describe"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +```sql +EXPLAIN SELECT * FROM documents WHERE identifier < 1000 +EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000 +``` + +`EXPLAIN` resolves the plan without running the query. `EXPLAIN ANALYZE` +executes it and annotates every operator with runtime metrics — elapsed time, +rows produced, bytes read, index comparisons. + +The operators are the same ones the SDK methods produce, so the reasoning about +index coverage and filter pushdown carries across unchanged. The two pages that +explain how to read them: + + + + Reading a plan: operators, pushdown, index usage. + + + Reading the metrics, and a worked before/after. + + + +## What to look for {#what-to-look-for} + +| Sign | Means | +| --- | --- | +| `ScalarIndexQuery` | the predicate used a scalar index | +| `LanceScan` with high `bytes_read` | no index, or too many columns selected | +| `FilterExec` above a search | the filter ran after search rather than pushing into it | +| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used | + +A `LanceScan` feeding a join is the common one worth acting on — see +[Joins](/sql/join#performance). + +## DESCRIBE {#describe} + +```sql +DESCRIBE documents +``` + +The table's columns and types, without a plan. diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx index 91bed36da..3e4056308 100644 --- a/docs/web/sql/index.mdx +++ b/docs/web/sql/index.mdx @@ -1,63 +1,85 @@ --- -title: "SQL" +title: "SQL reference" sidebarTitle: "Overview" -description: "Analytical SQL over the lakehouse: the dialect, where it runs, and what it can reach." +description: "The SQL surface: dialect, where a query runs, and a page per construct." icon: table-list -keywords: ["sql", "datafusion", "flight sql", "analytics", "dialect"] +keywords: ["sql", "datafusion", "flight sql", "reference", "dialect"] --- Enterprise SQL runs on a deployment. A local database has no engine to plan a statement, so `execute_query` raises there -- see [SDKs and SQL](/basics/apis). -The dialect is [Apache DataFusion's](/sql/functions), so anything its reference -documents for that version works here unchanged. What these pages cover is the -parts that are LanceDB's: how a query reaches the engine, how Lance columns -appear to it, and the extensions a lakehouse over multimodal data needs. +This is a reference rather than a guide: one page per construct, syntax first. +The guides elsewhere in these docs show the same operations from the SDKs. + +## The dialect {#dialect} + +LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org), +currently **version 54**. The grammar, built-in functions and type system are +DataFusion's, so anything its reference documents for that version works here +unchanged and is not reproduced. + +What these pages add is the part that is LanceDB's: how a query reaches the +engine, how Lance columns appear to it, and the statements a lakehouse needs +that DataFusion has no concept of. ## Where a query runs {#execution} -Queries execute on the server. The client sends a statement and receives Arrow -record batches -- nothing is converted to pandas or Python objects unless you -ask for it. A query feeding training or downstream processing therefore pays no -serialization cost to get there. +Queries execute on the server and results come back as Arrow record batches -- +nothing is converted to pandas or Python objects unless you ask. `execute_query` +hands back a reader as soon as the first batch is ready rather than when the +query finishes, so a large result set does not have to fit in memory before you +start reading it. See [Query and job](/basics/jobs#queries). -Results stream. `execute_query` hands back a reader as soon as the first batch -is ready rather than when the query finishes, so a large result set does not -have to fit in memory before you can start reading it. See -[Query and job](/basics/jobs#queries). +Served over an [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) +endpoint, so any Flight SQL client reaches it. -## What differs from a warehouse {#differences} +## Reference {#reference} -Two things, and both come from the storage being Lance rather than something -SQL-shaped: - -- **Columns are Arrow types**, not a SQL type system mapped onto storage. A - vector column is a `FixedSizeList` and behaves like one. See - [Data types](/basics/data-types). -- **Row metadata is available as columns** -- `_rowid`, `_rowaddr`, - `_rowoffset` -- but only when named, so they stay out of `SELECT *`. - -## The statements {#statements} +**Reading data** | | | | --- | --- | +| [Data types](/sql/data-types) | how Lance columns appear to SQL, and pseudo-columns | | [SELECT](/sql/select) | projection, filtering, ordering, set operations, CTEs | -| [Aggregation](/sql/aggregation) | `GROUP BY`, `HAVING`, aggregate functions | -| [Joins](/sql/join) | inner, outer, cross, and joining across namespaces | -| [Window functions](/sql/window) | `OVER (...)`, ranking, running totals | -| [Sampling](/sql/sampling) | `TABLESAMPLE`, for a random subset | -| [Functions and types](/sql/functions) | the function set, and what LanceDB adds | +| [Subqueries](/sql/subqueries) | scalar, `IN`, `EXISTS`, correlated, derived tables | +| [Joins](/sql/join) | inner, outer, cross, and namespace-qualified paths | +| [Aggregation](/sql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set | +| [Window functions](/sql/window) | `OVER`, framing, ranking and offset functions | +| [Sampling](/sql/sampling) | `TABLESAMPLE`, by rows or by fragments | -Beyond reading, SQL also writes (`INSERT`, `UPDATE`, `DELETE`, `COPY`), manages -the catalog (`CREATE TABLE`, `CREATE VIEW`, `DROP`, `SHOW`) and drives feature -engineering (`CREATE FUNCTION`, `REFRESH COLUMN`, `REFRESH MATERIALIZED VIEW`). -Those are documented with the operation rather than here -- see -[Management](/management) and [Feature engineering](/feature-engineering). +**Changing things** - -Vector search has no SQL form. It is a query-builder operation -- see -[Vector search](/search/vector-search). Full-text search does: the -[`fts()`](/sql/functions#fts) table function and the -[`contains_tokens`](/sql/functions#contains-tokens) predicate. - +| | | +| --- | --- | +| [DDL](/sql/ddl) | tables, views, indexes, functions, columns, WAL | +| [DML](/sql/dml) | `INSERT`, `UPDATE`, `DELETE`, `COPY` | +| [REFRESH](/sql/refresh) | filling computed columns and materialized views | + +**Introspection** + +| | | +| --- | --- | +| [SHOW](/sql/show) | jobs, functions, materialized views, errors | +| [EXPLAIN](/sql/explain) | plans, and measuring them | + +**Expressions** + +| | | +| --- | --- | +| [Operators](/sql/operators) | comparison, logical, arithmetic, literals, precedence | +| [Functions](/sql/functions) | the function set, and what Lance accelerates | + +## What SQL cannot do {#cannot} + +| | | +| --- | --- | +| Vector search | no SQL form -- a query-builder operation | +| Merge insert | no SQL form -- needs a source relation and a per-match policy | +| Run on a local database | no engine to plan the statement | + +Full-text search **is** available, as the [`fts()`](/sql/functions#fts) table +function and the [`contains_tokens`](/sql/functions#contains-tokens) predicate. + +See [Limits](/basics/limits#sql) for the bounds these pages enforce. diff --git a/docs/web/sql/operators.mdx b/docs/web/sql/operators.mdx new file mode 100644 index 000000000..804ed2ea7 --- /dev/null +++ b/docs/web/sql/operators.mdx @@ -0,0 +1,81 @@ +--- +title: "Operators and literals" +sidebarTitle: "Operators" +description: "Comparison, logical and arithmetic operators, literals, and precedence." +icon: plus-minus +keywords: ["operators", "literals", "precedence", "null", "cast", "interval"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +The operator set is DataFusion's. This page covers what is worth knowing when +the operands are Lance columns. + +## Comparison {#comparison} + +| | | +| --- | --- | +| `=` `!=` `<>` `<` `<=` `>` `>=` | ordinary comparison | +| `IS NULL` / `IS NOT NULL` | null test — `= NULL` is never true | +| `IS DISTINCT FROM` | comparison that treats null as a value | +| `BETWEEN a AND b` | inclusive range | +| `IN (...)` | set membership | +| `LIKE` / `ILIKE` | pattern match, `%` and `_` | +| `~` `~*` `!~` | regular expression match | + + +For keyword matching on a text column, [`contains_tokens`](/sql/functions#contains-tokens) +is index-accelerated where `LIKE` is not. Reach for `LIKE` only when you need +its wildcard semantics. + + +## Logical {#logical} + +`AND`, `OR`, `NOT`. Three-valued: a comparison against null is **unknown**, and +`NOT unknown` is still unknown. A `WHERE` clause keeps only rows where the +predicate is true, so unknown behaves like false there — but not inside `NOT +IN`, which is the usual place this bites. See +[Subqueries](/sql/subqueries#in-exists). + +## Arithmetic {#arithmetic} + +`+` `-` `*` `/` `%`, with the usual precedence. String concatenation is `||`. + + +Arithmetic over a vector column is **list** arithmetic, not scalar arithmetic — +a vector is a `FixedSizeList`. When the operation you want is a +similarity comparison, use [vector search](/search/vector-search); there is no +SQL form for it. + + +## Literals {#literals} + +| | | +| --- | --- | +| String | `'text'` — doubled quote to escape: `'it''s'` | +| Number | `42`, `3.14` | +| Boolean | `TRUE`, `FALSE` | +| Null | `NULL` | +| Date / timestamp | `DATE '2026-01-01'`, `TIMESTAMP '2026-01-01 12:00:00'` | +| Interval | `INTERVAL '7 days'` | +| Array | `[1, 2, 3]` | + +Identifiers are quoted with double quotes when they need to be — a reserved word +or a name with mixed case: `SELECT "order" FROM "MyTable"`. + +## Casting {#casting} + +```sql +SELECT CAST(identifier AS VARCHAR), identifier::VARCHAR FROM documents +``` + +Both forms work. `TRY_CAST` returns null instead of erroring on a value that +will not convert, which is the one to use over a column you do not control. + +## Precedence {#precedence} + +Highest to lowest: `::` → unary `-` → `* / %` → `+ -` → `||` → comparison → +`IS` → `NOT` → `AND` → `OR`. + +Parenthesize anything mixing `AND` and `OR`. `a OR b AND c` is `a OR (b AND c)`, +which is rarely what was meant. diff --git a/docs/web/sql/refresh.mdx b/docs/web/sql/refresh.mdx new file mode 100644 index 000000000..3208261eb --- /dev/null +++ b/docs/web/sql/refresh.mdx @@ -0,0 +1,95 @@ +--- +title: "REFRESH" +sidebarTitle: "Refresh" +description: "Filling computed columns and bringing materialized views up to date." +icon: arrows-rotate +keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +LanceDB extensions with no DataFusion equivalent. They are what makes +[feature engineering](/feature-engineering) reachable from SQL: a column +declared by [DDL](/sql/ddl#computed-column) starts empty, and a refresh fills it. + +## REFRESH COLUMN {#refresh-column} + +```sql +REFRESH COLUMN ON [TABLE] + [WHERE ] + [WITH (
[FOR JOB ''] [LIMIT ] +``` + +```sql +SHOW ERRORS ON documents +SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100 +``` + +Why rows failed. This is the statement to reach for when a +[refresh](/sql/refresh) reports fewer filled rows than you expected. + +**Errors and limits** + +| | | +| --- | --- | +| `LIMIT` | must be between 1 and 100,000 | +| Failed jobs scanned without `FOR JOB` | 32 most recent | +| Tables the server does not open directly | not supported | + +Without `FOR JOB` this surveys recent failures and stops at 32 jobs. When the +listing stops at your `LIMIT`, the message says so — narrow with `FOR JOB` +rather than raising the limit blindly. + +## See also {#see-also} + + + + The SDK listing calls, per object. + + + What a job handle exposes beyond these listings. + + diff --git a/docs/web/sql/subqueries.mdx b/docs/web/sql/subqueries.mdx new file mode 100644 index 000000000..d35c4c89f --- /dev/null +++ b/docs/web/sql/subqueries.mdx @@ -0,0 +1,76 @@ +--- +title: "Subqueries" +sidebarTitle: "Subqueries" +description: "Queries inside queries: scalar, IN, EXISTS, correlated, and derived tables." +icon: code-branch +keywords: ["subquery", "exists", "in", "correlated", "derived table", "lateral"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +A subquery is allowed anywhere an expression or a relation is. + +## As a relation {#derived} + +```sql +SELECT * FROM ( + SELECT category, count(*) AS n FROM documents GROUP BY category +) AS counts +WHERE n > 100 +``` + +A derived table needs an alias. It can be [sampled](/sql/sampling) like a base +table: + +```sql +SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10) +``` + +## As a scalar {#scalar} + +```sql +SELECT title, (SELECT count(*) FROM revisions r WHERE r.doc_id = d.id) AS revisions +FROM documents AS d +``` + +A scalar subquery must return one row and one column. Returning more is an +error rather than a silently-taken first row. + +## IN and EXISTS {#in-exists} + +```sql +SELECT title FROM documents +WHERE category_id IN (SELECT id FROM categories WHERE active) + +SELECT title FROM documents AS d +WHERE EXISTS (SELECT 1 FROM revisions r WHERE r.doc_id = d.id) +``` + +`EXISTS` stops at the first matching row, so it is the cheaper test when you +only care whether anything matches. `IN` builds the full set. + +`NOT IN` and `NOT EXISTS` differ on nulls: `NOT IN` against a set containing a +null yields no rows, because the comparison is unknown rather than false. Prefer +`NOT EXISTS` unless you have ruled nulls out. + +## Correlated subqueries {#correlated} + +A subquery referencing a column from the enclosing query — `d.id` above — is +correlated, and is conceptually evaluated per outer row. The planner usually +rewrites it into a join. + + +Check with [`EXPLAIN`](/sql/explain) when a correlated subquery is slow. A plan +showing a join has been optimized; one showing a repeated scan has not, and +rewriting it as an explicit [join](/sql/join) is the fix. + + +## CTEs as an alternative {#ctes} + +A [CTE](/sql/select#ctes) is a named subquery, and is usually more readable when +the same subquery appears twice or when nesting gets deep. + +```sql +WITH active AS (SELECT id FROM categories WHERE active) +SELECT title FROM documents WHERE category_id IN (SELECT id FROM active) +```