mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-15 08:12:26 +00:00
docs: bring the open-source documentation into this repository (#4122)
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
This commit is contained in:
Jack Ye
co-authored by
Brendan Clement
Brendan Clement
parent
de005f1d61
commit
47b7babc08
@@ -0,0 +1,154 @@
|
||||
---
|
||||
title: "SQL data types"
|
||||
sidebarTitle: "Data Types"
|
||||
description: "How Lance columns appear to SQL, including the types SQL has no name for."
|
||||
icon: shapes
|
||||
keywords: ["sql types", "arrow", "cast", "pseudo-columns", "vector", "blob", "json"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
There is no SQL type layer over Lance storage. The table provider hands
|
||||
DataFusion the dataset's Arrow schema directly, so a column is the same object
|
||||
to the query engine as it is to the format.
|
||||
|
||||
The consequence is that SQL type names here are **spellings of Arrow types**,
|
||||
not a system that maps onto them. [Data types](/sql/data-types) documents the
|
||||
storage side; this page covers what changes when you reach a column from SQL.
|
||||
|
||||
## The Arrow types a column can be {#supported}
|
||||
|
||||
| Category | Types |
|
||||
| --- | --- |
|
||||
| Null | `null` |
|
||||
| Boolean | `bool` |
|
||||
| Integer | `int8`, `int16`, `int32`, `int64` and their `uint` counterparts |
|
||||
| Floating point | `float16`, `float32`, `float64` |
|
||||
| String | `utf8`, `large_utf8` |
|
||||
| Binary | `binary`, `large_binary`, `fixed_size_binary(n)` |
|
||||
| Date and time | `date32`, `date64`, `time32`, `time64`, `timestamp` (with or without timezone), `duration` |
|
||||
| Nested | `struct`, `list`, `large_list`, `fixed_size_list`, `map` |
|
||||
| Dictionary | `dictionary(key, value)` |
|
||||
|
||||
Two limits are worth stating because they surface as schema errors rather than
|
||||
as documentation:
|
||||
|
||||
- A `map` with `keys_sorted = true` is rejected. Sorted-key maps do not survive
|
||||
the round trip through the Python bindings, so they are refused rather than
|
||||
silently unsorted.
|
||||
- Not every type is legal as a `fixed_size_list` child. The vector case is,
|
||||
which is what matters in practice.
|
||||
|
||||
## SQL names for Arrow types {#names}
|
||||
|
||||
| SQL | Arrow |
|
||||
| --- | --- |
|
||||
| `BOOLEAN` | `Boolean` |
|
||||
| `TINYINT` `SMALLINT` `INT` `BIGINT` | `Int8` `Int16` `Int32` `Int64` |
|
||||
| `UTINYINT` … `UBIGINT` | the unsigned counterparts |
|
||||
| `REAL` / `FLOAT` | `Float32` |
|
||||
| `DOUBLE` | `Float64` |
|
||||
| `VARCHAR` / `TEXT` / `STRING` | `Utf8` |
|
||||
| `BYTEA` / `BINARY` | `Binary` |
|
||||
| `DATE` | `Date32` |
|
||||
| `TIME` | `Time64` |
|
||||
| `TIMESTAMP` | `Timestamp`, with or without a timezone |
|
||||
| `INTERVAL` | `Interval` |
|
||||
| `STRUCT` / `LIST` / `MAP` | the nested equivalents |
|
||||
|
||||
DataFusion's [type reference](https://datafusion.apache.org/user-guide/sql/data_types.html)
|
||||
describes the full mapping.
|
||||
|
||||
## Types SQL has no name for {#no-name}
|
||||
|
||||
Three column kinds you will meet that no SQL type name covers.
|
||||
|
||||
### Vectors {#vectors}
|
||||
|
||||
A dense vector column has a fixed dimension, commonly represented as
|
||||
`FixedSizeList<Float32, dim>`. Other supported vector element types are
|
||||
possible; inspect the column with [`arrow_typeof`](/sql/scalar-functions#type) rather than assuming Float32.
|
||||
It is selectable and testable for nullness:
|
||||
|
||||
```sql
|
||||
SELECT id FROM documents WHERE vector IS NOT NULL
|
||||
```
|
||||
|
||||
<Warning>
|
||||
A list representation does not imply that ordinary arithmetic operators
|
||||
perform vector arithmetic. Use supported distance functions for similarity
|
||||
calculations. SQL vector search depends on the deployment version and query
|
||||
plan; see [SQL](/basics/sql#no-vector-sql).
|
||||
</Warning>
|
||||
|
||||
### JSON {#json}
|
||||
|
||||
Stored as JSONB in `large_binary` with the `lance.json` extension name. It reads
|
||||
through accessor functions rather than a `->` operator:
|
||||
|
||||
```sql
|
||||
SELECT json_get_string(data, 'name') FROM events
|
||||
WHERE json_get_int(data, 'status') = 200
|
||||
```
|
||||
|
||||
`json_get`, its typed variants, and `json_extract` are **index-accelerated** by a
|
||||
JSON scalar index. See [Functions](/sql/scalar-functions#json).
|
||||
|
||||
### Blobs {#blobs}
|
||||
|
||||
A blob column is read either as its binary contents or as a **description** — a
|
||||
pointer plus metadata, without fetching the bytes. That choice changes the
|
||||
column's Arrow type, and therefore the schema your query sees.
|
||||
|
||||
<Warning>
|
||||
The default reads blob columns as descriptions and other binary columns as
|
||||
binary. A query selecting a blob column and expecting bytes gets a struct.
|
||||
</Warning>
|
||||
|
||||
### bfloat16 {#bfloat16}
|
||||
|
||||
Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`.
|
||||
Useful as a vector element type when halving the memory matters more than the
|
||||
precision does.
|
||||
|
||||
## Geospatial {#geospatial}
|
||||
|
||||
<Warning>
|
||||
Lance defines geospatial types and `st_*` functions on
|
||||
[GeoArrow](https://geoarrow.org), but they sit behind an optional build feature
|
||||
that **LanceDB does not ship**. Calling `st_intersects` and friends returns an
|
||||
error naming the missing feature rather than an unknown-function error, and the
|
||||
spatial index is gated the same way.
|
||||
</Warning>
|
||||
|
||||
Geometry can still be stored -- it is binary, and binary columns work -- but
|
||||
nothing interprets it. Filter on a bounding box held in ordinary numeric
|
||||
columns if you need spatial predicates today.
|
||||
|
||||
## Pseudo-columns {#pseudo-columns}
|
||||
|
||||
Availability depends on the SQL table provider. The ordinary table provider
|
||||
exposes the stored dataset schema; naming a Lance scan metadata field does
|
||||
not make it available as a SQL column. The `fts()` provider adds `_score` to
|
||||
its schema, so `_score` is included in `SELECT * FROM fts(...)`.
|
||||
|
||||
| Column | Type | |
|
||||
| --- | --- | --- |
|
||||
| `_rowid` | `UInt64` | Lance scan row identifier; stability requires stable row ids |
|
||||
| `_rowaddr` | `UInt64` | physical row address |
|
||||
| `_rowoffset` | `UInt64` | offset within the fragment |
|
||||
| `_score` | `Float32` | relevance, from [`fts()`](/sql/functions#fts) |
|
||||
|
||||
`_rowid`, `_rowaddr`, and `_rowoffset` describe Lance scan metadata, not
|
||||
columns automatically registered on an ordinary SQL table. Select `_score`
|
||||
from an `fts()` relation explicitly when you want a projection containing it.
|
||||
|
||||
## Casting {#casting}
|
||||
|
||||
```sql
|
||||
SELECT CAST(identifier AS VARCHAR) FROM documents
|
||||
SELECT TRY_CAST(maybe_number AS BIGINT) FROM raw
|
||||
```
|
||||
|
||||
`TRY_CAST` yields null rather than failing the query on a value that will not
|
||||
convert. See [Operators](/sql/operators#casting).
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
title: "Function"
|
||||
sidebarTitle: "Function"
|
||||
description: "CREATE FUNCTION, SHOW FUNCTIONS, and DROP FUNCTION."
|
||||
icon: function
|
||||
keywords: ["create function", "drop function", "show functions", "sql", "udf"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
## CREATE FUNCTION {#create}
|
||||
|
||||
```sql
|
||||
CREATE FUNCTION f(x BIGINT) RETURNS BIGINT
|
||||
LANGUAGE python
|
||||
AS 'def f(x): return x * 2'
|
||||
WITH (pip = 'numpy==1.26.4', python_version = '3.11')
|
||||
```
|
||||
|
||||
The body is Python source. Its dependencies are declared with it and are part
|
||||
of the **immutable function version** -- changing one produces a new version
|
||||
rather than altering the registered one, so a column filled by an earlier
|
||||
version is unaffected.
|
||||
|
||||
| Option | |
|
||||
| --- | --- |
|
||||
| `pip` | pip requirements for the worker environment |
|
||||
| `conda` | conda packages instead of pip |
|
||||
| `conda_channels` | channels in priority order; requires `conda` |
|
||||
| `python_version` | worker Python version, defaulting to yours |
|
||||
| `gpu` | `'true'` when every execution needs a GPU |
|
||||
|
||||
**Errors and limits**
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `LANGUAGE` other than `python` | refused |
|
||||
| `pip` and `conda` together | refused -- pick one |
|
||||
| `conda_channels` without `conda` | refused |
|
||||
| `gpu` set to anything but `'true'` | refused |
|
||||
| An empty body | refused |
|
||||
| Any other option | refused, and the message lists what is supported |
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `create_function`. Registration returns a job because
|
||||
the function is baked into an image before the version becomes usable.
|
||||
</Note>
|
||||
|
||||
## SHOW FUNCTIONS {#show}
|
||||
|
||||
```sql
|
||||
SHOW FUNCTIONS
|
||||
```
|
||||
|
||||
Lists registered functions and their versions. Since a version is immutable
|
||||
and a column pins the version that fills it, this is how you find which
|
||||
versions are still in use before dropping one.
|
||||
|
||||
## DROP FUNCTION {#drop}
|
||||
|
||||
```sql
|
||||
DROP FUNCTION embed WITH (version = '1')
|
||||
DROP FUNCTION IF EXISTS embed WITH (version = '1')
|
||||
DROP FUNCTION embed WITH (version = '1', cascade = 'true')
|
||||
```
|
||||
|
||||
`version` is required -- a function is a set of immutable versions, and
|
||||
dropping is always of one specific version, never "the function" as a whole.
|
||||
|
||||
<Warning>
|
||||
Without `cascade = 'true'`, dropping a version that a column still pins fails
|
||||
rather than breaking that column silently. Passing it drops the version and
|
||||
lets columns pinned to it stop refreshing -- they keep their last-filled
|
||||
values, per [Guarantees](/basics/guarantees#jobs).
|
||||
</Warning>
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Function" icon="function" href="/management/function">
|
||||
Registering one from the SDK, and diagnosing environment mismatches.
|
||||
</Card>
|
||||
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/add-computed-column">
|
||||
Filling a column with a function instead of an expression.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
title: "Index"
|
||||
sidebarTitle: "Index"
|
||||
description: "CREATE INDEX and DROP INDEX — one column, server-chosen name."
|
||||
icon: bolt
|
||||
keywords: ["create index", "drop index", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
## CREATE INDEX {#create}
|
||||
|
||||
```sql
|
||||
CREATE INDEX ON documents (title)
|
||||
```
|
||||
|
||||
<Note>
|
||||
SQL `CREATE INDEX` covers **one column** and takes **no user-chosen name** --
|
||||
the server names it. For a multi-column or explicitly-named index, or to
|
||||
choose the algorithm, use the SDK's
|
||||
[`create_index`](/indexing).
|
||||
</Note>
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `create_ivf_pq`, `create_btree`, `create_fts`, and one
|
||||
more per algorithm -- the type names which algorithm is building, not a
|
||||
different feature. See [Build & Repair](/indexing/reindexing).
|
||||
</Note>
|
||||
|
||||
## DROP INDEX {#drop}
|
||||
|
||||
```sql
|
||||
DROP INDEX documents.title_idx
|
||||
```
|
||||
|
||||
Takes the qualified `table.index` name. Find it with the SDK's
|
||||
[`list_indices`](/indexing) if you created the index without
|
||||
naming it yourself.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Indexing" icon="bolt" href="/indexing">
|
||||
Choosing an index type from the SDK.
|
||||
</Card>
|
||||
<Card title="Build & Repair" icon="refresh" href="/indexing/reindexing">
|
||||
Keeping an index current as data arrives.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: "Job"
|
||||
sidebarTitle: "Job"
|
||||
description: "SHOW JOBS, SHOW ERRORS, and controlling a running job with PAUSE, RESUME and CANCEL."
|
||||
icon: hourglass-half
|
||||
keywords: ["show jobs", "show errors", "pause job", "resume job", "cancel job", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A [job](/basics/jobs) is not a catalog object -- it has no schema and is not
|
||||
namespaced -- but it has its own SQL statements, grouped here the same way the
|
||||
catalog objects are.
|
||||
|
||||
## SHOW JOBS {#show}
|
||||
|
||||
```sql
|
||||
SHOW JOBS
|
||||
```
|
||||
|
||||
Every job the server knows about -- id, type, state, creation time. The SQL
|
||||
counterpart of `list_jobs`.
|
||||
|
||||
## SHOW ERRORS {#errors}
|
||||
|
||||
```sql
|
||||
SHOW ERRORS ON <table> [FOR JOB '<job-id>'] [LIMIT <n>]
|
||||
```
|
||||
|
||||
```sql
|
||||
SHOW ERRORS ON documents
|
||||
SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100
|
||||
```
|
||||
|
||||
Why rows failed. Reach for this when a [refresh](/sql/dml/refresh) reports
|
||||
fewer filled rows than you expected.
|
||||
|
||||
**Errors and limits**
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `LIMIT` | must be between 1 and 100,000 |
|
||||
| Failed jobs scanned without `FOR JOB` | 32 most recent |
|
||||
| Tables the server does not open directly | not supported |
|
||||
|
||||
Without `FOR JOB` this surveys recent failures and stops at 32 jobs. When the
|
||||
listing stops at your `LIMIT`, the message says so -- narrow with `FOR JOB`
|
||||
rather than raising the limit blindly.
|
||||
|
||||
## PAUSE JOB {#pause}
|
||||
|
||||
```sql
|
||||
PAUSE JOB 'j-ci-abc-def'
|
||||
```
|
||||
|
||||
## RESUME JOB {#resume}
|
||||
|
||||
```sql
|
||||
RESUME JOB 'j-ci-abc-def'
|
||||
```
|
||||
|
||||
## CANCEL JOB {#cancel}
|
||||
|
||||
```sql
|
||||
CANCEL JOB 'j-ci-abc-def'
|
||||
```
|
||||
|
||||
The job id is always a quoted string literal. All three take exactly one and
|
||||
nothing after it -- a second token on the line is a syntax error rather than
|
||||
being ignored.
|
||||
|
||||
<Note>
|
||||
Cancellation is a request, matching the SDK's
|
||||
[`cancel()`](/basics/jobs#managing): it does not guarantee the job stops before
|
||||
doing more work, and asking twice or cancelling a finished job does nothing.
|
||||
`CANCEL JOB` itself can be unavailable on a deployment with no job registry
|
||||
configured, which is a deployment condition rather than something you can work
|
||||
around from SQL.
|
||||
</Note>
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Query and job" icon="hourglass-half" href="/basics/jobs">
|
||||
The handle these statements are the SQL view of.
|
||||
</Card>
|
||||
<Card title="Background jobs" icon="gears" href="/maintenance/background-jobs">
|
||||
Job types you did not start yourself.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
title: "Materialized View"
|
||||
sidebarTitle: "Materialized View"
|
||||
description: "CREATE MATERIALIZED VIEW and SHOW MATERIALIZED VIEWS; refreshing is a REFRESH statement."
|
||||
icon: table
|
||||
keywords: ["materialized view", "create materialized view", "show materialized views", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A materialized view stores a query's result as rows, kept current by a refresh
|
||||
-- the counterpart to a [view](/sql/ddl/view), which stores only the query.
|
||||
|
||||
<Note>
|
||||
Materialized views are off by default on a deployment and have to be enabled
|
||||
there. Which statements and query shapes a deployment accepts, whether it needs
|
||||
the source table to have stable row ids, and how it refreshes a view depend on
|
||||
its LanceDB Enterprise version and configuration.
|
||||
</Note>
|
||||
|
||||
## CREATE MATERIALIZED VIEW {#create}
|
||||
|
||||
```sql
|
||||
CREATE MATERIALIZED VIEW <name> AS SELECT ...
|
||||
```
|
||||
|
||||
On a `db://` connection, the SDK's
|
||||
[`create_materialized_view`](/management/materialized-view#create) sends this
|
||||
statement. Once created, a view is queried like a table, refreshed with SQL and
|
||||
listed with SQL.
|
||||
|
||||
## SHOW MATERIALIZED VIEWS {#show}
|
||||
|
||||
```sql
|
||||
SHOW MATERIALIZED VIEWS
|
||||
```
|
||||
|
||||
<Note>
|
||||
Cheaper than the SDK equivalent. `list_materialized_views` finds views by
|
||||
reading every table's schema, costing one open per table; this asks the server.
|
||||
</Note>
|
||||
|
||||
## Refreshing one {#refresh}
|
||||
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW chunks
|
||||
```
|
||||
|
||||
See [REFRESH](/sql/dml/refresh#refresh-materialized-view) for the full syntax,
|
||||
including `CASCADE` and `full = true`.
|
||||
|
||||
## Dropping one {#drop}
|
||||
|
||||
A materialized view **is** a table, so it is dropped with
|
||||
[`DROP TABLE`](/sql/ddl/table#drop) -- which is also why dropping one removes
|
||||
stored rows, unlike dropping a [view](/sql/ddl/view#drop).
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `mat_view_refresh` for a refresh, or `sql` on a
|
||||
deployment that rebuilds views through SQL; `drop_table` for dropping one, same
|
||||
as any table.
|
||||
</Note>
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Materialized View" icon="table" href="/management/materialized-view">
|
||||
Creating one from the SDK.
|
||||
</Card>
|
||||
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/materialized-views">
|
||||
How a refresh works, locally and on a deployment.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
title: "Namespace"
|
||||
sidebarTitle: "Namespace"
|
||||
description: "CREATE SCHEMA and DROP SCHEMA — the SQL spelling of namespace DDL."
|
||||
icon: folder-tree
|
||||
keywords: ["namespace", "schema", "create schema", "drop schema", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A [namespace](/management/namespace) is addressed in SQL as a `SCHEMA` -- the
|
||||
standard's name for the same concept.
|
||||
|
||||
## CREATE SCHEMA {#create}
|
||||
|
||||
```sql
|
||||
CREATE SCHEMA public
|
||||
CREATE SCHEMA db1.ns1.ns2
|
||||
CREATE SCHEMA IF NOT EXISTS `ns1$ns2`
|
||||
```
|
||||
|
||||
A dotted path creates the whole chain: `db1.ns1.ns2` creates `ns2` inside
|
||||
`ns1` inside `db1`, provided the parents already exist. A name containing the
|
||||
namespace path delimiter itself is backtick-quoted, as in `` `ns1$ns2` `` above.
|
||||
|
||||
`IF NOT EXISTS` makes the statement a no-op rather than an error when the
|
||||
namespace is already there.
|
||||
|
||||
## DROP SCHEMA {#drop}
|
||||
|
||||
```sql
|
||||
DROP SCHEMA db1.public
|
||||
DROP SCHEMA IF EXISTS `a$b`
|
||||
DROP SCHEMA `a$b` RESTRICT
|
||||
```
|
||||
|
||||
<Warning>
|
||||
`CASCADE` is rejected: **"DROP SCHEMA CASCADE is not supported; drop the
|
||||
objects inside first."** A namespace's tables, views and functions must be
|
||||
dropped explicitly before the namespace itself can go. `RESTRICT` -- the
|
||||
default -- states that requirement rather than changing it.
|
||||
</Warning>
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Namespace" icon="folder-tree" href="/management/namespace">
|
||||
The SDK equivalent, and how a namespace fits the catalog hierarchy.
|
||||
</Card>
|
||||
<Card title="Table" icon="table" href="/sql/ddl/table">
|
||||
Creating tables inside a namespace.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,178 @@
|
||||
---
|
||||
title: "Table"
|
||||
sidebarTitle: "Table"
|
||||
description: "CREATE, DESCRIBE, ALTER, CLONE and DROP TABLE."
|
||||
icon: table
|
||||
keywords: ["create table", "alter table", "drop table", "describe", "clone", "ddl"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
## CREATE TABLE {#create}
|
||||
|
||||
```sql
|
||||
CREATE TABLE documents (id BIGINT, title VARCHAR, created_at TIMESTAMP)
|
||||
CREATE TABLE recent AS SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
|
||||
```
|
||||
|
||||
Column types are Arrow types under SQL names -- see [Data types](/sql/data-types).
|
||||
A table inside a [namespace](/sql/ddl/namespace) is written as a dotted path:
|
||||
`prod.events.clicks`.
|
||||
|
||||
## DESCRIBE {#describe}
|
||||
|
||||
```sql
|
||||
DESCRIBE documents
|
||||
```
|
||||
|
||||
The table's columns and types, without a plan -- the SQL counterpart of
|
||||
`table.schema`.
|
||||
|
||||
## ALTER TABLE {#alter}
|
||||
|
||||
`ALTER TABLE ... ADD COLUMN` covers three different things, distinguished by
|
||||
what follows the column name.
|
||||
|
||||
### A plain column {#plain-column}
|
||||
|
||||
```sql
|
||||
ALTER TABLE t ADD COLUMN c INT
|
||||
```
|
||||
|
||||
Added empty and nullable. Nothing is computed.
|
||||
|
||||
### A computed column {#computed-column}
|
||||
|
||||
```sql
|
||||
ALTER TABLE t ADD COLUMN doubled AS value * 2
|
||||
```
|
||||
|
||||
`AS <expression>` declares a [computed column](/feature-engineering/add-computed-column).
|
||||
The column is committed empty and the expression is stored, not evaluated --
|
||||
declaring one on a billion-row table costs what declaring one on an empty table
|
||||
costs. [`REFRESH COLUMN`](/sql/dml/refresh) fills it.
|
||||
|
||||
### A function column {#function-column}
|
||||
|
||||
```sql
|
||||
ALTER TABLE t ADD COLUMN embedded AS embed(name) WITH (version = 'fv_1')
|
||||
```
|
||||
|
||||
When the expression calls a [registered function](/sql/ddl/function), the
|
||||
column is filled by running that function on workers. `version` pins which
|
||||
immutable function version fills it, so a later registration does not silently
|
||||
change what the column means.
|
||||
|
||||
Several modifiers narrow what gets computed:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `FILTER (WHERE pred)` | only rows matching `pred` are computed |
|
||||
| `IGNORE NULLS` | rows with null inputs are skipped |
|
||||
| `OVER (PARTITION BY col)` | the function sees rows grouped by `col` |
|
||||
|
||||
A function returning several values fills several columns at once:
|
||||
|
||||
```sql
|
||||
ALTER TABLE t ADD COLUMN (label, score) AS classify(text) WITH (version = 'fv_1')
|
||||
```
|
||||
|
||||
```sql
|
||||
ALTER TABLE t ADD COLUMN c AS f(x) FILTER (WHERE keep) WITH (version = 'fv_1')
|
||||
ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2')
|
||||
```
|
||||
|
||||
<Note>
|
||||
`WITH` on a plain computed column accepts only `version`. Resource options such
|
||||
as `num_workers` apply to function columns, where there are workers to size.
|
||||
</Note>
|
||||
|
||||
### Table properties {#properties}
|
||||
|
||||
```sql
|
||||
ALTER TABLE t SET UNENFORCED PRIMARY KEY (id)
|
||||
```
|
||||
|
||||
Records which column identifies a row. **Unenforced** is not a hedge in the
|
||||
naming: uniqueness is not checked on write, and declaring it does not make
|
||||
duplicates impossible. It tells the planner what to assume.
|
||||
|
||||
```sql
|
||||
ALTER TABLE t SET WAL
|
||||
ALTER TABLE t SET WAL WITH (sharding = 'bucket', buckets = 4)
|
||||
ALTER TABLE t SET WAL WITH (maintained_indexes = '')
|
||||
ALTER TABLE t UNSET WAL
|
||||
```
|
||||
|
||||
A write-ahead log makes small frequent writes cheap by batching them out of the
|
||||
commit path. `sharding` and `buckets` spread the log so concurrent writers do
|
||||
not contend on one place; `maintained_indexes` chooses which indexes are kept
|
||||
current against WAL rows. See
|
||||
[Background jobs](/maintenance/background-jobs#wal) for how the log gets
|
||||
folded back into the table.
|
||||
|
||||
<Warning>
|
||||
[`TABLESAMPLE`](/sql/dql/sampling) is refused on a WAL-backed table, because
|
||||
rows a sample would draw from are not all in the fragments yet.
|
||||
</Warning>
|
||||
|
||||
**Errors and limits**
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `ALTER TABLE ... DROP COLUMN` | not supported |
|
||||
|
||||
Dropping, renaming, or retyping a column is an SDK operation --
|
||||
[`drop_columns`](/management/table#schema) and
|
||||
[`alter_columns`](/management/table#schema) -- not a SQL statement.
|
||||
|
||||
## CLONE {#clone}
|
||||
|
||||
```sql
|
||||
CREATE TABLE docs_snapshot CLONE documents
|
||||
CREATE TABLE docs_at_v42 CLONE documents AT 42
|
||||
CREATE TABLE docs_at_release CLONE documents AT TAG 'release-1'
|
||||
CREATE TABLE docs_yesterday CLONE documents AT TIMESTAMP '2026-09-01T00:00:00Z'
|
||||
```
|
||||
|
||||
A **shallow clone**: the new table shares the source's data files rather than
|
||||
copying them, so it costs one metadata commit regardless of the source's size.
|
||||
|
||||
`AT` pins the point in the source's history to clone. A bare integer is a
|
||||
version and a bare quoted string is a timestamp; `VERSION`, `TAG` and
|
||||
`TIMESTAMP` say so explicitly, and a tag always needs its keyword -- a bare
|
||||
quoted string cannot be both a tag name and a timestamp.
|
||||
|
||||
<Warning>
|
||||
The clone depends on the source's files continuing to exist. If the source is
|
||||
[compacted](/maintenance/cleanup) past the retention window, the version the
|
||||
clone points at can be pruned. The SDK's
|
||||
[`clone_table`](/management/table#rename-clone) also offers a deep clone for
|
||||
when the copy has to outlive the source's history.
|
||||
</Warning>
|
||||
|
||||
## DROP TABLE {#drop}
|
||||
|
||||
```sql
|
||||
DROP TABLE documents
|
||||
```
|
||||
|
||||
<Warning>
|
||||
Removes the table and every version of it. There is no undrop.
|
||||
</Warning>
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `drop_table`. See
|
||||
[Background jobs](/maintenance/background-jobs#teardown).
|
||||
</Note>
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Table" icon="table" href="/management/table">
|
||||
The SDK equivalent, and what a table carries.
|
||||
</Card>
|
||||
<Card title="Schema evolution" icon="boxes-stacked" href="/management/table#schema">
|
||||
Renaming, retyping and dropping columns from the SDK.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
title: "View"
|
||||
sidebarTitle: "View"
|
||||
description: "CREATE VIEW and DROP VIEW — a saved query, evaluated when read."
|
||||
icon: eye
|
||||
keywords: ["create view", "drop view", "sql", "catalog"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A view stores a **query**, not rows. Reading it runs the query against current
|
||||
data, so a view is never stale -- and never faster than the query it wraps. It
|
||||
is the counterpart to a [materialized view](/sql/ddl/materialized-view), which
|
||||
stores rows and is brought up to date by a refresh; both are
|
||||
[catalog objects](/basics/catalog) defined by a query, differing only in when
|
||||
that query runs.
|
||||
|
||||
There is no SDK method for a view, because the thing it stores *is* a SQL
|
||||
statement.
|
||||
|
||||
## CREATE VIEW {#create}
|
||||
|
||||
```sql
|
||||
CREATE VIEW "recent_matches" AS
|
||||
SELECT t1.floats
|
||||
FROM "documents" AS t1
|
||||
INNER JOIN "categories" AS t2 ON t1.ints = t2.ints
|
||||
```
|
||||
|
||||
The statement returns the view's schema -- column name, data type, nullability
|
||||
-- resolved from the query at creation time. Anything the
|
||||
[dialect](/sql/functions) supports can define one: joins, predicates,
|
||||
aggregates, window functions.
|
||||
|
||||
## Reading one {#reading}
|
||||
|
||||
A view is queried like a table:
|
||||
|
||||
```sql
|
||||
SELECT * FROM "recent_matches" WHERE floats > 0.5
|
||||
```
|
||||
|
||||
Predicates applied to the view compose with the query inside it rather than
|
||||
filtering its output afterwards, so a selective filter does not pay for rows
|
||||
the view would have produced and then discarded.
|
||||
|
||||
## DROP VIEW {#drop}
|
||||
|
||||
```sql
|
||||
DROP VIEW "recent_matches"
|
||||
```
|
||||
|
||||
Removes the saved query. Unlike dropping a table, no data is removed -- there
|
||||
was none to begin with.
|
||||
|
||||
## Listing {#listing}
|
||||
|
||||
Views and tables share a namespace, and both are things you can query. The
|
||||
catalog distinguishes them:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| view names | the views in the database |
|
||||
| table names | the tables |
|
||||
| relation names | everything queryable -- both |
|
||||
|
||||
Use the relation listing when the question is "what can I select from," and the
|
||||
separate listings when the distinction matters -- before dropping, for
|
||||
instance, since dropping a view removes only the saved query while dropping a
|
||||
table removes data.
|
||||
|
||||
## Choosing between a view and a materialized view {#choosing}
|
||||
|
||||
| | View | Materialized view |
|
||||
| --- | --- | --- |
|
||||
| Stores | the query | the rows |
|
||||
| Freshness | always current | as of the last refresh |
|
||||
| Read cost | the full query, every read | a table scan |
|
||||
| Write cost | none | a refresh |
|
||||
|
||||
Reach for a **view** when the query is cheap, or when reading stale rows would
|
||||
be wrong. Reach for a **materialized view** when the query is expensive and the
|
||||
data changes less often than it is read.
|
||||
|
||||
The privileges are separate too: `CREATE_VIEW` and `CREATE_MATERIALIZED_VIEW`
|
||||
are distinct grants on a namespace, so the ability to create one does not imply
|
||||
the other. See [Authorization](/governance/authorization).
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Materialized View" icon="table" href="/sql/ddl/materialized-view">
|
||||
The stored-rows counterpart.
|
||||
</Card>
|
||||
<Card title="Management" icon="folder-tree" href="/management">
|
||||
Where views fit among the other catalog objects.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: "Copy"
|
||||
sidebarTitle: "Copy"
|
||||
description: "COPY ... TO — writing a query result to object storage as files."
|
||||
icon: file-export
|
||||
keywords: ["copy", "export", "parquet", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
COPY (SELECT id, title FROM documents) TO 's3://bucket/export/' STORED AS PARQUET
|
||||
```
|
||||
|
||||
Writes a query result to object storage. This is the export path when the
|
||||
consumer is another engine rather than an application -- results land as files
|
||||
rather than streaming to a client.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<Card title="Insert" icon="plus" href="/sql/dml/insert">
|
||||
Bringing rows in, rather than exporting them.
|
||||
</Card>
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
title: "Delete"
|
||||
sidebarTitle: "Delete"
|
||||
description: "DELETE FROM ... WHERE — rows are marked, not erased."
|
||||
icon: trash
|
||||
keywords: ["delete", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
DELETE FROM documents WHERE created_at < '2025-01-01'
|
||||
```
|
||||
|
||||
Deletion marks rows rather than erasing them, so the data remains recoverable
|
||||
for as long as the version holding it survives
|
||||
[retention](/maintenance/cleanup).
|
||||
|
||||
Commits a new table version, so [versioning](/management/table#versioning) and
|
||||
the [write guarantees](/basics/guarantees#writes) apply -- including that the
|
||||
version advances even when the statement affects zero rows.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<Card title="Delete" icon="trash" href="/ingestion/delete">
|
||||
The SDK equivalent.
|
||||
</Card>
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
title: "Insert"
|
||||
sidebarTitle: "Insert"
|
||||
description: "INSERT INTO, including INSERT ... SELECT to move rows server-side."
|
||||
icon: plus
|
||||
keywords: ["insert", "insert into", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur')
|
||||
|
||||
INSERT INTO archive SELECT * FROM documents WHERE created_at < '2026-01-01'
|
||||
```
|
||||
|
||||
`INSERT ... SELECT` is the form worth knowing: it moves rows between tables
|
||||
server-side, without a round trip through your process.
|
||||
|
||||
Commits a new table version, so [versioning](/management/table#versioning) and
|
||||
the [write guarantees](/basics/guarantees#writes) apply -- including that the
|
||||
version advances even when the statement affects zero rows.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Insert" icon="plus" href="/ingestion/insert">
|
||||
The SDK equivalent.
|
||||
</Card>
|
||||
<Card title="Merge Insert" icon="code-merge" href="/ingestion/merge-insert">
|
||||
Upsert -- matching incoming rows against existing keys -- has no SQL form.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
title: "REFRESH"
|
||||
sidebarTitle: "REFRESH"
|
||||
description: "Filling computed columns and bringing materialized views up to date."
|
||||
icon: arrows-rotate
|
||||
keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
LanceDB extensions with no DataFusion equivalent. They are what makes
|
||||
[feature engineering](/feature-engineering) reachable from SQL: a column
|
||||
declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it.
|
||||
|
||||
## REFRESH COLUMN {#refresh-column}
|
||||
|
||||
```sql
|
||||
REFRESH COLUMN <column> ON [TABLE] <table>
|
||||
[WHERE <predicate>]
|
||||
[WITH (<option> = <value> [, ...])]
|
||||
```
|
||||
|
||||
```sql
|
||||
REFRESH COLUMN doubled ON my_table
|
||||
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
|
||||
```
|
||||
|
||||
A refresh gives values to rows that lack them, recomputes rows whose inputs
|
||||
changed, and **leaves other filled rows alone**. That makes it idempotent, cheap
|
||||
to repeat as data arrives, and safe to re-run after a failure -- though a re-run
|
||||
can recompute values the failed run had already written.
|
||||
|
||||
`WHERE` scopes the refresh, which is how you fill a slice and check it before
|
||||
committing to the whole table.
|
||||
|
||||
### Options {#refresh-column-options}
|
||||
|
||||
| Option | Meaning |
|
||||
| --- | --- |
|
||||
| `num_workers` | how many workers to use |
|
||||
| `cpu_millis` | CPU budget per worker |
|
||||
| `memory_bytes` | memory budget per worker |
|
||||
| `gpu_count` | GPUs per worker |
|
||||
| `batch_size` | rows a worker takes at a time |
|
||||
| `error_policy` | `'skip'` continues past failing rows instead of failing the job |
|
||||
|
||||
```sql
|
||||
REFRESH COLUMN embedding ON documents
|
||||
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
|
||||
```
|
||||
|
||||
`num_workers` sets a count; the budget options size each worker. They are
|
||||
alternatives rather than a pair.
|
||||
|
||||
<Note>
|
||||
`error_policy` and `num_workers` are refused for a column filled by a registered
|
||||
[function](/management/function) with its own resource budget -- use the CPU,
|
||||
memory and GPU options there.
|
||||
</Note>
|
||||
|
||||
## REFRESH MATERIALIZED VIEW {#refresh-materialized-view}
|
||||
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW <name> [CASCADE] [WITH (<option> = <value>)]
|
||||
```
|
||||
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW chunks
|
||||
REFRESH MATERIALIZED VIEW chunks WITH (full = true)
|
||||
```
|
||||
|
||||
`full = true` asks for every row to be recomputed. Without it, whether a
|
||||
refresh rebuilds the whole view -- as every local SDK refresh does -- or
|
||||
reconciles only what changed in the source depends on the deployment's LanceDB
|
||||
Enterprise version and configuration.
|
||||
|
||||
<Warning>
|
||||
`CASCADE` asks for the views this one is built on to be refreshed first, in
|
||||
dependency order. Whether a deployment carries it out depends on its version;
|
||||
where it does not, refresh each view yourself, its sources first.
|
||||
</Warning>
|
||||
|
||||
## Watching a refresh {#watching}
|
||||
|
||||
Both statements submit work that outlives the call. Track it with
|
||||
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
|
||||
[`SHOW ERRORS`](/sql/ddl/job#errors).
|
||||
|
||||
## See also {#see-also}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
||||
The same operation from the SDKs, and what a refresh guarantees.
|
||||
</Card>
|
||||
<Card title="Table" icon="table" href="/sql/ddl/table#alter">
|
||||
Declaring the columns this fills.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
title: "Update"
|
||||
sidebarTitle: "Update"
|
||||
description: "UPDATE ... SET ... WHERE — and why it does not recompute derived columns."
|
||||
icon: pen
|
||||
keywords: ["update", "sql"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
UPDATE documents SET title = upper(title) WHERE category = 'film'
|
||||
```
|
||||
|
||||
Assignments take any expression the [dialect](/sql/functions) supports,
|
||||
including references to other columns of the same row. Without a `WHERE`
|
||||
clause every row is updated.
|
||||
|
||||
<Warning>
|
||||
An update does not recompute [computed columns](/feature-engineering/add-computed-column)
|
||||
that derive from the column you changed. Those rows keep the value computed from
|
||||
the old input until the next refresh recomputes them -- see
|
||||
[Guarantees](/basics/guarantees#jobs).
|
||||
</Warning>
|
||||
|
||||
Commits a new table version, so [versioning](/management/table#versioning) and
|
||||
the [write guarantees](/basics/guarantees#writes) apply -- including that the
|
||||
version advances even when the statement affects zero rows.
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<Card title="Update" icon="pen" href="/ingestion/update">
|
||||
The SDK equivalent.
|
||||
</Card>
|
||||
@@ -0,0 +1,118 @@
|
||||
---
|
||||
title: "Aggregation"
|
||||
sidebarTitle: "Aggregation"
|
||||
description: "GROUP BY, HAVING, and the aggregate function set."
|
||||
icon: layer-group
|
||||
keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximate"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Aggregation is one of the two main reasons to reach for SQL rather than a query
|
||||
builder -- the SDKs project, filter and search, but they do not group. The other
|
||||
is [joins](/sql/dql/join).
|
||||
|
||||
## Grouping {#grouping}
|
||||
|
||||
```sql
|
||||
SELECT category, count(*) AS n, avg(score) AS mean_score
|
||||
FROM documents
|
||||
GROUP BY category
|
||||
HAVING count(*) > 10
|
||||
ORDER BY n DESC
|
||||
```
|
||||
|
||||
`WHERE` filters rows before grouping; `HAVING` filters groups after. Putting a
|
||||
row-level predicate in `HAVING` still gives the right answer but does more work,
|
||||
because rows that could have been discarded early are grouped first.
|
||||
|
||||
Grouping by several expressions produces one row per distinct combination:
|
||||
|
||||
```sql
|
||||
SELECT category, source, count(*) FROM documents GROUP BY category, source
|
||||
```
|
||||
|
||||
## Aggregate without grouping {#whole-table}
|
||||
|
||||
Omitting `GROUP BY` aggregates the whole table into one row -- the usual way to
|
||||
ask how big something is:
|
||||
|
||||
```sql
|
||||
SELECT count(*), min(created_at), max(created_at) FROM documents
|
||||
```
|
||||
|
||||
`count(*)` counts rows; `count(col)` counts non-null values, which is how you
|
||||
measure how populated a [computed column](/feature-engineering/add-computed-column)
|
||||
is before deciding to refresh it again.
|
||||
|
||||
## The function set {#functions}
|
||||
|
||||
Registered in DataFusion 54. Names below are what is actually available.
|
||||
|
||||
**General**
|
||||
|
||||
```text
|
||||
count sum min max avg median
|
||||
bit_and bit_or bit_xor bool_and bool_or
|
||||
first_value last_value nth_value grouping
|
||||
```
|
||||
|
||||
**Statistical**
|
||||
|
||||
```text
|
||||
stddev stddev_pop stddev_samp
|
||||
var var_pop var_samp
|
||||
corr covar_pop covar_samp
|
||||
regr_slope regr_intercept regr_count regr_r2
|
||||
regr_avgx regr_avgy regr_sxx regr_syy regr_sxy
|
||||
```
|
||||
|
||||
**Approximate**
|
||||
|
||||
```text
|
||||
approx_distinct approx_median
|
||||
approx_percentile_cont approx_percentile_cont_with_weight
|
||||
percentile_cont
|
||||
```
|
||||
|
||||
**Collecting**
|
||||
|
||||
```text
|
||||
array_agg string_agg
|
||||
```
|
||||
|
||||
The approximate functions are the ones worth reaching for on a large table.
|
||||
`approx_distinct` answers "roughly how many distinct values" for a fraction of
|
||||
what `count(DISTINCT ...)` costs, which matters most where it is most expensive
|
||||
— a high-cardinality column.
|
||||
|
||||
`DISTINCT` works inside an aggregate: `count(DISTINCT category)`. `FILTER`
|
||||
narrows one aggregate without narrowing the others:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
count(*) AS total,
|
||||
count(*) FILTER (WHERE score > 0.9) AS confident
|
||||
FROM documents
|
||||
```
|
||||
|
||||
## Aggregating a sample {#sampling}
|
||||
|
||||
For an estimate over a very large table, aggregate a
|
||||
[sample](/sql/dql/sampling) rather than the whole thing:
|
||||
|
||||
```sql
|
||||
SELECT category, count(*) * 100 AS estimated
|
||||
FROM documents TABLESAMPLE (1)
|
||||
GROUP BY category
|
||||
```
|
||||
|
||||
Scale the result by the inverse of the sample fraction, and treat it as an
|
||||
estimate -- a 1% sample of a skewed column will miss rare categories entirely.
|
||||
|
||||
## What aggregation does not do {#not}
|
||||
|
||||
Aggregates run over rows the query already produced. They cannot combine with
|
||||
[vector search](/search/vector-search) in one statement, because similarity
|
||||
search has no SQL form. Run the search in an SDK and aggregate its result, or
|
||||
materialize the search output first.
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
title: "Explain"
|
||||
sidebarTitle: "Explain"
|
||||
description: "EXPLAIN, EXPLAIN ANALYZE and EXPLAIN VERBOSE — reading and measuring a plan from SQL."
|
||||
icon: map
|
||||
keywords: ["explain", "explain analyze", "explain verbose", "query plan"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
EXPLAIN SELECT * FROM documents WHERE identifier < 1000
|
||||
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
|
||||
```
|
||||
|
||||
| Statement | What it returns |
|
||||
| --- | --- |
|
||||
| `EXPLAIN <query>` | Logical and physical plan, without executing the query. |
|
||||
| `EXPLAIN ANALYZE <query>` | Executes the query and annotates each operator with runtime metrics (rows, timing). |
|
||||
| `EXPLAIN VERBOSE <query>` | Adds intermediate optimizer plans on top of `EXPLAIN`. |
|
||||
|
||||
The result is a small relation with `plan_type` and `plan` columns, read
|
||||
through the same client you use for regular queries -- see
|
||||
[Connecting](/sql#connecting).
|
||||
|
||||
The operators are the same ones the SDK's
|
||||
[`explain_plan` and `analyze_plan`](/search/explain) methods produce, so the
|
||||
reasoning about index coverage and filter pushdown carries across unchanged:
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="What it shows" icon="map" href="/search/explain#inspect-a-query-plan">
|
||||
Reading a plan: operators, pushdown, index usage.
|
||||
</Card>
|
||||
<Card title="What it measures" icon="gauge" href="/search/explain#analyze-a-query-plan">
|
||||
Reading the metrics, and a worked before/after.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## What to look for {#what-to-look-for}
|
||||
|
||||
| Sign | Means |
|
||||
| --- | --- |
|
||||
| `ScalarIndexQuery` | the predicate used a scalar index |
|
||||
| `LanceScan` with high `bytes_read` | no index, or too many columns selected |
|
||||
| `FilterExec` above a search | the filter ran after search rather than pushing into it |
|
||||
| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used |
|
||||
|
||||
A `LanceScan` feeding a join is the common one worth acting on -- see
|
||||
[Joins](/sql/dql/join#performance).
|
||||
|
||||
For a table's columns and types without a plan, see
|
||||
[`DESCRIBE`](/sql/ddl/table#describe).
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
title: "Full-Text Search"
|
||||
sidebarTitle: "Full-Text Search"
|
||||
description: "The fts() table function — match, fuzzy, phrase, boolean, boost and multi-match queries from SQL."
|
||||
icon: magnifying-glass
|
||||
keywords: ["fts", "sql", "full text search", "fuzzy", "phrase", "boolean", "boost", "multi-match"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Full-text search is a **table function**, `fts(table, query)`, rather than a
|
||||
predicate -- it produces a relevance score alongside the matching rows, and a
|
||||
`WHERE` clause has nowhere to put one. The two string-literal arguments are
|
||||
the table name and a JSON-encoded query, built with the same query-builder
|
||||
classes used from the SDK and serialized with `.to_json()`.
|
||||
|
||||
## Setup {#setup}
|
||||
|
||||
Run these statements in order, sending one statement per SQL request.
|
||||
Semicolons separate the statements for clients that execute a script.
|
||||
|
||||
```sql
|
||||
CREATE TABLE my_docs (id INT, text STRING, category STRING, author STRING);
|
||||
|
||||
INSERT INTO my_docs VALUES
|
||||
(1, 'The happy puppy runs merrily in the park', 'animals', 'Alice'),
|
||||
(2, 'A curious kitten jumps quickly over the fence', 'animals', 'Bob'),
|
||||
(3, 'The puppy catches a ball with great enthusiasm', 'sports', 'Alice'),
|
||||
(5, 'Puppy training requires patience and dedication', 'training', 'Alice');
|
||||
|
||||
CREATE INDEX ON my_docs USING fts (text);
|
||||
```
|
||||
|
||||
<Warning>
|
||||
Phrase queries need position information, which costs extra index size --
|
||||
add `WITH (with_position = true)` to `CREATE INDEX`. Without it,
|
||||
[phrase queries](#phrase) find nothing. See
|
||||
[FTS parameters](/indexing/fts-index#fts-parameters).
|
||||
</Warning>
|
||||
|
||||
## Basic search {#basic}
|
||||
|
||||
The query builder produces the JSON string used as the second SQL argument.
|
||||
The SQL examples below spell out the equivalent match object so they can be
|
||||
run directly after the setup above.
|
||||
|
||||
```python
|
||||
from lancedb.query import MatchQuery
|
||||
|
||||
query = MatchQuery("puppy", "text")
|
||||
json_query = query.to_json()
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT id, text, category
|
||||
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}')
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
By default the top 5 matches come back in **arbitrary order** -- select
|
||||
`_score` and order by it for relevance ranking:
|
||||
|
||||
```sql
|
||||
SELECT id, text, _score
|
||||
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}')
|
||||
ORDER BY _score DESC
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
`_score` is a pseudo-column carrying the query's BM25 relevance for the row;
|
||||
it behaves like any other column, but exists only in a relation that came
|
||||
from `fts()`. See [Data types](/sql/data-types#pseudo-columns).
|
||||
|
||||
## Fuzzy {#fuzzy}
|
||||
|
||||
Tolerates typos, up to a character-edit budget:
|
||||
|
||||
```python
|
||||
MatchQuery("pupy", "text", fuzziness=2)
|
||||
```
|
||||
|
||||
## Phrase {#phrase}
|
||||
|
||||
Exact phrase, or with `slop` to allow words between terms:
|
||||
|
||||
```python
|
||||
from lancedb.query import PhraseQuery
|
||||
|
||||
PhraseQuery("happy puppy", "text")
|
||||
PhraseQuery("puppy park", "text", slop=2) # up to 2 words between them
|
||||
```
|
||||
|
||||
## Boolean {#boolean}
|
||||
|
||||
Combine queries with `&` (AND) or `|` (OR):
|
||||
|
||||
```python
|
||||
MatchQuery("puppy", "text") & MatchQuery("happy", "text")
|
||||
MatchQuery("puppy", "text") | MatchQuery("kitten", "text")
|
||||
```
|
||||
|
||||
## Boost {#boost}
|
||||
|
||||
Weight a query up or down relative to another:
|
||||
|
||||
```python
|
||||
from lancedb.query import BoostQuery
|
||||
|
||||
BoostQuery(
|
||||
positive=MatchQuery("puppy", "text"),
|
||||
negative=MatchQuery("kitten", "text"),
|
||||
negative_boost=0.2,
|
||||
)
|
||||
```
|
||||
|
||||
## Multi-match {#multi-match}
|
||||
|
||||
Search several columns at once, optionally weighting them differently:
|
||||
|
||||
```python
|
||||
from lancedb.query import MultiMatchQuery
|
||||
|
||||
MultiMatchQuery("puppy", ["text", "category"])
|
||||
MultiMatchQuery("puppy", ["text", "category"], boosts=[2.0, 1.0]) # text weighted 2x
|
||||
```
|
||||
|
||||
## Combining with SQL {#combining}
|
||||
|
||||
`fts()` is a relation, so it composes with the rest of the SQL surface --
|
||||
filter, join, or aggregate over it same as any table:
|
||||
|
||||
```sql
|
||||
SELECT id, text, category
|
||||
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}')
|
||||
WHERE category = 'animals'
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Full-Text Search" icon="magnifying-glass" href="/search/full-text-search">
|
||||
The query-builder equivalent, and every query form in full.
|
||||
</Card>
|
||||
<Card title="FTS Index" icon="bolt" href="/indexing/fts-index">
|
||||
Tokenizers, and the position information phrase queries need.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
title: "Joins"
|
||||
sidebarTitle: "Joins"
|
||||
description: "Combining rows from more than one table, including across namespaces."
|
||||
icon: link
|
||||
keywords: ["join", "inner", "left", "outer", "cross", "namespace"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Joining is the other thing the query builders cannot do. A question spanning two
|
||||
tables is a SQL question.
|
||||
|
||||
## The join types {#types}
|
||||
|
||||
```sql
|
||||
SELECT d.title, c.name
|
||||
FROM documents AS d
|
||||
JOIN categories AS c ON c.id = d.category_id
|
||||
```
|
||||
|
||||
| | What it keeps |
|
||||
| --- | --- |
|
||||
| `[INNER] JOIN` | rows matching on both sides |
|
||||
| `LEFT [OUTER] JOIN` | every left row, nulls where the right has no match |
|
||||
| `RIGHT [OUTER] JOIN` | every right row |
|
||||
| `FULL [OUTER] JOIN` | every row from both |
|
||||
| `CROSS JOIN` | every combination -- the product of the two |
|
||||
|
||||
`USING (col)` is shorthand for an equality join on a shared column name, and
|
||||
avoids repeating it in the select list.
|
||||
|
||||
## Across namespaces {#namespaces}
|
||||
|
||||
Tables in different [namespaces](/management/namespace) join by their dotted paths -- the
|
||||
catalog is one catalog, so there is nothing to federate:
|
||||
|
||||
```sql
|
||||
SELECT c.id, u.email
|
||||
FROM prod.events.clicks AS c
|
||||
JOIN prod.reference.users AS u ON u.id = c.user_id
|
||||
```
|
||||
|
||||
## Joining a search result {#search}
|
||||
|
||||
[`fts()`](/sql/functions#fts) is a table function, so its result is a relation
|
||||
and joins like any other. This example assumes `my_docs` has `text` and
|
||||
`category_id` columns, a full-text index on `text`, and a `categories` table
|
||||
with `id` and `name` columns:
|
||||
|
||||
```sql
|
||||
SELECT c.name, count(*) AS hits
|
||||
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}') AS m
|
||||
JOIN categories AS c ON c.id = m.category_id
|
||||
GROUP BY c.name
|
||||
ORDER BY hits DESC
|
||||
```
|
||||
|
||||
That composition is the reason full-text search is a relation rather than a
|
||||
predicate -- a `WHERE` clause has nowhere to put the relevance score.
|
||||
|
||||
## Performance {#performance}
|
||||
|
||||
A [scalar index](/indexing/scalar-index) can accelerate supported filters on
|
||||
the input tables. An index on a join key does not by itself guarantee an
|
||||
index lookup join. Use SQL [`EXPLAIN`](/sql/dql/explain) to inspect the scans,
|
||||
filters, and join chosen for the complete statement.
|
||||
|
||||
<Warning>
|
||||
Avoid selecting blob or vector columns you do not need on either side of a join.
|
||||
The join multiplies rows before projection is applied in some plans, so a wide
|
||||
column can be materialized many more times than the result has rows.
|
||||
</Warning>
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: "Random sampling"
|
||||
sidebarTitle: "Sampling"
|
||||
description: "TABLESAMPLE: draw a random subset of a table, by rows or by fragments."
|
||||
icon: dice
|
||||
keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
`TABLESAMPLE` draws a random subset of a table. On a lakehouse table that is
|
||||
often the difference between a question you can afford to ask and one you
|
||||
cannot -- an aggregate over 1% of forty billion rows answers in a fraction of
|
||||
the time, and for most estimates that is enough.
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE (10) -- 10 percent
|
||||
SELECT * FROM documents TABLESAMPLE (500 ROWS) -- 500 rows
|
||||
```
|
||||
|
||||
A bare number is a **percentage**. Add `ROWS` for a count instead.
|
||||
|
||||
## Rows or fragments {#methods}
|
||||
|
||||
The standard calls these two methods of one clause. They are different
|
||||
operations, and the difference matters on Lance:
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE BERNOULLI (10)
|
||||
SELECT * FROM documents TABLESAMPLE SYSTEM (10)
|
||||
```
|
||||
|
||||
| Method | Aliases | What it picks |
|
||||
| --- | --- | --- |
|
||||
| `BERNOULLI` | `ROW` | rows, each considered independently |
|
||||
| `SYSTEM` | `BLOCK` | **Lance fragments**, and takes whatever rows they hold |
|
||||
|
||||
`BERNOULLI` gives a genuinely uniform sample and reads across the whole table to
|
||||
get it. `SYSTEM` reads far less -- it skips entire fragments -- but the sample is
|
||||
only as uniform as the data's layout. If rows were written in an order that
|
||||
correlates with what you are measuring, a fragment sample inherits that
|
||||
correlation.
|
||||
|
||||
Reach for `SYSTEM` when you want a cheap look at the data and `BERNOULLI` when
|
||||
the number you compute has to be defensible. Omitting the method gives
|
||||
`BERNOULLI`.
|
||||
|
||||
## Repeatability {#repeatable}
|
||||
|
||||
Without a seed, every execution resamples -- running the same query twice gives
|
||||
different rows. `REPEATABLE` fixes the draw:
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE (10) REPEATABLE (42)
|
||||
```
|
||||
|
||||
This is what makes a sample usable as a **fixed evaluation set**: the same seed
|
||||
against the same data gives the same rows, so a metric computed today is
|
||||
comparable with one computed last week.
|
||||
|
||||
## Where the clause can go {#placement}
|
||||
|
||||
It attaches to a table in `FROM`, including inside a subquery, a CTE, or either
|
||||
side of a join:
|
||||
|
||||
```sql
|
||||
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
|
||||
|
||||
SELECT * FROM a TABLESAMPLE (5 ROWS) JOIN b ON a.id = b.id
|
||||
|
||||
SELECT * FROM t a TABLESAMPLE (10) JOIN t b TABLESAMPLE (10) ON a.id = b.id
|
||||
```
|
||||
|
||||
Each side of a join samples independently. Sampling both sides of a join
|
||||
shrinks the *result* far more than either fraction suggests, because a row only
|
||||
survives if both its sides were drawn.
|
||||
|
||||
## What is refused {#refused}
|
||||
|
||||
Sampling is semantics, not a hint. A clause that were read and then not applied
|
||||
would be a wrong answer rather than a missing feature, so anything that cannot
|
||||
be honoured exactly is an error:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `TABLESAMPLE (101)` | a percentage outside `0..=100` is refused, not clamped |
|
||||
| `TABLESAMPLE (1.5 ROWS)` | a row count must be whole |
|
||||
| `TABLESAMPLE (BUCKET 1 OUT OF 4)` | bucket sampling is not supported |
|
||||
| the `SAMPLE` spelling | use `TABLESAMPLE` |
|
||||
| the `OFFSET` modifier | not supported |
|
||||
|
||||
<Note>
|
||||
`TABLESAMPLE` is also refused on a WAL-backed table, where the rows a sample
|
||||
would draw from are not all in the fragments yet.
|
||||
</Note>
|
||||
|
||||
## Sampling outside SQL {#sdk}
|
||||
|
||||
For a training loop rather than a query, the SDKs read a table in a shuffled
|
||||
order without going through SQL at all -- see [Training](/training/data-loading), where a
|
||||
permutation drives the data loader.
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
title: "SELECT"
|
||||
sidebarTitle: "SELECT"
|
||||
description: "Projection, filtering, ordering, and composing queries out of other queries."
|
||||
icon: table-columns
|
||||
keywords: ["select", "where", "order by", "limit", "cte", "union", "distinct"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
[WITH cte AS (...)]
|
||||
SELECT [ALL | DISTINCT] select_expr [, ...]
|
||||
FROM from_item [[JOIN ...] ON ...]
|
||||
[WHERE predicate]
|
||||
[GROUP BY expr [, ...]] [HAVING predicate]
|
||||
[WINDOW name AS (...)]
|
||||
[ORDER BY expr [ASC | DESC] [NULLS FIRST | LAST] [, ...]]
|
||||
[LIMIT n] [OFFSET n]
|
||||
```
|
||||
|
||||
## Projection {#projection}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL
|
||||
SELECT id, title, identifier FROM documents
|
||||
```
|
||||
|
||||
```python Python icon="python"
|
||||
tbl.search().select(["id", "title", "identifier"]).to_arrow()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await tbl.query().select(["id", "title", "identifier"]).toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Projection pushes into the scan in all three -- an unselected column is never
|
||||
read off storage. On a table of images or video that is the difference between a
|
||||
fast query and a slow one, so **prefer naming columns over `SELECT *`**.
|
||||
|
||||
Row metadata is available as [pseudo-columns](/sql/data-types#pseudo-columns)
|
||||
(`_rowid`, `_rowaddr`, `_rowoffset`) but only when named, so they stay out of
|
||||
`SELECT *`.
|
||||
|
||||
`DISTINCT` deduplicates the projected rows.
|
||||
|
||||
## Filtering {#filtering}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL
|
||||
SELECT * FROM documents
|
||||
WHERE identifier > 0 AND identifier < 1000000
|
||||
```
|
||||
|
||||
```python Python icon="python"
|
||||
tbl.search().where("identifier > 0 AND identifier < 1000000").to_arrow()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await tbl.query().where("identifier > 0 AND identifier < 1000000").toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The SDK `where` takes the same expression language, so a predicate moves between
|
||||
them unchanged. A predicate on an indexed column becomes a `ScalarIndexQuery`
|
||||
rather than a scan -- confirm with [`explain_plan`](/search/explain). See
|
||||
[Filtering](/search/filtering) for the operators and their pushdown behaviour.
|
||||
|
||||
## Ordering and limiting {#order-limit}
|
||||
|
||||
```sql
|
||||
SELECT id, title FROM documents
|
||||
ORDER BY created_at DESC NULLS LAST
|
||||
LIMIT 100 OFFSET 200
|
||||
```
|
||||
|
||||
`ORDER BY` on a large table is a sort over the whole result, so pair it with a
|
||||
`LIMIT` unless you genuinely need every row ordered.
|
||||
|
||||
## Set operations {#set-operations}
|
||||
|
||||
`UNION`, `UNION ALL`, `INTERSECT` and `EXCEPT` combine whole queries:
|
||||
|
||||
```sql
|
||||
SELECT id FROM documents WHERE category = 'film'
|
||||
UNION ALL
|
||||
SELECT id FROM archived_documents WHERE category = 'film'
|
||||
```
|
||||
|
||||
`UNION` deduplicates and `UNION ALL` does not, which makes `UNION ALL` the
|
||||
cheaper one when you already know the inputs are disjoint.
|
||||
|
||||
## Common table expressions {#ctes}
|
||||
|
||||
```sql
|
||||
WITH recent AS (
|
||||
SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
|
||||
)
|
||||
SELECT category, count(*) FROM recent GROUP BY category
|
||||
```
|
||||
|
||||
A CTE is scoped to the statement. One you want to keep is a
|
||||
[view](/management/view); one whose rows you want stored is a
|
||||
[materialized view](/management/materialized-view).
|
||||
|
||||
## Subqueries {#subqueries}
|
||||
|
||||
Subqueries are allowed anywhere an expression or a relation is -- in `FROM`, in
|
||||
`WHERE` with `IN` or `EXISTS`, and as a scalar in the select list:
|
||||
|
||||
```sql
|
||||
SELECT title
|
||||
FROM documents
|
||||
WHERE category_id IN (SELECT id FROM categories WHERE active)
|
||||
```
|
||||
|
||||
## Namespaced tables {#namespaces}
|
||||
|
||||
A table inside a [namespace](/management/namespace) is written as a dotted path:
|
||||
|
||||
```sql
|
||||
SELECT * FROM prod.events.clicks
|
||||
```
|
||||
|
||||
## Reading the plan {#plans}
|
||||
|
||||
`EXPLAIN` and `EXPLAIN ANALYZE` work as DataFusion defines them, and produce the
|
||||
same operators the SDK methods do:
|
||||
|
||||
```sql
|
||||
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
|
||||
```
|
||||
|
||||
See [Explain](/search/explain) and [Explain Analyze](/search/explain).
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: "Subqueries"
|
||||
sidebarTitle: "Subqueries"
|
||||
description: "Queries inside queries: scalar, IN, EXISTS, correlated, and derived tables."
|
||||
icon: code-branch
|
||||
keywords: ["subquery", "exists", "in", "correlated", "derived table", "lateral"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A subquery is allowed anywhere an expression or a relation is.
|
||||
|
||||
## As a relation {#derived}
|
||||
|
||||
```sql
|
||||
SELECT * FROM (
|
||||
SELECT category, count(*) AS n FROM documents GROUP BY category
|
||||
) AS counts
|
||||
WHERE n > 100
|
||||
```
|
||||
|
||||
A derived table needs an alias. It can be [sampled](/sql/dql/sampling) like a base
|
||||
table:
|
||||
|
||||
```sql
|
||||
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
|
||||
```
|
||||
|
||||
## As a scalar {#scalar}
|
||||
|
||||
```sql
|
||||
SELECT title, (SELECT count(*) FROM revisions r WHERE r.doc_id = d.id) AS revisions
|
||||
FROM documents AS d
|
||||
```
|
||||
|
||||
A scalar subquery must return one row and one column. Returning more is an
|
||||
error rather than a silently-taken first row.
|
||||
|
||||
## IN and EXISTS {#in-exists}
|
||||
|
||||
```sql
|
||||
SELECT title FROM documents
|
||||
WHERE category_id IN (SELECT id FROM categories WHERE active)
|
||||
|
||||
SELECT title FROM documents AS d
|
||||
WHERE EXISTS (SELECT 1 FROM revisions r WHERE r.doc_id = d.id)
|
||||
```
|
||||
|
||||
`EXISTS` stops at the first matching row, so it is the cheaper test when you
|
||||
only care whether anything matches. `IN` builds the full set.
|
||||
|
||||
`NOT IN` and `NOT EXISTS` differ on nulls: `NOT IN` against a set containing a
|
||||
null yields no rows, because the comparison is unknown rather than false. Prefer
|
||||
`NOT EXISTS` unless you have ruled nulls out.
|
||||
|
||||
## Correlated subqueries {#correlated}
|
||||
|
||||
A subquery referencing a column from the enclosing query — `d.id` above — is
|
||||
correlated, and is conceptually evaluated per outer row. The planner usually
|
||||
rewrites it into a join.
|
||||
|
||||
<Note>
|
||||
Check with [`EXPLAIN`](/sql/dql/explain) when a correlated subquery is slow. A plan
|
||||
showing a join has been optimized; one showing a repeated scan has not, and
|
||||
rewriting it as an explicit [join](/sql/dql/join) is the fix.
|
||||
</Note>
|
||||
|
||||
## CTEs as an alternative {#ctes}
|
||||
|
||||
A [CTE](/sql/dql/select#ctes) is a named subquery, and is usually more readable when
|
||||
the same subquery appears twice or when nesting gets deep.
|
||||
|
||||
```sql
|
||||
WITH active AS (SELECT id FROM categories WHERE active)
|
||||
SELECT title FROM documents WHERE category_id IN (SELECT id FROM active)
|
||||
```
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
title: "Window functions"
|
||||
sidebarTitle: "Window Functions"
|
||||
description: "Compute over a set of rows related to the current one, without collapsing them."
|
||||
icon: chart-simple
|
||||
keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A window function computes across a set of rows related to the current one and
|
||||
returns a value **per row**. That is the difference from
|
||||
[aggregation](/sql/dql/aggregation), which collapses the rows it covers.
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
title,
|
||||
score,
|
||||
row_number() OVER (PARTITION BY category ORDER BY score DESC) AS rank
|
||||
FROM documents
|
||||
```
|
||||
|
||||
Every row survives; each one gains its rank within its category.
|
||||
|
||||
## The window {#window}
|
||||
|
||||
```sql
|
||||
OVER (
|
||||
[PARTITION BY expr [, ...]]
|
||||
[ORDER BY expr [, ...]]
|
||||
[ROWS | RANGE BETWEEN frame_start AND frame_end]
|
||||
)
|
||||
```
|
||||
|
||||
`PARTITION BY` splits the rows into independent groups. `ORDER BY` orders them
|
||||
inside each partition -- required by the ranking functions, since a rank without
|
||||
an order is meaningless. The frame clause narrows the window further, which is
|
||||
how running totals and moving averages are expressed:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
created_at,
|
||||
sum(amount) OVER (ORDER BY created_at ROWS UNBOUNDED PRECEDING) AS running_total,
|
||||
avg(amount) OVER (ORDER BY created_at ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS trailing_7
|
||||
FROM transactions
|
||||
```
|
||||
|
||||
A `WINDOW` clause names a definition when several functions share one:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
row_number() OVER w,
|
||||
lag(score) OVER w
|
||||
FROM documents
|
||||
WINDOW w AS (PARTITION BY category ORDER BY score DESC)
|
||||
```
|
||||
|
||||
## The functions {#functions}
|
||||
|
||||
Registered in DataFusion 54.
|
||||
|
||||
**Ranking**
|
||||
|
||||
```text
|
||||
row_number rank dense_rank percent_rank cume_dist ntile
|
||||
```
|
||||
|
||||
**Offset and positional**
|
||||
|
||||
```text
|
||||
lag lead first_value last_value nth_value
|
||||
```
|
||||
|
||||
**Aggregates**
|
||||
|
||||
Any [aggregate function](/sql/dql/aggregation#functions) used with `OVER` becomes a
|
||||
window function — that is how running totals and moving averages are written.
|
||||
|
||||
## Deduplicating with a window {#dedup}
|
||||
|
||||
The common practical use: keep one row per key, chosen by some ordering. A
|
||||
window ranks, and an outer query filters:
|
||||
|
||||
```sql
|
||||
WITH ranked AS (
|
||||
SELECT *, row_number() OVER (PARTITION BY doc_id ORDER BY version DESC) AS rn
|
||||
FROM revisions
|
||||
)
|
||||
SELECT * FROM ranked WHERE rn = 1
|
||||
```
|
||||
|
||||
The outer query is needed because a window function cannot appear in `WHERE` --
|
||||
windows are computed after filtering, so the predicate would have nothing to
|
||||
read.
|
||||
|
||||
<Note>
|
||||
Window functions sort within each partition, so on a large table they are among
|
||||
the more expensive things you can ask for. Filter first, and check the cost with
|
||||
[Explain Analyze](/search/explain).
|
||||
</Note>
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
title: "Functions"
|
||||
sidebarTitle: "Functions"
|
||||
description: "Where to find each kind of function, and the dialect version they come from."
|
||||
icon: function
|
||||
keywords: ["functions", "datafusion", "scalar", "aggregate", "window", "fts", "table function"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org),
|
||||
currently **version 54**. The function set is DataFusion's default registration
|
||||
plus what Lance adds.
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| [Scalar functions](/sql/scalar-functions) | per row — math, string, regex, date/time, array, struct, JSON |
|
||||
| [Aggregate functions](/sql/dql/aggregation#functions) | per group — `count`, `sum`, statistical, approximate |
|
||||
| [Window functions](/sql/dql/window#functions) | per window — ranking, offset, running aggregates |
|
||||
| [Table functions](#table-functions) | produce a relation rather than a value |
|
||||
|
||||
<Note>
|
||||
Function availability is version-specific. DataFusion adds functions in most
|
||||
releases, so one documented on `main` may not exist in 54. When a name is not
|
||||
recognised, check it against the reference for that version before assuming a
|
||||
bug.
|
||||
</Note>
|
||||
|
||||
## Table functions {#table-functions}
|
||||
|
||||
A table function appears in `FROM` and produces a relation, which is what lets
|
||||
it return columns a scalar function has nowhere to put.
|
||||
|
||||
### `fts(table, query)` {#fts}
|
||||
|
||||
Full-text search is a table function rather than a predicate, because it
|
||||
produces relevance alongside the matching rows. The following query assumes
|
||||
`my_docs` has `id` and `text` columns and a full-text index on `text`; see the
|
||||
[setup example](/sql/dql/fts#setup):
|
||||
|
||||
```sql
|
||||
SELECT id, text, _score
|
||||
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}')
|
||||
ORDER BY _score DESC
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
The second argument is a string literal containing a JSON query object. A bare
|
||||
query string such as `'puppy'` is rejected. Match, fuzzy, phrase, boolean, boost
|
||||
and multi-match queries use this JSON encoding.
|
||||
|
||||
Its schema contains the dataset's columns plus `_score`, so every stored
|
||||
column is available in the result. Projections push into the scan.
|
||||
Being a relation is also what makes it compose — join it, filter it, aggregate
|
||||
over it. This join additionally requires `my_docs.category_id` and a
|
||||
`categories` table with `id` and `name` columns:
|
||||
|
||||
```sql
|
||||
SELECT c.name, count(*) AS hits
|
||||
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}') AS m
|
||||
JOIN categories AS c ON c.id = m.category_id
|
||||
GROUP BY c.name
|
||||
ORDER BY hits DESC
|
||||
```
|
||||
|
||||
For a keyword test that needs no score, the
|
||||
[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate is
|
||||
index-accelerated and usable directly in `WHERE`.
|
||||
|
||||
See [Full-Text Search](/sql/dql/fts) for the fuzzy, phrase, boolean, boost and
|
||||
multi-match query forms.
|
||||
|
||||
### `_score` {#score}
|
||||
|
||||
A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves
|
||||
like any other column — order by it, filter on it, project it — and exists only
|
||||
in a relation that came from `fts()`. See
|
||||
[Data types](/sql/data-types#pseudo-columns).
|
||||
|
||||
### `generate_series` {#generate-series}
|
||||
|
||||
```sql
|
||||
SELECT * FROM generate_series(1, 10)
|
||||
```
|
||||
|
||||
Rows from a range, useful for building a dense axis to left-join sparse data
|
||||
onto.
|
||||
|
||||
## What Lance accelerates {#accelerated}
|
||||
|
||||
Two function families use an index rather than scanning, which is the difference
|
||||
between a fast predicate and a slow one:
|
||||
|
||||
| | Index used |
|
||||
| --- | --- |
|
||||
| [`contains_tokens`](/sql/scalar-functions#contains-tokens) | full-text index |
|
||||
| [`json_get`, typed variants, `json_extract`](/sql/scalar-functions#json) | JSON scalar index |
|
||||
|
||||
Confirm with [`EXPLAIN`](/sql/dql/explain): an accelerated predicate shows an
|
||||
index node rather than a `LanceScan` followed by a filter.
|
||||
@@ -0,0 +1,150 @@
|
||||
---
|
||||
title: "SQL reference"
|
||||
sidebarTitle: "Overview"
|
||||
description: "The SQL surface: dialect, connecting a client, and a page per construct."
|
||||
icon: table-list
|
||||
keywords: ["sql", "datafusion", "flight sql", "reference", "dialect"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> SQL runs on a deployment. A local database
|
||||
has no engine to plan a statement, so `execute_query` raises there -- see
|
||||
[SQL](/basics/sql#sql-is-remote).
|
||||
|
||||
This is a reference rather than a guide: one page per construct, syntax first.
|
||||
The guides elsewhere in these docs show the same operations from the SDKs.
|
||||
|
||||
## The dialect {#dialect}
|
||||
|
||||
LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org),
|
||||
currently **version 54**. The grammar, built-in functions and type system are
|
||||
DataFusion's, so anything its reference documents for that version works here
|
||||
unchanged and is not reproduced.
|
||||
|
||||
What these pages add is the part that is LanceDB's: how a query reaches the
|
||||
engine, how Lance columns appear to it, and the statements a lakehouse needs
|
||||
that DataFusion has no concept of.
|
||||
|
||||
## Connecting {#connecting}
|
||||
|
||||
SQL is served over an
|
||||
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
|
||||
endpoint, so any Flight SQL client reaches it and results arrive as Arrow
|
||||
record batches -- nothing is converted to pandas or Python objects unless you
|
||||
ask.
|
||||
|
||||
From Python, the connection runs SQL directly, without a separate client:
|
||||
|
||||
```python
|
||||
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10")
|
||||
table = reader.read_all()
|
||||
```
|
||||
|
||||
`execute_query` hands back a reader as soon as the first batch is ready rather
|
||||
than when the query finishes, so a large result set does not have to fit in
|
||||
memory before you start reading it. See [Queries](/basics/query).
|
||||
|
||||
The TypeScript SDK has no SQL method; reach for a Flight SQL client instead --
|
||||
and in Python, reach for one too when the caller is outside the SDK's own
|
||||
process, such as a BI tool or a notebook in another language:
|
||||
|
||||
```bash
|
||||
pip install flightsql-dbapi # Python DB API 2 interface
|
||||
npm install --save @lancedb/flightsql-client
|
||||
```
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
from flightsql import FlightSQLClient
|
||||
|
||||
client = FlightSQLClient(
|
||||
host="your-enterprise-endpoint",
|
||||
port=10025,
|
||||
insecure=True,
|
||||
token="DATABASE_TOKEN",
|
||||
metadata={"database": "your-project-slug"},
|
||||
features={"metadata-reflection": "true"},
|
||||
)
|
||||
|
||||
info = client.execute("SELECT * FROM flights WHERE origin = 'SFO'")
|
||||
ticket = info.endpoints[0].ticket
|
||||
result = client.do_get(ticket).read_all()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
import { Client } from "@lancedb/flightsql-client";
|
||||
|
||||
const client = await Client.connect({
|
||||
host: "your-enterprise-endpoint:10025",
|
||||
username: "lancedb",
|
||||
password: "password",
|
||||
});
|
||||
|
||||
const result = await client.query("SELECT * FROM flights WHERE origin = 'SFO'");
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Usage {#usage}
|
||||
|
||||
The endpoint executes one statement per request and is meant for the queries
|
||||
and DDL/DML pages below. Reach for the SDKs instead for table-management
|
||||
operations that have no SQL form, such as building a vector index or a merge
|
||||
insert -- see [What SQL cannot do](#cannot).
|
||||
|
||||
## Reference {#reference}
|
||||
|
||||
### DDL, by object {#ddl}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| [Namespace](/sql/ddl/namespace) | `CREATE SCHEMA`, `DROP SCHEMA` |
|
||||
| [Table](/sql/ddl/table) | `CREATE`, `DESCRIBE`, `ALTER`, `CLONE`, `DROP TABLE` |
|
||||
| [View](/sql/ddl/view) | `CREATE VIEW`, `DROP VIEW` |
|
||||
| [Materialized view](/sql/ddl/materialized-view) | `CREATE MATERIALIZED VIEW`, `SHOW MATERIALIZED VIEWS` |
|
||||
| [Index](/sql/ddl/index) | `CREATE INDEX`, `DROP INDEX` |
|
||||
| [Function](/sql/ddl/function) | `CREATE FUNCTION`, `SHOW FUNCTIONS`, `DROP FUNCTION` |
|
||||
| [Job](/sql/ddl/job) | `SHOW JOBS`, `SHOW ERRORS`, `PAUSE`/`RESUME`/`CANCEL JOB` |
|
||||
|
||||
### DML {#dml}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| [Insert](/sql/dml/insert) | `INSERT INTO`, including `INSERT ... SELECT` |
|
||||
| [Update](/sql/dml/update) | `UPDATE ... SET` |
|
||||
| [Delete](/sql/dml/delete) | `DELETE FROM` |
|
||||
| [Copy](/sql/dml/copy) | `COPY INTO` / `COPY TO` |
|
||||
| [Refresh](/sql/dml/refresh) | filling computed columns and materialized views |
|
||||
|
||||
### DQL {#dql}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| [Data types](/sql/data-types) | how Lance columns appear to SQL, and pseudo-columns |
|
||||
| [Select](/sql/dql/select) | projection, filtering, ordering, set operations, CTEs |
|
||||
| [Subqueries](/sql/dql/subqueries) | scalar, `IN`, `EXISTS`, correlated, derived tables |
|
||||
| [Joins](/sql/dql/join) | inner, outer, cross, and namespace-qualified paths |
|
||||
| [Aggregation](/sql/dql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set |
|
||||
| [Window functions](/sql/dql/window) | `OVER`, framing, ranking and offset functions |
|
||||
| [Sampling](/sql/dql/sampling) | `TABLESAMPLE`, by rows or by fragments |
|
||||
| [Full-Text Search](/sql/dql/fts) | the `fts()` table function, and its query forms |
|
||||
| [Explain](/sql/dql/explain) | `EXPLAIN`, `EXPLAIN ANALYZE`, `EXPLAIN VERBOSE` |
|
||||
|
||||
### Expressions {#expressions}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| [Operators](/sql/operators) | comparison, logical, arithmetic, literals, precedence |
|
||||
| [Functions](/sql/functions) | where to find each kind, and table functions |
|
||||
| [Scalar functions](/sql/scalar-functions) | the per-row catalogue, and what Lance accelerates |
|
||||
| [Prepared statements](/sql/prepared-statements) | planning once, executing many times |
|
||||
|
||||
## What SQL cannot do {#cannot}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Vector search | no SQL form -- a query-builder operation |
|
||||
| Merge insert | no SQL form -- needs a source relation and a per-match policy |
|
||||
| `information_schema` | not enabled, so `SHOW TABLES` and `SHOW COLUMNS` fail |
|
||||
| `PREPARE` / `EXECUTE` | preparation is a [protocol feature](/sql/prepared-statements), not a statement |
|
||||
| Run on a local database | no engine to plan the statement |
|
||||
|
||||
See [Limits](/basics/limits#sql) for the bounds these pages enforce.
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
title: "Operators and literals"
|
||||
sidebarTitle: "Operators"
|
||||
description: "Comparison, logical and arithmetic operators, literals, and precedence."
|
||||
icon: plus-minus
|
||||
keywords: ["operators", "literals", "precedence", "null", "cast", "interval"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
The operator set is DataFusion's. This page covers what is worth knowing when
|
||||
the operands are Lance columns.
|
||||
|
||||
## Comparison {#comparison}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `=` `!=` `<>` `<` `<=` `>` `>=` | ordinary comparison |
|
||||
| `IS NULL` / `IS NOT NULL` | null test — `= NULL` is never true |
|
||||
| `IS DISTINCT FROM` | comparison that treats null as a value |
|
||||
| `BETWEEN a AND b` | inclusive range |
|
||||
| `IN (...)` | set membership |
|
||||
| `LIKE` / `ILIKE` | pattern match, `%` and `_` |
|
||||
| `~` `~*` `!~` | regular expression match |
|
||||
|
||||
<Note>
|
||||
For keyword matching on a text column, [`contains_tokens`](/sql/scalar-functions#contains-tokens)
|
||||
is index-accelerated where `LIKE` is not. Reach for `LIKE` only when you need
|
||||
its wildcard semantics.
|
||||
</Note>
|
||||
|
||||
## Logical {#logical}
|
||||
|
||||
`AND`, `OR`, `NOT`. Three-valued: a comparison against null is **unknown**, and
|
||||
`NOT unknown` is still unknown. A `WHERE` clause keeps only rows where the
|
||||
predicate is true, so unknown behaves like false there — but not inside `NOT
|
||||
IN`, which is the usual place this bites. See
|
||||
[Subqueries](/sql/dql/subqueries#in-exists).
|
||||
|
||||
## Arithmetic {#arithmetic}
|
||||
|
||||
`+` `-` `*` `/` `%`, with the usual precedence. String concatenation is `||`.
|
||||
|
||||
<Warning>
|
||||
Arithmetic over a vector column is **list** arithmetic, not scalar arithmetic —
|
||||
a vector is a `FixedSizeList<Float32>`. When the operation you want is a
|
||||
similarity comparison, use [vector search](/search/vector-search); there is no
|
||||
SQL form for it.
|
||||
</Warning>
|
||||
|
||||
## Literals {#literals}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| String | `'text'` — doubled quote to escape: `'it''s'` |
|
||||
| Number | `42`, `3.14` |
|
||||
| Boolean | `TRUE`, `FALSE` |
|
||||
| Null | `NULL` |
|
||||
| Date / timestamp | `DATE '2026-01-01'`, `TIMESTAMP '2026-01-01 12:00:00'` |
|
||||
| Interval | `INTERVAL '7 days'` |
|
||||
| Array | `[1, 2, 3]` |
|
||||
|
||||
Identifiers are quoted with double quotes when they need to be — a reserved word
|
||||
or a name with mixed case: `SELECT "order" FROM "MyTable"`.
|
||||
|
||||
## Casting {#casting}
|
||||
|
||||
```sql
|
||||
SELECT CAST(identifier AS VARCHAR), identifier::VARCHAR FROM documents
|
||||
```
|
||||
|
||||
Both forms work. `TRY_CAST` returns null instead of erroring on a value that
|
||||
will not convert, which is the one to use over a column you do not control.
|
||||
|
||||
## Precedence {#precedence}
|
||||
|
||||
Highest to lowest: `::` → unary `-` → `* / %` → `+ -` → `||` → comparison →
|
||||
`IS` → `NOT` → `AND` → `OR`.
|
||||
|
||||
Parenthesize anything mixing `AND` and `OR`. `a OR b AND c` is `a OR (b AND c)`,
|
||||
which is rarely what was meant.
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: "Prepared statements"
|
||||
sidebarTitle: "Prepared Statements"
|
||||
description: "Preparing a statement once and executing it many times, over the Flight SQL protocol."
|
||||
icon: bolt
|
||||
keywords: ["prepared statement", "flight sql", "parameters", "bind"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A prepared statement is planned once and executed many times, which saves
|
||||
re-planning a statement whose shape does not change between runs.
|
||||
|
||||
## It is a protocol feature, not a statement {#protocol}
|
||||
|
||||
<Warning>
|
||||
There is no SQL `PREPARE` / `EXECUTE` here. Preparation is part of the
|
||||
**Arrow Flight SQL protocol**, so you reach it through your client rather than
|
||||
by sending a statement.
|
||||
</Warning>
|
||||
|
||||
That distinction matters when porting SQL from another engine: a script
|
||||
containing `PREPARE name AS SELECT ...` will not run, while the same intent
|
||||
expressed through a Flight SQL client will.
|
||||
|
||||
## The flow {#flow}
|
||||
|
||||
Every Flight SQL client exposes the same three steps, whatever it calls them:
|
||||
|
||||
1. **Prepare** — send the SQL, get back a handle and the result schema. The
|
||||
statement is parsed and planned at this point, so a syntax or name error
|
||||
surfaces here rather than at execution.
|
||||
2. **Bind and execute** — supply parameter values for the handle and read the
|
||||
result. Repeat as often as you like.
|
||||
3. **Close** — release the handle. Servers do not hold them forever.
|
||||
|
||||
Knowing that the plan is built at step 1 is the practical part: if a statement
|
||||
is slow to prepare and fast to execute, the cost is planning, and reusing the
|
||||
handle is exactly the right fix.
|
||||
|
||||
## Parameters {#parameters}
|
||||
|
||||
Parameters are placeholders in the SQL text, bound to an Arrow record batch at
|
||||
execution. Binding rather than interpolating is worth doing for the usual two
|
||||
reasons — the value cannot change the statement's meaning, and the plan is
|
||||
reused across values.
|
||||
|
||||
```sql
|
||||
SELECT id, title FROM documents WHERE category = $1 AND score > $2
|
||||
```
|
||||
|
||||
The exact placeholder spelling and the binding call belong to your client; see
|
||||
[connecting a client](/sql#connecting).
|
||||
|
||||
## When it helps {#when}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| The same query with different values, many times | prepare once, bind per execution |
|
||||
| A complex plan — several joins, window functions | planning dominates; reuse it |
|
||||
| A one-off analytical query | no benefit; just execute it |
|
||||
|
||||
For a statement you want to keep rather than re-prepare, a
|
||||
[view](/management/view) stores the query server-side, and a
|
||||
[materialized view](/management/materialized-view) stores its rows.
|
||||
@@ -0,0 +1,190 @@
|
||||
---
|
||||
title: "Scalar functions"
|
||||
sidebarTitle: "Scalar Functions"
|
||||
description: "The per-row function set: math, string, regex, date/time, array, struct, and what Lance adds."
|
||||
icon: calculator
|
||||
keywords: ["scalar functions", "string functions", "math functions", "regex", "array functions", "json"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Functions that take a value and return a value, one row at a time. The set is
|
||||
based on DataFusion's registration plus Lance and deployment-specific
|
||||
functions. The names below are selected examples, not an exhaustive catalog
|
||||
for every deployment. Availability depends on the deployed version and build.
|
||||
|
||||
For per-group functions see [Aggregation](/sql/dql/aggregation); for per-window,
|
||||
[Window functions](/sql/dql/window).
|
||||
|
||||
## Math {#math}
|
||||
|
||||
```text
|
||||
abs ceil floor round trunc signum power sqrt cbrt exp
|
||||
ln log log2 log10 factorial gcd lcm pi random nanvl
|
||||
isnan iszero cot degrees radians to_degrees to_radians
|
||||
sin cos tan asin acos atan atan2
|
||||
sinh cosh tanh asinh acosh atanh
|
||||
```
|
||||
|
||||
## String {#string}
|
||||
|
||||
```text
|
||||
ascii bit_length btrim chr concat concat_ws contains ends_with
|
||||
starts_with levenshtein lower upper ltrim rtrim octet_length
|
||||
repeat replace split_part to_hex uuid
|
||||
```
|
||||
|
||||
Unicode-aware variants:
|
||||
|
||||
```text
|
||||
character_length find_in_set initcap left right lpad rpad
|
||||
reverse strpos substr substr_index substring translate
|
||||
```
|
||||
|
||||
<Note>
|
||||
For keyword matching on a text column, [`contains_tokens`](#contains-tokens) is
|
||||
index-accelerated where `contains` and `LIKE` are not.
|
||||
</Note>
|
||||
|
||||
## Regular expressions {#regex}
|
||||
|
||||
```text
|
||||
regexp_count regexp_instr regexp_like regexp_match regexp_replace
|
||||
```
|
||||
|
||||
## Date and time {#datetime}
|
||||
|
||||
```text
|
||||
current_date current_time date_bin date_part date_trunc
|
||||
from_unixtime to_unixtime make_date make_time to_char
|
||||
to_date to_time to_local_time
|
||||
```
|
||||
|
||||
`date_bin` is the one worth knowing for analytics: it buckets timestamps into
|
||||
fixed-width intervals, which is how you build a time series without a
|
||||
self-join.
|
||||
|
||||
```sql
|
||||
SELECT date_bin(INTERVAL '1 hour', created_at) AS hour, count(*)
|
||||
FROM events GROUP BY hour ORDER BY hour
|
||||
```
|
||||
|
||||
## Arrays and maps {#arrays}
|
||||
|
||||
```text
|
||||
make_array array_has array_filter array_transform array_any_match
|
||||
array_compact array_normalize array_slice arrays_zip cardinality
|
||||
flatten range generate_series
|
||||
```
|
||||
|
||||
```text
|
||||
map map_keys map_values map_entries map_extract
|
||||
```
|
||||
|
||||
Distance functions over arrays:
|
||||
|
||||
```text
|
||||
cosine_distance distance exact_distance inner_product
|
||||
```
|
||||
|
||||
<Warning>
|
||||
Distance functions can evaluate arrays per row. Some deployment versions also
|
||||
recognize `ORDER BY distance(vector, query_vector) LIMIT k` over a table scan
|
||||
and plan it as vector search, with indexed or flat search as appropriate.
|
||||
Where available, `exact_distance` evaluates distance without that rewrite.
|
||||
Confirm the
|
||||
functions supported by your deployment and inspect SQL
|
||||
[`EXPLAIN`](/sql/dql/explain); the function name alone does not guarantee an
|
||||
index-backed plan. See [SQL vector search](/basics/sql#no-vector-sql).
|
||||
</Warning>
|
||||
|
||||
## Structs and fields {#structs}
|
||||
|
||||
```text
|
||||
named_struct get_field coalesce nullif nvl nvl2 greatest least
|
||||
union_extract union_tag
|
||||
```
|
||||
|
||||
## Crypto and encoding {#crypto}
|
||||
|
||||
```text
|
||||
digest md5 sha224 sha256 sha384 sha512 encode decode
|
||||
```
|
||||
|
||||
## Type functions {#type}
|
||||
|
||||
```text
|
||||
arrow_cast arrow_try_cast arrow_typeof arrow_field arrow_metadata
|
||||
with_metadata version
|
||||
```
|
||||
|
||||
`arrow_typeof` is the fastest way to answer "what is this column, really" when a
|
||||
result does not have the shape you expected:
|
||||
|
||||
```sql
|
||||
SELECT arrow_typeof(vector) FROM documents LIMIT 1
|
||||
-- FixedSizeList(Field { name: "item", data_type: Float32, .. }, 1536)
|
||||
```
|
||||
|
||||
`arrow_cast` casts to an Arrow type by name, reaching types SQL has no spelling
|
||||
for. See [Data types](/sql/data-types).
|
||||
|
||||
## What Lance adds {#lance}
|
||||
|
||||
These are registered by Lance on top of DataFusion's set.
|
||||
|
||||
### `contains_tokens(column, tokens)` {#contains-tokens}
|
||||
|
||||
```sql
|
||||
SELECT * FROM my_docs WHERE contains_tokens(text, 'fox jumps dog')
|
||||
```
|
||||
|
||||
True when the column contains **all** the tokens, split on whitespace and
|
||||
punctuation. Equivalent to an `fts()` match query with fuzziness off, `AND`
|
||||
semantics and the simple tokenizer.
|
||||
|
||||
**Index-accelerated.** When the column has a full-text index that suits the
|
||||
query, the planner uses it instead of scanning — which is the reason to prefer
|
||||
it over `LIKE` for keyword tests.
|
||||
|
||||
### JSON accessors {#json}
|
||||
|
||||
| Function | Returns |
|
||||
| --- | --- |
|
||||
| `json_get(col, key)` | the value at `key`, still JSON |
|
||||
| `json_get_string(col, key)` | `Utf8` |
|
||||
| `json_get_int(col, key)` | `Int64` |
|
||||
| `json_get_float(col, key)` | `Float64` |
|
||||
| `json_get_bool(col, key)` | `Boolean` |
|
||||
| `json_extract(col, path)` | `Utf8`, by path |
|
||||
| `json_extract_with_type(col, path, type)` | the requested type |
|
||||
| `json_exists(col, path)` | `Boolean` |
|
||||
| `json_array_contains(col, value)` | `Boolean` |
|
||||
| `json_array_length(col)` | length |
|
||||
|
||||
`json_get` returns JSON, so accessors nest:
|
||||
|
||||
```sql
|
||||
SELECT * FROM events
|
||||
WHERE json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice'
|
||||
```
|
||||
|
||||
**`json_get`, its typed variants, and `json_extract` are index-accelerated** by a
|
||||
JSON scalar index on the column. The rest evaluate per row.
|
||||
|
||||
### Geospatial {#geo}
|
||||
|
||||
<Warning>
|
||||
Lance defines `st_*` functions on GeoArrow, but they sit behind a build feature
|
||||
**LanceDB does not ship**. Calling one returns an error naming the missing
|
||||
feature rather than an unknown-function error, and the spatial index is gated
|
||||
the same way.
|
||||
</Warning>
|
||||
|
||||
## Reaching a function not listed here {#escape-hatch}
|
||||
|
||||
Nothing wraps or restricts the set — a query reaches any registered function by
|
||||
writing it. When a name is not recognised, check it against
|
||||
[DataFusion's scalar function reference](https://datafusion.apache.org/user-guide/sql/scalar_functions.html)
|
||||
for version 54 before assuming it is unsupported; functions are added in most
|
||||
DataFusion releases, and a function documented on `main` may not exist here yet.
|
||||
Reference in New Issue
Block a user