rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-12 14:52:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: flip ownership of View/SQL-client/FTS-SQL/AuthZ/AuthN to OSS, restructure SQL Reference

Browse Source 
OSS docs now own the full usage content for features that only run against
an Enterprise deployment, since the client interface itself lives here;
Enterprise docs keep only the parts specific to its own behavior. Sophon's
overlay for these five pages is dropped since the OSS versions are now the
complete reference.

SQL Reference is restructured to match an object-type SQL manual (DDL by
object, then DML, then DQL, then Expressions) instead of the old flat,
loosely-ordered page list, absorbing the SQL client-connection guide and
FTS-via-SQL examples into it.
This commit is contained in:
Jack Ye
2026-09-28 17:17:09 -05:00
parent cb45b67c4c
commit 52684d50b9
57 changed files with 1412 additions and 590 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/basics/apis.mdx
+1 -1
View File
@@ -61,7 +61,7 @@ its result. Full-text search *is* available in SQL, as the
The TypeScript SDK does not expose SQL, and has no methods for
[functions](/management/function). Reach the endpoint with a Flight
SQL client instead -- see [connecting a client](/search/sql). This is a gap in
SQL client instead -- see [connecting a client](/sql#connecting). This is a gap in
the SDK rather than in the endpoint: the same statements work.
## Running SQL from Python {#sql-from-python}
docs/web/basics/limits.mdx
+1 -1
View File
@@ -77,7 +77,7 @@ See [Data types](/sql/data-types).
A sampling percentage over 100 is an error rather than a full scan, because a
query asking for 150 percent has a mistake in it and reading it as 100 would
hide the mistake. See [Sampling](/sql/sampling).
hide the mistake. See [Sampling](/sql/dql/sampling).
## What is unavailable where {#availability}
docs/web/curation/aggregate.mdx
+1 -1
View File
@@ -110,4 +110,4 @@ use this to check the shape of the head, not the existence of the tail.
## Reference {#reference}
The complete clause and function list is in
[Aggregation](/sql/aggregation).
[Aggregation](/sql/dql/aggregation).
docs/web/curation/join.mdx
+2 -2
View File
@@ -73,7 +73,7 @@ Two things dominate join cost on a lakehouse table.
**Index the join key.** A [scalar index](/indexing/scalar-index) on the key lets
the planner look rows up instead of scanning for them. Check with
[`EXPLAIN`](/sql/explain): an indexed key shows a `ScalarIndexQuery`, an
[`EXPLAIN`](/sql/dql/explain): an indexed key shows a `ScalarIndexQuery`, an
unindexed one shows a `LanceScan` feeding the join.
**Do not carry wide columns through.** Select the columns you need before
@@ -88,4 +88,4 @@ JOIN labels AS l ON l.image_id = i.id
## Reference {#reference}
The join types and their semantics are in [Joins](/sql/join).
The join types and their semantics are in [Joins](/sql/dql/join).
docs/web/curation/sampling.mdx
+1 -1
View File
@@ -95,4 +95,4 @@ rows. See [Training](/training/data-loading).
## Reference {#reference}
The full clause, its options and what it refuses are in
[Sampling](/sql/sampling).
[Sampling](/sql/dql/sampling).
docs/web/curation/windowing.mdx
+1 -1
View File
@@ -121,4 +121,4 @@ selective, and check the plan with
## Reference {#reference}
The full window syntax, framing, and function list are in
[Window functions](/sql/window).
[Window functions](/sql/dql/window).
docs/web/docs.json
+96 -17
View File
@@ -162,6 +162,58 @@
{
"source": "/enterprise/deployment",
"destination": "/enterprise/deployment/index"
},
{
"source": "/search/sql/index",
"destination": "/sql"
},
{
"source": "/search/sql/fts-sql",
"destination": "/sql/dql/fts"
},
{
"source": "/sql/select",
"destination": "/sql/dql/select"
},
{
"source": "/sql/subqueries",
"destination": "/sql/dql/subqueries"
},
{
"source": "/sql/join",
"destination": "/sql/dql/join"
},
{
"source": "/sql/aggregation",
"destination": "/sql/dql/aggregation"
},
{
"source": "/sql/window",
"destination": "/sql/dql/window"
},
{
"source": "/sql/sampling",
"destination": "/sql/dql/sampling"
},
{
"source": "/sql/explain",
"destination": "/sql/dql/explain"
},
{
"source": "/sql/ddl",
"destination": "/sql/ddl/table"
},
{
"source": "/sql/dml",
"destination": "/sql/dml/insert"
},
{
"source": "/sql/refresh",
"destination": "/sql/dml/refresh"
},
{
"source": "/sql/show",
"destination": "/sql/ddl/job"
}
],
"navigation": {
@@ -293,24 +345,51 @@
"group": "SQL Reference",
"pages": [
"sql/index",
"search/sql/index",
"sql/data-types",
"sql/select",
"search/sql/fts-sql",
"sql/subqueries",
"sql/join",
"sql/aggregation",
"sql/window",
"sql/sampling",
"sql/ddl",
"sql/dml",
"sql/refresh",
"sql/show",
"sql/explain",
"sql/prepared-statements",
"sql/operators",
"sql/functions",
"sql/scalar-functions"
{
"group": "DDL",
"pages": [
"sql/ddl/namespace",
"sql/ddl/table",
"sql/ddl/view",
"sql/ddl/materialized-view",
"sql/ddl/index",
"sql/ddl/function",
"sql/ddl/job"
]
},
{
"group": "DML",
"pages": [
"sql/dml/insert",
"sql/dml/update",
"sql/dml/delete",
"sql/dml/copy",
"sql/dml/refresh"
]
},
{
"group": "DQL",
"pages": [
"sql/dql/select",
"sql/dql/subqueries",
"sql/dql/join",
"sql/dql/aggregation",
"sql/dql/window",
"sql/dql/sampling",
"sql/dql/fts",
"sql/dql/explain"
]
},
{
"group": "Expressions",
"pages": [
"sql/operators",
"sql/functions",
"sql/scalar-functions",
"sql/prepared-statements"
]
}
]
},
{
docs/web/enterprise/authentication.mdx
+148 -4
View File
@@ -1,9 +1,9 @@
---
title: "Authentication"
sidebarTitle: "Authentication"
description: "Establishing who is calling: API keys and identity provider integration."
description: "Authentication modes for LanceDB Enterprise: API keys and OAuth credentials."
icon: key
keywords: ["authentication", "api key", "oidc", "identity"]
keywords: ["authentication", "oauth", "api key", "azure managed identity", "client credentials"]
---
<Badge color="red">Enterprise</Badge>
@@ -13,14 +13,158 @@ of LanceDB has no server to provide it -- see
[Architecture](/architecture) for what differs between the two.
A deployment authenticates every request before
[authorization](/governance/authorization) decides what it may do.
[authorization](/governance/authorization) decides what it may do, against a
`db://` remote table, in one of two ways:
- **API keys** -- a long-lived shared secret passed as `api_key` on connect. Works in every SDK.
- **OAuth 2.0** -- short-lived bearer tokens obtained from your identity provider, refreshed automatically by the client.
OAuth is the recommended option when you want to rotate credentials centrally,
plug into an existing identity provider, or run on Azure with managed
identities so no secret material lives on the client.
## API key {#api-key}
Pass the API key from your Enterprise tenant on `connect`. This works with
both the synchronous and asynchronous Python clients, as well as TypeScript
and Rust.
```python
db = lancedb.connect("db://my-database", api_key="...", region="us-east-1")
import lancedb
db = lancedb.connect(
uri="db://your-database-uri",
api_key="your-api-key",
region="us-east-1",
host_override="https://your-enterprise-endpoint.com",
)
```
An API key belongs in the environment rather than in source. See
[Database](/management/database#deployment) for the connection options.
## OAuth {#oauth}
The async Python client and the TypeScript client can obtain bearer tokens
from an OIDC issuer and attach them to every request. Token acquisition,
caching, and refresh are handled inside the client -- your application code
only provides the configuration.
<Info>
In Python, OAuth is supported through `lancedb.connect_async`. The synchronous
`connect` entry point continues to use API key authentication for `db://`
URIs. In TypeScript, `lancedb.connect` accepts `oauthConfig` directly.
</Info>
### Supported flows {#supported-flows}
`OAuthFlowType` selects how the client acquires tokens:
| Flow | Python value | TypeScript value | When to use |
| --- | --- | --- | --- |
| Client Credentials | `OAuthFlowType.CLIENT_CREDENTIALS` | `OAuthFlowType.ClientCredentials` | Service-to-service / machine-to-machine. Requires a client ID and client secret registered with your identity provider. |
| Azure Managed Identity | `OAuthFlowType.AZURE_MANAGED_IDENTITY` | `OAuthFlowType.AzureManagedIdentity` | Workloads running on Azure compute (VMs, AKS, App Service, Container Apps). Tokens are fetched from the Azure IMDS endpoint, so no client secret is stored on the client. |
### Configure OAuth {#configure-oauth}
Build an OAuth config and pass it on connect. Use `oauth_config` in Python and
`oauthConfig` in TypeScript.
<CodeGroup>
```python Python icon="python"
import lancedb
from lancedb.remote import OAuthConfig, OAuthFlowType
# Client Credentials (service-to-service)
oauth_config = OAuthConfig(
issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
client_id="your-application-client-id",
client_secret="your-application-client-secret",
scopes=["api://lancedb-api/.default"],
flow=OAuthFlowType.CLIENT_CREDENTIALS,
)
db = await lancedb.connect_async(
uri="db://your-database-uri",
region="us-east-1",
host_override="https://your-enterprise-endpoint.com",
oauth_config=oauth_config,
)
```
```typescript TypeScript icon="square-js"
import * as lancedb from "@lancedb/lancedb";
import { OAuthConfig, OAuthFlowType } from "@lancedb/lancedb";
// Client Credentials (service-to-service)
const oauthConfig: OAuthConfig = {
issuerUrl: "https://login.microsoftonline.com/{tenant}/v2.0",
clientId: "your-application-client-id",
clientSecret: "your-application-client-secret",
scopes: ["api://lancedb-api/.default"],
flow: OAuthFlowType.ClientCredentials,
};
const db = await lancedb.connect("db://your-database-uri", {
region: "us-east-1",
hostOverride: "https://your-enterprise-endpoint.com",
oauthConfig,
});
```
</CodeGroup>
For workloads running on Azure compute, use a managed identity instead of a
client secret:
<CodeGroup>
```python Python icon="python"
from lancedb.remote import OAuthConfig, OAuthFlowType
oauth_config = OAuthConfig(
issuer_url="https://login.microsoftonline.com/{tenant}/v2.0",
client_id="your-application-client-id",
scopes=["api://lancedb-api/.default"],
flow=OAuthFlowType.AZURE_MANAGED_IDENTITY,
# Optional: required only for user-assigned managed identities.
managed_identity_client_id="your-user-assigned-identity-client-id",
)
```
```typescript TypeScript icon="square-js"
import { OAuthConfig, OAuthFlowType } from "@lancedb/lancedb";
const oauthConfig: OAuthConfig = {
issuerUrl: "https://login.microsoftonline.com/{tenant}/v2.0",
clientId: "your-application-client-id",
scopes: ["api://lancedb-api/.default"],
flow: OAuthFlowType.AzureManagedIdentity,
// Optional: required only for user-assigned managed identities.
managedIdentityClientId: "your-user-assigned-identity-client-id",
};
```
</CodeGroup>
### Configuration reference {#configuration-reference}
The same configuration is available in both SDKs. Python uses `snake_case`
field names; TypeScript uses `camelCase`.
| Python field / TypeScript field | Required | Description |
| --- | --- | --- |
| `issuer_url` / `issuerUrl` | Yes | OIDC issuer URL or OAuth authority URL. For Azure, use `https://login.microsoftonline.com/{tenant_id}/v2.0`. |
| `client_id` / `clientId` | Yes | Application / client ID registered with your identity provider. |
| `scopes` | Yes | List of OAuth scopes to request. For Azure managed identity, provide exactly one scope or resource, for example `["api://your-app-id/.default"]`. |
| `flow` | No | Selects the OAuth flow. Defaults to Client Credentials. |
| `client_secret` / `clientSecret` | Conditional | Required for Client Credentials. Redacted from the config's `repr` (Python) and `Debug` output (TypeScript native binding) so it does not leak into logs. |
| `managed_identity_client_id` / `managedIdentityClientId` | No | Client ID of a user-assigned managed identity. Only used with Azure Managed Identity; omit for system-assigned identities. |
| `refresh_buffer_secs` / `refreshBufferSecs` | No | How many seconds before token expiry to proactively refresh. Defaults to 300. Keep this well below the token TTL -- setting it greater than or equal to the TTL forces a refresh on every request. |
<Warning>
Treat the client secret like any other production credential. Load it from a
secret manager or environment variable rather than committing it to source
control. Both SDKs deliberately omit the secret from their debug/repr output
so accidental log lines do not expose the value.
</Warning>
Embedded LanceDB has nothing to authenticate to -- your process opens the files
directly, and the object store's own credentials are the only gate.
docs/web/feature-engineering/data-evolution.mdx
+1 -1
View File
@@ -27,7 +27,7 @@ column -- the existing data, which is the expensive part on a multimodal table,
is never rewritten. See [why that works](/basics/lance#evolution).
<Note>
In SQL this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl#computed-column).
In SQL this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl/table#computed-column).
</Note>
## Set up the example table {#set-up-the-example-table}
docs/web/feature-engineering/index.mdx
+1 -1
View File
@@ -17,7 +17,7 @@ versioned, and refreshed like anything else, not a separate system bolted
alongside.
<Note>
Every operation here has a SQL form -- see [DDL](/sql/ddl#columns) to declare a column and [REFRESH](/sql/refresh) to fill it.
Every operation here has a SQL form -- see [DDL](/sql/ddl/table#alter) to declare a column and [REFRESH](/sql/dml/refresh) to fill it.
</Note>
## The shape everything shares {#the-shape}
docs/web/governance/authorization.mdx
+98 -9
View File
@@ -8,19 +8,108 @@ keywords: ["authorization", "authz", "privileges", "rbac", "access control"]
<Badge color="red">Enterprise</Badge>
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it -- see
This page describes a LanceDB Enterprise capability. Embedded LanceDB has no
such layer -- access is whatever the object store enforces on the bucket or
directory, which is coarser and has no concept of a table. See
[Architecture](/architecture) for what differs between the two.
[Authentication](/enterprise/authentication) establishes who is calling.
Authorization decides what they may do.
An access control entry grants **one privilege** on **one object** to **one
subject**. Objects are addressed by kind and path -- `system`, `database:db`,
`namespace:db/ns`, `table:db/ns/table` -- and the hierarchy is not decorative:
reaching a table requires `USAGE` on the namespace containing it and on the
database containing that.
subject**. Everything below is a consequence of that shape.
Embedded LanceDB has no such layer. Access is whatever the object store
enforces on the bucket or directory, which is coarser and has no concept of a
table.
## Subjects {#subjects}
A grant is held by one of four kinds of subject, distinguished by a prefix:
| Prefix | Subject |
| --- | --- |
| `p:` | a principal, by ID -- `P:` by name |
| `g:` | a group, by ID -- `G:` by name |
| `r:` | a role |
| `a:` | an API key |
Granting to a group or role rather than a principal is what keeps the entry
count manageable: a principal's effective privileges are the union of what it
holds directly and what it inherits.
Granting to an API key is narrower than granting to the principal that owns
it -- useful when a key is issued for one job and should not carry that
principal's full authority.
## Objects {#objects}
Objects are addressed by kind and path:
```
system
database:db
namespace:db/ns
table:db/ns/table
```
The hierarchy is not decorative. Reaching a table requires `USAGE` on the
namespace containing it and on the database containing that -- a grant on the
table alone is not enough to traverse to it.
## Privileges {#privileges}
| Privilege | Granted on | Allows |
| --- | --- | --- |
| `OWNERSHIP` | any object | ownership, conveying full authority |
| `USAGE` | database | accessing the namespaces inside |
| `USAGE` | namespace | accessing the tables inside |
| `SELECT` | table | reading rows |
| `INSERT` | table | appending rows |
| `UPDATE` | table | modifying existing rows |
| `DELETE` | table | deleting rows |
| `CREATE_DATABASE` | system | creating a database |
| `CREATE_NAMESPACE` | database, namespace | creating immediate child namespaces |
| `CREATE_TABLE` | namespace | creating a table in it |
| `CREATE_MATERIALIZED_VIEW` | namespace | creating a materialized view in it |
| `CREATE_VIEW` | namespace | creating a non-materialized view in it |
| `OPERATE` | system | reading server configuration |
The four data privileges map onto the four
[ingestion operations](/ingestion), so a principal that may append but not
modify is expressed by granting `INSERT` without `UPDATE`.
<Note>
`CREATE_MATERIALIZED_VIEW` and `CREATE_VIEW` are separate privileges because
[the two objects differ](/basics/catalog): one stores rows and is refreshed,
the other stores only a query. Granting the ability to create one does not
grant the other.
</Note>
## Creation privileges resolve upward {#upward}
A subtlety worth knowing, because it explains grants that appear to work on
objects that do not exist yet.
When `USAGE` is checked on a database and **no entries exist for it at all**,
the database is assumed not to exist yet, and `CREATE_DATABASE` on `system` is
checked instead. The same applies to the default namespace: absent any
entries, the check falls back to `CREATE_NAMESPACE` on the containing
database.
So a principal holding `CREATE_DATABASE` can act on a database that has not
been created -- which is what makes creation possible at all, and why an empty
ACL set is not the same as a deny.
## Listing grants {#listing}
Entries can be listed filtered by object, by subject, or both:
```
object = table:db/ns/documents
subject = G:data-science
```
Results paginate through a continuation token. One stored `(object, subject)`
tuple carrying several privileges expands into several entries in the
response, so the count reflects privileges rather than tuples.
Listing by subject answers "what can this team reach"; listing by object
answers "who can reach this table". Both are worth running before changing a
grant.
docs/web/ingestion/delete.mdx
+1 -1
View File
@@ -18,7 +18,7 @@ import {
} from '/snippets/tables.mdx';
<Note>
In SQL this is [`DELETE`](/sql/dml#delete).
In SQL this is [`DELETE`](/sql/dml/delete).
</Note>
## Delete rows {#delete-rows}
docs/web/ingestion/index.mdx
+1 -1
View File
@@ -30,7 +30,7 @@ writing to a table that exists. This page covers the first and introduces the
second.
<Note>
Every operation here has a SQL form -- see [DML](/sql/dml).
Every operation here has a SQL form -- see [DML](/sql#dml).
</Note>
Start by [creating a table](/ingestion/create-table); the rest of this
docs/web/ingestion/insert.mdx
+1 -1
View File
@@ -20,7 +20,7 @@ import {
} from '/snippets/tables.mdx';
<Note>
In SQL this is [`INSERT`](/sql/dml#insert), including `INSERT ... SELECT` to move rows server-side.
In SQL this is [`INSERT`](/sql/dml/insert), including `INSERT ... SELECT` to move rows server-side.
</Note>
## Ingesting large datasets {#large-datasets}
docs/web/ingestion/merge-insert.mdx
+1 -1
View File
@@ -27,7 +27,7 @@ import {
} from '/snippets/tables.mdx';
<Note>
Merge insert has **no SQL form** -- it needs a source relation and a per-match policy that one statement cannot express. See [DML](/sql/dml#merge-insert).
Merge insert has **no SQL form** -- it needs a source relation and a per-match policy that one statement cannot express. See [DML](/sql/dml/insert#next).
</Note>
## Merge incoming rows by key {#merge-incoming-rows-by-key}
docs/web/ingestion/update.mdx
+2 -2
View File
@@ -18,7 +18,7 @@ import {
} from '/snippets/tables.mdx';
<Note>
In SQL this is [`UPDATE`](/sql/dml#update).
In SQL this is [`UPDATE`](/sql/dml/update).
</Note>
## Update rows {#update-rows}
@@ -76,7 +76,7 @@ Expected table contents:
| 2 | Bob | 21 |
<Note title="SQL syntax">
See the [SQL queries](/search/sql/) page for more information on the supported SQL syntax.
See the [SQL queries](/sql#connecting) page for more information on the supported SQL syntax.
</Note>
When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows, consider triggering an index rebuild afterwards.
docs/web/maintenance/background-jobs.mdx
+3 -3
View File
@@ -13,14 +13,14 @@ Every job documented elsewhere in these docs -- [Build & Repair](/indexing/reind
[Column Refresh](/feature-engineering/refresh-column) -- is one you cause,
directly or by a schedule you set. This page is the rest: jobs the deployment
runs to keep itself correct and fast, which you may see named in
[`SHOW JOBS`](/sql/ddl#job) without having asked for any of them.
[`SHOW JOBS`](/sql/ddl/job#show) without having asked for any of them.
There is no action to take on these beyond recognizing the name. None of them
has a parameter you set or a call you make.
## Write-ahead log maintenance {#wal}
For a table with a [write-ahead log](/sql/ddl#table) enabled, two jobs keep the
For a table with a [write-ahead log](/sql/ddl/table#properties) enabled, two jobs keep the
log from growing without bound:
| Job type | What it does |
@@ -61,5 +61,5 @@ table.
| `drop_object` | Deletes a dropped catalog object's content, then clears it from the database's cleanup backlog. Covers namespaces, views, and functions -- everything [`drop_table`](/management/table#drop) does not, since that one fences a directory with its own marker instead. |
You trigger this indirectly, by dropping a [namespace](/management/namespace),
[view](/sql/ddl#view), or [function](/management/function#drop) -- the teardown
[view](/sql/ddl/view#drop), or [function](/management/function#drop) -- the teardown
itself runs as this job rather than inline with the call.
docs/web/management/database.mdx
+1 -1
View File
@@ -77,7 +77,7 @@ const db = await lancedb.connect("db://my-database", {
| `api_key` / `apiKey` | authenticates the connection -- see [Authentication](/enterprise/authentication) |
| `region` | which region hosts the database. Defaults to `us-east-1` |
| `host_override` / `hostOverride` | point at a specific endpoint instead of the regional one |
| `sql_host_override` | the same, for the [SQL endpoint](/search/sql). Python only |
| `sql_host_override` | the same, for the [SQL endpoint](/sql#connecting). Python only |
| `client_config` / `clientConfig` | retries, timeouts, and connection pooling |
An API key belongs in the environment, not in source. `region` is easy to get
docs/web/management/function.mdx
+1 -1
View File
@@ -18,7 +18,7 @@ raises `NotImplementedError`; use a
[computed column](/feature-engineering/refresh-column) instead.
<Note>
In SQL, a function is registered with [`CREATE FUNCTION`](/sql/ddl#functions) and listed with [`SHOW FUNCTIONS`](/sql/show#functions).
In SQL, a function is registered with [`CREATE FUNCTION`](/sql/ddl/function#create) and listed with [`SHOW FUNCTIONS`](/sql/ddl/function#show).
</Note>
## Define a function {#define}
docs/web/management/materialized-view.mdx
+1 -1
View File
@@ -17,7 +17,7 @@ This page covers creating and managing one. Keeping it current is
[Refresh a materialized view](/feature-engineering/materialized-views).
<Note>
In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view) and [`SHOW MATERIALIZED VIEWS`](/sql/show#materialized-views).
In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view) and [`SHOW MATERIALIZED VIEWS`](/sql/ddl/materialized-view#show).
</Note>
## Creating one {#create}
docs/web/management/namespace.mdx
+1 -1
View File
@@ -29,7 +29,7 @@ language: Python uses `namespace_path`, Rust uses builder methods like `.namespa
TypeScript uses `namespacePath` arguments.
<Note>
In SQL a namespaced table is addressed as a [dotted path](/sql/select#namespaces).
In SQL a namespaced table is addressed as a [dotted path](/sql/dql/select#namespaces).
</Note>
## Namespace hierarchy {#namespace-hierarchy}
docs/web/management/table.mdx
+2 -2
View File
@@ -121,7 +121,7 @@ This page covers the table as a catalog object. Getting data into one is
[Schema evolution](/management/table#schema).
<Note>
Tables are created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl#tables).
Tables are created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl/table).
</Note>
## What a table carries {#carries}
@@ -500,7 +500,7 @@ LanceDB supports ACID-compliant schema evolution through granular operations (ad
* Optimize Continuously: Remove unused fields or enforce new constraints without downtime
<Note>
In SQL, schema changes are [`ALTER TABLE`](/sql/ddl#columns).
In SQL, schema changes are [`ALTER TABLE`](/sql/ddl/table#alter).
</Note>
### Schema evolution operations {#schema-schema-evolution-operations}
docs/web/management/view.mdx
+3 -3
View File
@@ -27,6 +27,6 @@ defined by a query; the difference is entirely when that query runs.
| Read cost | the full query, every read | a table scan |
| Write cost | none | a refresh |
Views are created in SQL, over the [Flight SQL endpoint](/search/sql) -- there
is no SDK method, because the thing a view stores *is* a SQL statement. See
[DDL](/sql/ddl#views).
Views are created in SQL, over the [Flight SQL endpoint](/sql#connecting) --
there is no SDK method, because the thing a view stores *is* a SQL statement.
See [View](/sql/ddl/view) for `CREATE VIEW`, `DROP VIEW`, and listing.
docs/web/search/explain.mdx
+2 -2
View File
@@ -15,7 +15,7 @@ To measure a query rather than read it, use
operator with runtime metrics.
<Note>
The SQL equivalent is [`EXPLAIN`](/sql/explain).
The SQL equivalent is [`EXPLAIN`](/sql/dql/explain).
</Note>
## What it shows {#inspect-a-query-plan}
@@ -208,7 +208,7 @@ Where [`explain_plan`](#inspect-a-query-plan) tells you what a query intends to
this tells you what it cost.
<Note>
The SQL equivalent is [`EXPLAIN ANALYZE`](/sql/explain).
The SQL equivalent is [`EXPLAIN ANALYZE`](/sql/dql/explain).
</Note>
## What it measures {#analyze-a-query-plan}
docs/web/search/filtering.mdx
+1 -1
View File
@@ -15,7 +15,7 @@ with filtering capabilities even on datasets containing billions of records.
On hybrid queries, the same `where(...)` filter is applied to both the vector and full-text halves of the query. The prefilter or postfilter choice controls whether that happens before each subquery scores candidates or after the subquery top-k is produced.
<Note>
The same predicates work in SQL -- see [SELECT](/sql/select#filtering) and [Operators](/sql/operators).
The same predicates work in SQL -- see [SELECT](/sql/dql/select#filtering) and [Operators](/sql/operators).
</Note>
## Chaining `where` clauses {#chaining-where-clauses}
docs/web/search/index.mdx
+1 -1
View File
@@ -12,7 +12,7 @@ icon: "list"
| [Full-Text Search](/search/full-text-search/) | Keyword-based search with BM25 and pre-filtering |
| [Hybrid Search](/search/hybrid-search/) | Combines vector and full-text search with reranking |
| [Filtering](/search/filtering/) | Filter results based on metadata fields |
| [SQL Queries](/search/sql/index) | SQL query capabilities for data exploration and analytics |
| [SQL Queries](/sql) | SQL query capabilities for data exploration and analytics |
## Before you search {#before-you-search}
docs/web/search/sql/fts-sql.mdx
-32
View File
@@ -1,32 +0,0 @@
---
title: "Full-text search with SQL"
sidebarTitle: "Full-Text Search"
description: "Running full-text search from SQL, with fuzzy, phrase, boolean and boost queries."
icon: magnifying-glass
keywords: ["fts", "sql", "full text search", "fuzzy", "phrase"]
---
<Badge color="red">Enterprise</Badge>
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it -- see
[Architecture](/architecture) for what differs between the two.
Full-text search is available in SQL as the [`fts()`](/sql/functions#fts) table
function, which returns matching rows plus a `_score` column:
```sql
SELECT id, text, _score
FROM fts('my_docs', 'puppy')
ORDER BY _score DESC
LIMIT 5
```
The second argument is a bare query string, or a JSON query object for the
fuzzy, phrase, boolean, boost and multi-match forms.
For a keyword test that needs no score,
[`contains_tokens`](/sql/scalar-functions#contains-tokens) is a predicate usable
directly in `WHERE`, and is accelerated by the full-text index.
The query-builder equivalent is [Full-Text Search](/search/full-text-search).
docs/web/search/sql/index.mdx
-29
View File
@@ -1,29 +0,0 @@
---
title: "Connecting a SQL client"
sidebarTitle: "Client Connection"
description: "Reaching the Arrow Flight SQL endpoint from Python or TypeScript."
icon: plug
keywords: ["flight sql", "client", "connect", "arrow"]
---
<Badge color="red">Enterprise</Badge>
This page describes a LanceDB Enterprise capability. The embedded form
of LanceDB has no server to provide it -- see
[Architecture](/architecture) for what differs between the two.
SQL is served over an
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint, so any Flight SQL client reaches it and results arrive as Arrow
record batches.
From Python, the connection runs SQL directly without a separate client:
```python
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10")
table = reader.read_all()
```
The TypeScript SDK has no SQL method; use a Flight SQL client. See
[SDKs and SQL](/basics/apis) for the full comparison, and the
[SQL reference](/sql) for the dialect.
docs/web/sql/ddl.mdx
-192
View File
@@ -1,192 +0,0 @@
---
title: "DDL"
sidebarTitle: "DDL"
description: "Creating and altering catalog objects: tables, views, indexes, functions, and columns."
icon: pen-ruler
keywords: ["ddl", "create table", "alter table", "add column", "create function", "create index", "wal"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
Statements that change what exists rather than what it holds. For the values in
a table, see [DML](/sql/dml).
## Tables {#tables}
```sql
CREATE TABLE documents (id BIGINT, title VARCHAR, created_at TIMESTAMP)
CREATE TABLE recent AS SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
DROP TABLE documents
```
Column types are Arrow types under SQL names -- see
[Data types](/sql/data-types). A table in a [namespace](/management/namespace) is written
as a dotted path: `prod.events.clicks`.
## Views {#views}
```sql
CREATE VIEW recent_matches AS
SELECT t1.floats FROM documents AS t1
INNER JOIN categories AS t2 ON t1.ints = t2.ints
DROP VIEW recent_matches
```
`CREATE VIEW` returns the view's schema -- column name, type, nullability --
resolved from the query at creation time. A view stores only the query; see
[Views](/management/view).
## Columns {#columns}
`ALTER TABLE ... ADD COLUMN` covers three different things, distinguished by
what follows the column name.
### A plain column {#plain-column}
```sql
ALTER TABLE t ADD COLUMN c INT
```
Added empty and nullable. Nothing is computed.
### A computed column {#computed-column}
```sql
ALTER TABLE t ADD COLUMN doubled AS value * 2
```
`AS <expression>` declares a [computed column](/feature-engineering/add-computed-column).
The column is committed empty and the expression is stored, not evaluated --
declaring one on a billion-row table costs what declaring one on an empty table
costs. [`REFRESH COLUMN`](/sql/refresh) fills it.
### A function column {#function-column}
```sql
ALTER TABLE t ADD COLUMN embedded AS embed(name) WITH (version = 'fv_1')
```
When the expression calls a [registered function](#functions), the column is
filled by running that function on workers. `version` pins which immutable
function version fills it, so a later registration does not silently change what
the column means.
Several modifiers narrow what gets computed:
| | |
| --- | --- |
| `FILTER (WHERE pred)` | only rows matching `pred` are computed |
| `IGNORE NULLS` | rows with null inputs are skipped |
| `OVER (PARTITION BY col)` | the function sees rows grouped by `col` |
A function returning several values fills several columns at once:
```sql
ALTER TABLE t ADD COLUMN (label, score) AS classify(text) WITH (version = 'fv_1')
```
```sql
ALTER TABLE t ADD COLUMN c AS f(x) FILTER (WHERE keep) WITH (version = 'fv_1')
ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2')
```
<Note>
`WITH` on a plain computed column accepts only `version`. Resource options such
as `num_workers` apply to function columns, where there are workers to size.
</Note>
## Functions {#functions}
```sql
CREATE FUNCTION f(x BIGINT) RETURNS BIGINT
LANGUAGE python
AS 'def f(x): return x * 2'
WITH (pip = 'numpy==1.26.4', python_version = '3.11')
```
The body is Python source. Its dependencies are declared with it and are part of
the **immutable function version** -- changing one produces a new version rather
than altering the registered one, so a column filled by an earlier version is
unaffected.
| Option | |
| --- | --- |
| `pip` | pip requirements for the worker environment |
| `conda` | conda packages instead of pip |
| `conda_channels` | channels in priority order; requires `conda` |
| `python_version` | worker Python version, defaulting to yours |
| `gpu` | `'true'` when every execution needs a GPU |
**Errors and limits**
| | |
| --- | --- |
| `LANGUAGE` other than `python` | refused |
| `pip` and `conda` together | refused -- pick one |
| `conda_channels` without `conda` | refused |
| `gpu` set to anything but `'true'` | refused |
| An empty body | refused |
| Any other option | refused, and the message lists what is supported |
See [Functions](/management/function) for the SDK equivalent and for diagnosing
environment mismatches.
## Indexes {#indexes}
```sql
CREATE INDEX ON documents (title)
DROP INDEX documents.title_idx
```
<Note>
SQL `CREATE INDEX` covers **one column** and takes **no user-chosen name** --
the server names it. For a multi-column or named index, use the SDK. `DROP
INDEX` takes the qualified `table.index` name; find it with `list_indices`.
</Note>
Index type and parameters are chosen in the SDK; see [Indexing](/indexing).
## Table properties {#properties}
```sql
ALTER TABLE t SET UNENFORCED PRIMARY KEY (id)
```
Records which column identifies a row. **Unenforced** is not a hedge in the
naming: uniqueness is not checked on write, and declaring it does not make
duplicates impossible. It tells the planner what to assume.
```sql
ALTER TABLE t SET WAL
ALTER TABLE t SET WAL WITH (sharding = 'bucket', buckets = 4)
ALTER TABLE t SET WAL WITH (maintained_indexes = '')
ALTER TABLE t UNSET WAL
```
A write-ahead log makes small frequent writes cheap by batching them out of the
commit path. `sharding` and `buckets` spread the log so concurrent writers do
not contend on one place; `maintained_indexes` chooses which indexes are kept
current against WAL rows.
<Warning>
[`TABLESAMPLE`](/sql/sampling) is refused on a WAL-backed table, because rows a
sample would draw from are not all in the fragments yet.
</Warning>
## See also {#see-also}
<CardGroup cols={2}>
<Card title="DML" icon="pen-to-square" href="/sql/dml">
Changing rows rather than structure.
</Card>
<Card title="Refresh" icon="arrows-rotate" href="/sql/refresh">
Filling the columns declared here.
</Card>
<Card title="Management" icon="folder-tree" href="/management">
The same operations from the SDKs.
</Card>
<Card title="SHOW" icon="list" href="/sql/show">
Listing what exists.
</Card>
</CardGroup>
docs/web/sql/ddl/function.mdx
+86
View File
@@ -0,0 +1,86 @@
---
title: "Function"
sidebarTitle: "Function"
description: "CREATE FUNCTION, SHOW FUNCTIONS, and DROP FUNCTION."
icon: function
keywords: ["create function", "drop function", "show functions", "sql", "udf"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
## CREATE FUNCTION {#create}
```sql
CREATE FUNCTION f(x BIGINT) RETURNS BIGINT
LANGUAGE python
AS 'def f(x): return x * 2'
WITH (pip = 'numpy==1.26.4', python_version = '3.11')
```
The body is Python source. Its dependencies are declared with it and are part
of the **immutable function version** -- changing one produces a new version
rather than altering the registered one, so a column filled by an earlier
version is unaffected.
| Option | |
| --- | --- |
| `pip` | pip requirements for the worker environment |
| `conda` | conda packages instead of pip |
| `conda_channels` | channels in priority order; requires `conda` |
| `python_version` | worker Python version, defaulting to yours |
| `gpu` | `'true'` when every execution needs a GPU |
**Errors and limits**
| | |
| --- | --- |
| `LANGUAGE` other than `python` | refused |
| `pip` and `conda` together | refused -- pick one |
| `conda_channels` without `conda` | refused |
| `gpu` set to anything but `'true'` | refused |
| An empty body | refused |
| Any other option | refused, and the message lists what is supported |
<Note>
**Enterprise job type:** `create_function`. Registration returns a job because
the function is baked into an image before the version becomes usable.
</Note>
## SHOW FUNCTIONS {#show}
```sql
SHOW FUNCTIONS
```
Lists registered functions and their versions. Since a version is immutable
and a column pins the version that fills it, this is how you find which
versions are still in use before dropping one.
## DROP FUNCTION {#drop}
```sql
DROP FUNCTION embed WITH (version = '1')
DROP FUNCTION IF EXISTS embed WITH (version = '1')
DROP FUNCTION embed WITH (version = '1', cascade = 'true')
```
`version` is required -- a function is a set of immutable versions, and
dropping is always of one specific version, never "the function" as a whole.
<Warning>
Without `cascade = 'true'`, dropping a version that a column still pins fails
rather than breaking that column silently. Passing it drops the version and
lets columns pinned to it stop refreshing -- they keep their last-filled
values, per [Guarantees](/basics/guarantees#jobs).
</Warning>
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Function" icon="function" href="/management/function">
Registering one from the SDK, and diagnosing environment mismatches.
</Card>
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/add-computed-column">
Filling a column with a function instead of an expression.
</Card>
</CardGroup>
docs/web/sql/ddl/index.mdx
+49
View File
@@ -0,0 +1,49 @@
---
title: "Index"
sidebarTitle: "Index"
description: "CREATE INDEX and DROP INDEX — one column, server-chosen name."
icon: bolt
keywords: ["create index", "drop index", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
## CREATE INDEX {#create}
```sql
CREATE INDEX ON documents (title)
```
<Note>
SQL `CREATE INDEX` covers **one column** and takes **no user-chosen name** --
the server names it. For a multi-column or explicitly-named index, or to
choose the algorithm, use the SDK's
[`create_index`](/indexing#choosing-an-index).
</Note>
<Note>
**Enterprise job type:** `create_ivf_pq`, `create_btree`, `create_fts`, and one
more per algorithm -- the type names which algorithm is building, not a
different feature. See [Build & Repair](/indexing/reindexing).
</Note>
## DROP INDEX {#drop}
```sql
DROP INDEX documents.title_idx
```
Takes the qualified `table.index` name. Find it with the SDK's
[`list_indices`](/indexing#choosing-an-index) if you created the index without
naming it yourself.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Indexing" icon="bolt" href="/indexing">
Choosing an index type from the SDK.
</Card>
<Card title="Build & Repair" icon="refresh" href="/indexing/reindexing">
Keeping an index current as data arrives.
</Card>
</CardGroup>
docs/web/sql/ddl/job.mdx
+90
View File
@@ -0,0 +1,90 @@
---
title: "Job"
sidebarTitle: "Job"
description: "SHOW JOBS, SHOW ERRORS, and controlling a running job with PAUSE, RESUME and CANCEL."
icon: hourglass-half
keywords: ["show jobs", "show errors", "pause job", "resume job", "cancel job", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
A [job](/basics/jobs) is not a catalog object -- it has no schema and is not
namespaced -- but it has its own SQL statements, grouped here the same way the
catalog objects are.
## SHOW JOBS {#show}
```sql
SHOW JOBS
```
Every job the server knows about -- id, type, state, creation time. The SQL
counterpart of `list_jobs`.
## SHOW ERRORS {#errors}
```sql
SHOW ERRORS ON <table> [FOR JOB '<job-id>'] [LIMIT <n>]
```
```sql
SHOW ERRORS ON documents
SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100
```
Why rows failed. Reach for this when a [refresh](/sql/dml/refresh) reports
fewer filled rows than you expected.
**Errors and limits**
| | |
| --- | --- |
| `LIMIT` | must be between 1 and 100,000 |
| Failed jobs scanned without `FOR JOB` | 32 most recent |
| Tables the server does not open directly | not supported |
Without `FOR JOB` this surveys recent failures and stops at 32 jobs. When the
listing stops at your `LIMIT`, the message says so -- narrow with `FOR JOB`
rather than raising the limit blindly.
## PAUSE JOB {#pause}
```sql
PAUSE JOB 'j-ci-abc-def'
```
## RESUME JOB {#resume}
```sql
RESUME JOB 'j-ci-abc-def'
```
## CANCEL JOB {#cancel}
```sql
CANCEL JOB 'j-ci-abc-def'
```
The job id is always a quoted string literal. All three take exactly one and
nothing after it -- a second token on the line is a syntax error rather than
being ignored.
<Note>
Cancellation is a request, matching the SDK's
[`cancel()`](/basics/jobs#managing): it does not guarantee the job stops before
doing more work, and asking twice or cancelling a finished job does nothing.
`CANCEL JOB` itself can be unavailable on a deployment with no job registry
configured, which is a deployment condition rather than something you can work
around from SQL.
</Note>
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Query and job" icon="hourglass-half" href="/basics/jobs">
The handle these statements are the SQL view of.
</Card>
<Card title="Background jobs" icon="gears" href="/maintenance/background-jobs">
Job types you did not start yourself.
</Card>
</CardGroup>
docs/web/sql/ddl/materialized-view.mdx
+62
View File
@@ -0,0 +1,62 @@
---
title: "Materialized View"
sidebarTitle: "Materialized View"
description: "SHOW MATERIALIZED VIEWS — creation and refresh are SDK and REFRESH statements, not CREATE."
icon: table
keywords: ["materialized view", "show materialized views", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
A materialized view stores a query's result as rows, kept current by a refresh
-- the counterpart to a [view](/sql/ddl/view), which stores only the query.
<Note>
There is no `CREATE MATERIALIZED VIEW` statement. Creation is
[`create_materialized_view`](/management/materialized-view#create) in the SDK,
since it needs SDK-level source-table validation (stable row ids) that a bare
SQL statement has nowhere to check. Once created, it is entirely a SQL object:
queried like a table, refreshed with SQL, listed with SQL.
</Note>
## SHOW MATERIALIZED VIEWS {#show}
```sql
SHOW MATERIALIZED VIEWS
```
<Note>
Cheaper than the SDK equivalent. `list_materialized_views` finds views by
reading every table's schema, costing one open per table; this asks the server.
</Note>
## Refreshing one {#refresh}
```sql
REFRESH MATERIALIZED VIEW chunks
```
See [REFRESH](/sql/dml/refresh#refresh-materialized-view) for the full syntax,
including `full = true` and what `CASCADE` does not yet do.
## Dropping one {#drop}
A materialized view **is** a table, so it is dropped with
[`DROP TABLE`](/sql/ddl/table#drop) -- which is also why dropping one removes
stored rows, unlike dropping a [view](/sql/ddl/view#drop).
<Note>
**Enterprise job type:** `mat_view_refresh` for the refresh; `drop_table` for
dropping it, same as any table.
</Note>
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Materialized View" icon="table" href="/management/materialized-view">
Creating one from the SDK, and the stable-row-ids requirement.
</Card>
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/materialized-views">
Incremental refreshes, and what they cost.
</Card>
</CardGroup>
docs/web/sql/ddl/namespace.mdx
+53
View File
@@ -0,0 +1,53 @@
---
title: "Namespace"
sidebarTitle: "Namespace"
description: "CREATE SCHEMA and DROP SCHEMA — the SQL spelling of namespace DDL."
icon: folder-tree
keywords: ["namespace", "schema", "create schema", "drop schema", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
A [namespace](/management/namespace) is addressed in SQL as a `SCHEMA` -- the
standard's name for the same concept.
## CREATE SCHEMA {#create}
```sql
CREATE SCHEMA public
CREATE SCHEMA db1.ns1.ns2
CREATE SCHEMA IF NOT EXISTS `ns1$ns2`
```
A dotted path creates the whole chain: `db1.ns1.ns2` creates `ns2` inside
`ns1` inside `db1`, provided the parents already exist. A name containing the
namespace path delimiter itself is backtick-quoted, as in `` `ns1$ns2` `` above.
`IF NOT EXISTS` makes the statement a no-op rather than an error when the
namespace is already there.
## DROP SCHEMA {#drop}
```sql
DROP SCHEMA db1.public
DROP SCHEMA IF EXISTS `a$b`
DROP SCHEMA `a$b` RESTRICT
```
<Warning>
`CASCADE` is rejected: **"DROP SCHEMA CASCADE is not supported; drop the
objects inside first."** A namespace's tables, views and functions must be
dropped explicitly before the namespace itself can go. `RESTRICT` -- the
default -- states that requirement rather than changing it.
</Warning>
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Namespace" icon="folder-tree" href="/management/namespace">
The SDK equivalent, and how a namespace fits the catalog hierarchy.
</Card>
<Card title="Table" icon="table" href="/sql/ddl/table">
Creating tables inside a namespace.
</Card>
</CardGroup>
docs/web/sql/ddl/table.mdx
+178
View File
@@ -0,0 +1,178 @@
---
title: "Table"
sidebarTitle: "Table"
description: "CREATE, DESCRIBE, ALTER, CLONE and DROP TABLE."
icon: table
keywords: ["create table", "alter table", "drop table", "describe", "clone", "ddl"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
## CREATE TABLE {#create}
```sql
CREATE TABLE documents (id BIGINT, title VARCHAR, created_at TIMESTAMP)
CREATE TABLE recent AS SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
```
Column types are Arrow types under SQL names -- see [Data types](/sql/data-types).
A table inside a [namespace](/sql/ddl/namespace) is written as a dotted path:
`prod.events.clicks`.
## DESCRIBE {#describe}
```sql
DESCRIBE documents
```
The table's columns and types, without a plan -- the SQL counterpart of
`table.schema`.
## ALTER TABLE {#alter}
`ALTER TABLE ... ADD COLUMN` covers three different things, distinguished by
what follows the column name.
### A plain column {#plain-column}
```sql
ALTER TABLE t ADD COLUMN c INT
```
Added empty and nullable. Nothing is computed.
### A computed column {#computed-column}
```sql
ALTER TABLE t ADD COLUMN doubled AS value * 2
```
`AS <expression>` declares a [computed column](/feature-engineering/add-computed-column).
The column is committed empty and the expression is stored, not evaluated --
declaring one on a billion-row table costs what declaring one on an empty table
costs. [`REFRESH COLUMN`](/sql/dml/refresh) fills it.
### A function column {#function-column}
```sql
ALTER TABLE t ADD COLUMN embedded AS embed(name) WITH (version = 'fv_1')
```
When the expression calls a [registered function](/sql/ddl/function), the
column is filled by running that function on workers. `version` pins which
immutable function version fills it, so a later registration does not silently
change what the column means.
Several modifiers narrow what gets computed:
| | |
| --- | --- |
| `FILTER (WHERE pred)` | only rows matching `pred` are computed |
| `IGNORE NULLS` | rows with null inputs are skipped |
| `OVER (PARTITION BY col)` | the function sees rows grouped by `col` |
A function returning several values fills several columns at once:
```sql
ALTER TABLE t ADD COLUMN (label, score) AS classify(text) WITH (version = 'fv_1')
```
```sql
ALTER TABLE t ADD COLUMN c AS f(x) FILTER (WHERE keep) WITH (version = 'fv_1')
ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2')
```
<Note>
`WITH` on a plain computed column accepts only `version`. Resource options such
as `num_workers` apply to function columns, where there are workers to size.
</Note>
### Table properties {#properties}
```sql
ALTER TABLE t SET UNENFORCED PRIMARY KEY (id)
```
Records which column identifies a row. **Unenforced** is not a hedge in the
naming: uniqueness is not checked on write, and declaring it does not make
duplicates impossible. It tells the planner what to assume.
```sql
ALTER TABLE t SET WAL
ALTER TABLE t SET WAL WITH (sharding = 'bucket', buckets = 4)
ALTER TABLE t SET WAL WITH (maintained_indexes = '')
ALTER TABLE t UNSET WAL
```
A write-ahead log makes small frequent writes cheap by batching them out of the
commit path. `sharding` and `buckets` spread the log so concurrent writers do
not contend on one place; `maintained_indexes` chooses which indexes are kept
current against WAL rows. See
[Background jobs](/maintenance/background-jobs#wal) for how the log gets
folded back into the table.
<Warning>
[`TABLESAMPLE`](/sql/dql/sampling) is refused on a WAL-backed table, because
rows a sample would draw from are not all in the fragments yet.
</Warning>
**Errors and limits**
| | |
| --- | --- |
| `ALTER TABLE ... DROP COLUMN` | not supported |
Dropping, renaming, or retyping a column is an SDK operation --
[`drop_columns`](/management/table#schema) and
[`alter_columns`](/management/table#schema) -- not a SQL statement.
## CLONE {#clone}
```sql
CREATE TABLE docs_snapshot CLONE documents
CREATE TABLE docs_at_v42 CLONE documents AT 42
CREATE TABLE docs_at_release CLONE documents AT TAG 'release-1'
CREATE TABLE docs_yesterday CLONE documents AT TIMESTAMP '2026-09-01T00:00:00Z'
```
A **shallow clone**: the new table shares the source's data files rather than
copying them, so it costs one metadata commit regardless of the source's size.
`AT` pins the point in the source's history to clone. A bare integer is a
version and a bare quoted string is a timestamp; `VERSION`, `TAG` and
`TIMESTAMP` say so explicitly, and a tag always needs its keyword -- a bare
quoted string cannot be both a tag name and a timestamp.
<Warning>
The clone depends on the source's files continuing to exist. If the source is
[compacted](/maintenance/cleanup) past the retention window, the version the
clone points at can be pruned. The SDK's
[`clone_table`](/management/table#rename-clone) also offers a deep clone for
when the copy has to outlive the source's history.
</Warning>
## DROP TABLE {#drop}
```sql
DROP TABLE documents
```
<Warning>
Removes the table and every version of it. There is no undrop.
</Warning>
<Note>
**Enterprise job type:** `drop_table`. See
[Background jobs](/maintenance/background-jobs#teardown).
</Note>
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Table" icon="table" href="/management/table">
The SDK equivalent, and what a table carries.
</Card>
<Card title="Schema evolution" icon="boxes-stacked" href="/management/table#schema">
Renaming, retyping and dropping columns from the SDK.
</Card>
</CardGroup>
docs/web/sql/ddl/view.mdx
+98
View File
@@ -0,0 +1,98 @@
---
title: "View"
sidebarTitle: "View"
description: "CREATE VIEW and DROP VIEW — a saved query, evaluated when read."
icon: eye
keywords: ["create view", "drop view", "sql", "catalog"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
A view stores a **query**, not rows. Reading it runs the query against current
data, so a view is never stale -- and never faster than the query it wraps. It
is the counterpart to a [materialized view](/sql/ddl/materialized-view), which
stores rows and is brought up to date by a refresh; both are
[catalog objects](/basics/catalog) defined by a query, differing only in when
that query runs.
There is no SDK method for a view, because the thing it stores *is* a SQL
statement.
## CREATE VIEW {#create}
```sql
CREATE VIEW "recent_matches" AS
SELECT t1.floats
FROM "documents" AS t1
INNER JOIN "categories" AS t2 ON t1.ints = t2.ints
```
The statement returns the view's schema -- column name, data type, nullability
-- resolved from the query at creation time. Anything the
[dialect](/sql/functions) supports can define one: joins, predicates,
aggregates, window functions.
## Reading one {#reading}
A view is queried like a table:
```sql
SELECT * FROM "recent_matches" WHERE floats > 0.5
```
Predicates applied to the view compose with the query inside it rather than
filtering its output afterwards, so a selective filter does not pay for rows
the view would have produced and then discarded.
## DROP VIEW {#drop}
```sql
DROP VIEW "recent_matches"
```
Removes the saved query. Unlike dropping a table, no data is removed -- there
was none to begin with.
## Listing {#listing}
Views and tables share a namespace, and both are things you can query. The
catalog distinguishes them:
| | |
| --- | --- |
| view names | the views in the database |
| table names | the tables |
| relation names | everything queryable -- both |
Use the relation listing when the question is "what can I select from," and the
separate listings when the distinction matters -- before dropping, for
instance, since dropping a view removes only the saved query while dropping a
table removes data.
## Choosing between a view and a materialized view {#choosing}
| | View | Materialized view |
| --- | --- | --- |
| Stores | the query | the rows |
| Freshness | always current | as of the last refresh |
| Read cost | the full query, every read | a table scan |
| Write cost | none | a refresh |
Reach for a **view** when the query is cheap, or when reading stale rows would
be wrong. Reach for a **materialized view** when the query is expensive and the
data changes less often than it is read.
The privileges are separate too: `CREATE_VIEW` and `CREATE_MATERIALIZED_VIEW`
are distinct grants on a namespace, so the ability to create one does not imply
the other. See [Authorization](/governance/authorization).
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Materialized View" icon="table" href="/sql/ddl/materialized-view">
The stored-rows counterpart.
</Card>
<Card title="Management" icon="folder-tree" href="/management">
Where views fit among the other catalog objects.
</Card>
</CardGroup>
docs/web/sql/dml.mdx
-81
View File
@@ -1,81 +0,0 @@
---
title: "DML"
sidebarTitle: "DML"
description: "INSERT, UPDATE, DELETE and COPY — changing the rows a table holds."
icon: pen-to-square
keywords: ["insert", "update", "delete", "copy", "dml"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
Statements that change rows. For the shape of the table, see [DDL](/sql/ddl).
Each of these commits a new table version, so
[versioning](/management/table#versioning) and the
[write guarantees](/basics/guarantees#writes) apply -- including that the
version advances even when a statement affects zero rows.
## INSERT {#insert}
```sql
INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur')
INSERT INTO archive SELECT * FROM documents WHERE created_at < '2026-01-01'
```
`INSERT ... SELECT` is the form worth knowing: it moves rows between tables
server-side, without a round trip through your process.
## UPDATE {#update}
```sql
UPDATE documents SET title = upper(title) WHERE category = 'film'
```
Assignments take any expression the [dialect](/sql/functions) supports,
including references to other columns of the same row. Without a `WHERE` clause
every row is updated.
<Warning>
An update does not recompute [computed columns](/feature-engineering/add-computed-column)
that derive from the column you changed. A filled value keeps what it was
computed from -- see [Guarantees](/basics/guarantees#jobs).
</Warning>
## DELETE {#delete}
```sql
DELETE FROM documents WHERE created_at < '2025-01-01'
```
Deletion marks rows rather than erasing them, so the data remains recoverable
for as long as the version holding it survives
[retention](/maintenance/compaction).
## COPY {#copy}
```sql
COPY (SELECT id, title FROM documents) TO 's3://bucket/export/' STORED AS PARQUET
```
Writes a query result to object storage. This is the export path when the
consumer is another engine rather than an application -- results land as files
rather than streaming to a client.
## Merge insert {#merge-insert}
Upsert -- matching incoming rows against existing keys -- has no SQL form. It is
an SDK operation, because it needs a source relation and a per-match policy that
a single statement cannot express. See
[Merge insert](/ingestion/merge-insert).
## See also {#see-also}
<CardGroup cols={2}>
<Card title="Ingestion" icon="inbox" href="/ingestion">
The same operations from the SDKs.
</Card>
<Card title="DDL" icon="pen-ruler" href="/sql/ddl">
Changing structure rather than rows.
</Card>
</CardGroup>
docs/web/sql/dml/copy.mdx
+23
View File
@@ -0,0 +1,23 @@
---
title: "Copy"
sidebarTitle: "Copy"
description: "COPY ... TO — writing a query result to object storage as files."
icon: file-export
keywords: ["copy", "export", "parquet", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
```sql
COPY (SELECT id, title FROM documents) TO 's3://bucket/export/' STORED AS PARQUET
```
Writes a query result to object storage. This is the export path when the
consumer is another engine rather than an application -- results land as files
rather than streaming to a client.
## What to read next {#next}
<Card title="Insert" icon="plus" href="/sql/dml/insert">
Bringing rows in, rather than exporting them.
</Card>
docs/web/sql/dml/delete.mdx
+27
View File
@@ -0,0 +1,27 @@
---
title: "Delete"
sidebarTitle: "Delete"
description: "DELETE FROM ... WHERE — rows are marked, not erased."
icon: trash
keywords: ["delete", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
```sql
DELETE FROM documents WHERE created_at < '2025-01-01'
```
Deletion marks rows rather than erasing them, so the data remains recoverable
for as long as the version holding it survives
[retention](/maintenance/cleanup).
Commits a new table version, so [versioning](/management/table#versioning) and
the [write guarantees](/basics/guarantees#writes) apply -- including that the
version advances even when the statement affects zero rows.
## What to read next {#next}
<Card title="Delete" icon="trash" href="/ingestion/delete">
The SDK equivalent.
</Card>
docs/web/sql/dml/insert.mdx
+33
View File
@@ -0,0 +1,33 @@
---
title: "Insert"
sidebarTitle: "Insert"
description: "INSERT INTO, including INSERT ... SELECT to move rows server-side."
icon: plus
keywords: ["insert", "insert into", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
```sql
INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur')
INSERT INTO archive SELECT * FROM documents WHERE created_at < '2026-01-01'
```
`INSERT ... SELECT` is the form worth knowing: it moves rows between tables
server-side, without a round trip through your process.
Commits a new table version, so [versioning](/management/table#versioning) and
the [write guarantees](/basics/guarantees#writes) apply -- including that the
version advances even when the statement affects zero rows.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Insert" icon="plus" href="/ingestion/insert">
The SDK equivalent.
</Card>
<Card title="Merge Insert" icon="code-merge" href="/ingestion/merge-insert">
Upsert -- matching incoming rows against existing keys -- has no SQL form.
</Card>
</CardGroup>
docs/web/sql/refresh.mdx → docs/web/sql/dml/refresh.mdx
+4 -4
View File
@@ -10,7 +10,7 @@ keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"]
LanceDB extensions with no DataFusion equivalent. They are what makes
[feature engineering](/feature-engineering) reachable from SQL: a column
declared by [DDL](/sql/ddl#computed-column) starts empty, and a refresh fills it.
declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it.
## REFRESH COLUMN {#refresh-column}
@@ -80,8 +80,8 @@ refreshed by refreshing each in dependency order yourself.
## Watching a refresh {#watching}
Both statements submit work that outlives the call. Track it with
[`SHOW JOBS`](/sql/show#jobs), and diagnose failures with
[`SHOW ERRORS`](/sql/show#errors).
[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with
[`SHOW ERRORS`](/sql/ddl/job#errors).
## See also {#see-also}
@@ -89,7 +89,7 @@ Both statements submit work that outlives the call. Track it with
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
The same operation from the SDKs, and what a refresh guarantees.
</Card>
<Card title="DDL" icon="pen-ruler" href="/sql/ddl#columns">
<Card title="Table" icon="table" href="/sql/ddl/table#alter">
Declaring the columns this fills.
</Card>
</CardGroup>
docs/web/sql/dml/update.mdx
+33
View File
@@ -0,0 +1,33 @@
---
title: "Update"
sidebarTitle: "Update"
description: "UPDATE ... SET ... WHERE — and why it does not recompute derived columns."
icon: pen
keywords: ["update", "sql"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
```sql
UPDATE documents SET title = upper(title) WHERE category = 'film'
```
Assignments take any expression the [dialect](/sql/functions) supports,
including references to other columns of the same row. Without a `WHERE`
clause every row is updated.
<Warning>
An update does not recompute [computed columns](/feature-engineering/add-computed-column)
that derive from the column you changed. A filled value keeps what it was
computed from -- see [Guarantees](/basics/guarantees#jobs).
</Warning>
Commits a new table version, so [versioning](/management/table#versioning) and
the [write guarantees](/basics/guarantees#writes) apply -- including that the
version advances even when the statement affects zero rows.
## What to read next {#next}
<Card title="Update" icon="pen" href="/ingestion/update">
The SDK equivalent.
</Card>
docs/web/sql/aggregation.mdx → docs/web/sql/dql/aggregation.mdx
+2 -2
View File
@@ -10,7 +10,7 @@ keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximat
Aggregation is one of the two main reasons to reach for SQL rather than a query
builder -- the SDKs project, filter and search, but they do not group. The other
is [joins](/sql/join).
is [joins](/sql/dql/join).
## Grouping {#grouping}
@@ -99,7 +99,7 @@ FROM documents
## Aggregating a sample {#sampling}
For an estimate over a very large table, aggregate a
[sample](/sql/sampling) rather than the whole thing:
[sample](/sql/dql/sampling) rather than the whole thing:
```sql
SELECT category, count(*) * 100 AS estimated
docs/web/sql/dql/explain.mdx
+52
View File
@@ -0,0 +1,52 @@
---
title: "Explain"
sidebarTitle: "Explain"
description: "EXPLAIN, EXPLAIN ANALYZE and EXPLAIN VERBOSE — reading and measuring a plan from SQL."
icon: map
keywords: ["explain", "explain analyze", "explain verbose", "query plan"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
```sql
EXPLAIN SELECT * FROM documents WHERE identifier < 1000
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
```
| Statement | What it returns |
| --- | --- |
| `EXPLAIN <query>` | Logical and physical plan, without executing the query. |
| `EXPLAIN ANALYZE <query>` | Executes the query and annotates each operator with runtime metrics (rows, timing). |
| `EXPLAIN VERBOSE <query>` | Adds intermediate optimizer plans on top of `EXPLAIN`. |
The result is a small relation with `plan_type` and `plan` columns, read
through the same client you use for regular queries -- see
[Connecting](/sql#connecting).
The operators are the same ones the SDK's
[`explain_plan` and `analyze_plan`](/search/explain) methods produce, so the
reasoning about index coverage and filter pushdown carries across unchanged:
<CardGroup cols={2}>
<Card title="What it shows" icon="map" href="/search/explain#inspect-a-query-plan">
Reading a plan: operators, pushdown, index usage.
</Card>
<Card title="What it measures" icon="gauge" href="/search/explain#analyze-a-query-plan">
Reading the metrics, and a worked before/after.
</Card>
</CardGroup>
## What to look for {#what-to-look-for}
| Sign | Means |
| --- | --- |
| `ScalarIndexQuery` | the predicate used a scalar index |
| `LanceScan` with high `bytes_read` | no index, or too many columns selected |
| `FilterExec` above a search | the filter ran after search rather than pushing into it |
| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used |
A `LanceScan` feeding a join is the common one worth acting on -- see
[Joins](/sql/dql/join#performance).
For a table's columns and types without a plan, see
[`DESCRIBE`](/sql/ddl/table#describe).
docs/web/sql/dql/fts.mdx
+136
View File
@@ -0,0 +1,136 @@
---
title: "Full-Text Search"
sidebarTitle: "Full-Text Search"
description: "The fts() table function — match, fuzzy, phrase, boolean, boost and multi-match queries from SQL."
icon: magnifying-glass
keywords: ["fts", "sql", "full text search", "fuzzy", "phrase", "boolean", "boost", "multi-match"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
Full-text search is a **table function**, `fts(table, query)`, rather than a
predicate -- it produces a relevance score alongside the matching rows, and a
`WHERE` clause has nowhere to put one. The two string-literal arguments are
the table name and a JSON-encoded query, built with the same query-builder
classes used from the SDK and serialized with `.to_json()`.
## Setup {#setup}
```sql
CREATE TABLE my_docs (id INT, text STRING, category STRING, author STRING)
INSERT INTO my_docs VALUES
(1, 'The happy puppy runs merrily in the park', 'animals', 'Alice'),
(2, 'A curious kitten jumps quickly over the fence', 'animals', 'Bob'),
(3, 'The puppy catches a ball with great enthusiasm', 'sports', 'Alice'),
(5, 'Puppy training requires patience and dedication', 'training', 'Alice')
CREATE INDEX ON my_docs USING fts (text)
```
<Warning>
Phrase queries need position information, which costs extra index size --
add `WITH (with_position = true)` to `CREATE INDEX`. Without it,
[phrase queries](#phrase) find nothing. See
[FTS parameters](/indexing/fts-index#fts-parameters).
</Warning>
## Basic search {#basic}
```python
from lancedb.query import MatchQuery
query = MatchQuery("puppy", "text")
json_query = query.to_json()
```
```sql
SELECT id, text, category FROM fts('my_docs', '<json_query>')
LIMIT 5
```
By default the top 5 matches come back in **arbitrary order** -- select
`_score` and order by it for relevance ranking:
```sql
SELECT id, text, _score FROM fts('my_docs', '<json_query>')
ORDER BY _score DESC
LIMIT 5
```
`_score` is a pseudo-column carrying the query's BM25 relevance for the row;
it behaves like any other column, but exists only in a relation that came
from `fts()`. See [Data types](/sql/data-types#pseudo-columns).
## Fuzzy {#fuzzy}
Tolerates typos, up to a character-edit budget:
```python
MatchQuery("pupy", "text", fuzziness=2)
```
## Phrase {#phrase}
Exact phrase, or with `slop` to allow words between terms:
```python
PhraseQuery("happy puppy", "text")
PhraseQuery("puppy park", "text", slop=2) # up to 2 words between them
```
## Boolean {#boolean}
Combine queries with `&` (AND) or `|` (OR):
```python
MatchQuery("puppy", "text") & MatchQuery("happy", "text")
MatchQuery("puppy", "text") | MatchQuery("kitten", "text")
```
## Boost {#boost}
Weight a query up or down relative to another:
```python
from lancedb.query import BoostQuery
BoostQuery(
positive=MatchQuery("puppy", "text"),
negative=MatchQuery("kitten", "text"),
negative_boost=0.2,
)
```
## Multi-match {#multi-match}
Search several columns at once, optionally weighting them differently:
```python
from lancedb.query import MultiMatchQuery
MultiMatchQuery("puppy", ["text", "category"])
MultiMatchQuery("puppy", ["text", "category"], boosts=[2.0, 1.0]) # text weighted 2x
```
## Combining with SQL {#combining}
`fts()` is a relation, so it composes with the rest of the SQL surface --
filter, join, or aggregate over it same as any table:
```sql
SELECT id, text, category FROM fts('my_docs', '<json_query>')
WHERE category = 'animals'
LIMIT 5
```
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Full-Text Search" icon="magnifying-glass" href="/search/full-text-search">
The query-builder equivalent, and every query form in full.
</Card>
<Card title="FTS Index" icon="bolt" href="/indexing/fts-index">
Tokenizers, and the position information phrase queries need.
</Card>
</CardGroup>
docs/web/sql/join.mdx → docs/web/sql/dql/join.mdx
View File
docs/web/sql/sampling.mdx → docs/web/sql/dql/sampling.mdx
View File
docs/web/sql/select.mdx → docs/web/sql/dql/select.mdx
View File
docs/web/sql/subqueries.mdx → docs/web/sql/dql/subqueries.mdx
+4 -4
View File
@@ -19,7 +19,7 @@ SELECT * FROM (
WHERE n > 100
```
A derived table needs an alias. It can be [sampled](/sql/sampling) like a base
A derived table needs an alias. It can be [sampled](/sql/dql/sampling) like a base
table:
```sql
@@ -60,14 +60,14 @@ correlated, and is conceptually evaluated per outer row. The planner usually
rewrites it into a join.
<Note>
Check with [`EXPLAIN`](/sql/explain) when a correlated subquery is slow. A plan
Check with [`EXPLAIN`](/sql/dql/explain) when a correlated subquery is slow. A plan
showing a join has been optimized; one showing a repeated scan has not, and
rewriting it as an explicit [join](/sql/join) is the fix.
rewriting it as an explicit [join](/sql/dql/join) is the fix.
</Note>
## CTEs as an alternative {#ctes}
A [CTE](/sql/select#ctes) is a named subquery, and is usually more readable when
A [CTE](/sql/dql/select#ctes) is a named subquery, and is usually more readable when
the same subquery appears twice or when nesting gets deep.
```sql
docs/web/sql/window.mdx → docs/web/sql/dql/window.mdx
+2 -2
View File
@@ -10,7 +10,7 @@ keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead"
A window function computes across a set of rows related to the current one and
returns a value **per row**. That is the difference from
[aggregation](/sql/aggregation), which collapses the rows it covers.
[aggregation](/sql/dql/aggregation), which collapses the rows it covers.
```sql
SELECT
@@ -73,7 +73,7 @@ lag lead first_value last_value nth_value
**Aggregates**
Any [aggregate function](/sql/aggregation#functions) used with `OVER` becomes a
Any [aggregate function](/sql/dql/aggregation#functions) used with `OVER` becomes a
window function — that is how running totals and moving averages are written.
## Deduplicating with a window {#dedup}
docs/web/sql/explain.mdx
-51
View File
@@ -1,51 +0,0 @@
---
title: "EXPLAIN"
sidebarTitle: "EXPLAIN"
description: "Reading the plan for a statement, and measuring it."
icon: map
keywords: ["explain", "explain analyze", "query plan", "describe"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
```sql
EXPLAIN SELECT * FROM documents WHERE identifier < 1000
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
```
`EXPLAIN` resolves the plan without running the query. `EXPLAIN ANALYZE`
executes it and annotates every operator with runtime metrics — elapsed time,
rows produced, bytes read, index comparisons.
The operators are the same ones the SDK methods produce, so the reasoning about
index coverage and filter pushdown carries across unchanged. The two pages that
explain how to read them:
<CardGroup cols={2}>
<Card title="Explain" icon="map" href="/search/explain">
Reading a plan: operators, pushdown, index usage.
</Card>
<Card title="Explain Analyze" icon="gauge" href="/search/explain">
Reading the metrics, and a worked before/after.
</Card>
</CardGroup>
## What to look for {#what-to-look-for}
| Sign | Means |
| --- | --- |
| `ScalarIndexQuery` | the predicate used a scalar index |
| `LanceScan` with high `bytes_read` | no index, or too many columns selected |
| `FilterExec` above a search | the filter ran after search rather than pushing into it |
| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used |
A `LanceScan` feeding a join is the common one worth acting on — see
[Joins](/sql/join#performance).
## DESCRIBE {#describe}
```sql
DESCRIBE documents
```
The table's columns and types, without a plan.
docs/web/sql/functions.mdx
+7 -4
View File
@@ -15,8 +15,8 @@ plus what Lance adds.
| | |
| --- | --- |
| [Scalar functions](/sql/scalar-functions) | per row — math, string, regex, date/time, array, struct, JSON |
| [Aggregate functions](/sql/aggregation#functions) | per group — `count`, `sum`, statistical, approximate |
| [Window functions](/sql/window#functions) | per window — ranking, offset, running aggregates |
| [Aggregate functions](/sql/dql/aggregation#functions) | per group — `count`, `sum`, statistical, approximate |
| [Window functions](/sql/dql/window#functions) | per window — ranking, offset, running aggregates |
| [Table functions](#table-functions) | produce a relation rather than a value |
<Note>
@@ -63,6 +63,9 @@ For a keyword test that needs no score, the
[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate is
index-accelerated and usable directly in `WHERE`.
See [Full-Text Search](/sql/dql/fts) for the fuzzy, phrase, boolean, boost and
multi-match query forms.
### `_score` {#score}
A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves
@@ -89,5 +92,5 @@ between a fast predicate and a slow one:
| [`contains_tokens`](/sql/scalar-functions#contains-tokens) | full-text index |
| [`json_get`, typed variants, `json_extract`](/sql/scalar-functions#json) | JSON scalar index |
Confirm with [`EXPLAIN`](/sql/explain): an accelerated predicate shows an index
node rather than a `LanceScan` followed by a filter.
Confirm with [`EXPLAIN`](/sql/dql/explain): an accelerated predicate shows an
index node rather than a `LanceScan` followed by a filter.
docs/web/sql/index.mdx
+96 -35
View File
@@ -1,7 +1,7 @@
---
title: "SQL reference"
sidebarTitle: "Overview"
description: "The SQL surface: dialect, where a query runs, and a page per construct."
description: "The SQL surface: dialect, connecting a client, and a page per construct."
icon: table-list
keywords: ["sql", "datafusion", "flight sql", "reference", "dialect"]
---
@@ -24,47 +24,111 @@ What these pages add is the part that is LanceDB's: how a query reaches the
engine, how Lance columns appear to it, and the statements a lakehouse needs
that DataFusion has no concept of.
## Where a query runs {#execution}
## Connecting {#connecting}
Queries execute on the server and results come back as Arrow record batches --
nothing is converted to pandas or Python objects unless you ask. `execute_query`
hands back a reader as soon as the first batch is ready rather than when the
query finishes, so a large result set does not have to fit in memory before you
start reading it. See [Queries](/basics/query).
SQL is served over an
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint, so any Flight SQL client reaches it and results arrive as Arrow
record batches -- nothing is converted to pandas or Python objects unless you
ask.
Served over an [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint, so any Flight SQL client reaches it.
From Python, the connection runs SQL directly, without a separate client:
```python
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10")
table = reader.read_all()
```
`execute_query` hands back a reader as soon as the first batch is ready rather
than when the query finishes, so a large result set does not have to fit in
memory before you start reading it. See [Queries](/basics/query).
The TypeScript SDK has no SQL method; reach for a Flight SQL client instead --
and in Python, reach for one too when the caller is outside the SDK's own
process, such as a BI tool or a notebook in another language:
```bash
pip install flightsql-dbapi # Python DB API 2 interface
npm install --save @lancedb/flightsql-client
```
<CodeGroup>
```python Python icon="python"
from flightsql import FlightSQLClient
client = FlightSQLClient(
host="your-enterprise-endpoint",
port=10025,
insecure=True,
token="DATABASE_TOKEN",
metadata={"database": "your-project-slug"},
features={"metadata-reflection": "true"},
)
info = client.execute("SELECT * FROM flights WHERE origin = 'SFO'")
ticket = info.endpoints[0].ticket
result = client.do_get(ticket).read_all()
```
```typescript TypeScript icon="square-js"
import { Client } from "@lancedb/flightsql-client";
const client = await Client.connect({
host: "your-enterprise-endpoint:10025",
username: "lancedb",
password: "password",
});
const result = await client.query("SELECT * FROM flights WHERE origin = 'SFO'");
```
</CodeGroup>
## Usage {#usage}
The endpoint executes one statement per request and is meant for the queries
and DDL/DML pages below. Reach for the SDKs instead for table-management
operations that have no SQL form, such as building a vector index or a merge
insert -- see [What SQL cannot do](#cannot).
## Reference {#reference}
**Reading data**
### DDL, by object {#ddl}
| | |
| --- | --- |
| [Namespace](/sql/ddl/namespace) | `CREATE SCHEMA`, `DROP SCHEMA` |
| [Table](/sql/ddl/table) | `CREATE`, `DESCRIBE`, `ALTER`, `CLONE`, `DROP TABLE` |
| [View](/sql/ddl/view) | `CREATE VIEW`, `DROP VIEW` |
| [Materialized view](/sql/ddl/materialized-view) | `SHOW MATERIALIZED VIEWS` |
| [Index](/sql/ddl/index) | `CREATE INDEX`, `DROP INDEX` |
| [Function](/sql/ddl/function) | `CREATE FUNCTION`, `SHOW FUNCTIONS`, `DROP FUNCTION` |
| [Job](/sql/ddl/job) | `SHOW JOBS`, `SHOW ERRORS`, `PAUSE`/`RESUME`/`CANCEL JOB` |
### DML {#dml}
| | |
| --- | --- |
| [Insert](/sql/dml/insert) | `INSERT INTO`, including `INSERT ... SELECT` |
| [Update](/sql/dml/update) | `UPDATE ... SET` |
| [Delete](/sql/dml/delete) | `DELETE FROM` |
| [Copy](/sql/dml/copy) | `COPY INTO` / `COPY TO` |
| [Refresh](/sql/dml/refresh) | filling computed columns and materialized views |
### DQL {#dql}
| | |
| --- | --- |
| [Data types](/sql/data-types) | how Lance columns appear to SQL, and pseudo-columns |
| [SELECT](/sql/select) | projection, filtering, ordering, set operations, CTEs |
| [Subqueries](/sql/subqueries) | scalar, `IN`, `EXISTS`, correlated, derived tables |
| [Joins](/sql/join) | inner, outer, cross, and namespace-qualified paths |
| [Aggregation](/sql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set |
| [Window functions](/sql/window) | `OVER`, framing, ranking and offset functions |
| [Sampling](/sql/sampling) | `TABLESAMPLE`, by rows or by fragments |
| [Select](/sql/dql/select) | projection, filtering, ordering, set operations, CTEs |
| [Subqueries](/sql/dql/subqueries) | scalar, `IN`, `EXISTS`, correlated, derived tables |
| [Joins](/sql/dql/join) | inner, outer, cross, and namespace-qualified paths |
| [Aggregation](/sql/dql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set |
| [Window functions](/sql/dql/window) | `OVER`, framing, ranking and offset functions |
| [Sampling](/sql/dql/sampling) | `TABLESAMPLE`, by rows or by fragments |
| [Full-Text Search](/sql/dql/fts) | the `fts()` table function, and its query forms |
| [Explain](/sql/dql/explain) | `EXPLAIN`, `EXPLAIN ANALYZE`, `EXPLAIN VERBOSE` |
**Changing things**
| | |
| --- | --- |
| [DDL](/sql/ddl) | tables, views, indexes, functions, columns, WAL |
| [DML](/sql/dml) | `INSERT`, `UPDATE`, `DELETE`, `COPY` |
| [REFRESH](/sql/refresh) | filling computed columns and materialized views |
**Introspection**
| | |
| --- | --- |
| [SHOW](/sql/show) | jobs, functions, materialized views, errors |
| [EXPLAIN](/sql/explain) | plans, and measuring them |
**Expressions**
### Expressions {#expressions}
| | |
| --- | --- |
@@ -83,7 +147,4 @@ endpoint, so any Flight SQL client reaches it.
| `PREPARE` / `EXECUTE` | preparation is a [protocol feature](/sql/prepared-statements), not a statement |
| Run on a local database | no engine to plan the statement |
Full-text search **is** available, as the [`fts()`](/sql/functions#fts) table
function and the [`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate.
See [Limits](/basics/limits#sql) for the bounds these pages enforce.
docs/web/sql/operators.mdx
+1 -1
View File
@@ -35,7 +35,7 @@ its wildcard semantics.
`NOT unknown` is still unknown. A `WHERE` clause keeps only rows where the
predicate is true, so unknown behaves like false there — but not inside `NOT
IN`, which is the usual place this bites. See
[Subqueries](/sql/subqueries#in-exists).
[Subqueries](/sql/dql/subqueries#in-exists).
## Arithmetic {#arithmetic}
docs/web/sql/prepared-statements.mdx
+1 -1
View File
@@ -50,7 +50,7 @@ SELECT id, title FROM documents WHERE category = $1 AND score > $2
```
The exact placeholder spelling and the binding call belong to your client; see
[connecting a client](/search/sql).
[connecting a client](/sql#connecting).
## When it helps {#when}
docs/web/sql/scalar-functions.mdx
+2 -2
View File
@@ -12,8 +12,8 @@ Functions that take a value and return a value, one row at a time. The set is
DataFusion 54's default registration plus what Lance adds — the names below are
what is actually registered, not the union of everything DataFusion documents.
For per-group functions see [Aggregation](/sql/aggregation); for per-window,
[Window functions](/sql/window).
For per-group functions see [Aggregation](/sql/dql/aggregation); for per-window,
[Window functions](/sql/dql/window).
## Math {#math}
docs/web/sql/show.mdx
-89
View File
@@ -1,89 +0,0 @@
---
title: "SHOW"
sidebarTitle: "SHOW"
description: "Listing jobs, functions, materialized views, and the errors a job produced."
icon: list
keywords: ["show jobs", "show functions", "show materialized views", "show errors", "show tables"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
Introspection over objects DataFusion has no concept of. All four are LanceDB
extensions.
<Warning>
DataFusion's own `SHOW TABLES`, `SHOW COLUMNS` and `SHOW <variable>` are **not
available**. They are implemented over `information_schema`, which this session
does not enable, so they fail with
`"... is not supported unless information_schema is enabled"`.
List tables with the SDKs -- `table_names()` / `tableNames()`. For a table's
columns, use [`DESCRIBE`](/sql/explain#describe).
</Warning>
## SHOW MATERIALIZED VIEWS {#materialized-views}
```sql
SHOW MATERIALIZED VIEWS
```
<Note>
Cheaper than the SDK equivalent. `list_materialized_views` finds views by
reading every table's schema, costing one open per table; this asks the server.
</Note>
## SHOW FUNCTIONS {#functions}
```sql
SHOW FUNCTIONS
```
Lists registered [functions](/management/function) and their versions. Since a
version is immutable and a column pins the version that fills it, this is how
you find which versions are still in use before dropping one.
## SHOW JOBS {#jobs}
```sql
SHOW JOBS
```
Every job the server knows about — id, type, state, creation time. The SQL
counterpart of `list_jobs`; see [Jobs](/basics/jobs).
## SHOW ERRORS {#errors}
```sql
SHOW ERRORS ON <table> [FOR JOB '<job-id>'] [LIMIT <n>]
```
```sql
SHOW ERRORS ON documents
SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100
```
Why rows failed. This is the statement to reach for when a
[refresh](/sql/refresh) reports fewer filled rows than you expected.
**Errors and limits**
| | |
| --- | --- |
| `LIMIT` | must be between 1 and 100,000 |
| Failed jobs scanned without `FOR JOB` | 32 most recent |
| Tables the server does not open directly | not supported |
Without `FOR JOB` this surveys recent failures and stops at 32 jobs. When the
listing stops at your `LIMIT`, the message says so — narrow with `FOR JOB`
rather than raising the limit blindly.
## See also {#see-also}
<CardGroup cols={2}>
<Card title="Management" icon="folder-tree" href="/management">
The SDK listing calls, per object.
</Card>
<Card title="Query and job" icon="hourglass-half" href="/basics/jobs">
What a job handle exposes beyond these listings.
</Card>
</CardGroup>