From 52684d50b988ff81d0cfdb46d767d2301822a5cd Mon Sep 17 00:00:00 2001 From: Jack Ye Date: Mon, 28 Sep 2026 17:17:09 -0500 Subject: [PATCH] docs: flip ownership of View/SQL-client/FTS-SQL/AuthZ/AuthN to OSS, restructure SQL Reference OSS docs now own the full usage content for features that only run against an Enterprise deployment, since the client interface itself lives here; Enterprise docs keep only the parts specific to its own behavior. Sophon's overlay for these five pages is dropped since the OSS versions are now the complete reference. SQL Reference is restructured to match an object-type SQL manual (DDL by object, then DML, then DQL, then Expressions) instead of the old flat, loosely-ordered page list, absorbing the SQL client-connection guide and FTS-via-SQL examples into it. --- docs/web/basics/apis.mdx | 2 +- docs/web/basics/limits.mdx | 2 +- docs/web/curation/aggregate.mdx | 2 +- docs/web/curation/join.mdx | 4 +- docs/web/curation/sampling.mdx | 2 +- docs/web/curation/windowing.mdx | 2 +- docs/web/docs.json | 113 +++++++++-- docs/web/enterprise/authentication.mdx | 152 +++++++++++++- .../feature-engineering/data-evolution.mdx | 2 +- docs/web/feature-engineering/index.mdx | 2 +- docs/web/governance/authorization.mdx | 107 +++++++++- docs/web/ingestion/delete.mdx | 2 +- docs/web/ingestion/index.mdx | 2 +- docs/web/ingestion/insert.mdx | 2 +- docs/web/ingestion/merge-insert.mdx | 2 +- docs/web/ingestion/update.mdx | 4 +- docs/web/maintenance/background-jobs.mdx | 6 +- docs/web/management/database.mdx | 2 +- docs/web/management/function.mdx | 2 +- docs/web/management/materialized-view.mdx | 2 +- docs/web/management/namespace.mdx | 2 +- docs/web/management/table.mdx | 4 +- docs/web/management/view.mdx | 6 +- docs/web/search/explain.mdx | 4 +- docs/web/search/filtering.mdx | 2 +- docs/web/search/index.mdx | 2 +- docs/web/search/sql/fts-sql.mdx | 32 --- docs/web/search/sql/index.mdx | 29 --- docs/web/sql/ddl.mdx | 192 ------------------ docs/web/sql/ddl/function.mdx | 86 ++++++++ docs/web/sql/ddl/index.mdx | 49 +++++ docs/web/sql/ddl/job.mdx | 90 ++++++++ docs/web/sql/ddl/materialized-view.mdx | 62 ++++++ docs/web/sql/ddl/namespace.mdx | 53 +++++ docs/web/sql/ddl/table.mdx | 178 ++++++++++++++++ docs/web/sql/ddl/view.mdx | 98 +++++++++ docs/web/sql/dml.mdx | 81 -------- docs/web/sql/dml/copy.mdx | 23 +++ docs/web/sql/dml/delete.mdx | 27 +++ docs/web/sql/dml/insert.mdx | 33 +++ docs/web/sql/{ => dml}/refresh.mdx | 8 +- docs/web/sql/dml/update.mdx | 33 +++ docs/web/sql/{ => dql}/aggregation.mdx | 4 +- docs/web/sql/dql/explain.mdx | 52 +++++ docs/web/sql/dql/fts.mdx | 136 +++++++++++++ docs/web/sql/{ => dql}/join.mdx | 0 docs/web/sql/{ => dql}/sampling.mdx | 0 docs/web/sql/{ => dql}/select.mdx | 0 docs/web/sql/{ => dql}/subqueries.mdx | 8 +- docs/web/sql/{ => dql}/window.mdx | 4 +- docs/web/sql/explain.mdx | 51 ----- docs/web/sql/functions.mdx | 11 +- docs/web/sql/index.mdx | 131 ++++++++---- docs/web/sql/operators.mdx | 2 +- docs/web/sql/prepared-statements.mdx | 2 +- docs/web/sql/scalar-functions.mdx | 4 +- docs/web/sql/show.mdx | 89 -------- 57 files changed, 1412 insertions(+), 590 deletions(-) delete mode 100644 docs/web/search/sql/fts-sql.mdx delete mode 100644 docs/web/search/sql/index.mdx delete mode 100644 docs/web/sql/ddl.mdx create mode 100644 docs/web/sql/ddl/function.mdx create mode 100644 docs/web/sql/ddl/index.mdx create mode 100644 docs/web/sql/ddl/job.mdx create mode 100644 docs/web/sql/ddl/materialized-view.mdx create mode 100644 docs/web/sql/ddl/namespace.mdx create mode 100644 docs/web/sql/ddl/table.mdx create mode 100644 docs/web/sql/ddl/view.mdx delete mode 100644 docs/web/sql/dml.mdx create mode 100644 docs/web/sql/dml/copy.mdx create mode 100644 docs/web/sql/dml/delete.mdx create mode 100644 docs/web/sql/dml/insert.mdx rename docs/web/sql/{ => dml}/refresh.mdx (91%) create mode 100644 docs/web/sql/dml/update.mdx rename docs/web/sql/{ => dql}/aggregation.mdx (97%) create mode 100644 docs/web/sql/dql/explain.mdx create mode 100644 docs/web/sql/dql/fts.mdx rename docs/web/sql/{ => dql}/join.mdx (100%) rename docs/web/sql/{ => dql}/sampling.mdx (100%) rename docs/web/sql/{ => dql}/select.mdx (100%) rename docs/web/sql/{ => dql}/subqueries.mdx (86%) rename docs/web/sql/{ => dql}/window.mdx (94%) delete mode 100644 docs/web/sql/explain.mdx delete mode 100644 docs/web/sql/show.mdx diff --git a/docs/web/basics/apis.mdx b/docs/web/basics/apis.mdx index 9eca49d41..099566f90 100644 --- a/docs/web/basics/apis.mdx +++ b/docs/web/basics/apis.mdx @@ -61,7 +61,7 @@ its result. Full-text search *is* available in SQL, as the The TypeScript SDK does not expose SQL, and has no methods for [functions](/management/function). Reach the endpoint with a Flight -SQL client instead -- see [connecting a client](/search/sql). This is a gap in +SQL client instead -- see [connecting a client](/sql#connecting). This is a gap in the SDK rather than in the endpoint: the same statements work. ## Running SQL from Python {#sql-from-python} diff --git a/docs/web/basics/limits.mdx b/docs/web/basics/limits.mdx index d615fade5..34c84f9ee 100644 --- a/docs/web/basics/limits.mdx +++ b/docs/web/basics/limits.mdx @@ -77,7 +77,7 @@ See [Data types](/sql/data-types). A sampling percentage over 100 is an error rather than a full scan, because a query asking for 150 percent has a mistake in it and reading it as 100 would -hide the mistake. See [Sampling](/sql/sampling). +hide the mistake. See [Sampling](/sql/dql/sampling). ## What is unavailable where {#availability} diff --git a/docs/web/curation/aggregate.mdx b/docs/web/curation/aggregate.mdx index 7fab2c662..fde064a99 100644 --- a/docs/web/curation/aggregate.mdx +++ b/docs/web/curation/aggregate.mdx @@ -110,4 +110,4 @@ use this to check the shape of the head, not the existence of the tail. ## Reference {#reference} The complete clause and function list is in -[Aggregation](/sql/aggregation). +[Aggregation](/sql/dql/aggregation). diff --git a/docs/web/curation/join.mdx b/docs/web/curation/join.mdx index f8d55d93c..a05b93609 100644 --- a/docs/web/curation/join.mdx +++ b/docs/web/curation/join.mdx @@ -73,7 +73,7 @@ Two things dominate join cost on a lakehouse table. **Index the join key.** A [scalar index](/indexing/scalar-index) on the key lets the planner look rows up instead of scanning for them. Check with -[`EXPLAIN`](/sql/explain): an indexed key shows a `ScalarIndexQuery`, an +[`EXPLAIN`](/sql/dql/explain): an indexed key shows a `ScalarIndexQuery`, an unindexed one shows a `LanceScan` feeding the join. **Do not carry wide columns through.** Select the columns you need before @@ -88,4 +88,4 @@ JOIN labels AS l ON l.image_id = i.id ## Reference {#reference} -The join types and their semantics are in [Joins](/sql/join). +The join types and their semantics are in [Joins](/sql/dql/join). diff --git a/docs/web/curation/sampling.mdx b/docs/web/curation/sampling.mdx index a2c1289b3..4e8b000cd 100644 --- a/docs/web/curation/sampling.mdx +++ b/docs/web/curation/sampling.mdx @@ -95,4 +95,4 @@ rows. See [Training](/training/data-loading). ## Reference {#reference} The full clause, its options and what it refuses are in -[Sampling](/sql/sampling). +[Sampling](/sql/dql/sampling). diff --git a/docs/web/curation/windowing.mdx b/docs/web/curation/windowing.mdx index 2386ece3e..b8ad4aeca 100644 --- a/docs/web/curation/windowing.mdx +++ b/docs/web/curation/windowing.mdx @@ -121,4 +121,4 @@ selective, and check the plan with ## Reference {#reference} The full window syntax, framing, and function list are in -[Window functions](/sql/window). +[Window functions](/sql/dql/window). diff --git a/docs/web/docs.json b/docs/web/docs.json index 8859494e8..2406c8022 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -162,6 +162,58 @@ { "source": "/enterprise/deployment", "destination": "/enterprise/deployment/index" + }, + { + "source": "/search/sql/index", + "destination": "/sql" + }, + { + "source": "/search/sql/fts-sql", + "destination": "/sql/dql/fts" + }, + { + "source": "/sql/select", + "destination": "/sql/dql/select" + }, + { + "source": "/sql/subqueries", + "destination": "/sql/dql/subqueries" + }, + { + "source": "/sql/join", + "destination": "/sql/dql/join" + }, + { + "source": "/sql/aggregation", + "destination": "/sql/dql/aggregation" + }, + { + "source": "/sql/window", + "destination": "/sql/dql/window" + }, + { + "source": "/sql/sampling", + "destination": "/sql/dql/sampling" + }, + { + "source": "/sql/explain", + "destination": "/sql/dql/explain" + }, + { + "source": "/sql/ddl", + "destination": "/sql/ddl/table" + }, + { + "source": "/sql/dml", + "destination": "/sql/dml/insert" + }, + { + "source": "/sql/refresh", + "destination": "/sql/dml/refresh" + }, + { + "source": "/sql/show", + "destination": "/sql/ddl/job" } ], "navigation": { @@ -293,24 +345,51 @@ "group": "SQL Reference", "pages": [ "sql/index", - "search/sql/index", "sql/data-types", - "sql/select", - "search/sql/fts-sql", - "sql/subqueries", - "sql/join", - "sql/aggregation", - "sql/window", - "sql/sampling", - "sql/ddl", - "sql/dml", - "sql/refresh", - "sql/show", - "sql/explain", - "sql/prepared-statements", - "sql/operators", - "sql/functions", - "sql/scalar-functions" + { + "group": "DDL", + "pages": [ + "sql/ddl/namespace", + "sql/ddl/table", + "sql/ddl/view", + "sql/ddl/materialized-view", + "sql/ddl/index", + "sql/ddl/function", + "sql/ddl/job" + ] + }, + { + "group": "DML", + "pages": [ + "sql/dml/insert", + "sql/dml/update", + "sql/dml/delete", + "sql/dml/copy", + "sql/dml/refresh" + ] + }, + { + "group": "DQL", + "pages": [ + "sql/dql/select", + "sql/dql/subqueries", + "sql/dql/join", + "sql/dql/aggregation", + "sql/dql/window", + "sql/dql/sampling", + "sql/dql/fts", + "sql/dql/explain" + ] + }, + { + "group": "Expressions", + "pages": [ + "sql/operators", + "sql/functions", + "sql/scalar-functions", + "sql/prepared-statements" + ] + } ] }, { diff --git a/docs/web/enterprise/authentication.mdx b/docs/web/enterprise/authentication.mdx index e07d9b3bd..f41603fdf 100644 --- a/docs/web/enterprise/authentication.mdx +++ b/docs/web/enterprise/authentication.mdx @@ -1,9 +1,9 @@ --- title: "Authentication" sidebarTitle: "Authentication" -description: "Establishing who is calling: API keys and identity provider integration." +description: "Authentication modes for LanceDB Enterprise: API keys and OAuth credentials." icon: key -keywords: ["authentication", "api key", "oidc", "identity"] +keywords: ["authentication", "oauth", "api key", "azure managed identity", "client credentials"] --- Enterprise @@ -13,14 +13,158 @@ of LanceDB has no server to provide it -- see [Architecture](/architecture) for what differs between the two. A deployment authenticates every request before -[authorization](/governance/authorization) decides what it may do. +[authorization](/governance/authorization) decides what it may do, against a +`db://` remote table, in one of two ways: + +- **API keys** -- a long-lived shared secret passed as `api_key` on connect. Works in every SDK. +- **OAuth 2.0** -- short-lived bearer tokens obtained from your identity provider, refreshed automatically by the client. + +OAuth is the recommended option when you want to rotate credentials centrally, +plug into an existing identity provider, or run on Azure with managed +identities so no secret material lives on the client. + +## API key {#api-key} + +Pass the API key from your Enterprise tenant on `connect`. This works with +both the synchronous and asynchronous Python clients, as well as TypeScript +and Rust. ```python -db = lancedb.connect("db://my-database", api_key="...", region="us-east-1") +import lancedb + +db = lancedb.connect( + uri="db://your-database-uri", + api_key="your-api-key", + region="us-east-1", + host_override="https://your-enterprise-endpoint.com", +) ``` An API key belongs in the environment rather than in source. See [Database](/management/database#deployment) for the connection options. +## OAuth {#oauth} + +The async Python client and the TypeScript client can obtain bearer tokens +from an OIDC issuer and attach them to every request. Token acquisition, +caching, and refresh are handled inside the client -- your application code +only provides the configuration. + + +In Python, OAuth is supported through `lancedb.connect_async`. The synchronous +`connect` entry point continues to use API key authentication for `db://` +URIs. In TypeScript, `lancedb.connect` accepts `oauthConfig` directly. + + +### Supported flows {#supported-flows} + +`OAuthFlowType` selects how the client acquires tokens: + +| Flow | Python value | TypeScript value | When to use | +| --- | --- | --- | --- | +| Client Credentials | `OAuthFlowType.CLIENT_CREDENTIALS` | `OAuthFlowType.ClientCredentials` | Service-to-service / machine-to-machine. Requires a client ID and client secret registered with your identity provider. | +| Azure Managed Identity | `OAuthFlowType.AZURE_MANAGED_IDENTITY` | `OAuthFlowType.AzureManagedIdentity` | Workloads running on Azure compute (VMs, AKS, App Service, Container Apps). Tokens are fetched from the Azure IMDS endpoint, so no client secret is stored on the client. | + +### Configure OAuth {#configure-oauth} + +Build an OAuth config and pass it on connect. Use `oauth_config` in Python and +`oauthConfig` in TypeScript. + + +```python Python icon="python" +import lancedb +from lancedb.remote import OAuthConfig, OAuthFlowType + +# Client Credentials (service-to-service) +oauth_config = OAuthConfig( + issuer_url="https://login.microsoftonline.com/{tenant}/v2.0", + client_id="your-application-client-id", + client_secret="your-application-client-secret", + scopes=["api://lancedb-api/.default"], + flow=OAuthFlowType.CLIENT_CREDENTIALS, +) + +db = await lancedb.connect_async( + uri="db://your-database-uri", + region="us-east-1", + host_override="https://your-enterprise-endpoint.com", + oauth_config=oauth_config, +) +``` + +```typescript TypeScript icon="square-js" +import * as lancedb from "@lancedb/lancedb"; +import { OAuthConfig, OAuthFlowType } from "@lancedb/lancedb"; + +// Client Credentials (service-to-service) +const oauthConfig: OAuthConfig = { + issuerUrl: "https://login.microsoftonline.com/{tenant}/v2.0", + clientId: "your-application-client-id", + clientSecret: "your-application-client-secret", + scopes: ["api://lancedb-api/.default"], + flow: OAuthFlowType.ClientCredentials, +}; + +const db = await lancedb.connect("db://your-database-uri", { + region: "us-east-1", + hostOverride: "https://your-enterprise-endpoint.com", + oauthConfig, +}); +``` + + +For workloads running on Azure compute, use a managed identity instead of a +client secret: + + +```python Python icon="python" +from lancedb.remote import OAuthConfig, OAuthFlowType + +oauth_config = OAuthConfig( + issuer_url="https://login.microsoftonline.com/{tenant}/v2.0", + client_id="your-application-client-id", + scopes=["api://lancedb-api/.default"], + flow=OAuthFlowType.AZURE_MANAGED_IDENTITY, + # Optional: required only for user-assigned managed identities. + managed_identity_client_id="your-user-assigned-identity-client-id", +) +``` + +```typescript TypeScript icon="square-js" +import { OAuthConfig, OAuthFlowType } from "@lancedb/lancedb"; + +const oauthConfig: OAuthConfig = { + issuerUrl: "https://login.microsoftonline.com/{tenant}/v2.0", + clientId: "your-application-client-id", + scopes: ["api://lancedb-api/.default"], + flow: OAuthFlowType.AzureManagedIdentity, + // Optional: required only for user-assigned managed identities. + managedIdentityClientId: "your-user-assigned-identity-client-id", +}; +``` + + +### Configuration reference {#configuration-reference} + +The same configuration is available in both SDKs. Python uses `snake_case` +field names; TypeScript uses `camelCase`. + +| Python field / TypeScript field | Required | Description | +| --- | --- | --- | +| `issuer_url` / `issuerUrl` | Yes | OIDC issuer URL or OAuth authority URL. For Azure, use `https://login.microsoftonline.com/{tenant_id}/v2.0`. | +| `client_id` / `clientId` | Yes | Application / client ID registered with your identity provider. | +| `scopes` | Yes | List of OAuth scopes to request. For Azure managed identity, provide exactly one scope or resource, for example `["api://your-app-id/.default"]`. | +| `flow` | No | Selects the OAuth flow. Defaults to Client Credentials. | +| `client_secret` / `clientSecret` | Conditional | Required for Client Credentials. Redacted from the config's `repr` (Python) and `Debug` output (TypeScript native binding) so it does not leak into logs. | +| `managed_identity_client_id` / `managedIdentityClientId` | No | Client ID of a user-assigned managed identity. Only used with Azure Managed Identity; omit for system-assigned identities. | +| `refresh_buffer_secs` / `refreshBufferSecs` | No | How many seconds before token expiry to proactively refresh. Defaults to 300. Keep this well below the token TTL -- setting it greater than or equal to the TTL forces a refresh on every request. | + + +Treat the client secret like any other production credential. Load it from a +secret manager or environment variable rather than committing it to source +control. Both SDKs deliberately omit the secret from their debug/repr output +so accidental log lines do not expose the value. + + Embedded LanceDB has nothing to authenticate to -- your process opens the files directly, and the object store's own credentials are the only gate. diff --git a/docs/web/feature-engineering/data-evolution.mdx b/docs/web/feature-engineering/data-evolution.mdx index 6f5656870..b835dbf61 100644 --- a/docs/web/feature-engineering/data-evolution.mdx +++ b/docs/web/feature-engineering/data-evolution.mdx @@ -27,7 +27,7 @@ column -- the existing data, which is the expensive part on a multimodal table, is never rewritten. See [why that works](/basics/lance#evolution). -In SQL this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl#computed-column). +In SQL this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl/table#computed-column). ## Set up the example table {#set-up-the-example-table} diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx index 27f33e2f3..253f9734f 100644 --- a/docs/web/feature-engineering/index.mdx +++ b/docs/web/feature-engineering/index.mdx @@ -17,7 +17,7 @@ versioned, and refreshed like anything else, not a separate system bolted alongside. -Every operation here has a SQL form -- see [DDL](/sql/ddl#columns) to declare a column and [REFRESH](/sql/refresh) to fill it. +Every operation here has a SQL form -- see [DDL](/sql/ddl/table#alter) to declare a column and [REFRESH](/sql/dml/refresh) to fill it. ## The shape everything shares {#the-shape} diff --git a/docs/web/governance/authorization.mdx b/docs/web/governance/authorization.mdx index c19914908..044db158e 100644 --- a/docs/web/governance/authorization.mdx +++ b/docs/web/governance/authorization.mdx @@ -8,19 +8,108 @@ keywords: ["authorization", "authz", "privileges", "rbac", "access control"] Enterprise -This page describes a LanceDB Enterprise capability. The embedded form -of LanceDB has no server to provide it -- see +This page describes a LanceDB Enterprise capability. Embedded LanceDB has no +such layer -- access is whatever the object store enforces on the bucket or +directory, which is coarser and has no concept of a table. See [Architecture](/architecture) for what differs between the two. [Authentication](/enterprise/authentication) establishes who is calling. Authorization decides what they may do. An access control entry grants **one privilege** on **one object** to **one -subject**. Objects are addressed by kind and path -- `system`, `database:db`, -`namespace:db/ns`, `table:db/ns/table` -- and the hierarchy is not decorative: -reaching a table requires `USAGE` on the namespace containing it and on the -database containing that. +subject**. Everything below is a consequence of that shape. -Embedded LanceDB has no such layer. Access is whatever the object store -enforces on the bucket or directory, which is coarser and has no concept of a -table. +## Subjects {#subjects} + +A grant is held by one of four kinds of subject, distinguished by a prefix: + +| Prefix | Subject | +| --- | --- | +| `p:` | a principal, by ID -- `P:` by name | +| `g:` | a group, by ID -- `G:` by name | +| `r:` | a role | +| `a:` | an API key | + +Granting to a group or role rather than a principal is what keeps the entry +count manageable: a principal's effective privileges are the union of what it +holds directly and what it inherits. + +Granting to an API key is narrower than granting to the principal that owns +it -- useful when a key is issued for one job and should not carry that +principal's full authority. + +## Objects {#objects} + +Objects are addressed by kind and path: + +``` +system +database:db +namespace:db/ns +table:db/ns/table +``` + +The hierarchy is not decorative. Reaching a table requires `USAGE` on the +namespace containing it and on the database containing that -- a grant on the +table alone is not enough to traverse to it. + +## Privileges {#privileges} + +| Privilege | Granted on | Allows | +| --- | --- | --- | +| `OWNERSHIP` | any object | ownership, conveying full authority | +| `USAGE` | database | accessing the namespaces inside | +| `USAGE` | namespace | accessing the tables inside | +| `SELECT` | table | reading rows | +| `INSERT` | table | appending rows | +| `UPDATE` | table | modifying existing rows | +| `DELETE` | table | deleting rows | +| `CREATE_DATABASE` | system | creating a database | +| `CREATE_NAMESPACE` | database, namespace | creating immediate child namespaces | +| `CREATE_TABLE` | namespace | creating a table in it | +| `CREATE_MATERIALIZED_VIEW` | namespace | creating a materialized view in it | +| `CREATE_VIEW` | namespace | creating a non-materialized view in it | +| `OPERATE` | system | reading server configuration | + +The four data privileges map onto the four +[ingestion operations](/ingestion), so a principal that may append but not +modify is expressed by granting `INSERT` without `UPDATE`. + + +`CREATE_MATERIALIZED_VIEW` and `CREATE_VIEW` are separate privileges because +[the two objects differ](/basics/catalog): one stores rows and is refreshed, +the other stores only a query. Granting the ability to create one does not +grant the other. + + +## Creation privileges resolve upward {#upward} + +A subtlety worth knowing, because it explains grants that appear to work on +objects that do not exist yet. + +When `USAGE` is checked on a database and **no entries exist for it at all**, +the database is assumed not to exist yet, and `CREATE_DATABASE` on `system` is +checked instead. The same applies to the default namespace: absent any +entries, the check falls back to `CREATE_NAMESPACE` on the containing +database. + +So a principal holding `CREATE_DATABASE` can act on a database that has not +been created -- which is what makes creation possible at all, and why an empty +ACL set is not the same as a deny. + +## Listing grants {#listing} + +Entries can be listed filtered by object, by subject, or both: + +``` +object = table:db/ns/documents +subject = G:data-science +``` + +Results paginate through a continuation token. One stored `(object, subject)` +tuple carrying several privileges expands into several entries in the +response, so the count reflects privileges rather than tuples. + +Listing by subject answers "what can this team reach"; listing by object +answers "who can reach this table". Both are worth running before changing a +grant. diff --git a/docs/web/ingestion/delete.mdx b/docs/web/ingestion/delete.mdx index 43928b8f7..0e2a4fdfa 100644 --- a/docs/web/ingestion/delete.mdx +++ b/docs/web/ingestion/delete.mdx @@ -18,7 +18,7 @@ import { } from '/snippets/tables.mdx'; -In SQL this is [`DELETE`](/sql/dml#delete). +In SQL this is [`DELETE`](/sql/dml/delete). ## Delete rows {#delete-rows} diff --git a/docs/web/ingestion/index.mdx b/docs/web/ingestion/index.mdx index 1a8a94955..088d1f7ac 100644 --- a/docs/web/ingestion/index.mdx +++ b/docs/web/ingestion/index.mdx @@ -30,7 +30,7 @@ writing to a table that exists. This page covers the first and introduces the second. -Every operation here has a SQL form -- see [DML](/sql/dml). +Every operation here has a SQL form -- see [DML](/sql#dml). Start by [creating a table](/ingestion/create-table); the rest of this diff --git a/docs/web/ingestion/insert.mdx b/docs/web/ingestion/insert.mdx index b8f7fb6e5..a8559c88d 100644 --- a/docs/web/ingestion/insert.mdx +++ b/docs/web/ingestion/insert.mdx @@ -20,7 +20,7 @@ import { } from '/snippets/tables.mdx'; -In SQL this is [`INSERT`](/sql/dml#insert), including `INSERT ... SELECT` to move rows server-side. +In SQL this is [`INSERT`](/sql/dml/insert), including `INSERT ... SELECT` to move rows server-side. ## Ingesting large datasets {#large-datasets} diff --git a/docs/web/ingestion/merge-insert.mdx b/docs/web/ingestion/merge-insert.mdx index 9e0cc52d9..8bfb6084a 100644 --- a/docs/web/ingestion/merge-insert.mdx +++ b/docs/web/ingestion/merge-insert.mdx @@ -27,7 +27,7 @@ import { } from '/snippets/tables.mdx'; -Merge insert has **no SQL form** -- it needs a source relation and a per-match policy that one statement cannot express. See [DML](/sql/dml#merge-insert). +Merge insert has **no SQL form** -- it needs a source relation and a per-match policy that one statement cannot express. See [DML](/sql/dml/insert#next). ## Merge incoming rows by key {#merge-incoming-rows-by-key} diff --git a/docs/web/ingestion/update.mdx b/docs/web/ingestion/update.mdx index c739062f6..e77587ea8 100644 --- a/docs/web/ingestion/update.mdx +++ b/docs/web/ingestion/update.mdx @@ -18,7 +18,7 @@ import { } from '/snippets/tables.mdx'; -In SQL this is [`UPDATE`](/sql/dml#update). +In SQL this is [`UPDATE`](/sql/dml/update). ## Update rows {#update-rows} @@ -76,7 +76,7 @@ Expected table contents: | 2 | Bob | 21 | -See the [SQL queries](/search/sql/) page for more information on the supported SQL syntax. +See the [SQL queries](/sql#connecting) page for more information on the supported SQL syntax. When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows, consider triggering an index rebuild afterwards. diff --git a/docs/web/maintenance/background-jobs.mdx b/docs/web/maintenance/background-jobs.mdx index fc1c83767..ba88b7137 100644 --- a/docs/web/maintenance/background-jobs.mdx +++ b/docs/web/maintenance/background-jobs.mdx @@ -13,14 +13,14 @@ Every job documented elsewhere in these docs -- [Build & Repair](/indexing/reind [Column Refresh](/feature-engineering/refresh-column) -- is one you cause, directly or by a schedule you set. This page is the rest: jobs the deployment runs to keep itself correct and fast, which you may see named in -[`SHOW JOBS`](/sql/ddl#job) without having asked for any of them. +[`SHOW JOBS`](/sql/ddl/job#show) without having asked for any of them. There is no action to take on these beyond recognizing the name. None of them has a parameter you set or a call you make. ## Write-ahead log maintenance {#wal} -For a table with a [write-ahead log](/sql/ddl#table) enabled, two jobs keep the +For a table with a [write-ahead log](/sql/ddl/table#properties) enabled, two jobs keep the log from growing without bound: | Job type | What it does | @@ -61,5 +61,5 @@ table. | `drop_object` | Deletes a dropped catalog object's content, then clears it from the database's cleanup backlog. Covers namespaces, views, and functions -- everything [`drop_table`](/management/table#drop) does not, since that one fences a directory with its own marker instead. | You trigger this indirectly, by dropping a [namespace](/management/namespace), -[view](/sql/ddl#view), or [function](/management/function#drop) -- the teardown +[view](/sql/ddl/view#drop), or [function](/management/function#drop) -- the teardown itself runs as this job rather than inline with the call. diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx index 317a2aaec..1c3e1c084 100644 --- a/docs/web/management/database.mdx +++ b/docs/web/management/database.mdx @@ -77,7 +77,7 @@ const db = await lancedb.connect("db://my-database", { | `api_key` / `apiKey` | authenticates the connection -- see [Authentication](/enterprise/authentication) | | `region` | which region hosts the database. Defaults to `us-east-1` | | `host_override` / `hostOverride` | point at a specific endpoint instead of the regional one | -| `sql_host_override` | the same, for the [SQL endpoint](/search/sql). Python only | +| `sql_host_override` | the same, for the [SQL endpoint](/sql#connecting). Python only | | `client_config` / `clientConfig` | retries, timeouts, and connection pooling | An API key belongs in the environment, not in source. `region` is easy to get diff --git a/docs/web/management/function.mdx b/docs/web/management/function.mdx index 49d4b8b34..a244573d9 100644 --- a/docs/web/management/function.mdx +++ b/docs/web/management/function.mdx @@ -18,7 +18,7 @@ raises `NotImplementedError`; use a [computed column](/feature-engineering/refresh-column) instead. -In SQL, a function is registered with [`CREATE FUNCTION`](/sql/ddl#functions) and listed with [`SHOW FUNCTIONS`](/sql/show#functions). +In SQL, a function is registered with [`CREATE FUNCTION`](/sql/ddl/function#create) and listed with [`SHOW FUNCTIONS`](/sql/ddl/function#show). ## Define a function {#define} diff --git a/docs/web/management/materialized-view.mdx b/docs/web/management/materialized-view.mdx index f9b1e3ec0..14347d8ec 100644 --- a/docs/web/management/materialized-view.mdx +++ b/docs/web/management/materialized-view.mdx @@ -17,7 +17,7 @@ This page covers creating and managing one. Keeping it current is [Refresh a materialized view](/feature-engineering/materialized-views). -In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view) and [`SHOW MATERIALIZED VIEWS`](/sql/show#materialized-views). +In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/dml/refresh#refresh-materialized-view) and [`SHOW MATERIALIZED VIEWS`](/sql/ddl/materialized-view#show). ## Creating one {#create} diff --git a/docs/web/management/namespace.mdx b/docs/web/management/namespace.mdx index 0e7917be4..f1310b6da 100644 --- a/docs/web/management/namespace.mdx +++ b/docs/web/management/namespace.mdx @@ -29,7 +29,7 @@ language: Python uses `namespace_path`, Rust uses builder methods like `.namespa TypeScript uses `namespacePath` arguments. -In SQL a namespaced table is addressed as a [dotted path](/sql/select#namespaces). +In SQL a namespaced table is addressed as a [dotted path](/sql/dql/select#namespaces). ## Namespace hierarchy {#namespace-hierarchy} diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index ca4d6c767..be1da63b0 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -121,7 +121,7 @@ This page covers the table as a catalog object. Getting data into one is [Schema evolution](/management/table#schema). -Tables are created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl#tables). +Tables are created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl/table). ## What a table carries {#carries} @@ -500,7 +500,7 @@ LanceDB supports ACID-compliant schema evolution through granular operations (ad * Optimize Continuously: Remove unused fields or enforce new constraints without downtime -In SQL, schema changes are [`ALTER TABLE`](/sql/ddl#columns). +In SQL, schema changes are [`ALTER TABLE`](/sql/ddl/table#alter). ### Schema evolution operations {#schema-schema-evolution-operations} diff --git a/docs/web/management/view.mdx b/docs/web/management/view.mdx index 9db9e8a08..bfb3a0543 100644 --- a/docs/web/management/view.mdx +++ b/docs/web/management/view.mdx @@ -27,6 +27,6 @@ defined by a query; the difference is entirely when that query runs. | Read cost | the full query, every read | a table scan | | Write cost | none | a refresh | -Views are created in SQL, over the [Flight SQL endpoint](/search/sql) -- there -is no SDK method, because the thing a view stores *is* a SQL statement. See -[DDL](/sql/ddl#views). +Views are created in SQL, over the [Flight SQL endpoint](/sql#connecting) -- +there is no SDK method, because the thing a view stores *is* a SQL statement. +See [View](/sql/ddl/view) for `CREATE VIEW`, `DROP VIEW`, and listing. diff --git a/docs/web/search/explain.mdx b/docs/web/search/explain.mdx index f99b976e7..b2fc12c04 100644 --- a/docs/web/search/explain.mdx +++ b/docs/web/search/explain.mdx @@ -15,7 +15,7 @@ To measure a query rather than read it, use operator with runtime metrics. -The SQL equivalent is [`EXPLAIN`](/sql/explain). +The SQL equivalent is [`EXPLAIN`](/sql/dql/explain). ## What it shows {#inspect-a-query-plan} @@ -208,7 +208,7 @@ Where [`explain_plan`](#inspect-a-query-plan) tells you what a query intends to this tells you what it cost. -The SQL equivalent is [`EXPLAIN ANALYZE`](/sql/explain). +The SQL equivalent is [`EXPLAIN ANALYZE`](/sql/dql/explain). ## What it measures {#analyze-a-query-plan} diff --git a/docs/web/search/filtering.mdx b/docs/web/search/filtering.mdx index 0456ff9b9..d5bce618b 100644 --- a/docs/web/search/filtering.mdx +++ b/docs/web/search/filtering.mdx @@ -15,7 +15,7 @@ with filtering capabilities even on datasets containing billions of records. On hybrid queries, the same `where(...)` filter is applied to both the vector and full-text halves of the query. The prefilter or postfilter choice controls whether that happens before each subquery scores candidates or after the subquery top-k is produced. -The same predicates work in SQL -- see [SELECT](/sql/select#filtering) and [Operators](/sql/operators). +The same predicates work in SQL -- see [SELECT](/sql/dql/select#filtering) and [Operators](/sql/operators). ## Chaining `where` clauses {#chaining-where-clauses} diff --git a/docs/web/search/index.mdx b/docs/web/search/index.mdx index 4197acc78..886751331 100644 --- a/docs/web/search/index.mdx +++ b/docs/web/search/index.mdx @@ -12,7 +12,7 @@ icon: "list" | [Full-Text Search](/search/full-text-search/) | Keyword-based search with BM25 and pre-filtering | | [Hybrid Search](/search/hybrid-search/) | Combines vector and full-text search with reranking | | [Filtering](/search/filtering/) | Filter results based on metadata fields | -| [SQL Queries](/search/sql/index) | SQL query capabilities for data exploration and analytics | +| [SQL Queries](/sql) | SQL query capabilities for data exploration and analytics | ## Before you search {#before-you-search} diff --git a/docs/web/search/sql/fts-sql.mdx b/docs/web/search/sql/fts-sql.mdx deleted file mode 100644 index 836f724d8..000000000 --- a/docs/web/search/sql/fts-sql.mdx +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: "Full-text search with SQL" -sidebarTitle: "Full-Text Search" -description: "Running full-text search from SQL, with fuzzy, phrase, boolean and boost queries." -icon: magnifying-glass -keywords: ["fts", "sql", "full text search", "fuzzy", "phrase"] ---- - -Enterprise - -This page describes a LanceDB Enterprise capability. The embedded form -of LanceDB has no server to provide it -- see -[Architecture](/architecture) for what differs between the two. - -Full-text search is available in SQL as the [`fts()`](/sql/functions#fts) table -function, which returns matching rows plus a `_score` column: - -```sql -SELECT id, text, _score -FROM fts('my_docs', 'puppy') -ORDER BY _score DESC -LIMIT 5 -``` - -The second argument is a bare query string, or a JSON query object for the -fuzzy, phrase, boolean, boost and multi-match forms. - -For a keyword test that needs no score, -[`contains_tokens`](/sql/scalar-functions#contains-tokens) is a predicate usable -directly in `WHERE`, and is accelerated by the full-text index. - -The query-builder equivalent is [Full-Text Search](/search/full-text-search). diff --git a/docs/web/search/sql/index.mdx b/docs/web/search/sql/index.mdx deleted file mode 100644 index 06b46b717..000000000 --- a/docs/web/search/sql/index.mdx +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: "Connecting a SQL client" -sidebarTitle: "Client Connection" -description: "Reaching the Arrow Flight SQL endpoint from Python or TypeScript." -icon: plug -keywords: ["flight sql", "client", "connect", "arrow"] ---- - -Enterprise - -This page describes a LanceDB Enterprise capability. The embedded form -of LanceDB has no server to provide it -- see -[Architecture](/architecture) for what differs between the two. - -SQL is served over an -[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) -endpoint, so any Flight SQL client reaches it and results arrive as Arrow -record batches. - -From Python, the connection runs SQL directly without a separate client: - -```python -reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10") -table = reader.read_all() -``` - -The TypeScript SDK has no SQL method; use a Flight SQL client. See -[SDKs and SQL](/basics/apis) for the full comparison, and the -[SQL reference](/sql) for the dialect. diff --git a/docs/web/sql/ddl.mdx b/docs/web/sql/ddl.mdx deleted file mode 100644 index 3705065c4..000000000 --- a/docs/web/sql/ddl.mdx +++ /dev/null @@ -1,192 +0,0 @@ ---- -title: "DDL" -sidebarTitle: "DDL" -description: "Creating and altering catalog objects: tables, views, indexes, functions, and columns." -icon: pen-ruler -keywords: ["ddl", "create table", "alter table", "add column", "create function", "create index", "wal"] ---- - -Enterprise [SQL runs on a deployment.](/sql) - -Statements that change what exists rather than what it holds. For the values in -a table, see [DML](/sql/dml). - -## Tables {#tables} - -```sql -CREATE TABLE documents (id BIGINT, title VARCHAR, created_at TIMESTAMP) -CREATE TABLE recent AS SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days' -DROP TABLE documents -``` - -Column types are Arrow types under SQL names -- see -[Data types](/sql/data-types). A table in a [namespace](/management/namespace) is written -as a dotted path: `prod.events.clicks`. - -## Views {#views} - -```sql -CREATE VIEW recent_matches AS -SELECT t1.floats FROM documents AS t1 -INNER JOIN categories AS t2 ON t1.ints = t2.ints - -DROP VIEW recent_matches -``` - -`CREATE VIEW` returns the view's schema -- column name, type, nullability -- -resolved from the query at creation time. A view stores only the query; see -[Views](/management/view). - -## Columns {#columns} - -`ALTER TABLE ... ADD COLUMN` covers three different things, distinguished by -what follows the column name. - -### A plain column {#plain-column} - -```sql -ALTER TABLE t ADD COLUMN c INT -``` - -Added empty and nullable. Nothing is computed. - -### A computed column {#computed-column} - -```sql -ALTER TABLE t ADD COLUMN doubled AS value * 2 -``` - -`AS ` declares a [computed column](/feature-engineering/add-computed-column). -The column is committed empty and the expression is stored, not evaluated -- -declaring one on a billion-row table costs what declaring one on an empty table -costs. [`REFRESH COLUMN`](/sql/refresh) fills it. - -### A function column {#function-column} - -```sql -ALTER TABLE t ADD COLUMN embedded AS embed(name) WITH (version = 'fv_1') -``` - -When the expression calls a [registered function](#functions), the column is -filled by running that function on workers. `version` pins which immutable -function version fills it, so a later registration does not silently change what -the column means. - -Several modifiers narrow what gets computed: - -| | | -| --- | --- | -| `FILTER (WHERE pred)` | only rows matching `pred` are computed | -| `IGNORE NULLS` | rows with null inputs are skipped | -| `OVER (PARTITION BY col)` | the function sees rows grouped by `col` | - -A function returning several values fills several columns at once: - -```sql -ALTER TABLE t ADD COLUMN (label, score) AS classify(text) WITH (version = 'fv_1') -``` - -```sql -ALTER TABLE t ADD COLUMN c AS f(x) FILTER (WHERE keep) WITH (version = 'fv_1') -ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2') -``` - - -`WITH` on a plain computed column accepts only `version`. Resource options such -as `num_workers` apply to function columns, where there are workers to size. - - -## Functions {#functions} - -```sql -CREATE FUNCTION f(x BIGINT) RETURNS BIGINT -LANGUAGE python -AS 'def f(x): return x * 2' -WITH (pip = 'numpy==1.26.4', python_version = '3.11') -``` - -The body is Python source. Its dependencies are declared with it and are part of -the **immutable function version** -- changing one produces a new version rather -than altering the registered one, so a column filled by an earlier version is -unaffected. - -| Option | | -| --- | --- | -| `pip` | pip requirements for the worker environment | -| `conda` | conda packages instead of pip | -| `conda_channels` | channels in priority order; requires `conda` | -| `python_version` | worker Python version, defaulting to yours | -| `gpu` | `'true'` when every execution needs a GPU | - -**Errors and limits** - -| | | -| --- | --- | -| `LANGUAGE` other than `python` | refused | -| `pip` and `conda` together | refused -- pick one | -| `conda_channels` without `conda` | refused | -| `gpu` set to anything but `'true'` | refused | -| An empty body | refused | -| Any other option | refused, and the message lists what is supported | - -See [Functions](/management/function) for the SDK equivalent and for diagnosing -environment mismatches. - -## Indexes {#indexes} - -```sql -CREATE INDEX ON documents (title) -DROP INDEX documents.title_idx -``` - - -SQL `CREATE INDEX` covers **one column** and takes **no user-chosen name** -- -the server names it. For a multi-column or named index, use the SDK. `DROP -INDEX` takes the qualified `table.index` name; find it with `list_indices`. - - -Index type and parameters are chosen in the SDK; see [Indexing](/indexing). - -## Table properties {#properties} - -```sql -ALTER TABLE t SET UNENFORCED PRIMARY KEY (id) -``` - -Records which column identifies a row. **Unenforced** is not a hedge in the -naming: uniqueness is not checked on write, and declaring it does not make -duplicates impossible. It tells the planner what to assume. - -```sql -ALTER TABLE t SET WAL -ALTER TABLE t SET WAL WITH (sharding = 'bucket', buckets = 4) -ALTER TABLE t SET WAL WITH (maintained_indexes = '') -ALTER TABLE t UNSET WAL -``` - -A write-ahead log makes small frequent writes cheap by batching them out of the -commit path. `sharding` and `buckets` spread the log so concurrent writers do -not contend on one place; `maintained_indexes` chooses which indexes are kept -current against WAL rows. - - -[`TABLESAMPLE`](/sql/sampling) is refused on a WAL-backed table, because rows a -sample would draw from are not all in the fragments yet. - - -## See also {#see-also} - - - - Changing rows rather than structure. - - - Filling the columns declared here. - - - The same operations from the SDKs. - - - Listing what exists. - - diff --git a/docs/web/sql/ddl/function.mdx b/docs/web/sql/ddl/function.mdx new file mode 100644 index 000000000..39a661fc3 --- /dev/null +++ b/docs/web/sql/ddl/function.mdx @@ -0,0 +1,86 @@ +--- +title: "Function" +sidebarTitle: "Function" +description: "CREATE FUNCTION, SHOW FUNCTIONS, and DROP FUNCTION." +icon: function +keywords: ["create function", "drop function", "show functions", "sql", "udf"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +## CREATE FUNCTION {#create} + +```sql +CREATE FUNCTION f(x BIGINT) RETURNS BIGINT +LANGUAGE python +AS 'def f(x): return x * 2' +WITH (pip = 'numpy==1.26.4', python_version = '3.11') +``` + +The body is Python source. Its dependencies are declared with it and are part +of the **immutable function version** -- changing one produces a new version +rather than altering the registered one, so a column filled by an earlier +version is unaffected. + +| Option | | +| --- | --- | +| `pip` | pip requirements for the worker environment | +| `conda` | conda packages instead of pip | +| `conda_channels` | channels in priority order; requires `conda` | +| `python_version` | worker Python version, defaulting to yours | +| `gpu` | `'true'` when every execution needs a GPU | + +**Errors and limits** + +| | | +| --- | --- | +| `LANGUAGE` other than `python` | refused | +| `pip` and `conda` together | refused -- pick one | +| `conda_channels` without `conda` | refused | +| `gpu` set to anything but `'true'` | refused | +| An empty body | refused | +| Any other option | refused, and the message lists what is supported | + + +**Enterprise job type:** `create_function`. Registration returns a job because +the function is baked into an image before the version becomes usable. + + +## SHOW FUNCTIONS {#show} + +```sql +SHOW FUNCTIONS +``` + +Lists registered functions and their versions. Since a version is immutable +and a column pins the version that fills it, this is how you find which +versions are still in use before dropping one. + +## DROP FUNCTION {#drop} + +```sql +DROP FUNCTION embed WITH (version = '1') +DROP FUNCTION IF EXISTS embed WITH (version = '1') +DROP FUNCTION embed WITH (version = '1', cascade = 'true') +``` + +`version` is required -- a function is a set of immutable versions, and +dropping is always of one specific version, never "the function" as a whole. + + +Without `cascade = 'true'`, dropping a version that a column still pins fails +rather than breaking that column silently. Passing it drops the version and +lets columns pinned to it stop refreshing -- they keep their last-filled +values, per [Guarantees](/basics/guarantees#jobs). + + +## What to read next {#next} + + + + Registering one from the SDK, and diagnosing environment mismatches. + + + Filling a column with a function instead of an expression. + + diff --git a/docs/web/sql/ddl/index.mdx b/docs/web/sql/ddl/index.mdx new file mode 100644 index 000000000..f499f7681 --- /dev/null +++ b/docs/web/sql/ddl/index.mdx @@ -0,0 +1,49 @@ +--- +title: "Index" +sidebarTitle: "Index" +description: "CREATE INDEX and DROP INDEX — one column, server-chosen name." +icon: bolt +keywords: ["create index", "drop index", "sql"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +## CREATE INDEX {#create} + +```sql +CREATE INDEX ON documents (title) +``` + + +SQL `CREATE INDEX` covers **one column** and takes **no user-chosen name** -- +the server names it. For a multi-column or explicitly-named index, or to +choose the algorithm, use the SDK's +[`create_index`](/indexing#choosing-an-index). + + + +**Enterprise job type:** `create_ivf_pq`, `create_btree`, `create_fts`, and one +more per algorithm -- the type names which algorithm is building, not a +different feature. See [Build & Repair](/indexing/reindexing). + + +## DROP INDEX {#drop} + +```sql +DROP INDEX documents.title_idx +``` + +Takes the qualified `table.index` name. Find it with the SDK's +[`list_indices`](/indexing#choosing-an-index) if you created the index without +naming it yourself. + +## What to read next {#next} + + + + Choosing an index type from the SDK. + + + Keeping an index current as data arrives. + + diff --git a/docs/web/sql/ddl/job.mdx b/docs/web/sql/ddl/job.mdx new file mode 100644 index 000000000..ae99b014e --- /dev/null +++ b/docs/web/sql/ddl/job.mdx @@ -0,0 +1,90 @@ +--- +title: "Job" +sidebarTitle: "Job" +description: "SHOW JOBS, SHOW ERRORS, and controlling a running job with PAUSE, RESUME and CANCEL." +icon: hourglass-half +keywords: ["show jobs", "show errors", "pause job", "resume job", "cancel job", "sql"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +A [job](/basics/jobs) is not a catalog object -- it has no schema and is not +namespaced -- but it has its own SQL statements, grouped here the same way the +catalog objects are. + +## SHOW JOBS {#show} + +```sql +SHOW JOBS +``` + +Every job the server knows about -- id, type, state, creation time. The SQL +counterpart of `list_jobs`. + +## SHOW ERRORS {#errors} + +```sql +SHOW ERRORS ON [FOR JOB ''] [LIMIT ] +``` + +```sql +SHOW ERRORS ON documents +SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100 +``` + +Why rows failed. Reach for this when a [refresh](/sql/dml/refresh) reports +fewer filled rows than you expected. + +**Errors and limits** + +| | | +| --- | --- | +| `LIMIT` | must be between 1 and 100,000 | +| Failed jobs scanned without `FOR JOB` | 32 most recent | +| Tables the server does not open directly | not supported | + +Without `FOR JOB` this surveys recent failures and stops at 32 jobs. When the +listing stops at your `LIMIT`, the message says so -- narrow with `FOR JOB` +rather than raising the limit blindly. + +## PAUSE JOB {#pause} + +```sql +PAUSE JOB 'j-ci-abc-def' +``` + +## RESUME JOB {#resume} + +```sql +RESUME JOB 'j-ci-abc-def' +``` + +## CANCEL JOB {#cancel} + +```sql +CANCEL JOB 'j-ci-abc-def' +``` + +The job id is always a quoted string literal. All three take exactly one and +nothing after it -- a second token on the line is a syntax error rather than +being ignored. + + +Cancellation is a request, matching the SDK's +[`cancel()`](/basics/jobs#managing): it does not guarantee the job stops before +doing more work, and asking twice or cancelling a finished job does nothing. +`CANCEL JOB` itself can be unavailable on a deployment with no job registry +configured, which is a deployment condition rather than something you can work +around from SQL. + + +## What to read next {#next} + + + + The handle these statements are the SQL view of. + + + Job types you did not start yourself. + + diff --git a/docs/web/sql/ddl/materialized-view.mdx b/docs/web/sql/ddl/materialized-view.mdx new file mode 100644 index 000000000..e5920f0d5 --- /dev/null +++ b/docs/web/sql/ddl/materialized-view.mdx @@ -0,0 +1,62 @@ +--- +title: "Materialized View" +sidebarTitle: "Materialized View" +description: "SHOW MATERIALIZED VIEWS — creation and refresh are SDK and REFRESH statements, not CREATE." +icon: table +keywords: ["materialized view", "show materialized views", "sql"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +A materialized view stores a query's result as rows, kept current by a refresh +-- the counterpart to a [view](/sql/ddl/view), which stores only the query. + + +There is no `CREATE MATERIALIZED VIEW` statement. Creation is +[`create_materialized_view`](/management/materialized-view#create) in the SDK, +since it needs SDK-level source-table validation (stable row ids) that a bare +SQL statement has nowhere to check. Once created, it is entirely a SQL object: +queried like a table, refreshed with SQL, listed with SQL. + + +## SHOW MATERIALIZED VIEWS {#show} + +```sql +SHOW MATERIALIZED VIEWS +``` + + +Cheaper than the SDK equivalent. `list_materialized_views` finds views by +reading every table's schema, costing one open per table; this asks the server. + + +## Refreshing one {#refresh} + +```sql +REFRESH MATERIALIZED VIEW chunks +``` + +See [REFRESH](/sql/dml/refresh#refresh-materialized-view) for the full syntax, +including `full = true` and what `CASCADE` does not yet do. + +## Dropping one {#drop} + +A materialized view **is** a table, so it is dropped with +[`DROP TABLE`](/sql/ddl/table#drop) -- which is also why dropping one removes +stored rows, unlike dropping a [view](/sql/ddl/view#drop). + + +**Enterprise job type:** `mat_view_refresh` for the refresh; `drop_table` for +dropping it, same as any table. + + +## What to read next {#next} + + + + Creating one from the SDK, and the stable-row-ids requirement. + + + Incremental refreshes, and what they cost. + + diff --git a/docs/web/sql/ddl/namespace.mdx b/docs/web/sql/ddl/namespace.mdx new file mode 100644 index 000000000..daedf9720 --- /dev/null +++ b/docs/web/sql/ddl/namespace.mdx @@ -0,0 +1,53 @@ +--- +title: "Namespace" +sidebarTitle: "Namespace" +description: "CREATE SCHEMA and DROP SCHEMA — the SQL spelling of namespace DDL." +icon: folder-tree +keywords: ["namespace", "schema", "create schema", "drop schema", "sql"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +A [namespace](/management/namespace) is addressed in SQL as a `SCHEMA` -- the +standard's name for the same concept. + +## CREATE SCHEMA {#create} + +```sql +CREATE SCHEMA public +CREATE SCHEMA db1.ns1.ns2 +CREATE SCHEMA IF NOT EXISTS `ns1$ns2` +``` + +A dotted path creates the whole chain: `db1.ns1.ns2` creates `ns2` inside +`ns1` inside `db1`, provided the parents already exist. A name containing the +namespace path delimiter itself is backtick-quoted, as in `` `ns1$ns2` `` above. + +`IF NOT EXISTS` makes the statement a no-op rather than an error when the +namespace is already there. + +## DROP SCHEMA {#drop} + +```sql +DROP SCHEMA db1.public +DROP SCHEMA IF EXISTS `a$b` +DROP SCHEMA `a$b` RESTRICT +``` + + +`CASCADE` is rejected: **"DROP SCHEMA CASCADE is not supported; drop the +objects inside first."** A namespace's tables, views and functions must be +dropped explicitly before the namespace itself can go. `RESTRICT` -- the +default -- states that requirement rather than changing it. + + +## What to read next {#next} + + + + The SDK equivalent, and how a namespace fits the catalog hierarchy. + + + Creating tables inside a namespace. + + diff --git a/docs/web/sql/ddl/table.mdx b/docs/web/sql/ddl/table.mdx new file mode 100644 index 000000000..bd64458be --- /dev/null +++ b/docs/web/sql/ddl/table.mdx @@ -0,0 +1,178 @@ +--- +title: "Table" +sidebarTitle: "Table" +description: "CREATE, DESCRIBE, ALTER, CLONE and DROP TABLE." +icon: table +keywords: ["create table", "alter table", "drop table", "describe", "clone", "ddl"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +## CREATE TABLE {#create} + +```sql +CREATE TABLE documents (id BIGINT, title VARCHAR, created_at TIMESTAMP) +CREATE TABLE recent AS SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days' +``` + +Column types are Arrow types under SQL names -- see [Data types](/sql/data-types). +A table inside a [namespace](/sql/ddl/namespace) is written as a dotted path: +`prod.events.clicks`. + +## DESCRIBE {#describe} + +```sql +DESCRIBE documents +``` + +The table's columns and types, without a plan -- the SQL counterpart of +`table.schema`. + +## ALTER TABLE {#alter} + +`ALTER TABLE ... ADD COLUMN` covers three different things, distinguished by +what follows the column name. + +### A plain column {#plain-column} + +```sql +ALTER TABLE t ADD COLUMN c INT +``` + +Added empty and nullable. Nothing is computed. + +### A computed column {#computed-column} + +```sql +ALTER TABLE t ADD COLUMN doubled AS value * 2 +``` + +`AS ` declares a [computed column](/feature-engineering/add-computed-column). +The column is committed empty and the expression is stored, not evaluated -- +declaring one on a billion-row table costs what declaring one on an empty table +costs. [`REFRESH COLUMN`](/sql/dml/refresh) fills it. + +### A function column {#function-column} + +```sql +ALTER TABLE t ADD COLUMN embedded AS embed(name) WITH (version = 'fv_1') +``` + +When the expression calls a [registered function](/sql/ddl/function), the +column is filled by running that function on workers. `version` pins which +immutable function version fills it, so a later registration does not silently +change what the column means. + +Several modifiers narrow what gets computed: + +| | | +| --- | --- | +| `FILTER (WHERE pred)` | only rows matching `pred` are computed | +| `IGNORE NULLS` | rows with null inputs are skipped | +| `OVER (PARTITION BY col)` | the function sees rows grouped by `col` | + +A function returning several values fills several columns at once: + +```sql +ALTER TABLE t ADD COLUMN (label, score) AS classify(text) WITH (version = 'fv_1') +``` + +```sql +ALTER TABLE t ADD COLUMN c AS f(x) FILTER (WHERE keep) WITH (version = 'fv_1') +ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2') +``` + + +`WITH` on a plain computed column accepts only `version`. Resource options such +as `num_workers` apply to function columns, where there are workers to size. + + +### Table properties {#properties} + +```sql +ALTER TABLE t SET UNENFORCED PRIMARY KEY (id) +``` + +Records which column identifies a row. **Unenforced** is not a hedge in the +naming: uniqueness is not checked on write, and declaring it does not make +duplicates impossible. It tells the planner what to assume. + +```sql +ALTER TABLE t SET WAL +ALTER TABLE t SET WAL WITH (sharding = 'bucket', buckets = 4) +ALTER TABLE t SET WAL WITH (maintained_indexes = '') +ALTER TABLE t UNSET WAL +``` + +A write-ahead log makes small frequent writes cheap by batching them out of the +commit path. `sharding` and `buckets` spread the log so concurrent writers do +not contend on one place; `maintained_indexes` chooses which indexes are kept +current against WAL rows. See +[Background jobs](/maintenance/background-jobs#wal) for how the log gets +folded back into the table. + + +[`TABLESAMPLE`](/sql/dql/sampling) is refused on a WAL-backed table, because +rows a sample would draw from are not all in the fragments yet. + + +**Errors and limits** + +| | | +| --- | --- | +| `ALTER TABLE ... DROP COLUMN` | not supported | + +Dropping, renaming, or retyping a column is an SDK operation -- +[`drop_columns`](/management/table#schema) and +[`alter_columns`](/management/table#schema) -- not a SQL statement. + +## CLONE {#clone} + +```sql +CREATE TABLE docs_snapshot CLONE documents +CREATE TABLE docs_at_v42 CLONE documents AT 42 +CREATE TABLE docs_at_release CLONE documents AT TAG 'release-1' +CREATE TABLE docs_yesterday CLONE documents AT TIMESTAMP '2026-09-01T00:00:00Z' +``` + +A **shallow clone**: the new table shares the source's data files rather than +copying them, so it costs one metadata commit regardless of the source's size. + +`AT` pins the point in the source's history to clone. A bare integer is a +version and a bare quoted string is a timestamp; `VERSION`, `TAG` and +`TIMESTAMP` say so explicitly, and a tag always needs its keyword -- a bare +quoted string cannot be both a tag name and a timestamp. + + +The clone depends on the source's files continuing to exist. If the source is +[compacted](/maintenance/cleanup) past the retention window, the version the +clone points at can be pruned. The SDK's +[`clone_table`](/management/table#rename-clone) also offers a deep clone for +when the copy has to outlive the source's history. + + +## DROP TABLE {#drop} + +```sql +DROP TABLE documents +``` + + +Removes the table and every version of it. There is no undrop. + + + +**Enterprise job type:** `drop_table`. See +[Background jobs](/maintenance/background-jobs#teardown). + + +## What to read next {#next} + + + + The SDK equivalent, and what a table carries. + + + Renaming, retyping and dropping columns from the SDK. + + diff --git a/docs/web/sql/ddl/view.mdx b/docs/web/sql/ddl/view.mdx new file mode 100644 index 000000000..fc0807058 --- /dev/null +++ b/docs/web/sql/ddl/view.mdx @@ -0,0 +1,98 @@ +--- +title: "View" +sidebarTitle: "View" +description: "CREATE VIEW and DROP VIEW — a saved query, evaluated when read." +icon: eye +keywords: ["create view", "drop view", "sql", "catalog"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +A view stores a **query**, not rows. Reading it runs the query against current +data, so a view is never stale -- and never faster than the query it wraps. It +is the counterpart to a [materialized view](/sql/ddl/materialized-view), which +stores rows and is brought up to date by a refresh; both are +[catalog objects](/basics/catalog) defined by a query, differing only in when +that query runs. + +There is no SDK method for a view, because the thing it stores *is* a SQL +statement. + +## CREATE VIEW {#create} + +```sql +CREATE VIEW "recent_matches" AS +SELECT t1.floats +FROM "documents" AS t1 +INNER JOIN "categories" AS t2 ON t1.ints = t2.ints +``` + +The statement returns the view's schema -- column name, data type, nullability +-- resolved from the query at creation time. Anything the +[dialect](/sql/functions) supports can define one: joins, predicates, +aggregates, window functions. + +## Reading one {#reading} + +A view is queried like a table: + +```sql +SELECT * FROM "recent_matches" WHERE floats > 0.5 +``` + +Predicates applied to the view compose with the query inside it rather than +filtering its output afterwards, so a selective filter does not pay for rows +the view would have produced and then discarded. + +## DROP VIEW {#drop} + +```sql +DROP VIEW "recent_matches" +``` + +Removes the saved query. Unlike dropping a table, no data is removed -- there +was none to begin with. + +## Listing {#listing} + +Views and tables share a namespace, and both are things you can query. The +catalog distinguishes them: + +| | | +| --- | --- | +| view names | the views in the database | +| table names | the tables | +| relation names | everything queryable -- both | + +Use the relation listing when the question is "what can I select from," and the +separate listings when the distinction matters -- before dropping, for +instance, since dropping a view removes only the saved query while dropping a +table removes data. + +## Choosing between a view and a materialized view {#choosing} + +| | View | Materialized view | +| --- | --- | --- | +| Stores | the query | the rows | +| Freshness | always current | as of the last refresh | +| Read cost | the full query, every read | a table scan | +| Write cost | none | a refresh | + +Reach for a **view** when the query is cheap, or when reading stale rows would +be wrong. Reach for a **materialized view** when the query is expensive and the +data changes less often than it is read. + +The privileges are separate too: `CREATE_VIEW` and `CREATE_MATERIALIZED_VIEW` +are distinct grants on a namespace, so the ability to create one does not imply +the other. See [Authorization](/governance/authorization). + +## What to read next {#next} + + + + The stored-rows counterpart. + + + Where views fit among the other catalog objects. + + diff --git a/docs/web/sql/dml.mdx b/docs/web/sql/dml.mdx deleted file mode 100644 index dadb52589..000000000 --- a/docs/web/sql/dml.mdx +++ /dev/null @@ -1,81 +0,0 @@ ---- -title: "DML" -sidebarTitle: "DML" -description: "INSERT, UPDATE, DELETE and COPY — changing the rows a table holds." -icon: pen-to-square -keywords: ["insert", "update", "delete", "copy", "dml"] ---- - -Enterprise [SQL runs on a deployment.](/sql) - -Statements that change rows. For the shape of the table, see [DDL](/sql/ddl). - -Each of these commits a new table version, so -[versioning](/management/table#versioning) and the -[write guarantees](/basics/guarantees#writes) apply -- including that the -version advances even when a statement affects zero rows. - -## INSERT {#insert} - -```sql -INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur') - -INSERT INTO archive SELECT * FROM documents WHERE created_at < '2026-01-01' -``` - -`INSERT ... SELECT` is the form worth knowing: it moves rows between tables -server-side, without a round trip through your process. - -## UPDATE {#update} - -```sql -UPDATE documents SET title = upper(title) WHERE category = 'film' -``` - -Assignments take any expression the [dialect](/sql/functions) supports, -including references to other columns of the same row. Without a `WHERE` clause -every row is updated. - - -An update does not recompute [computed columns](/feature-engineering/add-computed-column) -that derive from the column you changed. A filled value keeps what it was -computed from -- see [Guarantees](/basics/guarantees#jobs). - - -## DELETE {#delete} - -```sql -DELETE FROM documents WHERE created_at < '2025-01-01' -``` - -Deletion marks rows rather than erasing them, so the data remains recoverable -for as long as the version holding it survives -[retention](/maintenance/compaction). - -## COPY {#copy} - -```sql -COPY (SELECT id, title FROM documents) TO 's3://bucket/export/' STORED AS PARQUET -``` - -Writes a query result to object storage. This is the export path when the -consumer is another engine rather than an application -- results land as files -rather than streaming to a client. - -## Merge insert {#merge-insert} - -Upsert -- matching incoming rows against existing keys -- has no SQL form. It is -an SDK operation, because it needs a source relation and a per-match policy that -a single statement cannot express. See -[Merge insert](/ingestion/merge-insert). - -## See also {#see-also} - - - - The same operations from the SDKs. - - - Changing structure rather than rows. - - diff --git a/docs/web/sql/dml/copy.mdx b/docs/web/sql/dml/copy.mdx new file mode 100644 index 000000000..837b92285 --- /dev/null +++ b/docs/web/sql/dml/copy.mdx @@ -0,0 +1,23 @@ +--- +title: "Copy" +sidebarTitle: "Copy" +description: "COPY ... TO — writing a query result to object storage as files." +icon: file-export +keywords: ["copy", "export", "parquet", "sql"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +```sql +COPY (SELECT id, title FROM documents) TO 's3://bucket/export/' STORED AS PARQUET +``` + +Writes a query result to object storage. This is the export path when the +consumer is another engine rather than an application -- results land as files +rather than streaming to a client. + +## What to read next {#next} + + + Bringing rows in, rather than exporting them. + diff --git a/docs/web/sql/dml/delete.mdx b/docs/web/sql/dml/delete.mdx new file mode 100644 index 000000000..da71dd9eb --- /dev/null +++ b/docs/web/sql/dml/delete.mdx @@ -0,0 +1,27 @@ +--- +title: "Delete" +sidebarTitle: "Delete" +description: "DELETE FROM ... WHERE — rows are marked, not erased." +icon: trash +keywords: ["delete", "sql"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +```sql +DELETE FROM documents WHERE created_at < '2025-01-01' +``` + +Deletion marks rows rather than erasing them, so the data remains recoverable +for as long as the version holding it survives +[retention](/maintenance/cleanup). + +Commits a new table version, so [versioning](/management/table#versioning) and +the [write guarantees](/basics/guarantees#writes) apply -- including that the +version advances even when the statement affects zero rows. + +## What to read next {#next} + + + The SDK equivalent. + diff --git a/docs/web/sql/dml/insert.mdx b/docs/web/sql/dml/insert.mdx new file mode 100644 index 000000000..2be017f06 --- /dev/null +++ b/docs/web/sql/dml/insert.mdx @@ -0,0 +1,33 @@ +--- +title: "Insert" +sidebarTitle: "Insert" +description: "INSERT INTO, including INSERT ... SELECT to move rows server-side." +icon: plus +keywords: ["insert", "insert into", "sql"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +```sql +INSERT INTO documents (id, title) VALUES (1, 'Merlin'), (2, 'Arthur') + +INSERT INTO archive SELECT * FROM documents WHERE created_at < '2026-01-01' +``` + +`INSERT ... SELECT` is the form worth knowing: it moves rows between tables +server-side, without a round trip through your process. + +Commits a new table version, so [versioning](/management/table#versioning) and +the [write guarantees](/basics/guarantees#writes) apply -- including that the +version advances even when the statement affects zero rows. + +## What to read next {#next} + + + + The SDK equivalent. + + + Upsert -- matching incoming rows against existing keys -- has no SQL form. + + diff --git a/docs/web/sql/refresh.mdx b/docs/web/sql/dml/refresh.mdx similarity index 91% rename from docs/web/sql/refresh.mdx rename to docs/web/sql/dml/refresh.mdx index 1cff471da..5d0101097 100644 --- a/docs/web/sql/refresh.mdx +++ b/docs/web/sql/dml/refresh.mdx @@ -10,7 +10,7 @@ keywords: ["refresh column", "refresh materialized view", "backfill", "cascade"] LanceDB extensions with no DataFusion equivalent. They are what makes [feature engineering](/feature-engineering) reachable from SQL: a column -declared by [DDL](/sql/ddl#computed-column) starts empty, and a refresh fills it. +declared by [DDL](/sql/ddl/table#computed-column) starts empty, and a refresh fills it. ## REFRESH COLUMN {#refresh-column} @@ -80,8 +80,8 @@ refreshed by refreshing each in dependency order yourself. ## Watching a refresh {#watching} Both statements submit work that outlives the call. Track it with -[`SHOW JOBS`](/sql/show#jobs), and diagnose failures with -[`SHOW ERRORS`](/sql/show#errors). +[`SHOW JOBS`](/sql/ddl/job#show), and diagnose failures with +[`SHOW ERRORS`](/sql/ddl/job#errors). ## See also {#see-also} @@ -89,7 +89,7 @@ Both statements submit work that outlives the call. Track it with The same operation from the SDKs, and what a refresh guarantees. - + Declaring the columns this fills. diff --git a/docs/web/sql/dml/update.mdx b/docs/web/sql/dml/update.mdx new file mode 100644 index 000000000..cef35ef4e --- /dev/null +++ b/docs/web/sql/dml/update.mdx @@ -0,0 +1,33 @@ +--- +title: "Update" +sidebarTitle: "Update" +description: "UPDATE ... SET ... WHERE — and why it does not recompute derived columns." +icon: pen +keywords: ["update", "sql"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +```sql +UPDATE documents SET title = upper(title) WHERE category = 'film' +``` + +Assignments take any expression the [dialect](/sql/functions) supports, +including references to other columns of the same row. Without a `WHERE` +clause every row is updated. + + +An update does not recompute [computed columns](/feature-engineering/add-computed-column) +that derive from the column you changed. A filled value keeps what it was +computed from -- see [Guarantees](/basics/guarantees#jobs). + + +Commits a new table version, so [versioning](/management/table#versioning) and +the [write guarantees](/basics/guarantees#writes) apply -- including that the +version advances even when the statement affects zero rows. + +## What to read next {#next} + + + The SDK equivalent. + diff --git a/docs/web/sql/aggregation.mdx b/docs/web/sql/dql/aggregation.mdx similarity index 97% rename from docs/web/sql/aggregation.mdx rename to docs/web/sql/dql/aggregation.mdx index 64c8d8920..2bd8ddd58 100644 --- a/docs/web/sql/aggregation.mdx +++ b/docs/web/sql/dql/aggregation.mdx @@ -10,7 +10,7 @@ keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximat Aggregation is one of the two main reasons to reach for SQL rather than a query builder -- the SDKs project, filter and search, but they do not group. The other -is [joins](/sql/join). +is [joins](/sql/dql/join). ## Grouping {#grouping} @@ -99,7 +99,7 @@ FROM documents ## Aggregating a sample {#sampling} For an estimate over a very large table, aggregate a -[sample](/sql/sampling) rather than the whole thing: +[sample](/sql/dql/sampling) rather than the whole thing: ```sql SELECT category, count(*) * 100 AS estimated diff --git a/docs/web/sql/dql/explain.mdx b/docs/web/sql/dql/explain.mdx new file mode 100644 index 000000000..3fc58861a --- /dev/null +++ b/docs/web/sql/dql/explain.mdx @@ -0,0 +1,52 @@ +--- +title: "Explain" +sidebarTitle: "Explain" +description: "EXPLAIN, EXPLAIN ANALYZE and EXPLAIN VERBOSE — reading and measuring a plan from SQL." +icon: map +keywords: ["explain", "explain analyze", "explain verbose", "query plan"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +```sql +EXPLAIN SELECT * FROM documents WHERE identifier < 1000 +EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000 +``` + +| Statement | What it returns | +| --- | --- | +| `EXPLAIN ` | Logical and physical plan, without executing the query. | +| `EXPLAIN ANALYZE ` | Executes the query and annotates each operator with runtime metrics (rows, timing). | +| `EXPLAIN VERBOSE ` | Adds intermediate optimizer plans on top of `EXPLAIN`. | + +The result is a small relation with `plan_type` and `plan` columns, read +through the same client you use for regular queries -- see +[Connecting](/sql#connecting). + +The operators are the same ones the SDK's +[`explain_plan` and `analyze_plan`](/search/explain) methods produce, so the +reasoning about index coverage and filter pushdown carries across unchanged: + + + + Reading a plan: operators, pushdown, index usage. + + + Reading the metrics, and a worked before/after. + + + +## What to look for {#what-to-look-for} + +| Sign | Means | +| --- | --- | +| `ScalarIndexQuery` | the predicate used a scalar index | +| `LanceScan` with high `bytes_read` | no index, or too many columns selected | +| `FilterExec` above a search | the filter ran after search rather than pushing into it | +| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used | + +A `LanceScan` feeding a join is the common one worth acting on -- see +[Joins](/sql/dql/join#performance). + +For a table's columns and types without a plan, see +[`DESCRIBE`](/sql/ddl/table#describe). diff --git a/docs/web/sql/dql/fts.mdx b/docs/web/sql/dql/fts.mdx new file mode 100644 index 000000000..fa6490259 --- /dev/null +++ b/docs/web/sql/dql/fts.mdx @@ -0,0 +1,136 @@ +--- +title: "Full-Text Search" +sidebarTitle: "Full-Text Search" +description: "The fts() table function — match, fuzzy, phrase, boolean, boost and multi-match queries from SQL." +icon: magnifying-glass +keywords: ["fts", "sql", "full text search", "fuzzy", "phrase", "boolean", "boost", "multi-match"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +Full-text search is a **table function**, `fts(table, query)`, rather than a +predicate -- it produces a relevance score alongside the matching rows, and a +`WHERE` clause has nowhere to put one. The two string-literal arguments are +the table name and a JSON-encoded query, built with the same query-builder +classes used from the SDK and serialized with `.to_json()`. + +## Setup {#setup} + +```sql +CREATE TABLE my_docs (id INT, text STRING, category STRING, author STRING) + +INSERT INTO my_docs VALUES + (1, 'The happy puppy runs merrily in the park', 'animals', 'Alice'), + (2, 'A curious kitten jumps quickly over the fence', 'animals', 'Bob'), + (3, 'The puppy catches a ball with great enthusiasm', 'sports', 'Alice'), + (5, 'Puppy training requires patience and dedication', 'training', 'Alice') + +CREATE INDEX ON my_docs USING fts (text) +``` + + +Phrase queries need position information, which costs extra index size -- +add `WITH (with_position = true)` to `CREATE INDEX`. Without it, +[phrase queries](#phrase) find nothing. See +[FTS parameters](/indexing/fts-index#fts-parameters). + + +## Basic search {#basic} + +```python +from lancedb.query import MatchQuery + +query = MatchQuery("puppy", "text") +json_query = query.to_json() +``` + +```sql +SELECT id, text, category FROM fts('my_docs', '') +LIMIT 5 +``` + +By default the top 5 matches come back in **arbitrary order** -- select +`_score` and order by it for relevance ranking: + +```sql +SELECT id, text, _score FROM fts('my_docs', '') +ORDER BY _score DESC +LIMIT 5 +``` + +`_score` is a pseudo-column carrying the query's BM25 relevance for the row; +it behaves like any other column, but exists only in a relation that came +from `fts()`. See [Data types](/sql/data-types#pseudo-columns). + +## Fuzzy {#fuzzy} + +Tolerates typos, up to a character-edit budget: + +```python +MatchQuery("pupy", "text", fuzziness=2) +``` + +## Phrase {#phrase} + +Exact phrase, or with `slop` to allow words between terms: + +```python +PhraseQuery("happy puppy", "text") +PhraseQuery("puppy park", "text", slop=2) # up to 2 words between them +``` + +## Boolean {#boolean} + +Combine queries with `&` (AND) or `|` (OR): + +```python +MatchQuery("puppy", "text") & MatchQuery("happy", "text") +MatchQuery("puppy", "text") | MatchQuery("kitten", "text") +``` + +## Boost {#boost} + +Weight a query up or down relative to another: + +```python +from lancedb.query import BoostQuery + +BoostQuery( + positive=MatchQuery("puppy", "text"), + negative=MatchQuery("kitten", "text"), + negative_boost=0.2, +) +``` + +## Multi-match {#multi-match} + +Search several columns at once, optionally weighting them differently: + +```python +from lancedb.query import MultiMatchQuery + +MultiMatchQuery("puppy", ["text", "category"]) +MultiMatchQuery("puppy", ["text", "category"], boosts=[2.0, 1.0]) # text weighted 2x +``` + +## Combining with SQL {#combining} + +`fts()` is a relation, so it composes with the rest of the SQL surface -- +filter, join, or aggregate over it same as any table: + +```sql +SELECT id, text, category FROM fts('my_docs', '') +WHERE category = 'animals' +LIMIT 5 +``` + +## What to read next {#next} + + + + The query-builder equivalent, and every query form in full. + + + Tokenizers, and the position information phrase queries need. + + diff --git a/docs/web/sql/join.mdx b/docs/web/sql/dql/join.mdx similarity index 100% rename from docs/web/sql/join.mdx rename to docs/web/sql/dql/join.mdx diff --git a/docs/web/sql/sampling.mdx b/docs/web/sql/dql/sampling.mdx similarity index 100% rename from docs/web/sql/sampling.mdx rename to docs/web/sql/dql/sampling.mdx diff --git a/docs/web/sql/select.mdx b/docs/web/sql/dql/select.mdx similarity index 100% rename from docs/web/sql/select.mdx rename to docs/web/sql/dql/select.mdx diff --git a/docs/web/sql/subqueries.mdx b/docs/web/sql/dql/subqueries.mdx similarity index 86% rename from docs/web/sql/subqueries.mdx rename to docs/web/sql/dql/subqueries.mdx index d35c4c89f..22eee986b 100644 --- a/docs/web/sql/subqueries.mdx +++ b/docs/web/sql/dql/subqueries.mdx @@ -19,7 +19,7 @@ SELECT * FROM ( WHERE n > 100 ``` -A derived table needs an alias. It can be [sampled](/sql/sampling) like a base +A derived table needs an alias. It can be [sampled](/sql/dql/sampling) like a base table: ```sql @@ -60,14 +60,14 @@ correlated, and is conceptually evaluated per outer row. The planner usually rewrites it into a join. -Check with [`EXPLAIN`](/sql/explain) when a correlated subquery is slow. A plan +Check with [`EXPLAIN`](/sql/dql/explain) when a correlated subquery is slow. A plan showing a join has been optimized; one showing a repeated scan has not, and -rewriting it as an explicit [join](/sql/join) is the fix. +rewriting it as an explicit [join](/sql/dql/join) is the fix. ## CTEs as an alternative {#ctes} -A [CTE](/sql/select#ctes) is a named subquery, and is usually more readable when +A [CTE](/sql/dql/select#ctes) is a named subquery, and is usually more readable when the same subquery appears twice or when nesting gets deep. ```sql diff --git a/docs/web/sql/window.mdx b/docs/web/sql/dql/window.mdx similarity index 94% rename from docs/web/sql/window.mdx rename to docs/web/sql/dql/window.mdx index 91e58ea95..42f7f1c0b 100644 --- a/docs/web/sql/window.mdx +++ b/docs/web/sql/dql/window.mdx @@ -10,7 +10,7 @@ keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead" A window function computes across a set of rows related to the current one and returns a value **per row**. That is the difference from -[aggregation](/sql/aggregation), which collapses the rows it covers. +[aggregation](/sql/dql/aggregation), which collapses the rows it covers. ```sql SELECT @@ -73,7 +73,7 @@ lag lead first_value last_value nth_value **Aggregates** -Any [aggregate function](/sql/aggregation#functions) used with `OVER` becomes a +Any [aggregate function](/sql/dql/aggregation#functions) used with `OVER` becomes a window function — that is how running totals and moving averages are written. ## Deduplicating with a window {#dedup} diff --git a/docs/web/sql/explain.mdx b/docs/web/sql/explain.mdx deleted file mode 100644 index efbae49e1..000000000 --- a/docs/web/sql/explain.mdx +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: "EXPLAIN" -sidebarTitle: "EXPLAIN" -description: "Reading the plan for a statement, and measuring it." -icon: map -keywords: ["explain", "explain analyze", "query plan", "describe"] ---- - -Enterprise [SQL runs on a deployment.](/sql) - -```sql -EXPLAIN SELECT * FROM documents WHERE identifier < 1000 -EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000 -``` - -`EXPLAIN` resolves the plan without running the query. `EXPLAIN ANALYZE` -executes it and annotates every operator with runtime metrics — elapsed time, -rows produced, bytes read, index comparisons. - -The operators are the same ones the SDK methods produce, so the reasoning about -index coverage and filter pushdown carries across unchanged. The two pages that -explain how to read them: - - - - Reading a plan: operators, pushdown, index usage. - - - Reading the metrics, and a worked before/after. - - - -## What to look for {#what-to-look-for} - -| Sign | Means | -| --- | --- | -| `ScalarIndexQuery` | the predicate used a scalar index | -| `LanceScan` with high `bytes_read` | no index, or too many columns selected | -| `FilterExec` above a search | the filter ran after search rather than pushing into it | -| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used | - -A `LanceScan` feeding a join is the common one worth acting on — see -[Joins](/sql/join#performance). - -## DESCRIBE {#describe} - -```sql -DESCRIBE documents -``` - -The table's columns and types, without a plan. diff --git a/docs/web/sql/functions.mdx b/docs/web/sql/functions.mdx index 03e05c038..f2a1ae9e9 100644 --- a/docs/web/sql/functions.mdx +++ b/docs/web/sql/functions.mdx @@ -15,8 +15,8 @@ plus what Lance adds. | | | | --- | --- | | [Scalar functions](/sql/scalar-functions) | per row — math, string, regex, date/time, array, struct, JSON | -| [Aggregate functions](/sql/aggregation#functions) | per group — `count`, `sum`, statistical, approximate | -| [Window functions](/sql/window#functions) | per window — ranking, offset, running aggregates | +| [Aggregate functions](/sql/dql/aggregation#functions) | per group — `count`, `sum`, statistical, approximate | +| [Window functions](/sql/dql/window#functions) | per window — ranking, offset, running aggregates | | [Table functions](#table-functions) | produce a relation rather than a value | @@ -63,6 +63,9 @@ For a keyword test that needs no score, the [`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate is index-accelerated and usable directly in `WHERE`. +See [Full-Text Search](/sql/dql/fts) for the fuzzy, phrase, boolean, boost and +multi-match query forms. + ### `_score` {#score} A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves @@ -89,5 +92,5 @@ between a fast predicate and a slow one: | [`contains_tokens`](/sql/scalar-functions#contains-tokens) | full-text index | | [`json_get`, typed variants, `json_extract`](/sql/scalar-functions#json) | JSON scalar index | -Confirm with [`EXPLAIN`](/sql/explain): an accelerated predicate shows an index -node rather than a `LanceScan` followed by a filter. +Confirm with [`EXPLAIN`](/sql/dql/explain): an accelerated predicate shows an +index node rather than a `LanceScan` followed by a filter. diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx index c8701c494..7958ea63e 100644 --- a/docs/web/sql/index.mdx +++ b/docs/web/sql/index.mdx @@ -1,7 +1,7 @@ --- title: "SQL reference" sidebarTitle: "Overview" -description: "The SQL surface: dialect, where a query runs, and a page per construct." +description: "The SQL surface: dialect, connecting a client, and a page per construct." icon: table-list keywords: ["sql", "datafusion", "flight sql", "reference", "dialect"] --- @@ -24,47 +24,111 @@ What these pages add is the part that is LanceDB's: how a query reaches the engine, how Lance columns appear to it, and the statements a lakehouse needs that DataFusion has no concept of. -## Where a query runs {#execution} +## Connecting {#connecting} -Queries execute on the server and results come back as Arrow record batches -- -nothing is converted to pandas or Python objects unless you ask. `execute_query` -hands back a reader as soon as the first batch is ready rather than when the -query finishes, so a large result set does not have to fit in memory before you -start reading it. See [Queries](/basics/query). +SQL is served over an +[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) +endpoint, so any Flight SQL client reaches it and results arrive as Arrow +record batches -- nothing is converted to pandas or Python objects unless you +ask. -Served over an [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) -endpoint, so any Flight SQL client reaches it. +From Python, the connection runs SQL directly, without a separate client: + +```python +reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10") +table = reader.read_all() +``` + +`execute_query` hands back a reader as soon as the first batch is ready rather +than when the query finishes, so a large result set does not have to fit in +memory before you start reading it. See [Queries](/basics/query). + +The TypeScript SDK has no SQL method; reach for a Flight SQL client instead -- +and in Python, reach for one too when the caller is outside the SDK's own +process, such as a BI tool or a notebook in another language: + +```bash +pip install flightsql-dbapi # Python DB API 2 interface +npm install --save @lancedb/flightsql-client +``` + + +```python Python icon="python" +from flightsql import FlightSQLClient + +client = FlightSQLClient( + host="your-enterprise-endpoint", + port=10025, + insecure=True, + token="DATABASE_TOKEN", + metadata={"database": "your-project-slug"}, + features={"metadata-reflection": "true"}, +) + +info = client.execute("SELECT * FROM flights WHERE origin = 'SFO'") +ticket = info.endpoints[0].ticket +result = client.do_get(ticket).read_all() +``` + +```typescript TypeScript icon="square-js" +import { Client } from "@lancedb/flightsql-client"; + +const client = await Client.connect({ + host: "your-enterprise-endpoint:10025", + username: "lancedb", + password: "password", +}); + +const result = await client.query("SELECT * FROM flights WHERE origin = 'SFO'"); +``` + + +## Usage {#usage} + +The endpoint executes one statement per request and is meant for the queries +and DDL/DML pages below. Reach for the SDKs instead for table-management +operations that have no SQL form, such as building a vector index or a merge +insert -- see [What SQL cannot do](#cannot). ## Reference {#reference} -**Reading data** +### DDL, by object {#ddl} + +| | | +| --- | --- | +| [Namespace](/sql/ddl/namespace) | `CREATE SCHEMA`, `DROP SCHEMA` | +| [Table](/sql/ddl/table) | `CREATE`, `DESCRIBE`, `ALTER`, `CLONE`, `DROP TABLE` | +| [View](/sql/ddl/view) | `CREATE VIEW`, `DROP VIEW` | +| [Materialized view](/sql/ddl/materialized-view) | `SHOW MATERIALIZED VIEWS` | +| [Index](/sql/ddl/index) | `CREATE INDEX`, `DROP INDEX` | +| [Function](/sql/ddl/function) | `CREATE FUNCTION`, `SHOW FUNCTIONS`, `DROP FUNCTION` | +| [Job](/sql/ddl/job) | `SHOW JOBS`, `SHOW ERRORS`, `PAUSE`/`RESUME`/`CANCEL JOB` | + +### DML {#dml} + +| | | +| --- | --- | +| [Insert](/sql/dml/insert) | `INSERT INTO`, including `INSERT ... SELECT` | +| [Update](/sql/dml/update) | `UPDATE ... SET` | +| [Delete](/sql/dml/delete) | `DELETE FROM` | +| [Copy](/sql/dml/copy) | `COPY INTO` / `COPY TO` | +| [Refresh](/sql/dml/refresh) | filling computed columns and materialized views | + +### DQL {#dql} | | | | --- | --- | | [Data types](/sql/data-types) | how Lance columns appear to SQL, and pseudo-columns | -| [SELECT](/sql/select) | projection, filtering, ordering, set operations, CTEs | -| [Subqueries](/sql/subqueries) | scalar, `IN`, `EXISTS`, correlated, derived tables | -| [Joins](/sql/join) | inner, outer, cross, and namespace-qualified paths | -| [Aggregation](/sql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set | -| [Window functions](/sql/window) | `OVER`, framing, ranking and offset functions | -| [Sampling](/sql/sampling) | `TABLESAMPLE`, by rows or by fragments | +| [Select](/sql/dql/select) | projection, filtering, ordering, set operations, CTEs | +| [Subqueries](/sql/dql/subqueries) | scalar, `IN`, `EXISTS`, correlated, derived tables | +| [Joins](/sql/dql/join) | inner, outer, cross, and namespace-qualified paths | +| [Aggregation](/sql/dql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set | +| [Window functions](/sql/dql/window) | `OVER`, framing, ranking and offset functions | +| [Sampling](/sql/dql/sampling) | `TABLESAMPLE`, by rows or by fragments | +| [Full-Text Search](/sql/dql/fts) | the `fts()` table function, and its query forms | +| [Explain](/sql/dql/explain) | `EXPLAIN`, `EXPLAIN ANALYZE`, `EXPLAIN VERBOSE` | -**Changing things** - -| | | -| --- | --- | -| [DDL](/sql/ddl) | tables, views, indexes, functions, columns, WAL | -| [DML](/sql/dml) | `INSERT`, `UPDATE`, `DELETE`, `COPY` | -| [REFRESH](/sql/refresh) | filling computed columns and materialized views | - -**Introspection** - -| | | -| --- | --- | -| [SHOW](/sql/show) | jobs, functions, materialized views, errors | -| [EXPLAIN](/sql/explain) | plans, and measuring them | - -**Expressions** +### Expressions {#expressions} | | | | --- | --- | @@ -83,7 +147,4 @@ endpoint, so any Flight SQL client reaches it. | `PREPARE` / `EXECUTE` | preparation is a [protocol feature](/sql/prepared-statements), not a statement | | Run on a local database | no engine to plan the statement | -Full-text search **is** available, as the [`fts()`](/sql/functions#fts) table -function and the [`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate. - See [Limits](/basics/limits#sql) for the bounds these pages enforce. diff --git a/docs/web/sql/operators.mdx b/docs/web/sql/operators.mdx index df3371a58..05cad2467 100644 --- a/docs/web/sql/operators.mdx +++ b/docs/web/sql/operators.mdx @@ -35,7 +35,7 @@ its wildcard semantics. `NOT unknown` is still unknown. A `WHERE` clause keeps only rows where the predicate is true, so unknown behaves like false there — but not inside `NOT IN`, which is the usual place this bites. See -[Subqueries](/sql/subqueries#in-exists). +[Subqueries](/sql/dql/subqueries#in-exists). ## Arithmetic {#arithmetic} diff --git a/docs/web/sql/prepared-statements.mdx b/docs/web/sql/prepared-statements.mdx index 2a4ed081f..8acbf085c 100644 --- a/docs/web/sql/prepared-statements.mdx +++ b/docs/web/sql/prepared-statements.mdx @@ -50,7 +50,7 @@ SELECT id, title FROM documents WHERE category = $1 AND score > $2 ``` The exact placeholder spelling and the binding call belong to your client; see -[connecting a client](/search/sql). +[connecting a client](/sql#connecting). ## When it helps {#when} diff --git a/docs/web/sql/scalar-functions.mdx b/docs/web/sql/scalar-functions.mdx index 70f22948d..271293cd1 100644 --- a/docs/web/sql/scalar-functions.mdx +++ b/docs/web/sql/scalar-functions.mdx @@ -12,8 +12,8 @@ Functions that take a value and return a value, one row at a time. The set is DataFusion 54's default registration plus what Lance adds — the names below are what is actually registered, not the union of everything DataFusion documents. -For per-group functions see [Aggregation](/sql/aggregation); for per-window, -[Window functions](/sql/window). +For per-group functions see [Aggregation](/sql/dql/aggregation); for per-window, +[Window functions](/sql/dql/window). ## Math {#math} diff --git a/docs/web/sql/show.mdx b/docs/web/sql/show.mdx deleted file mode 100644 index 14537d3da..000000000 --- a/docs/web/sql/show.mdx +++ /dev/null @@ -1,89 +0,0 @@ ---- -title: "SHOW" -sidebarTitle: "SHOW" -description: "Listing jobs, functions, materialized views, and the errors a job produced." -icon: list -keywords: ["show jobs", "show functions", "show materialized views", "show errors", "show tables"] ---- - -Enterprise [SQL runs on a deployment.](/sql) - -Introspection over objects DataFusion has no concept of. All four are LanceDB -extensions. - - -DataFusion's own `SHOW TABLES`, `SHOW COLUMNS` and `SHOW ` are **not -available**. They are implemented over `information_schema`, which this session -does not enable, so they fail with -`"... is not supported unless information_schema is enabled"`. - -List tables with the SDKs -- `table_names()` / `tableNames()`. For a table's -columns, use [`DESCRIBE`](/sql/explain#describe). - - -## SHOW MATERIALIZED VIEWS {#materialized-views} - -```sql -SHOW MATERIALIZED VIEWS -``` - - -Cheaper than the SDK equivalent. `list_materialized_views` finds views by -reading every table's schema, costing one open per table; this asks the server. - - -## SHOW FUNCTIONS {#functions} - -```sql -SHOW FUNCTIONS -``` - -Lists registered [functions](/management/function) and their versions. Since a -version is immutable and a column pins the version that fills it, this is how -you find which versions are still in use before dropping one. - -## SHOW JOBS {#jobs} - -```sql -SHOW JOBS -``` - -Every job the server knows about — id, type, state, creation time. The SQL -counterpart of `list_jobs`; see [Jobs](/basics/jobs). - -## SHOW ERRORS {#errors} - -```sql -SHOW ERRORS ON
[FOR JOB ''] [LIMIT ] -``` - -```sql -SHOW ERRORS ON documents -SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100 -``` - -Why rows failed. This is the statement to reach for when a -[refresh](/sql/refresh) reports fewer filled rows than you expected. - -**Errors and limits** - -| | | -| --- | --- | -| `LIMIT` | must be between 1 and 100,000 | -| Failed jobs scanned without `FOR JOB` | 32 most recent | -| Tables the server does not open directly | not supported | - -Without `FOR JOB` this surveys recent failures and stops at 32 jobs. When the -listing stops at your `LIMIT`, the message says so — narrow with `FOR JOB` -rather than raising the limit blindly. - -## See also {#see-also} - - - - The SDK listing calls, per object. - - - What a job handle exposes beyond these listings. - -