mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 22:26:53 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
104 lines
4.9 KiB
Plaintext
104 lines
4.9 KiB
Plaintext
---
|
|
title: "SQL reference"
|
|
sidebarTitle: "Overview"
|
|
description: "SQL statements, expressions and data types supported by LanceDB Enterprise."
|
|
icon: table-list
|
|
keywords: ["sql", "datafusion", "flight sql", "reference", "dialect"]
|
|
---
|
|
|
|
LanceDB Enterprise supports SQL through an Arrow Flight SQL endpoint. This
|
|
reference covers statement syntax, data types and expressions. To set up a
|
|
client and run your first query, see [SQL](/basics/sql).
|
|
|
|
## The dialect {#dialect}
|
|
|
|
LanceDB uses [Apache DataFusion](https://datafusion.apache.org) for SQL planning
|
|
and execution. These pages describe LanceDB's supported statements and
|
|
extensions. DataFusion's reference is useful for expression syntax, but does
|
|
not establish that every DataFusion statement or feature is exposed by
|
|
LanceDB.
|
|
|
|
## Connecting {#connecting}
|
|
|
|
Use Python's `execute_query` or a Flight SQL client after configuring the
|
|
[Enterprise connection](/basics/sql#sql-is-remote). TypeScript applications
|
|
use a Flight SQL client for SQL queries.
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
reader = db.execute_query("SELECT id, text FROM my_docs LIMIT 10")
|
|
for batch in reader:
|
|
print(batch)
|
|
```
|
|
</CodeGroup>
|
|
|
|
SQL results arrive as Arrow record batches. Iterating over the reader avoids
|
|
collecting the entire result in application memory. See
|
|
[Read and manage results](/basics/sql#results) for more.
|
|
|
|
## Usage {#usage}
|
|
|
|
The endpoint executes one statement per request and is meant for the queries
|
|
and DDL/DML pages below. Reach for the SDKs instead for table-management
|
|
operations that have no SQL form, such as a merge insert. See
|
|
[What SQL cannot do](#cannot).
|
|
|
|
## Reference {#reference}
|
|
|
|
### DDL, by object {#ddl}
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| [Database](/sql/ddl/database) | `CREATE`, `DESCRIBE`, `SHOW`, `ATTACH`, `DROP DATABASE` |
|
|
| [Namespace](/sql/ddl/namespace) | `CREATE`, `ALTER`, `DESCRIBE`, `SHOW`, `DROP SCHEMA`/`NAMESPACE` |
|
|
| [Table](/sql/ddl/table) | `CREATE`, `DESCRIBE`, `ALTER`, `CLONE`, `DROP TABLE` |
|
|
| [View](/sql/ddl/view) | `CREATE VIEW`, `DROP VIEW` |
|
|
| [Materialized view](/sql/ddl/materialized-view) | `CREATE MATERIALIZED VIEW`, `SHOW MATERIALIZED VIEWS` |
|
|
| [Secret](/sql/ddl/secret) | `CREATE`, `ALTER`, `DESCRIBE`, `SHOW`, `DROP SECRET` |
|
|
| [Storage Integration](/sql/ddl/storage-integration) | `CREATE`, `DESCRIBE`, `SHOW`, `DROP STORAGE INTEGRATION` |
|
|
| [Index](/sql/ddl/index) | `CREATE INDEX`, `DROP INDEX` |
|
|
| [Function](/sql/ddl/function) | `CREATE FUNCTION`, `SHOW FUNCTIONS`, `DROP FUNCTION` |
|
|
| [Job](/sql/ddl/job) | `SHOW JOBS`, `SHOW JOB EVENTS`, `DESCRIBE JOB`, `SHOW ERRORS`, `PAUSE`/`RESUME`/`CANCEL JOB` |
|
|
|
|
### DML {#dml}
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| [Insert](/sql/dml/insert) | `INSERT INTO`, including `INSERT ... SELECT` |
|
|
| [Update](/sql/dml/update) | `UPDATE ... SET` |
|
|
| [Delete](/sql/dml/delete) | `DELETE FROM` |
|
|
| [Copy](/sql/dml/copy) | `COPY` (refused by recent deployment versions) |
|
|
| [Refresh column](/sql/dml/refresh-column) | filling a computed column |
|
|
| [Refresh materialized view](/sql/dml/refresh-materialized-view) | bringing a view's stored result up to date |
|
|
|
|
### DQL {#dql}
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| [Data types](/sql/data-types) | how Lance columns appear to SQL, and pseudo-columns |
|
|
| [Select](/sql/dql/select) | projection, filtering, ordering, set operations, CTEs |
|
|
| [Subqueries](/sql/dql/subqueries) | scalar, `IN`, `EXISTS`, correlated, derived tables |
|
|
| [Joins](/sql/dql/join) | inner, outer, cross, and namespace-qualified paths |
|
|
| [Aggregation](/sql/dql/aggregation) | `GROUP BY`, `HAVING`, the aggregate set |
|
|
| [Window functions](/sql/dql/window) | `OVER`, framing, ranking and offset functions |
|
|
| [Sampling](/sql/dql/sampling) | `TABLESAMPLE`, by rows or by fragments |
|
|
| [Changes](/sql/dql/changes) | `CHANGES`: rows appended since an earlier version |
|
|
| [Full-Text Search](/sql/dql/fts) | the `fts()` table function, and its query forms |
|
|
| [Operators](/sql/dql/operators) | comparison, logical, arithmetic, literals, precedence |
|
|
| [Functions](/sql/dql/functions) | where to find each kind, and table functions |
|
|
| [Scalar functions](/sql/dql/scalar-functions) | the per-row catalogue, and what Lance accelerates |
|
|
| [Explain](/sql/dql/explain) | `EXPLAIN`, `EXPLAIN ANALYZE`, `EXPLAIN VERBOSE` |
|
|
|
|
## SQL limitations and alternatives {#cannot}
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| Vector search | See [SQL vector search](/basics/sql#no-vector-sql) for supported query forms and server-version requirements. The [SDK query builder](/search/vector-search) also supports vector search. |
|
|
| Merge insert | no SQL form: it needs a source relation and a per-match policy |
|
|
| `information_schema` | not enabled, so `SHOW TABLES` and `SHOW COLUMNS` fail |
|
|
| `COPY` | refused by recent deployment versions (see [Copy](/sql/dml/copy)) |
|
|
| `PREPARE` / `EXECUTE` | preparation is a [protocol feature](/basics/sql#prepared-statements), not a statement |
|
|
| Run on a local database | no engine to plan the statement |
|
|
|
|
See [Limits](/basics/limits#sql) for the bounds these pages enforce.
|