diff --git a/docs/web/basics/apis.mdx b/docs/web/basics/apis.mdx index b302a9a3e..6446fb223 100644 --- a/docs/web/basics/apis.mdx +++ b/docs/web/basics/apis.mdx @@ -55,7 +55,7 @@ Similarity search is a query-builder operation. A query that needs both vector similarity and SQL analytics runs the search in an SDK and the analytics over its result. Full-text search *is* available in SQL, as the [`fts()`](/sql/functions#fts) table function and the -[`contains_tokens`](/sql/functions#contains-tokens) predicate. +[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate. ## TypeScript has no SQL method {#ts-no-sql} diff --git a/docs/web/basics/data-types.mdx b/docs/web/basics/data-types.mdx index ee3cde9af..972aa6594 100644 --- a/docs/web/basics/data-types.mdx +++ b/docs/web/basics/data-types.mdx @@ -116,7 +116,7 @@ WHERE json_get_int(data, 'status') = 200 `json_get` returns JSON, so accessors nest for a nested document. **A JSON scalar index accelerates `json_get`, its typed variants, and `json_extract`** -- the rest evaluate per row. The full function list is in -[SQL functions](/sql/functions#json). +[SQL functions](/sql/scalar-functions#json). This is the type to reach for when a column's shape varies per row. When it does not, ordinary `struct` columns are faster and typed. diff --git a/docs/web/basics/jobs.mdx b/docs/web/basics/jobs.mdx index c774d6e28..925c21602 100644 --- a/docs/web/basics/jobs.mdx +++ b/docs/web/basics/jobs.mdx @@ -17,6 +17,10 @@ common source of confusion about how work gets done. They are independent. A synchronous Python call can start work that runs for an hour; an `await` can wait for something that finishes instantly. + +Jobs are listed in SQL with [`SHOW JOBS`](/sql/show#jobs), and their failures with [`SHOW ERRORS`](/sql/show#errors). + + ## Every `_async` method returns a handle {#handles} An `_async` method submits work and hands back a **handle** to it. Which handle diff --git a/docs/web/docs.json b/docs/web/docs.json index 72003bcbf..31f938dcf 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -240,8 +240,10 @@ "sql/refresh", "sql/show", "sql/explain", + "sql/prepared-statements", "sql/operators", - "sql/functions" + "sql/functions", + "sql/scalar-functions" ] }, { diff --git a/docs/web/feature-engineering/data-evolution.mdx b/docs/web/feature-engineering/data-evolution.mdx index a30b97d43..e7513174d 100644 --- a/docs/web/feature-engineering/data-evolution.mdx +++ b/docs/web/feature-engineering/data-evolution.mdx @@ -26,6 +26,10 @@ The reason it is cheap is the format. Adding a derived column writes only that column -- the existing data, which is the expensive part on a multimodal table, is never rewritten. See [why that works](/basics/lance#evolution). + +In SQL this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl#computed-column). + + ## Set up the example table {#set-up-the-example-table} diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx index ef4a5dc3f..27f33e2f3 100644 --- a/docs/web/feature-engineering/index.mdx +++ b/docs/web/feature-engineering/index.mdx @@ -16,6 +16,10 @@ column or a table should contain and let LanceDB fill it. versioned, and refreshed like anything else, not a separate system bolted alongside. + +Every operation here has a SQL form -- see [DDL](/sql/ddl#columns) to declare a column and [REFRESH](/sql/refresh) to fill it. + + ## The shape everything shares {#the-shape} Every mechanism here separates **declaring** from **filling**, and that diff --git a/docs/web/feature-engineering/materialized-views.mdx b/docs/web/feature-engineering/materialized-views.mdx index dc46becf3..9a6ef75ce 100644 --- a/docs/web/feature-engineering/materialized-views.mdx +++ b/docs/web/feature-engineering/materialized-views.mdx @@ -13,6 +13,10 @@ refresh computes the rows. The result is an ordinary table. It can be queried, indexed, and searched, and it appears in `table_names()` — nothing about reading it is special. + +In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view). + + ## Create and refresh {#create-and-refresh} ```python diff --git a/docs/web/feature-engineering/refresh-column.mdx b/docs/web/feature-engineering/refresh-column.mdx index 4db0fc6a8..874353674 100644 --- a/docs/web/feature-engineering/refresh-column.mdx +++ b/docs/web/feature-engineering/refresh-column.mdx @@ -28,6 +28,10 @@ REFRESH COLUMN doubled ON my_table The refresh commits an ordinary table version, so [versioning](/tables/versioning) applies to it like any other write. + +The SQL form is [`REFRESH COLUMN`](/sql/refresh#refresh-column), with the same resource options. + + ## It never revisits filled rows {#idempotent} A refresh fills rows that have no value and leaves the rest alone. Two diff --git a/docs/web/indexing/index.mdx b/docs/web/indexing/index.mdx index d3b123f9c..b48262fd5 100644 --- a/docs/web/indexing/index.mdx +++ b/docs/web/indexing/index.mdx @@ -20,6 +20,10 @@ Scalar indices serve as a foundational optimization layer, accelerating filterin - Key-value lookups (enabling rapid primary key-based retrievals) + +In SQL, see [`CREATE INDEX` and `DROP INDEX`](/sql/ddl#indexes) -- one column, server-chosen name. + + ## Supported Index Types {#supported-index-types} LanceDB provides a comprehensive suite of indexing strategies for different data types and use cases: diff --git a/docs/web/ingestion/delete.mdx b/docs/web/ingestion/delete.mdx index 08281b9fe..43928b8f7 100644 --- a/docs/web/ingestion/delete.mdx +++ b/docs/web/ingestion/delete.mdx @@ -17,6 +17,10 @@ import { RsUpdateOptimizeCleanup, } from '/snippets/tables.mdx'; + +In SQL this is [`DELETE`](/sql/dml#delete). + + ## Delete rows {#delete-rows} Delete operations **soft delete** rows that match a given condition. diff --git a/docs/web/ingestion/index.mdx b/docs/web/ingestion/index.mdx index 1676ce525..504810758 100644 --- a/docs/web/ingestion/index.mdx +++ b/docs/web/ingestion/index.mdx @@ -67,6 +67,10 @@ Getting data in has two halves: creating a table from data you already have, and writing to a table that exists. This page covers the first and introduces the second. + +Every operation here has a SQL form -- see [DML](/sql/dml). + + ## Create a table with data {#create-a-table-with-data} Initialize a LanceDB connection and create a table diff --git a/docs/web/ingestion/insert.mdx b/docs/web/ingestion/insert.mdx index 5290b17c6..efcee0194 100644 --- a/docs/web/ingestion/insert.mdx +++ b/docs/web/ingestion/insert.mdx @@ -19,6 +19,10 @@ import { TsAddProgress, } from '/snippets/tables.mdx'; + +In SQL this is [`INSERT`](/sql/dml#insert), including `INSERT ... SELECT` to move rows server-side. + + ## Ingesting large datasets {#large-datasets} When ingesting large datasets, use `table.add()` on an existing table rather than diff --git a/docs/web/ingestion/merge-insert.mdx b/docs/web/ingestion/merge-insert.mdx index 1db146574..a2c2cd7e1 100644 --- a/docs/web/ingestion/merge-insert.mdx +++ b/docs/web/ingestion/merge-insert.mdx @@ -26,6 +26,10 @@ import { RsMergePartialColumns, } from '/snippets/tables.mdx'; + +Merge insert has **no SQL form** -- it needs a source relation and a per-match policy that one statement cannot express. See [DML](/sql/dml#merge-insert). + + ## Merge incoming rows by key {#merge-incoming-rows-by-key} Merging is different from updating because it involves comparing incoming rows to existing rows by key, and then choosing what to do based on whether the key exists in the target table or not. diff --git a/docs/web/ingestion/update.mdx b/docs/web/ingestion/update.mdx index 04a4206de..c739062f6 100644 --- a/docs/web/ingestion/update.mdx +++ b/docs/web/ingestion/update.mdx @@ -17,6 +17,10 @@ import { RsUpdateUsingSql, } from '/snippets/tables.mdx'; + +In SQL this is [`UPDATE`](/sql/dml#update). + + ## Update rows {#update-rows} Use `update` when you already know which target rows to modify and you do not need to compare against an incoming dataset. diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx index cbbdf9849..2b2b47690 100644 --- a/docs/web/management/database.mdx +++ b/docs/web/management/database.mdx @@ -144,7 +144,7 @@ those writes. The cost is a metadata check on that interval. See | | Python | TypeScript | SQL | | --- | --- | --- | --- | -| list tables | `table_names`, `list_tables` | `tableNames`, `listTables` | `SHOW TABLES` | +| list tables | `table_names`, `list_tables` | `tableNames`, `listTables` | -- | | the URI it opened | `uri` | `display()` | -- | | drop every table | `drop_all_tables` | `dropAllTables` | -- | | drop the database | `drop_database` | -- | -- | @@ -164,7 +164,6 @@ await db.dropTable("documents"); ``` ```sql SQL -SHOW TABLES; DROP TABLE documents; ``` diff --git a/docs/web/management/function.mdx b/docs/web/management/function.mdx index 705603b64..b7e166610 100644 --- a/docs/web/management/function.mdx +++ b/docs/web/management/function.mdx @@ -17,6 +17,10 @@ Registration is a remote operation. On a local connection `create_function` raises `NotImplementedError`; use a [computed column](/feature-engineering/refresh-column) instead. + +In SQL, a function is registered with [`CREATE FUNCTION`](/sql/ddl#functions) and listed with [`SHOW FUNCTIONS`](/sql/show#functions). + + ## Define a function {#define} `@udf` packages a scalar Python callable. Input and output types are inferred diff --git a/docs/web/management/index.mdx b/docs/web/management/index.mdx index c4dc1855b..4bd88652d 100644 --- a/docs/web/management/index.mdx +++ b/docs/web/management/index.mdx @@ -39,7 +39,7 @@ See [Namespaces](/management/namespace). | create | `create_table` | `createTable` | `CREATE TABLE` | | create empty | `create_table(schema=)` | `createEmptyTable` | `CREATE TABLE` | | open | `open_table` | `openTable` | -- | -| list | `table_names`, `list_tables` | `tableNames`, `listTables` | `SHOW TABLES` | +| list | `table_names`, `list_tables` | `tableNames`, `listTables` | -- | | exists | `table_exists` | -- | -- | | rename | `rename_table` | `renameTable` | -- | | clone | `clone_table` | `cloneTable` | -- | diff --git a/docs/web/management/materialized-view.mdx b/docs/web/management/materialized-view.mdx index 4c61070fe..f9b1e3ec0 100644 --- a/docs/web/management/materialized-view.mdx +++ b/docs/web/management/materialized-view.mdx @@ -16,6 +16,10 @@ in `table_names()`. Nothing about reading it is special. This page covers creating and managing one. Keeping it current is [Refresh a materialized view](/feature-engineering/materialized-views). + +In SQL, see [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view) and [`SHOW MATERIALIZED VIEWS`](/sql/show#materialized-views). + + ## Creating one {#create} diff --git a/docs/web/management/namespace.mdx b/docs/web/management/namespace.mdx index 1450c66a7..bbaf2d26a 100644 --- a/docs/web/management/namespace.mdx +++ b/docs/web/management/namespace.mdx @@ -28,6 +28,10 @@ This is why many SDK methods in LanceDB, like `create_table`, `open_table`, `dro language: Python uses `namespace_path`, Rust uses builder methods like `.namespace(...)`, and TypeScript uses `namespacePath` arguments. + +In SQL a namespaced table is addressed as a [dotted path](/sql/select#namespaces). + + ## Namespace hierarchy {#namespace-hierarchy} Namespaces are generalizations of catalog specs that give platform developers a clean way to present Lance tables in the structures users expect. The diagram below shows how the hierarchy can go beyond a single level. diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index cd962c7d4..842912dd2 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -13,6 +13,10 @@ This page covers the table as a catalog object. Getting data into one is [Ingest](/ingestion); changing its shape is [Schema evolution](/tables/schema). + +Tables are created and dropped in SQL with [`CREATE TABLE` and `DROP TABLE`](/sql/ddl#tables). + + ## What a table carries {#carries} | | | @@ -65,9 +69,6 @@ const tbl = await db.openTable("documents"); await db.tableNames(); ``` -```sql SQL -SHOW TABLES -``` Each accepts `namespace_path` to act inside a [namespace](/management/namespace) @@ -134,7 +135,7 @@ On a large table the drop can take a while, so `drop_table_async` returns a | create | `create_table` | `createTable` | `CREATE TABLE` | | create empty | `create_table(schema=)` | `createEmptyTable` | `CREATE TABLE` | | open | `open_table` | `openTable` | -- | -| list | `table_names`, `list_tables` | `tableNames`, `listTables` | `SHOW TABLES` | +| list | `table_names`, `list_tables` | `tableNames`, `listTables` | -- | | exists | `table_exists` | -- | -- | | rename | `rename_table` | `renameTable` | -- | | clone | `clone_table` | `cloneTable` | -- | diff --git a/docs/web/search/explain-analyze.mdx b/docs/web/search/explain-analyze.mdx index 6e4ad7bbe..4234684a7 100644 --- a/docs/web/search/explain-analyze.mdx +++ b/docs/web/search/explain-analyze.mdx @@ -10,6 +10,10 @@ runtime metrics -- elapsed time, rows produced, bytes read, index comparisons. Where [`explain_plan`](/search/explain) tells you what a query intends to do, this tells you what it cost. + +The SQL equivalent is [`EXPLAIN ANALYZE`](/sql/explain). + + ## What it measures {#analyze-a-query-plan} Executes the query and provides detailed runtime metrics, including: diff --git a/docs/web/search/explain.mdx b/docs/web/search/explain.mdx index 6c73c8c65..c14d99f89 100644 --- a/docs/web/search/explain.mdx +++ b/docs/web/search/explain.mdx @@ -14,6 +14,10 @@ To measure a query rather than read it, use [`analyze_plan`](/search/explain-analyze), which executes it and annotates every operator with runtime metrics. + +The SQL equivalent is [`EXPLAIN`](/sql/explain). + + ## What it shows {#inspect-a-query-plan} Reveals the logical query plan before execution, helping you identify potential issues with query structure and index usage. This tool is useful for: diff --git a/docs/web/search/filtering.mdx b/docs/web/search/filtering.mdx index b8ebf37d2..0456ff9b9 100644 --- a/docs/web/search/filtering.mdx +++ b/docs/web/search/filtering.mdx @@ -14,6 +14,10 @@ with filtering capabilities even on datasets containing billions of records. On hybrid queries, the same `where(...)` filter is applied to both the vector and full-text halves of the query. The prefilter or postfilter choice controls whether that happens before each subquery scores candidates or after the subquery top-k is produced. + +The same predicates work in SQL -- see [SELECT](/sql/select#filtering) and [Operators](/sql/operators). + + ## Chaining `where` clauses {#chaining-where-clauses} In more recent LanceDB SDK versions (see the callout box below for the exact version numbers), you can call `where(...)` (Python and TypeScript) or `only_if(...)` (Rust) more than once on the same query builder. Each additional filter is combined with the previous one using logical `AND`, so `where("a > 0").where("b < 10")` is equivalent to `where("(a > 0) AND (b < 10)")`. diff --git a/docs/web/search/full-text-search.mdx b/docs/web/search/full-text-search.mdx index 77b1de7b6..dd6b3fd18 100644 --- a/docs/web/search/full-text-search.mdx +++ b/docs/web/search/full-text-search.mdx @@ -13,6 +13,10 @@ import { LanceDB provides support for Full-Text Search via Lance, allowing you to incorporate keyword-based search (based on BM25) in your retrieval solutions. + +In SQL, full-text search is the [`fts()`](/sql/functions#fts) table function, or the [`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate when you do not need a score. + + ## Basic Usage {#basic-usage} Consider that we have a LanceDB table named `my_table`, whose string column `text` we want to index and query via keyword search, the FTS index must be created before you can search via keywords. diff --git a/docs/web/search/vector-search.mdx b/docs/web/search/vector-search.mdx index 185761fbb..16e2fba9f 100644 --- a/docs/web/search/vector-search.mdx +++ b/docs/web/search/vector-search.mdx @@ -48,6 +48,10 @@ Vector search is a technique used to search for similar items based on their vec Raw data (e.g. text, images, audio, etc.) is converted into embeddings via an embedding model, which are then stored in a multimodal lakehouse like LanceDB. To perform similarity search at scale, an index is created on the stored embeddings, which can then used to perform fast lookups. + +Vector search has **no SQL form**. Run it here and use [SQL](/sql) over the result. + + ## Supported distance metrics {#supported-distance-metrics} Distance metrics determine how LanceDB compares vectors to find similar matches. Euclidean or `l2` is the default, and used for general-purpose similarity, `cosine` for unnormalized embeddings, `dot` for normalized embeddings (best performance), or `hamming` for binary vectors. diff --git a/docs/web/sql/aggregation.mdx b/docs/web/sql/aggregation.mdx index 71b9e8003..64c8d8920 100644 --- a/docs/web/sql/aggregation.mdx +++ b/docs/web/sql/aggregation.mdx @@ -47,20 +47,54 @@ is before deciding to refresh it again. ## The function set {#functions} -The aggregates are DataFusion's, and the full list is in its -[aggregate function reference](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html): +Registered in DataFusion 54. Names below are what is actually available. -| Kind | Examples | -| --- | --- | -| General | `count`, `sum`, `min`, `max`, `avg` | -| Statistical | `stddev`, `var`, `corr`, `covar`, `regr_*` | -| Approximate | `approx_distinct`, `approx_percentile_cont`, `approx_median` | -| Collecting | `array_agg`, `string_agg`, `first_value`, `last_value` | +**General** -The approximate ones are worth knowing about on a large table: +```text +count sum min max avg median +bit_and bit_or bit_xor bool_and bool_or +first_value last_value nth_value grouping +``` + +**Statistical** + +```text +stddev stddev_pop stddev_samp +var var_pop var_samp +corr covar_pop covar_samp +regr_slope regr_intercept regr_count regr_r2 +regr_avgx regr_avgy regr_sxx regr_syy regr_sxy +``` + +**Approximate** + +```text +approx_distinct approx_median +approx_percentile_cont approx_percentile_cont_with_weight +percentile_cont +``` + +**Collecting** + +```text +array_agg string_agg +``` + +The approximate functions are the ones worth reaching for on a large table. `approx_distinct` answers "roughly how many distinct values" for a fraction of -what `count(DISTINCT ...)` costs, which matters when the column has high -cardinality. +what `count(DISTINCT ...)` costs, which matters most where it is most expensive +— a high-cardinality column. + +`DISTINCT` works inside an aggregate: `count(DISTINCT category)`. `FILTER` +narrows one aggregate without narrowing the others: + +```sql +SELECT + count(*) AS total, + count(*) FILTER (WHERE score > 0.9) AS confident +FROM documents +``` ## Aggregating a sample {#sampling} diff --git a/docs/web/sql/data-types.mdx b/docs/web/sql/data-types.mdx index 5392641ab..709dfda83 100644 --- a/docs/web/sql/data-types.mdx +++ b/docs/web/sql/data-types.mdx @@ -66,7 +66,7 @@ WHERE json_get_int(data, 'status') = 200 ``` `json_get`, its typed variants, and `json_extract` are **index-accelerated** by a -JSON scalar index. See [Functions](/sql/functions#json). +JSON scalar index. See [Functions](/sql/scalar-functions#json). ### Blobs {#blobs} diff --git a/docs/web/sql/functions.mdx b/docs/web/sql/functions.mdx index 3a4e341ad..03e05c038 100644 --- a/docs/web/sql/functions.mdx +++ b/docs/web/sql/functions.mdx @@ -1,90 +1,40 @@ --- -title: "SQL functions and types" -sidebarTitle: "Functions and types" -description: "The dialect is DataFusion's. What that gives you, and what LanceDB adds on top." +title: "Functions" +sidebarTitle: "Functions" +description: "Where to find each kind of function, and the dialect version they come from." icon: function -keywords: ["sql functions", "datafusion", "scalar functions", "aggregate", "data types", "fts"] +keywords: ["functions", "datafusion", "scalar", "aggregate", "window", "fts", "table function"] --- +Enterprise [SQL runs on a deployment.](/sql) + LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org), -currently **version 54**. The dialect, the built-in functions, and the type -system are DataFusion's, registered from its default set — so anything the -DataFusion reference documents for that version is available, and this page does -not reproduce it. +currently **version 54**. The function set is DataFusion's default registration +plus what Lance adds. -What it does document is the version you are querying against, where the -authoritative lists are, and the parts that are LanceDB's rather than -DataFusion's. - -## Reference by category {#reference} - -| | DataFusion reference | +| | | | --- | --- | -| Math, conditional, string, binary string, regular expression, time and date | [Scalar functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) | -| `count`, `sum`, `avg`, `min`, `max`, statistical, approximate | [Aggregate functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) | -| `row_number`, `rank`, `lag`, `lead`, and the rest of `OVER (...)` | [Window functions](https://datafusion.apache.org/user-guide/sql/window_functions.html) | -| Arrow-backed types, and how SQL type names map onto them | [Data types](https://datafusion.apache.org/user-guide/sql/data_types.html) | -| `SELECT`, joins, `GROUP BY`, `ORDER BY`, `LIMIT`, CTEs, set operations | [SELECT syntax](https://datafusion.apache.org/user-guide/sql/select.html) | +| [Scalar functions](/sql/scalar-functions) | per row — math, string, regex, date/time, array, struct, JSON | +| [Aggregate functions](/sql/aggregation#functions) | per group — `count`, `sum`, statistical, approximate | +| [Window functions](/sql/window#functions) | per window — ranking, offset, running aggregates | +| [Table functions](#table-functions) | produce a relation rather than a value | -Function availability is version-specific: DataFusion adds functions in most -releases, so a function documented on `main` may not exist in 54. When a -function is not recognised, check it against the reference for the version above -before assuming a bug. +Function availability is version-specific. DataFusion adds functions in most +releases, so one documented on `main` may not exist in 54. When a name is not +recognised, check it against the reference for that version before assuming a +bug. -## Types are Arrow types {#types} +## Table functions {#table-functions} -There is no SQL type layer over Lance storage. The DataFusion table provider -returns the dataset's Arrow schema directly — `Schema::from(dataset.schema())` — -so a column is the same object to the query engine as it is to the format, and -DataFusion's [type reference](https://datafusion.apache.org/user-guide/sql/data_types.html) -describes it exactly. - -That identity is mostly invisible and occasionally load-bearing. A vector column -is a `FixedSizeList`: selectable, testable for nullness, joinable like -any other column — but arithmetic over it is list arithmetic, not scalar -arithmetic. When the operation is a similarity comparison, use vector search -rather than SQL. - -### Pseudo-columns {#pseudo-columns} - -Lance rows carry metadata the provider can expose as columns. They are appended -to the schema only when asked for, so they do not appear in `SELECT *`: - -| Column | Type | Meaning | -| --- | --- | --- | -| `_rowid` | `UInt64`, nullable | stable row identifier | -| `_rowaddr` | `UInt64`, nullable | physical row address | -| `_rowoffset` | `UInt64`, nullable | offset within the fragment | -| `_score` | `Float32` | relevance, from [`fts()`](#fts) | - -`_rowid` and `_rowaddr` are declared nullable, but not because a row lacks one — -their validity bitmap is reused as a selection vector internally. Do not read a -null there as meaningful. - -`_rowid` is the identifier [materialized views](/feature-engineering/materialized-views) -depend on, which is why a source table needs stable row ids enabled at creation. - -### Blob columns change shape {#blobs} - -A blob column can be read as its **binary contents** or as a **description** — -a pointer plus metadata, without fetching the bytes. That choice changes the -column's Arrow type, and therefore the schema a query sees. - -The default reads blobs as descriptions and other binary columns as binary. A -query that selects a blob column and expects bytes will get a struct instead -unless the read is configured otherwise. Worth knowing before writing a -predicate against one. - -## What LanceDB adds {#extensions} +A table function appears in `FROM` and produces a relation, which is what lets +it return columns a scalar function has nowhere to put. ### `fts(table, query)` {#fts} -Full-text search is a **table function**, not a predicate — implemented as a -DataFusion `TableProvider` over the dataset, so its result is a relation. That is -what lets it return relevance alongside the matching rows, which a `WHERE` clause -has nowhere to put: +Full-text search is a table function rather than a predicate, because it +produces relevance alongside the matching rows: ```sql SELECT id, text, _score @@ -93,13 +43,13 @@ ORDER BY _score DESC LIMIT 5 ``` -The second argument is either a bare query string or a JSON query object for +The second argument is a bare query string, or a JSON query object for the fuzzy, phrase, boolean, boost and multi-match forms. Its schema is the dataset's, so every column of the table is available in the -result — projections push down into the scan rather than being applied -afterwards. Being a relation is also what makes it compose: join it, filter it, -aggregate over it. +result and projections push into the scan rather than being applied afterwards. +Being a relation is also what makes it compose — join it, filter it, aggregate +over it: ```sql SELECT c.name, count(*) AS hits @@ -109,68 +59,35 @@ GROUP BY c.name ORDER BY hits DESC ``` +For a keyword test that needs no score, the +[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate is +index-accelerated and usable directly in `WHERE`. + ### `_score` {#score} A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves -like a normal column — order by it, filter on it, project it — and exists only -in a relation that came from `fts()`. +like any other column — order by it, filter on it, project it — and exists only +in a relation that came from `fts()`. See +[Data types](/sql/data-types#pseudo-columns). -### `contains_tokens(column, tokens)` {#contains-tokens} - -A predicate rather than a relation, for when you want keyword matching inside a -`WHERE` clause and do not need relevance: +### `generate_series` {#generate-series} ```sql -SELECT * FROM my_docs WHERE contains_tokens(text, 'fox jumps dog') +SELECT * FROM generate_series(1, 10) ``` -It is true when the column contains **all** the tokens, which are split on -whitespace and punctuation. Equivalent to an `fts()` match query with fuzziness -off, `AND` semantics, and the simple tokenizer. +Rows from a range, useful for building a dense axis to left-join sparse data +onto. -**It is index-accelerated.** When the column has a full-text index that suits -the query, the planner uses it instead of scanning. That is the reason to prefer -it over `LIKE` for keyword tests. +## What Lance accelerates {#accelerated} -Use `fts()` when you need `_score` or the fuzzy, phrase, or boost forms; use -`contains_tokens` when you just need a boolean. +Two function families use an index rather than scanning, which is the difference +between a fast predicate and a slow one: -### JSON functions {#json} - -Lance registers a set of JSON accessors over string columns holding JSON: - -| Function | Returns | +| | Index used | | --- | --- | -| `json_get(col, key)` | the value at `key`, still JSON | -| `json_get_string(col, key)` | `Utf8` | -| `json_get_int(col, key)` | `Int64` | -| `json_get_float(col, key)` | `Float64` | -| `json_get_bool(col, key)` | `Boolean` | -| `json_extract(col, path)` | `Utf8`, by path | -| `json_extract_with_type(col, path, type)` | the requested type | -| `json_exists(col, path)` | `Boolean` | -| `json_array_contains(col, value)` | `Boolean` | -| `json_array_length(col)` | length | +| [`contains_tokens`](/sql/scalar-functions#contains-tokens) | full-text index | +| [`json_get`, typed variants, `json_extract`](/sql/scalar-functions#json) | JSON scalar index | -`json_get` returns JSON, so accessors nest for a nested document: - -```sql -SELECT * FROM events -WHERE json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice' -``` - -**`json_get`, its typed variants, and `json_extract` are index-accelerated** by a -JSON scalar index on the column. The others evaluate per row. - - -Lance also defines geospatial `st_*` functions, but they are behind an optional -build feature that LanceDB does not ship. Calling one returns an error telling -you the feature is missing rather than an unknown-function error. - - -## Reaching a function LanceDB does not name {#escape-hatch} - -Nothing here wraps or restricts the DataFusion set — a query reaches any -function in the reference above by writing it. If a name is not recognised, -check it against the reference for the version this page pins rather than -assuming it is unsupported. +Confirm with [`EXPLAIN`](/sql/explain): an accelerated predicate shows an index +node rather than a `LanceScan` followed by a filter. diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx index 3e4056308..63f44116d 100644 --- a/docs/web/sql/index.mdx +++ b/docs/web/sql/index.mdx @@ -69,7 +69,9 @@ endpoint, so any Flight SQL client reaches it. | | | | --- | --- | | [Operators](/sql/operators) | comparison, logical, arithmetic, literals, precedence | -| [Functions](/sql/functions) | the function set, and what Lance accelerates | +| [Functions](/sql/functions) | where to find each kind, and table functions | +| [Scalar functions](/sql/scalar-functions) | the per-row catalogue, and what Lance accelerates | +| [Prepared statements](/sql/prepared-statements) | planning once, executing many times | ## What SQL cannot do {#cannot} @@ -77,9 +79,11 @@ endpoint, so any Flight SQL client reaches it. | --- | --- | | Vector search | no SQL form -- a query-builder operation | | Merge insert | no SQL form -- needs a source relation and a per-match policy | +| `information_schema` | not enabled, so `SHOW TABLES` and `SHOW COLUMNS` fail | +| `PREPARE` / `EXECUTE` | preparation is a [protocol feature](/sql/prepared-statements), not a statement | | Run on a local database | no engine to plan the statement | Full-text search **is** available, as the [`fts()`](/sql/functions#fts) table -function and the [`contains_tokens`](/sql/functions#contains-tokens) predicate. +function and the [`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate. See [Limits](/basics/limits#sql) for the bounds these pages enforce. diff --git a/docs/web/sql/operators.mdx b/docs/web/sql/operators.mdx index 804ed2ea7..df3371a58 100644 --- a/docs/web/sql/operators.mdx +++ b/docs/web/sql/operators.mdx @@ -24,7 +24,7 @@ the operands are Lance columns. | `~` `~*` `!~` | regular expression match | -For keyword matching on a text column, [`contains_tokens`](/sql/functions#contains-tokens) +For keyword matching on a text column, [`contains_tokens`](/sql/scalar-functions#contains-tokens) is index-accelerated where `LIKE` is not. Reach for `LIKE` only when you need its wildcard semantics. diff --git a/docs/web/sql/prepared-statements.mdx b/docs/web/sql/prepared-statements.mdx new file mode 100644 index 000000000..2a4ed081f --- /dev/null +++ b/docs/web/sql/prepared-statements.mdx @@ -0,0 +1,65 @@ +--- +title: "Prepared statements" +sidebarTitle: "Prepared Statements" +description: "Preparing a statement once and executing it many times, over the Flight SQL protocol." +icon: bolt +keywords: ["prepared statement", "flight sql", "parameters", "bind"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +A prepared statement is planned once and executed many times, which saves +re-planning a statement whose shape does not change between runs. + +## It is a protocol feature, not a statement {#protocol} + + +There is no SQL `PREPARE` / `EXECUTE` here. Preparation is part of the +**Arrow Flight SQL protocol**, so you reach it through your client rather than +by sending a statement. + + +That distinction matters when porting SQL from another engine: a script +containing `PREPARE name AS SELECT ...` will not run, while the same intent +expressed through a Flight SQL client will. + +## The flow {#flow} + +Every Flight SQL client exposes the same three steps, whatever it calls them: + +1. **Prepare** — send the SQL, get back a handle and the result schema. The + statement is parsed and planned at this point, so a syntax or name error + surfaces here rather than at execution. +2. **Bind and execute** — supply parameter values for the handle and read the + result. Repeat as often as you like. +3. **Close** — release the handle. Servers do not hold them forever. + +Knowing that the plan is built at step 1 is the practical part: if a statement +is slow to prepare and fast to execute, the cost is planning, and reusing the +handle is exactly the right fix. + +## Parameters {#parameters} + +Parameters are placeholders in the SQL text, bound to an Arrow record batch at +execution. Binding rather than interpolating is worth doing for the usual two +reasons — the value cannot change the statement's meaning, and the plan is +reused across values. + +```sql +SELECT id, title FROM documents WHERE category = $1 AND score > $2 +``` + +The exact placeholder spelling and the binding call belong to your client; see +[connecting a client](/search/sql). + +## When it helps {#when} + +| | | +| --- | --- | +| The same query with different values, many times | prepare once, bind per execution | +| A complex plan — several joins, window functions | planning dominates; reuse it | +| A one-off analytical query | no benefit; just execute it | + +For a statement you want to keep rather than re-prepare, a +[view](/management/view) stores the query server-side, and a +[materialized view](/management/materialized-view) stores its rows. diff --git a/docs/web/sql/scalar-functions.mdx b/docs/web/sql/scalar-functions.mdx new file mode 100644 index 000000000..70f22948d --- /dev/null +++ b/docs/web/sql/scalar-functions.mdx @@ -0,0 +1,185 @@ +--- +title: "Scalar functions" +sidebarTitle: "Scalar Functions" +description: "The per-row function set: math, string, regex, date/time, array, struct, and what Lance adds." +icon: calculator +keywords: ["scalar functions", "string functions", "math functions", "regex", "array functions", "json"] +--- + +Enterprise [SQL runs on a deployment.](/sql) + +Functions that take a value and return a value, one row at a time. The set is +DataFusion 54's default registration plus what Lance adds — the names below are +what is actually registered, not the union of everything DataFusion documents. + +For per-group functions see [Aggregation](/sql/aggregation); for per-window, +[Window functions](/sql/window). + +## Math {#math} + +```text +abs ceil floor round trunc signum power sqrt cbrt exp +ln log log2 log10 factorial gcd lcm pi random nanvl +isnan iszero cot degrees radians to_degrees to_radians +sin cos tan asin acos atan atan2 +sinh cosh tanh asinh acosh atanh +``` + +## String {#string} + +```text +ascii bit_length btrim chr concat concat_ws contains ends_with +starts_with levenshtein lower upper ltrim rtrim octet_length +repeat replace split_part to_hex uuid +``` + +Unicode-aware variants: + +```text +character_length find_in_set initcap left right lpad rpad +reverse strpos substr substr_index substring translate +``` + + +For keyword matching on a text column, [`contains_tokens`](#contains-tokens) is +index-accelerated where `contains` and `LIKE` are not. + + +## Regular expressions {#regex} + +```text +regexp_count regexp_instr regexp_like regexp_match regexp_replace +``` + +## Date and time {#datetime} + +```text +current_date current_time date_bin date_part date_trunc +from_unixtime to_unixtime make_date make_time to_char +to_date to_time to_local_time +``` + +`date_bin` is the one worth knowing for analytics: it buckets timestamps into +fixed-width intervals, which is how you build a time series without a +self-join. + +```sql +SELECT date_bin(INTERVAL '1 hour', created_at) AS hour, count(*) +FROM events GROUP BY hour ORDER BY hour +``` + +## Arrays and maps {#arrays} + +```text +make_array array_has array_filter array_transform array_any_match +array_compact array_normalize array_slice arrays_zip cardinality +flatten range generate_series +``` + +```text +map map_keys map_values map_entries map_extract +``` + +Distance functions over arrays: + +```text +cosine_distance distance inner_product +``` + + +These operate on ordinary arrays. They are **not** how you run +[vector search](/search/vector-search) — a search uses the vector index, while +these compute per row over whatever the query already produced. Use them to +score a small candidate set, not to retrieve one. + + +## Structs and fields {#structs} + +```text +named_struct get_field coalesce nullif nvl nvl2 greatest least +union_extract union_tag +``` + +## Crypto and encoding {#crypto} + +```text +digest md5 sha224 sha256 sha384 sha512 encode decode +``` + +## Type functions {#type} + +```text +arrow_cast arrow_try_cast arrow_typeof arrow_field arrow_metadata +with_metadata version +``` + +`arrow_typeof` is the fastest way to answer "what is this column, really" when a +result does not have the shape you expected: + +```sql +SELECT arrow_typeof(vector) FROM documents LIMIT 1 +-- FixedSizeList(Field { name: "item", data_type: Float32, .. }, 1536) +``` + +`arrow_cast` casts to an Arrow type by name, reaching types SQL has no spelling +for. See [Data types](/sql/data-types). + +## What Lance adds {#lance} + +These are registered by Lance on top of DataFusion's set. + +### `contains_tokens(column, tokens)` {#contains-tokens} + +```sql +SELECT * FROM my_docs WHERE contains_tokens(text, 'fox jumps dog') +``` + +True when the column contains **all** the tokens, split on whitespace and +punctuation. Equivalent to an `fts()` match query with fuzziness off, `AND` +semantics and the simple tokenizer. + +**Index-accelerated.** When the column has a full-text index that suits the +query, the planner uses it instead of scanning — which is the reason to prefer +it over `LIKE` for keyword tests. + +### JSON accessors {#json} + +| Function | Returns | +| --- | --- | +| `json_get(col, key)` | the value at `key`, still JSON | +| `json_get_string(col, key)` | `Utf8` | +| `json_get_int(col, key)` | `Int64` | +| `json_get_float(col, key)` | `Float64` | +| `json_get_bool(col, key)` | `Boolean` | +| `json_extract(col, path)` | `Utf8`, by path | +| `json_extract_with_type(col, path, type)` | the requested type | +| `json_exists(col, path)` | `Boolean` | +| `json_array_contains(col, value)` | `Boolean` | +| `json_array_length(col)` | length | + +`json_get` returns JSON, so accessors nest: + +```sql +SELECT * FROM events +WHERE json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice' +``` + +**`json_get`, its typed variants, and `json_extract` are index-accelerated** by a +JSON scalar index on the column. The rest evaluate per row. + +### Geospatial {#geo} + + +Lance defines `st_*` functions on GeoArrow, but they sit behind a build feature +**LanceDB does not ship**. Calling one returns an error naming the missing +feature rather than an unknown-function error, and the spatial index is gated +the same way. + + +## Reaching a function not listed here {#escape-hatch} + +Nothing wraps or restricts the set — a query reaches any registered function by +writing it. When a name is not recognised, check it against +[DataFusion's scalar function reference](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) +for version 54 before assuming it is unsupported; functions are added in most +DataFusion releases, and a function documented on `main` may not exist here yet. diff --git a/docs/web/sql/select.mdx b/docs/web/sql/select.mdx index 3bd084f46..aa81e3d25 100644 --- a/docs/web/sql/select.mdx +++ b/docs/web/sql/select.mdx @@ -39,7 +39,7 @@ Projection pushes into the scan in all three -- an unselected column is never read off storage. On a table of images or video that is the difference between a fast query and a slow one, so **prefer naming columns over `SELECT *`**. -Row metadata is available as [pseudo-columns](/sql/functions#pseudo-columns) +Row metadata is available as [pseudo-columns](/sql/data-types#pseudo-columns) (`_rowid`, `_rowaddr`, `_rowoffset`) but only when named, so they stay out of `SELECT *`. diff --git a/docs/web/sql/show.mdx b/docs/web/sql/show.mdx index 78b879738..bdecf6107 100644 --- a/docs/web/sql/show.mdx +++ b/docs/web/sql/show.mdx @@ -8,16 +8,18 @@ keywords: ["show jobs", "show functions", "show materialized views", "show error Enterprise [SQL runs on a deployment.](/sql) -Introspection. `SHOW TABLES` is DataFusion's; the rest are LanceDB extensions -covering objects DataFusion has no concept of. +Introspection over objects DataFusion has no concept of. All four are LanceDB +extensions. -## SHOW TABLES {#tables} + +DataFusion's own `SHOW TABLES`, `SHOW COLUMNS` and `SHOW ` are **not +available**. They are implemented over `information_schema`, which this session +does not enable, so they fail with +`"... is not supported unless information_schema is enabled"`. -```sql -SHOW TABLES -``` - -Lists the tables the session can see. +List tables with the SDKs -- `table_names()` / `tableNames()`. For a table's +columns, use [`DESCRIBE`](/sql/explain#describe). + ## SHOW MATERIALIZED VIEWS {#materialized-views} diff --git a/docs/web/sql/window.mdx b/docs/web/sql/window.mdx index ab92405d0..5a8c00c42 100644 --- a/docs/web/sql/window.mdx +++ b/docs/web/sql/window.mdx @@ -57,14 +57,24 @@ WINDOW w AS (PARTITION BY category ORDER BY score DESC) ## The functions {#functions} -| Kind | Functions | -| --- | --- | -| Ranking | `row_number`, `rank`, `dense_rank`, `percent_rank`, `ntile`, `cume_dist` | -| Offset | `lag`, `lead`, `first_value`, `last_value`, `nth_value` | -| Aggregate | any [aggregate](/sql/aggregation) used with `OVER` | +Registered in DataFusion 54. -The full set is DataFusion's -[window function reference](https://datafusion.apache.org/user-guide/sql/window_functions.html). +**Ranking** + +```text +row_number rank dense_rank percent_rank cume_dist ntile +``` + +**Offset and positional** + +```text +lag lead first_value last_value nth_value +``` + +**Aggregates** + +Any [aggregate function](/sql/aggregation#functions) used with `OVER` becomes a +window function — that is how running totals and moving averages are written. ## Deduplicating with a window {#dedup} diff --git a/docs/web/tables/schema.mdx b/docs/web/tables/schema.mdx index 9b110abd9..d31308a62 100644 --- a/docs/web/tables/schema.mdx +++ b/docs/web/tables/schema.mdx @@ -59,6 +59,10 @@ LanceDB supports ACID-compliant schema evolution through granular operations (ad * Scale Seamlessly: Handle ML model iterations, regulatory changes, or feature additions * Optimize Continuously: Remove unused fields or enforce new constraints without downtime + +In SQL, schema changes are [`ALTER TABLE`](/sql/ddl#columns). + + ## Schema evolution operations {#schema-evolution-operations} LanceDB supports four primary schema evolution operations: