rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-26 13:36:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: seed the SQL reference from DataFusion 54, and cross-link it

Browse Source 
The function catalogue is now real rather than a pointer. Scalar functions are
listed by taxonomy -- math, string, unicode, regex, date/time, array, map,
struct, crypto, type -- and the names come from what datafusion-functions 54
actually registers, not from what DataFusion documents on main. Aggregate and
window pages gain the same treatment.

Two things the audit corrected rather than copied.

SHOW TABLES does not work. DataFusion implements it over information_schema,
which this session does not enable, so it fails at plan time. It was documented
as working here, on the SHOW page and in three Manage tables; all four are
fixed, and the SQL overview now lists the absence alongside SHOW COLUMNS.

Prepared statements are a Flight SQL protocol feature, not a PREPARE statement.
A script carrying PREPARE ... AS SELECT will not run, while the same intent
expressed through a client will. The new page says so rather than copying
DataFusion's syntax.

Every section of the Documentation tab that has a SQL counterpart now links to
it -- 27 of 76 non-SQL pages, each pointing at the specific statement rather
than at the reference index.
This commit is contained in:
Jack Ye
2026-09-14 00:42:31 -07:00
parent cd0ebe3d12
commit fe651ae003
36 changed files with 468 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/sql/functions.mdx
+45 -128
View File
@@ -1,90 +1,40 @@
---
title: "SQL functions and types"
sidebarTitle: "Functions and types"
description: "The dialect is DataFusion's. What that gives you, and what LanceDB adds on top."
title: "Functions"
sidebarTitle: "Functions"
description: "Where to find each kind of function, and the dialect version they come from."
icon: function
keywords: ["sql functions", "datafusion", "scalar functions", "aggregate", "data types", "fts"]
keywords: ["functions", "datafusion", "scalar", "aggregate", "window", "fts", "table function"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org),
currently **version 54**. The dialect, the built-in functions, and the type
system are DataFusion's, registered from its default set — so anything the
DataFusion reference documents for that version is available, and this page does
not reproduce it.
currently **version 54**. The function set is DataFusion's default registration
plus what Lance adds.
What it does document is the version you are querying against, where the
authoritative lists are, and the parts that are LanceDB's rather than
DataFusion's.
## Reference by category {#reference}
| | DataFusion reference |
| | |
| --- | --- |
| Math, conditional, string, binary string, regular expression, time and date | [Scalar functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) |
| `count`, `sum`, `avg`, `min`, `max`, statistical, approximate | [Aggregate functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) |
| `row_number`, `rank`, `lag`, `lead`, and the rest of `OVER (...)` | [Window functions](https://datafusion.apache.org/user-guide/sql/window_functions.html) |
| Arrow-backed types, and how SQL type names map onto them | [Data types](https://datafusion.apache.org/user-guide/sql/data_types.html) |
| `SELECT`, joins, `GROUP BY`, `ORDER BY`, `LIMIT`, CTEs, set operations | [SELECT syntax](https://datafusion.apache.org/user-guide/sql/select.html) |
| [Scalar functions](/sql/scalar-functions) | per row — math, string, regex, date/time, array, struct, JSON |
| [Aggregate functions](/sql/aggregation#functions) | per group — `count`, `sum`, statistical, approximate |
| [Window functions](/sql/window#functions) | per window — ranking, offset, running aggregates |
| [Table functions](#table-functions) | produce a relation rather than a value |
<Note>
Function availability is version-specific: DataFusion adds functions in most
releases, so a function documented on `main` may not exist in 54. When a
function is not recognised, check it against the reference for the version above
before assuming a bug.
Function availability is version-specific. DataFusion adds functions in most
releases, so one documented on `main` may not exist in 54. When a name is not
recognised, check it against the reference for that version before assuming a
bug.
</Note>
## Types are Arrow types {#types}
## Table functions {#table-functions}
There is no SQL type layer over Lance storage. The DataFusion table provider
returns the dataset's Arrow schema directly — `Schema::from(dataset.schema())` —
so a column is the same object to the query engine as it is to the format, and
DataFusion's [type reference](https://datafusion.apache.org/user-guide/sql/data_types.html)
describes it exactly.
That identity is mostly invisible and occasionally load-bearing. A vector column
is a `FixedSizeList<Float32>`: selectable, testable for nullness, joinable like
any other column — but arithmetic over it is list arithmetic, not scalar
arithmetic. When the operation is a similarity comparison, use vector search
rather than SQL.
### Pseudo-columns {#pseudo-columns}
Lance rows carry metadata the provider can expose as columns. They are appended
to the schema only when asked for, so they do not appear in `SELECT *`:
| Column | Type | Meaning |
| --- | --- | --- |
| `_rowid` | `UInt64`, nullable | stable row identifier |
| `_rowaddr` | `UInt64`, nullable | physical row address |
| `_rowoffset` | `UInt64`, nullable | offset within the fragment |
| `_score` | `Float32` | relevance, from [`fts()`](#fts) |
`_rowid` and `_rowaddr` are declared nullable, but not because a row lacks one —
their validity bitmap is reused as a selection vector internally. Do not read a
null there as meaningful.
`_rowid` is the identifier [materialized views](/feature-engineering/materialized-views)
depend on, which is why a source table needs stable row ids enabled at creation.
### Blob columns change shape {#blobs}
A blob column can be read as its **binary contents** or as a **description** —
a pointer plus metadata, without fetching the bytes. That choice changes the
column's Arrow type, and therefore the schema a query sees.
The default reads blobs as descriptions and other binary columns as binary. A
query that selects a blob column and expects bytes will get a struct instead
unless the read is configured otherwise. Worth knowing before writing a
predicate against one.
## What LanceDB adds {#extensions}
A table function appears in `FROM` and produces a relation, which is what lets
it return columns a scalar function has nowhere to put.
### `fts(table, query)` {#fts}
Full-text search is a **table function**, not a predicate — implemented as a
DataFusion `TableProvider` over the dataset, so its result is a relation. That is
what lets it return relevance alongside the matching rows, which a `WHERE` clause
has nowhere to put:
Full-text search is a table function rather than a predicate, because it
produces relevance alongside the matching rows:
```sql
SELECT id, text, _score
@@ -93,13 +43,13 @@ ORDER BY _score DESC
LIMIT 5
```
The second argument is either a bare query string or a JSON query object for
The second argument is a bare query string, or a JSON query object for the
fuzzy, phrase, boolean, boost and multi-match forms.
Its schema is the dataset's, so every column of the table is available in the
result — projections push down into the scan rather than being applied
afterwards. Being a relation is also what makes it compose: join it, filter it,
aggregate over it.
result and projections push into the scan rather than being applied afterwards.
Being a relation is also what makes it compose — join it, filter it, aggregate
over it:
```sql
SELECT c.name, count(*) AS hits
@@ -109,68 +59,35 @@ GROUP BY c.name
ORDER BY hits DESC
```
For a keyword test that needs no score, the
[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate is
index-accelerated and usable directly in `WHERE`.
### `_score` {#score}
A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves
like a normal column — order by it, filter on it, project it — and exists only
in a relation that came from `fts()`.
like any other column — order by it, filter on it, project it — and exists only
in a relation that came from `fts()`. See
[Data types](/sql/data-types#pseudo-columns).
### `contains_tokens(column, tokens)` {#contains-tokens}
A predicate rather than a relation, for when you want keyword matching inside a
`WHERE` clause and do not need relevance:
### `generate_series` {#generate-series}
```sql
SELECT * FROM my_docs WHERE contains_tokens(text, 'fox jumps dog')
SELECT * FROM generate_series(1, 10)
```
It is true when the column contains **all** the tokens, which are split on
whitespace and punctuation. Equivalent to an `fts()` match query with fuzziness
off, `AND` semantics, and the simple tokenizer.
Rows from a range, useful for building a dense axis to left-join sparse data
onto.
**It is index-accelerated.** When the column has a full-text index that suits
the query, the planner uses it instead of scanning. That is the reason to prefer
it over `LIKE` for keyword tests.
## What Lance accelerates {#accelerated}
Use `fts()` when you need `_score` or the fuzzy, phrase, or boost forms; use
`contains_tokens` when you just need a boolean.
Two function families use an index rather than scanning, which is the difference
between a fast predicate and a slow one:
### JSON functions {#json}
Lance registers a set of JSON accessors over string columns holding JSON:
| Function | Returns |
| | Index used |
| --- | --- |
| `json_get(col, key)` | the value at `key`, still JSON |
| `json_get_string(col, key)` | `Utf8` |
| `json_get_int(col, key)` | `Int64` |
| `json_get_float(col, key)` | `Float64` |
| `json_get_bool(col, key)` | `Boolean` |
| `json_extract(col, path)` | `Utf8`, by path |
| `json_extract_with_type(col, path, type)` | the requested type |
| `json_exists(col, path)` | `Boolean` |
| `json_array_contains(col, value)` | `Boolean` |
| `json_array_length(col)` | length |
| [`contains_tokens`](/sql/scalar-functions#contains-tokens) | full-text index |
| [`json_get`, typed variants, `json_extract`](/sql/scalar-functions#json) | JSON scalar index |
`json_get` returns JSON, so accessors nest for a nested document:
```sql
SELECT * FROM events
WHERE json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice'
```
**`json_get`, its typed variants, and `json_extract` are index-accelerated** by a
JSON scalar index on the column. The others evaluate per row.
<Note>
Lance also defines geospatial `st_*` functions, but they are behind an optional
build feature that LanceDB does not ship. Calling one returns an error telling
you the feature is missing rather than an unknown-function error.
</Note>
## Reaching a function LanceDB does not name {#escape-hatch}
Nothing here wraps or restricts the DataFusion set — a query reaches any
function in the reference above by writing it. If a name is not recognised,
check it against the reference for the version this page pins rather than
assuming it is unsupported.
Confirm with [`EXPLAIN`](/sql/explain): an accelerated predicate shows an index
node rather than a `LanceScan` followed by a filter.