rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-13 07:12:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: reference the DataFusion dialect and Lance's type mapping

Browse Source 
The manual said the SQL dialect's function and type reference was out of scope.
It is not: the engine is DataFusion 54 with its default function set registered,
so the dialect is documented already and the useful thing is to say which
version and point at it.

The types are worth being concrete about rather than deferring. Lance's
DataFusion table provider returns the dataset's Arrow schema directly — there is
no SQL type layer over storage — so DataFusion's type reference describes Lance
columns exactly, and a vector column is a FixedSizeList that behaves like one.

Two consequences of that mapping are documented because they surprise people.
The pseudo-columns the provider can append are real and typed — `_rowid`,
`_rowaddr` and `_rowoffset` are nullable UInt64, and the nullability is a reused
selection bitmap rather than a missing value, so a null there means nothing.
And a blob column's Arrow type depends on how it is read: descriptions by
default, binary on request, which changes the schema a query sees.

`fts()` is recorded as what it is, a DataFusion TableProvider over the dataset.
That explains the parts that otherwise look arbitrary: why it is a table
function rather than a predicate, why every column of the table is available in
its result, and why projections push into the scan.
This commit is contained in:
Jack Ye
2026-09-04 00:49:37 -07:00
parent 6e1473278d
commit f7ed6c6b93
5 changed files with 542 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/docs.json
+9
View File
@@ -149,6 +149,15 @@
"feature-engineering/functions"
]
},
{
"group": "SQL and DataFrame",
"pages": [
"sql/index",
"sql/dataframe",
"sql/expressions",
"sql/functions"
]
},
{
"group": "Support",
"pages": [
docs/web/sql/dataframe.mdx
+157
View File
@@ -0,0 +1,157 @@
---
title: "DataFrame operations"
sidebarTitle: "DataFrame operations"
description: "Every relational operation on a LanceDB DataFrame, with signatures and semantics."
icon: table-list
keywords: ["dataframe", "select", "filter", "join", "aggregate", "substrait"]
---
A DataFrame is a lazy logical plan. Every operation returns a new plan; nothing
executes until [`execute()`](#execution).
Obtain one from a connection:
```python
import lancedb
from lancedb import col, lit
from lancedb import sql_functions as F
db = lancedb.connect("db://your-database", api_key="...")
df = db.table("documents")
```
`db.table(name, *, namespace_path=None)` scans a table. It opens the table to
read its schema, so the plan knows its columns before anything runs.
## Projection {#projection}
| Operation | Effect |
| --- | --- |
| `select(*expressions)` | Project columns or expressions. At least one required. |
| `with_column(name, expression)` | Add one derived column. |
| `with_columns(**expressions)` | Add several, named by keyword. |
| `drop(*columns)` | Remove columns by name. |
| `with_column_renamed(old, new)` | Rename one column. |
```python
df.select("id", col("score") * 2)
df.with_columns(doubled=col("score") * 2, flag=col("score") > 0.5)
```
A bare string is a column reference, so `select("id")` and `select(col("id"))`
mean the same thing. Use `col()` when you need methods or operators on it.
## Filtering {#filtering}
`filter(*predicates)` keeps rows matching **every** predicate — multiple
arguments are combined with `AND`:
```python
df.filter(col("score") > 0.5, col("lang") == "en")
df.filter((col("score") > 0.5) & (col("lang") == "en")) # equivalent
```
At least one predicate is required. For `OR`, combine with `|` inside a single
argument.
<Warning>
Python's `and` and `or` cannot be overloaded and will **not** do what you want on
expressions. Use `&`, `|` and `~`, and parenthesise them — they bind more
tightly than comparison operators, so `col("a") > 1 & col("b") < 2` parses as
`col("a") > (1 & col("b")) < 2`.
</Warning>
## Aggregation {#aggregation}
`aggregate(group_by, aggregates)` groups rows and computes aggregates over each
group.
```python
df.aggregate("lang", F.count(col("id")))
df.aggregate(["lang", "source"], [F.avg(col("score")), F.max(col("score"))])
df.aggregate(None, F.count(col("id"))) # whole relation, no grouping
```
Both arguments accept a single value or an iterable. `group_by=None` aggregates
over the whole relation. The available aggregates are in
[Expressions](/sql/expressions#aggregates).
## Ordering and slicing {#ordering}
| Operation | Effect |
| --- | --- |
| `sort(*expressions)` | Order rows. |
| `limit(count, offset=0)` | Take `count` rows after skipping `offset`. |
| `distinct()` | Remove duplicate rows. |
A bare expression sorts **ascending, nulls last**. For other orders use a
`SortExpr`, which follows DataFusion's convention: when `nulls_first` is not
given it defaults to the opposite of `ascending`, so descending sorts put nulls
first.
```python
df.sort(col("score")) # ascending, nulls last
df.sort(col("score").sort(ascending=False))
```
## Joins {#joins}
```python
df.join(other, on="id")
df.join(other, left_on="doc_id", right_on="id", how="left")
df.join(other, on=["tenant", "id"], how="inner")
```
Pass either `on` (same key names on both sides) or **both** `left_on` and
`right_on`. Passing `on` together with either raises, as does passing neither.
Keys may be a single name or a sequence.
`how` defaults to `"inner"`.
<Note>
Both plans must come from the **same connection** and the **same default
namespace**; otherwise the join raises before anything is submitted. The same
applies to set operations.
Use `alias(name)` to give a relation a name before joining it to itself.
</Note>
## Set operations {#set-operations}
| Operation | Effect |
| --- | --- |
| `union(other, distinct=False)` | Rows from both. |
| `intersect(other, distinct=False)` | Rows present in both. |
| `except_all(other, distinct=False)` | Rows in this plan and not the other. |
All three default to **keeping duplicates**. Pass `distinct=True` for set
semantics. Note this is the opposite default to SQL's bare `UNION`, which
deduplicates — `union()` here corresponds to `UNION ALL`.
## Inspecting a plan {#inspecting}
| | |
| --- | --- |
| `schema` | the plan's output schema, as a `pyarrow.Schema` |
| `to_substrait()` | the logical plan as a Substrait protobuf |
| `repr(df)` | a readable rendering of the plan |
`schema` and `to_substrait()` work on a local connection, so a plan can be
asserted against in tests without a server.
## Execution {#execution}
| | Returns |
| --- | --- |
| `execute()` | a blocking Arrow `RecordBatchReader` |
| `execute_async()` | a server-side query handle |
On `AsyncConnection` both are coroutines and `execute()` yields an async reader.
Execution requires a remote connection. A local one can build and serialize a
plan but not run it.
```python
reader = df.execute()
table = reader.read_all()
```
docs/web/sql/expressions.mdx
+135
View File
@@ -0,0 +1,135 @@
---
title: "Expressions"
sidebarTitle: "Expressions"
description: "Columns, literals, operators, string and null predicates, and the aggregate functions."
icon: function
keywords: ["expressions", "col", "lit", "operators", "aggregate", "sql functions"]
---
Expressions are the values a DataFrame operation takes: what to project, what to
filter on, what to group by. They are built from column references and literals,
combined with operators, and evaluated by the server.
```python
from lancedb import col, lit, func
from lancedb import sql_functions as F
```
## Building blocks {#building-blocks}
| | |
| --- | --- |
| `col(name)` | reference a column by name |
| `lit(value)` | a literal — `bool`, `int`, `float`, `str`, `bytes`, `date`, `datetime`, `Decimal` |
| `func(name, *args)` | call a server-side function by name |
`func()` is the escape hatch. Anything the engine exposes that has no Python
method here can be reached by name — at the cost of being checked only when the
query runs.
Where an operation accepts an expression, a bare Python value is coerced to a
literal and a bare string to a column reference. `col("age") > 18` needs no
`lit()`.
## Operators {#operators}
| Category | Operators |
| --- | --- |
| Comparison | `==` `!=` `<` `<=` `>` `>=` |
| Boolean | `&` (and) · `\|` (or) · `~` (not) |
| Arithmetic | `+` `-` `*` `/` |
All of them work with the expression on either side, so `1 + col("x")` is as
valid as `col("x") + 1`.
<Warning>
Use `&`, `|` and `~` — **not** `and`, `or` and `not`. Python cannot overload the
keywords, so they will silently evaluate the expression object's truthiness
instead of building a predicate.
Parenthesise them. `&` and `|` bind more tightly than comparisons, so
```python
col("a") > 1 & col("b") < 2 # wrong: parses as col("a") > (1 & col("b")) < 2
(col("a") > 1) & (col("b") < 2) # right
```
</Warning>
## Methods on an expression {#methods}
| Method | Result |
| --- | --- |
| `lower()` | lowercase a string |
| `upper()` | uppercase a string |
| `contains(substr)` | true when the string contains `substr` |
| `isin(values)` | true when the value is in `values` |
| `is_null()` | true when the value is null |
| `is_not_null()` | true when the value is not null |
| `alias(name)` | name the result, for projection |
| `sort(ascending=True, nulls_first=None)` | a sort key — see below |
```python
df.filter(col("title").lower().contains("puppy"))
df.filter(col("lang").isin(["en", "fr"]))
df.select(col("score").alias("relevance"))
```
## Sort keys {#sort}
`expr.sort()` produces a `SortExpr`, which carries the direction and the null
placement.
| | Default |
| --- | --- |
| `ascending` | `True` |
| `nulls_first` | the opposite of `ascending` |
That default is DataFusion's convention and is worth reading twice: an ascending
sort puts nulls **last**, and a descending sort puts them **first**. Set
`nulls_first` explicitly when it matters.
```python
df.sort(col("score").sort(ascending=False)) # desc, nulls first
df.sort(col("score").sort(ascending=False, nulls_first=False)) # desc, nulls last
```
Passing a bare expression to `sort()` is equivalent to `ascending=True,
nulls_first=False`.
## Aggregates {#aggregates}
From `lancedb.sql_functions`:
| Function | Result |
| --- | --- |
| `sum(expr)` | sum of non-null values |
| `avg(expr)` | mean of non-null values |
| `min(expr)` | minimum non-null value |
| `max(expr)` | maximum non-null value |
| `count(expr)` | number of non-null values |
All five ignore nulls, including `count`. To count rows rather than values,
count a column that cannot be null.
```python
from lancedb import sql_functions as F
df.aggregate("lang", [F.count(col("id")), F.avg(col("score"))])
```
These five have Python wrappers. Every other function in the
[DataFusion set](/sql/functions) — scalar, aggregate and window — is reachable
through `func(name, ...)`, unchecked until execution.
## Naming results {#naming}
An aggregate or derived column takes a generated name unless you give it one.
`alias()` on the expression, or the keyword in `with_columns()`, sets it:
```python
df.aggregate("lang", F.avg(col("score")).alias("mean_score"))
df.with_columns(mean_score=F.avg(col("score")))
```
Worth doing on anything downstream code refers to by name — a generated name is
not part of the API and may change.
docs/web/sql/functions.mdx
+133
View File
@@ -0,0 +1,133 @@
---
title: "SQL functions and types"
sidebarTitle: "Functions and types"
description: "The dialect is DataFusion's. What that gives you, and what LanceDB adds on top."
icon: function
keywords: ["sql functions", "datafusion", "scalar functions", "aggregate", "data types", "fts"]
---
LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org),
currently **version 54**. The dialect, the built-in functions, and the type
system are DataFusion's, registered from its default set — so anything the
DataFusion reference documents for that version is available, and this page does
not reproduce it.
What it does document is the version you are querying against, where the
authoritative lists are, and the parts that are LanceDB's rather than
DataFusion's.
## Reference by category {#reference}
| | DataFusion reference |
| --- | --- |
| Math, conditional, string, binary string, regular expression, time and date | [Scalar functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) |
| `count`, `sum`, `avg`, `min`, `max`, statistical, approximate | [Aggregate functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) |
| `row_number`, `rank`, `lag`, `lead`, and the rest of `OVER (...)` | [Window functions](https://datafusion.apache.org/user-guide/sql/window_functions.html) |
| Arrow-backed types, and how SQL type names map onto them | [Data types](https://datafusion.apache.org/user-guide/sql/data_types.html) |
| `SELECT`, joins, `GROUP BY`, `ORDER BY`, `LIMIT`, CTEs, set operations | [SELECT syntax](https://datafusion.apache.org/user-guide/sql/select.html) |
<Note>
Function availability is version-specific: DataFusion adds functions in most
releases, so a function documented on `main` may not exist in 54. When a
function is not recognised, check it against the reference for the version above
before assuming a bug.
</Note>
## Types are Arrow types {#types}
There is no SQL type layer over Lance storage. The DataFusion table provider
returns the dataset's Arrow schema directly — `Schema::from(dataset.schema())` —
so a column is the same object to the query engine as it is to the format, and
DataFusion's [type reference](https://datafusion.apache.org/user-guide/sql/data_types.html)
describes it exactly.
That identity is mostly invisible and occasionally load-bearing. A vector column
is a `FixedSizeList<Float32>`: selectable, testable for nullness, joinable like
any other column — but arithmetic over it is list arithmetic, not scalar
arithmetic. When the operation is a similarity comparison, use vector search
rather than SQL.
### Pseudo-columns {#pseudo-columns}
Lance rows carry metadata the provider can expose as columns. They are appended
to the schema only when asked for, so they do not appear in `SELECT *`:
| Column | Type | Meaning |
| --- | --- | --- |
| `_rowid` | `UInt64`, nullable | stable row identifier |
| `_rowaddr` | `UInt64`, nullable | physical row address |
| `_rowoffset` | `UInt64`, nullable | offset within the fragment |
| `_score` | `Float32` | relevance, from [`fts()`](#fts) |
`_rowid` and `_rowaddr` are declared nullable, but not because a row lacks one —
their validity bitmap is reused as a selection vector internally. Do not read a
null there as meaningful.
`_rowid` is the identifier [materialized views](/feature-engineering/materialized-views)
depend on, which is why a source table needs stable row ids enabled at creation.
### Blob columns change shape {#blobs}
A blob column can be read as its **binary contents** or as a **description** —
a pointer plus metadata, without fetching the bytes. That choice changes the
column's Arrow type, and therefore the schema a query sees.
The default reads blobs as descriptions and other binary columns as binary. A
query that selects a blob column and expects bytes will get a struct instead
unless the read is configured otherwise. Worth knowing before writing a
predicate against one.
## What LanceDB adds {#extensions}
### `fts(table, query)` {#fts}
Full-text search is a **table function**, not a predicate — implemented as a
DataFusion `TableProvider` over the dataset, so its result is a relation. That is
what lets it return relevance alongside the matching rows, which a `WHERE` clause
has nowhere to put:
```sql
SELECT id, text, _score
FROM fts('my_docs', 'puppy')
ORDER BY _score DESC
LIMIT 5
```
The second argument is either a bare query string or a JSON query object for
fuzzy, phrase, boolean, boost and multi-match forms.
Its schema is the dataset's, so every column of the table is available in the
result — projections push down into the scan rather than being applied
afterwards. Being a relation is also what makes it compose: join it, filter it,
aggregate over it.
```sql
SELECT c.name, count(*) AS hits
FROM fts('my_docs', 'puppy') AS m
JOIN categories AS c ON c.id = m.category_id
GROUP BY c.name
ORDER BY hits DESC
```
### `_score` {#score}
A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves
like a normal column — order by it, filter on it, project it — and exists only
in a relation that came from `fts()`.
## From the DataFrame API {#from-dataframe}
The [DataFrame API](/sql/dataframe) wraps five aggregates as Python functions —
`sum`, `avg`, `min`, `max`, `count`. Everything else in the DataFusion reference
is reachable through `func(name, *args)`:
```python
from lancedb import col, func
df.select(func("date_trunc", "month", col("created_at")))
df.filter(func("regexp_like", col("title"), "^intro"))
```
`func()` is not checked when you build the plan — an unknown name or a wrong
argument count surfaces when the query runs on the server. That is the trade for
reaching the whole function set without a wrapper per function.
docs/web/sql/index.mdx
+108
View File
@@ -0,0 +1,108 @@
---
title: "SQL and DataFrame manual"
sidebarTitle: "Overview"
description: "Two ways to express an analytical query over the lakehouse, one execution engine underneath."
icon: table-list
keywords: ["sql", "dataframe", "substrait", "flight sql", "query", "analytics"]
---
LanceDB answers analytical queries two ways. **SQL** goes over an Arrow Flight
SQL endpoint, so any Flight SQL client can reach it. The **DataFrame API**
builds the same kind of plan in Python, method by method, and submits it as
[Substrait](https://substrait.io).
They are not layered on each other — neither is sugar for the other. Both
produce a logical plan that the server executes, which is why they cost the same
and return the same Arrow data.
<Note>
The DataFrame API is new. Until it ships, the reference below describes the
merged surface rather than a released one; check the version you are running.
</Note>
## Choosing between them {#choosing}
| | SQL | DataFrame |
| --- | --- | --- |
| Written as | a query string | Python method calls |
| Reaches the server via | Arrow Flight SQL | Substrait |
| Composable in host code | by string building | by value |
| Available from | any Flight SQL client | the Python client |
| Checked | at the server | partly at build time |
Reach for **SQL** when the query is written by a person, pasted from a notebook,
or produced by a tool that already speaks Flight SQL. Reach for the **DataFrame
API** when the query is assembled by code — a filter that depends on user input,
a projection that varies by caller — because building a plan out of values
avoids the quoting and injection problems of building one out of strings.
## Where a query runs {#execution}
Both forms execute on the server.
A DataFrame is **lazy**: every method returns a new plan and nothing runs until
`execute()`. A local connection can build and serialize a plan — useful for
tests, and for inspecting what a query will do — but cannot execute one.
`db.table(...)` on a local connection gives you a plan you can print or convert
to Substrait; submitting it needs a remote connection.
```python
df = db.table("documents").filter(col("score") > 0.5).limit(10)
df.to_substrait() # works anywhere
df.execute() # needs a remote connection
```
That split is worth knowing before you write tests: plan construction is
testable locally, execution is not.
## Query lifecycle {#lifecycle}
`execute()` returns an Arrow `RecordBatchReader` and blocks. `execute_async()`
returns a server-side query handle instead, for queries long enough that holding
a reader open is the wrong shape.
Results are Arrow in both cases. Nothing converts to pandas or Python objects
unless you ask.
## LanceDB's SQL extensions {#extensions}
The dialect is [DataFusion's](/sql/functions) — `SELECT`, `WHERE`, `GROUP BY`,
`ORDER BY`, `LIMIT`, joins, CTEs and window functions — plus what a lakehouse
over multimodal data needs.
**Full-text search** is a table function rather than a predicate, because it
produces a relevance score alongside the matching rows:
```sql
SELECT id, text, _score
FROM fts('my_docs', 'puppy')
ORDER BY _score DESC
LIMIT 5
```
`fts()` takes a table name and a query — either a bare string or a JSON query
object for fuzzy, phrase, boolean, boost and multi-match forms. `_score` is a
pseudo-column carrying relevance, available for ordering and filtering like any
other column.
Because it is a table function, full-text search composes with the rest of a
query: join its results, filter them, aggregate over them.
## Reference {#reference}
<CardGroup cols={3}>
<Card title="DataFrame operations" icon="table-list" href="/sql/dataframe">
Every relational operation, with signatures and semantics.
</Card>
<Card title="Expressions" icon="function" href="/sql/expressions">
Columns, literals, operators, and the aggregate functions.
</Card>
<Card title="Functions and types" icon="calculator" href="/sql/functions">
The DataFusion dialect, and what LanceDB adds to it.
</Card>
</CardGroup>
## What this manual does not cover {#scope}
Connecting a Flight SQL client — drivers, authentication, the endpoint address —
is deployment-specific and documented with the deployment.