rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-13 07:12:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: drop the DataFrame API from the SQL manual

Browse Source 
DataFrame support is not going ahead, so the pages documenting it come out:
the relational operations and the expression system, both of which described
only that API.

The manual was framed around having two query surfaces and choosing between
them. With one, that framing is gone — the overview now leads with SQL over
Arrow Flight SQL and what the dialect is, rather than with a comparison to
something that will not exist.

The function reference keeps everything that was about SQL: the DataFusion
version, how Lance columns map onto its type system, the pseudo-columns, blob
handling, and `fts()`. Only the section on reaching functions through the
Python wrapper is replaced, since the wrapper is going too.

Mentions of pandas and polars DataFrames elsewhere in the documentation are
unrelated and untouched.
This commit is contained in:
Jack Ye
2026-09-09 20:49:50 -07:00
parent f7ed6c6b93
commit 96bfdcd937
5 changed files with 44 additions and 384 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/docs.json
+1 -3
View File
@@ -150,11 +150,9 @@
]
},
{
"group": "SQL and DataFrame",
"group": "SQL",
"pages": [
"sql/index",
"sql/dataframe",
"sql/expressions",
"sql/functions"
]
},
docs/web/sql/dataframe.mdx
-157
View File
@@ -1,157 +0,0 @@
---
title: "DataFrame operations"
sidebarTitle: "DataFrame operations"
description: "Every relational operation on a LanceDB DataFrame, with signatures and semantics."
icon: table-list
keywords: ["dataframe", "select", "filter", "join", "aggregate", "substrait"]
---
A DataFrame is a lazy logical plan. Every operation returns a new plan; nothing
executes until [`execute()`](#execution).
Obtain one from a connection:
```python
import lancedb
from lancedb import col, lit
from lancedb import sql_functions as F
db = lancedb.connect("db://your-database", api_key="...")
df = db.table("documents")
```
`db.table(name, *, namespace_path=None)` scans a table. It opens the table to
read its schema, so the plan knows its columns before anything runs.
## Projection {#projection}
| Operation | Effect |
| --- | --- |
| `select(*expressions)` | Project columns or expressions. At least one required. |
| `with_column(name, expression)` | Add one derived column. |
| `with_columns(**expressions)` | Add several, named by keyword. |
| `drop(*columns)` | Remove columns by name. |
| `with_column_renamed(old, new)` | Rename one column. |
```python
df.select("id", col("score") * 2)
df.with_columns(doubled=col("score") * 2, flag=col("score") > 0.5)
```
A bare string is a column reference, so `select("id")` and `select(col("id"))`
mean the same thing. Use `col()` when you need methods or operators on it.
## Filtering {#filtering}
`filter(*predicates)` keeps rows matching **every** predicate — multiple
arguments are combined with `AND`:
```python
df.filter(col("score") > 0.5, col("lang") == "en")
df.filter((col("score") > 0.5) & (col("lang") == "en")) # equivalent
```
At least one predicate is required. For `OR`, combine with `|` inside a single
argument.
<Warning>
Python's `and` and `or` cannot be overloaded and will **not** do what you want on
expressions. Use `&`, `|` and `~`, and parenthesise them — they bind more
tightly than comparison operators, so `col("a") > 1 & col("b") < 2` parses as
`col("a") > (1 & col("b")) < 2`.
</Warning>
## Aggregation {#aggregation}
`aggregate(group_by, aggregates)` groups rows and computes aggregates over each
group.
```python
df.aggregate("lang", F.count(col("id")))
df.aggregate(["lang", "source"], [F.avg(col("score")), F.max(col("score"))])
df.aggregate(None, F.count(col("id"))) # whole relation, no grouping
```
Both arguments accept a single value or an iterable. `group_by=None` aggregates
over the whole relation. The available aggregates are in
[Expressions](/sql/expressions#aggregates).
## Ordering and slicing {#ordering}
| Operation | Effect |
| --- | --- |
| `sort(*expressions)` | Order rows. |
| `limit(count, offset=0)` | Take `count` rows after skipping `offset`. |
| `distinct()` | Remove duplicate rows. |
A bare expression sorts **ascending, nulls last**. For other orders use a
`SortExpr`, which follows DataFusion's convention: when `nulls_first` is not
given it defaults to the opposite of `ascending`, so descending sorts put nulls
first.
```python
df.sort(col("score")) # ascending, nulls last
df.sort(col("score").sort(ascending=False))
```
## Joins {#joins}
```python
df.join(other, on="id")
df.join(other, left_on="doc_id", right_on="id", how="left")
df.join(other, on=["tenant", "id"], how="inner")
```
Pass either `on` (same key names on both sides) or **both** `left_on` and
`right_on`. Passing `on` together with either raises, as does passing neither.
Keys may be a single name or a sequence.
`how` defaults to `"inner"`.
<Note>
Both plans must come from the **same connection** and the **same default
namespace**; otherwise the join raises before anything is submitted. The same
applies to set operations.
Use `alias(name)` to give a relation a name before joining it to itself.
</Note>
## Set operations {#set-operations}
| Operation | Effect |
| --- | --- |
| `union(other, distinct=False)` | Rows from both. |
| `intersect(other, distinct=False)` | Rows present in both. |
| `except_all(other, distinct=False)` | Rows in this plan and not the other. |
All three default to **keeping duplicates**. Pass `distinct=True` for set
semantics. Note this is the opposite default to SQL's bare `UNION`, which
deduplicates — `union()` here corresponds to `UNION ALL`.
## Inspecting a plan {#inspecting}
| | |
| --- | --- |
| `schema` | the plan's output schema, as a `pyarrow.Schema` |
| `to_substrait()` | the logical plan as a Substrait protobuf |
| `repr(df)` | a readable rendering of the plan |
`schema` and `to_substrait()` work on a local connection, so a plan can be
asserted against in tests without a server.
## Execution {#execution}
| | Returns |
| --- | --- |
| `execute()` | a blocking Arrow `RecordBatchReader` |
| `execute_async()` | a server-side query handle |
On `AsyncConnection` both are coroutines and `execute()` yields an async reader.
Execution requires a remote connection. A local one can build and serialize a
plan but not run it.
```python
reader = df.execute()
table = reader.read_all()
```
docs/web/sql/expressions.mdx
-135
View File
@@ -1,135 +0,0 @@
---
title: "Expressions"
sidebarTitle: "Expressions"
description: "Columns, literals, operators, string and null predicates, and the aggregate functions."
icon: function
keywords: ["expressions", "col", "lit", "operators", "aggregate", "sql functions"]
---
Expressions are the values a DataFrame operation takes: what to project, what to
filter on, what to group by. They are built from column references and literals,
combined with operators, and evaluated by the server.
```python
from lancedb import col, lit, func
from lancedb import sql_functions as F
```
## Building blocks {#building-blocks}
| | |
| --- | --- |
| `col(name)` | reference a column by name |
| `lit(value)` | a literal — `bool`, `int`, `float`, `str`, `bytes`, `date`, `datetime`, `Decimal` |
| `func(name, *args)` | call a server-side function by name |
`func()` is the escape hatch. Anything the engine exposes that has no Python
method here can be reached by name — at the cost of being checked only when the
query runs.
Where an operation accepts an expression, a bare Python value is coerced to a
literal and a bare string to a column reference. `col("age") > 18` needs no
`lit()`.
## Operators {#operators}
| Category | Operators |
| --- | --- |
| Comparison | `==` `!=` `<` `<=` `>` `>=` |
| Boolean | `&` (and) · `\|` (or) · `~` (not) |
| Arithmetic | `+` `-` `*` `/` |
All of them work with the expression on either side, so `1 + col("x")` is as
valid as `col("x") + 1`.
<Warning>
Use `&`, `|` and `~` — **not** `and`, `or` and `not`. Python cannot overload the
keywords, so they will silently evaluate the expression object's truthiness
instead of building a predicate.
Parenthesise them. `&` and `|` bind more tightly than comparisons, so
```python
col("a") > 1 & col("b") < 2 # wrong: parses as col("a") > (1 & col("b")) < 2
(col("a") > 1) & (col("b") < 2) # right
```
</Warning>
## Methods on an expression {#methods}
| Method | Result |
| --- | --- |
| `lower()` | lowercase a string |
| `upper()` | uppercase a string |
| `contains(substr)` | true when the string contains `substr` |
| `isin(values)` | true when the value is in `values` |
| `is_null()` | true when the value is null |
| `is_not_null()` | true when the value is not null |
| `alias(name)` | name the result, for projection |
| `sort(ascending=True, nulls_first=None)` | a sort key — see below |
```python
df.filter(col("title").lower().contains("puppy"))
df.filter(col("lang").isin(["en", "fr"]))
df.select(col("score").alias("relevance"))
```
## Sort keys {#sort}
`expr.sort()` produces a `SortExpr`, which carries the direction and the null
placement.
| | Default |
| --- | --- |
| `ascending` | `True` |
| `nulls_first` | the opposite of `ascending` |
That default is DataFusion's convention and is worth reading twice: an ascending
sort puts nulls **last**, and a descending sort puts them **first**. Set
`nulls_first` explicitly when it matters.
```python
df.sort(col("score").sort(ascending=False)) # desc, nulls first
df.sort(col("score").sort(ascending=False, nulls_first=False)) # desc, nulls last
```
Passing a bare expression to `sort()` is equivalent to `ascending=True,
nulls_first=False`.
## Aggregates {#aggregates}
From `lancedb.sql_functions`:
| Function | Result |
| --- | --- |
| `sum(expr)` | sum of non-null values |
| `avg(expr)` | mean of non-null values |
| `min(expr)` | minimum non-null value |
| `max(expr)` | maximum non-null value |
| `count(expr)` | number of non-null values |
All five ignore nulls, including `count`. To count rows rather than values,
count a column that cannot be null.
```python
from lancedb import sql_functions as F
df.aggregate("lang", [F.count(col("id")), F.avg(col("score"))])
```
These five have Python wrappers. Every other function in the
[DataFusion set](/sql/functions) — scalar, aggregate and window — is reachable
through `func(name, ...)`, unchecked until execution.
## Naming results {#naming}
An aggregate or derived column takes a generated name unless you give it one.
`alias()` on the expression, or the keyword in `with_columns()`, sets it:
```python
df.aggregate("lang", F.avg(col("score")).alias("mean_score"))
df.with_columns(mean_score=F.avg(col("score")))
```
Worth doing on anything downstream code refers to by name — a generated name is
not part of the API and may change.
docs/web/sql/functions.mdx
+5 -15
View File
@@ -115,19 +115,9 @@ A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves
like a normal column — order by it, filter on it, project it — and exists only
in a relation that came from `fts()`.
## From the DataFrame API {#from-dataframe}
## Reaching a function LanceDB does not name {#escape-hatch}
The [DataFrame API](/sql/dataframe) wraps five aggregates as Python functions —
`sum`, `avg`, `min`, `max`, `count`. Everything else in the DataFusion reference
is reachable through `func(name, *args)`:
```python
from lancedb import col, func
df.select(func("date_trunc", "month", col("created_at")))
df.filter(func("regexp_like", col("title"), "^intro"))
```
`func()` is not checked when you build the plan — an unknown name or a wrong
argument count surfaces when the query runs on the server. That is the trade for
reaching the whole function set without a wrapper per function.
Nothing here wraps or restricts the DataFusion set — a query reaches any
function in the reference above by writing it. If a name is not recognised,
check it against the reference for the version this page pins rather than
assuming it is unsupported.
docs/web/sql/index.mdx
+38 -74
View File
@@ -1,77 +1,49 @@
---
title: "SQL and DataFrame manual"
title: "SQL manual"
sidebarTitle: "Overview"
description: "Two ways to express an analytical query over the lakehouse, one execution engine underneath."
description: "Analytical SQL over the lakehouse, served through Arrow Flight SQL."
icon: table-list
keywords: ["sql", "dataframe", "substrait", "flight sql", "query", "analytics"]
keywords: ["sql", "flight sql", "query", "analytics", "datafusion", "fts"]
---
LanceDB answers analytical queries two ways. **SQL** goes over an Arrow Flight
SQL endpoint, so any Flight SQL client can reach it. The **DataFrame API**
builds the same kind of plan in Python, method by method, and submits it as
[Substrait](https://substrait.io).
LanceDB answers analytical queries in SQL, served over an
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
endpoint. Any Flight SQL client can reach it, and results come back as Arrow.
They are not layered on each other — neither is sugar for the other. Both
produce a logical plan that the server executes, which is why they cost the same
and return the same Arrow data.
<Note>
The DataFrame API is new. Until it ships, the reference below describes the
merged surface rather than a released one; check the version you are running.
</Note>
## Choosing between them {#choosing}
| | SQL | DataFrame |
| --- | --- | --- |
| Written as | a query string | Python method calls |
| Reaches the server via | Arrow Flight SQL | Substrait |
| Composable in host code | by string building | by value |
| Available from | any Flight SQL client | the Python client |
| Checked | at the server | partly at build time |
Reach for **SQL** when the query is written by a person, pasted from a notebook,
or produced by a tool that already speaks Flight SQL. Reach for the **DataFrame
API** when the query is assembled by code — a filter that depends on user input,
a projection that varies by caller — because building a plan out of values
avoids the quoting and injection problems of building one out of strings.
The dialect is [DataFusion's](/sql/functions), so the syntax and functions are
the ones that reference documents. What this manual covers is the parts that are
LanceDB's: how a query reaches the engine, how Lance columns appear to it, and
the extensions a lakehouse over multimodal data needs.
## Where a query runs {#execution}
Both forms execute on the server.
Queries execute on the server. The client sends a statement and receives Arrow
record batches — nothing is converted to pandas or Python objects unless you ask
for it.
A DataFrame is **lazy**: every method returns a new plan and nothing runs until
`execute()`. A local connection can build and serialize a plan — useful for
tests, and for inspecting what a query will do — but cannot execute one.
`db.table(...)` on a local connection gives you a plan you can print or convert
to Substrait; submitting it needs a remote connection.
Because results are Arrow the whole way, a query that feeds training or
downstream processing does not pay a serialization cost to get there.
```python
df = db.table("documents").filter(col("score") > 0.5).limit(10)
df.to_substrait() # works anywhere
df.execute() # needs a remote connection
```
## The dialect {#dialect}
That split is worth knowing before you write tests: plan construction is
testable locally, execution is not.
Ordinary analytical SQL — `SELECT`, `WHERE`, `GROUP BY`, `ORDER BY`, `LIMIT`,
joins, CTEs and window functions — over Arrow-typed columns.
## Query lifecycle {#lifecycle}
Two things differ from a warehouse you may be used to, and both come from the
storage being Lance rather than something SQL-shaped:
`execute()` returns an Arrow `RecordBatchReader` and blocks. `execute_async()`
returns a server-side query handle instead, for queries long enough that holding
a reader open is the wrong shape.
- **Columns are Arrow types**, not a SQL type system mapped onto storage. A
vector column is a `FixedSizeList<Float32>` and behaves like one.
- **Row metadata is available as columns** — `_rowid`, `_rowaddr` — but only
when asked for, so they stay out of `SELECT *`.
Results are Arrow in both cases. Nothing converts to pandas or Python objects
unless you ask.
Both are covered in [Functions and types](/sql/functions).
## LanceDB's SQL extensions {#extensions}
## Full-text search {#fts}
The dialect is [DataFusion's](/sql/functions) — `SELECT`, `WHERE`, `GROUP BY`,
`ORDER BY`, `LIMIT`, joins, CTEs and window functions — plus what a lakehouse
over multimodal data needs.
**Full-text search** is a table function rather than a predicate, because it
produces a relevance score alongside the matching rows:
Full-text search is a **table function** rather than a predicate, because it
produces relevance alongside the matching rows and a `WHERE` clause has nowhere
to put that:
```sql
SELECT id, text, _score
@@ -80,27 +52,19 @@ ORDER BY _score DESC
LIMIT 5
```
`fts()` takes a table name and a query — either a bare string or a JSON query
object for fuzzy, phrase, boolean, boost and multi-match forms. `_score` is a
pseudo-column carrying relevance, available for ordering and filtering like any
other column.
`fts()` takes a table name and a query — a bare string, or a JSON query object
for fuzzy, phrase, boolean, boost and multi-match forms. `_score` carries
relevance and behaves like any other column.
Because it is a table function, full-text search composes with the rest of a
query: join its results, filter them, aggregate over them.
Being a relation is what makes it compose: join its result, filter it, aggregate
over it.
## Reference {#reference}
<CardGroup cols={3}>
<Card title="DataFrame operations" icon="table-list" href="/sql/dataframe">
Every relational operation, with signatures and semantics.
</Card>
<Card title="Expressions" icon="function" href="/sql/expressions">
Columns, literals, operators, and the aggregate functions.
</Card>
<Card title="Functions and types" icon="calculator" href="/sql/functions">
The DataFusion dialect, and what LanceDB adds to it.
</Card>
</CardGroup>
<Card title="Functions and types" icon="calculator" href="/sql/functions">
The DataFusion dialect and version, how Lance columns map onto it, and what
LanceDB adds.
</Card>
## What this manual does not cover {#scope}