diff --git a/docs/web/docs.json b/docs/web/docs.json
index ca59b1f40..0293bdb36 100644
--- a/docs/web/docs.json
+++ b/docs/web/docs.json
@@ -150,11 +150,9 @@
]
},
{
- "group": "SQL and DataFrame",
+ "group": "SQL",
"pages": [
"sql/index",
- "sql/dataframe",
- "sql/expressions",
"sql/functions"
]
},
diff --git a/docs/web/sql/dataframe.mdx b/docs/web/sql/dataframe.mdx
deleted file mode 100644
index 4f34c0677..000000000
--- a/docs/web/sql/dataframe.mdx
+++ /dev/null
@@ -1,157 +0,0 @@
----
-title: "DataFrame operations"
-sidebarTitle: "DataFrame operations"
-description: "Every relational operation on a LanceDB DataFrame, with signatures and semantics."
-icon: table-list
-keywords: ["dataframe", "select", "filter", "join", "aggregate", "substrait"]
----
-
-A DataFrame is a lazy logical plan. Every operation returns a new plan; nothing
-executes until [`execute()`](#execution).
-
-Obtain one from a connection:
-
-```python
-import lancedb
-from lancedb import col, lit
-from lancedb import sql_functions as F
-
-db = lancedb.connect("db://your-database", api_key="...")
-df = db.table("documents")
-```
-
-`db.table(name, *, namespace_path=None)` scans a table. It opens the table to
-read its schema, so the plan knows its columns before anything runs.
-
-## Projection {#projection}
-
-| Operation | Effect |
-| --- | --- |
-| `select(*expressions)` | Project columns or expressions. At least one required. |
-| `with_column(name, expression)` | Add one derived column. |
-| `with_columns(**expressions)` | Add several, named by keyword. |
-| `drop(*columns)` | Remove columns by name. |
-| `with_column_renamed(old, new)` | Rename one column. |
-
-```python
-df.select("id", col("score") * 2)
-df.with_columns(doubled=col("score") * 2, flag=col("score") > 0.5)
-```
-
-A bare string is a column reference, so `select("id")` and `select(col("id"))`
-mean the same thing. Use `col()` when you need methods or operators on it.
-
-## Filtering {#filtering}
-
-`filter(*predicates)` keeps rows matching **every** predicate — multiple
-arguments are combined with `AND`:
-
-```python
-df.filter(col("score") > 0.5, col("lang") == "en")
-df.filter((col("score") > 0.5) & (col("lang") == "en")) # equivalent
-```
-
-At least one predicate is required. For `OR`, combine with `|` inside a single
-argument.
-
-
-Python's `and` and `or` cannot be overloaded and will **not** do what you want on
-expressions. Use `&`, `|` and `~`, and parenthesise them — they bind more
-tightly than comparison operators, so `col("a") > 1 & col("b") < 2` parses as
-`col("a") > (1 & col("b")) < 2`.
-
-
-## Aggregation {#aggregation}
-
-`aggregate(group_by, aggregates)` groups rows and computes aggregates over each
-group.
-
-```python
-df.aggregate("lang", F.count(col("id")))
-df.aggregate(["lang", "source"], [F.avg(col("score")), F.max(col("score"))])
-df.aggregate(None, F.count(col("id"))) # whole relation, no grouping
-```
-
-Both arguments accept a single value or an iterable. `group_by=None` aggregates
-over the whole relation. The available aggregates are in
-[Expressions](/sql/expressions#aggregates).
-
-## Ordering and slicing {#ordering}
-
-| Operation | Effect |
-| --- | --- |
-| `sort(*expressions)` | Order rows. |
-| `limit(count, offset=0)` | Take `count` rows after skipping `offset`. |
-| `distinct()` | Remove duplicate rows. |
-
-A bare expression sorts **ascending, nulls last**. For other orders use a
-`SortExpr`, which follows DataFusion's convention: when `nulls_first` is not
-given it defaults to the opposite of `ascending`, so descending sorts put nulls
-first.
-
-```python
-df.sort(col("score")) # ascending, nulls last
-df.sort(col("score").sort(ascending=False))
-```
-
-## Joins {#joins}
-
-```python
-df.join(other, on="id")
-df.join(other, left_on="doc_id", right_on="id", how="left")
-df.join(other, on=["tenant", "id"], how="inner")
-```
-
-Pass either `on` (same key names on both sides) or **both** `left_on` and
-`right_on`. Passing `on` together with either raises, as does passing neither.
-Keys may be a single name or a sequence.
-
-`how` defaults to `"inner"`.
-
-
-Both plans must come from the **same connection** and the **same default
-namespace**; otherwise the join raises before anything is submitted. The same
-applies to set operations.
-
-Use `alias(name)` to give a relation a name before joining it to itself.
-
-
-## Set operations {#set-operations}
-
-| Operation | Effect |
-| --- | --- |
-| `union(other, distinct=False)` | Rows from both. |
-| `intersect(other, distinct=False)` | Rows present in both. |
-| `except_all(other, distinct=False)` | Rows in this plan and not the other. |
-
-All three default to **keeping duplicates**. Pass `distinct=True` for set
-semantics. Note this is the opposite default to SQL's bare `UNION`, which
-deduplicates — `union()` here corresponds to `UNION ALL`.
-
-## Inspecting a plan {#inspecting}
-
-| | |
-| --- | --- |
-| `schema` | the plan's output schema, as a `pyarrow.Schema` |
-| `to_substrait()` | the logical plan as a Substrait protobuf |
-| `repr(df)` | a readable rendering of the plan |
-
-`schema` and `to_substrait()` work on a local connection, so a plan can be
-asserted against in tests without a server.
-
-## Execution {#execution}
-
-| | Returns |
-| --- | --- |
-| `execute()` | a blocking Arrow `RecordBatchReader` |
-| `execute_async()` | a server-side query handle |
-
-On `AsyncConnection` both are coroutines and `execute()` yields an async reader.
-
-Execution requires a remote connection. A local one can build and serialize a
-plan but not run it.
-
-```python
-reader = df.execute()
-table = reader.read_all()
-```
diff --git a/docs/web/sql/expressions.mdx b/docs/web/sql/expressions.mdx
deleted file mode 100644
index 856a6bb50..000000000
--- a/docs/web/sql/expressions.mdx
+++ /dev/null
@@ -1,135 +0,0 @@
----
-title: "Expressions"
-sidebarTitle: "Expressions"
-description: "Columns, literals, operators, string and null predicates, and the aggregate functions."
-icon: function
-keywords: ["expressions", "col", "lit", "operators", "aggregate", "sql functions"]
----
-
-Expressions are the values a DataFrame operation takes: what to project, what to
-filter on, what to group by. They are built from column references and literals,
-combined with operators, and evaluated by the server.
-
-```python
-from lancedb import col, lit, func
-from lancedb import sql_functions as F
-```
-
-## Building blocks {#building-blocks}
-
-| | |
-| --- | --- |
-| `col(name)` | reference a column by name |
-| `lit(value)` | a literal — `bool`, `int`, `float`, `str`, `bytes`, `date`, `datetime`, `Decimal` |
-| `func(name, *args)` | call a server-side function by name |
-
-`func()` is the escape hatch. Anything the engine exposes that has no Python
-method here can be reached by name — at the cost of being checked only when the
-query runs.
-
-Where an operation accepts an expression, a bare Python value is coerced to a
-literal and a bare string to a column reference. `col("age") > 18` needs no
-`lit()`.
-
-## Operators {#operators}
-
-| Category | Operators |
-| --- | --- |
-| Comparison | `==` `!=` `<` `<=` `>` `>=` |
-| Boolean | `&` (and) · `\|` (or) · `~` (not) |
-| Arithmetic | `+` `-` `*` `/` |
-
-All of them work with the expression on either side, so `1 + col("x")` is as
-valid as `col("x") + 1`.
-
-
-Use `&`, `|` and `~` — **not** `and`, `or` and `not`. Python cannot overload the
-keywords, so they will silently evaluate the expression object's truthiness
-instead of building a predicate.
-
-Parenthesise them. `&` and `|` bind more tightly than comparisons, so
-
-```python
-col("a") > 1 & col("b") < 2 # wrong: parses as col("a") > (1 & col("b")) < 2
-(col("a") > 1) & (col("b") < 2) # right
-```
-
-
-## Methods on an expression {#methods}
-
-| Method | Result |
-| --- | --- |
-| `lower()` | lowercase a string |
-| `upper()` | uppercase a string |
-| `contains(substr)` | true when the string contains `substr` |
-| `isin(values)` | true when the value is in `values` |
-| `is_null()` | true when the value is null |
-| `is_not_null()` | true when the value is not null |
-| `alias(name)` | name the result, for projection |
-| `sort(ascending=True, nulls_first=None)` | a sort key — see below |
-
-```python
-df.filter(col("title").lower().contains("puppy"))
-df.filter(col("lang").isin(["en", "fr"]))
-df.select(col("score").alias("relevance"))
-```
-
-## Sort keys {#sort}
-
-`expr.sort()` produces a `SortExpr`, which carries the direction and the null
-placement.
-
-| | Default |
-| --- | --- |
-| `ascending` | `True` |
-| `nulls_first` | the opposite of `ascending` |
-
-That default is DataFusion's convention and is worth reading twice: an ascending
-sort puts nulls **last**, and a descending sort puts them **first**. Set
-`nulls_first` explicitly when it matters.
-
-```python
-df.sort(col("score").sort(ascending=False)) # desc, nulls first
-df.sort(col("score").sort(ascending=False, nulls_first=False)) # desc, nulls last
-```
-
-Passing a bare expression to `sort()` is equivalent to `ascending=True,
-nulls_first=False`.
-
-## Aggregates {#aggregates}
-
-From `lancedb.sql_functions`:
-
-| Function | Result |
-| --- | --- |
-| `sum(expr)` | sum of non-null values |
-| `avg(expr)` | mean of non-null values |
-| `min(expr)` | minimum non-null value |
-| `max(expr)` | maximum non-null value |
-| `count(expr)` | number of non-null values |
-
-All five ignore nulls, including `count`. To count rows rather than values,
-count a column that cannot be null.
-
-```python
-from lancedb import sql_functions as F
-
-df.aggregate("lang", [F.count(col("id")), F.avg(col("score"))])
-```
-
-These five have Python wrappers. Every other function in the
-[DataFusion set](/sql/functions) — scalar, aggregate and window — is reachable
-through `func(name, ...)`, unchecked until execution.
-
-## Naming results {#naming}
-
-An aggregate or derived column takes a generated name unless you give it one.
-`alias()` on the expression, or the keyword in `with_columns()`, sets it:
-
-```python
-df.aggregate("lang", F.avg(col("score")).alias("mean_score"))
-df.with_columns(mean_score=F.avg(col("score")))
-```
-
-Worth doing on anything downstream code refers to by name — a generated name is
-not part of the API and may change.
diff --git a/docs/web/sql/functions.mdx b/docs/web/sql/functions.mdx
index 701006d2c..cf947b014 100644
--- a/docs/web/sql/functions.mdx
+++ b/docs/web/sql/functions.mdx
@@ -115,19 +115,9 @@ A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves
like a normal column — order by it, filter on it, project it — and exists only
in a relation that came from `fts()`.
-## From the DataFrame API {#from-dataframe}
+## Reaching a function LanceDB does not name {#escape-hatch}
-The [DataFrame API](/sql/dataframe) wraps five aggregates as Python functions —
-`sum`, `avg`, `min`, `max`, `count`. Everything else in the DataFusion reference
-is reachable through `func(name, *args)`:
-
-```python
-from lancedb import col, func
-
-df.select(func("date_trunc", "month", col("created_at")))
-df.filter(func("regexp_like", col("title"), "^intro"))
-```
-
-`func()` is not checked when you build the plan — an unknown name or a wrong
-argument count surfaces when the query runs on the server. That is the trade for
-reaching the whole function set without a wrapper per function.
+Nothing here wraps or restricts the DataFusion set — a query reaches any
+function in the reference above by writing it. If a name is not recognised,
+check it against the reference for the version this page pins rather than
+assuming it is unsupported.
diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx
index 3a5684b4e..c769b52c1 100644
--- a/docs/web/sql/index.mdx
+++ b/docs/web/sql/index.mdx
@@ -1,77 +1,49 @@
---
-title: "SQL and DataFrame manual"
+title: "SQL manual"
sidebarTitle: "Overview"
-description: "Two ways to express an analytical query over the lakehouse, one execution engine underneath."
+description: "Analytical SQL over the lakehouse, served through Arrow Flight SQL."
icon: table-list
-keywords: ["sql", "dataframe", "substrait", "flight sql", "query", "analytics"]
+keywords: ["sql", "flight sql", "query", "analytics", "datafusion", "fts"]
---
-LanceDB answers analytical queries two ways. **SQL** goes over an Arrow Flight
-SQL endpoint, so any Flight SQL client can reach it. The **DataFrame API**
-builds the same kind of plan in Python, method by method, and submits it as
-[Substrait](https://substrait.io).
+LanceDB answers analytical queries in SQL, served over an
+[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
+endpoint. Any Flight SQL client can reach it, and results come back as Arrow.
-They are not layered on each other — neither is sugar for the other. Both
-produce a logical plan that the server executes, which is why they cost the same
-and return the same Arrow data.
-
-
-The DataFrame API is new. Until it ships, the reference below describes the
-merged surface rather than a released one; check the version you are running.
-
-
-## Choosing between them {#choosing}
-
-| | SQL | DataFrame |
-| --- | --- | --- |
-| Written as | a query string | Python method calls |
-| Reaches the server via | Arrow Flight SQL | Substrait |
-| Composable in host code | by string building | by value |
-| Available from | any Flight SQL client | the Python client |
-| Checked | at the server | partly at build time |
-
-Reach for **SQL** when the query is written by a person, pasted from a notebook,
-or produced by a tool that already speaks Flight SQL. Reach for the **DataFrame
-API** when the query is assembled by code — a filter that depends on user input,
-a projection that varies by caller — because building a plan out of values
-avoids the quoting and injection problems of building one out of strings.
+The dialect is [DataFusion's](/sql/functions), so the syntax and functions are
+the ones that reference documents. What this manual covers is the parts that are
+LanceDB's: how a query reaches the engine, how Lance columns appear to it, and
+the extensions a lakehouse over multimodal data needs.
## Where a query runs {#execution}
-Both forms execute on the server.
+Queries execute on the server. The client sends a statement and receives Arrow
+record batches — nothing is converted to pandas or Python objects unless you ask
+for it.
-A DataFrame is **lazy**: every method returns a new plan and nothing runs until
-`execute()`. A local connection can build and serialize a plan — useful for
-tests, and for inspecting what a query will do — but cannot execute one.
-`db.table(...)` on a local connection gives you a plan you can print or convert
-to Substrait; submitting it needs a remote connection.
+Because results are Arrow the whole way, a query that feeds training or
+downstream processing does not pay a serialization cost to get there.
-```python
-df = db.table("documents").filter(col("score") > 0.5).limit(10)
-df.to_substrait() # works anywhere
-df.execute() # needs a remote connection
-```
+## The dialect {#dialect}
-That split is worth knowing before you write tests: plan construction is
-testable locally, execution is not.
+Ordinary analytical SQL — `SELECT`, `WHERE`, `GROUP BY`, `ORDER BY`, `LIMIT`,
+joins, CTEs and window functions — over Arrow-typed columns.
-## Query lifecycle {#lifecycle}
+Two things differ from a warehouse you may be used to, and both come from the
+storage being Lance rather than something SQL-shaped:
-`execute()` returns an Arrow `RecordBatchReader` and blocks. `execute_async()`
-returns a server-side query handle instead, for queries long enough that holding
-a reader open is the wrong shape.
+- **Columns are Arrow types**, not a SQL type system mapped onto storage. A
+ vector column is a `FixedSizeList` and behaves like one.
+- **Row metadata is available as columns** — `_rowid`, `_rowaddr` — but only
+ when asked for, so they stay out of `SELECT *`.
-Results are Arrow in both cases. Nothing converts to pandas or Python objects
-unless you ask.
+Both are covered in [Functions and types](/sql/functions).
-## LanceDB's SQL extensions {#extensions}
+## Full-text search {#fts}
-The dialect is [DataFusion's](/sql/functions) — `SELECT`, `WHERE`, `GROUP BY`,
-`ORDER BY`, `LIMIT`, joins, CTEs and window functions — plus what a lakehouse
-over multimodal data needs.
-
-**Full-text search** is a table function rather than a predicate, because it
-produces a relevance score alongside the matching rows:
+Full-text search is a **table function** rather than a predicate, because it
+produces relevance alongside the matching rows and a `WHERE` clause has nowhere
+to put that:
```sql
SELECT id, text, _score
@@ -80,27 +52,19 @@ ORDER BY _score DESC
LIMIT 5
```
-`fts()` takes a table name and a query — either a bare string or a JSON query
-object for fuzzy, phrase, boolean, boost and multi-match forms. `_score` is a
-pseudo-column carrying relevance, available for ordering and filtering like any
-other column.
+`fts()` takes a table name and a query — a bare string, or a JSON query object
+for fuzzy, phrase, boolean, boost and multi-match forms. `_score` carries
+relevance and behaves like any other column.
-Because it is a table function, full-text search composes with the rest of a
-query: join its results, filter them, aggregate over them.
+Being a relation is what makes it compose: join its result, filter it, aggregate
+over it.
## Reference {#reference}
-
-
- Every relational operation, with signatures and semantics.
-
-
- Columns, literals, operators, and the aggregate functions.
-
-
- The DataFusion dialect, and what LanceDB adds to it.
-
-
+
+ The DataFusion dialect and version, how Lance columns map onto it, and what
+ LanceDB adds.
+
## What this manual does not cover {#scope}