From 96bfdcd93770369df75eea55a86783f38debf248 Mon Sep 17 00:00:00 2001 From: Jack Ye Date: Wed, 9 Sep 2026 20:49:50 -0700 Subject: [PATCH] docs: drop the DataFrame API from the SQL manual MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DataFrame support is not going ahead, so the pages documenting it come out: the relational operations and the expression system, both of which described only that API. The manual was framed around having two query surfaces and choosing between them. With one, that framing is gone — the overview now leads with SQL over Arrow Flight SQL and what the dialect is, rather than with a comparison to something that will not exist. The function reference keeps everything that was about SQL: the DataFusion version, how Lance columns map onto its type system, the pseudo-columns, blob handling, and `fts()`. Only the section on reaching functions through the Python wrapper is replaced, since the wrapper is going too. Mentions of pandas and polars DataFrames elsewhere in the documentation are unrelated and untouched. --- docs/web/docs.json | 4 +- docs/web/sql/dataframe.mdx | 157 ----------------------------------- docs/web/sql/expressions.mdx | 135 ------------------------------ docs/web/sql/functions.mdx | 20 ++--- docs/web/sql/index.mdx | 112 +++++++++---------------- 5 files changed, 44 insertions(+), 384 deletions(-) delete mode 100644 docs/web/sql/dataframe.mdx delete mode 100644 docs/web/sql/expressions.mdx diff --git a/docs/web/docs.json b/docs/web/docs.json index ca59b1f40..0293bdb36 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -150,11 +150,9 @@ ] }, { - "group": "SQL and DataFrame", + "group": "SQL", "pages": [ "sql/index", - "sql/dataframe", - "sql/expressions", "sql/functions" ] }, diff --git a/docs/web/sql/dataframe.mdx b/docs/web/sql/dataframe.mdx deleted file mode 100644 index 4f34c0677..000000000 --- a/docs/web/sql/dataframe.mdx +++ /dev/null @@ -1,157 +0,0 @@ ---- -title: "DataFrame operations" -sidebarTitle: "DataFrame operations" -description: "Every relational operation on a LanceDB DataFrame, with signatures and semantics." -icon: table-list -keywords: ["dataframe", "select", "filter", "join", "aggregate", "substrait"] ---- - -A DataFrame is a lazy logical plan. Every operation returns a new plan; nothing -executes until [`execute()`](#execution). - -Obtain one from a connection: - -```python -import lancedb -from lancedb import col, lit -from lancedb import sql_functions as F - -db = lancedb.connect("db://your-database", api_key="...") -df = db.table("documents") -``` - -`db.table(name, *, namespace_path=None)` scans a table. It opens the table to -read its schema, so the plan knows its columns before anything runs. - -## Projection {#projection} - -| Operation | Effect | -| --- | --- | -| `select(*expressions)` | Project columns or expressions. At least one required. | -| `with_column(name, expression)` | Add one derived column. | -| `with_columns(**expressions)` | Add several, named by keyword. | -| `drop(*columns)` | Remove columns by name. | -| `with_column_renamed(old, new)` | Rename one column. | - -```python -df.select("id", col("score") * 2) -df.with_columns(doubled=col("score") * 2, flag=col("score") > 0.5) -``` - -A bare string is a column reference, so `select("id")` and `select(col("id"))` -mean the same thing. Use `col()` when you need methods or operators on it. - -## Filtering {#filtering} - -`filter(*predicates)` keeps rows matching **every** predicate — multiple -arguments are combined with `AND`: - -```python -df.filter(col("score") > 0.5, col("lang") == "en") -df.filter((col("score") > 0.5) & (col("lang") == "en")) # equivalent -``` - -At least one predicate is required. For `OR`, combine with `|` inside a single -argument. - - -Python's `and` and `or` cannot be overloaded and will **not** do what you want on -expressions. Use `&`, `|` and `~`, and parenthesise them — they bind more -tightly than comparison operators, so `col("a") > 1 & col("b") < 2` parses as -`col("a") > (1 & col("b")) < 2`. - - -## Aggregation {#aggregation} - -`aggregate(group_by, aggregates)` groups rows and computes aggregates over each -group. - -```python -df.aggregate("lang", F.count(col("id"))) -df.aggregate(["lang", "source"], [F.avg(col("score")), F.max(col("score"))]) -df.aggregate(None, F.count(col("id"))) # whole relation, no grouping -``` - -Both arguments accept a single value or an iterable. `group_by=None` aggregates -over the whole relation. The available aggregates are in -[Expressions](/sql/expressions#aggregates). - -## Ordering and slicing {#ordering} - -| Operation | Effect | -| --- | --- | -| `sort(*expressions)` | Order rows. | -| `limit(count, offset=0)` | Take `count` rows after skipping `offset`. | -| `distinct()` | Remove duplicate rows. | - -A bare expression sorts **ascending, nulls last**. For other orders use a -`SortExpr`, which follows DataFusion's convention: when `nulls_first` is not -given it defaults to the opposite of `ascending`, so descending sorts put nulls -first. - -```python -df.sort(col("score")) # ascending, nulls last -df.sort(col("score").sort(ascending=False)) -``` - -## Joins {#joins} - -```python -df.join(other, on="id") -df.join(other, left_on="doc_id", right_on="id", how="left") -df.join(other, on=["tenant", "id"], how="inner") -``` - -Pass either `on` (same key names on both sides) or **both** `left_on` and -`right_on`. Passing `on` together with either raises, as does passing neither. -Keys may be a single name or a sequence. - -`how` defaults to `"inner"`. - - -Both plans must come from the **same connection** and the **same default -namespace**; otherwise the join raises before anything is submitted. The same -applies to set operations. - -Use `alias(name)` to give a relation a name before joining it to itself. - - -## Set operations {#set-operations} - -| Operation | Effect | -| --- | --- | -| `union(other, distinct=False)` | Rows from both. | -| `intersect(other, distinct=False)` | Rows present in both. | -| `except_all(other, distinct=False)` | Rows in this plan and not the other. | - -All three default to **keeping duplicates**. Pass `distinct=True` for set -semantics. Note this is the opposite default to SQL's bare `UNION`, which -deduplicates — `union()` here corresponds to `UNION ALL`. - -## Inspecting a plan {#inspecting} - -| | | -| --- | --- | -| `schema` | the plan's output schema, as a `pyarrow.Schema` | -| `to_substrait()` | the logical plan as a Substrait protobuf | -| `repr(df)` | a readable rendering of the plan | - -`schema` and `to_substrait()` work on a local connection, so a plan can be -asserted against in tests without a server. - -## Execution {#execution} - -| | Returns | -| --- | --- | -| `execute()` | a blocking Arrow `RecordBatchReader` | -| `execute_async()` | a server-side query handle | - -On `AsyncConnection` both are coroutines and `execute()` yields an async reader. - -Execution requires a remote connection. A local one can build and serialize a -plan but not run it. - -```python -reader = df.execute() -table = reader.read_all() -``` diff --git a/docs/web/sql/expressions.mdx b/docs/web/sql/expressions.mdx deleted file mode 100644 index 856a6bb50..000000000 --- a/docs/web/sql/expressions.mdx +++ /dev/null @@ -1,135 +0,0 @@ ---- -title: "Expressions" -sidebarTitle: "Expressions" -description: "Columns, literals, operators, string and null predicates, and the aggregate functions." -icon: function -keywords: ["expressions", "col", "lit", "operators", "aggregate", "sql functions"] ---- - -Expressions are the values a DataFrame operation takes: what to project, what to -filter on, what to group by. They are built from column references and literals, -combined with operators, and evaluated by the server. - -```python -from lancedb import col, lit, func -from lancedb import sql_functions as F -``` - -## Building blocks {#building-blocks} - -| | | -| --- | --- | -| `col(name)` | reference a column by name | -| `lit(value)` | a literal — `bool`, `int`, `float`, `str`, `bytes`, `date`, `datetime`, `Decimal` | -| `func(name, *args)` | call a server-side function by name | - -`func()` is the escape hatch. Anything the engine exposes that has no Python -method here can be reached by name — at the cost of being checked only when the -query runs. - -Where an operation accepts an expression, a bare Python value is coerced to a -literal and a bare string to a column reference. `col("age") > 18` needs no -`lit()`. - -## Operators {#operators} - -| Category | Operators | -| --- | --- | -| Comparison | `==` `!=` `<` `<=` `>` `>=` | -| Boolean | `&` (and) · `\|` (or) · `~` (not) | -| Arithmetic | `+` `-` `*` `/` | - -All of them work with the expression on either side, so `1 + col("x")` is as -valid as `col("x") + 1`. - - -Use `&`, `|` and `~` — **not** `and`, `or` and `not`. Python cannot overload the -keywords, so they will silently evaluate the expression object's truthiness -instead of building a predicate. - -Parenthesise them. `&` and `|` bind more tightly than comparisons, so - -```python -col("a") > 1 & col("b") < 2 # wrong: parses as col("a") > (1 & col("b")) < 2 -(col("a") > 1) & (col("b") < 2) # right -``` - - -## Methods on an expression {#methods} - -| Method | Result | -| --- | --- | -| `lower()` | lowercase a string | -| `upper()` | uppercase a string | -| `contains(substr)` | true when the string contains `substr` | -| `isin(values)` | true when the value is in `values` | -| `is_null()` | true when the value is null | -| `is_not_null()` | true when the value is not null | -| `alias(name)` | name the result, for projection | -| `sort(ascending=True, nulls_first=None)` | a sort key — see below | - -```python -df.filter(col("title").lower().contains("puppy")) -df.filter(col("lang").isin(["en", "fr"])) -df.select(col("score").alias("relevance")) -``` - -## Sort keys {#sort} - -`expr.sort()` produces a `SortExpr`, which carries the direction and the null -placement. - -| | Default | -| --- | --- | -| `ascending` | `True` | -| `nulls_first` | the opposite of `ascending` | - -That default is DataFusion's convention and is worth reading twice: an ascending -sort puts nulls **last**, and a descending sort puts them **first**. Set -`nulls_first` explicitly when it matters. - -```python -df.sort(col("score").sort(ascending=False)) # desc, nulls first -df.sort(col("score").sort(ascending=False, nulls_first=False)) # desc, nulls last -``` - -Passing a bare expression to `sort()` is equivalent to `ascending=True, -nulls_first=False`. - -## Aggregates {#aggregates} - -From `lancedb.sql_functions`: - -| Function | Result | -| --- | --- | -| `sum(expr)` | sum of non-null values | -| `avg(expr)` | mean of non-null values | -| `min(expr)` | minimum non-null value | -| `max(expr)` | maximum non-null value | -| `count(expr)` | number of non-null values | - -All five ignore nulls, including `count`. To count rows rather than values, -count a column that cannot be null. - -```python -from lancedb import sql_functions as F - -df.aggregate("lang", [F.count(col("id")), F.avg(col("score"))]) -``` - -These five have Python wrappers. Every other function in the -[DataFusion set](/sql/functions) — scalar, aggregate and window — is reachable -through `func(name, ...)`, unchecked until execution. - -## Naming results {#naming} - -An aggregate or derived column takes a generated name unless you give it one. -`alias()` on the expression, or the keyword in `with_columns()`, sets it: - -```python -df.aggregate("lang", F.avg(col("score")).alias("mean_score")) -df.with_columns(mean_score=F.avg(col("score"))) -``` - -Worth doing on anything downstream code refers to by name — a generated name is -not part of the API and may change. diff --git a/docs/web/sql/functions.mdx b/docs/web/sql/functions.mdx index 701006d2c..cf947b014 100644 --- a/docs/web/sql/functions.mdx +++ b/docs/web/sql/functions.mdx @@ -115,19 +115,9 @@ A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves like a normal column — order by it, filter on it, project it — and exists only in a relation that came from `fts()`. -## From the DataFrame API {#from-dataframe} +## Reaching a function LanceDB does not name {#escape-hatch} -The [DataFrame API](/sql/dataframe) wraps five aggregates as Python functions — -`sum`, `avg`, `min`, `max`, `count`. Everything else in the DataFusion reference -is reachable through `func(name, *args)`: - -```python -from lancedb import col, func - -df.select(func("date_trunc", "month", col("created_at"))) -df.filter(func("regexp_like", col("title"), "^intro")) -``` - -`func()` is not checked when you build the plan — an unknown name or a wrong -argument count surfaces when the query runs on the server. That is the trade for -reaching the whole function set without a wrapper per function. +Nothing here wraps or restricts the DataFusion set — a query reaches any +function in the reference above by writing it. If a name is not recognised, +check it against the reference for the version this page pins rather than +assuming it is unsupported. diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx index 3a5684b4e..c769b52c1 100644 --- a/docs/web/sql/index.mdx +++ b/docs/web/sql/index.mdx @@ -1,77 +1,49 @@ --- -title: "SQL and DataFrame manual" +title: "SQL manual" sidebarTitle: "Overview" -description: "Two ways to express an analytical query over the lakehouse, one execution engine underneath." +description: "Analytical SQL over the lakehouse, served through Arrow Flight SQL." icon: table-list -keywords: ["sql", "dataframe", "substrait", "flight sql", "query", "analytics"] +keywords: ["sql", "flight sql", "query", "analytics", "datafusion", "fts"] --- -LanceDB answers analytical queries two ways. **SQL** goes over an Arrow Flight -SQL endpoint, so any Flight SQL client can reach it. The **DataFrame API** -builds the same kind of plan in Python, method by method, and submits it as -[Substrait](https://substrait.io). +LanceDB answers analytical queries in SQL, served over an +[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) +endpoint. Any Flight SQL client can reach it, and results come back as Arrow. -They are not layered on each other — neither is sugar for the other. Both -produce a logical plan that the server executes, which is why they cost the same -and return the same Arrow data. - - -The DataFrame API is new. Until it ships, the reference below describes the -merged surface rather than a released one; check the version you are running. - - -## Choosing between them {#choosing} - -| | SQL | DataFrame | -| --- | --- | --- | -| Written as | a query string | Python method calls | -| Reaches the server via | Arrow Flight SQL | Substrait | -| Composable in host code | by string building | by value | -| Available from | any Flight SQL client | the Python client | -| Checked | at the server | partly at build time | - -Reach for **SQL** when the query is written by a person, pasted from a notebook, -or produced by a tool that already speaks Flight SQL. Reach for the **DataFrame -API** when the query is assembled by code — a filter that depends on user input, -a projection that varies by caller — because building a plan out of values -avoids the quoting and injection problems of building one out of strings. +The dialect is [DataFusion's](/sql/functions), so the syntax and functions are +the ones that reference documents. What this manual covers is the parts that are +LanceDB's: how a query reaches the engine, how Lance columns appear to it, and +the extensions a lakehouse over multimodal data needs. ## Where a query runs {#execution} -Both forms execute on the server. +Queries execute on the server. The client sends a statement and receives Arrow +record batches — nothing is converted to pandas or Python objects unless you ask +for it. -A DataFrame is **lazy**: every method returns a new plan and nothing runs until -`execute()`. A local connection can build and serialize a plan — useful for -tests, and for inspecting what a query will do — but cannot execute one. -`db.table(...)` on a local connection gives you a plan you can print or convert -to Substrait; submitting it needs a remote connection. +Because results are Arrow the whole way, a query that feeds training or +downstream processing does not pay a serialization cost to get there. -```python -df = db.table("documents").filter(col("score") > 0.5).limit(10) -df.to_substrait() # works anywhere -df.execute() # needs a remote connection -``` +## The dialect {#dialect} -That split is worth knowing before you write tests: plan construction is -testable locally, execution is not. +Ordinary analytical SQL — `SELECT`, `WHERE`, `GROUP BY`, `ORDER BY`, `LIMIT`, +joins, CTEs and window functions — over Arrow-typed columns. -## Query lifecycle {#lifecycle} +Two things differ from a warehouse you may be used to, and both come from the +storage being Lance rather than something SQL-shaped: -`execute()` returns an Arrow `RecordBatchReader` and blocks. `execute_async()` -returns a server-side query handle instead, for queries long enough that holding -a reader open is the wrong shape. +- **Columns are Arrow types**, not a SQL type system mapped onto storage. A + vector column is a `FixedSizeList` and behaves like one. +- **Row metadata is available as columns** — `_rowid`, `_rowaddr` — but only + when asked for, so they stay out of `SELECT *`. -Results are Arrow in both cases. Nothing converts to pandas or Python objects -unless you ask. +Both are covered in [Functions and types](/sql/functions). -## LanceDB's SQL extensions {#extensions} +## Full-text search {#fts} -The dialect is [DataFusion's](/sql/functions) — `SELECT`, `WHERE`, `GROUP BY`, -`ORDER BY`, `LIMIT`, joins, CTEs and window functions — plus what a lakehouse -over multimodal data needs. - -**Full-text search** is a table function rather than a predicate, because it -produces a relevance score alongside the matching rows: +Full-text search is a **table function** rather than a predicate, because it +produces relevance alongside the matching rows and a `WHERE` clause has nowhere +to put that: ```sql SELECT id, text, _score @@ -80,27 +52,19 @@ ORDER BY _score DESC LIMIT 5 ``` -`fts()` takes a table name and a query — either a bare string or a JSON query -object for fuzzy, phrase, boolean, boost and multi-match forms. `_score` is a -pseudo-column carrying relevance, available for ordering and filtering like any -other column. +`fts()` takes a table name and a query — a bare string, or a JSON query object +for fuzzy, phrase, boolean, boost and multi-match forms. `_score` carries +relevance and behaves like any other column. -Because it is a table function, full-text search composes with the rest of a -query: join its results, filter them, aggregate over them. +Being a relation is what makes it compose: join its result, filter it, aggregate +over it. ## Reference {#reference} - - - Every relational operation, with signatures and semantics. - - - Columns, literals, operators, and the aggregate functions. - - - The DataFusion dialect, and what LanceDB adds to it. - - + + The DataFusion dialect and version, how Lance columns map onto it, and what + LanceDB adds. + ## What this manual does not cover {#scope}