diff --git a/docs/web/docs.json b/docs/web/docs.json index 2490ff5b1..ca59b1f40 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -149,6 +149,15 @@ "feature-engineering/functions" ] }, + { + "group": "SQL and DataFrame", + "pages": [ + "sql/index", + "sql/dataframe", + "sql/expressions", + "sql/functions" + ] + }, { "group": "Support", "pages": [ diff --git a/docs/web/sql/dataframe.mdx b/docs/web/sql/dataframe.mdx new file mode 100644 index 000000000..4f34c0677 --- /dev/null +++ b/docs/web/sql/dataframe.mdx @@ -0,0 +1,157 @@ +--- +title: "DataFrame operations" +sidebarTitle: "DataFrame operations" +description: "Every relational operation on a LanceDB DataFrame, with signatures and semantics." +icon: table-list +keywords: ["dataframe", "select", "filter", "join", "aggregate", "substrait"] +--- + +A DataFrame is a lazy logical plan. Every operation returns a new plan; nothing +executes until [`execute()`](#execution). + +Obtain one from a connection: + +```python +import lancedb +from lancedb import col, lit +from lancedb import sql_functions as F + +db = lancedb.connect("db://your-database", api_key="...") +df = db.table("documents") +``` + +`db.table(name, *, namespace_path=None)` scans a table. It opens the table to +read its schema, so the plan knows its columns before anything runs. + +## Projection {#projection} + +| Operation | Effect | +| --- | --- | +| `select(*expressions)` | Project columns or expressions. At least one required. | +| `with_column(name, expression)` | Add one derived column. | +| `with_columns(**expressions)` | Add several, named by keyword. | +| `drop(*columns)` | Remove columns by name. | +| `with_column_renamed(old, new)` | Rename one column. | + +```python +df.select("id", col("score") * 2) +df.with_columns(doubled=col("score") * 2, flag=col("score") > 0.5) +``` + +A bare string is a column reference, so `select("id")` and `select(col("id"))` +mean the same thing. Use `col()` when you need methods or operators on it. + +## Filtering {#filtering} + +`filter(*predicates)` keeps rows matching **every** predicate — multiple +arguments are combined with `AND`: + +```python +df.filter(col("score") > 0.5, col("lang") == "en") +df.filter((col("score") > 0.5) & (col("lang") == "en")) # equivalent +``` + +At least one predicate is required. For `OR`, combine with `|` inside a single +argument. + + +Python's `and` and `or` cannot be overloaded and will **not** do what you want on +expressions. Use `&`, `|` and `~`, and parenthesise them — they bind more +tightly than comparison operators, so `col("a") > 1 & col("b") < 2` parses as +`col("a") > (1 & col("b")) < 2`. + + +## Aggregation {#aggregation} + +`aggregate(group_by, aggregates)` groups rows and computes aggregates over each +group. + +```python +df.aggregate("lang", F.count(col("id"))) +df.aggregate(["lang", "source"], [F.avg(col("score")), F.max(col("score"))]) +df.aggregate(None, F.count(col("id"))) # whole relation, no grouping +``` + +Both arguments accept a single value or an iterable. `group_by=None` aggregates +over the whole relation. The available aggregates are in +[Expressions](/sql/expressions#aggregates). + +## Ordering and slicing {#ordering} + +| Operation | Effect | +| --- | --- | +| `sort(*expressions)` | Order rows. | +| `limit(count, offset=0)` | Take `count` rows after skipping `offset`. | +| `distinct()` | Remove duplicate rows. | + +A bare expression sorts **ascending, nulls last**. For other orders use a +`SortExpr`, which follows DataFusion's convention: when `nulls_first` is not +given it defaults to the opposite of `ascending`, so descending sorts put nulls +first. + +```python +df.sort(col("score")) # ascending, nulls last +df.sort(col("score").sort(ascending=False)) +``` + +## Joins {#joins} + +```python +df.join(other, on="id") +df.join(other, left_on="doc_id", right_on="id", how="left") +df.join(other, on=["tenant", "id"], how="inner") +``` + +Pass either `on` (same key names on both sides) or **both** `left_on` and +`right_on`. Passing `on` together with either raises, as does passing neither. +Keys may be a single name or a sequence. + +`how` defaults to `"inner"`. + + +Both plans must come from the **same connection** and the **same default +namespace**; otherwise the join raises before anything is submitted. The same +applies to set operations. + +Use `alias(name)` to give a relation a name before joining it to itself. + + +## Set operations {#set-operations} + +| Operation | Effect | +| --- | --- | +| `union(other, distinct=False)` | Rows from both. | +| `intersect(other, distinct=False)` | Rows present in both. | +| `except_all(other, distinct=False)` | Rows in this plan and not the other. | + +All three default to **keeping duplicates**. Pass `distinct=True` for set +semantics. Note this is the opposite default to SQL's bare `UNION`, which +deduplicates — `union()` here corresponds to `UNION ALL`. + +## Inspecting a plan {#inspecting} + +| | | +| --- | --- | +| `schema` | the plan's output schema, as a `pyarrow.Schema` | +| `to_substrait()` | the logical plan as a Substrait protobuf | +| `repr(df)` | a readable rendering of the plan | + +`schema` and `to_substrait()` work on a local connection, so a plan can be +asserted against in tests without a server. + +## Execution {#execution} + +| | Returns | +| --- | --- | +| `execute()` | a blocking Arrow `RecordBatchReader` | +| `execute_async()` | a server-side query handle | + +On `AsyncConnection` both are coroutines and `execute()` yields an async reader. + +Execution requires a remote connection. A local one can build and serialize a +plan but not run it. + +```python +reader = df.execute() +table = reader.read_all() +``` diff --git a/docs/web/sql/expressions.mdx b/docs/web/sql/expressions.mdx new file mode 100644 index 000000000..856a6bb50 --- /dev/null +++ b/docs/web/sql/expressions.mdx @@ -0,0 +1,135 @@ +--- +title: "Expressions" +sidebarTitle: "Expressions" +description: "Columns, literals, operators, string and null predicates, and the aggregate functions." +icon: function +keywords: ["expressions", "col", "lit", "operators", "aggregate", "sql functions"] +--- + +Expressions are the values a DataFrame operation takes: what to project, what to +filter on, what to group by. They are built from column references and literals, +combined with operators, and evaluated by the server. + +```python +from lancedb import col, lit, func +from lancedb import sql_functions as F +``` + +## Building blocks {#building-blocks} + +| | | +| --- | --- | +| `col(name)` | reference a column by name | +| `lit(value)` | a literal — `bool`, `int`, `float`, `str`, `bytes`, `date`, `datetime`, `Decimal` | +| `func(name, *args)` | call a server-side function by name | + +`func()` is the escape hatch. Anything the engine exposes that has no Python +method here can be reached by name — at the cost of being checked only when the +query runs. + +Where an operation accepts an expression, a bare Python value is coerced to a +literal and a bare string to a column reference. `col("age") > 18` needs no +`lit()`. + +## Operators {#operators} + +| Category | Operators | +| --- | --- | +| Comparison | `==` `!=` `<` `<=` `>` `>=` | +| Boolean | `&` (and) · `\|` (or) · `~` (not) | +| Arithmetic | `+` `-` `*` `/` | + +All of them work with the expression on either side, so `1 + col("x")` is as +valid as `col("x") + 1`. + + +Use `&`, `|` and `~` — **not** `and`, `or` and `not`. Python cannot overload the +keywords, so they will silently evaluate the expression object's truthiness +instead of building a predicate. + +Parenthesise them. `&` and `|` bind more tightly than comparisons, so + +```python +col("a") > 1 & col("b") < 2 # wrong: parses as col("a") > (1 & col("b")) < 2 +(col("a") > 1) & (col("b") < 2) # right +``` + + +## Methods on an expression {#methods} + +| Method | Result | +| --- | --- | +| `lower()` | lowercase a string | +| `upper()` | uppercase a string | +| `contains(substr)` | true when the string contains `substr` | +| `isin(values)` | true when the value is in `values` | +| `is_null()` | true when the value is null | +| `is_not_null()` | true when the value is not null | +| `alias(name)` | name the result, for projection | +| `sort(ascending=True, nulls_first=None)` | a sort key — see below | + +```python +df.filter(col("title").lower().contains("puppy")) +df.filter(col("lang").isin(["en", "fr"])) +df.select(col("score").alias("relevance")) +``` + +## Sort keys {#sort} + +`expr.sort()` produces a `SortExpr`, which carries the direction and the null +placement. + +| | Default | +| --- | --- | +| `ascending` | `True` | +| `nulls_first` | the opposite of `ascending` | + +That default is DataFusion's convention and is worth reading twice: an ascending +sort puts nulls **last**, and a descending sort puts them **first**. Set +`nulls_first` explicitly when it matters. + +```python +df.sort(col("score").sort(ascending=False)) # desc, nulls first +df.sort(col("score").sort(ascending=False, nulls_first=False)) # desc, nulls last +``` + +Passing a bare expression to `sort()` is equivalent to `ascending=True, +nulls_first=False`. + +## Aggregates {#aggregates} + +From `lancedb.sql_functions`: + +| Function | Result | +| --- | --- | +| `sum(expr)` | sum of non-null values | +| `avg(expr)` | mean of non-null values | +| `min(expr)` | minimum non-null value | +| `max(expr)` | maximum non-null value | +| `count(expr)` | number of non-null values | + +All five ignore nulls, including `count`. To count rows rather than values, +count a column that cannot be null. + +```python +from lancedb import sql_functions as F + +df.aggregate("lang", [F.count(col("id")), F.avg(col("score"))]) +``` + +These five have Python wrappers. Every other function in the +[DataFusion set](/sql/functions) — scalar, aggregate and window — is reachable +through `func(name, ...)`, unchecked until execution. + +## Naming results {#naming} + +An aggregate or derived column takes a generated name unless you give it one. +`alias()` on the expression, or the keyword in `with_columns()`, sets it: + +```python +df.aggregate("lang", F.avg(col("score")).alias("mean_score")) +df.with_columns(mean_score=F.avg(col("score"))) +``` + +Worth doing on anything downstream code refers to by name — a generated name is +not part of the API and may change. diff --git a/docs/web/sql/functions.mdx b/docs/web/sql/functions.mdx new file mode 100644 index 000000000..701006d2c --- /dev/null +++ b/docs/web/sql/functions.mdx @@ -0,0 +1,133 @@ +--- +title: "SQL functions and types" +sidebarTitle: "Functions and types" +description: "The dialect is DataFusion's. What that gives you, and what LanceDB adds on top." +icon: function +keywords: ["sql functions", "datafusion", "scalar functions", "aggregate", "data types", "fts"] +--- + +LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org), +currently **version 54**. The dialect, the built-in functions, and the type +system are DataFusion's, registered from its default set — so anything the +DataFusion reference documents for that version is available, and this page does +not reproduce it. + +What it does document is the version you are querying against, where the +authoritative lists are, and the parts that are LanceDB's rather than +DataFusion's. + +## Reference by category {#reference} + +| | DataFusion reference | +| --- | --- | +| Math, conditional, string, binary string, regular expression, time and date | [Scalar functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) | +| `count`, `sum`, `avg`, `min`, `max`, statistical, approximate | [Aggregate functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) | +| `row_number`, `rank`, `lag`, `lead`, and the rest of `OVER (...)` | [Window functions](https://datafusion.apache.org/user-guide/sql/window_functions.html) | +| Arrow-backed types, and how SQL type names map onto them | [Data types](https://datafusion.apache.org/user-guide/sql/data_types.html) | +| `SELECT`, joins, `GROUP BY`, `ORDER BY`, `LIMIT`, CTEs, set operations | [SELECT syntax](https://datafusion.apache.org/user-guide/sql/select.html) | + + +Function availability is version-specific: DataFusion adds functions in most +releases, so a function documented on `main` may not exist in 54. When a +function is not recognised, check it against the reference for the version above +before assuming a bug. + + +## Types are Arrow types {#types} + +There is no SQL type layer over Lance storage. The DataFusion table provider +returns the dataset's Arrow schema directly — `Schema::from(dataset.schema())` — +so a column is the same object to the query engine as it is to the format, and +DataFusion's [type reference](https://datafusion.apache.org/user-guide/sql/data_types.html) +describes it exactly. + +That identity is mostly invisible and occasionally load-bearing. A vector column +is a `FixedSizeList`: selectable, testable for nullness, joinable like +any other column — but arithmetic over it is list arithmetic, not scalar +arithmetic. When the operation is a similarity comparison, use vector search +rather than SQL. + +### Pseudo-columns {#pseudo-columns} + +Lance rows carry metadata the provider can expose as columns. They are appended +to the schema only when asked for, so they do not appear in `SELECT *`: + +| Column | Type | Meaning | +| --- | --- | --- | +| `_rowid` | `UInt64`, nullable | stable row identifier | +| `_rowaddr` | `UInt64`, nullable | physical row address | +| `_rowoffset` | `UInt64`, nullable | offset within the fragment | +| `_score` | `Float32` | relevance, from [`fts()`](#fts) | + +`_rowid` and `_rowaddr` are declared nullable, but not because a row lacks one — +their validity bitmap is reused as a selection vector internally. Do not read a +null there as meaningful. + +`_rowid` is the identifier [materialized views](/feature-engineering/materialized-views) +depend on, which is why a source table needs stable row ids enabled at creation. + +### Blob columns change shape {#blobs} + +A blob column can be read as its **binary contents** or as a **description** — +a pointer plus metadata, without fetching the bytes. That choice changes the +column's Arrow type, and therefore the schema a query sees. + +The default reads blobs as descriptions and other binary columns as binary. A +query that selects a blob column and expects bytes will get a struct instead +unless the read is configured otherwise. Worth knowing before writing a +predicate against one. + +## What LanceDB adds {#extensions} + +### `fts(table, query)` {#fts} + +Full-text search is a **table function**, not a predicate — implemented as a +DataFusion `TableProvider` over the dataset, so its result is a relation. That is +what lets it return relevance alongside the matching rows, which a `WHERE` clause +has nowhere to put: + +```sql +SELECT id, text, _score +FROM fts('my_docs', 'puppy') +ORDER BY _score DESC +LIMIT 5 +``` + +The second argument is either a bare query string or a JSON query object for +fuzzy, phrase, boolean, boost and multi-match forms. + +Its schema is the dataset's, so every column of the table is available in the +result — projections push down into the scan rather than being applied +afterwards. Being a relation is also what makes it compose: join it, filter it, +aggregate over it. + +```sql +SELECT c.name, count(*) AS hits +FROM fts('my_docs', 'puppy') AS m +JOIN categories AS c ON c.id = m.category_id +GROUP BY c.name +ORDER BY hits DESC +``` + +### `_score` {#score} + +A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves +like a normal column — order by it, filter on it, project it — and exists only +in a relation that came from `fts()`. + +## From the DataFrame API {#from-dataframe} + +The [DataFrame API](/sql/dataframe) wraps five aggregates as Python functions — +`sum`, `avg`, `min`, `max`, `count`. Everything else in the DataFusion reference +is reachable through `func(name, *args)`: + +```python +from lancedb import col, func + +df.select(func("date_trunc", "month", col("created_at"))) +df.filter(func("regexp_like", col("title"), "^intro")) +``` + +`func()` is not checked when you build the plan — an unknown name or a wrong +argument count surfaces when the query runs on the server. That is the trade for +reaching the whole function set without a wrapper per function. diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx new file mode 100644 index 000000000..3a5684b4e --- /dev/null +++ b/docs/web/sql/index.mdx @@ -0,0 +1,108 @@ +--- +title: "SQL and DataFrame manual" +sidebarTitle: "Overview" +description: "Two ways to express an analytical query over the lakehouse, one execution engine underneath." +icon: table-list +keywords: ["sql", "dataframe", "substrait", "flight sql", "query", "analytics"] +--- + +LanceDB answers analytical queries two ways. **SQL** goes over an Arrow Flight +SQL endpoint, so any Flight SQL client can reach it. The **DataFrame API** +builds the same kind of plan in Python, method by method, and submits it as +[Substrait](https://substrait.io). + +They are not layered on each other — neither is sugar for the other. Both +produce a logical plan that the server executes, which is why they cost the same +and return the same Arrow data. + + +The DataFrame API is new. Until it ships, the reference below describes the +merged surface rather than a released one; check the version you are running. + + +## Choosing between them {#choosing} + +| | SQL | DataFrame | +| --- | --- | --- | +| Written as | a query string | Python method calls | +| Reaches the server via | Arrow Flight SQL | Substrait | +| Composable in host code | by string building | by value | +| Available from | any Flight SQL client | the Python client | +| Checked | at the server | partly at build time | + +Reach for **SQL** when the query is written by a person, pasted from a notebook, +or produced by a tool that already speaks Flight SQL. Reach for the **DataFrame +API** when the query is assembled by code — a filter that depends on user input, +a projection that varies by caller — because building a plan out of values +avoids the quoting and injection problems of building one out of strings. + +## Where a query runs {#execution} + +Both forms execute on the server. + +A DataFrame is **lazy**: every method returns a new plan and nothing runs until +`execute()`. A local connection can build and serialize a plan — useful for +tests, and for inspecting what a query will do — but cannot execute one. +`db.table(...)` on a local connection gives you a plan you can print or convert +to Substrait; submitting it needs a remote connection. + +```python +df = db.table("documents").filter(col("score") > 0.5).limit(10) +df.to_substrait() # works anywhere +df.execute() # needs a remote connection +``` + +That split is worth knowing before you write tests: plan construction is +testable locally, execution is not. + +## Query lifecycle {#lifecycle} + +`execute()` returns an Arrow `RecordBatchReader` and blocks. `execute_async()` +returns a server-side query handle instead, for queries long enough that holding +a reader open is the wrong shape. + +Results are Arrow in both cases. Nothing converts to pandas or Python objects +unless you ask. + +## LanceDB's SQL extensions {#extensions} + +The dialect is [DataFusion's](/sql/functions) — `SELECT`, `WHERE`, `GROUP BY`, +`ORDER BY`, `LIMIT`, joins, CTEs and window functions — plus what a lakehouse +over multimodal data needs. + +**Full-text search** is a table function rather than a predicate, because it +produces a relevance score alongside the matching rows: + +```sql +SELECT id, text, _score +FROM fts('my_docs', 'puppy') +ORDER BY _score DESC +LIMIT 5 +``` + +`fts()` takes a table name and a query — either a bare string or a JSON query +object for fuzzy, phrase, boolean, boost and multi-match forms. `_score` is a +pseudo-column carrying relevance, available for ordering and filtering like any +other column. + +Because it is a table function, full-text search composes with the rest of a +query: join its results, filter them, aggregate over them. + +## Reference {#reference} + + + + Every relational operation, with signatures and semantics. + + + Columns, literals, operators, and the aggregate functions. + + + The DataFusion dialect, and what LanceDB adds to it. + + + +## What this manual does not cover {#scope} + +Connecting a Flight SQL client — drivers, authentication, the endpoint address — +is deployment-specific and documented with the deployment.