rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-17 17:22:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: bring the open-source documentation into this repository (#4122)

Browse Source 
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).

This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.

Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.

**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.

**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.

Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.

**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.

The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.

---------

Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
This commit is contained in:
Jack Ye
2026-10-07 11:15:00 -07:00
committed by GitHub
co-authored by Brendan Clement
parent de005f1d61
commit 47b7babc08
307 changed files with 51361 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/sql/dql/aggregation.mdx
+118
View File
@@ -0,0 +1,118 @@
---
title: "Aggregation"
sidebarTitle: "Aggregation"
description: "GROUP BY, HAVING, and the aggregate function set."
icon: layer-group
keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximate"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
Aggregation is one of the two main reasons to reach for SQL rather than a query
builder -- the SDKs project, filter and search, but they do not group. The other
is [joins](/sql/dql/join).
## Grouping {#grouping}
```sql
SELECT category, count(*) AS n, avg(score) AS mean_score
FROM documents
GROUP BY category
HAVING count(*) > 10
ORDER BY n DESC
```
`WHERE` filters rows before grouping; `HAVING` filters groups after. Putting a
row-level predicate in `HAVING` still gives the right answer but does more work,
because rows that could have been discarded early are grouped first.
Grouping by several expressions produces one row per distinct combination:
```sql
SELECT category, source, count(*) FROM documents GROUP BY category, source
```
## Aggregate without grouping {#whole-table}
Omitting `GROUP BY` aggregates the whole table into one row -- the usual way to
ask how big something is:
```sql
SELECT count(*), min(created_at), max(created_at) FROM documents
```
`count(*)` counts rows; `count(col)` counts non-null values, which is how you
measure how populated a [computed column](/feature-engineering/add-computed-column)
is before deciding to refresh it again.
## The function set {#functions}
Registered in DataFusion 54. Names below are what is actually available.
**General**
```text
count sum min max avg median
bit_and bit_or bit_xor bool_and bool_or
first_value last_value nth_value grouping
```
**Statistical**
```text
stddev stddev_pop stddev_samp
var var_pop var_samp
corr covar_pop covar_samp
regr_slope regr_intercept regr_count regr_r2
regr_avgx regr_avgy regr_sxx regr_syy regr_sxy
```
**Approximate**
```text
approx_distinct approx_median
approx_percentile_cont approx_percentile_cont_with_weight
percentile_cont
```
**Collecting**
```text
array_agg string_agg
```
The approximate functions are the ones worth reaching for on a large table.
`approx_distinct` answers "roughly how many distinct values" for a fraction of
what `count(DISTINCT ...)` costs, which matters most where it is most expensive
— a high-cardinality column.
`DISTINCT` works inside an aggregate: `count(DISTINCT category)`. `FILTER`
narrows one aggregate without narrowing the others:
```sql
SELECT
count(*) AS total,
count(*) FILTER (WHERE score > 0.9) AS confident
FROM documents
```
## Aggregating a sample {#sampling}
For an estimate over a very large table, aggregate a
[sample](/sql/dql/sampling) rather than the whole thing:
```sql
SELECT category, count(*) * 100 AS estimated
FROM documents TABLESAMPLE (1)
GROUP BY category
```
Scale the result by the inverse of the sample fraction, and treat it as an
estimate -- a 1% sample of a skewed column will miss rare categories entirely.
## What aggregation does not do {#not}
Aggregates run over rows the query already produced. They cannot combine with
[vector search](/search/vector-search) in one statement, because similarity
search has no SQL form. Run the search in an SDK and aggregate its result, or
materialize the search output first.
docs/web/sql/dql/explain.mdx
+52
View File
@@ -0,0 +1,52 @@
---
title: "Explain"
sidebarTitle: "Explain"
description: "EXPLAIN, EXPLAIN ANALYZE and EXPLAIN VERBOSE — reading and measuring a plan from SQL."
icon: map
keywords: ["explain", "explain analyze", "explain verbose", "query plan"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
```sql
EXPLAIN SELECT * FROM documents WHERE identifier < 1000
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
```
| Statement | What it returns |
| --- | --- |
| `EXPLAIN <query>` | Logical and physical plan, without executing the query. |
| `EXPLAIN ANALYZE <query>` | Executes the query and annotates each operator with runtime metrics (rows, timing). |
| `EXPLAIN VERBOSE <query>` | Adds intermediate optimizer plans on top of `EXPLAIN`. |
The result is a small relation with `plan_type` and `plan` columns, read
through the same client you use for regular queries -- see
[Connecting](/sql#connecting).
The operators are the same ones the SDK's
[`explain_plan` and `analyze_plan`](/search/explain) methods produce, so the
reasoning about index coverage and filter pushdown carries across unchanged:
<CardGroup cols={2}>
<Card title="What it shows" icon="map" href="/search/explain#inspect-a-query-plan">
Reading a plan: operators, pushdown, index usage.
</Card>
<Card title="What it measures" icon="gauge" href="/search/explain#analyze-a-query-plan">
Reading the metrics, and a worked before/after.
</Card>
</CardGroup>
## What to look for {#what-to-look-for}
| Sign | Means |
| --- | --- |
| `ScalarIndexQuery` | the predicate used a scalar index |
| `LanceScan` with high `bytes_read` | no index, or too many columns selected |
| `FilterExec` above a search | the filter ran after search rather than pushing into it |
| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used |
A `LanceScan` feeding a join is the common one worth acting on -- see
[Joins](/sql/dql/join#performance).
For a table's columns and types without a plan, see
[`DESCRIBE`](/sql/ddl/table#describe).
docs/web/sql/dql/fts.mdx
+148
View File
@@ -0,0 +1,148 @@
---
title: "Full-Text Search"
sidebarTitle: "Full-Text Search"
description: "The fts() table function — match, fuzzy, phrase, boolean, boost and multi-match queries from SQL."
icon: magnifying-glass
keywords: ["fts", "sql", "full text search", "fuzzy", "phrase", "boolean", "boost", "multi-match"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
Full-text search is a **table function**, `fts(table, query)`, rather than a
predicate -- it produces a relevance score alongside the matching rows, and a
`WHERE` clause has nowhere to put one. The two string-literal arguments are
the table name and a JSON-encoded query, built with the same query-builder
classes used from the SDK and serialized with `.to_json()`.
## Setup {#setup}
Run these statements in order, sending one statement per SQL request.
Semicolons separate the statements for clients that execute a script.
```sql
CREATE TABLE my_docs (id INT, text STRING, category STRING, author STRING);
INSERT INTO my_docs VALUES
(1, 'The happy puppy runs merrily in the park', 'animals', 'Alice'),
(2, 'A curious kitten jumps quickly over the fence', 'animals', 'Bob'),
(3, 'The puppy catches a ball with great enthusiasm', 'sports', 'Alice'),
(5, 'Puppy training requires patience and dedication', 'training', 'Alice');
CREATE INDEX ON my_docs USING fts (text);
```
<Warning>
Phrase queries need position information, which costs extra index size --
add `WITH (with_position = true)` to `CREATE INDEX`. Without it,
[phrase queries](#phrase) find nothing. See
[FTS parameters](/indexing/fts-index#fts-parameters).
</Warning>
## Basic search {#basic}
The query builder produces the JSON string used as the second SQL argument.
The SQL examples below spell out the equivalent match object so they can be
run directly after the setup above.
```python
from lancedb.query import MatchQuery
query = MatchQuery("puppy", "text")
json_query = query.to_json()
```
```sql
SELECT id, text, category
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}')
LIMIT 5
```
By default the top 5 matches come back in **arbitrary order** -- select
`_score` and order by it for relevance ranking:
```sql
SELECT id, text, _score
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}')
ORDER BY _score DESC
LIMIT 5
```
`_score` is a pseudo-column carrying the query's BM25 relevance for the row;
it behaves like any other column, but exists only in a relation that came
from `fts()`. See [Data types](/sql/data-types#pseudo-columns).
## Fuzzy {#fuzzy}
Tolerates typos, up to a character-edit budget:
```python
MatchQuery("pupy", "text", fuzziness=2)
```
## Phrase {#phrase}
Exact phrase, or with `slop` to allow words between terms:
```python
from lancedb.query import PhraseQuery
PhraseQuery("happy puppy", "text")
PhraseQuery("puppy park", "text", slop=2) # up to 2 words between them
```
## Boolean {#boolean}
Combine queries with `&` (AND) or `|` (OR):
```python
MatchQuery("puppy", "text") & MatchQuery("happy", "text")
MatchQuery("puppy", "text") | MatchQuery("kitten", "text")
```
## Boost {#boost}
Weight a query up or down relative to another:
```python
from lancedb.query import BoostQuery
BoostQuery(
positive=MatchQuery("puppy", "text"),
negative=MatchQuery("kitten", "text"),
negative_boost=0.2,
)
```
## Multi-match {#multi-match}
Search several columns at once, optionally weighting them differently:
```python
from lancedb.query import MultiMatchQuery
MultiMatchQuery("puppy", ["text", "category"])
MultiMatchQuery("puppy", ["text", "category"], boosts=[2.0, 1.0]) # text weighted 2x
```
## Combining with SQL {#combining}
`fts()` is a relation, so it composes with the rest of the SQL surface --
filter, join, or aggregate over it same as any table:
```sql
SELECT id, text, category
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}')
WHERE category = 'animals'
LIMIT 5
```
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Full-Text Search" icon="magnifying-glass" href="/search/full-text-search">
The query-builder equivalent, and every query form in full.
</Card>
<Card title="FTS Index" icon="bolt" href="/indexing/fts-index">
Tokenizers, and the position information phrase queries need.
</Card>
</CardGroup>
docs/web/sql/dql/join.mdx
+73
View File
@@ -0,0 +1,73 @@
---
title: "Joins"
sidebarTitle: "Joins"
description: "Combining rows from more than one table, including across namespaces."
icon: link
keywords: ["join", "inner", "left", "outer", "cross", "namespace"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
Joining is the other thing the query builders cannot do. A question spanning two
tables is a SQL question.
## The join types {#types}
```sql
SELECT d.title, c.name
FROM documents AS d
JOIN categories AS c ON c.id = d.category_id
```
| | What it keeps |
| --- | --- |
| `[INNER] JOIN` | rows matching on both sides |
| `LEFT [OUTER] JOIN` | every left row, nulls where the right has no match |
| `RIGHT [OUTER] JOIN` | every right row |
| `FULL [OUTER] JOIN` | every row from both |
| `CROSS JOIN` | every combination -- the product of the two |
`USING (col)` is shorthand for an equality join on a shared column name, and
avoids repeating it in the select list.
## Across namespaces {#namespaces}
Tables in different [namespaces](/management/namespace) join by their dotted paths -- the
catalog is one catalog, so there is nothing to federate:
```sql
SELECT c.id, u.email
FROM prod.events.clicks AS c
JOIN prod.reference.users AS u ON u.id = c.user_id
```
## Joining a search result {#search}
[`fts()`](/sql/functions#fts) is a table function, so its result is a relation
and joins like any other. This example assumes `my_docs` has `text` and
`category_id` columns, a full-text index on `text`, and a `categories` table
with `id` and `name` columns:
```sql
SELECT c.name, count(*) AS hits
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}') AS m
JOIN categories AS c ON c.id = m.category_id
GROUP BY c.name
ORDER BY hits DESC
```
That composition is the reason full-text search is a relation rather than a
predicate -- a `WHERE` clause has nowhere to put the relevance score.
## Performance {#performance}
A [scalar index](/indexing/scalar-index) can accelerate supported filters on
the input tables. An index on a join key does not by itself guarantee an
index lookup join. Use SQL [`EXPLAIN`](/sql/dql/explain) to inspect the scans,
filters, and join chosen for the complete statement.
<Warning>
Avoid selecting blob or vector columns you do not need on either side of a join.
The join multiplies rows before projection is applied in some plans, so a wide
column can be materialized many more times than the result has rows.
</Warning>
docs/web/sql/dql/sampling.mdx
+101
View File
@@ -0,0 +1,101 @@
---
title: "Random sampling"
sidebarTitle: "Sampling"
description: "TABLESAMPLE: draw a random subset of a table, by rows or by fragments."
icon: dice
keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
`TABLESAMPLE` draws a random subset of a table. On a lakehouse table that is
often the difference between a question you can afford to ask and one you
cannot -- an aggregate over 1% of forty billion rows answers in a fraction of
the time, and for most estimates that is enough.
```sql
SELECT * FROM documents TABLESAMPLE (10) -- 10 percent
SELECT * FROM documents TABLESAMPLE (500 ROWS) -- 500 rows
```
A bare number is a **percentage**. Add `ROWS` for a count instead.
## Rows or fragments {#methods}
The standard calls these two methods of one clause. They are different
operations, and the difference matters on Lance:
```sql
SELECT * FROM documents TABLESAMPLE BERNOULLI (10)
SELECT * FROM documents TABLESAMPLE SYSTEM (10)
```
| Method | Aliases | What it picks |
| --- | --- | --- |
| `BERNOULLI` | `ROW` | rows, each considered independently |
| `SYSTEM` | `BLOCK` | **Lance fragments**, and takes whatever rows they hold |
`BERNOULLI` gives a genuinely uniform sample and reads across the whole table to
get it. `SYSTEM` reads far less -- it skips entire fragments -- but the sample is
only as uniform as the data's layout. If rows were written in an order that
correlates with what you are measuring, a fragment sample inherits that
correlation.
Reach for `SYSTEM` when you want a cheap look at the data and `BERNOULLI` when
the number you compute has to be defensible. Omitting the method gives
`BERNOULLI`.
## Repeatability {#repeatable}
Without a seed, every execution resamples -- running the same query twice gives
different rows. `REPEATABLE` fixes the draw:
```sql
SELECT * FROM documents TABLESAMPLE (10) REPEATABLE (42)
```
This is what makes a sample usable as a **fixed evaluation set**: the same seed
against the same data gives the same rows, so a metric computed today is
comparable with one computed last week.
## Where the clause can go {#placement}
It attaches to a table in `FROM`, including inside a subquery, a CTE, or either
side of a join:
```sql
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
SELECT * FROM a TABLESAMPLE (5 ROWS) JOIN b ON a.id = b.id
SELECT * FROM t a TABLESAMPLE (10) JOIN t b TABLESAMPLE (10) ON a.id = b.id
```
Each side of a join samples independently. Sampling both sides of a join
shrinks the *result* far more than either fraction suggests, because a row only
survives if both its sides were drawn.
## What is refused {#refused}
Sampling is semantics, not a hint. A clause that were read and then not applied
would be a wrong answer rather than a missing feature, so anything that cannot
be honoured exactly is an error:
| | |
| --- | --- |
| `TABLESAMPLE (101)` | a percentage outside `0..=100` is refused, not clamped |
| `TABLESAMPLE (1.5 ROWS)` | a row count must be whole |
| `TABLESAMPLE (BUCKET 1 OUT OF 4)` | bucket sampling is not supported |
| the `SAMPLE` spelling | use `TABLESAMPLE` |
| the `OFFSET` modifier | not supported |
<Note>
`TABLESAMPLE` is also refused on a WAL-backed table, where the rows a sample
would draw from are not all in the fragments yet.
</Note>
## Sampling outside SQL {#sdk}
For a training loop rather than a query, the SDKs read a table in a shuffled
order without going through SQL at all -- see [Training](/training/data-loading), where a
permutation drives the data loader.
docs/web/sql/dql/select.mdx
+135
View File
@@ -0,0 +1,135 @@
---
title: "SELECT"
sidebarTitle: "SELECT"
description: "Projection, filtering, ordering, and composing queries out of other queries."
icon: table-columns
keywords: ["select", "where", "order by", "limit", "cte", "union", "distinct"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
```sql
[WITH cte AS (...)]
SELECT [ALL | DISTINCT] select_expr [, ...]
FROM from_item [[JOIN ...] ON ...]
[WHERE predicate]
[GROUP BY expr [, ...]] [HAVING predicate]
[WINDOW name AS (...)]
[ORDER BY expr [ASC | DESC] [NULLS FIRST | LAST] [, ...]]
[LIMIT n] [OFFSET n]
```
## Projection {#projection}
<CodeGroup>
```sql SQL
SELECT id, title, identifier FROM documents
```
```python Python icon="python"
tbl.search().select(["id", "title", "identifier"]).to_arrow()
```
```typescript TypeScript icon="square-js"
await tbl.query().select(["id", "title", "identifier"]).toArrow();
```
</CodeGroup>
Projection pushes into the scan in all three -- an unselected column is never
read off storage. On a table of images or video that is the difference between a
fast query and a slow one, so **prefer naming columns over `SELECT *`**.
Row metadata is available as [pseudo-columns](/sql/data-types#pseudo-columns)
(`_rowid`, `_rowaddr`, `_rowoffset`) but only when named, so they stay out of
`SELECT *`.
`DISTINCT` deduplicates the projected rows.
## Filtering {#filtering}
<CodeGroup>
```sql SQL
SELECT * FROM documents
WHERE identifier > 0 AND identifier < 1000000
```
```python Python icon="python"
tbl.search().where("identifier > 0 AND identifier < 1000000").to_arrow()
```
```typescript TypeScript icon="square-js"
await tbl.query().where("identifier > 0 AND identifier < 1000000").toArrow();
```
</CodeGroup>
The SDK `where` takes the same expression language, so a predicate moves between
them unchanged. A predicate on an indexed column becomes a `ScalarIndexQuery`
rather than a scan -- confirm with [`explain_plan`](/search/explain). See
[Filtering](/search/filtering) for the operators and their pushdown behaviour.
## Ordering and limiting {#order-limit}
```sql
SELECT id, title FROM documents
ORDER BY created_at DESC NULLS LAST
LIMIT 100 OFFSET 200
```
`ORDER BY` on a large table is a sort over the whole result, so pair it with a
`LIMIT` unless you genuinely need every row ordered.
## Set operations {#set-operations}
`UNION`, `UNION ALL`, `INTERSECT` and `EXCEPT` combine whole queries:
```sql
SELECT id FROM documents WHERE category = 'film'
UNION ALL
SELECT id FROM archived_documents WHERE category = 'film'
```
`UNION` deduplicates and `UNION ALL` does not, which makes `UNION ALL` the
cheaper one when you already know the inputs are disjoint.
## Common table expressions {#ctes}
```sql
WITH recent AS (
SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
)
SELECT category, count(*) FROM recent GROUP BY category
```
A CTE is scoped to the statement. One you want to keep is a
[view](/management/view); one whose rows you want stored is a
[materialized view](/management/materialized-view).
## Subqueries {#subqueries}
Subqueries are allowed anywhere an expression or a relation is -- in `FROM`, in
`WHERE` with `IN` or `EXISTS`, and as a scalar in the select list:
```sql
SELECT title
FROM documents
WHERE category_id IN (SELECT id FROM categories WHERE active)
```
## Namespaced tables {#namespaces}
A table inside a [namespace](/management/namespace) is written as a dotted path:
```sql
SELECT * FROM prod.events.clicks
```
## Reading the plan {#plans}
`EXPLAIN` and `EXPLAIN ANALYZE` work as DataFusion defines them, and produce the
same operators the SDK methods do:
```sql
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
```
See [Explain](/search/explain) and [Explain Analyze](/search/explain).
docs/web/sql/dql/subqueries.mdx
+76
View File
@@ -0,0 +1,76 @@
---
title: "Subqueries"
sidebarTitle: "Subqueries"
description: "Queries inside queries: scalar, IN, EXISTS, correlated, and derived tables."
icon: code-branch
keywords: ["subquery", "exists", "in", "correlated", "derived table", "lateral"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
A subquery is allowed anywhere an expression or a relation is.
## As a relation {#derived}
```sql
SELECT * FROM (
SELECT category, count(*) AS n FROM documents GROUP BY category
) AS counts
WHERE n > 100
```
A derived table needs an alias. It can be [sampled](/sql/dql/sampling) like a base
table:
```sql
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
```
## As a scalar {#scalar}
```sql
SELECT title, (SELECT count(*) FROM revisions r WHERE r.doc_id = d.id) AS revisions
FROM documents AS d
```
A scalar subquery must return one row and one column. Returning more is an
error rather than a silently-taken first row.
## IN and EXISTS {#in-exists}
```sql
SELECT title FROM documents
WHERE category_id IN (SELECT id FROM categories WHERE active)
SELECT title FROM documents AS d
WHERE EXISTS (SELECT 1 FROM revisions r WHERE r.doc_id = d.id)
```
`EXISTS` stops at the first matching row, so it is the cheaper test when you
only care whether anything matches. `IN` builds the full set.
`NOT IN` and `NOT EXISTS` differ on nulls: `NOT IN` against a set containing a
null yields no rows, because the comparison is unknown rather than false. Prefer
`NOT EXISTS` unless you have ruled nulls out.
## Correlated subqueries {#correlated}
A subquery referencing a column from the enclosing query — `d.id` above — is
correlated, and is conceptually evaluated per outer row. The planner usually
rewrites it into a join.
<Note>
Check with [`EXPLAIN`](/sql/dql/explain) when a correlated subquery is slow. A plan
showing a join has been optimized; one showing a repeated scan has not, and
rewriting it as an explicit [join](/sql/dql/join) is the fix.
</Note>
## CTEs as an alternative {#ctes}
A [CTE](/sql/dql/select#ctes) is a named subquery, and is usually more readable when
the same subquery appears twice or when nesting gets deep.
```sql
WITH active AS (SELECT id FROM categories WHERE active)
SELECT title FROM documents WHERE category_id IN (SELECT id FROM active)
```
docs/web/sql/dql/window.mdx
+100
View File
@@ -0,0 +1,100 @@
---
title: "Window functions"
sidebarTitle: "Window Functions"
description: "Compute over a set of rows related to the current one, without collapsing them."
icon: chart-simple
keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead"]
---
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
A window function computes across a set of rows related to the current one and
returns a value **per row**. That is the difference from
[aggregation](/sql/dql/aggregation), which collapses the rows it covers.
```sql
SELECT
title,
score,
row_number() OVER (PARTITION BY category ORDER BY score DESC) AS rank
FROM documents
```
Every row survives; each one gains its rank within its category.
## The window {#window}
```sql
OVER (
[PARTITION BY expr [, ...]]
[ORDER BY expr [, ...]]
[ROWS | RANGE BETWEEN frame_start AND frame_end]
)
```
`PARTITION BY` splits the rows into independent groups. `ORDER BY` orders them
inside each partition -- required by the ranking functions, since a rank without
an order is meaningless. The frame clause narrows the window further, which is
how running totals and moving averages are expressed:
```sql
SELECT
created_at,
sum(amount) OVER (ORDER BY created_at ROWS UNBOUNDED PRECEDING) AS running_total,
avg(amount) OVER (ORDER BY created_at ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS trailing_7
FROM transactions
```
A `WINDOW` clause names a definition when several functions share one:
```sql
SELECT
row_number() OVER w,
lag(score) OVER w
FROM documents
WINDOW w AS (PARTITION BY category ORDER BY score DESC)
```
## The functions {#functions}
Registered in DataFusion 54.
**Ranking**
```text
row_number rank dense_rank percent_rank cume_dist ntile
```
**Offset and positional**
```text
lag lead first_value last_value nth_value
```
**Aggregates**
Any [aggregate function](/sql/dql/aggregation#functions) used with `OVER` becomes a
window function — that is how running totals and moving averages are written.
## Deduplicating with a window {#dedup}
The common practical use: keep one row per key, chosen by some ordering. A
window ranks, and an outer query filters:
```sql
WITH ranked AS (
SELECT *, row_number() OVER (PARTITION BY doc_id ORDER BY version DESC) AS rn
FROM revisions
)
SELECT * FROM ranked WHERE rn = 1
```
The outer query is needed because a window function cannot appear in `WHERE` --
windows are computed after filtering, so the predicate would have nothing to
read.
<Note>
Window functions sort within each partition, so on a large table they are among
the more expensive things you can ask for. Filter first, and check the cost with
[Explain Analyze](/search/explain).
</Note>