mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
docs: give SQL one section with a page per construct
SELECT and Enterprise SQL sat apart in Query while documenting the same engine. They are now one SQL group: overview, Select, Aggregation, Join, Window, Sampling, Functions -- one page per construct, with the Enterprise client-setup and FTS pages merged in rather than parallel to them. Sampling is new. TABLESAMPLE is implemented and was undocumented: BERNOULLI picks rows, SYSTEM picks Lance fragments, REPEATABLE fixes the draw so a sample works as a stable evaluation set. Its refusals are documented too -- a percentage over 100, a fractional row count, BUCKET, the SAMPLE spelling and the OFFSET modifier are all errors, because a sampling clause that were read and not applied would be a wrong answer rather than a missing feature. Schema & Data Types is one page again, with the multimodal content merged into its blob section rather than sitting beside it. Agentic Skills stops being a tab. Both its pages are tutorials, so they join the Tutorials tab; the Basics page that introduces them stays.
This commit is contained in:
Jack Ye
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: "Aggregation"
|
||||
sidebarTitle: "Aggregation"
|
||||
description: "GROUP BY, HAVING, and the aggregate function set."
|
||||
icon: layer-group
|
||||
keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximate"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Aggregation is one of the two main reasons to reach for SQL rather than a query
|
||||
builder -- the SDKs project, filter and search, but they do not group. The other
|
||||
is [joins](/sql/join).
|
||||
|
||||
## Grouping {#grouping}
|
||||
|
||||
```sql
|
||||
SELECT category, count(*) AS n, avg(score) AS mean_score
|
||||
FROM documents
|
||||
GROUP BY category
|
||||
HAVING count(*) > 10
|
||||
ORDER BY n DESC
|
||||
```
|
||||
|
||||
`WHERE` filters rows before grouping; `HAVING` filters groups after. Putting a
|
||||
row-level predicate in `HAVING` still gives the right answer but does more work,
|
||||
because rows that could have been discarded early are grouped first.
|
||||
|
||||
Grouping by several expressions produces one row per distinct combination:
|
||||
|
||||
```sql
|
||||
SELECT category, source, count(*) FROM documents GROUP BY category, source
|
||||
```
|
||||
|
||||
## Aggregate without grouping {#whole-table}
|
||||
|
||||
Omitting `GROUP BY` aggregates the whole table into one row -- the usual way to
|
||||
ask how big something is:
|
||||
|
||||
```sql
|
||||
SELECT count(*), min(created_at), max(created_at) FROM documents
|
||||
```
|
||||
|
||||
`count(*)` counts rows; `count(col)` counts non-null values, which is how you
|
||||
measure how populated a [computed column](/feature-engineering/add-computed-column)
|
||||
is before deciding to refresh it again.
|
||||
|
||||
## The function set {#functions}
|
||||
|
||||
The aggregates are DataFusion's, and the full list is in its
|
||||
[aggregate function reference](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html):
|
||||
|
||||
| Kind | Examples |
|
||||
| --- | --- |
|
||||
| General | `count`, `sum`, `min`, `max`, `avg` |
|
||||
| Statistical | `stddev`, `var`, `corr`, `covar`, `regr_*` |
|
||||
| Approximate | `approx_distinct`, `approx_percentile_cont`, `approx_median` |
|
||||
| Collecting | `array_agg`, `string_agg`, `first_value`, `last_value` |
|
||||
|
||||
The approximate ones are worth knowing about on a large table:
|
||||
`approx_distinct` answers "roughly how many distinct values" for a fraction of
|
||||
what `count(DISTINCT ...)` costs, which matters when the column has high
|
||||
cardinality.
|
||||
|
||||
## Aggregating a sample {#sampling}
|
||||
|
||||
For an estimate over a very large table, aggregate a
|
||||
[sample](/sql/sampling) rather than the whole thing:
|
||||
|
||||
```sql
|
||||
SELECT category, count(*) * 100 AS estimated
|
||||
FROM documents TABLESAMPLE (1)
|
||||
GROUP BY category
|
||||
```
|
||||
|
||||
Scale the result by the inverse of the sample fraction, and treat it as an
|
||||
estimate -- a 1% sample of a skewed column will miss rare categories entirely.
|
||||
|
||||
## What aggregation does not do {#not}
|
||||
|
||||
Aggregates run over rows the query already produced. They cannot combine with
|
||||
[vector search](/search/vector-search) in one statement, because similarity
|
||||
search has no SQL form. Run the search in an SDK and aggregate its result, or
|
||||
materialize the search output first.
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
title: "SQL"
|
||||
sidebarTitle: "Overview"
|
||||
description: "Analytical SQL over the lakehouse: the dialect, where it runs, and what it can reach."
|
||||
icon: table-list
|
||||
keywords: ["sql", "datafusion", "flight sql", "analytics", "dialect"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> SQL runs on a deployment. A local database
|
||||
has no engine to plan a statement, so `execute_query` raises there -- see
|
||||
[SDKs and SQL](/basics/apis).
|
||||
|
||||
The dialect is [Apache DataFusion's](/sql/functions), so anything its reference
|
||||
documents for that version works here unchanged. What these pages cover is the
|
||||
parts that are LanceDB's: how a query reaches the engine, how Lance columns
|
||||
appear to it, and the extensions a lakehouse over multimodal data needs.
|
||||
|
||||
## Where a query runs {#execution}
|
||||
|
||||
Queries execute on the server. The client sends a statement and receives Arrow
|
||||
record batches -- nothing is converted to pandas or Python objects unless you
|
||||
ask for it. A query feeding training or downstream processing therefore pays no
|
||||
serialization cost to get there.
|
||||
|
||||
Results stream. `execute_query` hands back a reader as soon as the first batch
|
||||
is ready rather than when the query finishes, so a large result set does not
|
||||
have to fit in memory before you can start reading it. See
|
||||
[Query and job](/basics/jobs#queries).
|
||||
|
||||
## What differs from a warehouse {#differences}
|
||||
|
||||
Two things, and both come from the storage being Lance rather than something
|
||||
SQL-shaped:
|
||||
|
||||
- **Columns are Arrow types**, not a SQL type system mapped onto storage. A
|
||||
vector column is a `FixedSizeList<Float32>` and behaves like one. See
|
||||
[Data types](/basics/data-types).
|
||||
- **Row metadata is available as columns** -- `_rowid`, `_rowaddr`,
|
||||
`_rowoffset` -- but only when named, so they stay out of `SELECT *`.
|
||||
|
||||
## The statements {#statements}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| [SELECT](/sql/select) | projection, filtering, ordering, set operations, CTEs |
|
||||
| [Aggregation](/sql/aggregation) | `GROUP BY`, `HAVING`, aggregate functions |
|
||||
| [Joins](/sql/join) | inner, outer, cross, and joining across namespaces |
|
||||
| [Window functions](/sql/window) | `OVER (...)`, ranking, running totals |
|
||||
| [Sampling](/sql/sampling) | `TABLESAMPLE`, for a random subset |
|
||||
| [Functions and types](/sql/functions) | the function set, and what LanceDB adds |
|
||||
|
||||
Beyond reading, SQL also writes (`INSERT`, `UPDATE`, `DELETE`, `COPY`), manages
|
||||
the catalog (`CREATE TABLE`, `CREATE VIEW`, `DROP`, `SHOW`) and drives feature
|
||||
engineering (`CREATE FUNCTION`, `REFRESH COLUMN`, `REFRESH MATERIALIZED VIEW`).
|
||||
Those are documented with the operation rather than here -- see
|
||||
[Management](/management) and [Feature engineering](/feature-engineering).
|
||||
|
||||
<Note>
|
||||
Vector search has no SQL form. It is a query-builder operation -- see
|
||||
[Vector search](/search/vector-search). Full-text search does: the
|
||||
[`fts()`](/sql/functions#fts) table function and the
|
||||
[`contains_tokens`](/sql/functions#contains-tokens) predicate.
|
||||
</Note>
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: "Joins"
|
||||
sidebarTitle: "Join"
|
||||
description: "Combining rows from more than one table, including across namespaces."
|
||||
icon: link
|
||||
keywords: ["join", "inner", "left", "outer", "cross", "namespace"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Joining is the other thing the query builders cannot do. A question spanning two
|
||||
tables is a SQL question.
|
||||
|
||||
## The join types {#types}
|
||||
|
||||
```sql
|
||||
SELECT d.title, c.name
|
||||
FROM documents AS d
|
||||
JOIN categories AS c ON c.id = d.category_id
|
||||
```
|
||||
|
||||
| | What it keeps |
|
||||
| --- | --- |
|
||||
| `[INNER] JOIN` | rows matching on both sides |
|
||||
| `LEFT [OUTER] JOIN` | every left row, nulls where the right has no match |
|
||||
| `RIGHT [OUTER] JOIN` | every right row |
|
||||
| `FULL [OUTER] JOIN` | every row from both |
|
||||
| `CROSS JOIN` | every combination -- the product of the two |
|
||||
|
||||
`USING (col)` is shorthand for an equality join on a shared column name, and
|
||||
avoids repeating it in the select list.
|
||||
|
||||
## Across namespaces {#namespaces}
|
||||
|
||||
Tables in different [namespaces](/namespaces) join by their dotted paths -- the
|
||||
catalog is one catalog, so there is nothing to federate:
|
||||
|
||||
```sql
|
||||
SELECT c.id, u.email
|
||||
FROM prod.events.clicks AS c
|
||||
JOIN prod.reference.users AS u ON u.id = c.user_id
|
||||
```
|
||||
|
||||
## Joining a search result {#search}
|
||||
|
||||
[`fts()`](/sql/functions#fts) is a table function, so its result is a relation
|
||||
and joins like any other:
|
||||
|
||||
```sql
|
||||
SELECT c.name, count(*) AS hits
|
||||
FROM fts('my_docs', 'puppy') AS m
|
||||
JOIN categories AS c ON c.id = m.category_id
|
||||
GROUP BY c.name
|
||||
ORDER BY hits DESC
|
||||
```
|
||||
|
||||
That composition is the reason full-text search is a relation rather than a
|
||||
predicate -- a `WHERE` clause has nowhere to put the relevance score.
|
||||
|
||||
## Performance {#performance}
|
||||
|
||||
A join key with a [scalar index](/indexing/scalar-index) on it lets the planner
|
||||
look rows up rather than scan for them. Check with
|
||||
[`explain_plan`](/search/explain): an indexed key shows a `ScalarIndexQuery`,
|
||||
an unindexed one shows a `LanceScan` feeding the join.
|
||||
|
||||
<Warning>
|
||||
Avoid selecting blob or vector columns you do not need on either side of a join.
|
||||
The join multiplies rows before projection is applied in some plans, so a wide
|
||||
column can be materialized many more times than the result has rows.
|
||||
</Warning>
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: "Random sampling"
|
||||
sidebarTitle: "Sampling"
|
||||
description: "TABLESAMPLE: draw a random subset of a table, by rows or by fragments."
|
||||
icon: dice
|
||||
keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
`TABLESAMPLE` draws a random subset of a table. On a lakehouse table that is
|
||||
often the difference between a question you can afford to ask and one you
|
||||
cannot -- an aggregate over 1% of forty billion rows answers in a fraction of
|
||||
the time, and for most estimates that is enough.
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE (10) -- 10 percent
|
||||
SELECT * FROM documents TABLESAMPLE (500 ROWS) -- 500 rows
|
||||
```
|
||||
|
||||
A bare number is a **percentage**. Add `ROWS` for a count instead.
|
||||
|
||||
## Rows or fragments {#methods}
|
||||
|
||||
The standard calls these two methods of one clause. They are different
|
||||
operations, and the difference matters on Lance:
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE BERNOULLI (10)
|
||||
SELECT * FROM documents TABLESAMPLE SYSTEM (10)
|
||||
```
|
||||
|
||||
| Method | Aliases | What it picks |
|
||||
| --- | --- | --- |
|
||||
| `BERNOULLI` | `ROW` | rows, each considered independently |
|
||||
| `SYSTEM` | `BLOCK` | **Lance fragments**, and takes whatever rows they hold |
|
||||
|
||||
`BERNOULLI` gives a genuinely uniform sample and reads across the whole table to
|
||||
get it. `SYSTEM` reads far less -- it skips entire fragments -- but the sample is
|
||||
only as uniform as the data's layout. If rows were written in an order that
|
||||
correlates with what you are measuring, a fragment sample inherits that
|
||||
correlation.
|
||||
|
||||
Reach for `SYSTEM` when you want a cheap look at the data and `BERNOULLI` when
|
||||
the number you compute has to be defensible. Omitting the method gives
|
||||
`BERNOULLI`.
|
||||
|
||||
## Repeatability {#repeatable}
|
||||
|
||||
Without a seed, every execution resamples -- running the same query twice gives
|
||||
different rows. `REPEATABLE` fixes the draw:
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE (10) REPEATABLE (42)
|
||||
```
|
||||
|
||||
This is what makes a sample usable as a **fixed evaluation set**: the same seed
|
||||
against the same data gives the same rows, so a metric computed today is
|
||||
comparable with one computed last week.
|
||||
|
||||
## Where the clause can go {#placement}
|
||||
|
||||
It attaches to a table in `FROM`, including inside a subquery, a CTE, or either
|
||||
side of a join:
|
||||
|
||||
```sql
|
||||
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
|
||||
|
||||
SELECT * FROM a TABLESAMPLE (5 ROWS) JOIN b ON a.id = b.id
|
||||
|
||||
SELECT * FROM t a TABLESAMPLE (10) JOIN t b TABLESAMPLE (10) ON a.id = b.id
|
||||
```
|
||||
|
||||
Each side of a join samples independently. Sampling both sides of a join
|
||||
shrinks the *result* far more than either fraction suggests, because a row only
|
||||
survives if both its sides were drawn.
|
||||
|
||||
## What is refused {#refused}
|
||||
|
||||
Sampling is semantics, not a hint. A clause that were read and then not applied
|
||||
would be a wrong answer rather than a missing feature, so anything that cannot
|
||||
be honoured exactly is an error:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `TABLESAMPLE (101)` | a percentage outside `0..=100` is refused, not clamped |
|
||||
| `TABLESAMPLE (1.5 ROWS)` | a row count must be whole |
|
||||
| `TABLESAMPLE (BUCKET 1 OUT OF 4)` | bucket sampling is not supported |
|
||||
| the `SAMPLE` spelling | use `TABLESAMPLE` |
|
||||
| the `OFFSET` modifier | not supported |
|
||||
|
||||
<Note>
|
||||
`TABLESAMPLE` is also refused on a WAL-backed table, where the rows a sample
|
||||
would draw from are not all in the fragments yet.
|
||||
</Note>
|
||||
|
||||
## Sampling outside SQL {#sdk}
|
||||
|
||||
For a training loop rather than a query, the SDKs read a table in a shuffled
|
||||
order without going through SQL at all -- see [Training](/training), where a
|
||||
permutation drives the data loader.
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
title: "SELECT"
|
||||
sidebarTitle: "Select"
|
||||
description: "Projection, filtering, ordering, and composing queries out of other queries."
|
||||
icon: table-columns
|
||||
keywords: ["select", "where", "order by", "limit", "cte", "union", "distinct"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
[WITH cte AS (...)]
|
||||
SELECT [ALL | DISTINCT] select_expr [, ...]
|
||||
FROM from_item [[JOIN ...] ON ...]
|
||||
[WHERE predicate]
|
||||
[GROUP BY expr [, ...]] [HAVING predicate]
|
||||
[WINDOW name AS (...)]
|
||||
[ORDER BY expr [ASC | DESC] [NULLS FIRST | LAST] [, ...]]
|
||||
[LIMIT n] [OFFSET n]
|
||||
```
|
||||
|
||||
## Projection {#projection}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL
|
||||
SELECT id, title, identifier FROM documents
|
||||
```
|
||||
|
||||
```python Python icon="python"
|
||||
tbl.search().select(["id", "title", "identifier"]).to_arrow()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await tbl.query().select(["id", "title", "identifier"]).toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Projection pushes into the scan in all three -- an unselected column is never
|
||||
read off storage. On a table of images or video that is the difference between a
|
||||
fast query and a slow one, so **prefer naming columns over `SELECT *`**.
|
||||
|
||||
Row metadata is available as [pseudo-columns](/sql/functions#pseudo-columns)
|
||||
(`_rowid`, `_rowaddr`, `_rowoffset`) but only when named, so they stay out of
|
||||
`SELECT *`.
|
||||
|
||||
`DISTINCT` deduplicates the projected rows.
|
||||
|
||||
## Filtering {#filtering}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL
|
||||
SELECT * FROM documents
|
||||
WHERE identifier > 0 AND identifier < 1000000
|
||||
```
|
||||
|
||||
```python Python icon="python"
|
||||
tbl.search().where("identifier > 0 AND identifier < 1000000").to_arrow()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await tbl.query().where("identifier > 0 AND identifier < 1000000").toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The SDK `where` takes the same expression language, so a predicate moves between
|
||||
them unchanged. A predicate on an indexed column becomes a `ScalarIndexQuery`
|
||||
rather than a scan -- confirm with [`explain_plan`](/search/explain). See
|
||||
[Filtering](/search/filtering) for the operators and their pushdown behaviour.
|
||||
|
||||
## Ordering and limiting {#order-limit}
|
||||
|
||||
```sql
|
||||
SELECT id, title FROM documents
|
||||
ORDER BY created_at DESC NULLS LAST
|
||||
LIMIT 100 OFFSET 200
|
||||
```
|
||||
|
||||
`ORDER BY` on a large table is a sort over the whole result, so pair it with a
|
||||
`LIMIT` unless you genuinely need every row ordered.
|
||||
|
||||
## Set operations {#set-operations}
|
||||
|
||||
`UNION`, `UNION ALL`, `INTERSECT` and `EXCEPT` combine whole queries:
|
||||
|
||||
```sql
|
||||
SELECT id FROM documents WHERE category = 'film'
|
||||
UNION ALL
|
||||
SELECT id FROM archived_documents WHERE category = 'film'
|
||||
```
|
||||
|
||||
`UNION` deduplicates and `UNION ALL` does not, which makes `UNION ALL` the
|
||||
cheaper one when you already know the inputs are disjoint.
|
||||
|
||||
## Common table expressions {#ctes}
|
||||
|
||||
```sql
|
||||
WITH recent AS (
|
||||
SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
|
||||
)
|
||||
SELECT category, count(*) FROM recent GROUP BY category
|
||||
```
|
||||
|
||||
A CTE is scoped to the statement. One you want to keep is a
|
||||
[view](/management/view); one whose rows you want stored is a
|
||||
[materialized view](/management/materialized-view).
|
||||
|
||||
## Subqueries {#subqueries}
|
||||
|
||||
Subqueries are allowed anywhere an expression or a relation is -- in `FROM`, in
|
||||
`WHERE` with `IN` or `EXISTS`, and as a scalar in the select list:
|
||||
|
||||
```sql
|
||||
SELECT title
|
||||
FROM documents
|
||||
WHERE category_id IN (SELECT id FROM categories WHERE active)
|
||||
```
|
||||
|
||||
## Namespaced tables {#namespaces}
|
||||
|
||||
A table inside a [namespace](/namespaces) is written as a dotted path:
|
||||
|
||||
```sql
|
||||
SELECT * FROM prod.events.clicks
|
||||
```
|
||||
|
||||
## Reading the plan {#plans}
|
||||
|
||||
`EXPLAIN` and `EXPLAIN ANALYZE` work as DataFusion defines them, and produce the
|
||||
same operators the SDK methods do:
|
||||
|
||||
```sql
|
||||
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
|
||||
```
|
||||
|
||||
See [Explain](/search/explain) and [Explain Analyze](/search/explain-analyze).
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: "Window functions"
|
||||
sidebarTitle: "Window"
|
||||
description: "Compute over a set of rows related to the current one, without collapsing them."
|
||||
icon: chart-simple
|
||||
keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A window function computes across a set of rows related to the current one and
|
||||
returns a value **per row**. That is the difference from
|
||||
[aggregation](/sql/aggregation), which collapses the rows it covers.
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
title,
|
||||
score,
|
||||
row_number() OVER (PARTITION BY category ORDER BY score DESC) AS rank
|
||||
FROM documents
|
||||
```
|
||||
|
||||
Every row survives; each one gains its rank within its category.
|
||||
|
||||
## The window {#window}
|
||||
|
||||
```sql
|
||||
OVER (
|
||||
[PARTITION BY expr [, ...]]
|
||||
[ORDER BY expr [, ...]]
|
||||
[ROWS | RANGE BETWEEN frame_start AND frame_end]
|
||||
)
|
||||
```
|
||||
|
||||
`PARTITION BY` splits the rows into independent groups. `ORDER BY` orders them
|
||||
inside each partition -- required by the ranking functions, since a rank without
|
||||
an order is meaningless. The frame clause narrows the window further, which is
|
||||
how running totals and moving averages are expressed:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
created_at,
|
||||
sum(amount) OVER (ORDER BY created_at ROWS UNBOUNDED PRECEDING) AS running_total,
|
||||
avg(amount) OVER (ORDER BY created_at ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS trailing_7
|
||||
FROM transactions
|
||||
```
|
||||
|
||||
A `WINDOW` clause names a definition when several functions share one:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
row_number() OVER w,
|
||||
lag(score) OVER w
|
||||
FROM documents
|
||||
WINDOW w AS (PARTITION BY category ORDER BY score DESC)
|
||||
```
|
||||
|
||||
## The functions {#functions}
|
||||
|
||||
| Kind | Functions |
|
||||
| --- | --- |
|
||||
| Ranking | `row_number`, `rank`, `dense_rank`, `percent_rank`, `ntile`, `cume_dist` |
|
||||
| Offset | `lag`, `lead`, `first_value`, `last_value`, `nth_value` |
|
||||
| Aggregate | any [aggregate](/sql/aggregation) used with `OVER` |
|
||||
|
||||
The full set is DataFusion's
|
||||
[window function reference](https://datafusion.apache.org/user-guide/sql/window_functions.html).
|
||||
|
||||
## Deduplicating with a window {#dedup}
|
||||
|
||||
The common practical use: keep one row per key, chosen by some ordering. A
|
||||
window ranks, and an outer query filters:
|
||||
|
||||
```sql
|
||||
WITH ranked AS (
|
||||
SELECT *, row_number() OVER (PARTITION BY doc_id ORDER BY version DESC) AS rn
|
||||
FROM revisions
|
||||
)
|
||||
SELECT * FROM ranked WHERE rn = 1
|
||||
```
|
||||
|
||||
The outer query is needed because a window function cannot appear in `WHERE` --
|
||||
windows are computed after filtering, so the predicate would have nothing to
|
||||
read.
|
||||
|
||||
<Note>
|
||||
Window functions sort within each partition, so on a large table they are among
|
||||
the more expensive things you can ask for. Filter first, and check the cost with
|
||||
[Explain Analyze](/search/explain-analyze).
|
||||
</Note>
|
||||
Reference in New Issue
Block a user