mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-17 09:12:28 +00:00
docs: flip ownership of View/SQL-client/FTS-SQL/AuthZ/AuthN to OSS, restructure SQL Reference
OSS docs now own the full usage content for features that only run against an Enterprise deployment, since the client interface itself lives here; Enterprise docs keep only the parts specific to its own behavior. Sophon's overlay for these five pages is dropped since the OSS versions are now the complete reference. SQL Reference is restructured to match an object-type SQL manual (DDL by object, then DML, then DQL, then Expressions) instead of the old flat, loosely-ordered page list, absorbing the SQL client-connection guide and FTS-via-SQL examples into it.
This commit is contained in:
Jack Ye
@@ -0,0 +1,118 @@
|
||||
---
|
||||
title: "Aggregation"
|
||||
sidebarTitle: "Aggregation"
|
||||
description: "GROUP BY, HAVING, and the aggregate function set."
|
||||
icon: layer-group
|
||||
keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximate"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Aggregation is one of the two main reasons to reach for SQL rather than a query
|
||||
builder -- the SDKs project, filter and search, but they do not group. The other
|
||||
is [joins](/sql/dql/join).
|
||||
|
||||
## Grouping {#grouping}
|
||||
|
||||
```sql
|
||||
SELECT category, count(*) AS n, avg(score) AS mean_score
|
||||
FROM documents
|
||||
GROUP BY category
|
||||
HAVING count(*) > 10
|
||||
ORDER BY n DESC
|
||||
```
|
||||
|
||||
`WHERE` filters rows before grouping; `HAVING` filters groups after. Putting a
|
||||
row-level predicate in `HAVING` still gives the right answer but does more work,
|
||||
because rows that could have been discarded early are grouped first.
|
||||
|
||||
Grouping by several expressions produces one row per distinct combination:
|
||||
|
||||
```sql
|
||||
SELECT category, source, count(*) FROM documents GROUP BY category, source
|
||||
```
|
||||
|
||||
## Aggregate without grouping {#whole-table}
|
||||
|
||||
Omitting `GROUP BY` aggregates the whole table into one row -- the usual way to
|
||||
ask how big something is:
|
||||
|
||||
```sql
|
||||
SELECT count(*), min(created_at), max(created_at) FROM documents
|
||||
```
|
||||
|
||||
`count(*)` counts rows; `count(col)` counts non-null values, which is how you
|
||||
measure how populated a [computed column](/feature-engineering/add-computed-column)
|
||||
is before deciding to refresh it again.
|
||||
|
||||
## The function set {#functions}
|
||||
|
||||
Registered in DataFusion 54. Names below are what is actually available.
|
||||
|
||||
**General**
|
||||
|
||||
```text
|
||||
count sum min max avg median
|
||||
bit_and bit_or bit_xor bool_and bool_or
|
||||
first_value last_value nth_value grouping
|
||||
```
|
||||
|
||||
**Statistical**
|
||||
|
||||
```text
|
||||
stddev stddev_pop stddev_samp
|
||||
var var_pop var_samp
|
||||
corr covar_pop covar_samp
|
||||
regr_slope regr_intercept regr_count regr_r2
|
||||
regr_avgx regr_avgy regr_sxx regr_syy regr_sxy
|
||||
```
|
||||
|
||||
**Approximate**
|
||||
|
||||
```text
|
||||
approx_distinct approx_median
|
||||
approx_percentile_cont approx_percentile_cont_with_weight
|
||||
percentile_cont
|
||||
```
|
||||
|
||||
**Collecting**
|
||||
|
||||
```text
|
||||
array_agg string_agg
|
||||
```
|
||||
|
||||
The approximate functions are the ones worth reaching for on a large table.
|
||||
`approx_distinct` answers "roughly how many distinct values" for a fraction of
|
||||
what `count(DISTINCT ...)` costs, which matters most where it is most expensive
|
||||
— a high-cardinality column.
|
||||
|
||||
`DISTINCT` works inside an aggregate: `count(DISTINCT category)`. `FILTER`
|
||||
narrows one aggregate without narrowing the others:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
count(*) AS total,
|
||||
count(*) FILTER (WHERE score > 0.9) AS confident
|
||||
FROM documents
|
||||
```
|
||||
|
||||
## Aggregating a sample {#sampling}
|
||||
|
||||
For an estimate over a very large table, aggregate a
|
||||
[sample](/sql/dql/sampling) rather than the whole thing:
|
||||
|
||||
```sql
|
||||
SELECT category, count(*) * 100 AS estimated
|
||||
FROM documents TABLESAMPLE (1)
|
||||
GROUP BY category
|
||||
```
|
||||
|
||||
Scale the result by the inverse of the sample fraction, and treat it as an
|
||||
estimate -- a 1% sample of a skewed column will miss rare categories entirely.
|
||||
|
||||
## What aggregation does not do {#not}
|
||||
|
||||
Aggregates run over rows the query already produced. They cannot combine with
|
||||
[vector search](/search/vector-search) in one statement, because similarity
|
||||
search has no SQL form. Run the search in an SDK and aggregate its result, or
|
||||
materialize the search output first.
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
title: "Explain"
|
||||
sidebarTitle: "Explain"
|
||||
description: "EXPLAIN, EXPLAIN ANALYZE and EXPLAIN VERBOSE — reading and measuring a plan from SQL."
|
||||
icon: map
|
||||
keywords: ["explain", "explain analyze", "explain verbose", "query plan"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
EXPLAIN SELECT * FROM documents WHERE identifier < 1000
|
||||
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
|
||||
```
|
||||
|
||||
| Statement | What it returns |
|
||||
| --- | --- |
|
||||
| `EXPLAIN <query>` | Logical and physical plan, without executing the query. |
|
||||
| `EXPLAIN ANALYZE <query>` | Executes the query and annotates each operator with runtime metrics (rows, timing). |
|
||||
| `EXPLAIN VERBOSE <query>` | Adds intermediate optimizer plans on top of `EXPLAIN`. |
|
||||
|
||||
The result is a small relation with `plan_type` and `plan` columns, read
|
||||
through the same client you use for regular queries -- see
|
||||
[Connecting](/sql#connecting).
|
||||
|
||||
The operators are the same ones the SDK's
|
||||
[`explain_plan` and `analyze_plan`](/search/explain) methods produce, so the
|
||||
reasoning about index coverage and filter pushdown carries across unchanged:
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="What it shows" icon="map" href="/search/explain#inspect-a-query-plan">
|
||||
Reading a plan: operators, pushdown, index usage.
|
||||
</Card>
|
||||
<Card title="What it measures" icon="gauge" href="/search/explain#analyze-a-query-plan">
|
||||
Reading the metrics, and a worked before/after.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
## What to look for {#what-to-look-for}
|
||||
|
||||
| Sign | Means |
|
||||
| --- | --- |
|
||||
| `ScalarIndexQuery` | the predicate used a scalar index |
|
||||
| `LanceScan` with high `bytes_read` | no index, or too many columns selected |
|
||||
| `FilterExec` above a search | the filter ran after search rather than pushing into it |
|
||||
| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used |
|
||||
|
||||
A `LanceScan` feeding a join is the common one worth acting on -- see
|
||||
[Joins](/sql/dql/join#performance).
|
||||
|
||||
For a table's columns and types without a plan, see
|
||||
[`DESCRIBE`](/sql/ddl/table#describe).
|
||||
@@ -0,0 +1,136 @@
|
||||
---
|
||||
title: "Full-Text Search"
|
||||
sidebarTitle: "Full-Text Search"
|
||||
description: "The fts() table function — match, fuzzy, phrase, boolean, boost and multi-match queries from SQL."
|
||||
icon: magnifying-glass
|
||||
keywords: ["fts", "sql", "full text search", "fuzzy", "phrase", "boolean", "boost", "multi-match"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Full-text search is a **table function**, `fts(table, query)`, rather than a
|
||||
predicate -- it produces a relevance score alongside the matching rows, and a
|
||||
`WHERE` clause has nowhere to put one. The two string-literal arguments are
|
||||
the table name and a JSON-encoded query, built with the same query-builder
|
||||
classes used from the SDK and serialized with `.to_json()`.
|
||||
|
||||
## Setup {#setup}
|
||||
|
||||
```sql
|
||||
CREATE TABLE my_docs (id INT, text STRING, category STRING, author STRING)
|
||||
|
||||
INSERT INTO my_docs VALUES
|
||||
(1, 'The happy puppy runs merrily in the park', 'animals', 'Alice'),
|
||||
(2, 'A curious kitten jumps quickly over the fence', 'animals', 'Bob'),
|
||||
(3, 'The puppy catches a ball with great enthusiasm', 'sports', 'Alice'),
|
||||
(5, 'Puppy training requires patience and dedication', 'training', 'Alice')
|
||||
|
||||
CREATE INDEX ON my_docs USING fts (text)
|
||||
```
|
||||
|
||||
<Warning>
|
||||
Phrase queries need position information, which costs extra index size --
|
||||
add `WITH (with_position = true)` to `CREATE INDEX`. Without it,
|
||||
[phrase queries](#phrase) find nothing. See
|
||||
[FTS parameters](/indexing/fts-index#fts-parameters).
|
||||
</Warning>
|
||||
|
||||
## Basic search {#basic}
|
||||
|
||||
```python
|
||||
from lancedb.query import MatchQuery
|
||||
|
||||
query = MatchQuery("puppy", "text")
|
||||
json_query = query.to_json()
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT id, text, category FROM fts('my_docs', '<json_query>')
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
By default the top 5 matches come back in **arbitrary order** -- select
|
||||
`_score` and order by it for relevance ranking:
|
||||
|
||||
```sql
|
||||
SELECT id, text, _score FROM fts('my_docs', '<json_query>')
|
||||
ORDER BY _score DESC
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
`_score` is a pseudo-column carrying the query's BM25 relevance for the row;
|
||||
it behaves like any other column, but exists only in a relation that came
|
||||
from `fts()`. See [Data types](/sql/data-types#pseudo-columns).
|
||||
|
||||
## Fuzzy {#fuzzy}
|
||||
|
||||
Tolerates typos, up to a character-edit budget:
|
||||
|
||||
```python
|
||||
MatchQuery("pupy", "text", fuzziness=2)
|
||||
```
|
||||
|
||||
## Phrase {#phrase}
|
||||
|
||||
Exact phrase, or with `slop` to allow words between terms:
|
||||
|
||||
```python
|
||||
PhraseQuery("happy puppy", "text")
|
||||
PhraseQuery("puppy park", "text", slop=2) # up to 2 words between them
|
||||
```
|
||||
|
||||
## Boolean {#boolean}
|
||||
|
||||
Combine queries with `&` (AND) or `|` (OR):
|
||||
|
||||
```python
|
||||
MatchQuery("puppy", "text") & MatchQuery("happy", "text")
|
||||
MatchQuery("puppy", "text") | MatchQuery("kitten", "text")
|
||||
```
|
||||
|
||||
## Boost {#boost}
|
||||
|
||||
Weight a query up or down relative to another:
|
||||
|
||||
```python
|
||||
from lancedb.query import BoostQuery
|
||||
|
||||
BoostQuery(
|
||||
positive=MatchQuery("puppy", "text"),
|
||||
negative=MatchQuery("kitten", "text"),
|
||||
negative_boost=0.2,
|
||||
)
|
||||
```
|
||||
|
||||
## Multi-match {#multi-match}
|
||||
|
||||
Search several columns at once, optionally weighting them differently:
|
||||
|
||||
```python
|
||||
from lancedb.query import MultiMatchQuery
|
||||
|
||||
MultiMatchQuery("puppy", ["text", "category"])
|
||||
MultiMatchQuery("puppy", ["text", "category"], boosts=[2.0, 1.0]) # text weighted 2x
|
||||
```
|
||||
|
||||
## Combining with SQL {#combining}
|
||||
|
||||
`fts()` is a relation, so it composes with the rest of the SQL surface --
|
||||
filter, join, or aggregate over it same as any table:
|
||||
|
||||
```sql
|
||||
SELECT id, text, category FROM fts('my_docs', '<json_query>')
|
||||
WHERE category = 'animals'
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Full-Text Search" icon="magnifying-glass" href="/search/full-text-search">
|
||||
The query-builder equivalent, and every query form in full.
|
||||
</Card>
|
||||
<Card title="FTS Index" icon="bolt" href="/indexing/fts-index">
|
||||
Tokenizers, and the position information phrase queries need.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: "Joins"
|
||||
sidebarTitle: "Joins"
|
||||
description: "Combining rows from more than one table, including across namespaces."
|
||||
icon: link
|
||||
keywords: ["join", "inner", "left", "outer", "cross", "namespace"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Joining is the other thing the query builders cannot do. A question spanning two
|
||||
tables is a SQL question.
|
||||
|
||||
## The join types {#types}
|
||||
|
||||
```sql
|
||||
SELECT d.title, c.name
|
||||
FROM documents AS d
|
||||
JOIN categories AS c ON c.id = d.category_id
|
||||
```
|
||||
|
||||
| | What it keeps |
|
||||
| --- | --- |
|
||||
| `[INNER] JOIN` | rows matching on both sides |
|
||||
| `LEFT [OUTER] JOIN` | every left row, nulls where the right has no match |
|
||||
| `RIGHT [OUTER] JOIN` | every right row |
|
||||
| `FULL [OUTER] JOIN` | every row from both |
|
||||
| `CROSS JOIN` | every combination -- the product of the two |
|
||||
|
||||
`USING (col)` is shorthand for an equality join on a shared column name, and
|
||||
avoids repeating it in the select list.
|
||||
|
||||
## Across namespaces {#namespaces}
|
||||
|
||||
Tables in different [namespaces](/management/namespace) join by their dotted paths -- the
|
||||
catalog is one catalog, so there is nothing to federate:
|
||||
|
||||
```sql
|
||||
SELECT c.id, u.email
|
||||
FROM prod.events.clicks AS c
|
||||
JOIN prod.reference.users AS u ON u.id = c.user_id
|
||||
```
|
||||
|
||||
## Joining a search result {#search}
|
||||
|
||||
[`fts()`](/sql/functions#fts) is a table function, so its result is a relation
|
||||
and joins like any other:
|
||||
|
||||
```sql
|
||||
SELECT c.name, count(*) AS hits
|
||||
FROM fts('my_docs', 'puppy') AS m
|
||||
JOIN categories AS c ON c.id = m.category_id
|
||||
GROUP BY c.name
|
||||
ORDER BY hits DESC
|
||||
```
|
||||
|
||||
That composition is the reason full-text search is a relation rather than a
|
||||
predicate -- a `WHERE` clause has nowhere to put the relevance score.
|
||||
|
||||
## Performance {#performance}
|
||||
|
||||
A join key with a [scalar index](/indexing/scalar-index) on it lets the planner
|
||||
look rows up rather than scan for them. Check with
|
||||
[`explain_plan`](/search/explain): an indexed key shows a `ScalarIndexQuery`,
|
||||
an unindexed one shows a `LanceScan` feeding the join.
|
||||
|
||||
<Warning>
|
||||
Avoid selecting blob or vector columns you do not need on either side of a join.
|
||||
The join multiplies rows before projection is applied in some plans, so a wide
|
||||
column can be materialized many more times than the result has rows.
|
||||
</Warning>
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: "Random sampling"
|
||||
sidebarTitle: "Sampling"
|
||||
description: "TABLESAMPLE: draw a random subset of a table, by rows or by fragments."
|
||||
icon: dice
|
||||
keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
`TABLESAMPLE` draws a random subset of a table. On a lakehouse table that is
|
||||
often the difference between a question you can afford to ask and one you
|
||||
cannot -- an aggregate over 1% of forty billion rows answers in a fraction of
|
||||
the time, and for most estimates that is enough.
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE (10) -- 10 percent
|
||||
SELECT * FROM documents TABLESAMPLE (500 ROWS) -- 500 rows
|
||||
```
|
||||
|
||||
A bare number is a **percentage**. Add `ROWS` for a count instead.
|
||||
|
||||
## Rows or fragments {#methods}
|
||||
|
||||
The standard calls these two methods of one clause. They are different
|
||||
operations, and the difference matters on Lance:
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE BERNOULLI (10)
|
||||
SELECT * FROM documents TABLESAMPLE SYSTEM (10)
|
||||
```
|
||||
|
||||
| Method | Aliases | What it picks |
|
||||
| --- | --- | --- |
|
||||
| `BERNOULLI` | `ROW` | rows, each considered independently |
|
||||
| `SYSTEM` | `BLOCK` | **Lance fragments**, and takes whatever rows they hold |
|
||||
|
||||
`BERNOULLI` gives a genuinely uniform sample and reads across the whole table to
|
||||
get it. `SYSTEM` reads far less -- it skips entire fragments -- but the sample is
|
||||
only as uniform as the data's layout. If rows were written in an order that
|
||||
correlates with what you are measuring, a fragment sample inherits that
|
||||
correlation.
|
||||
|
||||
Reach for `SYSTEM` when you want a cheap look at the data and `BERNOULLI` when
|
||||
the number you compute has to be defensible. Omitting the method gives
|
||||
`BERNOULLI`.
|
||||
|
||||
## Repeatability {#repeatable}
|
||||
|
||||
Without a seed, every execution resamples -- running the same query twice gives
|
||||
different rows. `REPEATABLE` fixes the draw:
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE (10) REPEATABLE (42)
|
||||
```
|
||||
|
||||
This is what makes a sample usable as a **fixed evaluation set**: the same seed
|
||||
against the same data gives the same rows, so a metric computed today is
|
||||
comparable with one computed last week.
|
||||
|
||||
## Where the clause can go {#placement}
|
||||
|
||||
It attaches to a table in `FROM`, including inside a subquery, a CTE, or either
|
||||
side of a join:
|
||||
|
||||
```sql
|
||||
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
|
||||
|
||||
SELECT * FROM a TABLESAMPLE (5 ROWS) JOIN b ON a.id = b.id
|
||||
|
||||
SELECT * FROM t a TABLESAMPLE (10) JOIN t b TABLESAMPLE (10) ON a.id = b.id
|
||||
```
|
||||
|
||||
Each side of a join samples independently. Sampling both sides of a join
|
||||
shrinks the *result* far more than either fraction suggests, because a row only
|
||||
survives if both its sides were drawn.
|
||||
|
||||
## What is refused {#refused}
|
||||
|
||||
Sampling is semantics, not a hint. A clause that were read and then not applied
|
||||
would be a wrong answer rather than a missing feature, so anything that cannot
|
||||
be honoured exactly is an error:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `TABLESAMPLE (101)` | a percentage outside `0..=100` is refused, not clamped |
|
||||
| `TABLESAMPLE (1.5 ROWS)` | a row count must be whole |
|
||||
| `TABLESAMPLE (BUCKET 1 OUT OF 4)` | bucket sampling is not supported |
|
||||
| the `SAMPLE` spelling | use `TABLESAMPLE` |
|
||||
| the `OFFSET` modifier | not supported |
|
||||
|
||||
<Note>
|
||||
`TABLESAMPLE` is also refused on a WAL-backed table, where the rows a sample
|
||||
would draw from are not all in the fragments yet.
|
||||
</Note>
|
||||
|
||||
## Sampling outside SQL {#sdk}
|
||||
|
||||
For a training loop rather than a query, the SDKs read a table in a shuffled
|
||||
order without going through SQL at all -- see [Training](/training/data-loading), where a
|
||||
permutation drives the data loader.
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
title: "SELECT"
|
||||
sidebarTitle: "SELECT"
|
||||
description: "Projection, filtering, ordering, and composing queries out of other queries."
|
||||
icon: table-columns
|
||||
keywords: ["select", "where", "order by", "limit", "cte", "union", "distinct"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
[WITH cte AS (...)]
|
||||
SELECT [ALL | DISTINCT] select_expr [, ...]
|
||||
FROM from_item [[JOIN ...] ON ...]
|
||||
[WHERE predicate]
|
||||
[GROUP BY expr [, ...]] [HAVING predicate]
|
||||
[WINDOW name AS (...)]
|
||||
[ORDER BY expr [ASC | DESC] [NULLS FIRST | LAST] [, ...]]
|
||||
[LIMIT n] [OFFSET n]
|
||||
```
|
||||
|
||||
## Projection {#projection}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL
|
||||
SELECT id, title, identifier FROM documents
|
||||
```
|
||||
|
||||
```python Python icon="python"
|
||||
tbl.search().select(["id", "title", "identifier"]).to_arrow()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await tbl.query().select(["id", "title", "identifier"]).toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Projection pushes into the scan in all three -- an unselected column is never
|
||||
read off storage. On a table of images or video that is the difference between a
|
||||
fast query and a slow one, so **prefer naming columns over `SELECT *`**.
|
||||
|
||||
Row metadata is available as [pseudo-columns](/sql/data-types#pseudo-columns)
|
||||
(`_rowid`, `_rowaddr`, `_rowoffset`) but only when named, so they stay out of
|
||||
`SELECT *`.
|
||||
|
||||
`DISTINCT` deduplicates the projected rows.
|
||||
|
||||
## Filtering {#filtering}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL
|
||||
SELECT * FROM documents
|
||||
WHERE identifier > 0 AND identifier < 1000000
|
||||
```
|
||||
|
||||
```python Python icon="python"
|
||||
tbl.search().where("identifier > 0 AND identifier < 1000000").to_arrow()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await tbl.query().where("identifier > 0 AND identifier < 1000000").toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The SDK `where` takes the same expression language, so a predicate moves between
|
||||
them unchanged. A predicate on an indexed column becomes a `ScalarIndexQuery`
|
||||
rather than a scan -- confirm with [`explain_plan`](/search/explain). See
|
||||
[Filtering](/search/filtering) for the operators and their pushdown behaviour.
|
||||
|
||||
## Ordering and limiting {#order-limit}
|
||||
|
||||
```sql
|
||||
SELECT id, title FROM documents
|
||||
ORDER BY created_at DESC NULLS LAST
|
||||
LIMIT 100 OFFSET 200
|
||||
```
|
||||
|
||||
`ORDER BY` on a large table is a sort over the whole result, so pair it with a
|
||||
`LIMIT` unless you genuinely need every row ordered.
|
||||
|
||||
## Set operations {#set-operations}
|
||||
|
||||
`UNION`, `UNION ALL`, `INTERSECT` and `EXCEPT` combine whole queries:
|
||||
|
||||
```sql
|
||||
SELECT id FROM documents WHERE category = 'film'
|
||||
UNION ALL
|
||||
SELECT id FROM archived_documents WHERE category = 'film'
|
||||
```
|
||||
|
||||
`UNION` deduplicates and `UNION ALL` does not, which makes `UNION ALL` the
|
||||
cheaper one when you already know the inputs are disjoint.
|
||||
|
||||
## Common table expressions {#ctes}
|
||||
|
||||
```sql
|
||||
WITH recent AS (
|
||||
SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
|
||||
)
|
||||
SELECT category, count(*) FROM recent GROUP BY category
|
||||
```
|
||||
|
||||
A CTE is scoped to the statement. One you want to keep is a
|
||||
[view](/management/view); one whose rows you want stored is a
|
||||
[materialized view](/management/materialized-view).
|
||||
|
||||
## Subqueries {#subqueries}
|
||||
|
||||
Subqueries are allowed anywhere an expression or a relation is -- in `FROM`, in
|
||||
`WHERE` with `IN` or `EXISTS`, and as a scalar in the select list:
|
||||
|
||||
```sql
|
||||
SELECT title
|
||||
FROM documents
|
||||
WHERE category_id IN (SELECT id FROM categories WHERE active)
|
||||
```
|
||||
|
||||
## Namespaced tables {#namespaces}
|
||||
|
||||
A table inside a [namespace](/management/namespace) is written as a dotted path:
|
||||
|
||||
```sql
|
||||
SELECT * FROM prod.events.clicks
|
||||
```
|
||||
|
||||
## Reading the plan {#plans}
|
||||
|
||||
`EXPLAIN` and `EXPLAIN ANALYZE` work as DataFusion defines them, and produce the
|
||||
same operators the SDK methods do:
|
||||
|
||||
```sql
|
||||
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
|
||||
```
|
||||
|
||||
See [Explain](/search/explain) and [Explain Analyze](/search/explain).
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: "Subqueries"
|
||||
sidebarTitle: "Subqueries"
|
||||
description: "Queries inside queries: scalar, IN, EXISTS, correlated, and derived tables."
|
||||
icon: code-branch
|
||||
keywords: ["subquery", "exists", "in", "correlated", "derived table", "lateral"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A subquery is allowed anywhere an expression or a relation is.
|
||||
|
||||
## As a relation {#derived}
|
||||
|
||||
```sql
|
||||
SELECT * FROM (
|
||||
SELECT category, count(*) AS n FROM documents GROUP BY category
|
||||
) AS counts
|
||||
WHERE n > 100
|
||||
```
|
||||
|
||||
A derived table needs an alias. It can be [sampled](/sql/dql/sampling) like a base
|
||||
table:
|
||||
|
||||
```sql
|
||||
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
|
||||
```
|
||||
|
||||
## As a scalar {#scalar}
|
||||
|
||||
```sql
|
||||
SELECT title, (SELECT count(*) FROM revisions r WHERE r.doc_id = d.id) AS revisions
|
||||
FROM documents AS d
|
||||
```
|
||||
|
||||
A scalar subquery must return one row and one column. Returning more is an
|
||||
error rather than a silently-taken first row.
|
||||
|
||||
## IN and EXISTS {#in-exists}
|
||||
|
||||
```sql
|
||||
SELECT title FROM documents
|
||||
WHERE category_id IN (SELECT id FROM categories WHERE active)
|
||||
|
||||
SELECT title FROM documents AS d
|
||||
WHERE EXISTS (SELECT 1 FROM revisions r WHERE r.doc_id = d.id)
|
||||
```
|
||||
|
||||
`EXISTS` stops at the first matching row, so it is the cheaper test when you
|
||||
only care whether anything matches. `IN` builds the full set.
|
||||
|
||||
`NOT IN` and `NOT EXISTS` differ on nulls: `NOT IN` against a set containing a
|
||||
null yields no rows, because the comparison is unknown rather than false. Prefer
|
||||
`NOT EXISTS` unless you have ruled nulls out.
|
||||
|
||||
## Correlated subqueries {#correlated}
|
||||
|
||||
A subquery referencing a column from the enclosing query — `d.id` above — is
|
||||
correlated, and is conceptually evaluated per outer row. The planner usually
|
||||
rewrites it into a join.
|
||||
|
||||
<Note>
|
||||
Check with [`EXPLAIN`](/sql/dql/explain) when a correlated subquery is slow. A plan
|
||||
showing a join has been optimized; one showing a repeated scan has not, and
|
||||
rewriting it as an explicit [join](/sql/dql/join) is the fix.
|
||||
</Note>
|
||||
|
||||
## CTEs as an alternative {#ctes}
|
||||
|
||||
A [CTE](/sql/dql/select#ctes) is a named subquery, and is usually more readable when
|
||||
the same subquery appears twice or when nesting gets deep.
|
||||
|
||||
```sql
|
||||
WITH active AS (SELECT id FROM categories WHERE active)
|
||||
SELECT title FROM documents WHERE category_id IN (SELECT id FROM active)
|
||||
```
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
title: "Window functions"
|
||||
sidebarTitle: "Window Functions"
|
||||
description: "Compute over a set of rows related to the current one, without collapsing them."
|
||||
icon: chart-simple
|
||||
keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A window function computes across a set of rows related to the current one and
|
||||
returns a value **per row**. That is the difference from
|
||||
[aggregation](/sql/dql/aggregation), which collapses the rows it covers.
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
title,
|
||||
score,
|
||||
row_number() OVER (PARTITION BY category ORDER BY score DESC) AS rank
|
||||
FROM documents
|
||||
```
|
||||
|
||||
Every row survives; each one gains its rank within its category.
|
||||
|
||||
## The window {#window}
|
||||
|
||||
```sql
|
||||
OVER (
|
||||
[PARTITION BY expr [, ...]]
|
||||
[ORDER BY expr [, ...]]
|
||||
[ROWS | RANGE BETWEEN frame_start AND frame_end]
|
||||
)
|
||||
```
|
||||
|
||||
`PARTITION BY` splits the rows into independent groups. `ORDER BY` orders them
|
||||
inside each partition -- required by the ranking functions, since a rank without
|
||||
an order is meaningless. The frame clause narrows the window further, which is
|
||||
how running totals and moving averages are expressed:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
created_at,
|
||||
sum(amount) OVER (ORDER BY created_at ROWS UNBOUNDED PRECEDING) AS running_total,
|
||||
avg(amount) OVER (ORDER BY created_at ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS trailing_7
|
||||
FROM transactions
|
||||
```
|
||||
|
||||
A `WINDOW` clause names a definition when several functions share one:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
row_number() OVER w,
|
||||
lag(score) OVER w
|
||||
FROM documents
|
||||
WINDOW w AS (PARTITION BY category ORDER BY score DESC)
|
||||
```
|
||||
|
||||
## The functions {#functions}
|
||||
|
||||
Registered in DataFusion 54.
|
||||
|
||||
**Ranking**
|
||||
|
||||
```text
|
||||
row_number rank dense_rank percent_rank cume_dist ntile
|
||||
```
|
||||
|
||||
**Offset and positional**
|
||||
|
||||
```text
|
||||
lag lead first_value last_value nth_value
|
||||
```
|
||||
|
||||
**Aggregates**
|
||||
|
||||
Any [aggregate function](/sql/dql/aggregation#functions) used with `OVER` becomes a
|
||||
window function — that is how running totals and moving averages are written.
|
||||
|
||||
## Deduplicating with a window {#dedup}
|
||||
|
||||
The common practical use: keep one row per key, chosen by some ordering. A
|
||||
window ranks, and an outer query filters:
|
||||
|
||||
```sql
|
||||
WITH ranked AS (
|
||||
SELECT *, row_number() OVER (PARTITION BY doc_id ORDER BY version DESC) AS rn
|
||||
FROM revisions
|
||||
)
|
||||
SELECT * FROM ranked WHERE rn = 1
|
||||
```
|
||||
|
||||
The outer query is needed because a window function cannot appear in `WHERE` --
|
||||
windows are computed after filtering, so the predicate would have nothing to
|
||||
read.
|
||||
|
||||
<Note>
|
||||
Window functions sort within each partition, so on a large table they are among
|
||||
the more expensive things you can ask for. Filter first, and check the cost with
|
||||
[Explain Analyze](/search/explain).
|
||||
</Note>
|
||||
Reference in New Issue
Block a user