mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-16 16:52:34 +00:00
docs: correct data workflows, SQL and operations (#4534)
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The curation, SQL and operations pages contain outdated claims and examples that do not reliably work as written. This corrects those pages against the reviewed SDK and local server evidence, including prepared statements, background jobs, cleanup and monitoring. Integration guides are grouped with their examples in #4533. Function and Secret-object documentation stays unpublished until its release. The two Function pages are listed in `.mintignore` and left out of the navigation, and the sections about them on other pages are MDX comments. Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after #4533 is squash-merged. Local assembly, Mint validation and link checks pass, and the assembled site shows no Function or Secret-object content. A local release rehearsal that uncomments the drafts and restores the navigation also passes. Hold publication until this PR and lancedb/sophon#8688 are on `main`.
This commit is contained in:
Brendan Clement
@@ -6,11 +6,10 @@ icon: layer-group
|
||||
keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximate"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Aggregation is one of the two main reasons to reach for SQL rather than a query
|
||||
builder -- the SDKs project, filter and search, but they do not group. The other
|
||||
is [joins](/sql/dql/join).
|
||||
Use SQL aggregates to summarize records, either across a whole table or
|
||||
within groups. Common examples include counts by category, totals and averages.
|
||||
|
||||
## Grouping {#grouping}
|
||||
|
||||
@@ -22,9 +21,8 @@ HAVING count(*) > 10
|
||||
ORDER BY n DESC
|
||||
```
|
||||
|
||||
`WHERE` filters rows before grouping; `HAVING` filters groups after. Putting a
|
||||
row-level predicate in `HAVING` still gives the right answer but does more work,
|
||||
because rows that could have been discarded early are grouped first.
|
||||
`WHERE` filters rows before grouping; `HAVING` filters groups after. Use `WHERE` for row-level conditions and `HAVING` for conditions on aggregate
|
||||
results. The planner may push eligible predicates earlier in the query.
|
||||
|
||||
Grouping by several expressions produces one row per distinct combination:
|
||||
|
||||
@@ -83,8 +81,8 @@ array_agg string_agg
|
||||
|
||||
The approximate functions are the ones worth reaching for on a large table.
|
||||
`approx_distinct` answers "roughly how many distinct values" for a fraction of
|
||||
what `count(DISTINCT ...)` costs, which matters most where it is most expensive
|
||||
— a high-cardinality column.
|
||||
what `count(DISTINCT ...)` costs, which can help with high-cardinality columns. Measure the accuracy and runtime
|
||||
for your workload.
|
||||
|
||||
`DISTINCT` works inside an aggregate: `count(DISTINCT category)`. `FILTER`
|
||||
narrows one aggregate without narrowing the others:
|
||||
@@ -107,12 +105,16 @@ FROM documents TABLESAMPLE (1)
|
||||
GROUP BY category
|
||||
```
|
||||
|
||||
Scale the result by the inverse of the sample fraction, and treat it as an
|
||||
estimate -- a 1% sample of a skewed column will miss rare categories entirely.
|
||||
For a uniform percentage sample, counts and sums can be scaled by the inverse
|
||||
of the sample fraction to estimate whole-table values. Do not apply that scaling
|
||||
to averages, minima or maxima. Small samples can miss rare categories; consider
|
||||
the sampling method and uncertainty when interpreting the result.
|
||||
|
||||
## What aggregation does not do {#not}
|
||||
|
||||
Aggregates run over rows the query already produced. They cannot combine with
|
||||
[vector search](/search/vector-search) in one statement, because similarity
|
||||
search has no SQL form. Run the search in an SDK and aggregate its result, or
|
||||
materialize the search output first.
|
||||
Aggregates run over rows the query already produced. Whether a statement can
|
||||
include [vector search](/search/vector-search)
|
||||
is described under [SQL vector search](/basics/sql#no-vector-sql). If your server
|
||||
does not support that query form, run the
|
||||
search in an SDK and aggregate its result, or materialize the search output
|
||||
first.
|
||||
|
||||
@@ -6,7 +6,7 @@ icon: map
|
||||
keywords: ["explain", "explain analyze", "explain verbose", "query plan"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
```sql
|
||||
EXPLAIN SELECT * FROM documents WHERE identifier < 1000
|
||||
@@ -23,9 +23,10 @@ The result is a small relation with `plan_type` and `plan` columns, read
|
||||
through the same client you use for regular queries -- see
|
||||
[Connecting](/sql#connecting).
|
||||
|
||||
The operators are the same ones the SDK's
|
||||
[`explain_plan` and `analyze_plan`](/search/explain) methods produce, so the
|
||||
reasoning about index coverage and filter pushdown carries across unchanged:
|
||||
SQL plans and the SDK's
|
||||
[`explain_plan` and `analyze_plan`](/search/explain) output share scan and index
|
||||
operators. Use the same approach to inspect index coverage and filter pushdown;
|
||||
SQL can also add joins, aggregates and other operators:
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="What it shows" icon="map" href="/search/explain#inspect-a-query-plan">
|
||||
@@ -41,7 +42,7 @@ reasoning about index coverage and filter pushdown carries across unchanged:
|
||||
| Sign | Means |
|
||||
| --- | --- |
|
||||
| `ScalarIndexQuery` | the predicate used a scalar index |
|
||||
| `LanceScan` with high `bytes_read` | no index, or too many columns selected |
|
||||
| `LanceScan` with high `bytes_read` | inspect the selected columns, filter pushdown and index use |
|
||||
| `FilterExec` above a search | the filter ran after search rather than pushing into it |
|
||||
| `ANNSubIndex` / `ANNIvfPartition` | the vector index was used |
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ icon: magnifying-glass
|
||||
keywords: ["fts", "sql", "full text search", "fuzzy", "phrase", "boolean", "boost", "multi-match"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Full-text search is a **table function**, `fts(table, query)`, rather than a
|
||||
predicate -- it produces a relevance score alongside the matching rows, and a
|
||||
|
||||
@@ -6,10 +6,9 @@ icon: link
|
||||
keywords: ["join", "inner", "left", "outer", "cross", "namespace"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
Joining is the other thing the query builders cannot do. A question spanning two
|
||||
tables is a SQL question.
|
||||
Use SQL joins to combine rows from two or more tables.
|
||||
|
||||
## The join types {#types}
|
||||
|
||||
|
||||
@@ -6,12 +6,11 @@ icon: dice
|
||||
keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
`TABLESAMPLE` draws a random subset of a table. On a lakehouse table that is
|
||||
often the difference between a question you can afford to ask and one you
|
||||
cannot -- an aggregate over 1% of forty billion rows answers in a fraction of
|
||||
the time, and for most estimates that is enough.
|
||||
Use `TABLESAMPLE` to explore a subset of a table or estimate aggregate
|
||||
statistics. The sampling method determines how much data is read and whether
|
||||
the table's physical layout influences the sample.
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE (10) -- 10 percent
|
||||
@@ -41,8 +40,8 @@ only as uniform as the data's layout. If rows were written in an order that
|
||||
correlates with what you are measuring, a fragment sample inherits that
|
||||
correlation.
|
||||
|
||||
Reach for `SYSTEM` when you want a cheap look at the data and `BERNOULLI` when
|
||||
the number you compute has to be defensible. Omitting the method gives
|
||||
Use `SYSTEM` for a quick look at the data and `BERNOULLI` when you need
|
||||
independent row sampling. Omitting the method gives
|
||||
`BERNOULLI`.
|
||||
|
||||
## Repeatability {#repeatable}
|
||||
@@ -54,9 +53,10 @@ different rows. `REPEATABLE` fixes the draw:
|
||||
SELECT * FROM documents TABLESAMPLE (10) REPEATABLE (42)
|
||||
```
|
||||
|
||||
This is what makes a sample usable as a **fixed evaluation set**: the same seed
|
||||
against the same data gives the same rows, so a metric computed today is
|
||||
comparable with one computed last week.
|
||||
Use the same seed and table snapshot to reproduce a sampling run. For a fixed
|
||||
evaluation set, save the selected row IDs or materialize the sample and retain
|
||||
its version. A seed alone does not preserve membership after source changes
|
||||
or changes to the physical layout.
|
||||
|
||||
## Where the clause can go {#placement}
|
||||
|
||||
@@ -64,7 +64,7 @@ It attaches to a table in `FROM`, including inside a subquery, a CTE, or either
|
||||
side of a join:
|
||||
|
||||
```sql
|
||||
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
|
||||
SELECT * FROM (SELECT * FROM documents TABLESAMPLE (10)) d
|
||||
|
||||
SELECT * FROM a TABLESAMPLE (5 ROWS) JOIN b ON a.id = b.id
|
||||
|
||||
@@ -75,7 +75,7 @@ Each side of a join samples independently. Sampling both sides of a join
|
||||
shrinks the *result* far more than either fraction suggests, because a row only
|
||||
survives if both its sides were drawn.
|
||||
|
||||
## What is refused {#refused}
|
||||
## Unsupported sampling forms {#refused}
|
||||
|
||||
Sampling is semantics, not a hint. A clause that were read and then not applied
|
||||
would be a wrong answer rather than a missing feature, so anything that cannot
|
||||
@@ -86,6 +86,7 @@ be honoured exactly is an error:
|
||||
| `TABLESAMPLE (101)` | a percentage outside `0..=100` is refused, not clamped |
|
||||
| `TABLESAMPLE (1.5 ROWS)` | a row count must be whole |
|
||||
| `TABLESAMPLE (BUCKET 1 OUT OF 4)` | bucket sampling is not supported |
|
||||
| `(SELECT ...) d TABLESAMPLE (10)` | a subquery's result cannot be sampled; sample the table inside it |
|
||||
| the `SAMPLE` spelling | use `TABLESAMPLE` |
|
||||
| the `OFFSET` modifier | not supported |
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ icon: table-columns
|
||||
keywords: ["select", "where", "order by", "limit", "cte", "union", "distinct"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
```sql
|
||||
[WITH cte AS (...)]
|
||||
@@ -35,9 +35,10 @@ await tbl.query().select(["id", "title", "identifier"]).toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Projection pushes into the scan in all three -- an unselected column is never
|
||||
read off storage. On a table of images or video that is the difference between a
|
||||
fast query and a slow one, so **prefer naming columns over `SELECT *`**.
|
||||
Name the columns you need instead of using `SELECT *`, particularly for
|
||||
tables containing images, video or large vectors. Projection lets the engine
|
||||
avoid loading unnecessary columns, although columns used by filters, joins or
|
||||
other expressions may still need to be read.
|
||||
|
||||
Row metadata is available as [pseudo-columns](/sql/data-types#pseudo-columns)
|
||||
(`_rowid`, `_rowaddr`, `_rowoffset`) but only when named, so they stay out of
|
||||
@@ -63,8 +64,8 @@ await tbl.query().where("identifier > 0 AND identifier < 1000000").toArrow();
|
||||
</CodeGroup>
|
||||
|
||||
The SDK `where` takes the same expression language, so a predicate moves between
|
||||
them unchanged. A predicate on an indexed column becomes a `ScalarIndexQuery`
|
||||
rather than a scan -- confirm with [`explain_plan`](/search/explain). See
|
||||
them unchanged. A supported predicate can use a scalar index; confirm the chosen plan with
|
||||
[`explain_plan`](/search/explain). See
|
||||
[Filtering](/search/filtering) for the operators and their pushdown behaviour.
|
||||
|
||||
## Ordering and limiting {#order-limit}
|
||||
|
||||
@@ -6,7 +6,7 @@ icon: code-branch
|
||||
keywords: ["subquery", "exists", "in", "correlated", "derived table", "lateral"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A subquery is allowed anywhere an expression or a relation is.
|
||||
|
||||
@@ -19,11 +19,11 @@ SELECT * FROM (
|
||||
WHERE n > 100
|
||||
```
|
||||
|
||||
A derived table needs an alias. It can be [sampled](/sql/dql/sampling) like a base
|
||||
table:
|
||||
A derived table needs an alias. Its result cannot be
|
||||
[sampled](/sql/dql/sampling); sample the table inside it instead:
|
||||
|
||||
```sql
|
||||
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
|
||||
SELECT * FROM (SELECT * FROM documents TABLESAMPLE (10)) d
|
||||
```
|
||||
|
||||
## As a scalar {#scalar}
|
||||
@@ -33,8 +33,8 @@ SELECT title, (SELECT count(*) FROM revisions r WHERE r.doc_id = d.id) AS revisi
|
||||
FROM documents AS d
|
||||
```
|
||||
|
||||
A scalar subquery must return one row and one column. Returning more is an
|
||||
error rather than a silently-taken first row.
|
||||
A scalar subquery returns one column and at most one row. No rows produces
|
||||
`NULL`; more than one row is an error.
|
||||
|
||||
## IN and EXISTS {#in-exists}
|
||||
|
||||
@@ -46,8 +46,9 @@ SELECT title FROM documents AS d
|
||||
WHERE EXISTS (SELECT 1 FROM revisions r WHERE r.doc_id = d.id)
|
||||
```
|
||||
|
||||
`EXISTS` stops at the first matching row, so it is the cheaper test when you
|
||||
only care whether anything matches. `IN` builds the full set.
|
||||
Use `EXISTS` to test whether a matching row exists and `IN` to test membership
|
||||
in a set of values. The optimizer may rewrite either into a join; inspect
|
||||
`EXPLAIN` before choosing one for performance reasons.
|
||||
|
||||
`NOT IN` and `NOT EXISTS` differ on nulls: `NOT IN` against a set containing a
|
||||
null yields no rows, because the comparison is unknown rather than false. Prefer
|
||||
@@ -55,8 +56,8 @@ null yields no rows, because the comparison is unknown rather than false. Prefer
|
||||
|
||||
## Correlated subqueries {#correlated}
|
||||
|
||||
A subquery referencing a column from the enclosing query — `d.id` above — is
|
||||
correlated, and is conceptually evaluated per outer row. The planner usually
|
||||
A subquery referencing a column from the enclosing query, such as `d.id` above,
|
||||
is correlated, and is conceptually evaluated per outer row. The planner usually
|
||||
rewrites it into a join.
|
||||
|
||||
<Note>
|
||||
|
||||
@@ -6,7 +6,7 @@ icon: chart-simple
|
||||
keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs in LanceDB Enterprise.](/sql)
|
||||
|
||||
A window function computes across a set of rows related to the current one and
|
||||
returns a value **per row**. That is the difference from
|
||||
@@ -33,8 +33,8 @@ OVER (
|
||||
```
|
||||
|
||||
`PARTITION BY` splits the rows into independent groups. `ORDER BY` orders them
|
||||
inside each partition -- required by the ranking functions, since a rank without
|
||||
an order is meaningless. The frame clause narrows the window further, which is
|
||||
inside each partition. Supply an ordering for useful ranks, and a stable
|
||||
tie-breaker when selecting one row from several equal values. The frame clause narrows the window further, which is
|
||||
how running totals and moving averages are expressed:
|
||||
|
||||
```sql
|
||||
@@ -74,7 +74,7 @@ lag lead first_value last_value nth_value
|
||||
**Aggregates**
|
||||
|
||||
Any [aggregate function](/sql/dql/aggregation#functions) used with `OVER` becomes a
|
||||
window function — that is how running totals and moving averages are written.
|
||||
window function. This is how running totals and moving averages are written.
|
||||
|
||||
## Deduplicating with a window {#dedup}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user