mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-31 15:56:52 +00:00
Jack Ye
ddb318567a
## Summary - Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause, and `dedupe()`/`duplicate_pairs()` (verified against current sophon source; these were recently renamed from `vector_dedup`/`vector_duplicate_pairs`). Fix the blob-column default-projection claim and add the previously undocumented `blob_descriptor()`. - Enrich the gated Functions draft (still `.mintignore`'d, pre-release): chained Functions over blob outputs, GPU-produced blobs, Secrets, multi-input/output binding, `code=[module]` packaging. - Upgrade the materialized-view and computed-column auto-refresh drafts with precise, verified catch-up semantics, and add the missing `ALTER MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the automatic-refresh scheduler is still dev/staging-only). - Thread one running example (video frames → GPU extract/caption → dedup → auto-refreshing materialized view) through Functions, materialized views, and the LeRobotDataset integration guide instead of a separate tutorial. - Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout (52 files); activate the existing availability-comparison mechanism on `basics/offerings.mdx` for a full feature-by-feature comparison table.
125 lines
4.6 KiB
Plaintext
125 lines
4.6 KiB
Plaintext
---
|
|
title: "Functions"
|
|
sidebarTitle: "Functions"
|
|
description: "Where to find each kind of function, and the dialect version they come from."
|
|
icon: function
|
|
keywords: ["functions", "datafusion", "scalar", "aggregate", "window", "fts", "table function"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org),
|
|
currently **version 54**. The function set is DataFusion's default registration
|
|
plus what Lance adds.
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| [Scalar functions](/sql/scalar-functions) | per row: math, string, regex, date/time, array, struct, JSON |
|
|
| [Aggregate functions](/sql/dql/aggregation#functions) | per group: `count`, `sum`, statistical, approximate |
|
|
| [Window functions](/sql/dql/window#functions) | per window: ranking, offset, running aggregates |
|
|
| [Table functions](#table-functions) | produce a relation rather than a value |
|
|
|
|
<Note>
|
|
Function availability is version-specific. DataFusion adds functions in most
|
|
releases, so one documented on `main` may not exist in 54. When a name is not
|
|
recognised, check it against the reference for that version before assuming a
|
|
bug.
|
|
</Note>
|
|
|
|
## Table functions {#table-functions}
|
|
|
|
A table function appears in `FROM` and produces a relation, which is what lets
|
|
it return columns a scalar function has nowhere to put.
|
|
|
|
### `fts(table, query)` {#fts}
|
|
|
|
Full-text search is a table function rather than a predicate, because it
|
|
produces relevance alongside the matching rows. The following query assumes
|
|
`my_docs` has `id` and `text` columns and a full-text index on `text`; see the
|
|
[setup example](/sql/dql/fts#setup):
|
|
|
|
```sql
|
|
SELECT id, text, _score
|
|
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}')
|
|
ORDER BY _score DESC
|
|
LIMIT 5
|
|
```
|
|
|
|
The second argument is a string literal containing a JSON query object. A bare
|
|
query string such as `'puppy'` is rejected. Match, fuzzy, phrase, boolean, boost
|
|
and multi-match queries use this JSON encoding.
|
|
|
|
Its schema contains the dataset's columns plus `_score`, so every stored
|
|
column is available in the result. Projections push into the scan.
|
|
Being a relation is also what makes it compose: join it, filter it, aggregate
|
|
over it. This join additionally requires `my_docs.category_id` and a
|
|
`categories` table with `id` and `name` columns:
|
|
|
|
```sql
|
|
SELECT c.name, count(*) AS hits
|
|
FROM fts('my_docs', '{"match":{"column":"text","terms":"puppy"}}') AS m
|
|
JOIN categories AS c ON c.id = m.category_id
|
|
GROUP BY c.name
|
|
ORDER BY hits DESC
|
|
```
|
|
|
|
For a keyword test that needs no score, the
|
|
[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate is
|
|
index-accelerated and usable directly in `WHERE`.
|
|
|
|
See [Full-Text Search](/sql/dql/fts) for the fuzzy, phrase, boolean, boost and
|
|
multi-match query forms.
|
|
|
|
### `_score` {#score}
|
|
|
|
A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves
|
|
like any other column (order by it, filter on it, project it) and exists only
|
|
in a relation that came from `fts()`. See
|
|
[Data types](/sql/data-types#pseudo-columns).
|
|
|
|
### `dedupe` and `duplicate_pairs` {#dedupe}
|
|
|
|
Scan a vector column's [index](/indexing/vector-index) partitions for rows
|
|
closer than a distance threshold, without a full pairwise scan or a shuffle:
|
|
|
|
```sql
|
|
SELECT * FROM duplicate_pairs('frames', 'vec', distance_threshold => 0.02)
|
|
SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02)
|
|
```
|
|
|
|
<Note>
|
|
Earlier releases spelled these `vector_duplicate_pairs` and `vector_dedup`;
|
|
those names are rejected on current deployments.
|
|
</Note>
|
|
|
|
`duplicate_pairs` returns every close pair it finds, in a fixed schema.
|
|
`dedupe` instead resolves overlapping pairs into duplicate groups and returns
|
|
one representative row **in the table's own schema** per group, so its result
|
|
composes directly into `CREATE TABLE ... AS` or
|
|
`CREATE MATERIALIZED VIEW ... AS`. See [`DEDUPE`](/sql/dql/dedupe) and
|
|
[`DUPLICATE_PAIRS`](/sql/dql/duplicate-pairs) for the full input and output
|
|
schema, resolution grouping, and partition sampling.
|
|
|
|
### `generate_series` {#generate-series}
|
|
|
|
```sql
|
|
SELECT * FROM generate_series(1, 10)
|
|
```
|
|
|
|
Rows from a range, useful for building a dense axis to left-join sparse data
|
|
onto.
|
|
|
|
## What Lance accelerates {#accelerated}
|
|
|
|
Two function families use an index rather than scanning, which is the difference
|
|
between a fast predicate and a slow one:
|
|
|
|
| | Index used |
|
|
| --- | --- |
|
|
| [`contains_tokens`](/sql/scalar-functions#contains-tokens) | full-text index |
|
|
| [`json_get`, typed variants, `json_extract`](/sql/scalar-functions#json) | JSON scalar index |
|
|
|
|
Confirm with [`EXPLAIN`](/sql/dql/explain): an accelerated predicate shows an
|
|
index node rather than a `LanceScan` followed by a filter.
|