mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
docs: seed the SQL reference from DataFusion 54, and cross-link it
The function catalogue is now real rather than a pointer. Scalar functions are listed by taxonomy -- math, string, unicode, regex, date/time, array, map, struct, crypto, type -- and the names come from what datafusion-functions 54 actually registers, not from what DataFusion documents on main. Aggregate and window pages gain the same treatment. Two things the audit corrected rather than copied. SHOW TABLES does not work. DataFusion implements it over information_schema, which this session does not enable, so it fails at plan time. It was documented as working here, on the SHOW page and in three Manage tables; all four are fixed, and the SQL overview now lists the absence alongside SHOW COLUMNS. Prepared statements are a Flight SQL protocol feature, not a PREPARE statement. A script carrying PREPARE ... AS SELECT will not run, while the same intent expressed through a client will. The new page says so rather than copying DataFusion's syntax. Every section of the Documentation tab that has a SQL counterpart now links to it -- 27 of 76 non-SQL pages, each pointing at the specific statement rather than at the reference index.
This commit is contained in:
Jack Ye
@@ -47,20 +47,54 @@ is before deciding to refresh it again.
|
||||
|
||||
## The function set {#functions}
|
||||
|
||||
The aggregates are DataFusion's, and the full list is in its
|
||||
[aggregate function reference](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html):
|
||||
Registered in DataFusion 54. Names below are what is actually available.
|
||||
|
||||
| Kind | Examples |
|
||||
| --- | --- |
|
||||
| General | `count`, `sum`, `min`, `max`, `avg` |
|
||||
| Statistical | `stddev`, `var`, `corr`, `covar`, `regr_*` |
|
||||
| Approximate | `approx_distinct`, `approx_percentile_cont`, `approx_median` |
|
||||
| Collecting | `array_agg`, `string_agg`, `first_value`, `last_value` |
|
||||
**General**
|
||||
|
||||
The approximate ones are worth knowing about on a large table:
|
||||
```text
|
||||
count sum min max avg median
|
||||
bit_and bit_or bit_xor bool_and bool_or
|
||||
first_value last_value nth_value grouping
|
||||
```
|
||||
|
||||
**Statistical**
|
||||
|
||||
```text
|
||||
stddev stddev_pop stddev_samp
|
||||
var var_pop var_samp
|
||||
corr covar_pop covar_samp
|
||||
regr_slope regr_intercept regr_count regr_r2
|
||||
regr_avgx regr_avgy regr_sxx regr_syy regr_sxy
|
||||
```
|
||||
|
||||
**Approximate**
|
||||
|
||||
```text
|
||||
approx_distinct approx_median
|
||||
approx_percentile_cont approx_percentile_cont_with_weight
|
||||
percentile_cont
|
||||
```
|
||||
|
||||
**Collecting**
|
||||
|
||||
```text
|
||||
array_agg string_agg
|
||||
```
|
||||
|
||||
The approximate functions are the ones worth reaching for on a large table.
|
||||
`approx_distinct` answers "roughly how many distinct values" for a fraction of
|
||||
what `count(DISTINCT ...)` costs, which matters when the column has high
|
||||
cardinality.
|
||||
what `count(DISTINCT ...)` costs, which matters most where it is most expensive
|
||||
— a high-cardinality column.
|
||||
|
||||
`DISTINCT` works inside an aggregate: `count(DISTINCT category)`. `FILTER`
|
||||
narrows one aggregate without narrowing the others:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
count(*) AS total,
|
||||
count(*) FILTER (WHERE score > 0.9) AS confident
|
||||
FROM documents
|
||||
```
|
||||
|
||||
## Aggregating a sample {#sampling}
|
||||
|
||||
|
||||
@@ -66,7 +66,7 @@ WHERE json_get_int(data, 'status') = 200
|
||||
```
|
||||
|
||||
`json_get`, its typed variants, and `json_extract` are **index-accelerated** by a
|
||||
JSON scalar index. See [Functions](/sql/functions#json).
|
||||
JSON scalar index. See [Functions](/sql/scalar-functions#json).
|
||||
|
||||
### Blobs {#blobs}
|
||||
|
||||
|
||||
+45
-128
@@ -1,90 +1,40 @@
|
||||
---
|
||||
title: "SQL functions and types"
|
||||
sidebarTitle: "Functions and types"
|
||||
description: "The dialect is DataFusion's. What that gives you, and what LanceDB adds on top."
|
||||
title: "Functions"
|
||||
sidebarTitle: "Functions"
|
||||
description: "Where to find each kind of function, and the dialect version they come from."
|
||||
icon: function
|
||||
keywords: ["sql functions", "datafusion", "scalar functions", "aggregate", "data types", "fts"]
|
||||
keywords: ["functions", "datafusion", "scalar", "aggregate", "window", "fts", "table function"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org),
|
||||
currently **version 54**. The dialect, the built-in functions, and the type
|
||||
system are DataFusion's, registered from its default set — so anything the
|
||||
DataFusion reference documents for that version is available, and this page does
|
||||
not reproduce it.
|
||||
currently **version 54**. The function set is DataFusion's default registration
|
||||
plus what Lance adds.
|
||||
|
||||
What it does document is the version you are querying against, where the
|
||||
authoritative lists are, and the parts that are LanceDB's rather than
|
||||
DataFusion's.
|
||||
|
||||
## Reference by category {#reference}
|
||||
|
||||
| | DataFusion reference |
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Math, conditional, string, binary string, regular expression, time and date | [Scalar functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) |
|
||||
| `count`, `sum`, `avg`, `min`, `max`, statistical, approximate | [Aggregate functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) |
|
||||
| `row_number`, `rank`, `lag`, `lead`, and the rest of `OVER (...)` | [Window functions](https://datafusion.apache.org/user-guide/sql/window_functions.html) |
|
||||
| Arrow-backed types, and how SQL type names map onto them | [Data types](https://datafusion.apache.org/user-guide/sql/data_types.html) |
|
||||
| `SELECT`, joins, `GROUP BY`, `ORDER BY`, `LIMIT`, CTEs, set operations | [SELECT syntax](https://datafusion.apache.org/user-guide/sql/select.html) |
|
||||
| [Scalar functions](/sql/scalar-functions) | per row — math, string, regex, date/time, array, struct, JSON |
|
||||
| [Aggregate functions](/sql/aggregation#functions) | per group — `count`, `sum`, statistical, approximate |
|
||||
| [Window functions](/sql/window#functions) | per window — ranking, offset, running aggregates |
|
||||
| [Table functions](#table-functions) | produce a relation rather than a value |
|
||||
|
||||
<Note>
|
||||
Function availability is version-specific: DataFusion adds functions in most
|
||||
releases, so a function documented on `main` may not exist in 54. When a
|
||||
function is not recognised, check it against the reference for the version above
|
||||
before assuming a bug.
|
||||
Function availability is version-specific. DataFusion adds functions in most
|
||||
releases, so one documented on `main` may not exist in 54. When a name is not
|
||||
recognised, check it against the reference for that version before assuming a
|
||||
bug.
|
||||
</Note>
|
||||
|
||||
## Types are Arrow types {#types}
|
||||
## Table functions {#table-functions}
|
||||
|
||||
There is no SQL type layer over Lance storage. The DataFusion table provider
|
||||
returns the dataset's Arrow schema directly — `Schema::from(dataset.schema())` —
|
||||
so a column is the same object to the query engine as it is to the format, and
|
||||
DataFusion's [type reference](https://datafusion.apache.org/user-guide/sql/data_types.html)
|
||||
describes it exactly.
|
||||
|
||||
That identity is mostly invisible and occasionally load-bearing. A vector column
|
||||
is a `FixedSizeList<Float32>`: selectable, testable for nullness, joinable like
|
||||
any other column — but arithmetic over it is list arithmetic, not scalar
|
||||
arithmetic. When the operation is a similarity comparison, use vector search
|
||||
rather than SQL.
|
||||
|
||||
### Pseudo-columns {#pseudo-columns}
|
||||
|
||||
Lance rows carry metadata the provider can expose as columns. They are appended
|
||||
to the schema only when asked for, so they do not appear in `SELECT *`:
|
||||
|
||||
| Column | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `_rowid` | `UInt64`, nullable | stable row identifier |
|
||||
| `_rowaddr` | `UInt64`, nullable | physical row address |
|
||||
| `_rowoffset` | `UInt64`, nullable | offset within the fragment |
|
||||
| `_score` | `Float32` | relevance, from [`fts()`](#fts) |
|
||||
|
||||
`_rowid` and `_rowaddr` are declared nullable, but not because a row lacks one —
|
||||
their validity bitmap is reused as a selection vector internally. Do not read a
|
||||
null there as meaningful.
|
||||
|
||||
`_rowid` is the identifier [materialized views](/feature-engineering/materialized-views)
|
||||
depend on, which is why a source table needs stable row ids enabled at creation.
|
||||
|
||||
### Blob columns change shape {#blobs}
|
||||
|
||||
A blob column can be read as its **binary contents** or as a **description** —
|
||||
a pointer plus metadata, without fetching the bytes. That choice changes the
|
||||
column's Arrow type, and therefore the schema a query sees.
|
||||
|
||||
The default reads blobs as descriptions and other binary columns as binary. A
|
||||
query that selects a blob column and expects bytes will get a struct instead
|
||||
unless the read is configured otherwise. Worth knowing before writing a
|
||||
predicate against one.
|
||||
|
||||
## What LanceDB adds {#extensions}
|
||||
A table function appears in `FROM` and produces a relation, which is what lets
|
||||
it return columns a scalar function has nowhere to put.
|
||||
|
||||
### `fts(table, query)` {#fts}
|
||||
|
||||
Full-text search is a **table function**, not a predicate — implemented as a
|
||||
DataFusion `TableProvider` over the dataset, so its result is a relation. That is
|
||||
what lets it return relevance alongside the matching rows, which a `WHERE` clause
|
||||
has nowhere to put:
|
||||
Full-text search is a table function rather than a predicate, because it
|
||||
produces relevance alongside the matching rows:
|
||||
|
||||
```sql
|
||||
SELECT id, text, _score
|
||||
@@ -93,13 +43,13 @@ ORDER BY _score DESC
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
The second argument is either a bare query string or a JSON query object for
|
||||
The second argument is a bare query string, or a JSON query object for the
|
||||
fuzzy, phrase, boolean, boost and multi-match forms.
|
||||
|
||||
Its schema is the dataset's, so every column of the table is available in the
|
||||
result — projections push down into the scan rather than being applied
|
||||
afterwards. Being a relation is also what makes it compose: join it, filter it,
|
||||
aggregate over it.
|
||||
result and projections push into the scan rather than being applied afterwards.
|
||||
Being a relation is also what makes it compose — join it, filter it, aggregate
|
||||
over it:
|
||||
|
||||
```sql
|
||||
SELECT c.name, count(*) AS hits
|
||||
@@ -109,68 +59,35 @@ GROUP BY c.name
|
||||
ORDER BY hits DESC
|
||||
```
|
||||
|
||||
For a keyword test that needs no score, the
|
||||
[`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate is
|
||||
index-accelerated and usable directly in `WHERE`.
|
||||
|
||||
### `_score` {#score}
|
||||
|
||||
A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves
|
||||
like a normal column — order by it, filter on it, project it — and exists only
|
||||
in a relation that came from `fts()`.
|
||||
like any other column — order by it, filter on it, project it — and exists only
|
||||
in a relation that came from `fts()`. See
|
||||
[Data types](/sql/data-types#pseudo-columns).
|
||||
|
||||
### `contains_tokens(column, tokens)` {#contains-tokens}
|
||||
|
||||
A predicate rather than a relation, for when you want keyword matching inside a
|
||||
`WHERE` clause and do not need relevance:
|
||||
### `generate_series` {#generate-series}
|
||||
|
||||
```sql
|
||||
SELECT * FROM my_docs WHERE contains_tokens(text, 'fox jumps dog')
|
||||
SELECT * FROM generate_series(1, 10)
|
||||
```
|
||||
|
||||
It is true when the column contains **all** the tokens, which are split on
|
||||
whitespace and punctuation. Equivalent to an `fts()` match query with fuzziness
|
||||
off, `AND` semantics, and the simple tokenizer.
|
||||
Rows from a range, useful for building a dense axis to left-join sparse data
|
||||
onto.
|
||||
|
||||
**It is index-accelerated.** When the column has a full-text index that suits
|
||||
the query, the planner uses it instead of scanning. That is the reason to prefer
|
||||
it over `LIKE` for keyword tests.
|
||||
## What Lance accelerates {#accelerated}
|
||||
|
||||
Use `fts()` when you need `_score` or the fuzzy, phrase, or boost forms; use
|
||||
`contains_tokens` when you just need a boolean.
|
||||
Two function families use an index rather than scanning, which is the difference
|
||||
between a fast predicate and a slow one:
|
||||
|
||||
### JSON functions {#json}
|
||||
|
||||
Lance registers a set of JSON accessors over string columns holding JSON:
|
||||
|
||||
| Function | Returns |
|
||||
| | Index used |
|
||||
| --- | --- |
|
||||
| `json_get(col, key)` | the value at `key`, still JSON |
|
||||
| `json_get_string(col, key)` | `Utf8` |
|
||||
| `json_get_int(col, key)` | `Int64` |
|
||||
| `json_get_float(col, key)` | `Float64` |
|
||||
| `json_get_bool(col, key)` | `Boolean` |
|
||||
| `json_extract(col, path)` | `Utf8`, by path |
|
||||
| `json_extract_with_type(col, path, type)` | the requested type |
|
||||
| `json_exists(col, path)` | `Boolean` |
|
||||
| `json_array_contains(col, value)` | `Boolean` |
|
||||
| `json_array_length(col)` | length |
|
||||
| [`contains_tokens`](/sql/scalar-functions#contains-tokens) | full-text index |
|
||||
| [`json_get`, typed variants, `json_extract`](/sql/scalar-functions#json) | JSON scalar index |
|
||||
|
||||
`json_get` returns JSON, so accessors nest for a nested document:
|
||||
|
||||
```sql
|
||||
SELECT * FROM events
|
||||
WHERE json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice'
|
||||
```
|
||||
|
||||
**`json_get`, its typed variants, and `json_extract` are index-accelerated** by a
|
||||
JSON scalar index on the column. The others evaluate per row.
|
||||
|
||||
<Note>
|
||||
Lance also defines geospatial `st_*` functions, but they are behind an optional
|
||||
build feature that LanceDB does not ship. Calling one returns an error telling
|
||||
you the feature is missing rather than an unknown-function error.
|
||||
</Note>
|
||||
|
||||
## Reaching a function LanceDB does not name {#escape-hatch}
|
||||
|
||||
Nothing here wraps or restricts the DataFusion set — a query reaches any
|
||||
function in the reference above by writing it. If a name is not recognised,
|
||||
check it against the reference for the version this page pins rather than
|
||||
assuming it is unsupported.
|
||||
Confirm with [`EXPLAIN`](/sql/explain): an accelerated predicate shows an index
|
||||
node rather than a `LanceScan` followed by a filter.
|
||||
|
||||
@@ -69,7 +69,9 @@ endpoint, so any Flight SQL client reaches it.
|
||||
| | |
|
||||
| --- | --- |
|
||||
| [Operators](/sql/operators) | comparison, logical, arithmetic, literals, precedence |
|
||||
| [Functions](/sql/functions) | the function set, and what Lance accelerates |
|
||||
| [Functions](/sql/functions) | where to find each kind, and table functions |
|
||||
| [Scalar functions](/sql/scalar-functions) | the per-row catalogue, and what Lance accelerates |
|
||||
| [Prepared statements](/sql/prepared-statements) | planning once, executing many times |
|
||||
|
||||
## What SQL cannot do {#cannot}
|
||||
|
||||
@@ -77,9 +79,11 @@ endpoint, so any Flight SQL client reaches it.
|
||||
| --- | --- |
|
||||
| Vector search | no SQL form -- a query-builder operation |
|
||||
| Merge insert | no SQL form -- needs a source relation and a per-match policy |
|
||||
| `information_schema` | not enabled, so `SHOW TABLES` and `SHOW COLUMNS` fail |
|
||||
| `PREPARE` / `EXECUTE` | preparation is a [protocol feature](/sql/prepared-statements), not a statement |
|
||||
| Run on a local database | no engine to plan the statement |
|
||||
|
||||
Full-text search **is** available, as the [`fts()`](/sql/functions#fts) table
|
||||
function and the [`contains_tokens`](/sql/functions#contains-tokens) predicate.
|
||||
function and the [`contains_tokens`](/sql/scalar-functions#contains-tokens) predicate.
|
||||
|
||||
See [Limits](/basics/limits#sql) for the bounds these pages enforce.
|
||||
|
||||
@@ -24,7 +24,7 @@ the operands are Lance columns.
|
||||
| `~` `~*` `!~` | regular expression match |
|
||||
|
||||
<Note>
|
||||
For keyword matching on a text column, [`contains_tokens`](/sql/functions#contains-tokens)
|
||||
For keyword matching on a text column, [`contains_tokens`](/sql/scalar-functions#contains-tokens)
|
||||
is index-accelerated where `LIKE` is not. Reach for `LIKE` only when you need
|
||||
its wildcard semantics.
|
||||
</Note>
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: "Prepared statements"
|
||||
sidebarTitle: "Prepared Statements"
|
||||
description: "Preparing a statement once and executing it many times, over the Flight SQL protocol."
|
||||
icon: bolt
|
||||
keywords: ["prepared statement", "flight sql", "parameters", "bind"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A prepared statement is planned once and executed many times, which saves
|
||||
re-planning a statement whose shape does not change between runs.
|
||||
|
||||
## It is a protocol feature, not a statement {#protocol}
|
||||
|
||||
<Warning>
|
||||
There is no SQL `PREPARE` / `EXECUTE` here. Preparation is part of the
|
||||
**Arrow Flight SQL protocol**, so you reach it through your client rather than
|
||||
by sending a statement.
|
||||
</Warning>
|
||||
|
||||
That distinction matters when porting SQL from another engine: a script
|
||||
containing `PREPARE name AS SELECT ...` will not run, while the same intent
|
||||
expressed through a Flight SQL client will.
|
||||
|
||||
## The flow {#flow}
|
||||
|
||||
Every Flight SQL client exposes the same three steps, whatever it calls them:
|
||||
|
||||
1. **Prepare** — send the SQL, get back a handle and the result schema. The
|
||||
statement is parsed and planned at this point, so a syntax or name error
|
||||
surfaces here rather than at execution.
|
||||
2. **Bind and execute** — supply parameter values for the handle and read the
|
||||
result. Repeat as often as you like.
|
||||
3. **Close** — release the handle. Servers do not hold them forever.
|
||||
|
||||
Knowing that the plan is built at step 1 is the practical part: if a statement
|
||||
is slow to prepare and fast to execute, the cost is planning, and reusing the
|
||||
handle is exactly the right fix.
|
||||
|
||||
## Parameters {#parameters}
|
||||
|
||||
Parameters are placeholders in the SQL text, bound to an Arrow record batch at
|
||||
execution. Binding rather than interpolating is worth doing for the usual two
|
||||
reasons — the value cannot change the statement's meaning, and the plan is
|
||||
reused across values.
|
||||
|
||||
```sql
|
||||
SELECT id, title FROM documents WHERE category = $1 AND score > $2
|
||||
```
|
||||
|
||||
The exact placeholder spelling and the binding call belong to your client; see
|
||||
[connecting a client](/search/sql).
|
||||
|
||||
## When it helps {#when}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| The same query with different values, many times | prepare once, bind per execution |
|
||||
| A complex plan — several joins, window functions | planning dominates; reuse it |
|
||||
| A one-off analytical query | no benefit; just execute it |
|
||||
|
||||
For a statement you want to keep rather than re-prepare, a
|
||||
[view](/management/view) stores the query server-side, and a
|
||||
[materialized view](/management/materialized-view) stores its rows.
|
||||
@@ -0,0 +1,185 @@
|
||||
---
|
||||
title: "Scalar functions"
|
||||
sidebarTitle: "Scalar Functions"
|
||||
description: "The per-row function set: math, string, regex, date/time, array, struct, and what Lance adds."
|
||||
icon: calculator
|
||||
keywords: ["scalar functions", "string functions", "math functions", "regex", "array functions", "json"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Functions that take a value and return a value, one row at a time. The set is
|
||||
DataFusion 54's default registration plus what Lance adds — the names below are
|
||||
what is actually registered, not the union of everything DataFusion documents.
|
||||
|
||||
For per-group functions see [Aggregation](/sql/aggregation); for per-window,
|
||||
[Window functions](/sql/window).
|
||||
|
||||
## Math {#math}
|
||||
|
||||
```text
|
||||
abs ceil floor round trunc signum power sqrt cbrt exp
|
||||
ln log log2 log10 factorial gcd lcm pi random nanvl
|
||||
isnan iszero cot degrees radians to_degrees to_radians
|
||||
sin cos tan asin acos atan atan2
|
||||
sinh cosh tanh asinh acosh atanh
|
||||
```
|
||||
|
||||
## String {#string}
|
||||
|
||||
```text
|
||||
ascii bit_length btrim chr concat concat_ws contains ends_with
|
||||
starts_with levenshtein lower upper ltrim rtrim octet_length
|
||||
repeat replace split_part to_hex uuid
|
||||
```
|
||||
|
||||
Unicode-aware variants:
|
||||
|
||||
```text
|
||||
character_length find_in_set initcap left right lpad rpad
|
||||
reverse strpos substr substr_index substring translate
|
||||
```
|
||||
|
||||
<Note>
|
||||
For keyword matching on a text column, [`contains_tokens`](#contains-tokens) is
|
||||
index-accelerated where `contains` and `LIKE` are not.
|
||||
</Note>
|
||||
|
||||
## Regular expressions {#regex}
|
||||
|
||||
```text
|
||||
regexp_count regexp_instr regexp_like regexp_match regexp_replace
|
||||
```
|
||||
|
||||
## Date and time {#datetime}
|
||||
|
||||
```text
|
||||
current_date current_time date_bin date_part date_trunc
|
||||
from_unixtime to_unixtime make_date make_time to_char
|
||||
to_date to_time to_local_time
|
||||
```
|
||||
|
||||
`date_bin` is the one worth knowing for analytics: it buckets timestamps into
|
||||
fixed-width intervals, which is how you build a time series without a
|
||||
self-join.
|
||||
|
||||
```sql
|
||||
SELECT date_bin(INTERVAL '1 hour', created_at) AS hour, count(*)
|
||||
FROM events GROUP BY hour ORDER BY hour
|
||||
```
|
||||
|
||||
## Arrays and maps {#arrays}
|
||||
|
||||
```text
|
||||
make_array array_has array_filter array_transform array_any_match
|
||||
array_compact array_normalize array_slice arrays_zip cardinality
|
||||
flatten range generate_series
|
||||
```
|
||||
|
||||
```text
|
||||
map map_keys map_values map_entries map_extract
|
||||
```
|
||||
|
||||
Distance functions over arrays:
|
||||
|
||||
```text
|
||||
cosine_distance distance inner_product
|
||||
```
|
||||
|
||||
<Warning>
|
||||
These operate on ordinary arrays. They are **not** how you run
|
||||
[vector search](/search/vector-search) — a search uses the vector index, while
|
||||
these compute per row over whatever the query already produced. Use them to
|
||||
score a small candidate set, not to retrieve one.
|
||||
</Warning>
|
||||
|
||||
## Structs and fields {#structs}
|
||||
|
||||
```text
|
||||
named_struct get_field coalesce nullif nvl nvl2 greatest least
|
||||
union_extract union_tag
|
||||
```
|
||||
|
||||
## Crypto and encoding {#crypto}
|
||||
|
||||
```text
|
||||
digest md5 sha224 sha256 sha384 sha512 encode decode
|
||||
```
|
||||
|
||||
## Type functions {#type}
|
||||
|
||||
```text
|
||||
arrow_cast arrow_try_cast arrow_typeof arrow_field arrow_metadata
|
||||
with_metadata version
|
||||
```
|
||||
|
||||
`arrow_typeof` is the fastest way to answer "what is this column, really" when a
|
||||
result does not have the shape you expected:
|
||||
|
||||
```sql
|
||||
SELECT arrow_typeof(vector) FROM documents LIMIT 1
|
||||
-- FixedSizeList(Field { name: "item", data_type: Float32, .. }, 1536)
|
||||
```
|
||||
|
||||
`arrow_cast` casts to an Arrow type by name, reaching types SQL has no spelling
|
||||
for. See [Data types](/sql/data-types).
|
||||
|
||||
## What Lance adds {#lance}
|
||||
|
||||
These are registered by Lance on top of DataFusion's set.
|
||||
|
||||
### `contains_tokens(column, tokens)` {#contains-tokens}
|
||||
|
||||
```sql
|
||||
SELECT * FROM my_docs WHERE contains_tokens(text, 'fox jumps dog')
|
||||
```
|
||||
|
||||
True when the column contains **all** the tokens, split on whitespace and
|
||||
punctuation. Equivalent to an `fts()` match query with fuzziness off, `AND`
|
||||
semantics and the simple tokenizer.
|
||||
|
||||
**Index-accelerated.** When the column has a full-text index that suits the
|
||||
query, the planner uses it instead of scanning — which is the reason to prefer
|
||||
it over `LIKE` for keyword tests.
|
||||
|
||||
### JSON accessors {#json}
|
||||
|
||||
| Function | Returns |
|
||||
| --- | --- |
|
||||
| `json_get(col, key)` | the value at `key`, still JSON |
|
||||
| `json_get_string(col, key)` | `Utf8` |
|
||||
| `json_get_int(col, key)` | `Int64` |
|
||||
| `json_get_float(col, key)` | `Float64` |
|
||||
| `json_get_bool(col, key)` | `Boolean` |
|
||||
| `json_extract(col, path)` | `Utf8`, by path |
|
||||
| `json_extract_with_type(col, path, type)` | the requested type |
|
||||
| `json_exists(col, path)` | `Boolean` |
|
||||
| `json_array_contains(col, value)` | `Boolean` |
|
||||
| `json_array_length(col)` | length |
|
||||
|
||||
`json_get` returns JSON, so accessors nest:
|
||||
|
||||
```sql
|
||||
SELECT * FROM events
|
||||
WHERE json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice'
|
||||
```
|
||||
|
||||
**`json_get`, its typed variants, and `json_extract` are index-accelerated** by a
|
||||
JSON scalar index on the column. The rest evaluate per row.
|
||||
|
||||
### Geospatial {#geo}
|
||||
|
||||
<Warning>
|
||||
Lance defines `st_*` functions on GeoArrow, but they sit behind a build feature
|
||||
**LanceDB does not ship**. Calling one returns an error naming the missing
|
||||
feature rather than an unknown-function error, and the spatial index is gated
|
||||
the same way.
|
||||
</Warning>
|
||||
|
||||
## Reaching a function not listed here {#escape-hatch}
|
||||
|
||||
Nothing wraps or restricts the set — a query reaches any registered function by
|
||||
writing it. When a name is not recognised, check it against
|
||||
[DataFusion's scalar function reference](https://datafusion.apache.org/user-guide/sql/scalar_functions.html)
|
||||
for version 54 before assuming it is unsupported; functions are added in most
|
||||
DataFusion releases, and a function documented on `main` may not exist here yet.
|
||||
@@ -39,7 +39,7 @@ Projection pushes into the scan in all three -- an unselected column is never
|
||||
read off storage. On a table of images or video that is the difference between a
|
||||
fast query and a slow one, so **prefer naming columns over `SELECT *`**.
|
||||
|
||||
Row metadata is available as [pseudo-columns](/sql/functions#pseudo-columns)
|
||||
Row metadata is available as [pseudo-columns](/sql/data-types#pseudo-columns)
|
||||
(`_rowid`, `_rowaddr`, `_rowoffset`) but only when named, so they stay out of
|
||||
`SELECT *`.
|
||||
|
||||
|
||||
+10
-8
@@ -8,16 +8,18 @@ keywords: ["show jobs", "show functions", "show materialized views", "show error
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Introspection. `SHOW TABLES` is DataFusion's; the rest are LanceDB extensions
|
||||
covering objects DataFusion has no concept of.
|
||||
Introspection over objects DataFusion has no concept of. All four are LanceDB
|
||||
extensions.
|
||||
|
||||
## SHOW TABLES {#tables}
|
||||
<Warning>
|
||||
DataFusion's own `SHOW TABLES`, `SHOW COLUMNS` and `SHOW <variable>` are **not
|
||||
available**. They are implemented over `information_schema`, which this session
|
||||
does not enable, so they fail with
|
||||
`"... is not supported unless information_schema is enabled"`.
|
||||
|
||||
```sql
|
||||
SHOW TABLES
|
||||
```
|
||||
|
||||
Lists the tables the session can see.
|
||||
List tables with the SDKs -- `table_names()` / `tableNames()`. For a table's
|
||||
columns, use [`DESCRIBE`](/sql/explain#describe).
|
||||
</Warning>
|
||||
|
||||
## SHOW MATERIALIZED VIEWS {#materialized-views}
|
||||
|
||||
|
||||
+17
-7
@@ -57,14 +57,24 @@ WINDOW w AS (PARTITION BY category ORDER BY score DESC)
|
||||
|
||||
## The functions {#functions}
|
||||
|
||||
| Kind | Functions |
|
||||
| --- | --- |
|
||||
| Ranking | `row_number`, `rank`, `dense_rank`, `percent_rank`, `ntile`, `cume_dist` |
|
||||
| Offset | `lag`, `lead`, `first_value`, `last_value`, `nth_value` |
|
||||
| Aggregate | any [aggregate](/sql/aggregation) used with `OVER` |
|
||||
Registered in DataFusion 54.
|
||||
|
||||
The full set is DataFusion's
|
||||
[window function reference](https://datafusion.apache.org/user-guide/sql/window_functions.html).
|
||||
**Ranking**
|
||||
|
||||
```text
|
||||
row_number rank dense_rank percent_rank cume_dist ntile
|
||||
```
|
||||
|
||||
**Offset and positional**
|
||||
|
||||
```text
|
||||
lag lead first_value last_value nth_value
|
||||
```
|
||||
|
||||
**Aggregates**
|
||||
|
||||
Any [aggregate function](/sql/aggregation#functions) used with `OVER` becomes a
|
||||
window function — that is how running totals and moving averages are written.
|
||||
|
||||
## Deduplicating with a window {#dedup}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user