mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
Brendan Clement
d0ed4d1a7a
- CREATE INDEX: the example left out USING, which the server requires, and the page said SQL cannot choose the algorithm or name the index. USING names the algorithm, the page lists them, and a name before ON is taken where the deployment's configuration allows it. - The search join passed fts() a bare string; fts() takes JSON. - Sampling a subquery's result is refused, so the sampling and subquery pages sample the table inside the subquery, and the refusals list it. - Two unaliased casts in one projection produce the same column name; the example aliases them. - A json_get nested in another json_get is refused. The page shows one level of nesting and json_extract with a JSONPath for deeper values, and corrects what json_extract and json_extract_with_type return.
200 lines
6.3 KiB
Plaintext
200 lines
6.3 KiB
Plaintext
---
|
|
title: "Scalar functions"
|
|
sidebarTitle: "Scalar Functions"
|
|
description: "The per-row function set: math, string, regex, date/time, array, struct, and what Lance adds."
|
|
icon: calculator
|
|
keywords: ["scalar functions", "string functions", "math functions", "regex", "array functions", "json"]
|
|
---
|
|
|
|
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
|
|
|
Functions that take a value and return a value, one row at a time. The set is
|
|
based on DataFusion's registration plus Lance and deployment-specific
|
|
functions. The names below are selected examples, not an exhaustive catalog
|
|
for every deployment. Availability depends on the deployed version and build.
|
|
|
|
For per-group functions see [Aggregation](/sql/dql/aggregation); for per-window,
|
|
[Window functions](/sql/dql/window).
|
|
|
|
## Math {#math}
|
|
|
|
```text
|
|
abs ceil floor round trunc signum power sqrt cbrt exp
|
|
ln log log2 log10 factorial gcd lcm pi random nanvl
|
|
isnan iszero cot degrees radians to_degrees to_radians
|
|
sin cos tan asin acos atan atan2
|
|
sinh cosh tanh asinh acosh atanh
|
|
```
|
|
|
|
## String {#string}
|
|
|
|
```text
|
|
ascii bit_length btrim chr concat concat_ws contains ends_with
|
|
starts_with levenshtein lower upper ltrim rtrim octet_length
|
|
repeat replace split_part to_hex uuid
|
|
```
|
|
|
|
Unicode-aware variants:
|
|
|
|
```text
|
|
character_length find_in_set initcap left right lpad rpad
|
|
reverse strpos substr substr_index substring translate
|
|
```
|
|
|
|
<Note>
|
|
For keyword matching on a text column, [`contains_tokens`](#contains-tokens) is
|
|
index-accelerated where `contains` and `LIKE` are not.
|
|
</Note>
|
|
|
|
## Regular expressions {#regex}
|
|
|
|
```text
|
|
regexp_count regexp_instr regexp_like regexp_match regexp_replace
|
|
```
|
|
|
|
## Date and time {#datetime}
|
|
|
|
```text
|
|
current_date current_time date_bin date_part date_trunc
|
|
from_unixtime to_unixtime make_date make_time to_char
|
|
to_date to_time to_local_time
|
|
```
|
|
|
|
`date_bin` is the one worth knowing for analytics: it buckets timestamps into
|
|
fixed-width intervals, which is how you build a time series without a
|
|
self-join.
|
|
|
|
```sql
|
|
SELECT date_bin(INTERVAL '1 hour', created_at) AS hour, count(*)
|
|
FROM events GROUP BY hour ORDER BY hour
|
|
```
|
|
|
|
## Arrays and maps {#arrays}
|
|
|
|
```text
|
|
make_array array_has array_filter array_transform array_any_match
|
|
array_compact array_normalize array_slice arrays_zip cardinality
|
|
flatten range generate_series
|
|
```
|
|
|
|
```text
|
|
map map_keys map_values map_entries map_extract
|
|
```
|
|
|
|
Distance functions over arrays:
|
|
|
|
```text
|
|
cosine_distance distance exact_distance inner_product
|
|
```
|
|
|
|
<Warning>
|
|
Distance functions can evaluate arrays per row. Some deployment versions also
|
|
recognize `ORDER BY distance(vector, query_vector) LIMIT k` over a table scan
|
|
and plan it as vector search, with indexed or flat search as appropriate.
|
|
Where available, `exact_distance` evaluates distance without that rewrite.
|
|
Confirm the
|
|
functions supported by your deployment and inspect SQL
|
|
[`EXPLAIN`](/sql/dql/explain); the function name alone does not guarantee an
|
|
index-backed plan. See [SQL vector search](/basics/sql#no-vector-sql).
|
|
</Warning>
|
|
|
|
## Structs and fields {#structs}
|
|
|
|
```text
|
|
named_struct get_field coalesce nullif nvl nvl2 greatest least
|
|
union_extract union_tag
|
|
```
|
|
|
|
## Crypto and encoding {#crypto}
|
|
|
|
```text
|
|
digest md5 sha224 sha256 sha384 sha512 encode decode
|
|
```
|
|
|
|
## Type functions {#type}
|
|
|
|
```text
|
|
arrow_cast arrow_try_cast arrow_typeof arrow_field arrow_metadata
|
|
with_metadata version
|
|
```
|
|
|
|
`arrow_typeof` is the fastest way to answer "what is this column, really" when a
|
|
result does not have the shape you expected:
|
|
|
|
```sql
|
|
SELECT arrow_typeof(vector) FROM documents LIMIT 1
|
|
-- FixedSizeList(Field { name: "item", data_type: Float32, .. }, 1536)
|
|
```
|
|
|
|
`arrow_cast` casts to an Arrow type by name, reaching types SQL has no spelling
|
|
for. See [Data types](/sql/data-types).
|
|
|
|
## What Lance adds {#lance}
|
|
|
|
These are registered by Lance on top of DataFusion's set.
|
|
|
|
### `contains_tokens(column, tokens)` {#contains-tokens}
|
|
|
|
```sql
|
|
SELECT * FROM my_docs WHERE contains_tokens(text, 'fox jumps dog')
|
|
```
|
|
|
|
True when the column contains **all** the tokens, split on whitespace and
|
|
punctuation. Equivalent to an `fts()` match query with fuzziness off, `AND`
|
|
semantics and the simple tokenizer.
|
|
|
|
**Index-accelerated.** When the column has a full-text index that suits the
|
|
query, the planner uses it instead of scanning — which is the reason to prefer
|
|
it over `LIKE` for keyword tests.
|
|
|
|
### JSON accessors {#json}
|
|
|
|
| Function | Returns |
|
|
| --- | --- |
|
|
| `json_get(col, key)` | the value at `key`, still JSON |
|
|
| `json_get_string(col, key)` | `Utf8` |
|
|
| `json_get_int(col, key)` | `Int64` |
|
|
| `json_get_float(col, key)` | `Float64` |
|
|
| `json_get_bool(col, key)` | `Boolean` |
|
|
| `json_extract(col, path)` | `Utf8`: the value as JSON text, by path |
|
|
| `json_extract_with_type(col, path)` | a struct of the value as JSON and a type tag, by path |
|
|
| `json_exists(col, path)` | `Boolean` |
|
|
| `json_array_contains(col, value)` | `Boolean` |
|
|
| `json_array_length(col)` | length |
|
|
|
|
`json_get` returns JSON, so a typed accessor can read a field inside it:
|
|
|
|
```sql
|
|
SELECT * FROM events
|
|
WHERE json_get_string(json_get(data, 'user'), 'name') = 'Alice'
|
|
```
|
|
|
|
That works one level down. A `json_get` nested inside another `json_get` is
|
|
refused, so reach a deeper value with `json_extract` and a JSONPath instead. It
|
|
returns JSON text, so a string compares with its quotes:
|
|
|
|
```sql
|
|
SELECT * FROM events
|
|
WHERE json_extract(data, '$.user.profile.name') = '"Alice"'
|
|
```
|
|
|
|
**`json_get`, its typed variants, and `json_extract` are index-accelerated** by a
|
|
JSON scalar index on the column. The rest evaluate per row.
|
|
|
|
### Geospatial {#geo}
|
|
|
|
<Warning>
|
|
Lance defines `st_*` functions on GeoArrow, but they sit behind a build feature
|
|
**LanceDB does not ship**. Calling one returns an error naming the missing
|
|
feature rather than an unknown-function error, and the spatial index is gated
|
|
the same way.
|
|
</Warning>
|
|
|
|
## Reaching a function not listed here {#escape-hatch}
|
|
|
|
Nothing wraps or restricts the set — a query reaches any registered function by
|
|
writing it. When a name is not recognised, check it against
|
|
[DataFusion's scalar function reference](https://datafusion.apache.org/user-guide/sql/scalar_functions.html)
|
|
for version 54 before assuming it is unsupported; functions are added in most
|
|
DataFusion releases, and a function documented on `main` may not exist here yet.
|