---
title: "SQL functions and types"
sidebarTitle: "Functions and types"
description: "The dialect is DataFusion's. What that gives you, and what LanceDB adds on top."
icon: function
keywords: ["sql functions", "datafusion", "scalar functions", "aggregate", "data types", "fts"]
---

LanceDB executes SQL with [Apache DataFusion](https://datafusion.apache.org),
currently **version 54**. The dialect, the built-in functions, and the type
system are DataFusion's, registered from its default set — so anything the
DataFusion reference documents for that version is available, and this page does
not reproduce it.

What it does document is the version you are querying against, where the
authoritative lists are, and the parts that are LanceDB's rather than
DataFusion's.

## Reference by category {#reference}

| | DataFusion reference |
| --- | --- |
| Math, conditional, string, binary string, regular expression, time and date | [Scalar functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) |
| `count`, `sum`, `avg`, `min`, `max`, statistical, approximate | [Aggregate functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) |
| `row_number`, `rank`, `lag`, `lead`, and the rest of `OVER (...)` | [Window functions](https://datafusion.apache.org/user-guide/sql/window_functions.html) |
| Arrow-backed types, and how SQL type names map onto them | [Data types](https://datafusion.apache.org/user-guide/sql/data_types.html) |
| `SELECT`, joins, `GROUP BY`, `ORDER BY`, `LIMIT`, CTEs, set operations | [SELECT syntax](https://datafusion.apache.org/user-guide/sql/select.html) |

<Note>
Function availability is version-specific: DataFusion adds functions in most
releases, so a function documented on `main` may not exist in 54. When a
function is not recognised, check it against the reference for the version above
before assuming a bug.
</Note>

## Types are Arrow types {#types}

There is no SQL type layer over Lance storage. The DataFusion table provider
returns the dataset's Arrow schema directly — `Schema::from(dataset.schema())` —
so a column is the same object to the query engine as it is to the format, and
DataFusion's [type reference](https://datafusion.apache.org/user-guide/sql/data_types.html)
describes it exactly.

That identity is mostly invisible and occasionally load-bearing. A vector column
is a `FixedSizeList<Float32>`: selectable, testable for nullness, joinable like
any other column — but arithmetic over it is list arithmetic, not scalar
arithmetic. When the operation is a similarity comparison, use vector search
rather than SQL.

### Pseudo-columns {#pseudo-columns}

Lance rows carry metadata the provider can expose as columns. They are appended
to the schema only when asked for, so they do not appear in `SELECT *`:

| Column | Type | Meaning |
| --- | --- | --- |
| `_rowid` | `UInt64`, nullable | stable row identifier |
| `_rowaddr` | `UInt64`, nullable | physical row address |
| `_rowoffset` | `UInt64`, nullable | offset within the fragment |
| `_score` | `Float32` | relevance, from [`fts()`](#fts) |

`_rowid` and `_rowaddr` are declared nullable, but not because a row lacks one —
their validity bitmap is reused as a selection vector internally. Do not read a
null there as meaningful.

`_rowid` is the identifier [materialized views](/feature-engineering/materialized-views)
depend on, which is why a source table needs stable row ids enabled at creation.

### Blob columns change shape {#blobs}

A blob column can be read as its **binary contents** or as a **description** —
a pointer plus metadata, without fetching the bytes. That choice changes the
column's Arrow type, and therefore the schema a query sees.

The default reads blobs as descriptions and other binary columns as binary. A
query that selects a blob column and expects bytes will get a struct instead
unless the read is configured otherwise. Worth knowing before writing a
predicate against one.

## What LanceDB adds {#extensions}

### `fts(table, query)` {#fts}

Full-text search is a **table function**, not a predicate — implemented as a
DataFusion `TableProvider` over the dataset, so its result is a relation. That is
what lets it return relevance alongside the matching rows, which a `WHERE` clause
has nowhere to put:

```sql
SELECT id, text, _score
FROM fts('my_docs', 'puppy')
ORDER BY _score DESC
LIMIT 5
```

The second argument is either a bare query string or a JSON query object for
fuzzy, phrase, boolean, boost and multi-match forms.

Its schema is the dataset's, so every column of the table is available in the
result — projections push down into the scan rather than being applied
afterwards. Being a relation is also what makes it compose: join it, filter it,
aggregate over it.

```sql
SELECT c.name, count(*) AS hits
FROM fts('my_docs', 'puppy') AS m
JOIN categories AS c ON c.id = m.category_id
GROUP BY c.name
ORDER BY hits DESC
```

### `_score` {#score}

A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves
like a normal column — order by it, filter on it, project it — and exists only
in a relation that came from `fts()`.

### `contains_tokens(column, tokens)` {#contains-tokens}

A predicate rather than a relation, for when you want keyword matching inside a
`WHERE` clause and do not need relevance:

```sql
SELECT * FROM my_docs WHERE contains_tokens(text, 'fox jumps dog')
```

It is true when the column contains **all** the tokens, which are split on
whitespace and punctuation. Equivalent to an `fts()` match query with fuzziness
off, `AND` semantics, and the simple tokenizer.

**It is index-accelerated.** When the column has a full-text index that suits
the query, the planner uses it instead of scanning. That is the reason to prefer
it over `LIKE` for keyword tests.

Use `fts()` when you need `_score` or the fuzzy, phrase, or boost forms; use
`contains_tokens` when you just need a boolean.

### JSON functions {#json}

Lance registers a set of JSON accessors over string columns holding JSON:

| Function | Returns |
| --- | --- |
| `json_get(col, key)` | the value at `key`, still JSON |
| `json_get_string(col, key)` | `Utf8` |
| `json_get_int(col, key)` | `Int64` |
| `json_get_float(col, key)` | `Float64` |
| `json_get_bool(col, key)` | `Boolean` |
| `json_extract(col, path)` | `Utf8`, by path |
| `json_extract_with_type(col, path, type)` | the requested type |
| `json_exists(col, path)` | `Boolean` |
| `json_array_contains(col, value)` | `Boolean` |
| `json_array_length(col)` | length |

`json_get` returns JSON, so accessors nest for a nested document:

```sql
SELECT * FROM events
WHERE json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice'
```

**`json_get`, its typed variants, and `json_extract` are index-accelerated** by a
JSON scalar index on the column. The others evaluate per row.

<Note>
Lance also defines geospatial `st_*` functions, but they are behind an optional
build feature that LanceDB does not ship. Calling one returns an error telling
you the feature is missing rather than an unknown-function error.
</Note>

## Reaching a function LanceDB does not name {#escape-hatch}

Nothing here wraps or restricts the DataFusion set — a query reaches any
function in the reference above by writing it. If a name is not recognised,
check it against the reference for the version this page pins rather than
assuming it is unsupported.