mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 07:12:27 +00:00
docs: restructure around the three API surfaces
The Documentation tab now reads as a manual over Python, TypeScript and SQL rather than a Python guide with SQL bolted on. Basics replaces the Catalog section: what the catalog holds, what each API surface can do, jobs, agentic skills, and what Lance is. Management is new and is the CRUD and listing reference for every catalog object, with a column per language so the gaps are visible -- TypeScript has no function or SQL methods, SQL needs a deployment. Search becomes Query and gains a SELECT reference, since an analytical query is a query whether it came from a builder or a statement. Jobs is new. Every long-running operation already returned the same handle from an _async method; nothing documented it. The SQL function reference gains what Lance registers on top of DataFusion: contains_tokens, backed by the FTS index, and the JSON accessors, backed by a JSON scalar index. Geospatial st_* functions are behind a build feature LanceDB does not ship, which is now stated rather than left to a confusing error. Removes the Geneva pages' orphaned snippets, merges FTS Examples into Full-Text Search, and splits Query Optimization into Explain and Explain Analyze.
This commit is contained in:
Jack Ye
@@ -115,6 +115,59 @@ A pseudo-column produced by `fts()`, carrying relevance for the row. It behaves
|
||||
like a normal column — order by it, filter on it, project it — and exists only
|
||||
in a relation that came from `fts()`.
|
||||
|
||||
### `contains_tokens(column, tokens)` {#contains-tokens}
|
||||
|
||||
A predicate rather than a relation, for when you want keyword matching inside a
|
||||
`WHERE` clause and do not need relevance:
|
||||
|
||||
```sql
|
||||
SELECT * FROM my_docs WHERE contains_tokens(text, 'fox jumps dog')
|
||||
```
|
||||
|
||||
It is true when the column contains **all** the tokens, which are split on
|
||||
whitespace and punctuation. Equivalent to an `fts()` match query with fuzziness
|
||||
off, `AND` semantics, and the simple tokenizer.
|
||||
|
||||
**It is index-accelerated.** When the column has a full-text index that suits
|
||||
the query, the planner uses it instead of scanning. That is the reason to prefer
|
||||
it over `LIKE` for keyword tests.
|
||||
|
||||
Use `fts()` when you need `_score` or the fuzzy, phrase, or boost forms; use
|
||||
`contains_tokens` when you just need a boolean.
|
||||
|
||||
### JSON functions {#json}
|
||||
|
||||
Lance registers a set of JSON accessors over string columns holding JSON:
|
||||
|
||||
| Function | Returns |
|
||||
| --- | --- |
|
||||
| `json_get(col, key)` | the value at `key`, still JSON |
|
||||
| `json_get_string(col, key)` | `Utf8` |
|
||||
| `json_get_int(col, key)` | `Int64` |
|
||||
| `json_get_float(col, key)` | `Float64` |
|
||||
| `json_get_bool(col, key)` | `Boolean` |
|
||||
| `json_extract(col, path)` | `Utf8`, by path |
|
||||
| `json_extract_with_type(col, path, type)` | the requested type |
|
||||
| `json_exists(col, path)` | `Boolean` |
|
||||
| `json_array_contains(col, value)` | `Boolean` |
|
||||
| `json_array_length(col)` | length |
|
||||
|
||||
`json_get` returns JSON, so accessors nest for a nested document:
|
||||
|
||||
```sql
|
||||
SELECT * FROM events
|
||||
WHERE json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice'
|
||||
```
|
||||
|
||||
**`json_get`, its typed variants, and `json_extract` are index-accelerated** by a
|
||||
JSON scalar index on the column. The others evaluate per row.
|
||||
|
||||
<Note>
|
||||
Lance also defines geospatial `st_*` functions, but they are behind an optional
|
||||
build feature that LanceDB does not ship. Calling one returns an error telling
|
||||
you the feature is missing rather than an unknown-function error.
|
||||
</Note>
|
||||
|
||||
## Reaching a function LanceDB does not name {#escape-hatch}
|
||||
|
||||
Nothing here wraps or restricts the DataFusion set — a query reaches any
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
---
|
||||
title: "SQL manual"
|
||||
sidebarTitle: "Overview"
|
||||
description: "Analytical SQL over the lakehouse, served through Arrow Flight SQL."
|
||||
icon: table-list
|
||||
keywords: ["sql", "flight sql", "query", "analytics", "datafusion", "fts"]
|
||||
---
|
||||
|
||||
LanceDB answers analytical queries in SQL, served over an
|
||||
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
|
||||
endpoint. Any Flight SQL client can reach it, and results come back as Arrow.
|
||||
|
||||
The dialect is [DataFusion's](/sql/functions), so the syntax and functions are
|
||||
the ones that reference documents. What this manual covers is the parts that are
|
||||
LanceDB's: how a query reaches the engine, how Lance columns appear to it, and
|
||||
the extensions a lakehouse over multimodal data needs.
|
||||
|
||||
## Where a query runs {#execution}
|
||||
|
||||
Queries execute on the server. The client sends a statement and receives Arrow
|
||||
record batches — nothing is converted to pandas or Python objects unless you ask
|
||||
for it.
|
||||
|
||||
Because results are Arrow the whole way, a query that feeds training or
|
||||
downstream processing does not pay a serialization cost to get there.
|
||||
|
||||
## The dialect {#dialect}
|
||||
|
||||
Ordinary analytical SQL — `SELECT`, `WHERE`, `GROUP BY`, `ORDER BY`, `LIMIT`,
|
||||
joins, CTEs and window functions — over Arrow-typed columns.
|
||||
|
||||
Two things differ from a warehouse you may be used to, and both come from the
|
||||
storage being Lance rather than something SQL-shaped:
|
||||
|
||||
- **Columns are Arrow types**, not a SQL type system mapped onto storage. A
|
||||
vector column is a `FixedSizeList<Float32>` and behaves like one.
|
||||
- **Row metadata is available as columns** — `_rowid`, `_rowaddr` — but only
|
||||
when asked for, so they stay out of `SELECT *`.
|
||||
|
||||
Both are covered in [Functions and types](/sql/functions).
|
||||
|
||||
## Full-text search {#fts}
|
||||
|
||||
Full-text search is a **table function** rather than a predicate, because it
|
||||
produces relevance alongside the matching rows and a `WHERE` clause has nowhere
|
||||
to put that:
|
||||
|
||||
```sql
|
||||
SELECT id, text, _score
|
||||
FROM fts('my_docs', 'puppy')
|
||||
ORDER BY _score DESC
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
`fts()` takes a table name and a query — a bare string, or a JSON query object
|
||||
for fuzzy, phrase, boolean, boost and multi-match forms. `_score` carries
|
||||
relevance and behaves like any other column.
|
||||
|
||||
Being a relation is what makes it compose: join its result, filter it, aggregate
|
||||
over it.
|
||||
|
||||
## Reference {#reference}
|
||||
|
||||
<Card title="Functions and types" icon="calculator" href="/sql/functions">
|
||||
The DataFusion dialect and version, how Lance columns map onto it, and what
|
||||
LanceDB adds.
|
||||
</Card>
|
||||
|
||||
## What this manual does not cover {#scope}
|
||||
|
||||
Connecting a Flight SQL client — drivers, authentication, the endpoint address —
|
||||
is deployment-specific and documented with the deployment.
|
||||
Reference in New Issue
Block a user