mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
Jack Ye
96bfdcd937
DataFrame support is not going ahead, so the pages documenting it come out: the relational operations and the expression system, both of which described only that API. The manual was framed around having two query surfaces and choosing between them. With one, that framing is gone — the overview now leads with SQL over Arrow Flight SQL and what the dialect is, rather than with a comparison to something that will not exist. The function reference keeps everything that was about SQL: the DataFusion version, how Lance columns map onto its type system, the pseudo-columns, blob handling, and `fts()`. Only the section on reaching functions through the Python wrapper is replaced, since the wrapper is going too. Mentions of pandas and polars DataFrames elsewhere in the documentation are unrelated and untouched.
73 lines
2.6 KiB
Plaintext
73 lines
2.6 KiB
Plaintext
---
|
|
title: "SQL manual"
|
|
sidebarTitle: "Overview"
|
|
description: "Analytical SQL over the lakehouse, served through Arrow Flight SQL."
|
|
icon: table-list
|
|
keywords: ["sql", "flight sql", "query", "analytics", "datafusion", "fts"]
|
|
---
|
|
|
|
LanceDB answers analytical queries in SQL, served over an
|
|
[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html)
|
|
endpoint. Any Flight SQL client can reach it, and results come back as Arrow.
|
|
|
|
The dialect is [DataFusion's](/sql/functions), so the syntax and functions are
|
|
the ones that reference documents. What this manual covers is the parts that are
|
|
LanceDB's: how a query reaches the engine, how Lance columns appear to it, and
|
|
the extensions a lakehouse over multimodal data needs.
|
|
|
|
## Where a query runs {#execution}
|
|
|
|
Queries execute on the server. The client sends a statement and receives Arrow
|
|
record batches — nothing is converted to pandas or Python objects unless you ask
|
|
for it.
|
|
|
|
Because results are Arrow the whole way, a query that feeds training or
|
|
downstream processing does not pay a serialization cost to get there.
|
|
|
|
## The dialect {#dialect}
|
|
|
|
Ordinary analytical SQL — `SELECT`, `WHERE`, `GROUP BY`, `ORDER BY`, `LIMIT`,
|
|
joins, CTEs and window functions — over Arrow-typed columns.
|
|
|
|
Two things differ from a warehouse you may be used to, and both come from the
|
|
storage being Lance rather than something SQL-shaped:
|
|
|
|
- **Columns are Arrow types**, not a SQL type system mapped onto storage. A
|
|
vector column is a `FixedSizeList<Float32>` and behaves like one.
|
|
- **Row metadata is available as columns** — `_rowid`, `_rowaddr` — but only
|
|
when asked for, so they stay out of `SELECT *`.
|
|
|
|
Both are covered in [Functions and types](/sql/functions).
|
|
|
|
## Full-text search {#fts}
|
|
|
|
Full-text search is a **table function** rather than a predicate, because it
|
|
produces relevance alongside the matching rows and a `WHERE` clause has nowhere
|
|
to put that:
|
|
|
|
```sql
|
|
SELECT id, text, _score
|
|
FROM fts('my_docs', 'puppy')
|
|
ORDER BY _score DESC
|
|
LIMIT 5
|
|
```
|
|
|
|
`fts()` takes a table name and a query — a bare string, or a JSON query object
|
|
for fuzzy, phrase, boolean, boost and multi-match forms. `_score` carries
|
|
relevance and behaves like any other column.
|
|
|
|
Being a relation is what makes it compose: join its result, filter it, aggregate
|
|
over it.
|
|
|
|
## Reference {#reference}
|
|
|
|
<Card title="Functions and types" icon="calculator" href="/sql/functions">
|
|
The DataFusion dialect and version, how Lance columns map onto it, and what
|
|
LanceDB adds.
|
|
</Card>
|
|
|
|
## What this manual does not cover {#scope}
|
|
|
|
Connecting a Flight SQL client — drivers, authentication, the endpoint address —
|
|
is deployment-specific and documented with the deployment.
|