mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
docs: give SQL one section with a page per construct
SELECT and Enterprise SQL sat apart in Query while documenting the same engine. They are now one SQL group: overview, Select, Aggregation, Join, Window, Sampling, Functions -- one page per construct, with the Enterprise client-setup and FTS pages merged in rather than parallel to them. Sampling is new. TABLESAMPLE is implemented and was undocumented: BERNOULLI picks rows, SYSTEM picks Lance fragments, REPEATABLE fixes the draw so a sample works as a stable evaluation set. Its refusals are documented too -- a percentage over 100, a fractional row count, BUCKET, the SAMPLE spelling and the OFFSET modifier are all errors, because a sampling clause that were read and not applied would be a wrong answer rather than a missing feature. Schema & Data Types is one page again, with the multimodal content merged into its blob section rather than sitting beside it. Agentic Skills stops being a tab. Both its pages are tutorials, so they join the Tutorials tab; the Basics page that introduces them stays.
This commit is contained in:
Jack Ye
@@ -6,6 +6,35 @@ icon: shapes
|
||||
keywords: ["schema", "data types", "arrow", "json", "blob", "bfloat16", "vector", "geospatial"]
|
||||
---
|
||||
|
||||
import {
|
||||
PyMultimodalImports as MultimodalImports,
|
||||
TsMultimodalImports as TsMultimodalImports,
|
||||
RsMultimodalImports as RsMultimodalImports,
|
||||
PyCreateDummyData as CreateDummyData,
|
||||
TsCreateDummyData as TsCreateDummyData,
|
||||
RsCreateDummyData as RsCreateDummyData,
|
||||
PyDefineSchema as DefineSchema,
|
||||
TsDefineSchema as TsDefineSchema,
|
||||
RsDefineSchema as RsDefineSchema,
|
||||
PyIngestData as IngestData,
|
||||
TsIngestData as TsIngestData,
|
||||
RsIngestData as RsIngestData,
|
||||
PySearchData as SearchData,
|
||||
TsSearchData as TsSearchData,
|
||||
RsSearchData as RsSearchData,
|
||||
PyProcessResults as ProcessResults,
|
||||
TsProcessResults as TsProcessResults,
|
||||
RsProcessResults as RsProcessResults,
|
||||
PyBlobApiSchema as BlobApiSchema,
|
||||
TsBlobApiSchema as TsBlobApiSchema,
|
||||
RsBlobApiSchema as RsBlobApiSchema,
|
||||
PyBlobApiIngest as BlobApiIngest,
|
||||
TsBlobApiIngest as TsBlobApiIngest,
|
||||
RsBlobApiIngest as RsBlobApiIngest,
|
||||
PyBlobApiToPandas as BlobApiToPandas,
|
||||
PyQueryToPandasKwargs as QueryToPandasKwargs,
|
||||
} from '/snippets/multimodal.mdx';
|
||||
|
||||
A table's schema is an **Arrow schema**. There is no separate LanceDB type
|
||||
system mapped onto storage -- a column is the same object to the query engine,
|
||||
to the format, and to any other engine reading the files.
|
||||
@@ -110,7 +139,202 @@ instead, unless the read is configured otherwise. Worth knowing before writing a
|
||||
predicate against one.
|
||||
</Warning>
|
||||
|
||||
See [Working with multimodal data](/tables/multimodal).
|
||||
### Store binary data {#store-binary-data}
|
||||
|
||||
To store binary data, define a binary Arrow field in your schema (`pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust).
|
||||
|
||||
#### 1\. Setup and imports {#1-setup-and-imports}
|
||||
|
||||
First, import the necessary libraries for LanceDB and Arrow in your SDK.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{MultimodalImports}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsMultimodalImports}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsMultimodalImports}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 2\. Prepare data {#2-prepare-data}
|
||||
|
||||
For this example, we'll create some dummy in-memory images. In a real application, you would read these from files or an API. The key is to convert your data (image, audio, etc.) into a raw `bytes` object.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateDummyData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateDummyData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateDummyData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 3\. Define the schema {#3-define-the-schema}
|
||||
|
||||
When creating the table, it is **highly recommended** to define the schema explicitly. This ensures that your binary data is correctly interpreted as a `binary` type by Arrow/LanceDB and not as a generic string or list.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{DefineSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDefineSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDefineSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 4\. Ingest data {#4-ingest-data}
|
||||
|
||||
Now, create the table using the data and the defined schema.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{IngestData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsIngestData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsIngestData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Retrieve and use blobs {#retrieve-and-use-blobs}
|
||||
|
||||
When you search your LanceDB table, you can retrieve the binary column just like any other metadata.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SearchData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSearchData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSearchData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### Convert bytes back to objects {#convert-bytes-back-to-objects}
|
||||
|
||||
Once you have the bytes back from the search result, you can decode them into the original format (for example, an image object or audio buffer).
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{ProcessResults}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsProcessResults}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsProcessResults}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Large Blobs (Blob API) {#large-blobs-blob-api}
|
||||
|
||||
For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront.
|
||||
|
||||
#### 1\. Define a blob schema {#1-define-a-blob-schema}
|
||||
|
||||
To use the Blob API, you must mark the column with `{"lance-encoding:blob": "true"}` metadata.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBlobApiSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBlobApiSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
#### 2\. Ingest large blobs {#2-ingest-large-blobs}
|
||||
|
||||
You can then ingest data normally, and Lance will handle the optimized storage.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiIngest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBlobApiIngest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBlobApiIngest}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Card>
|
||||
For more advanced usage, including random access and file-like reading of blobs, see the
|
||||
Lance format's [blob API documentation](https://lance.org/guide/blob/).
|
||||
</Card>
|
||||
|
||||
#### 3\. Convert blob tables to pandas {#3-convert-blob-tables-to-pandas}
|
||||
|
||||
When you call `to_pandas()` on a local LanceDB table that contains Blob API columns, the `blob_mode` argument controls how those columns materialize. This is available in the Python SDK on local tables; remote tables raise `NotImplementedError`.
|
||||
|
||||
`blob_mode` accepts:
|
||||
|
||||
- `"lazy"` (default): returns blob columns as lazy `BlobFile` objects without eagerly materializing their payloads. Use this when you want to stream blob bytes on demand or only inspect a subset of rows. Namespace-backed local tables also use the Lance native blob-aware pandas conversion for lazy blobs; in-memory datasets fall back to the standard PyArrow `to_pandas()` path.
|
||||
- `"bytes"`: eagerly materializes each blob as `bytes`. Use this when you need the raw payload in the DataFrame, for example to decode an image or audio clip in-process.
|
||||
- `"descriptions"`: returns blob descriptors (offsets, sizes, and positions) instead of the data itself. Use this when you want to plan I/O without paying the cost of loading every blob.
|
||||
|
||||
`"bytes"` and `"descriptions"` require a filesystem-backed Lance dataset and are not supported on in-memory tables.
|
||||
|
||||
Extra keyword arguments are forwarded to the underlying PyArrow / Lance pandas conversion, so you can also pass options like `split_blocks` or `self_destruct`:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiToPandas}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Query builders also accept `blob_mode` on their `to_pandas()` method:
|
||||
|
||||
- Plain scans support `"lazy"`, `"bytes"`, and `"descriptions"` with filters, projections, aliases, `limit`, and `offset`.
|
||||
- Vector, FTS, hybrid, and ordered queries can't materialize blob columns through `to_pandas()`; omit blob columns from the projection for those query shapes.
|
||||
- This works on both sync and async query builders. Extra PyArrow kwargs like `split_blocks` and `self_destruct` are still forwarded.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{QueryToPandasKwargs}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Other modalities {#other-modalities}
|
||||
|
||||
The `pa.binary()` and `pa.large_binary()` types are universal. You can use this same pattern for other types of multimodal data:
|
||||
|
||||
- **Audio:** Read `.wav` or `.mp3` files as bytes.
|
||||
- **Video:** Store video transitions or full clips using the Blob API.
|
||||
- **PDFs/Documents:** Store the raw file content for document search.
|
||||
|
||||
### bfloat16 {#bfloat16}
|
||||
|
||||
|
||||
+30
-27
@@ -55,10 +55,6 @@
|
||||
"source": "/catalog/database",
|
||||
"destination": "/basics/catalog"
|
||||
},
|
||||
{
|
||||
"source": "/sql",
|
||||
"destination": "/basics/apis"
|
||||
},
|
||||
{
|
||||
"source": "/api-reference",
|
||||
"destination": "/basics/apis"
|
||||
@@ -74,6 +70,14 @@
|
||||
{
|
||||
"source": "/tables/create",
|
||||
"destination": "/ingestion"
|
||||
},
|
||||
{
|
||||
"source": "/tables/multimodal",
|
||||
"destination": "/basics/data-types#blob"
|
||||
},
|
||||
{
|
||||
"source": "/search/select",
|
||||
"destination": "/sql/select"
|
||||
}
|
||||
],
|
||||
"navigation": {
|
||||
@@ -94,13 +98,7 @@
|
||||
"basics/lance",
|
||||
"basics/catalog",
|
||||
"basics/jobs",
|
||||
{
|
||||
"group": "Schema & Data Types",
|
||||
"pages": [
|
||||
"basics/data-types",
|
||||
"tables/multimodal"
|
||||
]
|
||||
},
|
||||
"basics/data-types",
|
||||
"tables/consistency",
|
||||
"basics/apis",
|
||||
"basics/agentic-skills"
|
||||
@@ -145,15 +143,11 @@
|
||||
"group": "Query",
|
||||
"pages": [
|
||||
"search/index",
|
||||
"search/select",
|
||||
"search/vector-search",
|
||||
"search/multivector-search",
|
||||
"search/full-text-search",
|
||||
"search/hybrid-search",
|
||||
"search/filtering",
|
||||
"search/explain",
|
||||
"search/explain-analyze",
|
||||
"sql/functions",
|
||||
{
|
||||
"group": "Reranking",
|
||||
"pages": [
|
||||
@@ -165,6 +159,20 @@
|
||||
"reranking/custom-reranker",
|
||||
"reranking/eval"
|
||||
]
|
||||
},
|
||||
"search/explain",
|
||||
"search/explain-analyze",
|
||||
{
|
||||
"group": "SQL",
|
||||
"pages": [
|
||||
"sql/index",
|
||||
"sql/select",
|
||||
"sql/aggregation",
|
||||
"sql/join",
|
||||
"sql/window",
|
||||
"sql/sampling",
|
||||
"sql/functions"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -329,6 +337,13 @@
|
||||
"training/vlm-finetuning"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "AI Agents",
|
||||
"pages": [
|
||||
"build-with-ai-agents",
|
||||
"agent-branch-experiments"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "Robotics",
|
||||
"pages": [
|
||||
@@ -356,18 +371,6 @@
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"tab": "Agentic Skills",
|
||||
"groups": [
|
||||
{
|
||||
"group": "Agentic Skills",
|
||||
"pages": [
|
||||
"build-with-ai-agents",
|
||||
"agent-branch-experiments"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
@@ -454,7 +454,7 @@ These snippets load the local image file and store the bytes in an `image` colum
|
||||
</CodeBlock>
|
||||
</CodeGroup >
|
||||
|
||||
For more examples, see the [multimodal data](/tables/multimodal) section.
|
||||
For more examples, see [Data types](/basics/data-types#blob).
|
||||
|
||||
## 8\. Train on the result {#8-train}
|
||||
|
||||
|
||||
@@ -1,176 +0,0 @@
|
||||
---
|
||||
title: "SELECT syntax"
|
||||
sidebarTitle: "SELECT"
|
||||
description: "The analytical query surface: projection, filtering, grouping, joins, and how each maps onto the SDKs."
|
||||
icon: table-columns
|
||||
keywords: ["select", "sql", "join", "group by", "window", "cte", "datafusion"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> SQL runs on a deployment. A local database
|
||||
has no engine to plan a statement -- see
|
||||
[Python, TypeScript, and SQL](/basics/apis).
|
||||
|
||||
The dialect is [DataFusion's](/sql/functions), so `SELECT` behaves as its
|
||||
[reference](https://datafusion.apache.org/user-guide/sql/select.html) documents.
|
||||
This page is the shape of that surface and where the SDKs meet it.
|
||||
|
||||
## The clauses {#clauses}
|
||||
|
||||
```sql
|
||||
[WITH cte AS (...)]
|
||||
SELECT [ALL | DISTINCT] select_expr [, ...]
|
||||
FROM from_item [[JOIN ...] ON ...]
|
||||
[WHERE predicate]
|
||||
[GROUP BY expr [, ...]]
|
||||
[HAVING predicate]
|
||||
[WINDOW name AS (...)]
|
||||
[ORDER BY expr [ASC | DESC] [NULLS FIRST | LAST] [, ...]]
|
||||
[LIMIT n] [OFFSET n]
|
||||
```
|
||||
|
||||
Set operations -- `UNION [ALL]`, `INTERSECT`, `EXCEPT` -- combine whole queries,
|
||||
and subqueries are allowed anywhere an expression or relation is.
|
||||
|
||||
## Projection {#projection}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL
|
||||
SELECT id, title, identifier FROM documents
|
||||
```
|
||||
|
||||
```python Python icon="python"
|
||||
tbl.search().select(["id", "title", "identifier"]).to_arrow()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await tbl.query().select(["id", "title", "identifier"]).toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Projection pushes into the scan in all three -- an unselected column is never
|
||||
read off storage. On a table of images or video that is the difference between a
|
||||
fast query and a slow one, so prefer naming columns over `SELECT *`.
|
||||
|
||||
Row metadata is available as [pseudo-columns](/sql/functions#pseudo-columns)
|
||||
(`_rowid`, `_rowaddr`, `_rowoffset`) but only when named, so they stay out of
|
||||
`SELECT *`.
|
||||
|
||||
## Filtering {#filtering}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL
|
||||
SELECT * FROM documents
|
||||
WHERE identifier > 0 AND identifier < 1000000
|
||||
```
|
||||
|
||||
```python Python icon="python"
|
||||
tbl.search().where("identifier > 0 AND identifier < 1000000").to_arrow()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await tbl.query().where("identifier > 0 AND identifier < 1000000").toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The SDK `where` takes the same expression language, so a predicate moves between
|
||||
them unchanged. A predicate on an indexed column becomes a
|
||||
`ScalarIndexQuery` rather than a scan -- confirm with
|
||||
[`explain_plan`](/search/explain). See [Filtering](/search/filtering) for the
|
||||
operators and their pushdown behaviour.
|
||||
|
||||
## Grouping and aggregation {#grouping}
|
||||
|
||||
```sql
|
||||
SELECT category, count(*) AS n, avg(score) AS mean_score
|
||||
FROM documents
|
||||
GROUP BY category
|
||||
HAVING count(*) > 10
|
||||
ORDER BY n DESC
|
||||
```
|
||||
|
||||
Aggregates, statistical and approximate functions come from DataFusion's
|
||||
[aggregate function set](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html).
|
||||
|
||||
<Note>
|
||||
The SDK query builders do not aggregate. Grouping, joins, and window functions
|
||||
are the reason to reach for SQL rather than a query builder -- see
|
||||
[choosing between them](/basics/apis#choosing).
|
||||
</Note>
|
||||
|
||||
## Joins {#joins}
|
||||
|
||||
```sql
|
||||
SELECT d.title, c.name
|
||||
FROM documents AS d
|
||||
JOIN categories AS c ON c.id = d.category_id
|
||||
```
|
||||
|
||||
`INNER`, `LEFT`, `RIGHT`, `FULL`, and `CROSS` joins are supported. Tables in
|
||||
different [namespaces](/namespaces) are joined by their dotted paths:
|
||||
|
||||
```sql
|
||||
SELECT * FROM prod.events.clicks AS c
|
||||
JOIN prod.reference.users AS u ON u.id = c.user_id
|
||||
```
|
||||
|
||||
## Window functions {#windows}
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
title,
|
||||
score,
|
||||
row_number() OVER (PARTITION BY category ORDER BY score DESC) AS rank
|
||||
FROM documents
|
||||
```
|
||||
|
||||
The full set is DataFusion's
|
||||
[window functions](https://datafusion.apache.org/user-guide/sql/window_functions.html).
|
||||
|
||||
## Common table expressions {#ctes}
|
||||
|
||||
```sql
|
||||
WITH recent AS (
|
||||
SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
|
||||
)
|
||||
SELECT category, count(*) FROM recent GROUP BY category
|
||||
```
|
||||
|
||||
A CTE that you want to keep is a [view](/management/view); one whose rows you
|
||||
want stored is a
|
||||
[materialized view](/feature-engineering/materialized-views).
|
||||
|
||||
## Search inside SELECT {#search}
|
||||
|
||||
Full-text search is a **table function**, not a predicate, because it produces
|
||||
relevance alongside the rows and `WHERE` has nowhere to put that:
|
||||
|
||||
```sql
|
||||
SELECT id, text, _score
|
||||
FROM fts('my_docs', 'puppy')
|
||||
ORDER BY _score DESC
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
Because it is a relation it composes -- join it, aggregate over it, wrap it in a
|
||||
CTE. For a keyword test that needs no score, the
|
||||
[`contains_tokens`](/sql/functions#contains-tokens) predicate is
|
||||
index-accelerated and usable directly in `WHERE`.
|
||||
|
||||
<Note>
|
||||
Vector search has no SQL form. It is a query-builder operation -- see
|
||||
[Vector search](/search/vector-search). A query that needs both vector
|
||||
similarity and SQL analytics runs the search in the SDK and the analytics over
|
||||
its result.
|
||||
</Note>
|
||||
|
||||
## Reading the plan {#plans}
|
||||
|
||||
`EXPLAIN` and `EXPLAIN ANALYZE` work as DataFusion defines them, and produce the
|
||||
same operators the SDK methods do:
|
||||
|
||||
```sql
|
||||
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
|
||||
```
|
||||
|
||||
See [Explain](/search/explain) and [Explain Analyze](/search/explain-analyze)
|
||||
for how to read the output.
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: "Aggregation"
|
||||
sidebarTitle: "Aggregation"
|
||||
description: "GROUP BY, HAVING, and the aggregate function set."
|
||||
icon: layer-group
|
||||
keywords: ["group by", "having", "count", "sum", "avg", "aggregate", "approximate"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Aggregation is one of the two main reasons to reach for SQL rather than a query
|
||||
builder -- the SDKs project, filter and search, but they do not group. The other
|
||||
is [joins](/sql/join).
|
||||
|
||||
## Grouping {#grouping}
|
||||
|
||||
```sql
|
||||
SELECT category, count(*) AS n, avg(score) AS mean_score
|
||||
FROM documents
|
||||
GROUP BY category
|
||||
HAVING count(*) > 10
|
||||
ORDER BY n DESC
|
||||
```
|
||||
|
||||
`WHERE` filters rows before grouping; `HAVING` filters groups after. Putting a
|
||||
row-level predicate in `HAVING` still gives the right answer but does more work,
|
||||
because rows that could have been discarded early are grouped first.
|
||||
|
||||
Grouping by several expressions produces one row per distinct combination:
|
||||
|
||||
```sql
|
||||
SELECT category, source, count(*) FROM documents GROUP BY category, source
|
||||
```
|
||||
|
||||
## Aggregate without grouping {#whole-table}
|
||||
|
||||
Omitting `GROUP BY` aggregates the whole table into one row -- the usual way to
|
||||
ask how big something is:
|
||||
|
||||
```sql
|
||||
SELECT count(*), min(created_at), max(created_at) FROM documents
|
||||
```
|
||||
|
||||
`count(*)` counts rows; `count(col)` counts non-null values, which is how you
|
||||
measure how populated a [computed column](/feature-engineering/add-computed-column)
|
||||
is before deciding to refresh it again.
|
||||
|
||||
## The function set {#functions}
|
||||
|
||||
The aggregates are DataFusion's, and the full list is in its
|
||||
[aggregate function reference](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html):
|
||||
|
||||
| Kind | Examples |
|
||||
| --- | --- |
|
||||
| General | `count`, `sum`, `min`, `max`, `avg` |
|
||||
| Statistical | `stddev`, `var`, `corr`, `covar`, `regr_*` |
|
||||
| Approximate | `approx_distinct`, `approx_percentile_cont`, `approx_median` |
|
||||
| Collecting | `array_agg`, `string_agg`, `first_value`, `last_value` |
|
||||
|
||||
The approximate ones are worth knowing about on a large table:
|
||||
`approx_distinct` answers "roughly how many distinct values" for a fraction of
|
||||
what `count(DISTINCT ...)` costs, which matters when the column has high
|
||||
cardinality.
|
||||
|
||||
## Aggregating a sample {#sampling}
|
||||
|
||||
For an estimate over a very large table, aggregate a
|
||||
[sample](/sql/sampling) rather than the whole thing:
|
||||
|
||||
```sql
|
||||
SELECT category, count(*) * 100 AS estimated
|
||||
FROM documents TABLESAMPLE (1)
|
||||
GROUP BY category
|
||||
```
|
||||
|
||||
Scale the result by the inverse of the sample fraction, and treat it as an
|
||||
estimate -- a 1% sample of a skewed column will miss rare categories entirely.
|
||||
|
||||
## What aggregation does not do {#not}
|
||||
|
||||
Aggregates run over rows the query already produced. They cannot combine with
|
||||
[vector search](/search/vector-search) in one statement, because similarity
|
||||
search has no SQL form. Run the search in an SDK and aggregate its result, or
|
||||
materialize the search output first.
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
title: "SQL"
|
||||
sidebarTitle: "Overview"
|
||||
description: "Analytical SQL over the lakehouse: the dialect, where it runs, and what it can reach."
|
||||
icon: table-list
|
||||
keywords: ["sql", "datafusion", "flight sql", "analytics", "dialect"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> SQL runs on a deployment. A local database
|
||||
has no engine to plan a statement, so `execute_query` raises there -- see
|
||||
[SDKs and SQL](/basics/apis).
|
||||
|
||||
The dialect is [Apache DataFusion's](/sql/functions), so anything its reference
|
||||
documents for that version works here unchanged. What these pages cover is the
|
||||
parts that are LanceDB's: how a query reaches the engine, how Lance columns
|
||||
appear to it, and the extensions a lakehouse over multimodal data needs.
|
||||
|
||||
## Where a query runs {#execution}
|
||||
|
||||
Queries execute on the server. The client sends a statement and receives Arrow
|
||||
record batches -- nothing is converted to pandas or Python objects unless you
|
||||
ask for it. A query feeding training or downstream processing therefore pays no
|
||||
serialization cost to get there.
|
||||
|
||||
Results stream. `execute_query` hands back a reader as soon as the first batch
|
||||
is ready rather than when the query finishes, so a large result set does not
|
||||
have to fit in memory before you can start reading it. See
|
||||
[Query and job](/basics/jobs#queries).
|
||||
|
||||
## What differs from a warehouse {#differences}
|
||||
|
||||
Two things, and both come from the storage being Lance rather than something
|
||||
SQL-shaped:
|
||||
|
||||
- **Columns are Arrow types**, not a SQL type system mapped onto storage. A
|
||||
vector column is a `FixedSizeList<Float32>` and behaves like one. See
|
||||
[Data types](/basics/data-types).
|
||||
- **Row metadata is available as columns** -- `_rowid`, `_rowaddr`,
|
||||
`_rowoffset` -- but only when named, so they stay out of `SELECT *`.
|
||||
|
||||
## The statements {#statements}
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| [SELECT](/sql/select) | projection, filtering, ordering, set operations, CTEs |
|
||||
| [Aggregation](/sql/aggregation) | `GROUP BY`, `HAVING`, aggregate functions |
|
||||
| [Joins](/sql/join) | inner, outer, cross, and joining across namespaces |
|
||||
| [Window functions](/sql/window) | `OVER (...)`, ranking, running totals |
|
||||
| [Sampling](/sql/sampling) | `TABLESAMPLE`, for a random subset |
|
||||
| [Functions and types](/sql/functions) | the function set, and what LanceDB adds |
|
||||
|
||||
Beyond reading, SQL also writes (`INSERT`, `UPDATE`, `DELETE`, `COPY`), manages
|
||||
the catalog (`CREATE TABLE`, `CREATE VIEW`, `DROP`, `SHOW`) and drives feature
|
||||
engineering (`CREATE FUNCTION`, `REFRESH COLUMN`, `REFRESH MATERIALIZED VIEW`).
|
||||
Those are documented with the operation rather than here -- see
|
||||
[Management](/management) and [Feature engineering](/feature-engineering).
|
||||
|
||||
<Note>
|
||||
Vector search has no SQL form. It is a query-builder operation -- see
|
||||
[Vector search](/search/vector-search). Full-text search does: the
|
||||
[`fts()`](/sql/functions#fts) table function and the
|
||||
[`contains_tokens`](/sql/functions#contains-tokens) predicate.
|
||||
</Note>
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: "Joins"
|
||||
sidebarTitle: "Join"
|
||||
description: "Combining rows from more than one table, including across namespaces."
|
||||
icon: link
|
||||
keywords: ["join", "inner", "left", "outer", "cross", "namespace"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
Joining is the other thing the query builders cannot do. A question spanning two
|
||||
tables is a SQL question.
|
||||
|
||||
## The join types {#types}
|
||||
|
||||
```sql
|
||||
SELECT d.title, c.name
|
||||
FROM documents AS d
|
||||
JOIN categories AS c ON c.id = d.category_id
|
||||
```
|
||||
|
||||
| | What it keeps |
|
||||
| --- | --- |
|
||||
| `[INNER] JOIN` | rows matching on both sides |
|
||||
| `LEFT [OUTER] JOIN` | every left row, nulls where the right has no match |
|
||||
| `RIGHT [OUTER] JOIN` | every right row |
|
||||
| `FULL [OUTER] JOIN` | every row from both |
|
||||
| `CROSS JOIN` | every combination -- the product of the two |
|
||||
|
||||
`USING (col)` is shorthand for an equality join on a shared column name, and
|
||||
avoids repeating it in the select list.
|
||||
|
||||
## Across namespaces {#namespaces}
|
||||
|
||||
Tables in different [namespaces](/namespaces) join by their dotted paths -- the
|
||||
catalog is one catalog, so there is nothing to federate:
|
||||
|
||||
```sql
|
||||
SELECT c.id, u.email
|
||||
FROM prod.events.clicks AS c
|
||||
JOIN prod.reference.users AS u ON u.id = c.user_id
|
||||
```
|
||||
|
||||
## Joining a search result {#search}
|
||||
|
||||
[`fts()`](/sql/functions#fts) is a table function, so its result is a relation
|
||||
and joins like any other:
|
||||
|
||||
```sql
|
||||
SELECT c.name, count(*) AS hits
|
||||
FROM fts('my_docs', 'puppy') AS m
|
||||
JOIN categories AS c ON c.id = m.category_id
|
||||
GROUP BY c.name
|
||||
ORDER BY hits DESC
|
||||
```
|
||||
|
||||
That composition is the reason full-text search is a relation rather than a
|
||||
predicate -- a `WHERE` clause has nowhere to put the relevance score.
|
||||
|
||||
## Performance {#performance}
|
||||
|
||||
A join key with a [scalar index](/indexing/scalar-index) on it lets the planner
|
||||
look rows up rather than scan for them. Check with
|
||||
[`explain_plan`](/search/explain): an indexed key shows a `ScalarIndexQuery`,
|
||||
an unindexed one shows a `LanceScan` feeding the join.
|
||||
|
||||
<Warning>
|
||||
Avoid selecting blob or vector columns you do not need on either side of a join.
|
||||
The join multiplies rows before projection is applied in some plans, so a wide
|
||||
column can be materialized many more times than the result has rows.
|
||||
</Warning>
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: "Random sampling"
|
||||
sidebarTitle: "Sampling"
|
||||
description: "TABLESAMPLE: draw a random subset of a table, by rows or by fragments."
|
||||
icon: dice
|
||||
keywords: ["tablesample", "sampling", "bernoulli", "system", "repeatable", "random"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
`TABLESAMPLE` draws a random subset of a table. On a lakehouse table that is
|
||||
often the difference between a question you can afford to ask and one you
|
||||
cannot -- an aggregate over 1% of forty billion rows answers in a fraction of
|
||||
the time, and for most estimates that is enough.
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE (10) -- 10 percent
|
||||
SELECT * FROM documents TABLESAMPLE (500 ROWS) -- 500 rows
|
||||
```
|
||||
|
||||
A bare number is a **percentage**. Add `ROWS` for a count instead.
|
||||
|
||||
## Rows or fragments {#methods}
|
||||
|
||||
The standard calls these two methods of one clause. They are different
|
||||
operations, and the difference matters on Lance:
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE BERNOULLI (10)
|
||||
SELECT * FROM documents TABLESAMPLE SYSTEM (10)
|
||||
```
|
||||
|
||||
| Method | Aliases | What it picks |
|
||||
| --- | --- | --- |
|
||||
| `BERNOULLI` | `ROW` | rows, each considered independently |
|
||||
| `SYSTEM` | `BLOCK` | **Lance fragments**, and takes whatever rows they hold |
|
||||
|
||||
`BERNOULLI` gives a genuinely uniform sample and reads across the whole table to
|
||||
get it. `SYSTEM` reads far less -- it skips entire fragments -- but the sample is
|
||||
only as uniform as the data's layout. If rows were written in an order that
|
||||
correlates with what you are measuring, a fragment sample inherits that
|
||||
correlation.
|
||||
|
||||
Reach for `SYSTEM` when you want a cheap look at the data and `BERNOULLI` when
|
||||
the number you compute has to be defensible. Omitting the method gives
|
||||
`BERNOULLI`.
|
||||
|
||||
## Repeatability {#repeatable}
|
||||
|
||||
Without a seed, every execution resamples -- running the same query twice gives
|
||||
different rows. `REPEATABLE` fixes the draw:
|
||||
|
||||
```sql
|
||||
SELECT * FROM documents TABLESAMPLE (10) REPEATABLE (42)
|
||||
```
|
||||
|
||||
This is what makes a sample usable as a **fixed evaluation set**: the same seed
|
||||
against the same data gives the same rows, so a metric computed today is
|
||||
comparable with one computed last week.
|
||||
|
||||
## Where the clause can go {#placement}
|
||||
|
||||
It attaches to a table in `FROM`, including inside a subquery, a CTE, or either
|
||||
side of a join:
|
||||
|
||||
```sql
|
||||
SELECT * FROM (SELECT * FROM documents) d TABLESAMPLE (10)
|
||||
|
||||
SELECT * FROM a TABLESAMPLE (5 ROWS) JOIN b ON a.id = b.id
|
||||
|
||||
SELECT * FROM t a TABLESAMPLE (10) JOIN t b TABLESAMPLE (10) ON a.id = b.id
|
||||
```
|
||||
|
||||
Each side of a join samples independently. Sampling both sides of a join
|
||||
shrinks the *result* far more than either fraction suggests, because a row only
|
||||
survives if both its sides were drawn.
|
||||
|
||||
## What is refused {#refused}
|
||||
|
||||
Sampling is semantics, not a hint. A clause that were read and then not applied
|
||||
would be a wrong answer rather than a missing feature, so anything that cannot
|
||||
be honoured exactly is an error:
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `TABLESAMPLE (101)` | a percentage outside `0..=100` is refused, not clamped |
|
||||
| `TABLESAMPLE (1.5 ROWS)` | a row count must be whole |
|
||||
| `TABLESAMPLE (BUCKET 1 OUT OF 4)` | bucket sampling is not supported |
|
||||
| the `SAMPLE` spelling | use `TABLESAMPLE` |
|
||||
| the `OFFSET` modifier | not supported |
|
||||
|
||||
<Note>
|
||||
`TABLESAMPLE` is also refused on a WAL-backed table, where the rows a sample
|
||||
would draw from are not all in the fragments yet.
|
||||
</Note>
|
||||
|
||||
## Sampling outside SQL {#sdk}
|
||||
|
||||
For a training loop rather than a query, the SDKs read a table in a shuffled
|
||||
order without going through SQL at all -- see [Training](/training), where a
|
||||
permutation drives the data loader.
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
title: "SELECT"
|
||||
sidebarTitle: "Select"
|
||||
description: "Projection, filtering, ordering, and composing queries out of other queries."
|
||||
icon: table-columns
|
||||
keywords: ["select", "where", "order by", "limit", "cte", "union", "distinct"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
```sql
|
||||
[WITH cte AS (...)]
|
||||
SELECT [ALL | DISTINCT] select_expr [, ...]
|
||||
FROM from_item [[JOIN ...] ON ...]
|
||||
[WHERE predicate]
|
||||
[GROUP BY expr [, ...]] [HAVING predicate]
|
||||
[WINDOW name AS (...)]
|
||||
[ORDER BY expr [ASC | DESC] [NULLS FIRST | LAST] [, ...]]
|
||||
[LIMIT n] [OFFSET n]
|
||||
```
|
||||
|
||||
## Projection {#projection}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL
|
||||
SELECT id, title, identifier FROM documents
|
||||
```
|
||||
|
||||
```python Python icon="python"
|
||||
tbl.search().select(["id", "title", "identifier"]).to_arrow()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await tbl.query().select(["id", "title", "identifier"]).toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Projection pushes into the scan in all three -- an unselected column is never
|
||||
read off storage. On a table of images or video that is the difference between a
|
||||
fast query and a slow one, so **prefer naming columns over `SELECT *`**.
|
||||
|
||||
Row metadata is available as [pseudo-columns](/sql/functions#pseudo-columns)
|
||||
(`_rowid`, `_rowaddr`, `_rowoffset`) but only when named, so they stay out of
|
||||
`SELECT *`.
|
||||
|
||||
`DISTINCT` deduplicates the projected rows.
|
||||
|
||||
## Filtering {#filtering}
|
||||
|
||||
<CodeGroup>
|
||||
```sql SQL
|
||||
SELECT * FROM documents
|
||||
WHERE identifier > 0 AND identifier < 1000000
|
||||
```
|
||||
|
||||
```python Python icon="python"
|
||||
tbl.search().where("identifier > 0 AND identifier < 1000000").to_arrow()
|
||||
```
|
||||
|
||||
```typescript TypeScript icon="square-js"
|
||||
await tbl.query().where("identifier > 0 AND identifier < 1000000").toArrow();
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The SDK `where` takes the same expression language, so a predicate moves between
|
||||
them unchanged. A predicate on an indexed column becomes a `ScalarIndexQuery`
|
||||
rather than a scan -- confirm with [`explain_plan`](/search/explain). See
|
||||
[Filtering](/search/filtering) for the operators and their pushdown behaviour.
|
||||
|
||||
## Ordering and limiting {#order-limit}
|
||||
|
||||
```sql
|
||||
SELECT id, title FROM documents
|
||||
ORDER BY created_at DESC NULLS LAST
|
||||
LIMIT 100 OFFSET 200
|
||||
```
|
||||
|
||||
`ORDER BY` on a large table is a sort over the whole result, so pair it with a
|
||||
`LIMIT` unless you genuinely need every row ordered.
|
||||
|
||||
## Set operations {#set-operations}
|
||||
|
||||
`UNION`, `UNION ALL`, `INTERSECT` and `EXCEPT` combine whole queries:
|
||||
|
||||
```sql
|
||||
SELECT id FROM documents WHERE category = 'film'
|
||||
UNION ALL
|
||||
SELECT id FROM archived_documents WHERE category = 'film'
|
||||
```
|
||||
|
||||
`UNION` deduplicates and `UNION ALL` does not, which makes `UNION ALL` the
|
||||
cheaper one when you already know the inputs are disjoint.
|
||||
|
||||
## Common table expressions {#ctes}
|
||||
|
||||
```sql
|
||||
WITH recent AS (
|
||||
SELECT * FROM documents WHERE created_at > now() - INTERVAL '7 days'
|
||||
)
|
||||
SELECT category, count(*) FROM recent GROUP BY category
|
||||
```
|
||||
|
||||
A CTE is scoped to the statement. One you want to keep is a
|
||||
[view](/management/view); one whose rows you want stored is a
|
||||
[materialized view](/management/materialized-view).
|
||||
|
||||
## Subqueries {#subqueries}
|
||||
|
||||
Subqueries are allowed anywhere an expression or a relation is -- in `FROM`, in
|
||||
`WHERE` with `IN` or `EXISTS`, and as a scalar in the select list:
|
||||
|
||||
```sql
|
||||
SELECT title
|
||||
FROM documents
|
||||
WHERE category_id IN (SELECT id FROM categories WHERE active)
|
||||
```
|
||||
|
||||
## Namespaced tables {#namespaces}
|
||||
|
||||
A table inside a [namespace](/namespaces) is written as a dotted path:
|
||||
|
||||
```sql
|
||||
SELECT * FROM prod.events.clicks
|
||||
```
|
||||
|
||||
## Reading the plan {#plans}
|
||||
|
||||
`EXPLAIN` and `EXPLAIN ANALYZE` work as DataFusion defines them, and produce the
|
||||
same operators the SDK methods do:
|
||||
|
||||
```sql
|
||||
EXPLAIN ANALYZE SELECT * FROM documents WHERE identifier < 1000
|
||||
```
|
||||
|
||||
See [Explain](/search/explain) and [Explain Analyze](/search/explain-analyze).
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: "Window functions"
|
||||
sidebarTitle: "Window"
|
||||
description: "Compute over a set of rows related to the current one, without collapsing them."
|
||||
icon: chart-simple
|
||||
keywords: ["window", "over", "partition by", "row_number", "rank", "lag", "lead"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge> [SQL runs on a deployment.](/sql)
|
||||
|
||||
A window function computes across a set of rows related to the current one and
|
||||
returns a value **per row**. That is the difference from
|
||||
[aggregation](/sql/aggregation), which collapses the rows it covers.
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
title,
|
||||
score,
|
||||
row_number() OVER (PARTITION BY category ORDER BY score DESC) AS rank
|
||||
FROM documents
|
||||
```
|
||||
|
||||
Every row survives; each one gains its rank within its category.
|
||||
|
||||
## The window {#window}
|
||||
|
||||
```sql
|
||||
OVER (
|
||||
[PARTITION BY expr [, ...]]
|
||||
[ORDER BY expr [, ...]]
|
||||
[ROWS | RANGE BETWEEN frame_start AND frame_end]
|
||||
)
|
||||
```
|
||||
|
||||
`PARTITION BY` splits the rows into independent groups. `ORDER BY` orders them
|
||||
inside each partition -- required by the ranking functions, since a rank without
|
||||
an order is meaningless. The frame clause narrows the window further, which is
|
||||
how running totals and moving averages are expressed:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
created_at,
|
||||
sum(amount) OVER (ORDER BY created_at ROWS UNBOUNDED PRECEDING) AS running_total,
|
||||
avg(amount) OVER (ORDER BY created_at ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS trailing_7
|
||||
FROM transactions
|
||||
```
|
||||
|
||||
A `WINDOW` clause names a definition when several functions share one:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
row_number() OVER w,
|
||||
lag(score) OVER w
|
||||
FROM documents
|
||||
WINDOW w AS (PARTITION BY category ORDER BY score DESC)
|
||||
```
|
||||
|
||||
## The functions {#functions}
|
||||
|
||||
| Kind | Functions |
|
||||
| --- | --- |
|
||||
| Ranking | `row_number`, `rank`, `dense_rank`, `percent_rank`, `ntile`, `cume_dist` |
|
||||
| Offset | `lag`, `lead`, `first_value`, `last_value`, `nth_value` |
|
||||
| Aggregate | any [aggregate](/sql/aggregation) used with `OVER` |
|
||||
|
||||
The full set is DataFusion's
|
||||
[window function reference](https://datafusion.apache.org/user-guide/sql/window_functions.html).
|
||||
|
||||
## Deduplicating with a window {#dedup}
|
||||
|
||||
The common practical use: keep one row per key, chosen by some ordering. A
|
||||
window ranks, and an outer query filters:
|
||||
|
||||
```sql
|
||||
WITH ranked AS (
|
||||
SELECT *, row_number() OVER (PARTITION BY doc_id ORDER BY version DESC) AS rn
|
||||
FROM revisions
|
||||
)
|
||||
SELECT * FROM ranked WHERE rn = 1
|
||||
```
|
||||
|
||||
The outer query is needed because a window function cannot appear in `WHERE` --
|
||||
windows are computed after filtering, so the predicate would have nothing to
|
||||
read.
|
||||
|
||||
<Note>
|
||||
Window functions sort within each partition, so on a large table they are among
|
||||
the more expensive things you can ask for. Filter first, and check the cost with
|
||||
[Explain Analyze](/search/explain-analyze).
|
||||
</Note>
|
||||
@@ -1,237 +0,0 @@
|
||||
---
|
||||
title: Multimodal Data (Blobs)
|
||||
sidebarTitle: "Working with multimodal data"
|
||||
description: Learn how to store and query multimodal data (images, audio, video) directly in LanceDB using binary columns.
|
||||
icon: "images"
|
||||
keywords: ["blob", "large binary", "blobs", "multimodal"]
|
||||
---
|
||||
|
||||
import {
|
||||
PyMultimodalImports as MultimodalImports,
|
||||
TsMultimodalImports as TsMultimodalImports,
|
||||
RsMultimodalImports as RsMultimodalImports,
|
||||
PyCreateDummyData as CreateDummyData,
|
||||
TsCreateDummyData as TsCreateDummyData,
|
||||
RsCreateDummyData as RsCreateDummyData,
|
||||
PyDefineSchema as DefineSchema,
|
||||
TsDefineSchema as TsDefineSchema,
|
||||
RsDefineSchema as RsDefineSchema,
|
||||
PyIngestData as IngestData,
|
||||
TsIngestData as TsIngestData,
|
||||
RsIngestData as RsIngestData,
|
||||
PySearchData as SearchData,
|
||||
TsSearchData as TsSearchData,
|
||||
RsSearchData as RsSearchData,
|
||||
PyProcessResults as ProcessResults,
|
||||
TsProcessResults as TsProcessResults,
|
||||
RsProcessResults as RsProcessResults,
|
||||
PyBlobApiSchema as BlobApiSchema,
|
||||
TsBlobApiSchema as TsBlobApiSchema,
|
||||
RsBlobApiSchema as RsBlobApiSchema,
|
||||
PyBlobApiIngest as BlobApiIngest,
|
||||
TsBlobApiIngest as TsBlobApiIngest,
|
||||
RsBlobApiIngest as RsBlobApiIngest,
|
||||
PyBlobApiToPandas as BlobApiToPandas,
|
||||
PyQueryToPandasKwargs as QueryToPandasKwargs,
|
||||
} from '/snippets/multimodal.mdx';
|
||||
|
||||
LanceDB handles multimodal data—images, audio, video, and PDF files—natively by storing the raw bytes in a binary column alongside your vectors and metadata. This approach simplifies your data infrastructure by keeping the raw assets and their embeddings in the same database, eliminating the need for separate object storage for many use cases.
|
||||
|
||||
This guide demonstrates how to ingest, store, and retrieve image data using standard binary columns, and also introduces the **Lance Blob API** for optimized handling of larger multimodal files.
|
||||
|
||||
## Store binary data {#store-binary-data}
|
||||
|
||||
To store binary data, define a binary Arrow field in your schema (`pa.binary()` in Python, `Binary` in TypeScript, and `DataType::Binary` in Rust).
|
||||
|
||||
### 1\. Setup and imports {#1-setup-and-imports}
|
||||
|
||||
First, import the necessary libraries for LanceDB and Arrow in your SDK.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{MultimodalImports}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsMultimodalImports}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsMultimodalImports}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### 2\. Prepare data {#2-prepare-data}
|
||||
|
||||
For this example, we'll create some dummy in-memory images. In a real application, you would read these from files or an API. The key is to convert your data (image, audio, etc.) into a raw `bytes` object.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{CreateDummyData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsCreateDummyData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsCreateDummyData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### 3\. Define the schema {#3-define-the-schema}
|
||||
|
||||
When creating the table, it is **highly recommended** to define the schema explicitly. This ensures that your binary data is correctly interpreted as a `binary` type by Arrow/LanceDB and not as a generic string or list.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{DefineSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsDefineSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsDefineSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### 4\. Ingest data {#4-ingest-data}
|
||||
|
||||
Now, create the table using the data and the defined schema.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{IngestData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsIngestData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsIngestData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Retrieve and use blobs {#retrieve-and-use-blobs}
|
||||
|
||||
When you search your LanceDB table, you can retrieve the binary column just like any other metadata.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{SearchData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsSearchData}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsSearchData}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### Convert bytes back to objects {#convert-bytes-back-to-objects}
|
||||
|
||||
Once you have the bytes back from the search result, you can decode them into the original format (for example, an image object or audio buffer).
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{ProcessResults}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsProcessResults}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsProcessResults}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Large Blobs (Blob API) {#large-blobs-blob-api}
|
||||
|
||||
For larger files like high-resolution images or videos, Lance provides a specialized **Blob API**. By using a large-binary Arrow type (`pa.large_binary()` in Python, `LargeBinary` in TypeScript, and `DataType::LargeBinary` in Rust) and specific metadata, you enable **lazy loading** and optimized encoding. This allows you to work with massive datasets without loading all binary data into memory upfront.
|
||||
|
||||
### 1\. Define a blob schema {#1-define-a-blob-schema}
|
||||
|
||||
To use the Blob API, you must mark the column with `{"lance-encoding:blob": "true"}` metadata.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBlobApiSchema}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBlobApiSchema}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
### 2\. Ingest large blobs {#2-ingest-large-blobs}
|
||||
|
||||
You can then ingest data normally, and Lance will handle the optimized storage.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiIngest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
||||
{TsBlobApiIngest}
|
||||
</CodeBlock>
|
||||
|
||||
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
||||
{RsBlobApiIngest}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
<Card>
|
||||
For more advanced usage, including random access and file-like reading of blobs, see the
|
||||
Lance format's [blob API documentation](https://lance.org/guide/blob/).
|
||||
</Card>
|
||||
|
||||
### 3\. Convert blob tables to pandas {#3-convert-blob-tables-to-pandas}
|
||||
|
||||
When you call `to_pandas()` on a local LanceDB table that contains Blob API columns, the `blob_mode` argument controls how those columns materialize. This is available in the Python SDK on local tables; remote tables raise `NotImplementedError`.
|
||||
|
||||
`blob_mode` accepts:
|
||||
|
||||
- `"lazy"` (default): returns blob columns as lazy `BlobFile` objects without eagerly materializing their payloads. Use this when you want to stream blob bytes on demand or only inspect a subset of rows. Namespace-backed local tables also use the Lance native blob-aware pandas conversion for lazy blobs; in-memory datasets fall back to the standard PyArrow `to_pandas()` path.
|
||||
- `"bytes"`: eagerly materializes each blob as `bytes`. Use this when you need the raw payload in the DataFrame, for example to decode an image or audio clip in-process.
|
||||
- `"descriptions"`: returns blob descriptors (offsets, sizes, and positions) instead of the data itself. Use this when you want to plan I/O without paying the cost of loading every blob.
|
||||
|
||||
`"bytes"` and `"descriptions"` require a filesystem-backed Lance dataset and are not supported on in-memory tables.
|
||||
|
||||
Extra keyword arguments are forwarded to the underlying PyArrow / Lance pandas conversion, so you can also pass options like `split_blocks` or `self_destruct`:
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{BlobApiToPandas}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
Query builders also accept `blob_mode` on their `to_pandas()` method:
|
||||
|
||||
- Plain scans support `"lazy"`, `"bytes"`, and `"descriptions"` with filters, projections, aliases, `limit`, and `offset`.
|
||||
- Vector, FTS, hybrid, and ordered queries can't materialize blob columns through `to_pandas()`; omit blob columns from the projection for those query shapes.
|
||||
- This works on both sync and async query builders. Extra PyArrow kwargs like `split_blocks` and `self_destruct` are still forwarded.
|
||||
|
||||
<CodeGroup>
|
||||
<CodeBlock filename="Python" language="Python" icon="python">
|
||||
{QueryToPandasKwargs}
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Other modalities {#other-modalities}
|
||||
|
||||
The `pa.binary()` and `pa.large_binary()` types are universal. You can use this same pattern for other types of multimodal data:
|
||||
|
||||
- **Audio:** Read `.wav` or `.mp3` files as bytes.
|
||||
- **Video:** Store video transitions or full clips using the Blob API.
|
||||
- **PDFs/Documents:** Store the raw file content for document search.
|
||||
Reference in New Issue
Block a user