rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-13 07:12:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: add data types to Basics, split the demo gallery, regroup tabs

Browse Source 
Basics gains a Schema & Data Types section -- the Arrow types a table can hold,
the extension types Lance adds on top (lance.json, lance.blob.v2,
lance.bfloat16), and where geospatial actually stands. The multimodal page moves
in beside it. Consistency joins Basics too, after Query & Job.

Query & Job now leads with the thing the _async convention actually gives you: a
handle. execute_query_async returns a query, everything else returns a job, and
both answer the same four questions -- which one, how is it going, give me the
result, stop. Only the third differs, because a query produces rows to stream
and a job produces an outcome to wait for.

Management gains a Materialized View page. Creating one was documented only on
the refresh page, and the stable-row-ids requirement is a creation-time
decision that cannot be undone.

Embeddings move to Integrations as the Embedding Registry, which is what they
are -- a client-side feature that wraps provider models, sitting above the
providers it registers.

The demo gallery was three apps on one page; it is now three pages. Use Cases
folds into Tutorials as Tutorials and Use Cases.
This commit is contained in:
Jack Ye
2026-09-09 23:38:50 -07:00
parent 139138c22e
commit 83595ac839
9 changed files with 401 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/basics/data-types.mdx
+143
View File
@@ -0,0 +1,143 @@
---
title: "Schema and data types"
sidebarTitle: "Data Types"
description: "The Arrow type system LanceDB stores, and the extension types Lance adds on top of it."
icon: shapes
keywords: ["schema", "data types", "arrow", "json", "blob", "bfloat16", "vector", "geospatial"]
---
A table's schema is an **Arrow schema**. There is no separate LanceDB type
system mapped onto storage -- a column is the same object to the query engine,
to the format, and to any other engine reading the files.
That identity is mostly invisible and occasionally load-bearing, so it is worth
knowing what it admits.
## Supported types {#supported}
| Category | Types |
| --- | --- |
| Null | `null` |
| Boolean | `bool` |
| Integer | `int8`, `int16`, `int32`, `int64` and their `uint` counterparts |
| Floating point | `float16`, `float32`, `float64` |
| String | `utf8`, `large_utf8` |
| Binary | `binary`, `large_binary`, `fixed_size_binary(n)` |
| Date and time | `date32`, `date64`, `time32`, `time64`, `timestamp` (with or without timezone), `duration` |
| Nested | `struct`, `list`, `large_list`, `fixed_size_list`, `map` |
| Dictionary | `dictionary(key, value)` |
Two limits are worth stating because they surface as schema errors rather than
as documentation:
- A `map` with `keys_sorted = true` is rejected. Sorted-key maps do not survive
the round trip through the Python bindings, so they are refused rather than
silently unsorted.
- Not every type is legal as a `fixed_size_list` child. The vector case is,
which is what matters in practice.
## Vectors are fixed-size lists {#vectors}
A vector column is a `fixed_size_list<float32, dim>` -- an ordinary Arrow type,
not a special one:
```python
import pyarrow as pa
schema = pa.schema([
pa.field("id", pa.int64()),
pa.field("vector", pa.list_(pa.float32(), 1536)),
])
```
Because it is ordinary, it is selectable, testable for nullness, and joinable
like any other column. The consequence people trip on is the other side of that:
arithmetic over it is **list** arithmetic, not scalar arithmetic. When the
operation you want is a similarity comparison, use
[vector search](/search/vector-search) rather than SQL.
Without an explicit schema, Python infers a list of floats as a fixed-size
`float32` vector from the dimension it observes in the data.
## Extension types {#extensions}
Lance adds a few types as **Arrow extension types** -- an underlying storage
type plus an `ARROW:extension:name` marker. Any Arrow reader can read the
storage; readers that know the extension get the richer behaviour.
| Type | Extension name | Stored as |
| --- | --- | --- |
| [JSON](#json) | `lance.json` | `large_binary`, JSONB-encoded |
| [Blob](#blob) | `lance.blob.v2` | `large_binary`, stored out of line |
| bfloat16 | `lance.bfloat16` | `fixed_size_binary(2)` |
`arrow.json` is accepted as well, for interoperability with writers that use
Arrow's own canonical JSON extension name.
### JSON {#json}
A JSON column stores documents in **JSONB**, a binary encoding, rather than as
text. Accessors read into a document without parsing the whole thing:
```sql
SELECT json_get_string(data, 'name') FROM events
WHERE json_get_int(data, 'status') = 200
```
`json_get` returns JSON, so accessors nest for a nested document. **A JSON
scalar index accelerates `json_get`, its typed variants, and `json_extract`** --
the rest evaluate per row. The full function list is in
[SQL functions](/sql/functions#json).
This is the type to reach for when a column's shape varies per row. When it does
not, ordinary `struct` columns are faster and typed.
### Blob {#blob}
A blob column holds large binary objects -- images, video, audio, model
artifacts -- **out of line** from the rest of the row. That is what keeps a scan
over the other columns fast on a table whose rows are megabytes each.
A blob can be read two ways, and the choice changes the column's Arrow type:
- as its **binary contents**, or
- as a **description** -- a pointer plus metadata, without fetching the bytes.
<Warning>
The default reads blob columns as descriptions and other binary columns as
binary. A query that selects a blob column expecting bytes gets a struct
instead, unless the read is configured otherwise. Worth knowing before writing a
predicate against one.
</Warning>
See [Working with multimodal data](/tables/multimodal).
### bfloat16 {#bfloat16}
Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`.
Useful as a vector element type when halving the memory matters more than the
precision does.
## Geospatial {#geospatial}
<Warning>
Lance defines geospatial types and `st_*` functions on
[GeoArrow](https://geoarrow.org), but they sit behind an optional build feature
that **LanceDB does not ship**. Calling `st_intersects` and friends returns an
error naming the missing feature rather than an unknown-function error, and the
spatial index is gated the same way.
</Warning>
Geometry can still be stored -- it is binary, and binary columns work -- but
nothing interprets it. Filter on a bounding box held in ordinary numeric
columns if you need spatial predicates today.
## Changing types later {#evolution}
A schema is not fixed at creation. Columns can be added, renamed, retyped, made
nullable, and dropped, usually without rewriting existing data -- see
[Schema evolution](/tables/schema). Deriving new values from existing columns is
[data evolution](/feature-engineering/data-evolution).
Nullability is worth deciding up front: if later writes omit a non-nullable
column, or write actual nulls into it, ingestion fails.
docs/web/basics/jobs.mdx
+40 -6
View File
@@ -12,11 +12,34 @@ common source of confusion about how work gets done.
| | What it is | What it changes |
| --- | --- | --- |
| **Async client** | `connect_async`, `await` | whether *your thread* blocks |
| **Async execution** | `_async` methods returning a [job](#jobs) | whether *the work* outlives the call |
| **Async execution** | `_async` methods returning a handle | whether *the work* outlives the call |
They are independent. A synchronous Python call can start a job that runs for an
They are independent. A synchronous Python call can start work that runs for an
hour; an `await` can wait for something that finishes instantly.
## Every `_async` method returns a handle {#handles}
An `_async` method submits work and hands back a **handle** to it. Which handle
depends on what you submitted -- `execute_query_async` returns a query,
everything else returns a [job](#jobs) -- but they are the same idea, and they
answer the same four questions:
| | Query handle | Job handle |
| --- | --- | --- |
| Which one is this? | `id` | `id` |
| How is it going? | `describe()` | `status()`, `refresh()`, `events()` |
| Give me the result | `reader()` — streams batches as they arrive | `wait()` — blocks for the terminal result |
| Stop it | `cancel()` | `cancel()` |
The difference is in the third row, and it follows from what the work produces.
A query produces **rows**, so its handle streams them and you can start reading
before the query finishes. A job produces **an outcome** -- an index built, a
column filled -- so its handle has nothing to stream and gives you the result
once, at the end.
Everything else is shared: both identify the work on the server, both report on
it without blocking, and both can be told to stop.
## Async client: whether you block {#async-client}
A concern of your language, not of LanceDB. The work is identical either way.
@@ -61,11 +84,22 @@ SELECT id, text FROM my_docs LIMIT 1000
`execute_query` returns a blocking Arrow reader **once results start arriving**
-- it does not wait for the whole query to finish. Results stream as record
batches, so a large result set does not have to fit in memory before you see
its first row.
batches, so a large result set does not have to fit in memory before you see its
first row.
`execute_query_async` returns the query handle instead, for a statement you
would rather not sit on at all. `describe_query` reports on one by id.
`execute_query_async` returns the query handle instead:
```python
query = db.execute_query_async("SELECT id, text FROM my_docs")
query.id # identifies it on the server
query.describe() # what it is doing, without consuming results
reader = query.reader() # stream the rows when you want them
query.cancel() # stop it
```
`describe_query(id)` reports on one from elsewhere, the way `open_job(id)` does
for a job.
<Note>
SQL requires a deployment. See [SDKs and SQL](/basics/apis).
docs/web/demos/index.mdx
+11 -74
View File
@@ -1,78 +1,15 @@
---
title: "Demo Application Gallery"
sidebarTitle: "App Gallery"
description: "Demo apps showcasing end-to-end applications built with LanceDB for production use cases."
title: "Demo application gallery"
sidebarTitle: "Overview"
description: "End-to-end applications built with LanceDB."
icon: grid-2
---
Explore the demo applications built with LanceDB below.
Applications built on LanceDB, each with a live demo and a write-up of how it
works.
App | Description
--- | ---
[Semantic.Art](#semantic-art) | A multimodal art discovery platform using feelings, phrases, and images.
[Wikipedia 41M Hybrid Search](#wikipedia-41m-hybrid-search) | An interactive hybrid search demo combining full-text search and vector search.
[Video Search](#video-search) | A video search application that allows searching through a library of videos using natural language queries.
## Semantic.art {#semantic-art}
<Badge color="green">multimodal</Badge>
<Badge color="red">hybrid-search</Badge>
<Badge color="purple">vector-search</Badge>
<Badge color="surface-destructive">semantic-routing</Badge>
<Card
title="Semantic.art"
img="/static/assets/images/demos/sda_hero.jpg"
href="https://www.semantic.art/"
>
Semantic.art turns real, human-made art discovery into a multimodal search experience using
feelings, phrases, and images. It's built with LanceDB hybrid search and semantic routing.
<Card
title="Learn how it's built"
href="https://lancedb.com/blog/semanticdotart/"
>
Read in detail about how Semantic.art is built in this blog post.
</Card>
</Card>
## Wikipedia 41M Hybrid Search {#wikipedia-41m-hybrid-search}
<Badge color="green">multimodal</Badge>
<Badge color="red">hybrid-search</Badge>
<Badge color="purple">vector-search</Badge>
<Badge color="gray">fts</Badge>
<Card
title="Wikipedia 41M Hybrid Search"
img="/static/assets/images/demos/wiki_hero.png"
href="https://lancedb-demos.vercel.app/demo/wikipedia-search"
>
Interactive hybrid search, full-text search (FTS) and vector search demo with 41M+ Wikipedia entries.
Explore the power of combining FTS with vector search for more relevant results.
<Card
title="Learn how it's built"
href="https://lancedb.com/blog/feature-full-text-search/"
>
Read in detail about how the Wikipedia 41M hybrid search demo is built in this blog post.
</Card>
</Card>
## Video Search {#video-search}
<Badge color="green">multimodal</Badge>
<Badge color="orange">video-search</Badge>
<Badge color="purple">vector-search</Badge>
<Card
title="Video Search"
img="/static/assets/images/demos/video_demo_hero.png"
href="https://lancedb-demos.vercel.app/demo/video-search"
>
Search through a library of videos using natural language queries. This demo showcases how to use LanceDB
to perform semantic search on video content.
</Card>
| App | What it shows |
| --- | --- |
| [Semantic.art](/demos/semantic-art) | Multimodal art discovery using feelings, phrases, and images. |
| [Wikipedia 41M hybrid search](/demos/wikipedia-search) | Full-text and vector search combined, over 41M+ entries. |
| [Video search](/demos/video-search) | Natural-language search across a video library. |
docs/web/demos/semantic-art.mdx
+28
View File
@@ -0,0 +1,28 @@
---
title: "Semantic.art"
sidebarTitle: "Semantic.art"
description: "Multimodal art discovery through feelings, phrases, and images."
icon: palette
---
<Badge color="green">multimodal</Badge>
<Badge color="red">hybrid-search</Badge>
<Badge color="purple">vector-search</Badge>
<Badge color="surface-destructive">semantic-routing</Badge>
<Card
title="Semantic.art"
img="/static/assets/images/demos/sda_hero.jpg"
href="https://www.semantic.art/"
>
Semantic.art turns real, human-made art discovery into a multimodal search experience using
feelings, phrases, and images. It's built with LanceDB hybrid search and semantic routing.
<Card
title="Learn how it's built"
href="https://lancedb.com/blog/semanticdotart/"
>
Read in detail about how Semantic.art is built in this blog post.
</Card>
</Card>
docs/web/demos/video-search.mdx
+20
View File
@@ -0,0 +1,20 @@
---
title: "Video search"
sidebarTitle: "Video Search"
description: "Search a video library with natural language queries."
icon: video
---
<Badge color="green">multimodal</Badge>
<Badge color="orange">video-search</Badge>
<Badge color="purple">vector-search</Badge>
<Card
title="Video Search"
img="/static/assets/images/demos/video_demo_hero.png"
href="https://lancedb-demos.vercel.app/demo/video-search"
>
Search through a library of videos using natural language queries. This demo showcases how to use LanceDB
to perform semantic search on video content.
</Card>
docs/web/demos/wikipedia-search.mdx
+28
View File
@@ -0,0 +1,28 @@
---
title: "Wikipedia 41M hybrid search"
sidebarTitle: "Wikipedia Search"
description: "Hybrid full-text and vector search across 41M+ Wikipedia entries."
icon: magnifying-glass
---
<Badge color="green">multimodal</Badge>
<Badge color="red">hybrid-search</Badge>
<Badge color="purple">vector-search</Badge>
<Badge color="gray">fts</Badge>
<Card
title="Wikipedia 41M Hybrid Search"
img="/static/assets/images/demos/wiki_hero.png"
href="https://lancedb-demos.vercel.app/demo/wikipedia-search"
>
Interactive hybrid search, full-text search (FTS) and vector search demo with 41M+ Wikipedia entries.
Explore the power of combining FTS with vector search for more relevant results.
<Card
title="Learn how it's built"
href="https://lancedb.com/blog/feature-full-text-search/"
>
Read in detail about how the Wikipedia 41M hybrid search demo is built in this blog post.
</Card>
</Card>
docs/web/docs.json
+33 -28
View File
@@ -94,6 +94,14 @@
"basics/lance",
"basics/catalog",
"basics/jobs",
{
"group": "Schema & Data Types",
"pages": [
"basics/data-types",
"tables/multimodal"
]
},
"tables/consistency",
"basics/apis",
"basics/agentic-skills"
]
@@ -115,11 +123,11 @@
"pages": [
"tables/index",
"tables/schema",
"tables/multimodal",
"tables/versioning",
"tables/branching"
]
},
"management/materialized-view",
"management/function"
]
},
@@ -130,8 +138,7 @@
"ingestion/insert",
"ingestion/merge-insert",
"ingestion/update",
"ingestion/delete",
"tables/consistency"
"ingestion/delete"
]
},
{
@@ -158,13 +165,6 @@
"reranking/custom-reranker",
"reranking/eval"
]
},
{
"group": "Embedding",
"pages": [
"embedding/index",
"embedding/quickstart"
]
}
]
},
@@ -229,6 +229,13 @@
"group": "Integrations",
"pages": [
"integrations/index",
{
"group": "Embedding Registry",
"pages": [
"embedding/index",
"embedding/quickstart"
]
},
{
"group": "Embedding Providers",
"pages": [
@@ -293,7 +300,7 @@
]
},
{
"tab": "Tutorials",
"tab": "Tutorials and Use Cases",
"groups": [
{
"group": "Tutorials",
@@ -321,23 +328,7 @@
"training/object-detection",
"training/vlm-finetuning"
]
}
]
},
{
"tab": "Demos",
"groups": [
{
"group": "Demos",
"pages": [
"demos/index"
]
}
]
},
{
"tab": "Use Cases",
"groups": [
},
{
"group": "Robotics",
"pages": [
@@ -352,6 +343,20 @@
}
]
},
{
"tab": "Demos",
"groups": [
{
"group": "Demos",
"pages": [
"demos/index",
"demos/semantic-art",
"demos/wikipedia-search",
"demos/video-search"
]
}
]
},
{
"tab": "Agentic Skills",
"groups": [
docs/web/management/index.mdx
+3 -1
View File
@@ -63,7 +63,9 @@ See [Tables](/tables) for creating one and shaping its schema.
one open per table. `SHOW MATERIALIZED VIEWS` asks the server instead.
</Note>
See [Materialized views](/feature-engineering/materialized-views).
See [Materialized views](/management/materialized-view) for creating one, and
[Refresh a materialized view](/feature-engineering/materialized-views) for keeping
it current.
### Views {#views}
docs/web/management/materialized-view.mdx
+95
View File
@@ -0,0 +1,95 @@
---
title: "Materialized view"
sidebarTitle: "Materialized View"
description: "Create, open, list and drop a table whose contents are defined by a query."
icon: table
keywords: ["materialized view", "create_materialized_view", "catalog", "ddl"]
---
A materialized view is a table whose contents are defined by a query over
another table. The query is recorded in the view's schema; a refresh computes
the rows.
The result is an ordinary table -- queryable, indexable, searchable, and present
in `table_names()`. Nothing about reading it is special.
This page covers creating and managing one. Keeping it current is
[Refresh a materialized view](/feature-engineering/materialized-views).
## Creating one {#create}
<CodeGroup>
```python Python icon="python"
view = db.create_materialized_view(
"recent_docs",
source="documents",
select=["id", "title", "embedding"],
where="created_at > '2026-01-01'",
)
```
```typescript TypeScript icon="square-js"
const view = await db.createMaterializedView("recent_docs", {
source: "documents",
select: ["id", "title", "embedding"],
where: "created_at > '2026-01-01'",
});
```
</CodeGroup>
`source` names the table it derives from. `select`, `where` and `limit` record
the query -- there is no separate definition step.
<Warning>
The source table must have **stable row ids**, and they cannot be enabled after
a table exists. A table created without them cannot become a view's source
without being recreated.
Set `new_table_enable_stable_row_ids` in
[storage options](/management/database#storage-options) when you create the
source, not when you create the view.
</Warning>
## Opening, listing and dropping {#manage}
| | Python | TypeScript | SQL |
| --- | --- | --- | --- |
| create | `create_materialized_view` | `createMaterializedView` | -- |
| open | `open_materialized_view` | `openMaterializedView` | -- |
| list | `list_materialized_views` | `listMaterializedViews` | `SHOW MATERIALIZED VIEWS` |
| refresh | `refresh` | `refresh` | `REFRESH MATERIALIZED VIEW` |
| drop | `drop_table` | `dropTable` | `DROP TABLE` |
<Note>
`list_materialized_views` finds views by reading every table's schema, so it
costs one open per table. `SHOW MATERIALIZED VIEWS` asks the server instead and
is cheaper on a catalog with many tables.
</Note>
A view is dropped with `drop_table`, because a materialized view **is** a table.
That also means dropping one removes stored rows -- unlike dropping a
[view](/management/view), which removes only a saved query.
## Working with the view {#working}
`view.table()` returns the underlying table, for querying, indexing, or search.
`view.definition()` returns the recorded query -- select list, predicate, limit
-- which is how you check what a view contains without inferring it from the
rows.
## Against a deployment {#deployment}
<Badge color="red">Enterprise</Badge> `create_materialized_view` works on a
`db://` connection as well as a local one. What changes is that a refresh runs
across the cluster rather than in your process, and SQL can drive it.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/materialized-views">
Incremental refreshes, and what they cost.
</Card>
<Card title="Views" icon="eye" href="/management/view">
The non-materialized counterpart, defined in SQL.
</Card>
</CardGroup>