diff --git a/docs/web/basics/data-types.mdx b/docs/web/basics/data-types.mdx new file mode 100644 index 000000000..0b0d35faa --- /dev/null +++ b/docs/web/basics/data-types.mdx @@ -0,0 +1,143 @@ +--- +title: "Schema and data types" +sidebarTitle: "Data Types" +description: "The Arrow type system LanceDB stores, and the extension types Lance adds on top of it." +icon: shapes +keywords: ["schema", "data types", "arrow", "json", "blob", "bfloat16", "vector", "geospatial"] +--- + +A table's schema is an **Arrow schema**. There is no separate LanceDB type +system mapped onto storage -- a column is the same object to the query engine, +to the format, and to any other engine reading the files. + +That identity is mostly invisible and occasionally load-bearing, so it is worth +knowing what it admits. + +## Supported types {#supported} + +| Category | Types | +| --- | --- | +| Null | `null` | +| Boolean | `bool` | +| Integer | `int8`, `int16`, `int32`, `int64` and their `uint` counterparts | +| Floating point | `float16`, `float32`, `float64` | +| String | `utf8`, `large_utf8` | +| Binary | `binary`, `large_binary`, `fixed_size_binary(n)` | +| Date and time | `date32`, `date64`, `time32`, `time64`, `timestamp` (with or without timezone), `duration` | +| Nested | `struct`, `list`, `large_list`, `fixed_size_list`, `map` | +| Dictionary | `dictionary(key, value)` | + +Two limits are worth stating because they surface as schema errors rather than +as documentation: + +- A `map` with `keys_sorted = true` is rejected. Sorted-key maps do not survive + the round trip through the Python bindings, so they are refused rather than + silently unsorted. +- Not every type is legal as a `fixed_size_list` child. The vector case is, + which is what matters in practice. + +## Vectors are fixed-size lists {#vectors} + +A vector column is a `fixed_size_list` -- an ordinary Arrow type, +not a special one: + +```python +import pyarrow as pa + +schema = pa.schema([ + pa.field("id", pa.int64()), + pa.field("vector", pa.list_(pa.float32(), 1536)), +]) +``` + +Because it is ordinary, it is selectable, testable for nullness, and joinable +like any other column. The consequence people trip on is the other side of that: +arithmetic over it is **list** arithmetic, not scalar arithmetic. When the +operation you want is a similarity comparison, use +[vector search](/search/vector-search) rather than SQL. + +Without an explicit schema, Python infers a list of floats as a fixed-size +`float32` vector from the dimension it observes in the data. + +## Extension types {#extensions} + +Lance adds a few types as **Arrow extension types** -- an underlying storage +type plus an `ARROW:extension:name` marker. Any Arrow reader can read the +storage; readers that know the extension get the richer behaviour. + +| Type | Extension name | Stored as | +| --- | --- | --- | +| [JSON](#json) | `lance.json` | `large_binary`, JSONB-encoded | +| [Blob](#blob) | `lance.blob.v2` | `large_binary`, stored out of line | +| bfloat16 | `lance.bfloat16` | `fixed_size_binary(2)` | + +`arrow.json` is accepted as well, for interoperability with writers that use +Arrow's own canonical JSON extension name. + +### JSON {#json} + +A JSON column stores documents in **JSONB**, a binary encoding, rather than as +text. Accessors read into a document without parsing the whole thing: + +```sql +SELECT json_get_string(data, 'name') FROM events +WHERE json_get_int(data, 'status') = 200 +``` + +`json_get` returns JSON, so accessors nest for a nested document. **A JSON +scalar index accelerates `json_get`, its typed variants, and `json_extract`** -- +the rest evaluate per row. The full function list is in +[SQL functions](/sql/functions#json). + +This is the type to reach for when a column's shape varies per row. When it does +not, ordinary `struct` columns are faster and typed. + +### Blob {#blob} + +A blob column holds large binary objects -- images, video, audio, model +artifacts -- **out of line** from the rest of the row. That is what keeps a scan +over the other columns fast on a table whose rows are megabytes each. + +A blob can be read two ways, and the choice changes the column's Arrow type: + +- as its **binary contents**, or +- as a **description** -- a pointer plus metadata, without fetching the bytes. + + +The default reads blob columns as descriptions and other binary columns as +binary. A query that selects a blob column expecting bytes gets a struct +instead, unless the read is configured otherwise. Worth knowing before writing a +predicate against one. + + +See [Working with multimodal data](/tables/multimodal). + +### bfloat16 {#bfloat16} + +Sixteen-bit floats in the bfloat16 layout, stored as `fixed_size_binary(2)`. +Useful as a vector element type when halving the memory matters more than the +precision does. + +## Geospatial {#geospatial} + + +Lance defines geospatial types and `st_*` functions on +[GeoArrow](https://geoarrow.org), but they sit behind an optional build feature +that **LanceDB does not ship**. Calling `st_intersects` and friends returns an +error naming the missing feature rather than an unknown-function error, and the +spatial index is gated the same way. + + +Geometry can still be stored -- it is binary, and binary columns work -- but +nothing interprets it. Filter on a bounding box held in ordinary numeric +columns if you need spatial predicates today. + +## Changing types later {#evolution} + +A schema is not fixed at creation. Columns can be added, renamed, retyped, made +nullable, and dropped, usually without rewriting existing data -- see +[Schema evolution](/tables/schema). Deriving new values from existing columns is +[data evolution](/feature-engineering/data-evolution). + +Nullability is worth deciding up front: if later writes omit a non-nullable +column, or write actual nulls into it, ingestion fails. diff --git a/docs/web/basics/jobs.mdx b/docs/web/basics/jobs.mdx index db4be33d0..c774d6e28 100644 --- a/docs/web/basics/jobs.mdx +++ b/docs/web/basics/jobs.mdx @@ -12,11 +12,34 @@ common source of confusion about how work gets done. | | What it is | What it changes | | --- | --- | --- | | **Async client** | `connect_async`, `await` | whether *your thread* blocks | -| **Async execution** | `_async` methods returning a [job](#jobs) | whether *the work* outlives the call | +| **Async execution** | `_async` methods returning a handle | whether *the work* outlives the call | -They are independent. A synchronous Python call can start a job that runs for an +They are independent. A synchronous Python call can start work that runs for an hour; an `await` can wait for something that finishes instantly. +## Every `_async` method returns a handle {#handles} + +An `_async` method submits work and hands back a **handle** to it. Which handle +depends on what you submitted -- `execute_query_async` returns a query, +everything else returns a [job](#jobs) -- but they are the same idea, and they +answer the same four questions: + +| | Query handle | Job handle | +| --- | --- | --- | +| Which one is this? | `id` | `id` | +| How is it going? | `describe()` | `status()`, `refresh()`, `events()` | +| Give me the result | `reader()` — streams batches as they arrive | `wait()` — blocks for the terminal result | +| Stop it | `cancel()` | `cancel()` | + +The difference is in the third row, and it follows from what the work produces. +A query produces **rows**, so its handle streams them and you can start reading +before the query finishes. A job produces **an outcome** -- an index built, a +column filled -- so its handle has nothing to stream and gives you the result +once, at the end. + +Everything else is shared: both identify the work on the server, both report on +it without blocking, and both can be told to stop. + ## Async client: whether you block {#async-client} A concern of your language, not of LanceDB. The work is identical either way. @@ -61,11 +84,22 @@ SELECT id, text FROM my_docs LIMIT 1000 `execute_query` returns a blocking Arrow reader **once results start arriving** -- it does not wait for the whole query to finish. Results stream as record -batches, so a large result set does not have to fit in memory before you see -its first row. +batches, so a large result set does not have to fit in memory before you see its +first row. -`execute_query_async` returns the query handle instead, for a statement you -would rather not sit on at all. `describe_query` reports on one by id. +`execute_query_async` returns the query handle instead: + +```python +query = db.execute_query_async("SELECT id, text FROM my_docs") + +query.id # identifies it on the server +query.describe() # what it is doing, without consuming results +reader = query.reader() # stream the rows when you want them +query.cancel() # stop it +``` + +`describe_query(id)` reports on one from elsewhere, the way `open_job(id)` does +for a job. SQL requires a deployment. See [SDKs and SQL](/basics/apis). diff --git a/docs/web/demos/index.mdx b/docs/web/demos/index.mdx index 95f1ab365..a6a3e284a 100644 --- a/docs/web/demos/index.mdx +++ b/docs/web/demos/index.mdx @@ -1,78 +1,15 @@ --- -title: "Demo Application Gallery" -sidebarTitle: "App Gallery" -description: "Demo apps showcasing end-to-end applications built with LanceDB for production use cases." +title: "Demo application gallery" +sidebarTitle: "Overview" +description: "End-to-end applications built with LanceDB." +icon: grid-2 --- -Explore the demo applications built with LanceDB below. +Applications built on LanceDB, each with a live demo and a write-up of how it +works. -App | Description ---- | --- -[Semantic.Art](#semantic-art) | A multimodal art discovery platform using feelings, phrases, and images. -[Wikipedia 41M Hybrid Search](#wikipedia-41m-hybrid-search) | An interactive hybrid search demo combining full-text search and vector search. -[Video Search](#video-search) | A video search application that allows searching through a library of videos using natural language queries. - - -## Semantic.art {#semantic-art} - -multimodal -hybrid-search -vector-search -semantic-routing - - -Semantic.art turns real, human-made art discovery into a multimodal search experience using -feelings, phrases, and images. It's built with LanceDB hybrid search and semantic routing. - - -Read in detail about how Semantic.art is built in this blog post. - - - - -## Wikipedia 41M Hybrid Search {#wikipedia-41m-hybrid-search} - -multimodal -hybrid-search -vector-search -fts - - -Interactive hybrid search, full-text search (FTS) and vector search demo with 41M+ Wikipedia entries. -Explore the power of combining FTS with vector search for more relevant results. - - -Read in detail about how the Wikipedia 41M hybrid search demo is built in this blog post. - - - - -## Video Search {#video-search} - -multimodal -video-search -vector-search - - -Search through a library of videos using natural language queries. This demo showcases how to use LanceDB -to perform semantic search on video content. - - \ No newline at end of file +| App | What it shows | +| --- | --- | +| [Semantic.art](/demos/semantic-art) | Multimodal art discovery using feelings, phrases, and images. | +| [Wikipedia 41M hybrid search](/demos/wikipedia-search) | Full-text and vector search combined, over 41M+ entries. | +| [Video search](/demos/video-search) | Natural-language search across a video library. | diff --git a/docs/web/demos/semantic-art.mdx b/docs/web/demos/semantic-art.mdx new file mode 100644 index 000000000..a7ed3859f --- /dev/null +++ b/docs/web/demos/semantic-art.mdx @@ -0,0 +1,28 @@ +--- +title: "Semantic.art" +sidebarTitle: "Semantic.art" +description: "Multimodal art discovery through feelings, phrases, and images." +icon: palette +--- + +multimodal +hybrid-search +vector-search +semantic-routing + + +Semantic.art turns real, human-made art discovery into a multimodal search experience using +feelings, phrases, and images. It's built with LanceDB hybrid search and semantic routing. + + +Read in detail about how Semantic.art is built in this blog post. + + + diff --git a/docs/web/demos/video-search.mdx b/docs/web/demos/video-search.mdx new file mode 100644 index 000000000..4a8ec3d4c --- /dev/null +++ b/docs/web/demos/video-search.mdx @@ -0,0 +1,20 @@ +--- +title: "Video search" +sidebarTitle: "Video Search" +description: "Search a video library with natural language queries." +icon: video +--- + +multimodal +video-search +vector-search + + +Search through a library of videos using natural language queries. This demo showcases how to use LanceDB +to perform semantic search on video content. + + diff --git a/docs/web/demos/wikipedia-search.mdx b/docs/web/demos/wikipedia-search.mdx new file mode 100644 index 000000000..73dac4998 --- /dev/null +++ b/docs/web/demos/wikipedia-search.mdx @@ -0,0 +1,28 @@ +--- +title: "Wikipedia 41M hybrid search" +sidebarTitle: "Wikipedia Search" +description: "Hybrid full-text and vector search across 41M+ Wikipedia entries." +icon: magnifying-glass +--- + +multimodal +hybrid-search +vector-search +fts + + +Interactive hybrid search, full-text search (FTS) and vector search demo with 41M+ Wikipedia entries. +Explore the power of combining FTS with vector search for more relevant results. + + +Read in detail about how the Wikipedia 41M hybrid search demo is built in this blog post. + + + diff --git a/docs/web/docs.json b/docs/web/docs.json index b5152fe24..036c022d1 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -94,6 +94,14 @@ "basics/lance", "basics/catalog", "basics/jobs", + { + "group": "Schema & Data Types", + "pages": [ + "basics/data-types", + "tables/multimodal" + ] + }, + "tables/consistency", "basics/apis", "basics/agentic-skills" ] @@ -115,11 +123,11 @@ "pages": [ "tables/index", "tables/schema", - "tables/multimodal", "tables/versioning", "tables/branching" ] }, + "management/materialized-view", "management/function" ] }, @@ -130,8 +138,7 @@ "ingestion/insert", "ingestion/merge-insert", "ingestion/update", - "ingestion/delete", - "tables/consistency" + "ingestion/delete" ] }, { @@ -158,13 +165,6 @@ "reranking/custom-reranker", "reranking/eval" ] - }, - { - "group": "Embedding", - "pages": [ - "embedding/index", - "embedding/quickstart" - ] } ] }, @@ -229,6 +229,13 @@ "group": "Integrations", "pages": [ "integrations/index", + { + "group": "Embedding Registry", + "pages": [ + "embedding/index", + "embedding/quickstart" + ] + }, { "group": "Embedding Providers", "pages": [ @@ -293,7 +300,7 @@ ] }, { - "tab": "Tutorials", + "tab": "Tutorials and Use Cases", "groups": [ { "group": "Tutorials", @@ -321,23 +328,7 @@ "training/object-detection", "training/vlm-finetuning" ] - } - ] - }, - { - "tab": "Demos", - "groups": [ - { - "group": "Demos", - "pages": [ - "demos/index" - ] - } - ] - }, - { - "tab": "Use Cases", - "groups": [ + }, { "group": "Robotics", "pages": [ @@ -352,6 +343,20 @@ } ] }, + { + "tab": "Demos", + "groups": [ + { + "group": "Demos", + "pages": [ + "demos/index", + "demos/semantic-art", + "demos/wikipedia-search", + "demos/video-search" + ] + } + ] + }, { "tab": "Agentic Skills", "groups": [ diff --git a/docs/web/management/index.mdx b/docs/web/management/index.mdx index 38ef22367..8a78ec2fd 100644 --- a/docs/web/management/index.mdx +++ b/docs/web/management/index.mdx @@ -63,7 +63,9 @@ See [Tables](/tables) for creating one and shaping its schema. one open per table. `SHOW MATERIALIZED VIEWS` asks the server instead. -See [Materialized views](/feature-engineering/materialized-views). +See [Materialized views](/management/materialized-view) for creating one, and +[Refresh a materialized view](/feature-engineering/materialized-views) for keeping +it current. ### Views {#views} diff --git a/docs/web/management/materialized-view.mdx b/docs/web/management/materialized-view.mdx new file mode 100644 index 000000000..4c61070fe --- /dev/null +++ b/docs/web/management/materialized-view.mdx @@ -0,0 +1,95 @@ +--- +title: "Materialized view" +sidebarTitle: "Materialized View" +description: "Create, open, list and drop a table whose contents are defined by a query." +icon: table +keywords: ["materialized view", "create_materialized_view", "catalog", "ddl"] +--- + +A materialized view is a table whose contents are defined by a query over +another table. The query is recorded in the view's schema; a refresh computes +the rows. + +The result is an ordinary table -- queryable, indexable, searchable, and present +in `table_names()`. Nothing about reading it is special. + +This page covers creating and managing one. Keeping it current is +[Refresh a materialized view](/feature-engineering/materialized-views). + +## Creating one {#create} + + +```python Python icon="python" +view = db.create_materialized_view( + "recent_docs", + source="documents", + select=["id", "title", "embedding"], + where="created_at > '2026-01-01'", +) +``` + +```typescript TypeScript icon="square-js" +const view = await db.createMaterializedView("recent_docs", { + source: "documents", + select: ["id", "title", "embedding"], + where: "created_at > '2026-01-01'", +}); +``` + + +`source` names the table it derives from. `select`, `where` and `limit` record +the query -- there is no separate definition step. + + +The source table must have **stable row ids**, and they cannot be enabled after +a table exists. A table created without them cannot become a view's source +without being recreated. + +Set `new_table_enable_stable_row_ids` in +[storage options](/management/database#storage-options) when you create the +source, not when you create the view. + + +## Opening, listing and dropping {#manage} + +| | Python | TypeScript | SQL | +| --- | --- | --- | --- | +| create | `create_materialized_view` | `createMaterializedView` | -- | +| open | `open_materialized_view` | `openMaterializedView` | -- | +| list | `list_materialized_views` | `listMaterializedViews` | `SHOW MATERIALIZED VIEWS` | +| refresh | `refresh` | `refresh` | `REFRESH MATERIALIZED VIEW` | +| drop | `drop_table` | `dropTable` | `DROP TABLE` | + + +`list_materialized_views` finds views by reading every table's schema, so it +costs one open per table. `SHOW MATERIALIZED VIEWS` asks the server instead and +is cheaper on a catalog with many tables. + + +A view is dropped with `drop_table`, because a materialized view **is** a table. +That also means dropping one removes stored rows -- unlike dropping a +[view](/management/view), which removes only a saved query. + +## Working with the view {#working} + +`view.table()` returns the underlying table, for querying, indexing, or search. +`view.definition()` returns the recorded query -- select list, predicate, limit +-- which is how you check what a view contains without inferring it from the +rows. + +## Against a deployment {#deployment} + +Enterprise `create_materialized_view` works on a +`db://` connection as well as a local one. What changes is that a refresh runs +across the cluster rather than in your process, and SQL can drive it. + +## What to read next {#next} + + + + Incremental refreshes, and what they cost. + + + The non-materialized counterpart, defined in SQL. + +