Files
lancedb/docs/web/quickstart.mdx
T
Jack YeandClaude Sonnet 5 5e38120bdc docs: add storage integration SDK examples, abstract RTE write, simplify cache pages (#4865)
## Summary

- **Define Storage Integration**: un-gates the Python (and Rust, for S3)
`create_storage_integration` examples, drafted against #4618 and held
back until it shipped -- it's now in v0.42.0-beta.0+, and the signature
matches the drafts exactly.
- **Naming rules**: drops the `Branch` row and its "Learn more" link,
consistent with de-emphasizing branches elsewhere in this nav.
- **Distributed RTE Write**: removes the REST path, the
`__lance_mem_wal` index name, and "WAL node" terminology in favor of an
abstract framing -- a query node routes writes to RTE nodes, each of
which can host shards from many different tables. Both diagram variants
are updated to match.
- **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop
Helm-specific and exact-percentage detail (disk-budget math,
`startup_prewarm_block_query_ready`/`healthPort`/failure-policy table)
in favor of a short feature → architecture → why-it-matters shape, each
closing with a contact-sales pointer. Data Page Cache's on-demand
prewarm section is removed outright.

## Note

I wasn't able to get a clean visual render of the updated Distributed
RTE Write diagram in local preview -- Mermaid isn't rendering in
headless Chrome in this environment for *any* page right now, confirmed
against an unmodified reference page showing the identical symptom. The
diagram source follows the exact pattern already shipped and working in
the 21 diagrams from the earlier brand-theming pass, and `mint
validate`/`mint broken-links` both pass clean.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-05 01:40:54 -08:00

397 lines
16 KiB
Plaintext

---
title: Quickstart
sidebarTitle: "Quickstart"
description: "Install LanceDB, create a table, and run your first vector search."
icon: rocket
---
import { PyConnect, TsConnect, RsConnect, PyConnectObjectStorage, TsConnectObjectStorage, RsConnectObjectStorage, PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders, RsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuickstartCurateSample, PyQuickstartCurateDedupeSetup, PyQuickstartCurateDedupe, PyQuickstartCurateDedupeQuery, PyQuickstartFlattenStats, PyQuickstartAddFeature, PyQuickstartAddFeatureEnterprise, PyQuickstartMultimodalBytes, PyQuickstartTraining, TsQuickstartData, TsQuickstartCreateTable, TsQuickstartAddData, TsQuickstartVectorSearch1, TsQuickstartMultimodalBytes, RsQuickstartDefineStruct, RsQuickstartData, RsQuickstartCreateTable, RsQuickstartAddData, RsQuickstartVectorSearch1, RsQuickstartMultimodalBytes } from '/snippets/quickstart.mdx';
This quickstart uses LanceDB Embedded with a local database and works without an account or API key.
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
## 1\. Install LanceDB {#1-install-lancedb}
Choose your language. Python uses Pandas to display the search results. TypeScript needs Node.js 22.18 or later to run the `.mts` example directly.
<CodeGroup>
```bash Python (pip) icon="python"
pip install lancedb pandas
```
```bash Python (uv) icon="python"
uv venv
uv pip install lancedb pandas
```
```bash TypeScript icon="square-js"
npm install @lancedb/lancedb
```
```bash Rust icon="rust"
cargo init --name lancedb_quickstart
cargo add lancedb@0.31.0 --features polars,remote
cargo add tokio --features macros,rt-multi-thread
```
</CodeGroup>
Run the Rust commands in a new, empty directory. The `polars` feature provides DataFrame output; `remote` also lets this example connect to Enterprise.
<Accordion title="Python pre-release builds and older CPUs">
### Python pre-release builds {#python-pre-release-builds}
To pick up the latest features and bug fixes
before the next stable release, install a pre-release from LanceDB's Fury index.
<CodeGroup>
```bash Python (pip) icon="terminal"
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
```
```bash Python (uv) icon="terminal"
uv venv
uv pip install --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
# To add to pyproject.toml, use:
uv add --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
```
</CodeGroup>
Most applications only need `lancedb`. Install the optional `pylance` extra and
include the Lance preview index only when building query engines or connectors,
calling format-level `lance` APIs directly, or converting plain-scan lazy or
legacy blob columns with `to_pandas()`:
<CodeGroup>
```bash Python (pip) icon="terminal"
pip install --pre \
--extra-index-url https://pypi.fury.io/lancedb/ \
--extra-index-url https://pypi.fury.io/lance-format/ \
"lancedb[pylance]"
```
```bash Python (uv) icon="terminal"
uv pip install --prerelease allow \
--index https://pypi.fury.io/lancedb/ \
--index https://pypi.fury.io/lance-format/ \
--index-strategy unsafe-best-match \
"lancedb[pylance]"
```
</CodeGroup>
The LanceDB preview index contains old `pylance` releases; current Lance previews
are published to the `lance-format` index. By default, uv only considers versions
from the first index containing a package, which can select an old `pylance` or
backtrack to an older LanceDB release. `unsafe-best-match` searches all configured
indexes, like pip; use it only with indexes you trust. Install the extra rather
than an unconstrained `pylance` so LanceDB's minimum supported version is enforced.
<Note>
Pre-release builds receive the same level of testing as stable releases, but their availability is not guaranteed
for more than 6 months after release. For real-world workloads, we recommend you use the latest stable release
as far as possible.
</Note>
### Older x86_64 CPUs {#older-x86-cpus}
The default `lancedb` wheel targets `x86-64-haswell` (AVX2 + FMA + F16C). On older x86_64 CPUs without AVX2
(Intel Sandy Bridge, Ivy Bridge, Westmere; AMD Bulldozer, Piledriver, Steamroller), `import lancedb` crashes
with `Illegal instruction`.
Install `lancedb-compat` instead. It exposes the same API (`import lancedb` still works), is built at the
`x86-64-v2` baseline, and uses runtime SIMD dispatch to still use AVX2, FMA, or AVX-512 when available:
<CodeGroup>
```bash Python (pip) icon="terminal"
pip install lancedb-compat
```
</CodeGroup>
The two packages share the same `lancedb/` namespace and conflict at install time, so pick one. To switch,
uninstall the other first (`pip uninstall lancedb && pip install lancedb-compat`).
</Accordion>
## 2\. Connect to a database {#2-connect-to-a-lancedb-database}
### Local directory {#connect-via-local-directory-path}
LanceDB creates an `ex_lancedb` directory to store your tables. No server is needed.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyConnect}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
{TsConnect}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{"use lancedb::connect;\n\n"}
{RsConnect}
</CodeBlock>
</CodeGroup>
For Python's async API, see [Connecting to a database](/basics/catalog).
### Object storage {#connect-via-object-storage-uris}
Use an `s3://`, `gs://`, or `az://` URI in place of the local path. See [Object storage](/basics/catalog#object-storage) for credentials and provider-specific options.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyConnectObjectStorage}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
{TsConnectObjectStorage}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{"use lancedb::connect;\n\n"}
{RsConnectObjectStorage}
</CodeBlock>
</CodeGroup>
### Enterprise {#connect-to-lancedb-enterprise}
Use the database URI, API key, and endpoint provided by your administrator or the LanceDB team. Replace the placeholders below with those details. Use the deployment's region if provided; otherwise keep `us-east-1`.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyConnectEnterprisePlaceholders}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
{TsConnectEnterprisePlaceholders}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{"use lancedb::connect;\n\n"}
{RsConnectEnterprisePlaceholders}
</CodeBlock>
</CodeGroup>
<Note>
In application code, load credentials from environment variables or a secrets manager. See [connection settings](/basics/catalog#enterprise) for an example.
</Note>
Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between local and server execution, see [LanceDB Enterprise](/index#architecture).
## 3\. Create a table and add data {#3-create-a-new-table}
Each row describes a frame from a robot video clip and stores an embedding alongside its video, caption, and quality stats.
<Info>
These embeddings are made up for this example. In your application, use an embedding model to generate them from your data.
</Info>
<Accordion title="Frame records">
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartData}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartData}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartDefineStruct}
{RsQuickstartData}
</CodeBlock>
</CodeGroup>
</Accordion>
Create a `frames` table from those records:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartCreateTable}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartCreateTable}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartCreateTable}
</CodeBlock>
</CodeGroup>
This example uses overwrite mode so you can run it again. It replaces any existing `frames` table in the database you connect to.
Add one more frame:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartAddData}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartAddData}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartAddData}
</CodeBlock>
</CodeGroup>
## 4\. Search the table {#4-semantic-search}
Find the two embeddings closest to the query embedding. This example uses a fixed embedding; it does not call an embedding model.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{"# Search for frames similar to \"a gripper closing on an object\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install pandas` beforehand\nresult = (\n table.search(query_vector)\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_pandas()\n)\nprint(result)\n"}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartVectorSearch1}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartVectorSearch1}
</CodeBlock>
</CodeGroup>
Frames 2 and 4 come back first -- the gripper closing on the cup, then lowering it. Python displays a Pandas DataFrame, Rust displays a Polars DataFrame, and TypeScript displays an array of results. The `_distance` column measures how far each result is from your query embedding, with smaller values indicating a closer match.
## 5\. Curate data with SQL {#5-curate-with-sql}
<Info>
Step 5 needs LanceDB Enterprise: SQL is an Enterprise feature. If you're
using LanceDB Embedded, skip ahead to
[Flatten nested fields](#flatten-nested-fields) below.
</Info>
LanceDB Enterprise accepts SQL over the same connection, for sampling, joins, and
curation tasks that go beyond a single table scan. Draw a quick sample to look
at the data:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartCurateSample}
</CodeBlock>
</CodeGroup>
See [SQL](/basics/sql) for the Flight SQL client and CLI alternatives to `execute_query()`.
Now add a near-duplicate frame -- frame 5 is one video frame after frame 2, with
an almost identical embedding -- and build the vector index that `dedupe` reads:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartCurateDedupeSetup}
</CodeBlock>
</CodeGroup>
Collapse near-duplicates into a materialized view that stays current as new frames arrive, then read it back:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartCurateDedupe}
</CodeBlock>
</CodeGroup>
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartCurateDedupeQuery}
</CodeBlock>
</CodeGroup>
Frames 2 and 5 collapse into one representative -- five rows become four.
Which of the two ids survives is not guaranteed to stay the same across
appends, so don't treat it as a stable identifier for the cluster. See
[Deduplicate near-duplicate records](/curation/dedupe) for `duplicate_pairs`,
resolution strategies, and tuning `distance_threshold`, and
[Statistical Sampling](/curation/statistical-sampling) for percentage samples,
repeatable draws, and sampling across a join.
## 6\. Add a derived feature {#6-add-a-derived-feature}
### Flatten nested fields {#flatten-nested-fields}
A UDF binds to top-level columns, so flatten the fields inside `stats` first.
This is a plain eager `add_columns` call, so it works the same on LanceDB
Embedded and LanceDB Enterprise -- [step 8](#8-train) needs these columns
either way:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartFlattenStats}
</CodeBlock>
</CodeGroup>
<Info>
The rest of this step needs LanceDB Enterprise -- Functions are an
Enterprise feature. If you're using LanceDB Embedded, skip ahead to
[step 7](#7-store-multimodal-data).
</Info>
Register a UDF as a Function, then call it to fill a column computed from
those flattened fields. This fills every existing row once, immediately:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartAddFeature}
</CodeBlock>
</CodeGroup>
Read it back like any other column:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{"features = table.search().select([\"frame_id\", \"video_id\", \"quality_score\"]).limit(10).to_pandas()\nprint(features)\n"}
</CodeBlock>
</CodeGroup>
`refresh_column` fills only rows added since the last refresh and recomputes rows whose inputs changed, leaving unchanged rows alone. Run the same refresh as a background job instead, so it doesn't block:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartAddFeatureEnterprise}
</CodeBlock>
</CodeGroup>
See [Create a Function](/curation/create-function) for UDFs that call external
models or services, and [Computed columns](/feature-engineering/add-computed-column)
for the SQL-expression form that doesn't need a registered Function.
## 7\. Store multimodal data {#7-store-multimodal-data}
Store images, audio, or other binary data alongside vectors and metadata, in a column of raw bytes. This creates a new table from one image file -- replace the path with one of your own:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartMultimodalBytes}
</CodeBlock>
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
{TsQuickstartMultimodalBytes}
</CodeBlock>
<CodeBlock filename="Rust" language="Rust" icon="rust">
{RsQuickstartMultimodalBytes}
</CodeBlock>
</CodeGroup>
See [Multimodal data types](/management/table#multimodal) for larger payloads and the Blob v2 column type, which avoids loading binary data on every scan.
## 8\. Train on your data {#8-train}
Read table data into PyTorch, projecting and shuffling only the columns a training step needs:
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{PyQuickstartTraining}
</CodeBlock>
</CodeGroup>
`StreamingDataset` wraps a table for PyTorch training reads; `columns` projects before reading, so a training step touches only the columns it selects, not unused ones such as `caption` or `embedding`. It works the same way against LanceDB Embedded and LanceDB Enterprise. See [PyTorch Data Loader](/training/data-loading) for shuffling, transforms, resuming, and multi-worker loading, and [Permutation](/training/permutation) for the lower-level row-ordering primitive it's built on.
## Where to go next {#next-steps}
<Columns cols={2}>
<Card title="Basic table operations" icon="table" href="/management/table">
Create, update, and manage your tables.
</Card>
<Card title="Build a RAG app" icon="search" href="/tutorials/agents/contextual-rag">
Use retrieved context in a language model application.
</Card>
<Card title="Indexing" icon="layer-group" href="/indexing/">
Speed up queries on larger datasets.
</Card>
<Card title="Embedding models" icon="vector-square" href="/embedding/">
Generate vectors from your own data.
</Card>
</Columns>