mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 22:26:53 +00:00
## Summary - **Define Storage Integration**: un-gates the Python (and Rust, for S3) `create_storage_integration` examples, drafted against #4618 and held back until it shipped -- it's now in v0.42.0-beta.0+, and the signature matches the drafts exactly. - **Naming rules**: drops the `Branch` row and its "Learn more" link, consistent with de-emphasizing branches elsewhere in this nav. - **Distributed RTE Write**: removes the REST path, the `__lance_mem_wal` index name, and "WAL node" terminology in favor of an abstract framing -- a query node routes writes to RTE nodes, each of which can host shards from many different tables. Both diagram variants are updated to match. - **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop Helm-specific and exact-percentage detail (disk-budget math, `startup_prewarm_block_query_ready`/`healthPort`/failure-policy table) in favor of a short feature → architecture → why-it-matters shape, each closing with a contact-sales pointer. Data Page Cache's on-demand prewarm section is removed outright. ## Note I wasn't able to get a clean visual render of the updated Distributed RTE Write diagram in local preview -- Mermaid isn't rendering in headless Chrome in this environment for *any* page right now, confirmed against an unmodified reference page showing the identical symptom. The diagram source follows the exact pattern already shipped and working in the 21 diagrams from the earlier brand-theming pass, and `mint validate`/`mint broken-links` both pass clean. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
397 lines
16 KiB
Plaintext
397 lines
16 KiB
Plaintext
---
|
|
title: Quickstart
|
|
sidebarTitle: "Quickstart"
|
|
description: "Install LanceDB, create a table, and run your first vector search."
|
|
icon: rocket
|
|
---
|
|
import { PyConnect, TsConnect, RsConnect, PyConnectObjectStorage, TsConnectObjectStorage, RsConnectObjectStorage, PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders, RsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
|
|
import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuickstartCurateSample, PyQuickstartCurateDedupeSetup, PyQuickstartCurateDedupe, PyQuickstartCurateDedupeQuery, PyQuickstartFlattenStats, PyQuickstartAddFeature, PyQuickstartAddFeatureEnterprise, PyQuickstartMultimodalBytes, PyQuickstartTraining, TsQuickstartData, TsQuickstartCreateTable, TsQuickstartAddData, TsQuickstartVectorSearch1, TsQuickstartMultimodalBytes, RsQuickstartDefineStruct, RsQuickstartData, RsQuickstartCreateTable, RsQuickstartAddData, RsQuickstartVectorSearch1, RsQuickstartMultimodalBytes } from '/snippets/quickstart.mdx';
|
|
|
|
This quickstart uses LanceDB Embedded with a local database and works without an account or API key.
|
|
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
|
|
|
|
## 1\. Install LanceDB {#1-install-lancedb}
|
|
|
|
Choose your language. Python uses Pandas to display the search results. TypeScript needs Node.js 22.18 or later to run the `.mts` example directly.
|
|
|
|
<CodeGroup>
|
|
```bash Python (pip) icon="python"
|
|
pip install lancedb pandas
|
|
```
|
|
|
|
```bash Python (uv) icon="python"
|
|
uv venv
|
|
uv pip install lancedb pandas
|
|
```
|
|
|
|
```bash TypeScript icon="square-js"
|
|
npm install @lancedb/lancedb
|
|
```
|
|
|
|
```bash Rust icon="rust"
|
|
cargo init --name lancedb_quickstart
|
|
cargo add lancedb@0.31.0 --features polars,remote
|
|
cargo add tokio --features macros,rt-multi-thread
|
|
```
|
|
</CodeGroup>
|
|
|
|
Run the Rust commands in a new, empty directory. The `polars` feature provides DataFrame output; `remote` also lets this example connect to Enterprise.
|
|
|
|
<Accordion title="Python pre-release builds and older CPUs">
|
|
### Python pre-release builds {#python-pre-release-builds}
|
|
|
|
To pick up the latest features and bug fixes
|
|
before the next stable release, install a pre-release from LanceDB's Fury index.
|
|
|
|
<CodeGroup>
|
|
```bash Python (pip) icon="terminal"
|
|
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
|
|
```
|
|
|
|
```bash Python (uv) icon="terminal"
|
|
uv venv
|
|
uv pip install --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
|
|
|
|
# To add to pyproject.toml, use:
|
|
uv add --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
|
|
```
|
|
</CodeGroup>
|
|
|
|
Most applications only need `lancedb`. Install the optional `pylance` extra and
|
|
include the Lance preview index only when building query engines or connectors,
|
|
calling format-level `lance` APIs directly, or converting plain-scan lazy or
|
|
legacy blob columns with `to_pandas()`:
|
|
|
|
<CodeGroup>
|
|
```bash Python (pip) icon="terminal"
|
|
pip install --pre \
|
|
--extra-index-url https://pypi.fury.io/lancedb/ \
|
|
--extra-index-url https://pypi.fury.io/lance-format/ \
|
|
"lancedb[pylance]"
|
|
```
|
|
|
|
```bash Python (uv) icon="terminal"
|
|
uv pip install --prerelease allow \
|
|
--index https://pypi.fury.io/lancedb/ \
|
|
--index https://pypi.fury.io/lance-format/ \
|
|
--index-strategy unsafe-best-match \
|
|
"lancedb[pylance]"
|
|
```
|
|
</CodeGroup>
|
|
|
|
The LanceDB preview index contains old `pylance` releases; current Lance previews
|
|
are published to the `lance-format` index. By default, uv only considers versions
|
|
from the first index containing a package, which can select an old `pylance` or
|
|
backtrack to an older LanceDB release. `unsafe-best-match` searches all configured
|
|
indexes, like pip; use it only with indexes you trust. Install the extra rather
|
|
than an unconstrained `pylance` so LanceDB's minimum supported version is enforced.
|
|
|
|
<Note>
|
|
Pre-release builds receive the same level of testing as stable releases, but their availability is not guaranteed
|
|
for more than 6 months after release. For real-world workloads, we recommend you use the latest stable release
|
|
as far as possible.
|
|
</Note>
|
|
|
|
### Older x86_64 CPUs {#older-x86-cpus}
|
|
The default `lancedb` wheel targets `x86-64-haswell` (AVX2 + FMA + F16C). On older x86_64 CPUs without AVX2
|
|
(Intel Sandy Bridge, Ivy Bridge, Westmere; AMD Bulldozer, Piledriver, Steamroller), `import lancedb` crashes
|
|
with `Illegal instruction`.
|
|
|
|
Install `lancedb-compat` instead. It exposes the same API (`import lancedb` still works), is built at the
|
|
`x86-64-v2` baseline, and uses runtime SIMD dispatch to still use AVX2, FMA, or AVX-512 when available:
|
|
|
|
<CodeGroup>
|
|
```bash Python (pip) icon="terminal"
|
|
pip install lancedb-compat
|
|
```
|
|
</CodeGroup>
|
|
|
|
The two packages share the same `lancedb/` namespace and conflict at install time, so pick one. To switch,
|
|
uninstall the other first (`pip uninstall lancedb && pip install lancedb-compat`).
|
|
|
|
|
|
</Accordion>
|
|
|
|
## 2\. Connect to a database {#2-connect-to-a-lancedb-database}
|
|
|
|
### Local directory {#connect-via-local-directory-path}
|
|
|
|
LanceDB creates an `ex_lancedb` directory to store your tables. No server is needed.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyConnect}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
|
|
{TsConnect}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{"use lancedb::connect;\n\n"}
|
|
{RsConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
For Python's async API, see [Connecting to a database](/basics/catalog).
|
|
|
|
### Object storage {#connect-via-object-storage-uris}
|
|
|
|
Use an `s3://`, `gs://`, or `az://` URI in place of the local path. See [Object storage](/basics/catalog#object-storage) for credentials and provider-specific options.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyConnectObjectStorage}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
|
|
{TsConnectObjectStorage}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{"use lancedb::connect;\n\n"}
|
|
{RsConnectObjectStorage}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### Enterprise {#connect-to-lancedb-enterprise}
|
|
|
|
Use the database URI, API key, and endpoint provided by your administrator or the LanceDB team. Replace the placeholders below with those details. Use the deployment's region if provided; otherwise keep `us-east-1`.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyConnectEnterprisePlaceholders}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
|
|
{TsConnectEnterprisePlaceholders}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{"use lancedb::connect;\n\n"}
|
|
{RsConnectEnterprisePlaceholders}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<Note>
|
|
In application code, load credentials from environment variables or a secrets manager. See [connection settings](/basics/catalog#enterprise) for an example.
|
|
</Note>
|
|
|
|
Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between local and server execution, see [LanceDB Enterprise](/index#architecture).
|
|
|
|
## 3\. Create a table and add data {#3-create-a-new-table}
|
|
|
|
Each row describes a frame from a robot video clip and stores an embedding alongside its video, caption, and quality stats.
|
|
|
|
<Info>
|
|
These embeddings are made up for this example. In your application, use an embedding model to generate them from your data.
|
|
</Info>
|
|
|
|
<Accordion title="Frame records">
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartData}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsQuickstartData}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsQuickstartDefineStruct}
|
|
{RsQuickstartData}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
</Accordion>
|
|
|
|
Create a `frames` table from those records:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartCreateTable}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsQuickstartCreateTable}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsQuickstartCreateTable}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
This example uses overwrite mode so you can run it again. It replaces any existing `frames` table in the database you connect to.
|
|
|
|
Add one more frame:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartAddData}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsQuickstartAddData}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsQuickstartAddData}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## 4\. Search the table {#4-semantic-search}
|
|
|
|
Find the two embeddings closest to the query embedding. This example uses a fixed embedding; it does not call an embedding model.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{"# Search for frames similar to \"a gripper closing on an object\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install pandas` beforehand\nresult = (\n table.search(query_vector)\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_pandas()\n)\nprint(result)\n"}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsQuickstartVectorSearch1}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsQuickstartVectorSearch1}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Frames 2 and 4 come back first -- the gripper closing on the cup, then lowering it. Python displays a Pandas DataFrame, Rust displays a Polars DataFrame, and TypeScript displays an array of results. The `_distance` column measures how far each result is from your query embedding, with smaller values indicating a closer match.
|
|
|
|
## 5\. Curate data with SQL {#5-curate-with-sql}
|
|
|
|
<Info>
|
|
Step 5 needs LanceDB Enterprise: SQL is an Enterprise feature. If you're
|
|
using LanceDB Embedded, skip ahead to
|
|
[Flatten nested fields](#flatten-nested-fields) below.
|
|
</Info>
|
|
|
|
LanceDB Enterprise accepts SQL over the same connection, for sampling, joins, and
|
|
curation tasks that go beyond a single table scan. Draw a quick sample to look
|
|
at the data:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartCurateSample}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
See [SQL](/basics/sql) for the Flight SQL client and CLI alternatives to `execute_query()`.
|
|
|
|
Now add a near-duplicate frame -- frame 5 is one video frame after frame 2, with
|
|
an almost identical embedding -- and build the vector index that `dedupe` reads:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartCurateDedupeSetup}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Collapse near-duplicates into a materialized view that stays current as new frames arrive, then read it back:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartCurateDedupe}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartCurateDedupeQuery}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Frames 2 and 5 collapse into one representative -- five rows become four.
|
|
Which of the two ids survives is not guaranteed to stay the same across
|
|
appends, so don't treat it as a stable identifier for the cluster. See
|
|
[Deduplicate near-duplicate records](/curation/dedupe) for `duplicate_pairs`,
|
|
resolution strategies, and tuning `distance_threshold`, and
|
|
[Statistical Sampling](/curation/statistical-sampling) for percentage samples,
|
|
repeatable draws, and sampling across a join.
|
|
|
|
## 6\. Add a derived feature {#6-add-a-derived-feature}
|
|
|
|
### Flatten nested fields {#flatten-nested-fields}
|
|
|
|
A UDF binds to top-level columns, so flatten the fields inside `stats` first.
|
|
This is a plain eager `add_columns` call, so it works the same on LanceDB
|
|
Embedded and LanceDB Enterprise -- [step 8](#8-train) needs these columns
|
|
either way:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartFlattenStats}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<Info>
|
|
The rest of this step needs LanceDB Enterprise -- Functions are an
|
|
Enterprise feature. If you're using LanceDB Embedded, skip ahead to
|
|
[step 7](#7-store-multimodal-data).
|
|
</Info>
|
|
|
|
Register a UDF as a Function, then call it to fill a column computed from
|
|
those flattened fields. This fills every existing row once, immediately:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartAddFeature}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Read it back like any other column:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{"features = table.search().select([\"frame_id\", \"video_id\", \"quality_score\"]).limit(10).to_pandas()\nprint(features)\n"}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
`refresh_column` fills only rows added since the last refresh and recomputes rows whose inputs changed, leaving unchanged rows alone. Run the same refresh as a background job instead, so it doesn't block:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartAddFeatureEnterprise}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
See [Create a Function](/curation/create-function) for UDFs that call external
|
|
models or services, and [Computed columns](/feature-engineering/add-computed-column)
|
|
for the SQL-expression form that doesn't need a registered Function.
|
|
|
|
## 7\. Store multimodal data {#7-store-multimodal-data}
|
|
|
|
Store images, audio, or other binary data alongside vectors and metadata, in a column of raw bytes. This creates a new table from one image file -- replace the path with one of your own:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartMultimodalBytes}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsQuickstartMultimodalBytes}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsQuickstartMultimodalBytes}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
See [Multimodal data types](/management/table#multimodal) for larger payloads and the Blob v2 column type, which avoids loading binary data on every scan.
|
|
|
|
## 8\. Train on your data {#8-train}
|
|
|
|
Read table data into PyTorch, projecting and shuffling only the columns a training step needs:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartTraining}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
`StreamingDataset` wraps a table for PyTorch training reads; `columns` projects before reading, so a training step touches only the columns it selects, not unused ones such as `caption` or `embedding`. It works the same way against LanceDB Embedded and LanceDB Enterprise. See [PyTorch Data Loader](/training/data-loading) for shuffling, transforms, resuming, and multi-worker loading, and [Permutation](/training/permutation) for the lower-level row-ordering primitive it's built on.
|
|
|
|
## Where to go next {#next-steps}
|
|
|
|
<Columns cols={2}>
|
|
<Card title="Basic table operations" icon="table" href="/management/table">
|
|
Create, update, and manage your tables.
|
|
</Card>
|
|
<Card title="Build a RAG app" icon="search" href="/tutorials/agents/contextual-rag">
|
|
Use retrieved context in a language model application.
|
|
</Card>
|
|
<Card title="Indexing" icon="layer-group" href="/indexing/">
|
|
Speed up queries on larger datasets.
|
|
</Card>
|
|
<Card title="Embedding models" icon="vector-square" href="/embedding/">
|
|
Generate vectors from your own data.
|
|
</Card>
|
|
</Columns>
|