mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
404 lines
16 KiB
Plaintext
404 lines
16 KiB
Plaintext
---
|
|
title: Quickstart
|
|
sidebarTitle: "Quickstart"
|
|
description: "Install LanceDB, create a table, and run your first vector search."
|
|
icon: rocket
|
|
---
|
|
import { PyConnect, TsConnect, RsConnect, PyConnectObjectStorage, TsConnectObjectStorage, RsConnectObjectStorage, PyConnectEnterprisePlaceholders, TsConnectEnterprisePlaceholders, RsConnectEnterprisePlaceholders } from '/snippets/connection.mdx';
|
|
import { PyQuickstartData, PyQuickstartCreateTable, PyQuickstartAddData, PyQuickstartCurateSample, PyQuickstartCurateDedupeSetup, PyQuickstartCurateDedupe, PyQuickstartCurateDedupeQuery, PyQuickstartFlattenStats, PyQuickstartAddFeature, PyQuickstartAddFeatureEnterprise, PyQuickstartMultimodalBytes, PyQuickstartTraining, TsQuickstartData, TsQuickstartCreateTable, TsQuickstartAddData, TsQuickstartVectorSearch1, TsQuickstartMultimodalBytes, RsQuickstartDefineStruct, RsQuickstartData, RsQuickstartCreateTable, RsQuickstartAddData, RsQuickstartVectorSearch1, RsQuickstartMultimodalBytes } from '/snippets/quickstart.mdx';
|
|
|
|
This quickstart uses LanceDB Embedded with a local database and works without an account or API key.
|
|
LanceDB Embedded runs in your application's process; your data can live on local disk or in object storage.
|
|
|
|
<Tip>
|
|
Using LanceDB Enterprise? Only the connection changes. In step 2, use the
|
|
[Enterprise](#connect-to-lancedb-enterprise) connection with your deployment's database URI, API key, and endpoint
|
|
in place of the local directory. The remaining steps use the same API, except
|
|
steps 5 and 6, which need Enterprise either way.
|
|
</Tip>
|
|
|
|
## 1\. Install LanceDB {#1-install-lancedb}
|
|
|
|
Choose your language. Python uses Pandas to display the search results. TypeScript needs Node.js 22.18 or later to run the `.mts` example directly.
|
|
|
|
<CodeGroup>
|
|
```bash Python (pip) icon="python"
|
|
pip install lancedb pandas
|
|
```
|
|
|
|
```bash Python (uv) icon="python"
|
|
uv venv
|
|
uv pip install lancedb pandas
|
|
```
|
|
|
|
```bash TypeScript icon="square-js"
|
|
npm install @lancedb/lancedb
|
|
```
|
|
|
|
```bash Rust icon="rust"
|
|
cargo init --name lancedb_quickstart
|
|
cargo add lancedb@0.31.0 --features polars,remote
|
|
cargo add tokio --features macros,rt-multi-thread
|
|
```
|
|
</CodeGroup>
|
|
|
|
Run the Rust commands in a new, empty directory. The `polars` feature provides DataFrame output; `remote` also lets this example connect to Enterprise.
|
|
|
|
<Accordion title="Python pre-release builds and older CPUs">
|
|
### Python pre-release builds {#python-pre-release-builds}
|
|
|
|
To pick up the latest features and bug fixes
|
|
before the next stable release, install a pre-release from LanceDB's Fury index.
|
|
|
|
<CodeGroup>
|
|
```bash Python (pip) icon="terminal"
|
|
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
|
|
```
|
|
|
|
```bash Python (uv) icon="terminal"
|
|
uv venv
|
|
uv pip install --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
|
|
|
|
# To add to pyproject.toml, use:
|
|
uv add --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
|
|
```
|
|
</CodeGroup>
|
|
|
|
Most applications only need `lancedb`. Install the optional `pylance` extra and
|
|
include the Lance preview index only when building query engines or connectors,
|
|
calling format-level `lance` APIs directly, or converting plain-scan lazy or
|
|
legacy blob columns with `to_pandas()`:
|
|
|
|
<CodeGroup>
|
|
```bash Python (pip) icon="terminal"
|
|
pip install --pre \
|
|
--extra-index-url https://pypi.fury.io/lancedb/ \
|
|
--extra-index-url https://pypi.fury.io/lance-format/ \
|
|
"lancedb[pylance]"
|
|
```
|
|
|
|
```bash Python (uv) icon="terminal"
|
|
uv pip install --prerelease allow \
|
|
--index https://pypi.fury.io/lancedb/ \
|
|
--index https://pypi.fury.io/lance-format/ \
|
|
--index-strategy unsafe-best-match \
|
|
"lancedb[pylance]"
|
|
```
|
|
</CodeGroup>
|
|
|
|
The LanceDB preview index contains old `pylance` releases; current Lance previews
|
|
are published to the `lance-format` index. By default, uv only considers versions
|
|
from the first index containing a package, which can select an old `pylance` or
|
|
backtrack to an older LanceDB release. `unsafe-best-match` searches all configured
|
|
indexes, like pip; use it only with indexes you trust. Install the extra rather
|
|
than an unconstrained `pylance` so LanceDB's minimum supported version is enforced.
|
|
|
|
<Note>
|
|
Pre-release builds receive the same level of testing as stable releases, but their availability is not guaranteed
|
|
for more than 6 months after release. For real-world workloads, we recommend you use the latest stable release
|
|
as far as possible.
|
|
</Note>
|
|
|
|
### Older x86_64 CPUs {#older-x86-cpus}
|
|
The default `lancedb` wheel targets `x86-64-haswell` (AVX2 + FMA + F16C). On older x86_64 CPUs without AVX2
|
|
(Intel Sandy Bridge, Ivy Bridge, Westmere; AMD Bulldozer, Piledriver, Steamroller), `import lancedb` crashes
|
|
with `Illegal instruction`.
|
|
|
|
Install `lancedb-compat` instead. It exposes the same API (`import lancedb` still works), is built at the
|
|
`x86-64-v2` baseline, and uses runtime SIMD dispatch to still use AVX2, FMA, or AVX-512 when available:
|
|
|
|
<CodeGroup>
|
|
```bash Python (pip) icon="terminal"
|
|
pip install lancedb-compat
|
|
```
|
|
</CodeGroup>
|
|
|
|
The two packages share the same `lancedb/` namespace and conflict at install time, so pick one. To switch,
|
|
uninstall the other first (`pip uninstall lancedb && pip install lancedb-compat`).
|
|
|
|
|
|
</Accordion>
|
|
|
|
## 2\. Connect to a database {#2-connect-to-a-lancedb-database}
|
|
|
|
### Local directory {#connect-via-local-directory-path}
|
|
|
|
LanceDB creates an `ex_lancedb` directory to store your tables. No server is needed.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyConnect}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
|
|
{TsConnect}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{"use lancedb::connect;\n\n"}
|
|
{RsConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
For Python's async API, see [Connecting to a database](/basics/catalog).
|
|
|
|
### Object storage {#connect-via-object-storage-uris}
|
|
|
|
Use an `s3://`, `gs://`, or `az://` URI in place of the local path. See [Object storage](/basics/catalog#object-storage) for credentials and provider-specific options.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyConnectObjectStorage}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
|
|
{TsConnectObjectStorage}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{"use lancedb::connect;\n\n"}
|
|
{RsConnectObjectStorage}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### Enterprise {#connect-to-lancedb-enterprise}
|
|
|
|
Use the database URI, API key, and endpoint provided by your administrator or the LanceDB team. Replace the placeholders below with those details. Use the deployment's region if provided; otherwise keep `us-east-1`.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyConnectEnterprisePlaceholders}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{"import * as lancedb from \"@lancedb/lancedb\";\n\n"}
|
|
{TsConnectEnterprisePlaceholders}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{"use lancedb::connect;\n\n"}
|
|
{RsConnectEnterprisePlaceholders}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<Note>
|
|
In application code, load credentials from environment variables or a secrets manager. See [connection settings](/basics/catalog#enterprise) for an example.
|
|
</Note>
|
|
|
|
Need a deployment? [Contact the LanceDB team](https://www.lancedb.com/contact). For differences between local and server execution, see [LanceDB Enterprise](/index#architecture).
|
|
|
|
## 3\. Create a table and add data {#3-create-a-new-table}
|
|
|
|
Each row describes a frame from a robot video clip and stores an embedding alongside its video, caption, and quality stats.
|
|
|
|
<Info>
|
|
These embeddings are made up for this example. In your application, use an embedding model to generate them from your data.
|
|
</Info>
|
|
|
|
<Accordion title="Frame records">
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartData}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsQuickstartData}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsQuickstartDefineStruct}
|
|
{RsQuickstartData}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
</Accordion>
|
|
|
|
Create a `frames` table from those records:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartCreateTable}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsQuickstartCreateTable}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsQuickstartCreateTable}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
This example uses overwrite mode so you can run it again. It replaces any existing `frames` table in the database you connect to.
|
|
|
|
Add one more frame:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartAddData}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsQuickstartAddData}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsQuickstartAddData}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## 4\. Search the table {#4-semantic-search}
|
|
|
|
Find the two embeddings closest to the query embedding. This example uses a fixed embedding; it does not call an embedding model.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{"# Search for frames similar to \"a gripper closing on an object\"\nquery_vector = [0.2, 0.8, 0.4, 0.9]\n\n# Ensure you run `pip install pandas` beforehand\nresult = (\n table.search(query_vector)\n .select([\"frame_id\", \"video_id\", \"caption\", \"_distance\"])\n .limit(2)\n .to_pandas()\n)\nprint(result)\n"}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsQuickstartVectorSearch1}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsQuickstartVectorSearch1}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Frames 2 and 4 come back first -- the gripper closing on the cup, then lowering it. Python displays a Pandas DataFrame, Rust displays a Polars DataFrame, and TypeScript displays an array of results. The `_distance` column measures how far each result is from your query embedding, with smaller values indicating a closer match.
|
|
|
|
## 5\. Curate data with SQL {#5-curate-with-sql}
|
|
|
|
<Info>
|
|
Step 5 needs LanceDB Enterprise: SQL is an Enterprise feature. If you're
|
|
using LanceDB Embedded, skip ahead to
|
|
[Flatten nested fields](#flatten-nested-fields) below.
|
|
</Info>
|
|
|
|
LanceDB Enterprise accepts SQL over the same connection, for sampling, joins, and
|
|
curation tasks that go beyond a single table scan. Draw a quick sample to look
|
|
at the data:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartCurateSample}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
See [SQL](/basics/sql) for the Flight SQL client and CLI alternatives to `execute_query()`.
|
|
|
|
Now add a near-duplicate frame -- frame 5 is one video frame after frame 2, with
|
|
an almost identical embedding -- and build the vector index that `dedupe` reads:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartCurateDedupeSetup}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Collapse near-duplicates into a materialized view that stays current as new frames arrive, then read it back:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartCurateDedupe}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartCurateDedupeQuery}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Frames 2 and 5 collapse into one representative -- five rows become four.
|
|
Which of the two ids survives is not guaranteed to stay the same across
|
|
appends, so don't treat it as a stable identifier for the cluster. See
|
|
[Deduplicate near-duplicate records](/curation/dedupe) for `duplicate_pairs`,
|
|
resolution strategies, and tuning `distance_threshold`, and
|
|
[Statistical Sampling](/curation/statistical-sampling) for percentage samples,
|
|
repeatable draws, and sampling across a join.
|
|
|
|
## 6\. Add a derived feature {#6-add-a-derived-feature}
|
|
|
|
### Flatten nested fields {#flatten-nested-fields}
|
|
|
|
A UDF binds to top-level columns, so flatten the fields inside `stats` first.
|
|
This is a plain eager `add_columns` call, so it works the same on LanceDB
|
|
Embedded and LanceDB Enterprise -- [step 8](#8-train) needs these columns
|
|
either way:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartFlattenStats}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<Info>
|
|
The rest of this step needs LanceDB Enterprise -- Functions are an
|
|
Enterprise feature. If you're using LanceDB Embedded, skip ahead to
|
|
[step 7](#7-store-multimodal-data).
|
|
</Info>
|
|
|
|
Register a UDF as a Function, then call it to fill a column computed from
|
|
those flattened fields. This fills every existing row once, immediately:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartAddFeature}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Read it back like any other column:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{"features = table.search().select([\"frame_id\", \"video_id\", \"quality_score\"]).limit(10).to_pandas()\nprint(features)\n"}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
`refresh_column` fills only rows added since the last refresh and recomputes rows whose inputs changed, leaving unchanged rows alone. Run the same refresh as a background job instead, so it doesn't block:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartAddFeatureEnterprise}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
See [Create a Function](/curation/create-function) for UDFs that call external
|
|
models or services, and [Computed columns](/feature-engineering/add-computed-column)
|
|
for the SQL-expression form that doesn't need a registered Function.
|
|
|
|
## 7\. Store multimodal data {#7-store-multimodal-data}
|
|
|
|
Store images, audio, or other binary data alongside vectors and metadata, in a column of raw bytes. This creates a new table from one image file -- replace the path with one of your own:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartMultimodalBytes}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsQuickstartMultimodalBytes}
|
|
</CodeBlock>
|
|
<CodeBlock filename="Rust" language="Rust" icon="rust">
|
|
{RsQuickstartMultimodalBytes}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
See [Multimodal data types](/management/table#multimodal) for larger payloads and the Blob v2 column type, which avoids loading binary data on every scan.
|
|
|
|
## 8\. Train on your data {#8-train}
|
|
|
|
Read table data into PyTorch, projecting and shuffling only the columns a training step needs:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyQuickstartTraining}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
`StreamingDataset` wraps a table for PyTorch training reads; `columns` projects before reading, so a training step touches only the columns it selects, not unused ones such as `caption` or `embedding`. It works the same way against LanceDB Embedded and LanceDB Enterprise. See [PyTorch Data Loader](/training/data-loading) for shuffling, transforms, resuming, and multi-worker loading, and [Permutation](/training/permutation) for the lower-level row-ordering primitive it's built on.
|
|
|
|
## Where to go next {#next-steps}
|
|
|
|
<Columns cols={2}>
|
|
<Card title="Basic table operations" icon="table" href="/management/table">
|
|
Create, update, and manage your tables.
|
|
</Card>
|
|
<Card title="Build a RAG app" icon="search" href="/tutorials/agents/contextual-rag">
|
|
Use retrieved context in a language model application.
|
|
</Card>
|
|
<Card title="Indexing" icon="layer-group" href="/indexing/">
|
|
Speed up queries on larger datasets.
|
|
</Card>
|
|
<Card title="Embedding models" icon="vector-square" href="/embedding/">
|
|
Generate vectors from your own data.
|
|
</Card>
|
|
</Columns>
|