Files
lancedb/docs/web/embedding/quickstart.mdx
T
Jack YeandClaude Sonnet 5 de95ed3f6d docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:

**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.

**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.

**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.

**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.

**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.

Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-03 16:17:24 -08:00

155 lines
5.5 KiB
Plaintext

---
title: "Embeddings: Quickstart"
sidebarTitle: "Quickstart"
description: "Quickstart guide for generating and working with embeddings."
icon: "rocket"
---
import {
TsQuickstartConnect,
TsQuickstartCreateTable,
TsQuickstartImports,
TsQuickstartInitModel,
TsQuickstartQuery,
TsQuickstartSchema,
} from '/snippets/sentence-transformers.mdx';
LanceDB will automatically vectorize the data both at ingestion and query time. All you need to do is specify which model to use.
Popular embedding models like OpenAI, HuggingFace, Sentence Transformers, CLIP, and more, are supported.
## Step 1: Import Required Libraries {#step-1-import-required-libraries}
First, import the necessary LanceDB components:
<CodeGroup>
```python Python icon="python"
from lancedb.pydantic import LanceModel, Vector
from lancedb.embeddings import get_registry
```
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
{TsQuickstartImports}
</CodeBlock>
</CodeGroup>
- `lancedb`: The main database connection and operations
- `LanceModel`: Pydantic model for defining table schemas
- `Vector`: Field type for storing vector embeddings
- `get_registry()`: Access to the embedding function registry. It has all the supported as well as custom embedding functions registered by the user
- TypeScript uses `lancedb.embedding.getRegistry()` and `lancedb.embedding.LanceSchema()` for the same registry/schema workflow
- In TypeScript, import the provider module before calling `getRegistry().get(...)`; the provider import is what registers names such as `"huggingface"` or `"openai"`
## Step 2: Connect to LanceDB {#step-2-connect-to-lancedb}
Establish a connection to your LanceDB Embedded directory or Enterprise cluster:
<CodeGroup>
```python Python icon="python"
# Enter your LanceDB connection URI for LanceDB Embedded or Enterprise here
db = lancedb.connect(...)
```
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
{TsQuickstartConnect}
</CodeBlock>
</CodeGroup>
## Step 3: Initialize the Embedding Function {#step-3-initialize-the-embedding-function}
Choose and configure your embedding model:
<CodeGroup>
```python Python icon="python"
model = get_registry().get("sentence-transformers").create(name="BAAI/bge-small-en-v1.5", )
```
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
{TsQuickstartInitModel}
</CodeBlock>
</CodeGroup>
This creates an embedding function from the local embedding registry. The Python snippet uses the
`sentence-transformers` provider with the BGE model; the TypeScript snippet uses the Transformers-backed
`huggingface` provider. You can:
- Change `"sentence-transformers"` to other providers like `"openai"`, `"cohere"`, etc.
- Modify the model name for different embedding models
- Set `device="cuda"` for GPU acceleration if available
## Step 4: Define Your Schema {#step-4-define-your-schema}
Create a Pydantic model that defines your table structure:
<CodeGroup>
```python Python icon="python"
class Words(LanceModel):
text: str = model.SourceField()
vector: Vector(model.ndims()) = model.VectorField()
```
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
{TsQuickstartSchema}
</CodeBlock>
</CodeGroup>
- `SourceField()`: This field will be embedded
- `VectorField()`: This stores the embeddings
- `model.ndims()`: Sets vector dimensions for your model
- In TypeScript, use `model.sourceField(...)` and `model.vectorField()` inside `LanceSchema(...)`
## Step 5: Create Table and Ingest Data {#step-5-create-table-and-ingest-data}
Create a table with your schema and add data:
<CodeGroup>
```python Python icon="python"
table = db.create_table("words", schema=Words)
table.add([
{"text": "hello world"},
{"text": "goodbye world"}
])
```
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
{TsQuickstartCreateTable}
</CodeBlock>
</CodeGroup>
The `table.add()` call automatically:
- Takes the text from each document
- Generates embeddings using your chosen model
- Stores both the original text and the vector embeddings
If your input already includes the vector column, automatic embedding only runs when that column is
absent or entirely null for the batch. Partially supplied vectors are treated as manual data, so
LanceDB preserves them instead of filling only the missing rows.
## Step 6: Query with Automatic Embedding {#step-6-query-with-automatic-embedding}
Note: On LanceDB Enterprise, the server does not generate embeddings from query text. When the Python
SDK connects to Enterprise, `table.search("greetings")` can still work when the table schema includes embedding metadata, because
the client computes the query embedding before sending the vector search. If there is no embedding metadata for
that search, `search("greetings")` in `auto` mode is treated as FTS instead.
Search your data using natural language queries:
<CodeGroup>
```python Python icon="python"
query = "greetings"
actual = table.search(query).limit(1).to_pydantic(Words)[0]
print(actual.text)
```
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
{TsQuickstartQuery}
</CodeBlock>
</CodeGroup>
The search process:
1. Automatically converts your query text to embeddings
2. Finds the most similar vectors in your table
3. Returns the matching documents
Automatic text search depends on the table's embedding metadata. If the client cannot reconstruct an
embedding function from that metadata, compute the query embedding yourself and search with the
vector directly.