mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
155 lines
5.5 KiB
Plaintext
155 lines
5.5 KiB
Plaintext
---
|
|
title: "Embeddings: Quickstart"
|
|
sidebarTitle: "Quickstart"
|
|
description: "Quickstart guide for generating and working with embeddings."
|
|
icon: "rocket"
|
|
---
|
|
|
|
import {
|
|
TsQuickstartConnect,
|
|
TsQuickstartCreateTable,
|
|
TsQuickstartImports,
|
|
TsQuickstartInitModel,
|
|
TsQuickstartQuery,
|
|
TsQuickstartSchema,
|
|
} from '/snippets/sentence-transformers.mdx';
|
|
|
|
LanceDB will automatically vectorize the data both at ingestion and query time. All you need to do is specify which model to use.
|
|
Popular embedding models like OpenAI, HuggingFace, Sentence Transformers, CLIP, and more, are supported.
|
|
|
|
## Step 1: Import Required Libraries {#step-1-import-required-libraries}
|
|
|
|
First, import the necessary LanceDB components:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
from lancedb.pydantic import LanceModel, Vector
|
|
from lancedb.embeddings import get_registry
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartImports}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
- `lancedb`: The main database connection and operations
|
|
- `LanceModel`: Pydantic model for defining table schemas
|
|
- `Vector`: Field type for storing vector embeddings
|
|
- `get_registry()`: Access to the embedding function registry. It has all the supported as well as custom embedding functions registered by the user
|
|
- TypeScript uses `lancedb.embedding.getRegistry()` and `lancedb.embedding.LanceSchema()` for the same registry/schema workflow
|
|
- In TypeScript, import the provider module before calling `getRegistry().get(...)`; the provider import is what registers names such as `"huggingface"` or `"openai"`
|
|
|
|
## Step 2: Connect to LanceDB {#step-2-connect-to-lancedb}
|
|
|
|
Establish a connection to your LanceDB Embedded directory or Enterprise cluster:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
# Enter your LanceDB connection URI for LanceDB Embedded or Enterprise here
|
|
db = lancedb.connect(...)
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Step 3: Initialize the Embedding Function {#step-3-initialize-the-embedding-function}
|
|
|
|
Choose and configure your embedding model:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
model = get_registry().get("sentence-transformers").create(name="BAAI/bge-small-en-v1.5", )
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartInitModel}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
This creates an embedding function from the local embedding registry. The Python snippet uses the
|
|
`sentence-transformers` provider with the BGE model; the TypeScript snippet uses the Transformers-backed
|
|
`huggingface` provider. You can:
|
|
- Change `"sentence-transformers"` to other providers like `"openai"`, `"cohere"`, etc.
|
|
- Modify the model name for different embedding models
|
|
- Set `device="cuda"` for GPU acceleration if available
|
|
|
|
## Step 4: Define Your Schema {#step-4-define-your-schema}
|
|
|
|
Create a Pydantic model that defines your table structure:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
class Words(LanceModel):
|
|
text: str = model.SourceField()
|
|
vector: Vector(model.ndims()) = model.VectorField()
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartSchema}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
- `SourceField()`: This field will be embedded
|
|
- `VectorField()`: This stores the embeddings
|
|
- `model.ndims()`: Sets vector dimensions for your model
|
|
- In TypeScript, use `model.sourceField(...)` and `model.vectorField()` inside `LanceSchema(...)`
|
|
|
|
## Step 5: Create Table and Ingest Data {#step-5-create-table-and-ingest-data}
|
|
|
|
Create a table with your schema and add data:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
table = db.create_table("words", schema=Words)
|
|
table.add([
|
|
{"text": "hello world"},
|
|
{"text": "goodbye world"}
|
|
])
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartCreateTable}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
The `table.add()` call automatically:
|
|
- Takes the text from each document
|
|
- Generates embeddings using your chosen model
|
|
- Stores both the original text and the vector embeddings
|
|
|
|
If your input already includes the vector column, automatic embedding only runs when that column is
|
|
absent or entirely null for the batch. Partially supplied vectors are treated as manual data, so
|
|
LanceDB preserves them instead of filling only the missing rows.
|
|
|
|
## Step 6: Query with Automatic Embedding {#step-6-query-with-automatic-embedding}
|
|
|
|
Note: On LanceDB Enterprise, the server does not generate embeddings from query text. When the Python
|
|
SDK connects to Enterprise, `table.search("greetings")` can still work when the table schema includes embedding metadata, because
|
|
the client computes the query embedding before sending the vector search. If there is no embedding metadata for
|
|
that search, `search("greetings")` in `auto` mode is treated as FTS instead.
|
|
|
|
Search your data using natural language queries:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
query = "greetings"
|
|
actual = table.search(query).limit(1).to_pydantic(Words)[0]
|
|
print(actual.text)
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartQuery}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
The search process:
|
|
1. Automatically converts your query text to embeddings
|
|
2. Finds the most similar vectors in your table
|
|
3. Returns the matching documents
|
|
|
|
Automatic text search depends on the table's embedding metadata. If the client cannot reconstruct an
|
|
embedding function from that metadata, compute the query embedding yourself and search with the
|
|
vector directly.
|