mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-30 23:36:52 +00:00
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
156 lines
5.5 KiB
Plaintext
156 lines
5.5 KiB
Plaintext
---
|
|
title: "Embeddings: Quickstart"
|
|
sidebarTitle: "Quickstart"
|
|
description: "Quickstart guide for generating and working with embeddings."
|
|
icon: "rocket"
|
|
---
|
|
|
|
import {
|
|
TsQuickstartConnect,
|
|
TsQuickstartCreateTable,
|
|
TsQuickstartImports,
|
|
TsQuickstartInitModel,
|
|
TsQuickstartQuery,
|
|
TsQuickstartSchema,
|
|
} from '/snippets/sentence-transformers.mdx';
|
|
|
|
LanceDB will automatically vectorize the data both at ingestion and query time. All you need to do is specify which model to use.
|
|
Popular embedding models like OpenAI, Hugging Face, Sentence Transformers, CLIP, and more, are supported.
|
|
|
|
## Step 1: Import Required Libraries {#step-1-import-required-libraries}
|
|
|
|
First, import the necessary LanceDB components:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
import lancedb
|
|
from lancedb.pydantic import LanceModel, Vector
|
|
from lancedb.embeddings import get_registry
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartImports}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
- `lancedb`: The main database connection and operations
|
|
- `LanceModel`: Pydantic model for defining table schemas
|
|
- `Vector`: Field type for storing vector embeddings
|
|
- `get_registry()`: Access to the embedding function registry. It has all the supported as well as custom embedding functions registered by the user
|
|
- TypeScript uses `lancedb.embedding.getRegistry()` and `lancedb.embedding.LanceSchema()` for the same registry/schema workflow
|
|
- In TypeScript, import the provider module before calling `getRegistry().get(...)`; the provider import is what registers names such as `"huggingface"` or `"openai"`
|
|
|
|
## Step 2: Connect to LanceDB {#step-2-connect-to-lancedb}
|
|
|
|
Establish a connection to your LanceDB Embedded directory or Enterprise cluster:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
# Enter your LanceDB connection URI for LanceDB Embedded or Enterprise here
|
|
db = lancedb.connect(...)
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Step 3: Initialize the Embedding Function {#step-3-initialize-the-embedding-function}
|
|
|
|
Choose and configure your embedding model:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
model = get_registry().get("sentence-transformers").create(name="BAAI/bge-small-en-v1.5", )
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartInitModel}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
This creates an embedding function from the local embedding registry. The Python snippet uses the
|
|
`sentence-transformers` provider with the BGE model; the TypeScript snippet uses the Transformers-backed
|
|
`huggingface` provider. You can:
|
|
- Change `"sentence-transformers"` to other providers like `"openai"`, `"cohere"`, etc.
|
|
- Modify the model name for different embedding models
|
|
- Set `device="cuda"` for GPU acceleration if available
|
|
|
|
## Step 4: Define Your Schema {#step-4-define-your-schema}
|
|
|
|
Create a Pydantic model that defines your table structure:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
class Words(LanceModel):
|
|
text: str = model.SourceField()
|
|
vector: Vector(model.ndims()) = model.VectorField()
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartSchema}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
- `SourceField()`: This field will be embedded
|
|
- `VectorField()`: This stores the embeddings
|
|
- `model.ndims()`: Sets vector dimensions for your model
|
|
- In TypeScript, use `model.sourceField(...)` and `model.vectorField()` inside `LanceSchema(...)`
|
|
|
|
## Step 5: Create Table and Ingest Data {#step-5-create-table-and-ingest-data}
|
|
|
|
Create a table with your schema and add data:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
table = db.create_table("words", schema=Words)
|
|
table.add([
|
|
{"text": "hello world"},
|
|
{"text": "goodbye world"}
|
|
])
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartCreateTable}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
The `table.add()` call automatically:
|
|
- Takes the text from each document
|
|
- Generates embeddings using your chosen model
|
|
- Stores both the original text and the vector embeddings
|
|
|
|
If your input already includes the vector column, automatic embedding only runs when that column is
|
|
absent or entirely null for the batch. Partially supplied vectors are treated as manual data, so
|
|
LanceDB preserves them instead of filling only the missing rows.
|
|
|
|
## Step 6: Query with Automatic Embedding {#step-6-query-with-automatic-embedding}
|
|
|
|
Note: On LanceDB Enterprise, the server does not generate embeddings from query text. When the Python
|
|
SDK connects to Enterprise, `table.search("greetings")` can still work when the table schema includes embedding metadata, because
|
|
the client computes the query embedding before sending the vector search. If there is no embedding metadata for
|
|
that search, `search("greetings")` in `auto` mode is treated as FTS instead.
|
|
|
|
Search your data using natural language queries:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
query = "greetings"
|
|
actual = table.search(query).limit(1).to_pydantic(Words)[0]
|
|
print(actual.text)
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartQuery}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
The search process:
|
|
1. Automatically converts your query text to embeddings
|
|
2. Finds the most similar vectors in your table
|
|
3. Returns the matching documents
|
|
|
|
Automatic text search depends on the table's embedding metadata. If the client cannot reconstruct an
|
|
embedding function from that metadata, compute the query embedding yourself and search with the
|
|
vector directly.
|