mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
The pages describing this code lived in lancedb/docs, a repository with ten stars that neither contributing guide mentioned. A user who hit a wrong page while reading the source had to find a separate, near-invisible repository to fix it. These 102 pages now sit beside the code, and a documentation fix is a pull request in the repository the reader already had open. `docs/web` is a complete Mintlify site: `cd docs/web && mint dev` serves it with no other checkout. Enterprise pages stay private and are merged in when the published site is assembled, so links into them do not resolve here — every open-source page does. `docs/web-tests` holds the tests the code examples are extracted from. They ran against a published wheel before; here they can run against this repository's own SDK, so an example that stops compiling fails the pull request that broke it rather than being noticed later. Both directories sit beside `docs/src` and are invisible to it: mkdocs reads `docs_dir: src`, so the SDK reference builds and deploys exactly as before.
156 lines
5.5 KiB
Plaintext
156 lines
5.5 KiB
Plaintext
---
|
|
title: "Embeddings: Quickstart"
|
|
sidebarTitle: "Quickstart"
|
|
description: "Quickstart guide for generating and working with embeddings."
|
|
icon: "rocket"
|
|
---
|
|
|
|
import {
|
|
TsQuickstartConnect,
|
|
TsQuickstartCreateTable,
|
|
TsQuickstartImports,
|
|
TsQuickstartInitModel,
|
|
TsQuickstartQuery,
|
|
TsQuickstartSchema,
|
|
} from '/snippets/sentence-transformers.mdx';
|
|
|
|
LanceDB will automatically vectorize the data both at ingestion and query time. All you need to do is specify which model to use.
|
|
Popular embedding models like OpenAI, Hugging Face, Sentence Transformers, CLIP, and more, are supported.
|
|
|
|
## Step 1: Import Required Libraries {#step-1-import-required-libraries}
|
|
|
|
First, import the necessary LanceDB components:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
import lancedb
|
|
from lancedb.pydantic import LanceModel, Vector
|
|
from lancedb.embeddings import get_registry
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartImports}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
- `lancedb`: The main database connection and operations
|
|
- `LanceModel`: Pydantic model for defining table schemas
|
|
- `Vector`: Field type for storing vector embeddings
|
|
- `get_registry()`: Access to the embedding function registry. It has all the supported as well as custom embedding functions registered by the user
|
|
- TypeScript uses `lancedb.embedding.getRegistry()` and `lancedb.embedding.LanceSchema()` for the same registry/schema workflow
|
|
- In TypeScript, import the provider module before calling `getRegistry().get(...)`; the provider import is what registers names such as `"huggingface"` or `"openai"`
|
|
|
|
## Step 2: Connect to LanceDB {#step-2-connect-to-lancedb}
|
|
|
|
Establish a connection to your LanceDB OSS directory or Enterprise cluster:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
# Enter your LanceDB connection URI for OSS or Enterprise here
|
|
db = lancedb.connect(...)
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Step 3: Initialize the Embedding Function {#step-3-initialize-the-embedding-function}
|
|
|
|
Choose and configure your embedding model:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
model = get_registry().get("sentence-transformers").create(name="BAAI/bge-small-en-v1.5", )
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartInitModel}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
This creates an embedding function from the local embedding registry. The Python snippet uses the
|
|
`sentence-transformers` provider with the BGE model; the TypeScript snippet uses the Transformers-backed
|
|
`huggingface` provider. You can:
|
|
- Change `"sentence-transformers"` to other providers like `"openai"`, `"cohere"`, etc.
|
|
- Modify the model name for different embedding models
|
|
- Set `device="cuda"` for GPU acceleration if available
|
|
|
|
## Step 4: Define Your Schema {#step-4-define-your-schema}
|
|
|
|
Create a Pydantic model that defines your table structure:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
class Words(LanceModel):
|
|
text: str = model.SourceField()
|
|
vector: Vector(model.ndims()) = model.VectorField()
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartSchema}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
- `SourceField()`: This field will be embedded
|
|
- `VectorField()`: This stores the embeddings
|
|
- `model.ndims()`: Sets vector dimensions for your model
|
|
- In TypeScript, use `model.sourceField(...)` and `model.vectorField()` inside `LanceSchema(...)`
|
|
|
|
## Step 5: Create Table and Ingest Data {#step-5-create-table-and-ingest-data}
|
|
|
|
Create a table with your schema and add data:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
table = db.create_table("words", schema=Words)
|
|
table.add([
|
|
{"text": "hello world"},
|
|
{"text": "goodbye world"}
|
|
])
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartCreateTable}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
The `table.add()` call automatically:
|
|
- Takes the text from each document
|
|
- Generates embeddings using your chosen model
|
|
- Stores both the original text and the vector embeddings
|
|
|
|
If your input already includes the vector column, automatic embedding only runs when that column is
|
|
absent or entirely null for the batch. Partially supplied vectors are treated as manual data, so
|
|
LanceDB preserves them instead of filling only the missing rows.
|
|
|
|
## Step 6: Query with Automatic Embedding {#step-6-query-with-automatic-embedding}
|
|
|
|
Note: On LanceDB Enterprise, the server does not generate embeddings from query text. In the Python remote
|
|
client, `table.search("greetings")` can still work when the table schema includes embedding metadata, because
|
|
the client computes the query embedding before sending the vector search. If there is no embedding metadata for
|
|
that search, `search("greetings")` in `auto` mode is treated as FTS instead.
|
|
|
|
Search your data using natural language queries:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
query = "greetings"
|
|
actual = table.search(query).limit(1).to_pydantic(Words)[0]
|
|
print(actual.text)
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartQuery}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
The search process:
|
|
1. Automatically converts your query text to embeddings
|
|
2. Finds the most similar vectors in your table
|
|
3. Returns the matching documents
|
|
|
|
Automatic text search depends on the table's embedding metadata. If the client cannot reconstruct an
|
|
embedding function from that metadata, compute the query embedding yourself and search with the
|
|
vector directly.
|