mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
156 lines
5.5 KiB
Plaintext
156 lines
5.5 KiB
Plaintext
---
|
|
title: "Embeddings: Quickstart"
|
|
sidebarTitle: "Quickstart"
|
|
description: "Quickstart guide for generating and working with embeddings."
|
|
icon: "rocket"
|
|
---
|
|
|
|
import {
|
|
TsQuickstartConnect,
|
|
TsQuickstartCreateTable,
|
|
TsQuickstartImports,
|
|
TsQuickstartInitModel,
|
|
TsQuickstartQuery,
|
|
TsQuickstartSchema,
|
|
} from '/snippets/sentence-transformers.mdx';
|
|
|
|
LanceDB will automatically vectorize the data both at ingestion and query time. All you need to do is specify which model to use.
|
|
Popular embedding models like OpenAI, Hugging Face, Sentence Transformers, CLIP, and more, are supported.
|
|
|
|
## Step 1: Import Required Libraries {#step-1-import-required-libraries}
|
|
|
|
First, import the necessary LanceDB components:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
import lancedb
|
|
from lancedb.pydantic import LanceModel, Vector
|
|
from lancedb.embeddings import get_registry
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartImports}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
- `lancedb`: The main database connection and operations
|
|
- `LanceModel`: Pydantic model for defining table schemas
|
|
- `Vector`: Field type for storing vector embeddings
|
|
- `get_registry()`: Access to the embedding function registry. It has all the supported as well as custom embedding functions registered by the user
|
|
- TypeScript uses `lancedb.embedding.getRegistry()` and `lancedb.embedding.LanceSchema()` for the same registry/schema workflow
|
|
- In TypeScript, import the provider module before calling `getRegistry().get(...)`; the provider import is what registers names such as `"huggingface"` or `"openai"`
|
|
|
|
## Step 2: Connect to LanceDB {#step-2-connect-to-lancedb}
|
|
|
|
Establish a connection to your LanceDB OSS directory or Enterprise cluster:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
# Enter your LanceDB connection URI for OSS or Enterprise here
|
|
db = lancedb.connect(...)
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Step 3: Initialize the Embedding Function {#step-3-initialize-the-embedding-function}
|
|
|
|
Choose and configure your embedding model:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
model = get_registry().get("sentence-transformers").create(name="BAAI/bge-small-en-v1.5", )
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartInitModel}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
This creates an embedding function from the local embedding registry. The Python snippet uses the
|
|
`sentence-transformers` provider with the BGE model; the TypeScript snippet uses the Transformers-backed
|
|
`huggingface` provider. You can:
|
|
- Change `"sentence-transformers"` to other providers like `"openai"`, `"cohere"`, etc.
|
|
- Modify the model name for different embedding models
|
|
- Set `device="cuda"` for GPU acceleration if available
|
|
|
|
## Step 4: Define Your Schema {#step-4-define-your-schema}
|
|
|
|
Create a Pydantic model that defines your table structure:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
class Words(LanceModel):
|
|
text: str = model.SourceField()
|
|
vector: Vector(model.ndims()) = model.VectorField()
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartSchema}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
- `SourceField()`: This field will be embedded
|
|
- `VectorField()`: This stores the embeddings
|
|
- `model.ndims()`: Sets vector dimensions for your model
|
|
- In TypeScript, use `model.sourceField(...)` and `model.vectorField()` inside `LanceSchema(...)`
|
|
|
|
## Step 5: Create Table and Ingest Data {#step-5-create-table-and-ingest-data}
|
|
|
|
Create a table with your schema and add data:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
table = db.create_table("words", schema=Words)
|
|
table.add([
|
|
{"text": "hello world"},
|
|
{"text": "goodbye world"}
|
|
])
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartCreateTable}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
The `table.add()` call automatically:
|
|
- Takes the text from each document
|
|
- Generates embeddings using your chosen model
|
|
- Stores both the original text and the vector embeddings
|
|
|
|
If your input already includes the vector column, automatic embedding only runs when that column is
|
|
absent or entirely null for the batch. Partially supplied vectors are treated as manual data, so
|
|
LanceDB preserves them instead of filling only the missing rows.
|
|
|
|
## Step 6: Query with Automatic Embedding {#step-6-query-with-automatic-embedding}
|
|
|
|
Note: On LanceDB Enterprise, the server does not generate embeddings from query text. In the Python remote
|
|
client, `table.search("greetings")` can still work when the table schema includes embedding metadata, because
|
|
the client computes the query embedding before sending the vector search. If there is no embedding metadata for
|
|
that search, `search("greetings")` in `auto` mode is treated as FTS instead.
|
|
|
|
Search your data using natural language queries:
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
query = "greetings"
|
|
actual = table.search(query).limit(1).to_pydantic(Words)[0]
|
|
print(actual.text)
|
|
```
|
|
|
|
<CodeBlock filename="TypeScript" language="typescript" icon="square-js">
|
|
{TsQuickstartQuery}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
The search process:
|
|
1. Automatically converts your query text to embeddings
|
|
2. Finds the most similar vectors in your table
|
|
3. Returns the matching documents
|
|
|
|
Automatic text search depends on the table's embedding metadata. If the client cannot reconstruct an
|
|
embedding function from that metadata, compute the query embedding yourself and search with the
|
|
vector directly.
|