rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-11-13 14:16:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: fix table, accuracy and structure problems found in the pre-launch review (#4745)

Browse Source 
## Summary

Fixes the problems a page-by-page review of the docs found before
launch, after checking each one against the current source and, where
noted, the code. The matching fixes for the dataset and Geneva pages,
which live in `lancedb/docs`, are in lancedb/docs#402.

**Tables**
- `integrations/embedding/cohere`: each input-type description was split
across two table rows and wrapped in literal quotes
(`"`search_document`"`). Now one row per type.
- `integrations/embedding/gemini`: removes the same stray quotes from
the task-type table.
- `integrations/ai/langchain`: the MMR search table's `kwargs` row was
missing a cell, so its description sat under Type.
- Six reranker pages (answerdotai, cohere, colbert, jina, openai,
voyageai): closes the backtick before `_relevance_score`.
- `indexing/vector-index`: finishes a cell cut off at "with scalar".
- `sql/dql/dedupe`: labels the empty header cell in the resolution
table.

**Corrections**
- `curation/sampling`: sampling works in LanceDB Embedded through the
Python SDK, so the page is available in both, with the SQL sections
marked Enterprise.
- `feature-engineering/index`, `add-computed-column`: automatic refresh
is on by default for each computed column and takes effect once a
deployment enables its scheduler. `refresh-column` already said so;
these two implied it was off.
- `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates
`array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also
merges scalar-index's two "nested fields" sections into one.
- `integrations/embedding/ibm`: lists the four current models from
`CURRENT_MODELS` (it listed a deprecated one) and explains that the
default stays a legacy ID so existing tables reload with their model.
- `integrations/reranking/openai`: the class is `OpenaiReranker`, not
`OpenAI()`.
- `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit`
throughout, each step runs one command, each step passes the file it
writes to the next, and `save-as` uses `-f`/`-o`.
- `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the
pipeline reads, and the run command runs the file the page creates.
- `search/vector-search`: the multivector walkthrough connected to
Enterprise in step 1 and a local path in step 2. It now runs locally
throughout, with a note on connecting to Enterprise, and is renamed so
it doesn't read as a duplicate of the short section above.

**Structure and wording**
- `integrations/embedding/nvidia`: adds headings, and drops test-suite
details aimed at contributors.
- `integrations/ai/genkit`: makes the retrieval test a real heading and
describes the query step.
- `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a
`<Note>`.
- `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading,
not bold text.
- `curation/windowing`: sidebar title matches the curation overview
("Rank and Deduplicate").
- `deployment/storage`, `basics/agentic-skills`: fixes a garbled
sentence and a stale reference to the quickstart.


**Checked:** no new Vale alerts on the changed pages. Assembled together
with lancedb/docs#402, the site passes `mint validate` and `mint
broken-links --check-anchors --check-redirects`. Every changed page was
reviewed in a local preview.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brendan Clement
2026-10-29 21:26:37 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 3306e5fbf2
commit ca343a5c06
34 changed files with 107 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/src/js/classes/Index.md
+1 -1
View File
@@ -252,7 +252,7 @@ static labelList(): Index
Create a label list index.
LabelList index is a scalar index that can be used on `List<T>` columns to
support queries with `array_contains_all` and `array_contains_any`
support queries with `array_has_all` and `array_has_any`
using an underlying bitmap index.
#### Returns
docs/web/basics/agentic-skills.mdx
+3 -3
View File
@@ -84,8 +84,8 @@ found and what each target supports.
### Get started with the LanceDB agent plugin {#get-started-with-the-lancedb-agent-plugin}
This tutorial uses the Camelot dataset from the [quickstart](/quickstart), with
a portrait added for each character. Each LanceDB row contains validated
This tutorial uses a small Camelot dataset of characters from Arthurian legend,
with a portrait for each character. Each LanceDB row contains validated
metadata and raw JPEG bytes. Text, images, and any embeddings you add later
remain in the same table.
@@ -166,7 +166,7 @@ embedded retrieval library. A simple prompt like this should work:
Use the lancedb plugin to ingest the dataset in `data/` into a LanceDB
Embedded table.
# If using enterprise
# If using LanceDB Enterprise
Use the lancedb plugin to ingest the dataset in `data/` into a LanceDB
Enterprise table using the connection information in `.env`.
```
docs/web/curation/sampling.mdx
+14 -3
View File
@@ -5,7 +5,7 @@ description: "Choose a sample for exploration or evaluation and preserve the sel
icon: dice
keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"]
availability:
embedded: unavailable
embedded: available
enterprise: available
---
@@ -15,11 +15,14 @@ Sampling selects records from a larger table. Use it to explore unfamiliar
data, run a smaller experiment, or prepare an evaluation dataset. Enterprise
SQL provides `TABLESAMPLE` for selecting a percentage or number of rows.
This page covers Enterprise SQL sampling and a complete
[Python SDK example](#training) that runs locally.
Sampling works in LanceDB Embedded and LanceDB Enterprise through the
[Python SDK](#training). SQL sampling, the sections marked Enterprise below, needs
LanceDB Enterprise.
## Example data {#example-data}
<Badge color="red">Enterprise</Badge>
Connect with an [Enterprise SQL client](/basics/sql). The examples use a small
document table and a separate label table. Expand the setup below to create them.
@@ -58,6 +61,8 @@ has no label. These cases let you check the effect of each operation.
## Basic sampling {#quick}
<Badge color="red">Enterprise</Badge>
A percentage selects a fraction of the source; `ROWS` requests a row count:
```sql
@@ -75,6 +80,8 @@ change a production table's write configuration just to try an example.
## Sampling methods {#methods}
<Badge color="red">Enterprise</Badge>
Choose a method according to how records should be selected:
| Method | Selection | When to use it |
@@ -97,6 +104,8 @@ represent rare groups; check that the resulting sample covers the groups you nee
## Repeatable samples {#fixed}
<Badge color="red">Enterprise</Badge>
A seed requests a repeatable draw:
```sql
@@ -121,6 +130,8 @@ training or evaluation.
## Sampling with joins {#joins}
<Badge color="red">Enterprise</Badge>
Sampling both input tables independently changes which pairs can match.
For a one-to-one match, independent 10% row samples retain a given pair with
about 1% probability, not 10%.
docs/web/curation/windowing.mdx
+1 -1
View File
@@ -1,6 +1,6 @@
---
title: "Ranking and deduplication"
sidebarTitle: "Windowing"
sidebarTitle: "Rank and Deduplicate"
description: "Choose records with explicit ranking rules and deterministic tie handling."
icon: chart-simple
keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"]
docs/web/deployment/storage.mdx
+1 -1
View File
@@ -76,7 +76,7 @@ Use local disk only when you need extremely low latency and are comfortable owni
### File-format choices that interact with the backend {#file-format-choices}
A few `storage_options` keys shape new tables in ways that depend on the backend you picked above. They are documented in full on [below](#new-table-configuration); the architecture-level summary is:
A few `storage_options` keys shape new tables in ways that depend on the backend you picked above. They are documented in full [below](#new-table-configuration); the architecture-level summary is:
- `new_table_enable_v2_manifest_paths` matters most on object stores, where opening a table with many versions is dominated by listing cost. Leave it off for backward compatibility with clients older than LanceDB 0.10.0.
- `new_table_enable_stable_row_ids` keeps row IDs stable across compaction, delete, and merge. The choice is independent of the backend but affects any system that joins on row ID.
docs/web/feature-engineering/add-computed-column.mdx
+7 -7
View File
@@ -71,21 +71,21 @@ including that you can travel back to before the column existed.
`computed` and `transforms` cannot be combined in one call. Reach for `computed` when the table is
large enough that you would rather control when the work happens.
Pass `auto_refresh_enabled` and `auto_refresh_interval_seconds` alongside
`computed` to have the server keep the column refreshed instead of calling
[`refresh_column`](/feature-engineering/refresh-column) yourself:
Automatic refresh is on by default for each computed column, and takes effect
when your Enterprise deployment enables its refresh scheduler. Until then, and in
LanceDB Embedded, which never refreshes on a schedule, call
[`refresh_column`](/feature-engineering/refresh-column) yourself. Pass
`auto_refresh_enabled=False` to opt a column out, or
`auto_refresh_interval_seconds` to set how often it refreshes:
```python
table.add_columns(
computed={"doubled": "x * 2"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
A local table stores these settings but does not act on them -- automatic
scheduling runs on LanceDB Cloud or Enterprise. See
[Automatic refresh](/feature-engineering/refresh-column#automatic) for the
See [Automatic refresh](/feature-engineering/refresh-column#automatic) for the
full behavior, including how to change the setting later.
An eager `transforms` call looks like this, starting from a small product
docs/web/feature-engineering/index.mdx
+3 -2
View File
@@ -133,8 +133,9 @@ result.rows_assigned
A backfill over a large table can therefore be started, left, and checked on
from a different process: the connection does not have to stay open.
Enterprise manages the compute for the refresh. You still control when to
submit it; declaring a computed column does not schedule recurring refreshes.
Enterprise manages the compute for the refresh. Declaring a computed column
doesn't refresh it on a schedule unless your deployment enables automatic
refresh; see [Automatic refresh](/feature-engineering/refresh-column#automatic).
{/* Draft for the Functions release: That is also why a function must declare the environment it needs. */}
## Recovering from a failure {#failures}
docs/web/index.mdx
+1 -1
View File
@@ -133,7 +133,7 @@ To set up LanceDB Enterprise in your organization, [reach out to us](https://www
| | [Scalar indexes](/indexing/scalar-index) | ✅ | ✅ |
| **SQL and curation** | [SQL (Arrow Flight SQL)](/basics/sql) | ❌ | ✅ |
| | [Aggregation and profiling](/curation/aggregate) | ❌ | ✅ |
| | [Sampling](/curation/sampling) | ❌ | ✅ |
| | [Sampling](/curation/sampling) | ✅ | ✅ |
| | [Joins](/curation/join) | ❌ | ✅ |
| | [Ranking and deduplication](/curation/windowing) | ❌ | ✅ |
| | [Near-duplicate detection](/curation/dedupe) | ❌ | ✅ |
docs/web/indexing/index.mdx
+1 -1
View File
@@ -24,7 +24,7 @@ patterns. LanceDB supports three main index families:
| `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. |
| `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. |
| `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. |
| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_contains_all` or `array_contains_any` filters. | Scalar index for `List<T>` and `LargeList<T>` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. |
| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_has_all` or `array_has_any` filters. | Scalar index for `List<T>` and `LargeList<T>` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. |
| `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. |
A deployment builds the vector indexes `IVF_FLAT`, `IVF_PQ`, `IVF_RQ` and
docs/web/indexing/scalar-index.mdx
+4 -18
View File
@@ -30,7 +30,7 @@ LanceDB supports four types of scalar indexes:
- `BTREE`: Stores column data in sorted order for binary search. Best for columns with many unique values.
- `BITMAP`: Uses bitmaps to track value presence. Ideal for columns with few unique values (e.g., categories, tags).
- `LABEL_LIST`: Special index for `List<T>` and `LargeList<T>` columns of primitive values supporting `array_contains_all` and `array_contains_any` queries.
- `LABEL_LIST`: Special index for `List<T>` and `LargeList<T>` columns of primitive values supporting `array_has_all` and `array_has_any` queries.
- `FM`: FM-Index over string or binary columns that accelerates substring search via `contains(col, 'needle')`.
<Tip title="Working with scalar indices">
@@ -109,22 +109,6 @@ Scalar indexes can also speed up scans containing a vector search or full text s
</CodeBlock>
</CodeGroup>
## Indexing nested fields
Scalar indexes can target a scalar field inside a struct by passing its full dotted path. The path is preserved end to end: it's the value you pass to `create_scalar_index`, it's what `list_indices()` reports under `columns`, and it's the column reference you use in filter predicates.
```python
# Schema: pa.struct([pa.field("user_id", pa.int32())]) stored under the `metadata` column.
table.create_scalar_index("metadata.user_id", name="metadata_user_id_idx")
# The same dotted path works in WHERE clauses.
table.search().where("metadata.user_id = 42").limit(1).to_list()
```
<Note>
Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `metadata.author.name`). The same convention applies to FTS and vector indexes.
</Note>
## FM-Index for substring search
The `FM` index is a scalar index built over string or binary columns that
@@ -246,7 +230,9 @@ named `user.id` nested inside `metadata`), wrap that segment in backticks so
LanceDB can tell the dot apart from the path separator: `` metadata.`user.id` ``.
`list_indices()` echoes the same canonical path back, so the column you pass in
round-trips through index metadata regardless of nesting depth or escaping.
round-trips through index metadata regardless of nesting depth or escaping. Use the
same path in filter predicates, for example `metadata.user_id = 42`. FTS and vector
indexes follow the same path convention.
<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
docs/web/indexing/vector-index.mdx
+1 -1
View File
@@ -55,7 +55,7 @@ Lance with `index_type`.
| :--- | :--- | :--- | :--- | :--- |
| Maximum compression | `IVF_RQ` | IVF indexing with RaBitQ quantization | Around `1/32` of raw size with `num_bits=1` | `IvfRq` |
| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | IVF indexing with product quantization | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | `IvfPq` |
| Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar | Typically a little larger than `1/4` of raw size | `IvfHnswSq` |
| Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar quantization | Typically a little larger than `1/4` of raw size | `IvfHnswSq` |
| Highest recall / no quantization | `IVF_HNSW_FLAT` | IVF-HNSW indexing with no quantization; embedded library only | Around raw vector size plus HNSW graph overhead | `IvfHnswFlat` |
<Warning>
docs/web/integrations/ai/genkit.mdx
+7 -3
View File
@@ -37,11 +37,15 @@ genkit start -- tsx --watch src/index.ts
This'll add LanceDB as a retriever and indexer to the genkit instance. You can see it in the GUI view
<img width="1710" alt="Screenshot 2025-05-11 at 7 21 05 PM" src="https://github.com/user-attachments/assets/e752f7f4-785b-4797-a11e-72ab06a531b7" />
**Testing retrieval on a sample table**
Let's see the raw retrieval results
### Testing retrieval on a sample table {#testing-retrieval-on-a-sample-table}
To see the raw retrieval results, open the LanceDB retriever in the Genkit Developer UI,
enter a query and run it:
<img width="1710" alt="Screenshot 2025-05-11 at 7 21 05 PM" src="https://github.com/user-attachments/assets/b8d356ed-8421-4790-8fc0-d6af563b9657" />
On running this query, you'll get 5 results fetched from the lancedb table, where each result looks something like this:
The retriever returns 5 results from the LanceDB table. Each result looks something like this:
<img width="1417" alt="Screenshot 2025-05-11 at 7 21 18 PM" src="https://github.com/user-attachments/assets/77429525-36e2-4da6-a694-e58c1cf9eb83" />
docs/web/integrations/ai/langchain.mdx
+1 -1
View File
@@ -200,7 +200,7 @@ Maximal marginal relevance optimizes for similarity to query AND diversity among
| `fetch_k`| `Optional[int]`| Number of Documents to fetch to pass to MMR algorithm.| `None` |
| `lambda_mult` | `float` | Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. | `0.5` |
| `filter`| `Optional[Dict[str, str]]`| Filter by metadata. | `None` |
|`kwargs`| Other keyworded arguments provided by the user. |-|
| `kwargs` | `Any` | Other keyworded arguments provided by the user. | N/A |
Similarly, `max_marginal_relevance_search_by_vector()` function returns docs most similar to the embedding passed to the function using MMR. instead of a string query you need to pass the embedding to be searched for.
docs/web/integrations/ai/synthetic-data-kit.mdx
+8 -9
View File
@@ -35,23 +35,21 @@ Here is an example of the end-to-end workflow:
1. **Ingest Data into a LanceDB dataset**
This command takes a directory of source files and creates a LanceDB dataset from them.
This command reads a source document and creates a Lance dataset from it.
```bash
synthetic-data-kit ingest docs/report.pdf --multimodal
# This will create a Lance dataset at data/parsed/report.lance
# with 'text' and 'image' columns.
#Generate multimodal-qa pairs from the ingested data
synthetic-data-kit create data/parsed/report.lance --type multimodal-qa
```
2. **Create fine-tuning data**
This command uses the LanceDB dataset created in the previous step to generate synthetic data in the desired format.
This command uses the Lance dataset created in the previous step to generate multimodal question-answer pairs.
```bash
synthetic_data create data/parsed/report.lance
synthetic-data-kit create data/parsed/report.lance --type multimodal-qa
# This writes data/generated/report.json
```
3. **Curate the data**
@@ -59,15 +57,16 @@ Here is an example of the end-to-end workflow:
This step uses a language model to curate the generated data and ensure its quality.
```bash
synthetic_data curate report.json
synthetic-data-kit curate data/generated/report.json
# This writes data/curated/report_cleaned.json
```
4. **Save the final dataset**
Finally, save the curated data to a file in the desired format.
Finally, save the curated data in the format you need: `jsonl`, `alpaca`, `ft` or `chatml`.
```bash
synthetic_data save-as report.json --save_path ./my_finetuning_data.jsonl
synthetic-data-kit save-as data/curated/report_cleaned.json -f jsonl -o ./my_finetuning_data.jsonl
```
This workflow allows you to go from a collection of documents to a high-quality, fine-tuning dataset with just a few commands. The use of LanceDB in the background makes the process efficient and scalable.
docs/web/integrations/data/dlt.mdx
+2 -2
View File
@@ -45,7 +45,7 @@ In this example, we will be fetching movie information from the [Open Movie Data
dlt's LanceDB integration also allows you to automatically embed the data during ingestion. Depending on the embedding model chosen, you may need to paste the necessary credentials inside `.dlt/secrets.toml`:
```toml
[sources.rest_api]
api_key = "api_key" # Enter the API key for the OMDb API
api_token = "api_token" # Enter the API key for the OMDb API
[destination.lancedb]
embedding_model_provider = "sentence-transformers"
@@ -91,7 +91,7 @@ In this example, we will be fetching movie information from the [Open Movie Data
6. **Run the pipeline:**
Finally, running the following command will ingest the data into your LanceDB instance.
```sh
python custom_source.py
python rest_api_pipeline.py
```
For more information and advanced usage of dlt's LanceDB integration, read [the dlt documentation](https://dlthub.com/docs/dlt-ecosystem/destinations/lancedb).
docs/web/integrations/data/voxel51.mdx
+3 -2
View File
@@ -105,8 +105,9 @@ Make sure you install torch ([guide here](https://pytorch.org/get-started/locall
{PyPlatformsVoxel51ComputeSimilarity}
</CodeBlock>
!!! note
Running the code above will download the clip model (2.6Gb)
<Note>
Running the code above downloads the CLIP model (2.6 GB).
</Note>
Once the similarity index has been generated, we can query our data in FiftyOne by specifying the `brain_key`:
docs/web/integrations/embedding/cohere.mdx
+7 -12
View File
@@ -37,18 +37,13 @@ Supported parameters (to be passed in `create` method) are:
Cohere supports following input types:
| Input Type | Description |
|-------------------------|---------------------------------------|
| "`search_document`" | Used for embeddings stored in a vector|
| | database for search use-cases. |
| "`search_query`" | Used for embeddings of search queries |
| | run against a vector DB |
| "`semantic_similarity`" | Specifies the given text will be used |
| | for Semantic Textual Similarity (STS) |
| "`classification`" | Used for embeddings passed through a |
| | text classifier. |
| "`clustering`" | Used for the embeddings run through a |
| | clustering algorithm |
| Input type | Description |
| --- | --- |
| `search_document` | Used for embeddings stored in a vector database for search use cases. |
| `search_query` | Used for embeddings of search queries run against a vector database. |
| `semantic_similarity` | Specifies that the text will be used for semantic textual similarity (STS). |
| `classification` | Used for embeddings passed through a text classifier. |
| `clustering` | Used for embeddings run through a clustering algorithm. |
Usage Example:
docs/web/integrations/embedding/gemini.mdx
+5 -5
View File
@@ -10,11 +10,11 @@ The Gemini Embedding Model API supports various task types:
| Task Type | Description |
|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
| "`retrieval_query`" | Specifies the given text is a query in a search/retrieval setting. |
| "`retrieval_document`" | Specifies the given text is a document in a search/retrieval setting. Using this task type requires a title but is automatically provided by Embeddings API |
| "`semantic_similarity`" | Specifies the given text will be used for Semantic Textual Similarity (STS). |
| "`classification`" | Specifies that the embeddings will be used for classification. |
| "`clustering`" | Specifies that the embeddings will be used for clustering. |
| `retrieval_query` | Specifies the given text is a query in a search/retrieval setting. |
| `retrieval_document` | Specifies the given text is a document in a search/retrieval setting. Using this task type requires a title but is automatically provided by Embeddings API |
| `semantic_similarity` | Specifies the given text will be used for Semantic Textual Similarity (STS). |
| `classification` | Specifies that the embeddings will be used for classification. |
| `clustering` | Specifies that the embeddings will be used for clustering. |
Usage Example:
docs/web/integrations/embedding/ibm.mdx
+1 -2
View File
@@ -17,10 +17,9 @@ You can find a list of supported models at [IBM watsonx.ai Documentation](https:
| `ibm/slate-125m-english-rtrvr-v2` | 768 |
| `ibm/slate-30m-english-rtrvr-v2` | 384 |
| `intfloat/multilingual-e5-large` | 1024 |
| `sentence-transformers/all-minilm-l6-v2` | 384 |
<Info>
For new tables, `ibm/granite-embedding-278m-multilingual` is the recommended default. Older model IDs (such as `ibm/slate-125m-english-rtrvr` and `sentence-transformers/all-minilm-l12-v2`) remain resolvable for tables whose stored metadata references them, but they are no longer advertised for new use.
For new tables, pass `name="ibm/granite-embedding-278m-multilingual"`. The default `name` is still the legacy `ibm/slate-125m-english-rtrvr`, so tables created with it reload with the same model. Legacy and deprecated IDs (such as `ibm/slate-125m-english-rtrvr`, `sentence-transformers/all-minilm-l6-v2` and `sentence-transformers/all-minilm-l12-v2`) still work for tables whose stored metadata references them, but they aren't listed above for new use.
</Info>
## Parameters {#parameters}
docs/web/integrations/embedding/nvidia.mdx
+11 -8
View File
@@ -80,6 +80,8 @@ func = get_registry().get("nvidia").create(
)
```
### Authenticated endpoints {#authenticated-endpoints}
Custom endpoints may run without a key. `NVIDIA_API_KEY` is only sent to the
hosted API, so for authenticated endpoints use
`get_registry().set_var("nvidia-key", key)` with `api_key="$var:nvidia-key"`. Registry references, rather than secret values, are
@@ -87,20 +89,21 @@ stored in the schema. Restore registry variables in each process. Before using
a custom endpoint restored from table metadata in a new process, call
`get_registry().allow_endpoint("http://localhost:8000/v1")`.
### Output dimensions {#output-dimensions}
The NIM 2.4 support matrix lists `nvidia/Nemotron-3.5-Embed-8B-BF16` at 2048
dimensions, whereas the hosted 8B reference specifies 4096. For self-hosted model
IDs, this provider requires an explicit `dim` and checks the response instead of
assuming the hosted defaults.
SDK coverage: Python. TypeScript, Rust, and Java do not gain a NVIDIA provider
from this integration. Automated tests use mocked NVIDIA and Enterprise HTTP
responses plus real local tables. The opt-in hosted smoke test runs with
`NVIDIA_API_KEY` and `pytest python/tests/test_nvidia_embeddings.py -m slow -k nvidia_live`
from the repository's Python environment. It sends five small requests per model
(ten total), with retries disabled, to check local ingestion, reopening, search,
and reduced-dimensional embeddings.
## SDK support {#sdk-support}
NVIDIA references (checked October 23, 2026):
The NVIDIA provider is available in the Python SDK only. TypeScript, Rust and Java
don't have an NVIDIA embedding provider.
## References {#references}
NVIDIA documentation:
[hosted 1B API](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3_5-embed-1b-infer),
[hosted 8B API](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3_5-embed-8b-infer),
[NIM request/response format](https://docs.nvidia.com/nim/nemo-retriever/embedding/2.4/use-the-api-openai.html),
docs/web/integrations/reranking/answerdotai.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ This integration uses [AnswersDotAI's rerankers](https://github.com/AnswerDotAI/
| `model_type` | `str` | `"colbert"` | The type of model to use. Supported model types can be found here: https://github.com/AnswerDotAI/rerankers. |
| `model_name` | `str` | `"answerdotai/answerai-colbert-small-v1"` | The name of the reranker model to use. |
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
docs/web/integrations/reranking/cohere.mdx
+1 -1
View File
@@ -33,7 +33,7 @@ Replace the credential placeholders with your provider credentials. In applicati
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `top_n` | `str` | `None` | The number of results to return. If None, will return all results. |
| `api_key` | `str` | `None` | The API key for the Cohere API. If not provided, the `COHERE_API_KEY` environment variable is used. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type |
docs/web/integrations/reranking/colbert.mdx
+1 -1
View File
@@ -25,7 +25,7 @@ This reranker uses ColBERT model to rerank the search results. You can use this
| `model_name` | `str` | `"colbert-ir/colbertv2.0"` | The name of the reranker model to use.|
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `device` | `str` | `None` | The device to use for the cross encoder model. If None, will use "cuda" if available, otherwise "cpu". |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
## Supported Scores for each query type {#supported-scores-for-each-query-type}
docs/web/integrations/reranking/jina.mdx
+1 -1
View File
@@ -30,7 +30,7 @@ Replace the credential placeholders with your provider credentials. In applicati
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `top_n` | `str` | `None` | The number of results to return. If None, will return all results. |
| `api_key` | `str` | `None` | The API key for the Jina API. If not provided, the `JINA_API_KEY` environment variable is used. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
docs/web/integrations/reranking/openai.mdx
+2 -2
View File
@@ -9,7 +9,7 @@ import { PyRerankingOpenaiUsage } from '/snippets/provider_examples.mdx';
# OpenAI Reranker (Experimental)
This reranker uses OpenAI chat model to rerank the search results. You can use this reranker by passing `OpenAI()` to the `rerank()` method.
This reranker uses OpenAI chat model to rerank the search results. You can use this reranker by passing `OpenaiReranker()` to the `rerank()` method.
> **Note:** Supported query types: Hybrid, Vector, and FTS.
> **Warning:** This reranker is experimental. OpenAI does not have a dedicated reranking model, so it uses a chat model under the hood.
@@ -28,7 +28,7 @@ credentials from environment variables or a secrets manager.
| --- | --- | --- | --- |
| `model_name` | `str` | `"gpt-5.6-sol"` | The name of the reranker model to use.|
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
| `api_key` | `str` | `None` | The API key to use. If None, will use the OPENAI_API_KEY environment variable.
docs/web/integrations/reranking/voyageai.mdx
+1 -1
View File
@@ -33,7 +33,7 @@ credentials from environment variables or a secrets manager.
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `top_n` | `str` | `None` | The number of results to return. If None, will return all results. |
| `api_key` | `str` | `None` | The API key for the Voyage AI API. If not provided, the `VOYAGE_API_KEY` environment variable is used. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type |
| `truncation` | `bool` | `None` | Whether to truncate the input to satisfy the "context length limit" on the query and the documents. |
docs/web/search/vector-search.mdx
+11 -12
View File
@@ -317,9 +317,6 @@ the filter condition after obtaining the nearest neighbors based on vector simil
Use multivector search when your documents contain multiple embeddings and you need sophisticated matching between query and document vector pairs. The late interaction approach finds the most relevant combinations across all available embeddings and provides nuanced similarity scoring.
Only `cosine` similarity is supported as the distance metric for multivector search operations.
Every query vector must match the inner dimension of the multivector column.
<CodeGroup>
<CodeBlock filename="Python" language="python" icon="python">
{MultivectorSearch}
@@ -328,7 +325,8 @@ Every query vector must match the inner dimension of the multivector column.
Here you can see how to take 2 query vectors and find the best matching pairs between them and document vectors using late interaction. The `np.random.random(size=(2, 256))` creates a 2×256 array with two random query vectors, `.limit(5)` returns the top 5 best document-query combinations, and `.to_pandas()` provides results in a DataFrame format.
**Read more:** [Multivector search walkthrough](#multivector)
For the supported metric, how scoring works and a complete example, see the
[multivector search walkthrough](#multivector).
## Advanced Search Scenarios
@@ -495,7 +493,7 @@ This approach is particularly useful when:
- Calculating recall metrics to tune index parameters
- Ensuring exact results for critical applications
## Multi-vector search {#multivector}
## Multivector search walkthrough {#multivector}
LanceDB's multivector support enables you to store and search multiple vector embeddings for a single item.
@@ -539,7 +537,7 @@ $Q$ represents the query embeddings, and $D = \{d_1, d_2, ..., d_{|D|}\}$ repres
#### 1. Setup
Connect to LanceDB and import the required libraries.
Connect to a local LanceDB database and import the required libraries.
<CodeGroup>
```python Python icon="python"
@@ -547,14 +545,16 @@ import lancedb
import numpy as np
import pyarrow as pa
db = lancedb.connect(
uri="db://your-project-slug",
api_key="your-api-key",
region="your-region"
)
db = lancedb.connect("data/multivector_demo")
```
</CodeGroup>
<Note>
To run this walkthrough on LanceDB Enterprise, connect with
`lancedb.connect(uri="db://your-database", api_key="your-api-key", region="your-region")`
instead. The remaining steps are the same.
</Note>
#### 2. Define Schema
Define a schema that specifies a multivector field. A multivector field is a nested list structure in which each document contains multiple vectors. In this case, we'll create a schema with:
@@ -567,7 +567,6 @@ Define a schema that specifies a multivector field. A multivector field is a nes
<CodeGroup>
```python Python icon="python"
db = lancedb.connect("data/multivector_demo")
schema = pa.schema(
[
pa.field("id", pa.int64()),
docs/web/sql/ddl/job.mdx
+1 -1
View File
@@ -36,7 +36,7 @@ SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100
Why rows failed. Reach for this when a [refresh](/sql/dml/refresh-column) reports
fewer filled rows than you expected.
**Errors and limits**
### Errors and limits {#errors-and-limits}
| | |
| --- | --- |
docs/web/sql/ddl/table.mdx
+1 -1
View File
@@ -182,7 +182,7 @@ folded back into the table.
rows a sample would draw from are not all in the fragments yet.
</Warning>
**Errors and limits**
### Errors and limits {#errors-and-limits}
| | |
| --- | --- |
docs/web/sql/dql/dedupe.mdx
+1 -1
View File
@@ -70,7 +70,7 @@ SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02)
`resolution` chooses how overlapping pairs group together:
| `resolution` | |
| `resolution` value | How pairs group |
| --- | --- |
| `connected_components` (default) | groups transitively: if a~b and b~c are both close, all three resolve together |
| `direct_representative` | only pairs within the threshold of each other resolve together |
nodejs/lancedb/indices.ts
+1 -1
View File
@@ -816,7 +816,7 @@ export class Index {
* Create a label list index.
*
* LabelList index is a scalar index that can be used on `List<T>` columns to
* support queries with `array_contains_all` and `array_contains_any`
* support queries with `array_has_all` and `array_has_any`
* using an underlying bitmap index.
*/
static labelList() {
python/python/lancedb/index.py
+1 -1
View File
@@ -85,7 +85,7 @@ class LabelList:
"""Describe a LabelList index configuration.
`LabelList` is a scalar index that can be used on `List<T>` columns to
support queries with `array_contains_all` and `array_contains_any`
support queries with `array_has_all` and `array_has_any`
using an underlying bitmap index.
For example, it works with `tags`, `categories`, `keywords`, etc.
rust/lancedb/src/index.rs
+1 -1
View File
@@ -49,7 +49,7 @@ pub enum Index {
Bitmap(BitmapIndexBuilder),
/// [LabelListIndexBuilder] is a scalar index that can be used on `List<T>` columns to
/// support queries with `array_contains_all` and `array_contains_any`
/// support queries with `array_has_all` and `array_has_any`
/// using an underlying bitmap index.
LabelList(LabelListIndexBuilder),
rust/lancedb/src/index/scalar.rs
+1 -1
View File
@@ -45,7 +45,7 @@ pub struct BitmapIndexBuilder {}
/// Builder for LabelList index.
///
/// [LabeListIndexBuilder] is a scalar index that can be used on `List<T>` columns to
/// support queries with `array_contains_all` and `array_contains_any`
/// support queries with `array_has_all` and `array_has_any`
/// using an underlying bitmap index.
///
#[derive(Debug, Clone, Default, serde::Serialize)]