diff --git a/docs/src/js/classes/Index.md b/docs/src/js/classes/Index.md index 7b5033160..e613882e4 100644 --- a/docs/src/js/classes/Index.md +++ b/docs/src/js/classes/Index.md @@ -252,7 +252,7 @@ static labelList(): Index Create a label list index. LabelList index is a scalar index that can be used on `List` columns to -support queries with `array_contains_all` and `array_contains_any` +support queries with `array_has_all` and `array_has_any` using an underlying bitmap index. #### Returns diff --git a/docs/web/basics/agentic-skills.mdx b/docs/web/basics/agentic-skills.mdx index b87574dff..06a126daa 100644 --- a/docs/web/basics/agentic-skills.mdx +++ b/docs/web/basics/agentic-skills.mdx @@ -84,8 +84,8 @@ found and what each target supports. ### Get started with the LanceDB agent plugin {#get-started-with-the-lancedb-agent-plugin} -This tutorial uses the Camelot dataset from the [quickstart](/quickstart), with -a portrait added for each character. Each LanceDB row contains validated +This tutorial uses a small Camelot dataset of characters from Arthurian legend, +with a portrait for each character. Each LanceDB row contains validated metadata and raw JPEG bytes. Text, images, and any embeddings you add later remain in the same table. @@ -166,7 +166,7 @@ embedded retrieval library. A simple prompt like this should work: Use the lancedb plugin to ingest the dataset in `data/` into a LanceDB Embedded table. -# If using enterprise +# If using LanceDB Enterprise Use the lancedb plugin to ingest the dataset in `data/` into a LanceDB Enterprise table using the connection information in `.env`. ``` diff --git a/docs/web/curation/sampling.mdx b/docs/web/curation/sampling.mdx index 82f1c1ecb..0658a2d8a 100644 --- a/docs/web/curation/sampling.mdx +++ b/docs/web/curation/sampling.mdx @@ -5,7 +5,7 @@ description: "Choose a sample for exploration or evaluation and preserve the sel icon: dice keywords: ["sampling", "tablesample", "subset", "eval set", "reproducible", "shuffle"] availability: - embedded: unavailable + embedded: available enterprise: available --- @@ -15,11 +15,14 @@ Sampling selects records from a larger table. Use it to explore unfamiliar data, run a smaller experiment, or prepare an evaluation dataset. Enterprise SQL provides `TABLESAMPLE` for selecting a percentage or number of rows. -This page covers Enterprise SQL sampling and a complete -[Python SDK example](#training) that runs locally. +Sampling works in LanceDB Embedded and LanceDB Enterprise through the +[Python SDK](#training). SQL sampling, the sections marked Enterprise below, needs +LanceDB Enterprise. ## Example data {#example-data} +Enterprise + Connect with an [Enterprise SQL client](/basics/sql). The examples use a small document table and a separate label table. Expand the setup below to create them. @@ -58,6 +61,8 @@ has no label. These cases let you check the effect of each operation. ## Basic sampling {#quick} +Enterprise + A percentage selects a fraction of the source; `ROWS` requests a row count: ```sql @@ -75,6 +80,8 @@ change a production table's write configuration just to try an example. ## Sampling methods {#methods} +Enterprise + Choose a method according to how records should be selected: | Method | Selection | When to use it | @@ -97,6 +104,8 @@ represent rare groups; check that the resulting sample covers the groups you nee ## Repeatable samples {#fixed} +Enterprise + A seed requests a repeatable draw: ```sql @@ -121,6 +130,8 @@ training or evaluation. ## Sampling with joins {#joins} +Enterprise + Sampling both input tables independently changes which pairs can match. For a one-to-one match, independent 10% row samples retain a given pair with about 1% probability, not 10%. diff --git a/docs/web/curation/windowing.mdx b/docs/web/curation/windowing.mdx index 6cb5322d6..7e303ec51 100644 --- a/docs/web/curation/windowing.mdx +++ b/docs/web/curation/windowing.mdx @@ -1,6 +1,6 @@ --- title: "Ranking and deduplication" -sidebarTitle: "Windowing" +sidebarTitle: "Rank and Deduplicate" description: "Choose records with explicit ranking rules and deterministic tie handling." icon: chart-simple keywords: ["window", "deduplication", "row_number", "rank", "train test split", "per class"] diff --git a/docs/web/deployment/storage.mdx b/docs/web/deployment/storage.mdx index 50746d778..31f3521e7 100644 --- a/docs/web/deployment/storage.mdx +++ b/docs/web/deployment/storage.mdx @@ -76,7 +76,7 @@ Use local disk only when you need extremely low latency and are comfortable owni ### File-format choices that interact with the backend {#file-format-choices} -A few `storage_options` keys shape new tables in ways that depend on the backend you picked above. They are documented in full on [below](#new-table-configuration); the architecture-level summary is: +A few `storage_options` keys shape new tables in ways that depend on the backend you picked above. They are documented in full [below](#new-table-configuration); the architecture-level summary is: - `new_table_enable_v2_manifest_paths` matters most on object stores, where opening a table with many versions is dominated by listing cost. Leave it off for backward compatibility with clients older than LanceDB 0.10.0. - `new_table_enable_stable_row_ids` keeps row IDs stable across compaction, delete, and merge. The choice is independent of the backend but affects any system that joins on row ID. diff --git a/docs/web/feature-engineering/add-computed-column.mdx b/docs/web/feature-engineering/add-computed-column.mdx index aecb0acf0..8dd8b3c23 100644 --- a/docs/web/feature-engineering/add-computed-column.mdx +++ b/docs/web/feature-engineering/add-computed-column.mdx @@ -71,21 +71,21 @@ including that you can travel back to before the column existed. `computed` and `transforms` cannot be combined in one call. Reach for `computed` when the table is large enough that you would rather control when the work happens. -Pass `auto_refresh_enabled` and `auto_refresh_interval_seconds` alongside -`computed` to have the server keep the column refreshed instead of calling -[`refresh_column`](/feature-engineering/refresh-column) yourself: +Automatic refresh is on by default for each computed column, and takes effect +when your Enterprise deployment enables its refresh scheduler. Until then, and in +LanceDB Embedded, which never refreshes on a schedule, call +[`refresh_column`](/feature-engineering/refresh-column) yourself. Pass +`auto_refresh_enabled=False` to opt a column out, or +`auto_refresh_interval_seconds` to set how often it refreshes: ```python table.add_columns( computed={"doubled": "x * 2"}, - auto_refresh_enabled=True, auto_refresh_interval_seconds=300, ) ``` -A local table stores these settings but does not act on them -- automatic -scheduling runs on LanceDB Cloud or Enterprise. See -[Automatic refresh](/feature-engineering/refresh-column#automatic) for the +See [Automatic refresh](/feature-engineering/refresh-column#automatic) for the full behavior, including how to change the setting later. An eager `transforms` call looks like this, starting from a small product diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx index 6807998ec..c164645e4 100644 --- a/docs/web/feature-engineering/index.mdx +++ b/docs/web/feature-engineering/index.mdx @@ -133,8 +133,9 @@ result.rows_assigned A backfill over a large table can therefore be started, left, and checked on from a different process: the connection does not have to stay open. -Enterprise manages the compute for the refresh. You still control when to -submit it; declaring a computed column does not schedule recurring refreshes. +Enterprise manages the compute for the refresh. Declaring a computed column +doesn't refresh it on a schedule unless your deployment enables automatic +refresh; see [Automatic refresh](/feature-engineering/refresh-column#automatic). {/* Draft for the Functions release: That is also why a function must declare the environment it needs. */} ## Recovering from a failure {#failures} diff --git a/docs/web/index.mdx b/docs/web/index.mdx index 21c9134af..85ee3de0d 100644 --- a/docs/web/index.mdx +++ b/docs/web/index.mdx @@ -133,7 +133,7 @@ To set up LanceDB Enterprise in your organization, [reach out to us](https://www | | [Scalar indexes](/indexing/scalar-index) | ✅ | ✅ | | **SQL and curation** | [SQL (Arrow Flight SQL)](/basics/sql) | ❌ | ✅ | | | [Aggregation and profiling](/curation/aggregate) | ❌ | ✅ | -| | [Sampling](/curation/sampling) | ❌ | ✅ | +| | [Sampling](/curation/sampling) | ✅ | ✅ | | | [Joins](/curation/join) | ❌ | ✅ | | | [Ranking and deduplication](/curation/windowing) | ❌ | ✅ | | | [Near-duplicate detection](/curation/dedupe) | ❌ | ✅ | diff --git a/docs/web/indexing/index.mdx b/docs/web/indexing/index.mdx index e408acb78..df8109f5b 100644 --- a/docs/web/indexing/index.mdx +++ b/docs/web/indexing/index.mdx @@ -24,7 +24,7 @@ patterns. LanceDB supports three main index families: | `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. | | `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. | | `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. | -| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_contains_all` or `array_contains_any` filters. | Scalar index for `List` and `LargeList` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. | +| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_has_all` or `array_has_any` filters. | Scalar index for `List` and `LargeList` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. | | `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. | A deployment builds the vector indexes `IVF_FLAT`, `IVF_PQ`, `IVF_RQ` and diff --git a/docs/web/indexing/scalar-index.mdx b/docs/web/indexing/scalar-index.mdx index d99bb6003..5c483111b 100644 --- a/docs/web/indexing/scalar-index.mdx +++ b/docs/web/indexing/scalar-index.mdx @@ -30,7 +30,7 @@ LanceDB supports four types of scalar indexes: - `BTREE`: Stores column data in sorted order for binary search. Best for columns with many unique values. - `BITMAP`: Uses bitmaps to track value presence. Ideal for columns with few unique values (e.g., categories, tags). -- `LABEL_LIST`: Special index for `List` and `LargeList` columns of primitive values supporting `array_contains_all` and `array_contains_any` queries. +- `LABEL_LIST`: Special index for `List` and `LargeList` columns of primitive values supporting `array_has_all` and `array_has_any` queries. - `FM`: FM-Index over string or binary columns that accelerates substring search via `contains(col, 'needle')`. @@ -109,22 +109,6 @@ Scalar indexes can also speed up scans containing a vector search or full text s -## Indexing nested fields - -Scalar indexes can target a scalar field inside a struct by passing its full dotted path. The path is preserved end to end: it's the value you pass to `create_scalar_index`, it's what `list_indices()` reports under `columns`, and it's the column reference you use in filter predicates. - -```python -# Schema: pa.struct([pa.field("user_id", pa.int32())]) stored under the `metadata` column. -table.create_scalar_index("metadata.user_id", name="metadata_user_id_idx") - -# The same dotted path works in WHERE clauses. -table.search().where("metadata.user_id = 42").limit(1).to_list() -``` - - -Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `metadata.author.name`). The same convention applies to FTS and vector indexes. - - ## FM-Index for substring search The `FM` index is a scalar index built over string or binary columns that @@ -246,7 +230,9 @@ named `user.id` nested inside `metadata`), wrap that segment in backticks so LanceDB can tell the dot apart from the path separator: `` metadata.`user.id` ``. `list_indices()` echoes the same canonical path back, so the column you pass in -round-trips through index metadata regardless of nesting depth or escaping. +round-trips through index metadata regardless of nesting depth or escaping. Use the +same path in filter predicates, for example `metadata.user_id = 42`. FTS and vector +indexes follow the same path convention. diff --git a/docs/web/indexing/vector-index.mdx b/docs/web/indexing/vector-index.mdx index 3e2b7d6f0..9366a09bc 100644 --- a/docs/web/indexing/vector-index.mdx +++ b/docs/web/indexing/vector-index.mdx @@ -55,7 +55,7 @@ Lance with `index_type`. | :--- | :--- | :--- | :--- | :--- | | Maximum compression | `IVF_RQ` | IVF indexing with RaBitQ quantization | Around `1/32` of raw size with `num_bits=1` | `IvfRq` | | Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | IVF indexing with product quantization | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | `IvfPq` | -| Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar | Typically a little larger than `1/4` of raw size | `IvfHnswSq` | +| Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar quantization | Typically a little larger than `1/4` of raw size | `IvfHnswSq` | | Highest recall / no quantization | `IVF_HNSW_FLAT` | IVF-HNSW indexing with no quantization; embedded library only | Around raw vector size plus HNSW graph overhead | `IvfHnswFlat` | diff --git a/docs/web/integrations/ai/genkit.mdx b/docs/web/integrations/ai/genkit.mdx index 1ccbf834d..fbc3d3382 100644 --- a/docs/web/integrations/ai/genkit.mdx +++ b/docs/web/integrations/ai/genkit.mdx @@ -37,11 +37,15 @@ genkit start -- tsx --watch src/index.ts This'll add LanceDB as a retriever and indexer to the genkit instance. You can see it in the GUI view Screenshot 2025-05-11 at 7 21 05 PM -**Testing retrieval on a sample table** -Let's see the raw retrieval results +### Testing retrieval on a sample table {#testing-retrieval-on-a-sample-table} + +To see the raw retrieval results, open the LanceDB retriever in the Genkit Developer UI, +enter a query and run it: Screenshot 2025-05-11 at 7 21 05 PM -On running this query, you'll get 5 results fetched from the lancedb table, where each result looks something like this: + +The retriever returns 5 results from the LanceDB table. Each result looks something like this: + Screenshot 2025-05-11 at 7 21 18 PM diff --git a/docs/web/integrations/ai/langchain.mdx b/docs/web/integrations/ai/langchain.mdx index 8d012e01e..c87253169 100644 --- a/docs/web/integrations/ai/langchain.mdx +++ b/docs/web/integrations/ai/langchain.mdx @@ -200,7 +200,7 @@ Maximal marginal relevance optimizes for similarity to query AND diversity among | `fetch_k`| `Optional[int]`| Number of Documents to fetch to pass to MMR algorithm.| `None` | | `lambda_mult` | `float` | Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. | `0.5` | | `filter`| `Optional[Dict[str, str]]`| Filter by metadata. | `None` | -|`kwargs`| Other keyworded arguments provided by the user. |-| +| `kwargs` | `Any` | Other keyworded arguments provided by the user. | N/A | Similarly, `max_marginal_relevance_search_by_vector()` function returns docs most similar to the embedding passed to the function using MMR. instead of a string query you need to pass the embedding to be searched for. diff --git a/docs/web/integrations/ai/synthetic-data-kit.mdx b/docs/web/integrations/ai/synthetic-data-kit.mdx index 18b90674a..7055efb80 100644 --- a/docs/web/integrations/ai/synthetic-data-kit.mdx +++ b/docs/web/integrations/ai/synthetic-data-kit.mdx @@ -35,23 +35,21 @@ Here is an example of the end-to-end workflow: 1. **Ingest Data into a LanceDB dataset** - This command takes a directory of source files and creates a LanceDB dataset from them. + This command reads a source document and creates a Lance dataset from it. ```bash synthetic-data-kit ingest docs/report.pdf --multimodal # This will create a Lance dataset at data/parsed/report.lance # with 'text' and 'image' columns. - - #Generate multimodal-qa pairs from the ingested data - synthetic-data-kit create data/parsed/report.lance --type multimodal-qa ``` 2. **Create fine-tuning data** - This command uses the LanceDB dataset created in the previous step to generate synthetic data in the desired format. + This command uses the Lance dataset created in the previous step to generate multimodal question-answer pairs. ```bash - synthetic_data create data/parsed/report.lance + synthetic-data-kit create data/parsed/report.lance --type multimodal-qa + # This writes data/generated/report.json ``` 3. **Curate the data** @@ -59,15 +57,16 @@ Here is an example of the end-to-end workflow: This step uses a language model to curate the generated data and ensure its quality. ```bash - synthetic_data curate report.json + synthetic-data-kit curate data/generated/report.json + # This writes data/curated/report_cleaned.json ``` 4. **Save the final dataset** - Finally, save the curated data to a file in the desired format. + Finally, save the curated data in the format you need: `jsonl`, `alpaca`, `ft` or `chatml`. ```bash - synthetic_data save-as report.json --save_path ./my_finetuning_data.jsonl + synthetic-data-kit save-as data/curated/report_cleaned.json -f jsonl -o ./my_finetuning_data.jsonl ``` This workflow allows you to go from a collection of documents to a high-quality, fine-tuning dataset with just a few commands. The use of LanceDB in the background makes the process efficient and scalable. diff --git a/docs/web/integrations/data/dlt.mdx b/docs/web/integrations/data/dlt.mdx index f5ede936b..96c2479fd 100644 --- a/docs/web/integrations/data/dlt.mdx +++ b/docs/web/integrations/data/dlt.mdx @@ -45,7 +45,7 @@ In this example, we will be fetching movie information from the [Open Movie Data dlt's LanceDB integration also allows you to automatically embed the data during ingestion. Depending on the embedding model chosen, you may need to paste the necessary credentials inside `.dlt/secrets.toml`: ```toml [sources.rest_api] - api_key = "api_key" # Enter the API key for the OMDb API + api_token = "api_token" # Enter the API key for the OMDb API [destination.lancedb] embedding_model_provider = "sentence-transformers" @@ -91,7 +91,7 @@ In this example, we will be fetching movie information from the [Open Movie Data 6. **Run the pipeline:** Finally, running the following command will ingest the data into your LanceDB instance. ```sh - python custom_source.py + python rest_api_pipeline.py ``` For more information and advanced usage of dlt's LanceDB integration, read [the dlt documentation](https://dlthub.com/docs/dlt-ecosystem/destinations/lancedb). diff --git a/docs/web/integrations/data/voxel51.mdx b/docs/web/integrations/data/voxel51.mdx index a04cd0c3e..30fc4728d 100644 --- a/docs/web/integrations/data/voxel51.mdx +++ b/docs/web/integrations/data/voxel51.mdx @@ -105,8 +105,9 @@ Make sure you install torch ([guide here](https://pytorch.org/get-started/locall {PyPlatformsVoxel51ComputeSimilarity} -!!! note - Running the code above will download the clip model (2.6Gb) + +Running the code above downloads the CLIP model (2.6 GB). + Once the similarity index has been generated, we can query our data in FiftyOne by specifying the `brain_key`: diff --git a/docs/web/integrations/embedding/cohere.mdx b/docs/web/integrations/embedding/cohere.mdx index 3c56fec76..68278828c 100644 --- a/docs/web/integrations/embedding/cohere.mdx +++ b/docs/web/integrations/embedding/cohere.mdx @@ -37,18 +37,13 @@ Supported parameters (to be passed in `create` method) are: Cohere supports following input types: -| Input Type | Description | -|-------------------------|---------------------------------------| -| "`search_document`" | Used for embeddings stored in a vector| -| | database for search use-cases. | -| "`search_query`" | Used for embeddings of search queries | -| | run against a vector DB | -| "`semantic_similarity`" | Specifies the given text will be used | -| | for Semantic Textual Similarity (STS) | -| "`classification`" | Used for embeddings passed through a | -| | text classifier. | -| "`clustering`" | Used for the embeddings run through a | -| | clustering algorithm | +| Input type | Description | +| --- | --- | +| `search_document` | Used for embeddings stored in a vector database for search use cases. | +| `search_query` | Used for embeddings of search queries run against a vector database. | +| `semantic_similarity` | Specifies that the text will be used for semantic textual similarity (STS). | +| `classification` | Used for embeddings passed through a text classifier. | +| `clustering` | Used for embeddings run through a clustering algorithm. | Usage Example: diff --git a/docs/web/integrations/embedding/gemini.mdx b/docs/web/integrations/embedding/gemini.mdx index e3359ec0d..532969122 100644 --- a/docs/web/integrations/embedding/gemini.mdx +++ b/docs/web/integrations/embedding/gemini.mdx @@ -10,11 +10,11 @@ The Gemini Embedding Model API supports various task types: | Task Type | Description | |-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------| -| "`retrieval_query`" | Specifies the given text is a query in a search/retrieval setting. | -| "`retrieval_document`" | Specifies the given text is a document in a search/retrieval setting. Using this task type requires a title but is automatically provided by Embeddings API | -| "`semantic_similarity`" | Specifies the given text will be used for Semantic Textual Similarity (STS). | -| "`classification`" | Specifies that the embeddings will be used for classification. | -| "`clustering`" | Specifies that the embeddings will be used for clustering. | +| `retrieval_query` | Specifies the given text is a query in a search/retrieval setting. | +| `retrieval_document` | Specifies the given text is a document in a search/retrieval setting. Using this task type requires a title but is automatically provided by Embeddings API | +| `semantic_similarity` | Specifies the given text will be used for Semantic Textual Similarity (STS). | +| `classification` | Specifies that the embeddings will be used for classification. | +| `clustering` | Specifies that the embeddings will be used for clustering. | Usage Example: diff --git a/docs/web/integrations/embedding/ibm.mdx b/docs/web/integrations/embedding/ibm.mdx index 6dd3d57e3..66a2128d9 100644 --- a/docs/web/integrations/embedding/ibm.mdx +++ b/docs/web/integrations/embedding/ibm.mdx @@ -17,10 +17,9 @@ You can find a list of supported models at [IBM watsonx.ai Documentation](https: | `ibm/slate-125m-english-rtrvr-v2` | 768 | | `ibm/slate-30m-english-rtrvr-v2` | 384 | | `intfloat/multilingual-e5-large` | 1024 | -| `sentence-transformers/all-minilm-l6-v2` | 384 | -For new tables, `ibm/granite-embedding-278m-multilingual` is the recommended default. Older model IDs (such as `ibm/slate-125m-english-rtrvr` and `sentence-transformers/all-minilm-l12-v2`) remain resolvable for tables whose stored metadata references them, but they are no longer advertised for new use. +For new tables, pass `name="ibm/granite-embedding-278m-multilingual"`. The default `name` is still the legacy `ibm/slate-125m-english-rtrvr`, so tables created with it reload with the same model. Legacy and deprecated IDs (such as `ibm/slate-125m-english-rtrvr`, `sentence-transformers/all-minilm-l6-v2` and `sentence-transformers/all-minilm-l12-v2`) still work for tables whose stored metadata references them, but they aren't listed above for new use. ## Parameters {#parameters} diff --git a/docs/web/integrations/embedding/nvidia.mdx b/docs/web/integrations/embedding/nvidia.mdx index c0664684a..00265aa5f 100644 --- a/docs/web/integrations/embedding/nvidia.mdx +++ b/docs/web/integrations/embedding/nvidia.mdx @@ -80,6 +80,8 @@ func = get_registry().get("nvidia").create( ) ``` +### Authenticated endpoints {#authenticated-endpoints} + Custom endpoints may run without a key. `NVIDIA_API_KEY` is only sent to the hosted API, so for authenticated endpoints use `get_registry().set_var("nvidia-key", key)` with `api_key="$var:nvidia-key"`. Registry references, rather than secret values, are @@ -87,20 +89,21 @@ stored in the schema. Restore registry variables in each process. Before using a custom endpoint restored from table metadata in a new process, call `get_registry().allow_endpoint("http://localhost:8000/v1")`. +### Output dimensions {#output-dimensions} + The NIM 2.4 support matrix lists `nvidia/Nemotron-3.5-Embed-8B-BF16` at 2048 dimensions, whereas the hosted 8B reference specifies 4096. For self-hosted model IDs, this provider requires an explicit `dim` and checks the response instead of assuming the hosted defaults. -SDK coverage: Python. TypeScript, Rust, and Java do not gain a NVIDIA provider -from this integration. Automated tests use mocked NVIDIA and Enterprise HTTP -responses plus real local tables. The opt-in hosted smoke test runs with -`NVIDIA_API_KEY` and `pytest python/tests/test_nvidia_embeddings.py -m slow -k nvidia_live` -from the repository's Python environment. It sends five small requests per model -(ten total), with retries disabled, to check local ingestion, reopening, search, -and reduced-dimensional embeddings. +## SDK support {#sdk-support} -NVIDIA references (checked October 23, 2026): +The NVIDIA provider is available in the Python SDK only. TypeScript, Rust and Java +don't have an NVIDIA embedding provider. + +## References {#references} + +NVIDIA documentation: [hosted 1B API](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3_5-embed-1b-infer), [hosted 8B API](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3_5-embed-8b-infer), [NIM request/response format](https://docs.nvidia.com/nim/nemo-retriever/embedding/2.4/use-the-api-openai.html), diff --git a/docs/web/integrations/reranking/answerdotai.mdx b/docs/web/integrations/reranking/answerdotai.mdx index 22814e822..9d2b6f0f6 100644 --- a/docs/web/integrations/reranking/answerdotai.mdx +++ b/docs/web/integrations/reranking/answerdotai.mdx @@ -26,7 +26,7 @@ This integration uses [AnswersDotAI's rerankers](https://github.com/AnswerDotAI/ | `model_type` | `str` | `"colbert"` | The type of model to use. Supported model types can be found here: https://github.com/AnswerDotAI/rerankers. | | `model_name` | `str` | `"answerdotai/answerai-colbert-small-v1"` | The name of the reranker model to use. | | `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. | -| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. | +| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. | diff --git a/docs/web/integrations/reranking/cohere.mdx b/docs/web/integrations/reranking/cohere.mdx index bc731b39e..b79420c92 100644 --- a/docs/web/integrations/reranking/cohere.mdx +++ b/docs/web/integrations/reranking/cohere.mdx @@ -33,7 +33,7 @@ Replace the credential placeholders with your provider credentials. In applicati | `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. | | `top_n` | `str` | `None` | The number of results to return. If None, will return all results. | | `api_key` | `str` | `None` | The API key for the Cohere API. If not provided, the `COHERE_API_KEY` environment variable is used. | -| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type | +| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type | diff --git a/docs/web/integrations/reranking/colbert.mdx b/docs/web/integrations/reranking/colbert.mdx index ffb3f8d2c..fa369d392 100644 --- a/docs/web/integrations/reranking/colbert.mdx +++ b/docs/web/integrations/reranking/colbert.mdx @@ -25,7 +25,7 @@ This reranker uses ColBERT model to rerank the search results. You can use this | `model_name` | `str` | `"colbert-ir/colbertv2.0"` | The name of the reranker model to use.| | `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. | | `device` | `str` | `None` | The device to use for the cross encoder model. If None, will use "cuda" if available, otherwise "cpu". | -| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. | +| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. | ## Supported Scores for each query type {#supported-scores-for-each-query-type} diff --git a/docs/web/integrations/reranking/jina.mdx b/docs/web/integrations/reranking/jina.mdx index c2b1db6db..bf7f2d4da 100644 --- a/docs/web/integrations/reranking/jina.mdx +++ b/docs/web/integrations/reranking/jina.mdx @@ -30,7 +30,7 @@ Replace the credential placeholders with your provider credentials. In applicati | `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. | | `top_n` | `str` | `None` | The number of results to return. If None, will return all results. | | `api_key` | `str` | `None` | The API key for the Jina API. If not provided, the `JINA_API_KEY` environment variable is used. | -| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. | +| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. | diff --git a/docs/web/integrations/reranking/openai.mdx b/docs/web/integrations/reranking/openai.mdx index 71fe63117..0d7a56de7 100644 --- a/docs/web/integrations/reranking/openai.mdx +++ b/docs/web/integrations/reranking/openai.mdx @@ -9,7 +9,7 @@ import { PyRerankingOpenaiUsage } from '/snippets/provider_examples.mdx'; # OpenAI Reranker (Experimental) -This reranker uses OpenAI chat model to rerank the search results. You can use this reranker by passing `OpenAI()` to the `rerank()` method. +This reranker uses OpenAI chat model to rerank the search results. You can use this reranker by passing `OpenaiReranker()` to the `rerank()` method. > **Note:** Supported query types: Hybrid, Vector, and FTS. > **Warning:** This reranker is experimental. OpenAI does not have a dedicated reranking model, so it uses a chat model under the hood. @@ -28,7 +28,7 @@ credentials from environment variables or a secrets manager. | --- | --- | --- | --- | | `model_name` | `str` | `"gpt-5.6-sol"` | The name of the reranker model to use.| | `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. | -| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. | +| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. | | `api_key` | `str` | `None` | The API key to use. If None, will use the OPENAI_API_KEY environment variable. diff --git a/docs/web/integrations/reranking/voyageai.mdx b/docs/web/integrations/reranking/voyageai.mdx index eaf2d83f8..18a59ec94 100644 --- a/docs/web/integrations/reranking/voyageai.mdx +++ b/docs/web/integrations/reranking/voyageai.mdx @@ -33,7 +33,7 @@ credentials from environment variables or a secrets manager. | `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. | | `top_n` | `str` | `None` | The number of results to return. If None, will return all results. | | `api_key` | `str` | `None` | The API key for the Voyage AI API. If not provided, the `VOYAGE_API_KEY` environment variable is used. | -| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type | +| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score` column. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type | | `truncation` | `bool` | `None` | Whether to truncate the input to satisfy the "context length limit" on the query and the documents. | diff --git a/docs/web/search/vector-search.mdx b/docs/web/search/vector-search.mdx index 8c43fd0e6..eeaff631c 100644 --- a/docs/web/search/vector-search.mdx +++ b/docs/web/search/vector-search.mdx @@ -317,9 +317,6 @@ the filter condition after obtaining the nearest neighbors based on vector simil Use multivector search when your documents contain multiple embeddings and you need sophisticated matching between query and document vector pairs. The late interaction approach finds the most relevant combinations across all available embeddings and provides nuanced similarity scoring. -Only `cosine` similarity is supported as the distance metric for multivector search operations. -Every query vector must match the inner dimension of the multivector column. - {MultivectorSearch} @@ -328,7 +325,8 @@ Every query vector must match the inner dimension of the multivector column. Here you can see how to take 2 query vectors and find the best matching pairs between them and document vectors using late interaction. The `np.random.random(size=(2, 256))` creates a 2×256 array with two random query vectors, `.limit(5)` returns the top 5 best document-query combinations, and `.to_pandas()` provides results in a DataFrame format. -**Read more:** [Multivector search walkthrough](#multivector) +For the supported metric, how scoring works and a complete example, see the +[multivector search walkthrough](#multivector). ## Advanced Search Scenarios @@ -495,7 +493,7 @@ This approach is particularly useful when: - Calculating recall metrics to tune index parameters - Ensuring exact results for critical applications -## Multi-vector search {#multivector} +## Multivector search walkthrough {#multivector} LanceDB's multivector support enables you to store and search multiple vector embeddings for a single item. @@ -539,7 +537,7 @@ $Q$ represents the query embeddings, and $D = \{d_1, d_2, ..., d_{|D|}\}$ repres #### 1. Setup -Connect to LanceDB and import the required libraries. +Connect to a local LanceDB database and import the required libraries. ```python Python icon="python" @@ -547,14 +545,16 @@ import lancedb import numpy as np import pyarrow as pa -db = lancedb.connect( - uri="db://your-project-slug", - api_key="your-api-key", - region="your-region" -) +db = lancedb.connect("data/multivector_demo") ``` + +To run this walkthrough on LanceDB Enterprise, connect with +`lancedb.connect(uri="db://your-database", api_key="your-api-key", region="your-region")` +instead. The remaining steps are the same. + + #### 2. Define Schema Define a schema that specifies a multivector field. A multivector field is a nested list structure in which each document contains multiple vectors. In this case, we'll create a schema with: @@ -567,7 +567,6 @@ Define a schema that specifies a multivector field. A multivector field is a nes ```python Python icon="python" -db = lancedb.connect("data/multivector_demo") schema = pa.schema( [ pa.field("id", pa.int64()), diff --git a/docs/web/sql/ddl/job.mdx b/docs/web/sql/ddl/job.mdx index 1b66e1e29..6ba177564 100644 --- a/docs/web/sql/ddl/job.mdx +++ b/docs/web/sql/ddl/job.mdx @@ -36,7 +36,7 @@ SHOW ERRORS ON documents FOR JOB 'job-abc' LIMIT 100 Why rows failed. Reach for this when a [refresh](/sql/dml/refresh-column) reports fewer filled rows than you expected. -**Errors and limits** +### Errors and limits {#errors-and-limits} | | | | --- | --- | diff --git a/docs/web/sql/ddl/table.mdx b/docs/web/sql/ddl/table.mdx index b1a5e8f1e..4becafcdc 100644 --- a/docs/web/sql/ddl/table.mdx +++ b/docs/web/sql/ddl/table.mdx @@ -182,7 +182,7 @@ folded back into the table. rows a sample would draw from are not all in the fragments yet. -**Errors and limits** +### Errors and limits {#errors-and-limits} | | | | --- | --- | diff --git a/docs/web/sql/dql/dedupe.mdx b/docs/web/sql/dql/dedupe.mdx index c82e72c13..eff3db950 100644 --- a/docs/web/sql/dql/dedupe.mdx +++ b/docs/web/sql/dql/dedupe.mdx @@ -70,7 +70,7 @@ SELECT * FROM dedupe('frames', 'vec', distance_threshold => 0.02) `resolution` chooses how overlapping pairs group together: -| `resolution` | | +| `resolution` value | How pairs group | | --- | --- | | `connected_components` (default) | groups transitively: if a~b and b~c are both close, all three resolve together | | `direct_representative` | only pairs within the threshold of each other resolve together | diff --git a/nodejs/lancedb/indices.ts b/nodejs/lancedb/indices.ts index 4b1b422bf..ea64fc712 100644 --- a/nodejs/lancedb/indices.ts +++ b/nodejs/lancedb/indices.ts @@ -816,7 +816,7 @@ export class Index { * Create a label list index. * * LabelList index is a scalar index that can be used on `List` columns to - * support queries with `array_contains_all` and `array_contains_any` + * support queries with `array_has_all` and `array_has_any` * using an underlying bitmap index. */ static labelList() { diff --git a/python/python/lancedb/index.py b/python/python/lancedb/index.py index 8c192a2f0..01c8ea3f5 100644 --- a/python/python/lancedb/index.py +++ b/python/python/lancedb/index.py @@ -85,7 +85,7 @@ class LabelList: """Describe a LabelList index configuration. `LabelList` is a scalar index that can be used on `List` columns to - support queries with `array_contains_all` and `array_contains_any` + support queries with `array_has_all` and `array_has_any` using an underlying bitmap index. For example, it works with `tags`, `categories`, `keywords`, etc. diff --git a/rust/lancedb/src/index.rs b/rust/lancedb/src/index.rs index e9056863a..c34ed1407 100644 --- a/rust/lancedb/src/index.rs +++ b/rust/lancedb/src/index.rs @@ -49,7 +49,7 @@ pub enum Index { Bitmap(BitmapIndexBuilder), /// [LabelListIndexBuilder] is a scalar index that can be used on `List` columns to - /// support queries with `array_contains_all` and `array_contains_any` + /// support queries with `array_has_all` and `array_has_any` /// using an underlying bitmap index. LabelList(LabelListIndexBuilder), diff --git a/rust/lancedb/src/index/scalar.rs b/rust/lancedb/src/index/scalar.rs index 5d2256381..2555c1439 100644 --- a/rust/lancedb/src/index/scalar.rs +++ b/rust/lancedb/src/index/scalar.rs @@ -45,7 +45,7 @@ pub struct BitmapIndexBuilder {} /// Builder for LabelList index. /// /// [LabeListIndexBuilder] is a scalar index that can be used on `List` columns to -/// support queries with `array_contains_all` and `array_contains_any` +/// support queries with `array_has_all` and `array_has_any` /// using an underlying bitmap index. /// #[derive(Debug, Clone, Default, serde::Serialize)]