rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-08-18 03:58:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(python): fill gaps in the Python API reference (#3746)

Browse Source 
`docs/src/python/python.md` is the whole Python API reference, but it is
maintained by hand and had drifted from the public API. Anything not
listed there simply doesn't get rendered, so a number of public,
documented, tested APIs were invisible to users — most notably branch
management, where `diff` and `merge` live.

I audited every public symbol reachable from `lancedb` and its
subpackages against the `:::` directives on the page. This adds the
missing ones:

- **Branching** — `Branches`, `AsyncBranches` (`list` / `create` /
`checkout` / `delete` / `diff` / `merge`)
- **Tables** — `TableStatistics` (returned by `Table.stats()`; the
fragment-level stats classes were already listed)
- **Full text queries** — `FullTextQuery`, `MatchQuery`, `PhraseQuery`,
`BoostQuery`, `MultiMatchQuery`, `BooleanQuery`, `FullTextOperator`,
`Occur`
- **Querying** — `LanceEmptyQueryBuilder`, `LanceTakeQueryBuilder`,
`AsyncTakeQuery`
- **Indices** — `Fm` (the FM-index for substring search), `IndexConfig`
- **Blobs** — `blob`, `BlobType`, `BlobFile`
- **Namespaces** — `connect_namespace`, `connect_namespace_async`, and
both namespace connection classes
- **Remote config** — `TlsConfig`, `HeaderProvider`, `OAuthConfig`,
`OAuthFlowType`
- **Rerankers** — the `Reranker` base class plus `JinaReranker`,
`RRFReranker`, `MRRReranker`, `AnswerdotaiRerankers`,
`VoyageAIReranker`, `WatsonxReranker` (5 of 12 were listed)
- **Embeddings** — `get_registry`, `register`, and the 14 embedding
functions that were missing (3 of 17 were listed)
- **PyTorch** — `StreamingDataset` and the permutation API it is built
on
- **Misc** — `Session`, `tokenize`, `FtsToken`, `pydantic.Vector`,
`pydantic.MultiVector`, `instrument_lancedb_metrics`, and the two
exception types

It also repairs cross-references in docstrings that no longer resolve:
links into guide pages that have since moved to lancedb.com
(`querying-an-ann-index`, `experimental-full-text-search`),
`lance.dataset` references with no inventory behind them, and the
relative targets `[Table](Table)` and `[PyArrow Table](pyarrow.Table)`.

Deliberately left out: concrete implementation classes reached through
their abstract base (`LanceTable`, `LanceDBConnection`,
`RemoteDBConnection`), query base classes already covered by
`inherited_members: true`, and internal plumbing such as
`FullTextSearchQuery` and `ColumnOrdering`.

## Testing

The docs job only runs on pushes to `main`, so I built the site locally
and compared against a build of `upstream/main`: every added entry
resolves, and no symbol that was rendered before stopped being rendered
when the four packages moved to automodule. `mkdocs build --strict`
exits 0 on this branch, against 61 warnings on `main`.

## Also in this PR

`lancedb.index`, `lancedb.embeddings`, `lancedb.remote` and
`lancedb.rerankers` are now rendered by a single mkdocstrings directive
each, driven by the module's `__all__`, rather than a hand-maintained
list. These four are where most of the drift was, and `__all__` is
harder to forget than a docs page. `lancedb.embeddings` had no
`__all__`; without one mkdocstrings renders no members at all for a
re-export package, so one is added. AGENTS.md gains a section on how the
page is wired up and how to build the docs locally.

Rendering all that code for the first time surfaced ~100 more build
warnings, which would have made #3707 (turning on `mkdocs build
--strict`) harder to land, so the warning backlog is cleared here too.
97 of the 158 warnings were one systematic false positive — griffe
cannot see the generated `__init__` of a pydantic dataclass, so every
documented parameter looks unknown — switched off via
`warn_unknown_params`. The remaining 61 came from 15 docstrings with
real bugs: prose trailing a `Parameters` section (we were rendering
parameters called `The`, `you` and `To`), types dropped because numpydoc
needs spaces around the colon, `num_partitions, default sqrt(num_rows)`
parsing as a list of names and inventing a `default` parameter, and one
parameter indented five spaces. `mkdocs build --strict` now exits 0.

---

#3747 (the coverage test that keeps this from happening again) is
stacked on this branch, so review it after this one.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Will Jones
2026-07-30 16:50:05 -07:00
committed by GitHub
parent 48945d0658
commit 5a1015ba72
24 changed files with 309 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+29
View File
@@ -92,6 +92,8 @@ Python bindings changes:
* Should use `LOOP.run()` to call the corresponding `AsyncTable` method.
6. Add concrete sync method to `RemoteTable` class in `python/python/lancedb/remote/table.py`.
7. Add unit test in `python/tests/test_table.py`.
8. If you added a new public class or module-level function (not just a method on an
existing class), expose it in the API reference. See "Python API reference" below.
TypeScript bindings changes:
@@ -103,6 +105,33 @@ TypeScript bindings changes:
5. Add test in `nodejs/__test__/table.test.ts`.
6. Run `npm run docs` to generate TypeScript documentation.
## Python API reference
`docs/src/python/python.md` is the entire Python API reference. It is maintained by
hand, and anything not listed there is not rendered at all, so new public classes and
module-level functions have to be added explicitly. How depends on the module:
* `lancedb.index`, `lancedb.embeddings`, `lancedb.remote`, and `lancedb.rerankers` are
rendered by a single directive each, driven by the module's `__all__`. Add the new
name to `__all__` and it appears; forget, and it is silently omitted.
* Everything else (`lancedb`, `lancedb.table`, `lancedb.query`, `lancedb.db`, ...) is
listed symbol by symbol. Add a `::: lancedb.<module>.<Name>` line to the matching
section, and remember that the page separates synchronous and asynchronous APIs.
Deliberately undocumented: concrete implementations reached through an abstract base
(`LanceTable`, `LanceDBConnection`, `RemoteDBConnection`), query base classes already
covered by `inherited_members`, and internal helpers.
Cross-references in docstrings use mkdocstrings syntax, `[text][lancedb.table.Table]`.
Plain relative links such as `[Table](Table)` do not resolve. To check your work:
```shell
pip install -r docs/requirements.txt
cd docs && PYTHONPATH=. mkdocs build
```
The docs site only builds on pushes to `main`, so this is not covered by PR CI.
## Review Guidelines
Please consider the following when reviewing code contributions.
docs/mkdocs.yml
+5
View File
@@ -51,6 +51,11 @@ plugins:
paths: [../python/python]
options:
docstring_style: numpy
docstring_options:
# Attributes documented in a `Parameters` section, and pydantic
# dataclasses whose `__init__` griffe cannot see statically, both
# trip this check. It reports nothing actionable here.
warn_unknown_params: false
heading_level: 3
show_signature_annotations: true
show_root_heading: true
docs/src/js/_media/CONTRIBUTING.md
+1 -1
View File
@@ -1,7 +1,7 @@
# Contributing to LanceDB Typescript
This document outlines the process for contributing to LanceDB Typescript.
For general contribution guidelines, see [CONTRIBUTING.md](../CONTRIBUTING.md).
For general contribution guidelines, see [CONTRIBUTING.md](https://github.com/lancedb/lancedb/blob/main/CONTRIBUTING.md).
## Project layout
docs/src/python/python.md
+114 -49
View File
@@ -26,6 +26,18 @@ is also an [asynchronous API client](#connections-asynchronous).
::: lancedb.db.DBConnection
::: lancedb.Session
## Namespaces (Synchronous)
A namespace-backed connection resolves tables through a
[Lance namespace](https://lancedb.github.io/lance-namespace/) service instead of
listing a storage directory.
::: lancedb.connect_namespace
::: lancedb.namespace.LanceNamespaceDBConnection
## Tables (Synchronous)
::: lancedb.table.Table
@@ -34,8 +46,12 @@ is also an [asynchronous API client](#connections-asynchronous).
::: lancedb.table.FragmentSummaryStats
::: lancedb.table.TableStatistics
::: lancedb.table.Tags
::: lancedb.table.Branches
## Expressions
Type-safe expression builder for filters and projections. Use these instead
@@ -62,29 +78,46 @@ of raw SQL strings with [where][lancedb.query.LanceQueryBuilder.where] and
::: lancedb.query.LanceHybridQueryBuilder
::: lancedb.query.LanceEmptyQueryBuilder
::: lancedb.query.LanceTakeQueryBuilder
## Full text queries
Structured full text queries can be passed to
[Table.search][lancedb.table.Table.search] or
[AsyncTable.search][lancedb.table.AsyncTable.search] in place of a query string,
and combined with [BooleanQuery][lancedb.query.BooleanQuery].
::: lancedb.query.FullTextQuery
::: lancedb.query.MatchQuery
::: lancedb.query.PhraseQuery
::: lancedb.query.BoostQuery
::: lancedb.query.MultiMatchQuery
::: lancedb.query.BooleanQuery
::: lancedb.query.FullTextOperator
::: lancedb.query.Occur
## Embeddings
::: lancedb.embeddings.registry.EmbeddingFunctionRegistry
::: lancedb.embeddings.base.EmbeddingFunctionConfig
::: lancedb.embeddings.base.EmbeddingFunction
::: lancedb.embeddings.base.TextEmbeddingFunction
::: lancedb.embeddings.sentence_transformers.SentenceTransformerEmbeddings
::: lancedb.embeddings.openai.OpenAIEmbeddings
::: lancedb.embeddings.open_clip.OpenClipEmbeddings
::: lancedb.embeddings
options:
show_root_heading: false
show_root_toc_entry: false
## Remote configuration
::: lancedb.remote.ClientConfig
::: lancedb.remote.TimeoutConfig
::: lancedb.remote.RetryConfig
::: lancedb.remote
options:
show_root_heading: false
show_root_toc_entry: false
## Context
@@ -122,7 +155,22 @@ tokens = list(lancedb.tokenize("acme makes searchable data",
custom_stop_words=["acme"]))
```
::: lancedb.index.FTS
::: lancedb.tokenize
::: lancedb.FtsToken
## Blobs
Blob columns store large binary values out of line so they can be read lazily
instead of being materialized with the rest of the row.
::: lancedb.blob
::: lancedb.BlobType
::: lancedb._blob.BlobFile
options:
show_root_full_path: false
## Utilities
@@ -130,6 +178,14 @@ tokens = list(lancedb.tokenize("acme makes searchable data",
::: lancedb.merge.LanceMergeInsertBuilder
::: lancedb.otel.instrument_lancedb_metrics
## Exceptions
::: lancedb.exceptions.MissingValueError
::: lancedb.exceptions.MissingColumnError
## Integrations
## Pydantic
@@ -138,19 +194,30 @@ tokens = list(lancedb.tokenize("acme makes searchable data",
::: lancedb.pydantic.vector
::: lancedb.pydantic.Vector
::: lancedb.pydantic.MultiVector
::: lancedb.pydantic.LanceModel
## PyTorch
::: lancedb.streaming.StreamingDataset
::: lancedb.permutation.permutation_builder
::: lancedb.permutation.PermutationBuilder
::: lancedb.permutation.Permutation
::: lancedb.permutation.Transforms
## Reranking
::: lancedb.rerankers.linear_combination.LinearCombinationReranker
::: lancedb.rerankers.cohere.CohereReranker
::: lancedb.rerankers.colbert.ColbertReranker
::: lancedb.rerankers.cross_encoder.CrossEncoderReranker
::: lancedb.rerankers.openai.OpenaiReranker
::: lancedb.rerankers
options:
show_root_heading: false
show_root_toc_entry: false
## Connections (Asynchronous)
@@ -161,6 +228,12 @@ can be used to create, list, or open tables.
::: lancedb.db.AsyncConnection
## Namespaces (Asynchronous)
::: lancedb.connect_namespace_async
::: lancedb.namespace.AsyncLanceNamespaceDBConnection
## Tables (Asynchronous)
Table hold your actual data as a collection of records / rows.
@@ -169,32 +242,20 @@ Table hold your actual data as a collection of records / rows.
::: lancedb.table.AsyncTags
::: lancedb.table.AsyncBranches
## Indices (Asynchronous)
Indices can be created on a table to speed up queries. This section
lists the indices that LanceDb supports.
::: lancedb.index.BTree
::: lancedb.index.Bitmap
::: lancedb.index.LabelList
::: lancedb.index.FTS
::: lancedb.index.IvfPq
::: lancedb.index.HnswPq
::: lancedb.index.HnswSq
::: lancedb.index.IvfFlat
::: lancedb.index.IvfSq
::: lancedb.index.IvfRq
::: lancedb.index.HnswFlat
::: lancedb.index
options:
show_root_heading: false
show_root_toc_entry: false
# `lang_mapping` is defined in the module rather than imported, so it is
# picked up despite not being in `__all__`. It is an internal lookup table.
filters: ["!^_", "!^lang_mapping$"]
::: lancedb.table.IndexStatistics
@@ -222,3 +283,7 @@ rows nearest to a query vector and can be created with the
::: lancedb.query.AsyncHybridQuery
options:
inherited_members: true
::: lancedb.query.AsyncTakeQuery
options:
inherited_members: true
nodejs/CONTRIBUTING.md
+1 -1
View File
@@ -1,7 +1,7 @@
# Contributing to LanceDB Typescript
This document outlines the process for contributing to LanceDB Typescript.
For general contribution guidelines, see [CONTRIBUTING.md](../CONTRIBUTING.md).
For general contribution guidelines, see [CONTRIBUTING.md](https://github.com/lancedb/lancedb/blob/main/CONTRIBUTING.md).
## Project layout
python/python/lancedb/db.py
+2 -2
View File
@@ -404,7 +404,7 @@ class DBConnection(EnforceOverrides):
Data is converted to Arrow before being written to disk. For maximum
control over how data is saved, either provide the PyArrow schema to
convert to or else provide a [PyArrow Table](pyarrow.Table) directly.
convert to or else provide a [PyArrow Table][pyarrow.Table] directly.
>>> import pyarrow as pa
>>> custom_schema = pa.schema([
@@ -1570,7 +1570,7 @@ class AsyncConnection(object):
Data is converted to Arrow before being written to disk. For maximum
control over how data is saved, either provide the PyArrow schema to
convert to or else provide a [PyArrow Table](pyarrow.Table) directly.
convert to or else provide a [PyArrow Table][pyarrow.Table] directly.
>>> import pyarrow as pa
>>> custom_schema = pa.schema([
python/python/lancedb/embeddings/__init__.py
+29
View File
@@ -21,3 +21,32 @@ from .watsonx import WatsonxEmbeddings
from .voyageai import VoyageAIEmbeddingFunction
from .colpali import ColPaliEmbeddings
from .siglip import SigLipEmbeddings
# The API reference renders this package with a single mkdocstrings directive,
# which only picks up names listed here. New embedding functions must be added
# to both the imports above and this list, or they will silently go undocumented.
__all__ = [
"EmbeddingFunction",
"EmbeddingFunctionConfig",
"TextEmbeddingFunction",
"EmbeddingFunctionRegistry",
"get_registry",
"register",
"SentenceTransformerEmbeddings",
"OpenAIEmbeddings",
"OpenClipEmbeddings",
"BedRockText",
"CohereEmbeddingFunction",
"GeminiText",
"GteEmbeddings",
"InstructorEmbeddingFunction",
"JinaEmbeddings",
"OllamaEmbeddings",
"TransformersEmbeddingFunction",
"ColbertEmbeddings",
"VoyageAIEmbeddingFunction",
"WatsonxEmbeddings",
"ColPaliEmbeddings",
"ImageBindEmbeddings",
"SigLipEmbeddings",
]
python/python/lancedb/embeddings/bedrock.py
+5 -5
View File
@@ -21,20 +21,20 @@ class BedRockText(TextEmbeddingFunction):
"""
Parameters
----------
name: str, default "amazon.titan-embed-text-v1"
name : str, default "amazon.titan-embed-text-v1"
The model ID of the bedrock model to use. Supported models for are:
- amazon.titan-embed-text-v1
- cohere.embed-english-v3
- cohere.embed-multilingual-v3
region: str, default "us-east-1"
region : str, default "us-east-1"
Optional name of the AWS Region in which the service should be called.
profile_name: str, default None
profile_name : str, default None
Optional name of the AWS profile to use for calling the Bedrock service.
If not specified, the default profile will be used.
assumed_role: str, default None
assumed_role : str, default None
Optional ARN of an AWS IAM role to assume for calling the Bedrock service.
If not specified, the current active credentials will be used.
role_session_name: str, default "lancedb-embeddings"
role_session_name : str, default "lancedb-embeddings"
Optional name of the AWS IAM role session to use for calling the Bedrock
service. If not specified, "lancedb-embeddings" name will be used.
python/python/lancedb/embeddings/cohere.py
+5 -3
View File
@@ -22,7 +22,7 @@ class CohereEmbeddingFunction(TextEmbeddingFunction):
Parameters
----------
name: str, default "embed-multilingual-v2.0"
name : str, default "embed-multilingual-v2.0"
The name of the model to use. List of acceptable models:
* embed-english-v3.0
@@ -33,12 +33,14 @@ class CohereEmbeddingFunction(TextEmbeddingFunction):
* embed-english-light-v2.0
* embed-multilingual-v2.0
source_input_type: str, default "search_document"
source_input_type : str, default "search_document"
The input type for the source column in the database
query_input_type: str, default "search_query"
query_input_type : str, default "search_query"
The input type for the query column in the database
Notes
-----
Cohere supports following input types:
| Input Type | Description |
python/python/lancedb/embeddings/colpali.py
+2 -2
View File
@@ -44,7 +44,7 @@ class ColPaliEmbeddings(EmbeddingFunction):
The token pooling strategy to use, by default "hierarchical".
- "hierarchical": Progressively pools tokens to reduce sequence length.
- "lambda": A simpler pooling that uses a custom `pooling_func`.
pooling_func: typing.Callable, optional
pooling_func : typing.Callable, optional
A function to use for pooling when `pooling_strategy` is "lambda".
pool_factor : int
Factor to reduce sequence length if token pooling is enabled (default 2).
@@ -52,7 +52,7 @@ class ColPaliEmbeddings(EmbeddingFunction):
Quantization configuration for the model. (default None, bitsandbytes needed)
batch_size : int
Batch size for processing inputs (default 2).
offload_folder: str, optional
offload_folder : str, optional
Folder to offload model weights if using CPU offloading (default None). This is
useful for large models that do not fit in memory.
"""
python/python/lancedb/embeddings/gemini_text.py
+3 -3
View File
@@ -48,16 +48,16 @@ class GeminiText(TextEmbeddingFunction):
Parameters
----------
name: str, default "gemini-embedding-001"
name : str, default "gemini-embedding-001"
The name of the model to use. Supported models include:
- "gemini-embedding-001" (768 dimensions)
Note: The legacy "models/embedding-001" format is also supported but
"gemini-embedding-001" is recommended.
query_task_type: str, default "retrieval_query"
query_task_type : str, default "retrieval_query"
Sets the task type for the queries.
source_task_type: str, default "retrieval_document"
source_task_type : str, default "retrieval_document"
Sets the task type for ingestion.
Examples
python/python/lancedb/embeddings/gte.py
+4 -4
View File
@@ -26,13 +26,13 @@ class GteEmbeddings(TextEmbeddingFunction):
Parameters
----------
name: str, default "thenlper/gte-large"
name : str, default "thenlper/gte-large"
The name of the model to use.
device: str, default "cpu"
device : str, default "cpu"
Sets the device type for the model.
normalize: str, default "True"
normalize : str, default "True"
Controls normalize param in encode function for the transformer.
mlx: bool, default False
mlx : bool, default False
Controls which model to use. False for gte-large,True for the mlx version.
Examples
python/python/lancedb/embeddings/instructor.py
+8 -8
View File
@@ -35,23 +35,23 @@ class InstructorEmbeddingFunction(TextEmbeddingFunction):
Parameters
----------
name: str
name : str
The name of the model to use. Available models are listed at
https://github.com/xlang-ai/instructor-embedding#model-list;
The default model is hkunlp/instructor-base
batch_size: int, default 32
batch_size : int, default 32
The batch size to use when generating embeddings
device: str, default "cpu"
device : str, default "cpu"
The device to use when generating embeddings
show_progress_bar: bool, default True
show_progress_bar : bool, default True
Whether to show a progress bar when generating embeddings
normalize_embeddings: bool, default True
normalize_embeddings : bool, default True
Whether to normalize the embeddings
quantize: bool, default False
quantize : bool, default False
Whether to quantize the model
source_instruction: str, default "represent the document for retrieval"
source_instruction : str, default "represent the document for retrieval"
The instruction for the source column
query_instruction: str, default "represent the document for retrieving the most
query_instruction : str, default "represent the document for retrieving the most
similar documents"
The instruction for the query
python/python/lancedb/embeddings/jinaai.py
+2 -2
View File
@@ -40,10 +40,10 @@ class JinaEmbeddings(EmbeddingFunction):
Parameters
----------
name: str, default "jina-clip-v1". Note that some models support both image
name : str, default "jina-clip-v1". Note that some models support both image
and text embeddings and some just text embedding
api_key: str, default None
api_key : str, default None
The api key to access Jina API. If you pass None, you can set JINA_API_KEY
environment variable
python/python/lancedb/embeddings/sentence_transformers.py
+4 -4
View File
@@ -21,13 +21,13 @@ class SentenceTransformerEmbeddings(TextEmbeddingFunction):
Parameters
----------
name: str, default "all-MiniLM-L6-v2"
name : str, default "all-MiniLM-L6-v2"
The name of the model to use.
device: str, default "cpu"
device : str, default "cpu"
The device to use for the model
normalize: bool, default True
normalize : bool, default True
Whether to normalize the embeddings
trust_remote_code: bool, default True
trust_remote_code : bool, default True
Whether to trust the remote code
"""
python/python/lancedb/embeddings/voyageai.py
+2 -2
View File
@@ -167,7 +167,7 @@ class VoyageAIEmbeddingFunction(EmbeddingFunction):
Parameters
----------
name: str
name : str
The name of the model to use. List of acceptable models:
* voyage-4 (1024 dims, general-purpose and multilingual retrieval)
@@ -185,7 +185,7 @@ class VoyageAIEmbeddingFunction(EmbeddingFunction):
* voyage-law-2
* voyage-code-2
output_dimension: int, optional
output_dimension : int, optional
The output dimension for models that support flexible dimensions.
Currently only voyage-multimodal-3.5 supports this feature.
Valid options: 256, 512, 1024 (default), 2048.
python/python/lancedb/index.py
+26 -23
View File
@@ -219,7 +219,7 @@ class HnswPq:
distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their
l2 norm is 1), then dot distance is equivalent to the cosine distance.
num_partitions, default sqrt(num_rows)
num_partitions: int, default sqrt(num_rows)
The number of IVF partitions to create.
@@ -228,7 +228,7 @@ class HnswPq:
will require too much memory. Each partition becomes its own HNSW graph, so
setting this value higher reduces the peak memory use of training.
num_sub_vectors, default is vector dimension / 16
num_sub_vectors: int, default is vector dimension / 16
Number of sub-vectors of PQ.
@@ -244,13 +244,13 @@ class HnswPq:
If the dimension is not visible by 8 then we use 1 subvector. This is not
ideal and will likely result in poor performance.
num_bits: int, default 8
num_bits: int, default 8
Number of bits to encode each sub-vector.
This value controls how much the sub-vectors are compressed. The more bits
the more accurate the index but the slower search. Only 4 and 8 are supported.
max_iterations, default 50
max_iterations: int, default 50
Max iterations to train kmeans.
@@ -263,7 +263,7 @@ class HnswPq:
those cases it is unlikely that setting this larger will lead to the index
converging anyways.
sample_rate, default 256
sample_rate: int, default 256
The rate used to calculate the number of training vectors for kmeans.
@@ -279,14 +279,14 @@ class HnswPq:
Increasing this value might improve the quality of the index but in
most cases the default should be sufficient.
m, default 20
m: int, default 20
The number of neighbors to select for each vector in the HNSW graph.
This value controls the tradeoff between search speed and accuracy.
The higher the value the more accurate the search but the slower it will be.
ef_construction, default 300
ef_construction: int, default 300
The number of candidates to evaluate during the construction of the HNSW graph.
@@ -297,7 +297,7 @@ class HnswPq:
This value should be set to a value that is not less than `ef` in the
search phase.
target_partition_size, default is 1,048,576
target_partition_size: int, default is 1,048,576
The target size of each partition.
@@ -351,7 +351,7 @@ class HnswSq:
distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their
l2 norm is 1), then dot distance is equivalent to the cosine distance.
num_partitions, default sqrt(num_rows)
num_partitions: int, default sqrt(num_rows)
The number of IVF partitions to create.
@@ -360,7 +360,7 @@ class HnswSq:
will require too much memory. Each partition becomes its own HNSW graph, so
setting this value higher reduces the peak memory use of training.
max_iterations, default 50
max_iterations: int, default 50
Max iterations to train kmeans.
@@ -373,7 +373,7 @@ class HnswSq:
In those cases it is unlikely that setting this larger will lead to
the index converging anyways.
sample_rate, default 256
sample_rate: int, default 256
The rate used to calculate the number of training vectors for kmeans.
@@ -389,14 +389,14 @@ class HnswSq:
Increasing this value might improve the quality of the index but in
most cases the default should be sufficient.
m, default 20
m: int, default 20
The number of neighbors to select for each vector in the HNSW graph.
This value controls the tradeoff between search speed and accuracy.
The higher the value the more accurate the search but the slower it will be.
ef_construction, default 300
ef_construction: int, default 300
The number of candidates to evaluate during the construction of the HNSW graph.
@@ -407,7 +407,7 @@ class HnswSq:
This value should be set to a value that is not less than `ef` in the search
phase.
target_partition_size, default is 1,048,576
target_partition_size: int, default is 1,048,576
The target size of each partition.
@@ -460,7 +460,7 @@ class HnswFlat:
distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their
l2 norm is 1), then dot distance is equivalent to the cosine distance.
num_partitions, default sqrt(num_rows)
num_partitions: int, default sqrt(num_rows)
The number of IVF partitions to create.
@@ -470,18 +470,18 @@ class HnswFlat:
graph, so setting this value higher reduces the peak memory use of
training.
max_iterations, default 50
max_iterations: int, default 50
Max iterations to train kmeans.
When training an IVF index we use kmeans to calculate the partitions.
This parameter controls how many iterations of kmeans to run.
sample_rate, default 256
sample_rate: int, default 256
The rate used to calculate the number of training vectors for kmeans.
m, default 20
m: int, default 20
The number of neighbors to select for each vector in the HNSW graph.
@@ -489,7 +489,7 @@ class HnswFlat:
The higher the value the more accurate the search but the slower it
will be.
ef_construction, default 300
ef_construction: int, default 300
The number of candidates to evaluate during the construction of the HNSW
graph.
@@ -501,7 +501,7 @@ class HnswFlat:
than 500. This value should be set to a value that is not less than `ef`
in the search phase.
target_partition_size, default is 1,048,576
target_partition_size: int, default is 1,048,576
The target size of each partition.
"""
@@ -605,7 +605,7 @@ class IvfFlat:
The default value is 256.
target_partition_size, default is 8192
target_partition_size: int, default is 8192
The target size of each partition.
@@ -769,7 +769,7 @@ class IvfPq:
The default value is 256.
target_partition_size, default is 8192
target_partition_size: int, default is 8192
The target size of each partition.
@@ -830,7 +830,7 @@ class IvfRq:
sample_rate: int, default 256
Controls the number of training vectors: sample_rate * num_partitions.
target_partition_size, default is 8192
target_partition_size: int, default is 8192
Target size of each partition.
"""
@@ -845,6 +845,9 @@ class IvfRq:
accelerator: Optional[str] = None
# The API reference renders this module with a single mkdocstrings directive,
# which only picks up names listed here. New public names must be added to this
# list, or they will silently go undocumented.
__all__ = [
"BTree",
"IvfPq",
python/python/lancedb/permutation.py
+11 -6
View File
@@ -438,7 +438,8 @@ class Permutation:
_reader: Optional[PermutationReader] = None,
):
"""
Internal constructor. Use [from_tables](#from_tables) instead.
Internal constructor. Use
[from_tables][lancedb.permutation.Permutation.from_tables] instead.
"""
assert base_table is not None, "base_table is required"
assert selection is not None, "selection is required"
@@ -985,8 +986,9 @@ class Permutation:
types. Conversion of strings, lists, and structs will require creating python
objects and this is not zero-copy.
For custom formatting, use [with_transform](#with_transform) which overrides
this method.
For custom formatting, use
[with_transform][lancedb.permutation.Permutation.with_transform] which
overrides this method.
"""
assert format is not None, "format is required"
if format == "python":
@@ -1061,7 +1063,8 @@ class Permutation:
Note: this method returns a new permutation and does not modify `self`
It is provided for compatibility with the huggingface Dataset API.
Use [with_skip](#with_skip) instead to avoid confusion.
Use [with_skip][lancedb.permutation.Permutation.with_skip] instead to
avoid confusion.
"""
return self.with_skip(skip)
@@ -1084,7 +1087,8 @@ class Permutation:
Note: this method returns a new permutation and does not modify `self`
It is provided for compatibility with the huggingface Dataset API.
Use [with_take](#with_take) instead to avoid confusion.
Use [with_take][lancedb.permutation.Permutation.with_take] instead to
avoid confusion.
"""
return self.with_take(limit)
@@ -1107,7 +1111,8 @@ class Permutation:
Note: this method returns a new permutation and does not modify `self`
It is provided for compatibility with the huggingface Dataset API.
Use [with_repeat](#with_repeat) instead to avoid confusion.
Use [with_repeat][lancedb.permutation.Permutation.with_repeat] instead
to avoid confusion.
"""
return self.with_repeat(times)
python/python/lancedb/query.py
+18 -13
View File
@@ -650,7 +650,8 @@ class Query(pydantic.BaseModel):
distance_type : Optional[str]
the distance type to use for vector search
This can be l2 (default), cosine and dot. See [metric definitions][search] for
This can be l2 (default), cosine and dot. See
[metric definitions](https://lancedb.com/docs/search/vector-search/) for
more details.
If this is not a vector search this will be None.
@@ -663,8 +664,9 @@ class Query(pydantic.BaseModel):
- A higher number makes search more accurate but also slower.
- See discussion in [Querying an ANN Index][querying-an-ann-index] for
tuning advice.
- See discussion in
[Querying an ANN Index](https://lancedb.com/docs/indexing/)
for tuning advice.
Will be None if this is not a vector search.
refine_factor : Optional[int]
@@ -672,8 +674,9 @@ class Query(pydantic.BaseModel):
- A higher number makes search more accurate but also slower.
- See discussion in [Querying an ANN Index][querying-an-ann-index] for
tuning advice.
- See discussion in
[Querying an ANN Index](https://lancedb.com/docs/indexing/)
for tuning advice.
Will be None if this is not a vector search.
lower_bound : Optional[float]
@@ -1647,8 +1650,8 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
Higher values will yield better recall (more likely to find vectors if
they exist) at the expense of latency.
See discussion in [Querying an ANN Index][querying-an-ann-index] for
tuning advice.
See discussion in [Querying an ANN Index](https://lancedb.com/docs/indexing/)
for tuning advice.
This method sets both the minimum and maximum number of probes to the same
value. See `minimum_nprobes` and `maximum_nprobes` for more fine-grained
@@ -1748,8 +1751,8 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
As an example, a refine factor of 2 will sample 2x as many vectors as
requested, re-ranks them, and returns the top half most relevant results.
See discussion in [Querying an ANN Index][querying-an-ann-index] for
tuning advice.
See discussion in [Querying an ANN Index](https://lancedb.com/docs/indexing/)
for tuning advice.
Parameters
----------
@@ -3373,8 +3376,9 @@ class AsyncQuery(AsyncStandardQuery):
are various ANN search parameters that will let you fine tune your recall
accuracy vs search latency.
Vector searches always have a [limit][]. If `limit` has not been called then
a default `limit` of 10 will be used.
Vector searches always have a
[limit][lancedb.query.AsyncVectorQuery.limit]. If `limit` has not been
called then a default `limit` of 10 will be used.
Typically, a single vector is passed in as the query. However, you can also
pass in multiple vectors. When multiple vectors are passed in, if the vector
@@ -3505,8 +3509,9 @@ class AsyncFTSQuery(AsyncStandardQuery):
are various ANN search parameters that will let you fine tune your recall
accuracy vs search latency.
Hybrid searches always have a [limit][]. If `limit` has not been called then
a default `limit` of 10 will be used.
Hybrid searches always have a
[limit][lancedb.query.AsyncHybridQuery.limit]. If `limit` has not been
called then a default `limit` of 10 will be used.
Typically, a single vector is passed in as the query. However, you can also
pass in multiple vectors. This can be useful if you want to find the nearest
python/python/lancedb/remote/__init__.py
+3
View File
@@ -11,6 +11,9 @@ from lancedb import __version__
from .header import HeaderProvider
from .oauth import OAuthConfig, OAuthFlowType
# The API reference renders this module with a single mkdocstrings directive,
# which only picks up names listed here. New public names must be added to this
# list, or they will silently go undocumented.
__all__ = [
"TimeoutConfig",
"RetryConfig",
python/python/lancedb/remote/errors.py
+2 -2
View File
@@ -53,9 +53,9 @@ class RetryError(LanceDBClientError):
"""An error that occurs when the client has exceeded the maximum number of retries.
The retry strategy can be adjusted by setting the
[retry_config](lancedb.remote.ClientConfig.retry_config) in the client
[retry_config][lancedb.remote.ClientConfig.retry_config] in the client
configuration. This is passed in the `client_config` argument of
[connect](lancedb.connect) and [connect_async](lancedb.connect_async).
[connect][lancedb.connect] and [connect_async][lancedb.connect_async].
The __cause__ attribute of this exception will be the last exception that
caused the retry to fail. It will be an
python/python/lancedb/remote/table.py
+5 -3
View File
@@ -581,8 +581,9 @@ class RemoteTable(Table):
progress: Optional[Union[bool, Callable, Any]] = None,
write_parallelism: Optional[int] = None,
) -> AddResult:
"""Add more data to the [Table](Table). It has the same API signature as
the OSS version.
"""Add more data to the [Table][lancedb.table.Table].
It has the same API signature as the OSS version.
Parameters
----------
@@ -642,7 +643,8 @@ class RemoteTable(Table):
fast_search: bool = False,
) -> LanceVectorQueryBuilder:
"""Create a search query to find the nearest neighbors
of the given query vector. We currently support [vector search][search]
of the given query vector. We currently support
[vector search](https://lancedb.com/docs/search/vector-search/)
All query options are defined in
[LanceVectorQueryBuilder][lancedb.query.LanceVectorQueryBuilder].
python/python/lancedb/rerankers/__init__.py
+3
View File
@@ -14,6 +14,9 @@ from .answerdotai import AnswerdotaiRerankers
from .voyageai import VoyageAIReranker
from .watsonx import WatsonxReranker
# The API reference renders this module with a single mkdocstrings directive,
# which only picks up names listed here. New public names must be added to this
# list, or they will silently go undocumented.
__all__ = [
"Reranker",
"CrossEncoderReranker",
python/python/lancedb/table.py
+25 -18
View File
@@ -1211,7 +1211,7 @@ class Table(ABC):
progress: Optional[Union[bool, Callable, Any]] = None,
write_parallelism: Optional[int] = None,
) -> AddResult:
"""Add more data to the [Table](Table).
"""Add more data to the [Table][lancedb.table.Table].
Parameters
----------
@@ -1343,8 +1343,8 @@ class Table(ABC):
fts_columns: Optional[Union[str, List[str]]] = None,
) -> LanceQueryBuilder:
"""Create a search query to find the nearest neighbors
of the given query vector. We currently support [vector search][search]
and [full-text search][experimental-full-text-search].
of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/)
and [full-text search](https://lancedb.com/docs/search/full-text-search/).
All query options are defined in
[LanceQueryBuilder][lancedb.query.LanceQueryBuilder].
@@ -1780,7 +1780,7 @@ class Table(ABC):
for faster reads.
Arguments are passed onto Lance's
[compact_files][lance.dataset.DatasetOptimizer.compact_files].
`lance.dataset.DatasetOptimizer.compact_files`.
For most cases, the default should be fine.
See Also
@@ -1834,6 +1834,8 @@ class Table(ABC):
retrain: bool, default False
This parameter is no longer used and is deprecated.
Notes
-----
The frequency an application should call optimize is based on the frequency of
data modifications. If data is frequently added, deleted, or updated then
optimize should be run frequently. A good rule of thumb is to run optimize if
@@ -1988,15 +1990,14 @@ class Table(ABC):
change permanent you can use the `[Self::restore]` method.
Any operation that modifies the table will fail while the table is in a checked
out state.
out state. To return the table to a normal state use
`[Self::checkout_latest]`.
Parameters
----------
version: int | str,
The version to check out. A version number (`int`) or a tag
(`str`) can be provided.
To return the table to a normal state use `[Self::checkout_latest]`
"""
@abstractmethod
@@ -3383,8 +3384,8 @@ class LanceTable(Table):
fts_columns: Optional[Union[str, List[str]]] = None,
) -> LanceQueryBuilder:
"""Create a search query to find the nearest neighbors
of the given query vector. We currently support [vector search][search]
and [full-text search][search].
of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/)
and [full-text search](https://lancedb.com/docs/search/full-text-search/).
Examples
--------
@@ -3414,8 +3415,9 @@ class LanceTable(Table):
- *default None*.
Acceptable types are: list, np.ndarray, PIL.Image.Image
- If None then the select/[where][sql]/limit clauses are applied
to filter the table
- If None then the
select/[where][lancedb.query.LanceQueryBuilder.where]/limit clauses
are applied to filter the table
vector_column_name: str, optional
The name of the vector column to search.
@@ -3809,6 +3811,8 @@ class LanceTable(Table):
retrain: bool, default False
This parameter is no longer used and is deprecated.
Notes
-----
The frequency an application should call optimize is based on the frequency of
data modifications. If data is frequently added, deleted, or updated then
optimize should be run frequently. A good rule of thumb is to run optimize if
@@ -4687,7 +4691,7 @@ class AsyncTable:
Parameters
----------
**kwargs
Forwarded to [`lance.dataset`][lance.dataset].
Forwarded to `lance.dataset`.
Returns
-------
@@ -5006,7 +5010,7 @@ class AsyncTable:
progress: Optional[Union[bool, Callable, Any]] = None,
write_parallelism: Optional[int] = None,
) -> AddResult:
"""Add more data to the [Table](Table).
"""Add more data to the [AsyncTable][lancedb.table.AsyncTable].
Parameters
----------
@@ -5208,8 +5212,8 @@ class AsyncTable:
fts_columns: Optional[Union[str, List[str]]] = None,
) -> Union[AsyncHybridQuery, AsyncFTSQuery, AsyncVectorQuery]:
"""Create a search query to find the nearest neighbors
of the given query vector. We currently support [vector search][search]
and [full-text search][experimental-full-text-search].
of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/)
and [full-text search](https://lancedb.com/docs/search/full-text-search/).
All query options are defined in [AsyncQuery][lancedb.query.AsyncQuery].
@@ -5770,15 +5774,14 @@ class AsyncTable:
change permanent you can use the `[Self::restore]` method.
Any operation that modifies the table will fail while the table is in a checked
out state.
out state. To return the table to a normal state use
`[Self::checkout_latest]`.
Parameters
----------
version: int | str,
The version to check out. A version number (`int`) or a tag
(`str`) can be provided.
To return the table to a normal state use `[Self::checkout_latest]`
"""
try:
await self._inner.checkout(version)
@@ -5962,6 +5965,8 @@ class AsyncTable:
retrain: bool, default False
This parameter is no longer used and is deprecated.
Notes
-----
The frequency an application should call optimize is based on the frequency of
data modifications. If data is frequently added, deleted, or updated then
optimize should be run frequently. A good rule of thumb is to run optimize if
@@ -6342,6 +6347,8 @@ class Branches:
dry_run: bool, default False
When True, only preview. When False, attempt the merge.
Notes
-----
A rejected merge returns ``status="rejected"`` instead of raising.
"""
return LOOP.run(self._table.branches.merge(from_branch, dry_run))