From 5a1015ba72ec98170b3c10b737ecaa5b7eb3869b Mon Sep 17 00:00:00 2001 From: Will Jones Date: Thu, 30 Jul 2026 16:50:05 -0700 Subject: [PATCH] docs(python): fill gaps in the Python API reference (#3746) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `docs/src/python/python.md` is the whole Python API reference, but it is maintained by hand and had drifted from the public API. Anything not listed there simply doesn't get rendered, so a number of public, documented, tested APIs were invisible to users — most notably branch management, where `diff` and `merge` live. I audited every public symbol reachable from `lancedb` and its subpackages against the `:::` directives on the page. This adds the missing ones: - **Branching** — `Branches`, `AsyncBranches` (`list` / `create` / `checkout` / `delete` / `diff` / `merge`) - **Tables** — `TableStatistics` (returned by `Table.stats()`; the fragment-level stats classes were already listed) - **Full text queries** — `FullTextQuery`, `MatchQuery`, `PhraseQuery`, `BoostQuery`, `MultiMatchQuery`, `BooleanQuery`, `FullTextOperator`, `Occur` - **Querying** — `LanceEmptyQueryBuilder`, `LanceTakeQueryBuilder`, `AsyncTakeQuery` - **Indices** — `Fm` (the FM-index for substring search), `IndexConfig` - **Blobs** — `blob`, `BlobType`, `BlobFile` - **Namespaces** — `connect_namespace`, `connect_namespace_async`, and both namespace connection classes - **Remote config** — `TlsConfig`, `HeaderProvider`, `OAuthConfig`, `OAuthFlowType` - **Rerankers** — the `Reranker` base class plus `JinaReranker`, `RRFReranker`, `MRRReranker`, `AnswerdotaiRerankers`, `VoyageAIReranker`, `WatsonxReranker` (5 of 12 were listed) - **Embeddings** — `get_registry`, `register`, and the 14 embedding functions that were missing (3 of 17 were listed) - **PyTorch** — `StreamingDataset` and the permutation API it is built on - **Misc** — `Session`, `tokenize`, `FtsToken`, `pydantic.Vector`, `pydantic.MultiVector`, `instrument_lancedb_metrics`, and the two exception types It also repairs cross-references in docstrings that no longer resolve: links into guide pages that have since moved to lancedb.com (`querying-an-ann-index`, `experimental-full-text-search`), `lance.dataset` references with no inventory behind them, and the relative targets `[Table](Table)` and `[PyArrow Table](pyarrow.Table)`. Deliberately left out: concrete implementation classes reached through their abstract base (`LanceTable`, `LanceDBConnection`, `RemoteDBConnection`), query base classes already covered by `inherited_members: true`, and internal plumbing such as `FullTextSearchQuery` and `ColumnOrdering`. ## Testing The docs job only runs on pushes to `main`, so I built the site locally and compared against a build of `upstream/main`: every added entry resolves, and no symbol that was rendered before stopped being rendered when the four packages moved to automodule. `mkdocs build --strict` exits 0 on this branch, against 61 warnings on `main`. ## Also in this PR `lancedb.index`, `lancedb.embeddings`, `lancedb.remote` and `lancedb.rerankers` are now rendered by a single mkdocstrings directive each, driven by the module's `__all__`, rather than a hand-maintained list. These four are where most of the drift was, and `__all__` is harder to forget than a docs page. `lancedb.embeddings` had no `__all__`; without one mkdocstrings renders no members at all for a re-export package, so one is added. AGENTS.md gains a section on how the page is wired up and how to build the docs locally. Rendering all that code for the first time surfaced ~100 more build warnings, which would have made #3707 (turning on `mkdocs build --strict`) harder to land, so the warning backlog is cleared here too. 97 of the 158 warnings were one systematic false positive — griffe cannot see the generated `__init__` of a pydantic dataclass, so every documented parameter looks unknown — switched off via `warn_unknown_params`. The remaining 61 came from 15 docstrings with real bugs: prose trailing a `Parameters` section (we were rendering parameters called `The`, `you` and `To`), types dropped because numpydoc needs spaces around the colon, `num_partitions, default sqrt(num_rows)` parsing as a list of names and inventing a `default` parameter, and one parameter indented five spaces. `mkdocs build --strict` now exits 0. --- #3747 (the coverage test that keeps this from happening again) is stacked on this branch, so review it after this one. --------- Co-authored-by: Claude Opus 5 (1M context) --- AGENTS.md | 29 ++++ docs/mkdocs.yml | 5 + docs/src/js/_media/CONTRIBUTING.md | 2 +- docs/src/python/python.md | 163 ++++++++++++------ nodejs/CONTRIBUTING.md | 2 +- python/python/lancedb/db.py | 4 +- python/python/lancedb/embeddings/__init__.py | 29 ++++ python/python/lancedb/embeddings/bedrock.py | 10 +- python/python/lancedb/embeddings/cohere.py | 8 +- python/python/lancedb/embeddings/colpali.py | 4 +- .../python/lancedb/embeddings/gemini_text.py | 6 +- python/python/lancedb/embeddings/gte.py | 8 +- .../python/lancedb/embeddings/instructor.py | 16 +- python/python/lancedb/embeddings/jinaai.py | 4 +- .../embeddings/sentence_transformers.py | 8 +- python/python/lancedb/embeddings/voyageai.py | 4 +- python/python/lancedb/index.py | 49 +++--- python/python/lancedb/permutation.py | 17 +- python/python/lancedb/query.py | 31 ++-- python/python/lancedb/remote/__init__.py | 3 + python/python/lancedb/remote/errors.py | 4 +- python/python/lancedb/remote/table.py | 8 +- python/python/lancedb/rerankers/__init__.py | 3 + python/python/lancedb/table.py | 43 +++-- 24 files changed, 309 insertions(+), 151 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 14ec35441..21631a2cd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,6 +92,8 @@ Python bindings changes: * Should use `LOOP.run()` to call the corresponding `AsyncTable` method. 6. Add concrete sync method to `RemoteTable` class in `python/python/lancedb/remote/table.py`. 7. Add unit test in `python/tests/test_table.py`. +8. If you added a new public class or module-level function (not just a method on an + existing class), expose it in the API reference. See "Python API reference" below. TypeScript bindings changes: @@ -103,6 +105,33 @@ TypeScript bindings changes: 5. Add test in `nodejs/__test__/table.test.ts`. 6. Run `npm run docs` to generate TypeScript documentation. +## Python API reference + +`docs/src/python/python.md` is the entire Python API reference. It is maintained by +hand, and anything not listed there is not rendered at all, so new public classes and +module-level functions have to be added explicitly. How depends on the module: + +* `lancedb.index`, `lancedb.embeddings`, `lancedb.remote`, and `lancedb.rerankers` are + rendered by a single directive each, driven by the module's `__all__`. Add the new + name to `__all__` and it appears; forget, and it is silently omitted. +* Everything else (`lancedb`, `lancedb.table`, `lancedb.query`, `lancedb.db`, ...) is + listed symbol by symbol. Add a `::: lancedb..` line to the matching + section, and remember that the page separates synchronous and asynchronous APIs. + +Deliberately undocumented: concrete implementations reached through an abstract base +(`LanceTable`, `LanceDBConnection`, `RemoteDBConnection`), query base classes already +covered by `inherited_members`, and internal helpers. + +Cross-references in docstrings use mkdocstrings syntax, `[text][lancedb.table.Table]`. +Plain relative links such as `[Table](Table)` do not resolve. To check your work: + +```shell +pip install -r docs/requirements.txt +cd docs && PYTHONPATH=. mkdocs build +``` + +The docs site only builds on pushes to `main`, so this is not covered by PR CI. + ## Review Guidelines Please consider the following when reviewing code contributions. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 324b6fda5..ad6fc4b65 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -51,6 +51,11 @@ plugins: paths: [../python/python] options: docstring_style: numpy + docstring_options: + # Attributes documented in a `Parameters` section, and pydantic + # dataclasses whose `__init__` griffe cannot see statically, both + # trip this check. It reports nothing actionable here. + warn_unknown_params: false heading_level: 3 show_signature_annotations: true show_root_heading: true diff --git a/docs/src/js/_media/CONTRIBUTING.md b/docs/src/js/_media/CONTRIBUTING.md index d784b9f75..2faa8cd37 100644 --- a/docs/src/js/_media/CONTRIBUTING.md +++ b/docs/src/js/_media/CONTRIBUTING.md @@ -1,7 +1,7 @@ # Contributing to LanceDB Typescript This document outlines the process for contributing to LanceDB Typescript. -For general contribution guidelines, see [CONTRIBUTING.md](../CONTRIBUTING.md). +For general contribution guidelines, see [CONTRIBUTING.md](https://github.com/lancedb/lancedb/blob/main/CONTRIBUTING.md). ## Project layout diff --git a/docs/src/python/python.md b/docs/src/python/python.md index dd60451cc..36044d35d 100644 --- a/docs/src/python/python.md +++ b/docs/src/python/python.md @@ -26,6 +26,18 @@ is also an [asynchronous API client](#connections-asynchronous). ::: lancedb.db.DBConnection +::: lancedb.Session + +## Namespaces (Synchronous) + +A namespace-backed connection resolves tables through a +[Lance namespace](https://lancedb.github.io/lance-namespace/) service instead of +listing a storage directory. + +::: lancedb.connect_namespace + +::: lancedb.namespace.LanceNamespaceDBConnection + ## Tables (Synchronous) ::: lancedb.table.Table @@ -34,8 +46,12 @@ is also an [asynchronous API client](#connections-asynchronous). ::: lancedb.table.FragmentSummaryStats +::: lancedb.table.TableStatistics + ::: lancedb.table.Tags +::: lancedb.table.Branches + ## Expressions Type-safe expression builder for filters and projections. Use these instead @@ -62,29 +78,46 @@ of raw SQL strings with [where][lancedb.query.LanceQueryBuilder.where] and ::: lancedb.query.LanceHybridQueryBuilder +::: lancedb.query.LanceEmptyQueryBuilder + +::: lancedb.query.LanceTakeQueryBuilder + +## Full text queries + +Structured full text queries can be passed to +[Table.search][lancedb.table.Table.search] or +[AsyncTable.search][lancedb.table.AsyncTable.search] in place of a query string, +and combined with [BooleanQuery][lancedb.query.BooleanQuery]. + +::: lancedb.query.FullTextQuery + +::: lancedb.query.MatchQuery + +::: lancedb.query.PhraseQuery + +::: lancedb.query.BoostQuery + +::: lancedb.query.MultiMatchQuery + +::: lancedb.query.BooleanQuery + +::: lancedb.query.FullTextOperator + +::: lancedb.query.Occur + ## Embeddings -::: lancedb.embeddings.registry.EmbeddingFunctionRegistry - -::: lancedb.embeddings.base.EmbeddingFunctionConfig - -::: lancedb.embeddings.base.EmbeddingFunction - -::: lancedb.embeddings.base.TextEmbeddingFunction - -::: lancedb.embeddings.sentence_transformers.SentenceTransformerEmbeddings - -::: lancedb.embeddings.openai.OpenAIEmbeddings - -::: lancedb.embeddings.open_clip.OpenClipEmbeddings +::: lancedb.embeddings + options: + show_root_heading: false + show_root_toc_entry: false ## Remote configuration -::: lancedb.remote.ClientConfig - -::: lancedb.remote.TimeoutConfig - -::: lancedb.remote.RetryConfig +::: lancedb.remote + options: + show_root_heading: false + show_root_toc_entry: false ## Context @@ -122,7 +155,22 @@ tokens = list(lancedb.tokenize("acme makes searchable data", custom_stop_words=["acme"])) ``` -::: lancedb.index.FTS +::: lancedb.tokenize + +::: lancedb.FtsToken + +## Blobs + +Blob columns store large binary values out of line so they can be read lazily +instead of being materialized with the rest of the row. + +::: lancedb.blob + +::: lancedb.BlobType + +::: lancedb._blob.BlobFile + options: + show_root_full_path: false ## Utilities @@ -130,6 +178,14 @@ tokens = list(lancedb.tokenize("acme makes searchable data", ::: lancedb.merge.LanceMergeInsertBuilder +::: lancedb.otel.instrument_lancedb_metrics + +## Exceptions + +::: lancedb.exceptions.MissingValueError + +::: lancedb.exceptions.MissingColumnError + ## Integrations ## Pydantic @@ -138,19 +194,30 @@ tokens = list(lancedb.tokenize("acme makes searchable data", ::: lancedb.pydantic.vector +::: lancedb.pydantic.Vector + +::: lancedb.pydantic.MultiVector + ::: lancedb.pydantic.LanceModel +## PyTorch + +::: lancedb.streaming.StreamingDataset + +::: lancedb.permutation.permutation_builder + +::: lancedb.permutation.PermutationBuilder + +::: lancedb.permutation.Permutation + +::: lancedb.permutation.Transforms + ## Reranking -::: lancedb.rerankers.linear_combination.LinearCombinationReranker - -::: lancedb.rerankers.cohere.CohereReranker - -::: lancedb.rerankers.colbert.ColbertReranker - -::: lancedb.rerankers.cross_encoder.CrossEncoderReranker - -::: lancedb.rerankers.openai.OpenaiReranker +::: lancedb.rerankers + options: + show_root_heading: false + show_root_toc_entry: false ## Connections (Asynchronous) @@ -161,6 +228,12 @@ can be used to create, list, or open tables. ::: lancedb.db.AsyncConnection +## Namespaces (Asynchronous) + +::: lancedb.connect_namespace_async + +::: lancedb.namespace.AsyncLanceNamespaceDBConnection + ## Tables (Asynchronous) Table hold your actual data as a collection of records / rows. @@ -169,32 +242,20 @@ Table hold your actual data as a collection of records / rows. ::: lancedb.table.AsyncTags +::: lancedb.table.AsyncBranches + ## Indices (Asynchronous) Indices can be created on a table to speed up queries. This section lists the indices that LanceDb supports. -::: lancedb.index.BTree - -::: lancedb.index.Bitmap - -::: lancedb.index.LabelList - -::: lancedb.index.FTS - -::: lancedb.index.IvfPq - -::: lancedb.index.HnswPq - -::: lancedb.index.HnswSq - -::: lancedb.index.IvfFlat - -::: lancedb.index.IvfSq - -::: lancedb.index.IvfRq - -::: lancedb.index.HnswFlat +::: lancedb.index + options: + show_root_heading: false + show_root_toc_entry: false + # `lang_mapping` is defined in the module rather than imported, so it is + # picked up despite not being in `__all__`. It is an internal lookup table. + filters: ["!^_", "!^lang_mapping$"] ::: lancedb.table.IndexStatistics @@ -222,3 +283,7 @@ rows nearest to a query vector and can be created with the ::: lancedb.query.AsyncHybridQuery options: inherited_members: true + +::: lancedb.query.AsyncTakeQuery + options: + inherited_members: true diff --git a/nodejs/CONTRIBUTING.md b/nodejs/CONTRIBUTING.md index d784b9f75..2faa8cd37 100644 --- a/nodejs/CONTRIBUTING.md +++ b/nodejs/CONTRIBUTING.md @@ -1,7 +1,7 @@ # Contributing to LanceDB Typescript This document outlines the process for contributing to LanceDB Typescript. -For general contribution guidelines, see [CONTRIBUTING.md](../CONTRIBUTING.md). +For general contribution guidelines, see [CONTRIBUTING.md](https://github.com/lancedb/lancedb/blob/main/CONTRIBUTING.md). ## Project layout diff --git a/python/python/lancedb/db.py b/python/python/lancedb/db.py index 75ff7239d..90b8dc75b 100644 --- a/python/python/lancedb/db.py +++ b/python/python/lancedb/db.py @@ -404,7 +404,7 @@ class DBConnection(EnforceOverrides): Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to - convert to or else provide a [PyArrow Table](pyarrow.Table) directly. + convert to or else provide a [PyArrow Table][pyarrow.Table] directly. >>> import pyarrow as pa >>> custom_schema = pa.schema([ @@ -1570,7 +1570,7 @@ class AsyncConnection(object): Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to - convert to or else provide a [PyArrow Table](pyarrow.Table) directly. + convert to or else provide a [PyArrow Table][pyarrow.Table] directly. >>> import pyarrow as pa >>> custom_schema = pa.schema([ diff --git a/python/python/lancedb/embeddings/__init__.py b/python/python/lancedb/embeddings/__init__.py index f70aa57b2..4a39b7900 100644 --- a/python/python/lancedb/embeddings/__init__.py +++ b/python/python/lancedb/embeddings/__init__.py @@ -21,3 +21,32 @@ from .watsonx import WatsonxEmbeddings from .voyageai import VoyageAIEmbeddingFunction from .colpali import ColPaliEmbeddings from .siglip import SigLipEmbeddings + +# The API reference renders this package with a single mkdocstrings directive, +# which only picks up names listed here. New embedding functions must be added +# to both the imports above and this list, or they will silently go undocumented. +__all__ = [ + "EmbeddingFunction", + "EmbeddingFunctionConfig", + "TextEmbeddingFunction", + "EmbeddingFunctionRegistry", + "get_registry", + "register", + "SentenceTransformerEmbeddings", + "OpenAIEmbeddings", + "OpenClipEmbeddings", + "BedRockText", + "CohereEmbeddingFunction", + "GeminiText", + "GteEmbeddings", + "InstructorEmbeddingFunction", + "JinaEmbeddings", + "OllamaEmbeddings", + "TransformersEmbeddingFunction", + "ColbertEmbeddings", + "VoyageAIEmbeddingFunction", + "WatsonxEmbeddings", + "ColPaliEmbeddings", + "ImageBindEmbeddings", + "SigLipEmbeddings", +] diff --git a/python/python/lancedb/embeddings/bedrock.py b/python/python/lancedb/embeddings/bedrock.py index ffa44f0c8..dc2badceb 100644 --- a/python/python/lancedb/embeddings/bedrock.py +++ b/python/python/lancedb/embeddings/bedrock.py @@ -21,20 +21,20 @@ class BedRockText(TextEmbeddingFunction): """ Parameters ---------- - name: str, default "amazon.titan-embed-text-v1" + name : str, default "amazon.titan-embed-text-v1" The model ID of the bedrock model to use. Supported models for are: - amazon.titan-embed-text-v1 - cohere.embed-english-v3 - cohere.embed-multilingual-v3 - region: str, default "us-east-1" + region : str, default "us-east-1" Optional name of the AWS Region in which the service should be called. - profile_name: str, default None + profile_name : str, default None Optional name of the AWS profile to use for calling the Bedrock service. If not specified, the default profile will be used. - assumed_role: str, default None + assumed_role : str, default None Optional ARN of an AWS IAM role to assume for calling the Bedrock service. If not specified, the current active credentials will be used. - role_session_name: str, default "lancedb-embeddings" + role_session_name : str, default "lancedb-embeddings" Optional name of the AWS IAM role session to use for calling the Bedrock service. If not specified, "lancedb-embeddings" name will be used. diff --git a/python/python/lancedb/embeddings/cohere.py b/python/python/lancedb/embeddings/cohere.py index 8769ddaaa..8fc460f9a 100644 --- a/python/python/lancedb/embeddings/cohere.py +++ b/python/python/lancedb/embeddings/cohere.py @@ -22,7 +22,7 @@ class CohereEmbeddingFunction(TextEmbeddingFunction): Parameters ---------- - name: str, default "embed-multilingual-v2.0" + name : str, default "embed-multilingual-v2.0" The name of the model to use. List of acceptable models: * embed-english-v3.0 @@ -33,12 +33,14 @@ class CohereEmbeddingFunction(TextEmbeddingFunction): * embed-english-light-v2.0 * embed-multilingual-v2.0 - source_input_type: str, default "search_document" + source_input_type : str, default "search_document" The input type for the source column in the database - query_input_type: str, default "search_query" + query_input_type : str, default "search_query" The input type for the query column in the database + Notes + ----- Cohere supports following input types: | Input Type | Description | diff --git a/python/python/lancedb/embeddings/colpali.py b/python/python/lancedb/embeddings/colpali.py index 4b614bce3..7232d32ce 100644 --- a/python/python/lancedb/embeddings/colpali.py +++ b/python/python/lancedb/embeddings/colpali.py @@ -44,7 +44,7 @@ class ColPaliEmbeddings(EmbeddingFunction): The token pooling strategy to use, by default "hierarchical". - "hierarchical": Progressively pools tokens to reduce sequence length. - "lambda": A simpler pooling that uses a custom `pooling_func`. - pooling_func: typing.Callable, optional + pooling_func : typing.Callable, optional A function to use for pooling when `pooling_strategy` is "lambda". pool_factor : int Factor to reduce sequence length if token pooling is enabled (default 2). @@ -52,7 +52,7 @@ class ColPaliEmbeddings(EmbeddingFunction): Quantization configuration for the model. (default None, bitsandbytes needed) batch_size : int Batch size for processing inputs (default 2). - offload_folder: str, optional + offload_folder : str, optional Folder to offload model weights if using CPU offloading (default None). This is useful for large models that do not fit in memory. """ diff --git a/python/python/lancedb/embeddings/gemini_text.py b/python/python/lancedb/embeddings/gemini_text.py index 1102e92d1..32f2d4d04 100644 --- a/python/python/lancedb/embeddings/gemini_text.py +++ b/python/python/lancedb/embeddings/gemini_text.py @@ -48,16 +48,16 @@ class GeminiText(TextEmbeddingFunction): Parameters ---------- - name: str, default "gemini-embedding-001" + name : str, default "gemini-embedding-001" The name of the model to use. Supported models include: - "gemini-embedding-001" (768 dimensions) Note: The legacy "models/embedding-001" format is also supported but "gemini-embedding-001" is recommended. - query_task_type: str, default "retrieval_query" + query_task_type : str, default "retrieval_query" Sets the task type for the queries. - source_task_type: str, default "retrieval_document" + source_task_type : str, default "retrieval_document" Sets the task type for ingestion. Examples diff --git a/python/python/lancedb/embeddings/gte.py b/python/python/lancedb/embeddings/gte.py index 4f547a30b..9bad4b54f 100644 --- a/python/python/lancedb/embeddings/gte.py +++ b/python/python/lancedb/embeddings/gte.py @@ -26,13 +26,13 @@ class GteEmbeddings(TextEmbeddingFunction): Parameters ---------- - name: str, default "thenlper/gte-large" + name : str, default "thenlper/gte-large" The name of the model to use. - device: str, default "cpu" + device : str, default "cpu" Sets the device type for the model. - normalize: str, default "True" + normalize : str, default "True" Controls normalize param in encode function for the transformer. - mlx: bool, default False + mlx : bool, default False Controls which model to use. False for gte-large,True for the mlx version. Examples diff --git a/python/python/lancedb/embeddings/instructor.py b/python/python/lancedb/embeddings/instructor.py index a694fd382..675a0139c 100644 --- a/python/python/lancedb/embeddings/instructor.py +++ b/python/python/lancedb/embeddings/instructor.py @@ -35,23 +35,23 @@ class InstructorEmbeddingFunction(TextEmbeddingFunction): Parameters ---------- - name: str + name : str The name of the model to use. Available models are listed at https://github.com/xlang-ai/instructor-embedding#model-list; The default model is hkunlp/instructor-base - batch_size: int, default 32 + batch_size : int, default 32 The batch size to use when generating embeddings - device: str, default "cpu" + device : str, default "cpu" The device to use when generating embeddings - show_progress_bar: bool, default True + show_progress_bar : bool, default True Whether to show a progress bar when generating embeddings - normalize_embeddings: bool, default True + normalize_embeddings : bool, default True Whether to normalize the embeddings - quantize: bool, default False + quantize : bool, default False Whether to quantize the model - source_instruction: str, default "represent the document for retrieval" + source_instruction : str, default "represent the document for retrieval" The instruction for the source column - query_instruction: str, default "represent the document for retrieving the most + query_instruction : str, default "represent the document for retrieving the most similar documents" The instruction for the query diff --git a/python/python/lancedb/embeddings/jinaai.py b/python/python/lancedb/embeddings/jinaai.py index 720e80e85..9656f041f 100644 --- a/python/python/lancedb/embeddings/jinaai.py +++ b/python/python/lancedb/embeddings/jinaai.py @@ -40,10 +40,10 @@ class JinaEmbeddings(EmbeddingFunction): Parameters ---------- - name: str, default "jina-clip-v1". Note that some models support both image + name : str, default "jina-clip-v1". Note that some models support both image and text embeddings and some just text embedding - api_key: str, default None + api_key : str, default None The api key to access Jina API. If you pass None, you can set JINA_API_KEY environment variable diff --git a/python/python/lancedb/embeddings/sentence_transformers.py b/python/python/lancedb/embeddings/sentence_transformers.py index 30d283cf7..8be02e9f7 100644 --- a/python/python/lancedb/embeddings/sentence_transformers.py +++ b/python/python/lancedb/embeddings/sentence_transformers.py @@ -21,13 +21,13 @@ class SentenceTransformerEmbeddings(TextEmbeddingFunction): Parameters ---------- - name: str, default "all-MiniLM-L6-v2" + name : str, default "all-MiniLM-L6-v2" The name of the model to use. - device: str, default "cpu" + device : str, default "cpu" The device to use for the model - normalize: bool, default True + normalize : bool, default True Whether to normalize the embeddings - trust_remote_code: bool, default True + trust_remote_code : bool, default True Whether to trust the remote code """ diff --git a/python/python/lancedb/embeddings/voyageai.py b/python/python/lancedb/embeddings/voyageai.py index 3cf70d332..4a721c998 100644 --- a/python/python/lancedb/embeddings/voyageai.py +++ b/python/python/lancedb/embeddings/voyageai.py @@ -167,7 +167,7 @@ class VoyageAIEmbeddingFunction(EmbeddingFunction): Parameters ---------- - name: str + name : str The name of the model to use. List of acceptable models: * voyage-4 (1024 dims, general-purpose and multilingual retrieval) @@ -185,7 +185,7 @@ class VoyageAIEmbeddingFunction(EmbeddingFunction): * voyage-law-2 * voyage-code-2 - output_dimension: int, optional + output_dimension : int, optional The output dimension for models that support flexible dimensions. Currently only voyage-multimodal-3.5 supports this feature. Valid options: 256, 512, 1024 (default), 2048. diff --git a/python/python/lancedb/index.py b/python/python/lancedb/index.py index 5ced0600f..aa7846892 100644 --- a/python/python/lancedb/index.py +++ b/python/python/lancedb/index.py @@ -219,7 +219,7 @@ class HnswPq: distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their l2 norm is 1), then dot distance is equivalent to the cosine distance. - num_partitions, default sqrt(num_rows) + num_partitions: int, default sqrt(num_rows) The number of IVF partitions to create. @@ -228,7 +228,7 @@ class HnswPq: will require too much memory. Each partition becomes its own HNSW graph, so setting this value higher reduces the peak memory use of training. - num_sub_vectors, default is vector dimension / 16 + num_sub_vectors: int, default is vector dimension / 16 Number of sub-vectors of PQ. @@ -244,13 +244,13 @@ class HnswPq: If the dimension is not visible by 8 then we use 1 subvector. This is not ideal and will likely result in poor performance. - num_bits: int, default 8 + num_bits: int, default 8 Number of bits to encode each sub-vector. This value controls how much the sub-vectors are compressed. The more bits the more accurate the index but the slower search. Only 4 and 8 are supported. - max_iterations, default 50 + max_iterations: int, default 50 Max iterations to train kmeans. @@ -263,7 +263,7 @@ class HnswPq: those cases it is unlikely that setting this larger will lead to the index converging anyways. - sample_rate, default 256 + sample_rate: int, default 256 The rate used to calculate the number of training vectors for kmeans. @@ -279,14 +279,14 @@ class HnswPq: Increasing this value might improve the quality of the index but in most cases the default should be sufficient. - m, default 20 + m: int, default 20 The number of neighbors to select for each vector in the HNSW graph. This value controls the tradeoff between search speed and accuracy. The higher the value the more accurate the search but the slower it will be. - ef_construction, default 300 + ef_construction: int, default 300 The number of candidates to evaluate during the construction of the HNSW graph. @@ -297,7 +297,7 @@ class HnswPq: This value should be set to a value that is not less than `ef` in the search phase. - target_partition_size, default is 1,048,576 + target_partition_size: int, default is 1,048,576 The target size of each partition. @@ -351,7 +351,7 @@ class HnswSq: distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their l2 norm is 1), then dot distance is equivalent to the cosine distance. - num_partitions, default sqrt(num_rows) + num_partitions: int, default sqrt(num_rows) The number of IVF partitions to create. @@ -360,7 +360,7 @@ class HnswSq: will require too much memory. Each partition becomes its own HNSW graph, so setting this value higher reduces the peak memory use of training. - max_iterations, default 50 + max_iterations: int, default 50 Max iterations to train kmeans. @@ -373,7 +373,7 @@ class HnswSq: In those cases it is unlikely that setting this larger will lead to the index converging anyways. - sample_rate, default 256 + sample_rate: int, default 256 The rate used to calculate the number of training vectors for kmeans. @@ -389,14 +389,14 @@ class HnswSq: Increasing this value might improve the quality of the index but in most cases the default should be sufficient. - m, default 20 + m: int, default 20 The number of neighbors to select for each vector in the HNSW graph. This value controls the tradeoff between search speed and accuracy. The higher the value the more accurate the search but the slower it will be. - ef_construction, default 300 + ef_construction: int, default 300 The number of candidates to evaluate during the construction of the HNSW graph. @@ -407,7 +407,7 @@ class HnswSq: This value should be set to a value that is not less than `ef` in the search phase. - target_partition_size, default is 1,048,576 + target_partition_size: int, default is 1,048,576 The target size of each partition. @@ -460,7 +460,7 @@ class HnswFlat: distance has a range of (-∞, ∞). If the vectors are normalized (i.e. their l2 norm is 1), then dot distance is equivalent to the cosine distance. - num_partitions, default sqrt(num_rows) + num_partitions: int, default sqrt(num_rows) The number of IVF partitions to create. @@ -470,18 +470,18 @@ class HnswFlat: graph, so setting this value higher reduces the peak memory use of training. - max_iterations, default 50 + max_iterations: int, default 50 Max iterations to train kmeans. When training an IVF index we use kmeans to calculate the partitions. This parameter controls how many iterations of kmeans to run. - sample_rate, default 256 + sample_rate: int, default 256 The rate used to calculate the number of training vectors for kmeans. - m, default 20 + m: int, default 20 The number of neighbors to select for each vector in the HNSW graph. @@ -489,7 +489,7 @@ class HnswFlat: The higher the value the more accurate the search but the slower it will be. - ef_construction, default 300 + ef_construction: int, default 300 The number of candidates to evaluate during the construction of the HNSW graph. @@ -501,7 +501,7 @@ class HnswFlat: than 500. This value should be set to a value that is not less than `ef` in the search phase. - target_partition_size, default is 1,048,576 + target_partition_size: int, default is 1,048,576 The target size of each partition. """ @@ -605,7 +605,7 @@ class IvfFlat: The default value is 256. - target_partition_size, default is 8192 + target_partition_size: int, default is 8192 The target size of each partition. @@ -769,7 +769,7 @@ class IvfPq: The default value is 256. - target_partition_size, default is 8192 + target_partition_size: int, default is 8192 The target size of each partition. @@ -830,7 +830,7 @@ class IvfRq: sample_rate: int, default 256 Controls the number of training vectors: sample_rate * num_partitions. - target_partition_size, default is 8192 + target_partition_size: int, default is 8192 Target size of each partition. """ @@ -845,6 +845,9 @@ class IvfRq: accelerator: Optional[str] = None +# The API reference renders this module with a single mkdocstrings directive, +# which only picks up names listed here. New public names must be added to this +# list, or they will silently go undocumented. __all__ = [ "BTree", "IvfPq", diff --git a/python/python/lancedb/permutation.py b/python/python/lancedb/permutation.py index 8aa764c4b..b3d95c352 100644 --- a/python/python/lancedb/permutation.py +++ b/python/python/lancedb/permutation.py @@ -438,7 +438,8 @@ class Permutation: _reader: Optional[PermutationReader] = None, ): """ - Internal constructor. Use [from_tables](#from_tables) instead. + Internal constructor. Use + [from_tables][lancedb.permutation.Permutation.from_tables] instead. """ assert base_table is not None, "base_table is required" assert selection is not None, "selection is required" @@ -985,8 +986,9 @@ class Permutation: types. Conversion of strings, lists, and structs will require creating python objects and this is not zero-copy. - For custom formatting, use [with_transform](#with_transform) which overrides - this method. + For custom formatting, use + [with_transform][lancedb.permutation.Permutation.with_transform] which + overrides this method. """ assert format is not None, "format is required" if format == "python": @@ -1061,7 +1063,8 @@ class Permutation: Note: this method returns a new permutation and does not modify `self` It is provided for compatibility with the huggingface Dataset API. - Use [with_skip](#with_skip) instead to avoid confusion. + Use [with_skip][lancedb.permutation.Permutation.with_skip] instead to + avoid confusion. """ return self.with_skip(skip) @@ -1084,7 +1087,8 @@ class Permutation: Note: this method returns a new permutation and does not modify `self` It is provided for compatibility with the huggingface Dataset API. - Use [with_take](#with_take) instead to avoid confusion. + Use [with_take][lancedb.permutation.Permutation.with_take] instead to + avoid confusion. """ return self.with_take(limit) @@ -1107,7 +1111,8 @@ class Permutation: Note: this method returns a new permutation and does not modify `self` It is provided for compatibility with the huggingface Dataset API. - Use [with_repeat](#with_repeat) instead to avoid confusion. + Use [with_repeat][lancedb.permutation.Permutation.with_repeat] instead + to avoid confusion. """ return self.with_repeat(times) diff --git a/python/python/lancedb/query.py b/python/python/lancedb/query.py index 181eb9349..60c5d0c68 100644 --- a/python/python/lancedb/query.py +++ b/python/python/lancedb/query.py @@ -650,7 +650,8 @@ class Query(pydantic.BaseModel): distance_type : Optional[str] the distance type to use for vector search - This can be l2 (default), cosine and dot. See [metric definitions][search] for + This can be l2 (default), cosine and dot. See + [metric definitions](https://lancedb.com/docs/search/vector-search/) for more details. If this is not a vector search this will be None. @@ -663,8 +664,9 @@ class Query(pydantic.BaseModel): - A higher number makes search more accurate but also slower. - - See discussion in [Querying an ANN Index][querying-an-ann-index] for - tuning advice. + - See discussion in + [Querying an ANN Index](https://lancedb.com/docs/indexing/) + for tuning advice. Will be None if this is not a vector search. refine_factor : Optional[int] @@ -672,8 +674,9 @@ class Query(pydantic.BaseModel): - A higher number makes search more accurate but also slower. - - See discussion in [Querying an ANN Index][querying-an-ann-index] for - tuning advice. + - See discussion in + [Querying an ANN Index](https://lancedb.com/docs/indexing/) + for tuning advice. Will be None if this is not a vector search. lower_bound : Optional[float] @@ -1647,8 +1650,8 @@ class LanceVectorQueryBuilder(LanceQueryBuilder): Higher values will yield better recall (more likely to find vectors if they exist) at the expense of latency. - See discussion in [Querying an ANN Index][querying-an-ann-index] for - tuning advice. + See discussion in [Querying an ANN Index](https://lancedb.com/docs/indexing/) + for tuning advice. This method sets both the minimum and maximum number of probes to the same value. See `minimum_nprobes` and `maximum_nprobes` for more fine-grained @@ -1748,8 +1751,8 @@ class LanceVectorQueryBuilder(LanceQueryBuilder): As an example, a refine factor of 2 will sample 2x as many vectors as requested, re-ranks them, and returns the top half most relevant results. - See discussion in [Querying an ANN Index][querying-an-ann-index] for - tuning advice. + See discussion in [Querying an ANN Index](https://lancedb.com/docs/indexing/) + for tuning advice. Parameters ---------- @@ -3373,8 +3376,9 @@ class AsyncQuery(AsyncStandardQuery): are various ANN search parameters that will let you fine tune your recall accuracy vs search latency. - Vector searches always have a [limit][]. If `limit` has not been called then - a default `limit` of 10 will be used. + Vector searches always have a + [limit][lancedb.query.AsyncVectorQuery.limit]. If `limit` has not been + called then a default `limit` of 10 will be used. Typically, a single vector is passed in as the query. However, you can also pass in multiple vectors. When multiple vectors are passed in, if the vector @@ -3505,8 +3509,9 @@ class AsyncFTSQuery(AsyncStandardQuery): are various ANN search parameters that will let you fine tune your recall accuracy vs search latency. - Hybrid searches always have a [limit][]. If `limit` has not been called then - a default `limit` of 10 will be used. + Hybrid searches always have a + [limit][lancedb.query.AsyncHybridQuery.limit]. If `limit` has not been + called then a default `limit` of 10 will be used. Typically, a single vector is passed in as the query. However, you can also pass in multiple vectors. This can be useful if you want to find the nearest diff --git a/python/python/lancedb/remote/__init__.py b/python/python/lancedb/remote/__init__.py index a6ef55eb5..1f255b991 100644 --- a/python/python/lancedb/remote/__init__.py +++ b/python/python/lancedb/remote/__init__.py @@ -11,6 +11,9 @@ from lancedb import __version__ from .header import HeaderProvider from .oauth import OAuthConfig, OAuthFlowType +# The API reference renders this module with a single mkdocstrings directive, +# which only picks up names listed here. New public names must be added to this +# list, or they will silently go undocumented. __all__ = [ "TimeoutConfig", "RetryConfig", diff --git a/python/python/lancedb/remote/errors.py b/python/python/lancedb/remote/errors.py index 2f2dad9b4..5aa13a9c7 100644 --- a/python/python/lancedb/remote/errors.py +++ b/python/python/lancedb/remote/errors.py @@ -53,9 +53,9 @@ class RetryError(LanceDBClientError): """An error that occurs when the client has exceeded the maximum number of retries. The retry strategy can be adjusted by setting the - [retry_config](lancedb.remote.ClientConfig.retry_config) in the client + [retry_config][lancedb.remote.ClientConfig.retry_config] in the client configuration. This is passed in the `client_config` argument of - [connect](lancedb.connect) and [connect_async](lancedb.connect_async). + [connect][lancedb.connect] and [connect_async][lancedb.connect_async]. The __cause__ attribute of this exception will be the last exception that caused the retry to fail. It will be an diff --git a/python/python/lancedb/remote/table.py b/python/python/lancedb/remote/table.py index 20a952c53..268e0d5a4 100644 --- a/python/python/lancedb/remote/table.py +++ b/python/python/lancedb/remote/table.py @@ -581,8 +581,9 @@ class RemoteTable(Table): progress: Optional[Union[bool, Callable, Any]] = None, write_parallelism: Optional[int] = None, ) -> AddResult: - """Add more data to the [Table](Table). It has the same API signature as - the OSS version. + """Add more data to the [Table][lancedb.table.Table]. + + It has the same API signature as the OSS version. Parameters ---------- @@ -642,7 +643,8 @@ class RemoteTable(Table): fast_search: bool = False, ) -> LanceVectorQueryBuilder: """Create a search query to find the nearest neighbors - of the given query vector. We currently support [vector search][search] + of the given query vector. We currently support + [vector search](https://lancedb.com/docs/search/vector-search/) All query options are defined in [LanceVectorQueryBuilder][lancedb.query.LanceVectorQueryBuilder]. diff --git a/python/python/lancedb/rerankers/__init__.py b/python/python/lancedb/rerankers/__init__.py index 86b0ac884..aef2d331e 100644 --- a/python/python/lancedb/rerankers/__init__.py +++ b/python/python/lancedb/rerankers/__init__.py @@ -14,6 +14,9 @@ from .answerdotai import AnswerdotaiRerankers from .voyageai import VoyageAIReranker from .watsonx import WatsonxReranker +# The API reference renders this module with a single mkdocstrings directive, +# which only picks up names listed here. New public names must be added to this +# list, or they will silently go undocumented. __all__ = [ "Reranker", "CrossEncoderReranker", diff --git a/python/python/lancedb/table.py b/python/python/lancedb/table.py index 26d78a427..399b075f0 100644 --- a/python/python/lancedb/table.py +++ b/python/python/lancedb/table.py @@ -1211,7 +1211,7 @@ class Table(ABC): progress: Optional[Union[bool, Callable, Any]] = None, write_parallelism: Optional[int] = None, ) -> AddResult: - """Add more data to the [Table](Table). + """Add more data to the [Table][lancedb.table.Table]. Parameters ---------- @@ -1343,8 +1343,8 @@ class Table(ABC): fts_columns: Optional[Union[str, List[str]]] = None, ) -> LanceQueryBuilder: """Create a search query to find the nearest neighbors - of the given query vector. We currently support [vector search][search] - and [full-text search][experimental-full-text-search]. + of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/) + and [full-text search](https://lancedb.com/docs/search/full-text-search/). All query options are defined in [LanceQueryBuilder][lancedb.query.LanceQueryBuilder]. @@ -1780,7 +1780,7 @@ class Table(ABC): for faster reads. Arguments are passed onto Lance's - [compact_files][lance.dataset.DatasetOptimizer.compact_files]. + `lance.dataset.DatasetOptimizer.compact_files`. For most cases, the default should be fine. See Also @@ -1834,6 +1834,8 @@ class Table(ABC): retrain: bool, default False This parameter is no longer used and is deprecated. + Notes + ----- The frequency an application should call optimize is based on the frequency of data modifications. If data is frequently added, deleted, or updated then optimize should be run frequently. A good rule of thumb is to run optimize if @@ -1988,15 +1990,14 @@ class Table(ABC): change permanent you can use the `[Self::restore]` method. Any operation that modifies the table will fail while the table is in a checked - out state. + out state. To return the table to a normal state use + `[Self::checkout_latest]`. Parameters ---------- version: int | str, The version to check out. A version number (`int`) or a tag (`str`) can be provided. - - To return the table to a normal state use `[Self::checkout_latest]` """ @abstractmethod @@ -3383,8 +3384,8 @@ class LanceTable(Table): fts_columns: Optional[Union[str, List[str]]] = None, ) -> LanceQueryBuilder: """Create a search query to find the nearest neighbors - of the given query vector. We currently support [vector search][search] - and [full-text search][search]. + of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/) + and [full-text search](https://lancedb.com/docs/search/full-text-search/). Examples -------- @@ -3414,8 +3415,9 @@ class LanceTable(Table): - *default None*. Acceptable types are: list, np.ndarray, PIL.Image.Image - - If None then the select/[where][sql]/limit clauses are applied - to filter the table + - If None then the + select/[where][lancedb.query.LanceQueryBuilder.where]/limit clauses + are applied to filter the table vector_column_name: str, optional The name of the vector column to search. @@ -3809,6 +3811,8 @@ class LanceTable(Table): retrain: bool, default False This parameter is no longer used and is deprecated. + Notes + ----- The frequency an application should call optimize is based on the frequency of data modifications. If data is frequently added, deleted, or updated then optimize should be run frequently. A good rule of thumb is to run optimize if @@ -4687,7 +4691,7 @@ class AsyncTable: Parameters ---------- **kwargs - Forwarded to [`lance.dataset`][lance.dataset]. + Forwarded to `lance.dataset`. Returns ------- @@ -5006,7 +5010,7 @@ class AsyncTable: progress: Optional[Union[bool, Callable, Any]] = None, write_parallelism: Optional[int] = None, ) -> AddResult: - """Add more data to the [Table](Table). + """Add more data to the [AsyncTable][lancedb.table.AsyncTable]. Parameters ---------- @@ -5208,8 +5212,8 @@ class AsyncTable: fts_columns: Optional[Union[str, List[str]]] = None, ) -> Union[AsyncHybridQuery, AsyncFTSQuery, AsyncVectorQuery]: """Create a search query to find the nearest neighbors - of the given query vector. We currently support [vector search][search] - and [full-text search][experimental-full-text-search]. + of the given query vector. We currently support [vector search](https://lancedb.com/docs/search/vector-search/) + and [full-text search](https://lancedb.com/docs/search/full-text-search/). All query options are defined in [AsyncQuery][lancedb.query.AsyncQuery]. @@ -5770,15 +5774,14 @@ class AsyncTable: change permanent you can use the `[Self::restore]` method. Any operation that modifies the table will fail while the table is in a checked - out state. + out state. To return the table to a normal state use + `[Self::checkout_latest]`. Parameters ---------- version: int | str, The version to check out. A version number (`int`) or a tag (`str`) can be provided. - - To return the table to a normal state use `[Self::checkout_latest]` """ try: await self._inner.checkout(version) @@ -5962,6 +5965,8 @@ class AsyncTable: retrain: bool, default False This parameter is no longer used and is deprecated. + Notes + ----- The frequency an application should call optimize is based on the frequency of data modifications. If data is frequently added, deleted, or updated then optimize should be run frequently. A good rule of thumb is to run optimize if @@ -6342,6 +6347,8 @@ class Branches: dry_run: bool, default False When True, only preview. When False, attempt the merge. + Notes + ----- A rejected merge returns ``status="rejected"`` instead of raising. """ return LOOP.run(self._table.branches.merge(from_branch, dry_run))