Will Jones
5a1015ba72
`docs/src/python/python.md` is the whole Python API reference, but it is maintained by hand and had drifted from the public API. Anything not listed there simply doesn't get rendered, so a number of public, documented, tested APIs were invisible to users — most notably branch management, where `diff` and `merge` live. I audited every public symbol reachable from `lancedb` and its subpackages against the `:::` directives on the page. This adds the missing ones: - **Branching** — `Branches`, `AsyncBranches` (`list` / `create` / `checkout` / `delete` / `diff` / `merge`) - **Tables** — `TableStatistics` (returned by `Table.stats()`; the fragment-level stats classes were already listed) - **Full text queries** — `FullTextQuery`, `MatchQuery`, `PhraseQuery`, `BoostQuery`, `MultiMatchQuery`, `BooleanQuery`, `FullTextOperator`, `Occur` - **Querying** — `LanceEmptyQueryBuilder`, `LanceTakeQueryBuilder`, `AsyncTakeQuery` - **Indices** — `Fm` (the FM-index for substring search), `IndexConfig` - **Blobs** — `blob`, `BlobType`, `BlobFile` - **Namespaces** — `connect_namespace`, `connect_namespace_async`, and both namespace connection classes - **Remote config** — `TlsConfig`, `HeaderProvider`, `OAuthConfig`, `OAuthFlowType` - **Rerankers** — the `Reranker` base class plus `JinaReranker`, `RRFReranker`, `MRRReranker`, `AnswerdotaiRerankers`, `VoyageAIReranker`, `WatsonxReranker` (5 of 12 were listed) - **Embeddings** — `get_registry`, `register`, and the 14 embedding functions that were missing (3 of 17 were listed) - **PyTorch** — `StreamingDataset` and the permutation API it is built on - **Misc** — `Session`, `tokenize`, `FtsToken`, `pydantic.Vector`, `pydantic.MultiVector`, `instrument_lancedb_metrics`, and the two exception types It also repairs cross-references in docstrings that no longer resolve: links into guide pages that have since moved to lancedb.com (`querying-an-ann-index`, `experimental-full-text-search`), `lance.dataset` references with no inventory behind them, and the relative targets `[Table](Table)` and `[PyArrow Table](pyarrow.Table)`. Deliberately left out: concrete implementation classes reached through their abstract base (`LanceTable`, `LanceDBConnection`, `RemoteDBConnection`), query base classes already covered by `inherited_members: true`, and internal plumbing such as `FullTextSearchQuery` and `ColumnOrdering`. ## Testing The docs job only runs on pushes to `main`, so I built the site locally and compared against a build of `upstream/main`: every added entry resolves, and no symbol that was rendered before stopped being rendered when the four packages moved to automodule. `mkdocs build --strict` exits 0 on this branch, against 61 warnings on `main`. ## Also in this PR `lancedb.index`, `lancedb.embeddings`, `lancedb.remote` and `lancedb.rerankers` are now rendered by a single mkdocstrings directive each, driven by the module's `__all__`, rather than a hand-maintained list. These four are where most of the drift was, and `__all__` is harder to forget than a docs page. `lancedb.embeddings` had no `__all__`; without one mkdocstrings renders no members at all for a re-export package, so one is added. AGENTS.md gains a section on how the page is wired up and how to build the docs locally. Rendering all that code for the first time surfaced ~100 more build warnings, which would have made #3707 (turning on `mkdocs build --strict`) harder to land, so the warning backlog is cleared here too. 97 of the 158 warnings were one systematic false positive — griffe cannot see the generated `__init__` of a pydantic dataclass, so every documented parameter looks unknown — switched off via `warn_unknown_params`. The remaining 61 came from 15 docstrings with real bugs: prose trailing a `Parameters` section (we were rendering parameters called `The`, `you` and `To`), types dropped because numpydoc needs spaces around the colon, `num_partitions, default sqrt(num_rows)` parsing as a list of names and inventing a `default` parameter, and one parameter indented five spaces. `mkdocs build --strict` now exits 0. --- #3747 (the coverage test that keeps this from happening again) is stacked on this branch, so review it after this one. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
6.2 KiB
Python API Reference
This section contains the API reference for the Python API of LanceDB. Both synchronous and asynchronous APIs are available.
The general flow of using the API is:
- Use [lancedb.connect][] or [lancedb.connect_async][] to connect to a database.
- Use the returned [lancedb.DBConnection][] or [lancedb.AsyncConnection][] to create or open tables.
- Use the returned [lancedb.table.Table][] or [lancedb.AsyncTable][] to query or modify tables.
Installation
pip install lancedb
The following methods describe the synchronous API client. There is also an asynchronous API client.
Connections (Synchronous)
::: lancedb.connect
::: lancedb.db.DBConnection
::: lancedb.Session
Namespaces (Synchronous)
A namespace-backed connection resolves tables through a Lance namespace service instead of listing a storage directory.
::: lancedb.connect_namespace
::: lancedb.namespace.LanceNamespaceDBConnection
Tables (Synchronous)
::: lancedb.table.Table
::: lancedb.table.FragmentStatistics
::: lancedb.table.FragmentSummaryStats
::: lancedb.table.TableStatistics
::: lancedb.table.Tags
::: lancedb.table.Branches
Expressions
Type-safe expression builder for filters and projections. Use these instead of raw SQL strings with [where][lancedb.query.LanceQueryBuilder.where] and [select][lancedb.query.LanceQueryBuilder.select].
::: lancedb.expr.Expr
::: lancedb.expr.col
::: lancedb.expr.lit
::: lancedb.expr.func
Querying (Synchronous)
::: lancedb.query.Query
::: lancedb.query.LanceQueryBuilder
::: lancedb.query.LanceVectorQueryBuilder
::: lancedb.query.LanceFtsQueryBuilder
::: lancedb.query.LanceHybridQueryBuilder
::: lancedb.query.LanceEmptyQueryBuilder
::: lancedb.query.LanceTakeQueryBuilder
Full text queries
Structured full text queries can be passed to [Table.search][lancedb.table.Table.search] or [AsyncTable.search][lancedb.table.AsyncTable.search] in place of a query string, and combined with [BooleanQuery][lancedb.query.BooleanQuery].
::: lancedb.query.FullTextQuery
::: lancedb.query.MatchQuery
::: lancedb.query.PhraseQuery
::: lancedb.query.BoostQuery
::: lancedb.query.MultiMatchQuery
::: lancedb.query.BooleanQuery
::: lancedb.query.FullTextOperator
::: lancedb.query.Occur
Embeddings
::: lancedb.embeddings options: show_root_heading: false show_root_toc_entry: false
Remote configuration
::: lancedb.remote options: show_root_heading: false show_root_toc_entry: false
Context
::: lancedb.context.contextualize
::: lancedb.context.Contextualizer
Full text search
Pass custom_stop_words to [lancedb.index.FTS][]:
from lancedb.index import FTS
table.create_index(
"text",
config=FTS(remove_stop_words=True, custom_stop_words=["acme", "internal"]),
)
The list replaces the built-in stop words and is used only when
remove_stop_words=True:
custom_stop_words=Noneuses the built-in list forlanguage.custom_stop_words=[]removes no words.- Values are passed through without trimming, lowercasing, or other rewriting.
The same option is available on lancedb.tokenize(...) and the deprecated
[lancedb.table.Table.create_fts_index][] compatibility helper:
import lancedb
tokens = list(lancedb.tokenize("acme makes searchable data",
custom_stop_words=["acme"]))
::: lancedb.tokenize
::: lancedb.FtsToken
Blobs
Blob columns store large binary values out of line so they can be read lazily instead of being materialized with the rest of the row.
::: lancedb.blob
::: lancedb.BlobType
::: lancedb._blob.BlobFile options: show_root_full_path: false
Utilities
::: lancedb.schema.vector
::: lancedb.merge.LanceMergeInsertBuilder
::: lancedb.otel.instrument_lancedb_metrics
Exceptions
::: lancedb.exceptions.MissingValueError
::: lancedb.exceptions.MissingColumnError
Integrations
Pydantic
::: lancedb.pydantic.pydantic_to_schema
::: lancedb.pydantic.vector
::: lancedb.pydantic.Vector
::: lancedb.pydantic.MultiVector
::: lancedb.pydantic.LanceModel
PyTorch
::: lancedb.streaming.StreamingDataset
::: lancedb.permutation.permutation_builder
::: lancedb.permutation.PermutationBuilder
::: lancedb.permutation.Permutation
::: lancedb.permutation.Transforms
Reranking
::: lancedb.rerankers options: show_root_heading: false show_root_toc_entry: false
Connections (Asynchronous)
Connections represent a connection to a LanceDb database and can be used to create, list, or open tables.
::: lancedb.connect_async
::: lancedb.db.AsyncConnection
Namespaces (Asynchronous)
::: lancedb.connect_namespace_async
::: lancedb.namespace.AsyncLanceNamespaceDBConnection
Tables (Asynchronous)
Table hold your actual data as a collection of records / rows.
::: lancedb.table.AsyncTable
::: lancedb.table.AsyncTags
::: lancedb.table.AsyncBranches
Indices (Asynchronous)
Indices can be created on a table to speed up queries. This section lists the indices that LanceDb supports.
::: lancedb.index
options:
show_root_heading: false
show_root_toc_entry: false
# lang_mapping is defined in the module rather than imported, so it is
# picked up despite not being in __all__. It is an internal lookup table.
filters: ["!^_", "!^lang_mapping$"]
::: lancedb.table.IndexStatistics
Querying (Asynchronous)
Queries allow you to return data from your database. Basic queries can be created with the [AsyncTable.query][lancedb.table.AsyncTable.query] method to return the entire (typically filtered) table. Vector searches return the rows nearest to a query vector and can be created with the [AsyncTable.vector_search][lancedb.table.AsyncTable.vector_search] method.
::: lancedb.query.AsyncQuery options: inherited_members: true
::: lancedb.query.AsyncVectorQuery options: inherited_members: true
::: lancedb.query.AsyncFTSQuery options: inherited_members: true
::: lancedb.query.AsyncHybridQuery options: inherited_members: true
::: lancedb.query.AsyncTakeQuery options: inherited_members: true