mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-27 16:38:31 +00:00
Will Jones
5a1015ba72
`docs/src/python/python.md` is the whole Python API reference, but it is maintained by hand and had drifted from the public API. Anything not listed there simply doesn't get rendered, so a number of public, documented, tested APIs were invisible to users — most notably branch management, where `diff` and `merge` live. I audited every public symbol reachable from `lancedb` and its subpackages against the `:::` directives on the page. This adds the missing ones: - **Branching** — `Branches`, `AsyncBranches` (`list` / `create` / `checkout` / `delete` / `diff` / `merge`) - **Tables** — `TableStatistics` (returned by `Table.stats()`; the fragment-level stats classes were already listed) - **Full text queries** — `FullTextQuery`, `MatchQuery`, `PhraseQuery`, `BoostQuery`, `MultiMatchQuery`, `BooleanQuery`, `FullTextOperator`, `Occur` - **Querying** — `LanceEmptyQueryBuilder`, `LanceTakeQueryBuilder`, `AsyncTakeQuery` - **Indices** — `Fm` (the FM-index for substring search), `IndexConfig` - **Blobs** — `blob`, `BlobType`, `BlobFile` - **Namespaces** — `connect_namespace`, `connect_namespace_async`, and both namespace connection classes - **Remote config** — `TlsConfig`, `HeaderProvider`, `OAuthConfig`, `OAuthFlowType` - **Rerankers** — the `Reranker` base class plus `JinaReranker`, `RRFReranker`, `MRRReranker`, `AnswerdotaiRerankers`, `VoyageAIReranker`, `WatsonxReranker` (5 of 12 were listed) - **Embeddings** — `get_registry`, `register`, and the 14 embedding functions that were missing (3 of 17 were listed) - **PyTorch** — `StreamingDataset` and the permutation API it is built on - **Misc** — `Session`, `tokenize`, `FtsToken`, `pydantic.Vector`, `pydantic.MultiVector`, `instrument_lancedb_metrics`, and the two exception types It also repairs cross-references in docstrings that no longer resolve: links into guide pages that have since moved to lancedb.com (`querying-an-ann-index`, `experimental-full-text-search`), `lance.dataset` references with no inventory behind them, and the relative targets `[Table](Table)` and `[PyArrow Table](pyarrow.Table)`. Deliberately left out: concrete implementation classes reached through their abstract base (`LanceTable`, `LanceDBConnection`, `RemoteDBConnection`), query base classes already covered by `inherited_members: true`, and internal plumbing such as `FullTextSearchQuery` and `ColumnOrdering`. ## Testing The docs job only runs on pushes to `main`, so I built the site locally and compared against a build of `upstream/main`: every added entry resolves, and no symbol that was rendered before stopped being rendered when the four packages moved to automodule. `mkdocs build --strict` exits 0 on this branch, against 61 warnings on `main`. ## Also in this PR `lancedb.index`, `lancedb.embeddings`, `lancedb.remote` and `lancedb.rerankers` are now rendered by a single mkdocstrings directive each, driven by the module's `__all__`, rather than a hand-maintained list. These four are where most of the drift was, and `__all__` is harder to forget than a docs page. `lancedb.embeddings` had no `__all__`; without one mkdocstrings renders no members at all for a re-export package, so one is added. AGENTS.md gains a section on how the page is wired up and how to build the docs locally. Rendering all that code for the first time surfaced ~100 more build warnings, which would have made #3707 (turning on `mkdocs build --strict`) harder to land, so the warning backlog is cleared here too. 97 of the 158 warnings were one systematic false positive — griffe cannot see the generated `__init__` of a pydantic dataclass, so every documented parameter looks unknown — switched off via `warn_unknown_params`. The remaining 61 came from 15 docstrings with real bugs: prose trailing a `Parameters` section (we were rendering parameters called `The`, `you` and `To`), types dropped because numpydoc needs spaces around the colon, `num_partitions, default sqrt(num_rows)` parsing as a list of names and inventing a `default` parameter, and one parameter indented five spaces. `mkdocs build --strict` now exits 0. --- #3747 (the coverage test that keeps this from happening again) is stacked on this branch, so review it after this one. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
290 lines
6.2 KiB
Markdown
290 lines
6.2 KiB
Markdown
# Python API Reference
|
|
|
|
This section contains the API reference for the Python API of [LanceDB](https://github.com/lancedb/lancedb). Both synchronous and asynchronous APIs are available.
|
|
|
|
The general flow of using the API is:
|
|
|
|
1. Use [lancedb.connect][] or [lancedb.connect_async][] to connect to a database.
|
|
2. Use the returned [lancedb.DBConnection][] or [lancedb.AsyncConnection][] to
|
|
create or open tables.
|
|
3. Use the returned [lancedb.table.Table][] or [lancedb.AsyncTable][] to query
|
|
or modify tables.
|
|
|
|
|
|
## Installation
|
|
|
|
```shell
|
|
pip install lancedb
|
|
```
|
|
|
|
The following methods describe the synchronous API client. There
|
|
is also an [asynchronous API client](#connections-asynchronous).
|
|
|
|
## Connections (Synchronous)
|
|
|
|
::: lancedb.connect
|
|
|
|
::: lancedb.db.DBConnection
|
|
|
|
::: lancedb.Session
|
|
|
|
## Namespaces (Synchronous)
|
|
|
|
A namespace-backed connection resolves tables through a
|
|
[Lance namespace](https://lancedb.github.io/lance-namespace/) service instead of
|
|
listing a storage directory.
|
|
|
|
::: lancedb.connect_namespace
|
|
|
|
::: lancedb.namespace.LanceNamespaceDBConnection
|
|
|
|
## Tables (Synchronous)
|
|
|
|
::: lancedb.table.Table
|
|
|
|
::: lancedb.table.FragmentStatistics
|
|
|
|
::: lancedb.table.FragmentSummaryStats
|
|
|
|
::: lancedb.table.TableStatistics
|
|
|
|
::: lancedb.table.Tags
|
|
|
|
::: lancedb.table.Branches
|
|
|
|
## Expressions
|
|
|
|
Type-safe expression builder for filters and projections. Use these instead
|
|
of raw SQL strings with [where][lancedb.query.LanceQueryBuilder.where] and
|
|
[select][lancedb.query.LanceQueryBuilder.select].
|
|
|
|
::: lancedb.expr.Expr
|
|
|
|
::: lancedb.expr.col
|
|
|
|
::: lancedb.expr.lit
|
|
|
|
::: lancedb.expr.func
|
|
|
|
## Querying (Synchronous)
|
|
|
|
::: lancedb.query.Query
|
|
|
|
::: lancedb.query.LanceQueryBuilder
|
|
|
|
::: lancedb.query.LanceVectorQueryBuilder
|
|
|
|
::: lancedb.query.LanceFtsQueryBuilder
|
|
|
|
::: lancedb.query.LanceHybridQueryBuilder
|
|
|
|
::: lancedb.query.LanceEmptyQueryBuilder
|
|
|
|
::: lancedb.query.LanceTakeQueryBuilder
|
|
|
|
## Full text queries
|
|
|
|
Structured full text queries can be passed to
|
|
[Table.search][lancedb.table.Table.search] or
|
|
[AsyncTable.search][lancedb.table.AsyncTable.search] in place of a query string,
|
|
and combined with [BooleanQuery][lancedb.query.BooleanQuery].
|
|
|
|
::: lancedb.query.FullTextQuery
|
|
|
|
::: lancedb.query.MatchQuery
|
|
|
|
::: lancedb.query.PhraseQuery
|
|
|
|
::: lancedb.query.BoostQuery
|
|
|
|
::: lancedb.query.MultiMatchQuery
|
|
|
|
::: lancedb.query.BooleanQuery
|
|
|
|
::: lancedb.query.FullTextOperator
|
|
|
|
::: lancedb.query.Occur
|
|
|
|
## Embeddings
|
|
|
|
::: lancedb.embeddings
|
|
options:
|
|
show_root_heading: false
|
|
show_root_toc_entry: false
|
|
|
|
## Remote configuration
|
|
|
|
::: lancedb.remote
|
|
options:
|
|
show_root_heading: false
|
|
show_root_toc_entry: false
|
|
|
|
## Context
|
|
|
|
::: lancedb.context.contextualize
|
|
|
|
::: lancedb.context.Contextualizer
|
|
|
|
## Full text search
|
|
|
|
Pass `custom_stop_words` to [lancedb.index.FTS][]:
|
|
|
|
```python
|
|
from lancedb.index import FTS
|
|
|
|
table.create_index(
|
|
"text",
|
|
config=FTS(remove_stop_words=True, custom_stop_words=["acme", "internal"]),
|
|
)
|
|
```
|
|
|
|
The list replaces the built-in stop words and is used only when
|
|
`remove_stop_words=True`:
|
|
|
|
- `custom_stop_words=None` uses the built-in list for `language`.
|
|
- `custom_stop_words=[]` removes no words.
|
|
- Values are passed through without trimming, lowercasing, or other rewriting.
|
|
|
|
The same option is available on `lancedb.tokenize(...)` and the deprecated
|
|
[lancedb.table.Table.create_fts_index][] compatibility helper:
|
|
|
|
```python
|
|
import lancedb
|
|
|
|
tokens = list(lancedb.tokenize("acme makes searchable data",
|
|
custom_stop_words=["acme"]))
|
|
```
|
|
|
|
::: lancedb.tokenize
|
|
|
|
::: lancedb.FtsToken
|
|
|
|
## Blobs
|
|
|
|
Blob columns store large binary values out of line so they can be read lazily
|
|
instead of being materialized with the rest of the row.
|
|
|
|
::: lancedb.blob
|
|
|
|
::: lancedb.BlobType
|
|
|
|
::: lancedb._blob.BlobFile
|
|
options:
|
|
show_root_full_path: false
|
|
|
|
## Utilities
|
|
|
|
::: lancedb.schema.vector
|
|
|
|
::: lancedb.merge.LanceMergeInsertBuilder
|
|
|
|
::: lancedb.otel.instrument_lancedb_metrics
|
|
|
|
## Exceptions
|
|
|
|
::: lancedb.exceptions.MissingValueError
|
|
|
|
::: lancedb.exceptions.MissingColumnError
|
|
|
|
## Integrations
|
|
|
|
## Pydantic
|
|
|
|
::: lancedb.pydantic.pydantic_to_schema
|
|
|
|
::: lancedb.pydantic.vector
|
|
|
|
::: lancedb.pydantic.Vector
|
|
|
|
::: lancedb.pydantic.MultiVector
|
|
|
|
::: lancedb.pydantic.LanceModel
|
|
|
|
## PyTorch
|
|
|
|
::: lancedb.streaming.StreamingDataset
|
|
|
|
::: lancedb.permutation.permutation_builder
|
|
|
|
::: lancedb.permutation.PermutationBuilder
|
|
|
|
::: lancedb.permutation.Permutation
|
|
|
|
::: lancedb.permutation.Transforms
|
|
|
|
## Reranking
|
|
|
|
::: lancedb.rerankers
|
|
options:
|
|
show_root_heading: false
|
|
show_root_toc_entry: false
|
|
|
|
## Connections (Asynchronous)
|
|
|
|
Connections represent a connection to a LanceDb database and
|
|
can be used to create, list, or open tables.
|
|
|
|
::: lancedb.connect_async
|
|
|
|
::: lancedb.db.AsyncConnection
|
|
|
|
## Namespaces (Asynchronous)
|
|
|
|
::: lancedb.connect_namespace_async
|
|
|
|
::: lancedb.namespace.AsyncLanceNamespaceDBConnection
|
|
|
|
## Tables (Asynchronous)
|
|
|
|
Table hold your actual data as a collection of records / rows.
|
|
|
|
::: lancedb.table.AsyncTable
|
|
|
|
::: lancedb.table.AsyncTags
|
|
|
|
::: lancedb.table.AsyncBranches
|
|
|
|
## Indices (Asynchronous)
|
|
|
|
Indices can be created on a table to speed up queries. This section
|
|
lists the indices that LanceDb supports.
|
|
|
|
::: lancedb.index
|
|
options:
|
|
show_root_heading: false
|
|
show_root_toc_entry: false
|
|
# `lang_mapping` is defined in the module rather than imported, so it is
|
|
# picked up despite not being in `__all__`. It is an internal lookup table.
|
|
filters: ["!^_", "!^lang_mapping$"]
|
|
|
|
::: lancedb.table.IndexStatistics
|
|
|
|
## Querying (Asynchronous)
|
|
|
|
Queries allow you to return data from your database. Basic queries can be
|
|
created with the [AsyncTable.query][lancedb.table.AsyncTable.query] method
|
|
to return the entire (typically filtered) table. Vector searches return the
|
|
rows nearest to a query vector and can be created with the
|
|
[AsyncTable.vector_search][lancedb.table.AsyncTable.vector_search] method.
|
|
|
|
|
|
::: lancedb.query.AsyncQuery
|
|
options:
|
|
inherited_members: true
|
|
|
|
::: lancedb.query.AsyncVectorQuery
|
|
options:
|
|
inherited_members: true
|
|
|
|
::: lancedb.query.AsyncFTSQuery
|
|
options:
|
|
inherited_members: true
|
|
|
|
::: lancedb.query.AsyncHybridQuery
|
|
options:
|
|
inherited_members: true
|
|
|
|
::: lancedb.query.AsyncTakeQuery
|
|
options:
|
|
inherited_members: true
|