mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 12:08:35 +00:00
Will Jones
5a1015ba72
`docs/src/python/python.md` is the whole Python API reference, but it is maintained by hand and had drifted from the public API. Anything not listed there simply doesn't get rendered, so a number of public, documented, tested APIs were invisible to users — most notably branch management, where `diff` and `merge` live. I audited every public symbol reachable from `lancedb` and its subpackages against the `:::` directives on the page. This adds the missing ones: - **Branching** — `Branches`, `AsyncBranches` (`list` / `create` / `checkout` / `delete` / `diff` / `merge`) - **Tables** — `TableStatistics` (returned by `Table.stats()`; the fragment-level stats classes were already listed) - **Full text queries** — `FullTextQuery`, `MatchQuery`, `PhraseQuery`, `BoostQuery`, `MultiMatchQuery`, `BooleanQuery`, `FullTextOperator`, `Occur` - **Querying** — `LanceEmptyQueryBuilder`, `LanceTakeQueryBuilder`, `AsyncTakeQuery` - **Indices** — `Fm` (the FM-index for substring search), `IndexConfig` - **Blobs** — `blob`, `BlobType`, `BlobFile` - **Namespaces** — `connect_namespace`, `connect_namespace_async`, and both namespace connection classes - **Remote config** — `TlsConfig`, `HeaderProvider`, `OAuthConfig`, `OAuthFlowType` - **Rerankers** — the `Reranker` base class plus `JinaReranker`, `RRFReranker`, `MRRReranker`, `AnswerdotaiRerankers`, `VoyageAIReranker`, `WatsonxReranker` (5 of 12 were listed) - **Embeddings** — `get_registry`, `register`, and the 14 embedding functions that were missing (3 of 17 were listed) - **PyTorch** — `StreamingDataset` and the permutation API it is built on - **Misc** — `Session`, `tokenize`, `FtsToken`, `pydantic.Vector`, `pydantic.MultiVector`, `instrument_lancedb_metrics`, and the two exception types It also repairs cross-references in docstrings that no longer resolve: links into guide pages that have since moved to lancedb.com (`querying-an-ann-index`, `experimental-full-text-search`), `lance.dataset` references with no inventory behind them, and the relative targets `[Table](Table)` and `[PyArrow Table](pyarrow.Table)`. Deliberately left out: concrete implementation classes reached through their abstract base (`LanceTable`, `LanceDBConnection`, `RemoteDBConnection`), query base classes already covered by `inherited_members: true`, and internal plumbing such as `FullTextSearchQuery` and `ColumnOrdering`. ## Testing The docs job only runs on pushes to `main`, so I built the site locally and compared against a build of `upstream/main`: every added entry resolves, and no symbol that was rendered before stopped being rendered when the four packages moved to automodule. `mkdocs build --strict` exits 0 on this branch, against 61 warnings on `main`. ## Also in this PR `lancedb.index`, `lancedb.embeddings`, `lancedb.remote` and `lancedb.rerankers` are now rendered by a single mkdocstrings directive each, driven by the module's `__all__`, rather than a hand-maintained list. These four are where most of the drift was, and `__all__` is harder to forget than a docs page. `lancedb.embeddings` had no `__all__`; without one mkdocstrings renders no members at all for a re-export package, so one is added. AGENTS.md gains a section on how the page is wired up and how to build the docs locally. Rendering all that code for the first time surfaced ~100 more build warnings, which would have made #3707 (turning on `mkdocs build --strict`) harder to land, so the warning backlog is cleared here too. 97 of the 158 warnings were one systematic false positive — griffe cannot see the generated `__init__` of a pydantic dataclass, so every documented parameter looks unknown — switched off via `warn_unknown_params`. The remaining 61 came from 15 docstrings with real bugs: prose trailing a `Parameters` section (we were rendering parameters called `The`, `you` and `To`), types dropped because numpydoc needs spaces around the colon, `num_partitions, default sqrt(num_rows)` parsing as a list of names and inventing a `default` parameter, and one parameter indented five spaces. `mkdocs build --strict` now exits 0. --- #3747 (the coverage test that keeps this from happening again) is stacked on this branch, so review it after this one. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
79 lines
1.8 KiB
Markdown
79 lines
1.8 KiB
Markdown
# Contributing to LanceDB Typescript
|
|
|
|
This document outlines the process for contributing to LanceDB Typescript.
|
|
For general contribution guidelines, see [CONTRIBUTING.md](https://github.com/lancedb/lancedb/blob/main/CONTRIBUTING.md).
|
|
|
|
## Project layout
|
|
|
|
The Typescript package is a wrapper around the Rust library, `lancedb`. We use
|
|
the [napi-rs](https://napi.rs/) library to create the bindings between Rust and
|
|
Typescript.
|
|
|
|
* `src/`: Rust bindings source code
|
|
* `lancedb/`: Typescript package source code
|
|
* `__test__/`: Unit tests
|
|
* `examples/`: A pnpm package with the examples shown in the documentation
|
|
|
|
## Development environment
|
|
|
|
To set up your development environment, you will need to install the following:
|
|
|
|
1. Node.js 22 or later (required by pnpm 11)
|
|
2. [pnpm](https://pnpm.io/installation) 11 or later (or run via `corepack enable`,
|
|
which uses the `packageManager` field in `package.json`)
|
|
3. Rust's package manager, Cargo. Use [rustup](https://rustup.rs/) to install.
|
|
4. [protoc](https://grpc.io/docs/protoc-installation/) (Protocol Buffers compiler)
|
|
|
|
Initial setup:
|
|
|
|
```shell
|
|
pnpm install
|
|
```
|
|
|
|
### Commit Hooks
|
|
|
|
It is **highly recommended** to install the [pre-commit](https://pre-commit.com/) hooks to ensure that your
|
|
code is formatted correctly and passes basic checks before committing:
|
|
|
|
```shell
|
|
pre-commit install
|
|
```
|
|
|
|
## Development
|
|
|
|
Most common development commands can be run using the pnpm scripts.
|
|
|
|
Build the package
|
|
|
|
```shell
|
|
pnpm install
|
|
pnpm build
|
|
```
|
|
|
|
Lint:
|
|
|
|
```shell
|
|
pnpm lint
|
|
```
|
|
|
|
Format and fix lints:
|
|
|
|
```shell
|
|
pnpm lint-fix
|
|
```
|
|
|
|
Run tests:
|
|
|
|
```shell
|
|
pnpm test
|
|
```
|
|
|
|
To run a single test:
|
|
|
|
```shell
|
|
# Single file: table.test.ts
|
|
pnpm test -- table.test.ts
|
|
# Single test: 'merge insert' in table.test.ts
|
|
pnpm test -- table.test.ts --testNamePattern=merge\ insert
|
|
```
|