`docs/src/python/python.md` is the whole Python API reference, but it is
maintained by hand and had drifted from the public API. Anything not
listed there simply doesn't get rendered, so a number of public,
documented, tested APIs were invisible to users — most notably branch
management, where `diff` and `merge` live.
I audited every public symbol reachable from `lancedb` and its
subpackages against the `:::` directives on the page. This adds the
missing ones:
- **Branching** — `Branches`, `AsyncBranches` (`list` / `create` /
`checkout` / `delete` / `diff` / `merge`)
- **Tables** — `TableStatistics` (returned by `Table.stats()`; the
fragment-level stats classes were already listed)
- **Full text queries** — `FullTextQuery`, `MatchQuery`, `PhraseQuery`,
`BoostQuery`, `MultiMatchQuery`, `BooleanQuery`, `FullTextOperator`,
`Occur`
- **Querying** — `LanceEmptyQueryBuilder`, `LanceTakeQueryBuilder`,
`AsyncTakeQuery`
- **Indices** — `Fm` (the FM-index for substring search), `IndexConfig`
- **Blobs** — `blob`, `BlobType`, `BlobFile`
- **Namespaces** — `connect_namespace`, `connect_namespace_async`, and
both namespace connection classes
- **Remote config** — `TlsConfig`, `HeaderProvider`, `OAuthConfig`,
`OAuthFlowType`
- **Rerankers** — the `Reranker` base class plus `JinaReranker`,
`RRFReranker`, `MRRReranker`, `AnswerdotaiRerankers`,
`VoyageAIReranker`, `WatsonxReranker` (5 of 12 were listed)
- **Embeddings** — `get_registry`, `register`, and the 14 embedding
functions that were missing (3 of 17 were listed)
- **PyTorch** — `StreamingDataset` and the permutation API it is built
on
- **Misc** — `Session`, `tokenize`, `FtsToken`, `pydantic.Vector`,
`pydantic.MultiVector`, `instrument_lancedb_metrics`, and the two
exception types
It also repairs cross-references in docstrings that no longer resolve:
links into guide pages that have since moved to lancedb.com
(`querying-an-ann-index`, `experimental-full-text-search`),
`lance.dataset` references with no inventory behind them, and the
relative targets `[Table](Table)` and `[PyArrow Table](pyarrow.Table)`.
Deliberately left out: concrete implementation classes reached through
their abstract base (`LanceTable`, `LanceDBConnection`,
`RemoteDBConnection`), query base classes already covered by
`inherited_members: true`, and internal plumbing such as
`FullTextSearchQuery` and `ColumnOrdering`.
## Testing
The docs job only runs on pushes to `main`, so I built the site locally
and compared against a build of `upstream/main`: every added entry
resolves, and no symbol that was rendered before stopped being rendered
when the four packages moved to automodule. `mkdocs build --strict`
exits 0 on this branch, against 61 warnings on `main`.
## Also in this PR
`lancedb.index`, `lancedb.embeddings`, `lancedb.remote` and
`lancedb.rerankers` are now rendered by a single mkdocstrings directive
each, driven by the module's `__all__`, rather than a hand-maintained
list. These four are where most of the drift was, and `__all__` is
harder to forget than a docs page. `lancedb.embeddings` had no
`__all__`; without one mkdocstrings renders no members at all for a
re-export package, so one is added. AGENTS.md gains a section on how the
page is wired up and how to build the docs locally.
Rendering all that code for the first time surfaced ~100 more build
warnings, which would have made #3707 (turning on `mkdocs build
--strict`) harder to land, so the warning backlog is cleared here too.
97 of the 158 warnings were one systematic false positive — griffe
cannot see the generated `__init__` of a pydantic dataclass, so every
documented parameter looks unknown — switched off via
`warn_unknown_params`. The remaining 61 came from 15 docstrings with
real bugs: prose trailing a `Parameters` section (we were rendering
parameters called `The`, `you` and `To`), types dropped because numpydoc
needs spaces around the colon, `num_partitions, default sqrt(num_rows)`
parsing as a list of names and inventing a `default` parameter, and one
parameter indented five spaces. `mkdocs build --strict` now exits 0.
---
#3747 (the coverage test that keeps this from happening again) is
stacked on this branch, so review it after this one.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
Switch the nodejs bindings and examples package from npm to pnpm 11 to
pick up its stronger supply-chain defaults:
- `minimumReleaseAge` defaults to 1 day, so newly-published (potentially
compromised) versions aren't resolved into installs for at least 24h.
- Install lifecycle scripts (`preinstall`/`install`/`postinstall`) are
no longer run for arbitrary transitive deps; only an explicit allowlist
may run them, and unapproved scripts cause install to fail
(`strictDepBuilds: true`).
- Audit uses GHSA IDs and `--fix=update` to add patched versions to
`minimumReleaseAgeExclude`.
This is the same class of protection that would have blunted the recent
TanStack/`@uipath`/etc. compromise discussed in the [Aikido
write-up](https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised).
## Changes
- Replace `nodejs/package-lock.json` and
`nodejs/examples/package-lock.json` with `pnpm-lock.yaml`.
- Pin pnpm via `packageManager: pnpm@11.1.1` in both `package.json`s.
- Add `pnpm-workspace.yaml` with the four build-script packages we
actually need: `@biomejs/biome`, `onnxruntime-node`, `protobufjs`,
`sharp`. Everything else is blocked from running install scripts.
- Update package.json scripts (`npm run X` → `pnpm X`).
- Update workflows: `.github/workflows/nodejs.yml`,
`.github/workflows/npm-publish.yml`, and
`.github/workflows/codex-fix-ci.yml` — install pnpm via
`pnpm/action-setup@v4` and switch `setup-node` caches to
`pnpm-lock.yaml`.
- Refresh `nodejs/AGENTS.md`, `nodejs/CLAUDE.md`, and
`nodejs/CONTRIBUTING.md`.
`docs/package-lock.json` is **not** touched — out of scope for this PR.
## Test plan
- [ ] `Lint` job (lint Rust/TS + examples lint) passes on CI.
- [ ] `Linux (NodeJS 18/20)` build+test passes, including the examples
test step.
- [ ] `macos` build+test passes.
- [ ] `NPM Publish` workflow's PR dry-run completes (build matrix + test
matrix + dry `npm publish`).
- [ ] No new install-script approvals are required at install time.
## Follow-ups
- `update_package_lock_run_nodejs.yml` references a composite action
path that doesn't exist
(`./.github/workflows/update_package_lock_nodejs`); it was already
broken pre-PR. We may want to either delete this workflow or rewrite it
for pnpm in a follow-up.
- Consider migrating `docs/` to pnpm in a separate PR.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Will Jones