mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-29 14:56:51 +00:00
8887c82789c6e8db8f03848b66e0f6f58898c03e
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Xuanwo
|
8a4effa06a |
fix: gate experimental local functions behind an opt-in feature (#4688)
Published SDK builds currently expose local SQL computed columns and materialized-view execution that are not ready for release. Add the opt-in `experimental-functions` Cargo feature to the Rust core and forward it through the Python and Node bindings; published packages leave it disabled. Local declarations and refreshes now return `NotSupported` before mutating tables or spawning jobs, including prepared materialized-view creation. Remote Cloud/Enterprise client operations, Python UDF packaging, ordinary eager `add_columns`, and access to existing data remain available. Stored metadata and write protections stay active, and shared server commit helpers remain usable. CI covers both feature configurations. Existing local execution tests run with the feature enabled; default-build regressions cover early rejection, unchanged table versions, existing metadata protections, and normal eager transforms. API and source-build documentation explain the opt-in. Validation: Rust library tests pass with the feature disabled (1,869 tests) and enabled (1,947 tests), plus three default-build integration regressions. Python and TypeScript bindings were rebuilt and tested in both configurations, including mocked remote requests; Rust formatting, Python package lint, TypeScript lint/build, and generated docs were checked. Repository-wide Ruff still reports 48 pre-existing errors in unchanged `docs/web-tests` files. |
||
  Will JonesandClaude Opus 5
|
5a1015ba72 |
docs(python): fill gaps in the Python API reference (#3746)
`docs/src/python/python.md` is the whole Python API reference, but it is maintained by hand and had drifted from the public API. Anything not listed there simply doesn't get rendered, so a number of public, documented, tested APIs were invisible to users — most notably branch management, where `diff` and `merge` live. I audited every public symbol reachable from `lancedb` and its subpackages against the `:::` directives on the page. This adds the missing ones: - **Branching** — `Branches`, `AsyncBranches` (`list` / `create` / `checkout` / `delete` / `diff` / `merge`) - **Tables** — `TableStatistics` (returned by `Table.stats()`; the fragment-level stats classes were already listed) - **Full text queries** — `FullTextQuery`, `MatchQuery`, `PhraseQuery`, `BoostQuery`, `MultiMatchQuery`, `BooleanQuery`, `FullTextOperator`, `Occur` - **Querying** — `LanceEmptyQueryBuilder`, `LanceTakeQueryBuilder`, `AsyncTakeQuery` - **Indices** — `Fm` (the FM-index for substring search), `IndexConfig` - **Blobs** — `blob`, `BlobType`, `BlobFile` - **Namespaces** — `connect_namespace`, `connect_namespace_async`, and both namespace connection classes - **Remote config** — `TlsConfig`, `HeaderProvider`, `OAuthConfig`, `OAuthFlowType` - **Rerankers** — the `Reranker` base class plus `JinaReranker`, `RRFReranker`, `MRRReranker`, `AnswerdotaiRerankers`, `VoyageAIReranker`, `WatsonxReranker` (5 of 12 were listed) - **Embeddings** — `get_registry`, `register`, and the 14 embedding functions that were missing (3 of 17 were listed) - **PyTorch** — `StreamingDataset` and the permutation API it is built on - **Misc** — `Session`, `tokenize`, `FtsToken`, `pydantic.Vector`, `pydantic.MultiVector`, `instrument_lancedb_metrics`, and the two exception types It also repairs cross-references in docstrings that no longer resolve: links into guide pages that have since moved to lancedb.com (`querying-an-ann-index`, `experimental-full-text-search`), `lance.dataset` references with no inventory behind them, and the relative targets `[Table](Table)` and `[PyArrow Table](pyarrow.Table)`. Deliberately left out: concrete implementation classes reached through their abstract base (`LanceTable`, `LanceDBConnection`, `RemoteDBConnection`), query base classes already covered by `inherited_members: true`, and internal plumbing such as `FullTextSearchQuery` and `ColumnOrdering`. ## Testing The docs job only runs on pushes to `main`, so I built the site locally and compared against a build of `upstream/main`: every added entry resolves, and no symbol that was rendered before stopped being rendered when the four packages moved to automodule. `mkdocs build --strict` exits 0 on this branch, against 61 warnings on `main`. ## Also in this PR `lancedb.index`, `lancedb.embeddings`, `lancedb.remote` and `lancedb.rerankers` are now rendered by a single mkdocstrings directive each, driven by the module's `__all__`, rather than a hand-maintained list. These four are where most of the drift was, and `__all__` is harder to forget than a docs page. `lancedb.embeddings` had no `__all__`; without one mkdocstrings renders no members at all for a re-export package, so one is added. AGENTS.md gains a section on how the page is wired up and how to build the docs locally. Rendering all that code for the first time surfaced ~100 more build warnings, which would have made #3707 (turning on `mkdocs build --strict`) harder to land, so the warning backlog is cleared here too. 97 of the 158 warnings were one systematic false positive — griffe cannot see the generated `__init__` of a pydantic dataclass, so every documented parameter looks unknown — switched off via `warn_unknown_params`. The remaining 61 came from 15 docstrings with real bugs: prose trailing a `Parameters` section (we were rendering parameters called `The`, `you` and `To`), types dropped because numpydoc needs spaces around the colon, `num_partitions, default sqrt(num_rows)` parsing as a list of names and inventing a `default` parameter, and one parameter indented five spaces. `mkdocs build --strict` now exits 0. --- #3747 (the coverage test that keeps this from happening again) is stacked on this branch, so review it after this one. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Will JonesandClaude Opus 4.7
|
81617fd3d9 |
ci(nodejs): switch from npm to pnpm 11 (#3373)
## Summary Switch the nodejs bindings and examples package from npm to pnpm 11 to pick up its stronger supply-chain defaults: - `minimumReleaseAge` defaults to 1 day, so newly-published (potentially compromised) versions aren't resolved into installs for at least 24h. - Install lifecycle scripts (`preinstall`/`install`/`postinstall`) are no longer run for arbitrary transitive deps; only an explicit allowlist may run them, and unapproved scripts cause install to fail (`strictDepBuilds: true`). - Audit uses GHSA IDs and `--fix=update` to add patched versions to `minimumReleaseAgeExclude`. This is the same class of protection that would have blunted the recent TanStack/`@uipath`/etc. compromise discussed in the [Aikido write-up](https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised). ## Changes - Replace `nodejs/package-lock.json` and `nodejs/examples/package-lock.json` with `pnpm-lock.yaml`. - Pin pnpm via `packageManager: pnpm@11.1.1` in both `package.json`s. - Add `pnpm-workspace.yaml` with the four build-script packages we actually need: `@biomejs/biome`, `onnxruntime-node`, `protobufjs`, `sharp`. Everything else is blocked from running install scripts. - Update package.json scripts (`npm run X` → `pnpm X`). - Update workflows: `.github/workflows/nodejs.yml`, `.github/workflows/npm-publish.yml`, and `.github/workflows/codex-fix-ci.yml` — install pnpm via `pnpm/action-setup@v4` and switch `setup-node` caches to `pnpm-lock.yaml`. - Refresh `nodejs/AGENTS.md`, `nodejs/CLAUDE.md`, and `nodejs/CONTRIBUTING.md`. `docs/package-lock.json` is **not** touched — out of scope for this PR. ## Test plan - [ ] `Lint` job (lint Rust/TS + examples lint) passes on CI. - [ ] `Linux (NodeJS 18/20)` build+test passes, including the examples test step. - [ ] `macos` build+test passes. - [ ] `NPM Publish` workflow's PR dry-run completes (build matrix + test matrix + dry `npm publish`). - [ ] No new install-script approvals are required at install time. ## Follow-ups - `update_package_lock_run_nodejs.yml` references a composite action path that doesn't exist (`./.github/workflows/update_package_lock_nodejs`); it was already broken pre-PR. We may want to either delete this workflow or rewrite it for pnpm in a follow-up. - Consider migrating `docs/` to pnpm in a separate PR. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Will Jones
|
0e496ed3b5 |
docs: contributing guide (#1970)
* Adds basic contributing guides. * Simplifies Python development with a Makefile. |