The bindings are built, installed and published with pnpm everywhere,
but a parallel npm dependency graph was still being maintained beside
it. This removes it, raises the supported Node floor to the versions we
actually test, and gives Dependabot the npm coverage it was missing.
## Dropping npm
`nodejs/package-lock.json` was regenerated by `ci/update_lockfiles.sh`
on every release commit and read by nothing — no workflow runs `npm ci`
or `npm install` in `nodejs/`, and npm never publishes a lockfile in a
package tarball. It could not even agree with the real install, since
npm does not see pnpm's `overrides`. Because GitHub's dependency graph
parses `package-lock.json`, it was also reporting vulnerabilities for a
tree we neither install nor ship.
`docs/package.json`, `docs/package-lock.json` and `docs/tsconfig.json`
go too. They depend on `file:../node` and
`file:../node/node_modules/apache-arrow` — the `node/` directory was
removed long ago — the tsconfig compiles `src/*.ts` where no TypeScript
files exist, and nothing installs any of it. `docs.yml` only referenced
the lockfile to configure an npm cache for an install it never ran.
Two `workflow_dispatch` workflows for regenerating those lockfiles are
removed as well. Both were already broken: they `uses:` composite
actions at `.github/workflows/update_package_lock{,_nodejs}` that do not
exist, so dispatching either failed immediately.
The remaining `npx` calls become direct `node_modules/.bin/...`
invocations. These were already running locally installed binaries
rather than resolving anything, but naming the binary removes the npm
CLI from the loop and does not depend on which Node version is active.
`dev.yml`'s commitlint check was the last place doing real npm
dependency resolution — an unpinned `npm install
@commitlint/config-conventional` that also bypassed the
`minimumReleaseAge` hold configured for `nodejs/` — and is now a pinned
`pnpm dlx`.
## Node support
Node 18 and 20 both reached end-of-life, in April 2025 and April 2026.
The matrix moves to 22, 24 and 26, and `engines` rises from `>= 18` to
`>= 22` so the declared floor is one the matrix actually covers. Node 22
is LTS until April 2027; 24 is LTS; 26 is Current and becomes LTS in
October 2026.
This also removes the reason the workflows reached for `npx` in the
first place: pnpm 11 requires Node >= 22.13, which every matrix version
now satisfies.
The prebuilt-binary smoke test in `npm-publish.yml` moves from Node 20
to Node 22 — the floor, where a napi ABI problem would surface first —
rather than fanning out across all three, to keep the publish matrix
from tripling.
## Dependabot
There were no npm-ecosystem entries at all, which is why the advisories
behind #4073 went unnoticed. Both pnpm lockfiles are now watched —
`nodejs/` and `nodejs/examples/`, which is a separate install — using
the same `lockfile-only` strategy as the existing cargo and pip entries,
so version ranges in `package.json` are left alone.
## Pre-commit biome
The hook ran `npx @biomejs/biome@1.8.3` while `nodejs/package.json`
resolved 1.9.4. The two disagree about formatting, so the hook rejected
code that `pnpm lint` accepts, and failed on unmodified `main` for
anyone touching `nodejs/`. It now uses the pnpm-managed biome, which
fixes the drift with no source changes.
## Testing
`dev.yml`'s commitlint job does not check out the repo, so it runs in an
empty workspace, and I could not verify `pnpm/action-setup` there
locally. It triggers on `pull_request_target`, so this PR exercises it
directly — worth confirming green before merge. I did verify the `pnpm
dlx` invocation itself locally: it accepts a conventional title and
rejects a non-conventional one with exit 1.
Node 26 is new enough that the examples job may surface gaps in prebuilt
native binaries (`onnxruntime-node`, `sharp`) before their maintainers
publish for it.
## Not included
`nodejs/examples/` still pins `sharp: "0.33.5"` and has its own audit
findings. Raising the Node floor unblocks that work — sharp 0.35
requires Node >= 20.9, which the matrix now satisfies — but it is a
dependency bump rather than tooling cleanup, so it is left separate.
## Breaking changes
`@lancedb/lancedb` now requires Node >= 22; previously >= 18. The
`@types/node` peer range moves from `>=18` to `>=22` to match. Users on
Node 18 or 20 must upgrade their runtime; both have been end-of-life for
some time. Existing installs are unaffected, since `engines` is only
checked on install.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@wjones127 is there a standard way you guys setup your virtualenv? I can
either relist all the dependencies in the pyright precommit section, or
specify a venv, or the user has to be in the virtual environment when
they run git commit. If the venv location was standardized or a python
manager like `uv` was used it would be easier to avoid duplicating the
pyright dependency list.
Per your suggestion, in `pyproject.toml` I added in all the passing
files to the `includes` section.
For ruff I upgraded the version and removed "TCH" which doesn't exist as
an option.
I added a `pyright_report.csv` which contains a list of all files sorted
by pyright errors ascending as a todo list to work on.
I fixed about 30 issues in `table.py` stemming from str's being passed
into methods that required a string within a set of string Literals by
extracting them into `types.py`
Can you verify in the rust bridge that the schema should be a property
and not a method here? If it's a method, then there's another place in
the code where `inner.schema` should be `inner.schema()`
``` python
class RecordBatchStream:
@property
def schema(self) -> pa.Schema: ...
```
Also unless the `_lancedb.pyi` file is wrong, then there is no
`__anext__` here for `__inner` when it's not an `AsyncGenerator` and
only `next` is defined:
``` python
async def __anext__(self) -> pa.RecordBatch:
return await self._inner.__anext__()
if isinstance(self._inner, AsyncGenerator):
batch = await self._inner.__anext__()
else:
batch = await self._inner.next()
if batch is None:
raise StopAsyncIteration
return batch
```
in the else statement, `_inner` is a `RecordBatchStream`
```python
class RecordBatchStream:
@property
def schema(self) -> pa.Schema: ...
async def next(self) -> Optional[pa.RecordBatch]: ...
```
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
This opens up the door for more custom database implementations than the
two we have today. The biggest change should be inivisble:
`ConnectionInternal` has been renamed to `Database`, made public, and
refactored
However, there are a few breaking changes. `data_storage_version` and
`enable_v2_manifest_paths` have been moved from options on
`create_table` to options for the database which are now set via
`storage_options`.
Before:
```
db = connect(uri)
tbl = db.create_table("my_table", data, data_storage_version="legacy", enable_v2_manifest_paths=True)
```
After:
```
db = connect(uri, storage_options={
"new_table_enable_v2_manifest_paths": "true",
"new_table_data_storage_version": "legacy"
})
tbl = db.create_table("my_table", data)
```
BREAKING CHANGE: the data_storage_version, enable_v2_manifest_paths
options have moved from options to create_table to storage_options.
BREAKING CHANGE: the use_legacy_format option has been removed,
data_storage_version has replaced it for some time now
This allows users to specify URIs like:
```
s3+ddb://my_bucket/path?ddbTableName=myCommitTable
```
and it will support concurrent writes in S3.
* [x] Add dynamodb integration tests
* [x] Add modifications to get it working in Python sync API
* [x] Added section in documentation describing how to configure.
Closes#534
---------
Co-authored-by: universalmind303 <cory.grinstead@gmail.com>
I've been noticing a lot of friction with the current toolchain for
'/nodejs'. Particularly with the usage of eslint and prettier.
[Biome](https://biomejs.dev/) is an all in one formatter & linter that
replaces the need for two different ones that can potentially clash with
one another.
I've been using it in the
[nodejs-polars](https://github.com/pola-rs/nodejs-polars) repo for quite
some time & have found it much more pleasant to work with.
---
One other small change included in this PR:
use [ts-jest](https://www.npmjs.com/package/ts-jest) so we can run our
tests without having to rebuild typescript code first
The synchronous table_names function in python lancedb relies on arrow's
filesystem which behaves slightly differently than object_store. As a
result, the function would not work properly in GCS.
However, the async table_names function uses object_store directly and
thus is accurate. In most cases we can fallback to using the async
table_names function and so this PR does so. The one case we cannot is
if the user is already in an async context (we can't start a new async
event loop). Soon, we can just redirect those users to use the async API
instead of the sync API and so that case will eventually go away. For
now, we fallback to the old behavior.
The eslint rules specify some formatting requirements that are rather
strict and conflict with vscode's default formatter. I was unable to get
auto-formatting to setup correctly. Also, eslint has quite recently
[given up on
formatting](https://eslint.org/blog/2023/10/deprecating-formatting-rules/)
and recommends using a 3rd party formatter.
This PR adds prettier as the formatter. It restores the eslint rules to
their defaults. This does mean we now have the "no explicit any" check
back on. I know that rule is pedantic but it did help me catch a few
corner cases in type testing that weren't covered in the current code.
Leaving in draft as this is dependent on other PRs.
Sometimes LangChain would insert a single `[np.nan]` as a placeholder if
the embedding function failed. This causes a problem for Lance format
because then the array can't be stored as a FixedSizedListArray.
Instead:
1. By default we remove rows with embedding lengths less than the
maximum length in the batch
2. If `strict=True` kwargs is set to True, then a `ValueError` is raised
if the embeddings aren't all the same length
---------
Co-authored-by: Chang She <chang@lancedb.com>
Will Jones
msu-reevo
Weston Pace
Cory Grinstead
Chang She