mirror of
https://github.com/lancedb/lancedb.git
synced 2026-05-14 18:40:39 +00:00
ci(nodejs): switch from npm to pnpm 11 (#3373)
## Summary Switch the nodejs bindings and examples package from npm to pnpm 11 to pick up its stronger supply-chain defaults: - `minimumReleaseAge` defaults to 1 day, so newly-published (potentially compromised) versions aren't resolved into installs for at least 24h. - Install lifecycle scripts (`preinstall`/`install`/`postinstall`) are no longer run for arbitrary transitive deps; only an explicit allowlist may run them, and unapproved scripts cause install to fail (`strictDepBuilds: true`). - Audit uses GHSA IDs and `--fix=update` to add patched versions to `minimumReleaseAgeExclude`. This is the same class of protection that would have blunted the recent TanStack/`@uipath`/etc. compromise discussed in the [Aikido write-up](https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised). ## Changes - Replace `nodejs/package-lock.json` and `nodejs/examples/package-lock.json` with `pnpm-lock.yaml`. - Pin pnpm via `packageManager: pnpm@11.1.1` in both `package.json`s. - Add `pnpm-workspace.yaml` with the four build-script packages we actually need: `@biomejs/biome`, `onnxruntime-node`, `protobufjs`, `sharp`. Everything else is blocked from running install scripts. - Update package.json scripts (`npm run X` → `pnpm X`). - Update workflows: `.github/workflows/nodejs.yml`, `.github/workflows/npm-publish.yml`, and `.github/workflows/codex-fix-ci.yml` — install pnpm via `pnpm/action-setup@v4` and switch `setup-node` caches to `pnpm-lock.yaml`. - Refresh `nodejs/AGENTS.md`, `nodejs/CLAUDE.md`, and `nodejs/CONTRIBUTING.md`. `docs/package-lock.json` is **not** touched — out of scope for this PR. ## Test plan - [ ] `Lint` job (lint Rust/TS + examples lint) passes on CI. - [ ] `Linux (NodeJS 18/20)` build+test passes, including the examples test step. - [ ] `macos` build+test passes. - [ ] `NPM Publish` workflow's PR dry-run completes (build matrix + test matrix + dry `npm publish`). - [ ] No new install-script approvals are required at install time. ## Follow-ups - `update_package_lock_run_nodejs.yml` references a composite action path that doesn't exist (`./.github/workflows/update_package_lock_nodejs`); it was already broken pre-PR. We may want to either delete this workflow or rewrite it for pnpm in a follow-up. - Consider migrating `docs/` to pnpm in a separate PR. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Will Jones
GitHub
@@ -3,11 +3,11 @@ The core Rust library is in the `../rust/lancedb` directory, the rust binding
|
||||
code is in the `src/` directory and the typescript bindings are in
|
||||
the `lancedb/` directory.
|
||||
|
||||
Whenever you change the Rust code, you will need to recompile: `npm run build`.
|
||||
Whenever you change the Rust code, you will need to recompile: `pnpm build`.
|
||||
|
||||
Common commands:
|
||||
* Build: `npm run build`
|
||||
* Lint: `npm run lint`
|
||||
* Fix lints: `npm run lint-fix`
|
||||
* Test: `npm test`
|
||||
* Run single test file: `npm test __test__/arrow.test.ts`
|
||||
* Build: `pnpm build`
|
||||
* Lint: `pnpm lint`
|
||||
* Fix lints: `pnpm lint-fix`
|
||||
* Test: `pnpm test`
|
||||
* Run single test file: `pnpm test __test__/arrow.test.ts`
|
||||
|
||||
@@ -12,20 +12,22 @@ Typescript.
|
||||
* `src/`: Rust bindings source code
|
||||
* `lancedb/`: Typescript package source code
|
||||
* `__test__/`: Unit tests
|
||||
* `examples/`: An npm package with the examples shown in the documentation
|
||||
* `examples/`: A pnpm package with the examples shown in the documentation
|
||||
|
||||
## Development environment
|
||||
|
||||
To set up your development environment, you will need to install the following:
|
||||
|
||||
1. Node.js 14 or later
|
||||
2. Rust's package manager, Cargo. Use [rustup](https://rustup.rs/) to install.
|
||||
3. [protoc](https://grpc.io/docs/protoc-installation/) (Protocol Buffers compiler)
|
||||
1. Node.js 22 or later (required by pnpm 11)
|
||||
2. [pnpm](https://pnpm.io/installation) 11 or later (or run via `corepack enable`,
|
||||
which uses the `packageManager` field in `package.json`)
|
||||
3. Rust's package manager, Cargo. Use [rustup](https://rustup.rs/) to install.
|
||||
4. [protoc](https://grpc.io/docs/protoc-installation/) (Protocol Buffers compiler)
|
||||
|
||||
Initial setup:
|
||||
|
||||
```shell
|
||||
npm install
|
||||
pnpm install
|
||||
```
|
||||
|
||||
### Commit Hooks
|
||||
@@ -39,38 +41,38 @@ pre-commit install
|
||||
|
||||
## Development
|
||||
|
||||
Most common development commands can be run using the npm scripts.
|
||||
Most common development commands can be run using the pnpm scripts.
|
||||
|
||||
Build the package
|
||||
|
||||
```shell
|
||||
npm install
|
||||
npm run build
|
||||
pnpm install
|
||||
pnpm build
|
||||
```
|
||||
|
||||
Lint:
|
||||
|
||||
```shell
|
||||
npm run lint
|
||||
pnpm lint
|
||||
```
|
||||
|
||||
Format and fix lints:
|
||||
|
||||
```shell
|
||||
npm run lint-fix
|
||||
pnpm lint-fix
|
||||
```
|
||||
|
||||
Run tests:
|
||||
|
||||
```shell
|
||||
npm test
|
||||
pnpm test
|
||||
```
|
||||
|
||||
To run a single test:
|
||||
|
||||
```shell
|
||||
# Single file: table.test.ts
|
||||
npm test -- table.test.ts
|
||||
pnpm test -- table.test.ts
|
||||
# Single test: 'merge insert' in table.test.ts
|
||||
npm test -- table.test.ts --testNamePattern=merge\ insert
|
||||
pnpm test -- table.test.ts --testNamePattern=merge\ insert
|
||||
```
|
||||
|
||||
4810
nodejs/examples/package-lock.json
generated
4810
nodejs/examples/package-lock.json
generated
File diff suppressed because it is too large
Load Diff
@@ -11,16 +11,17 @@
|
||||
"test": "node --experimental-vm-modules node_modules/.bin/jest --testEnvironment jest-environment-node-single-context --verbose",
|
||||
"lint": "biome check *.ts && biome format *.ts",
|
||||
"lint-ci": "biome ci .",
|
||||
"lint-fix": "biome check --write *.ts && npm run format",
|
||||
"lint-fix": "biome check --write *.ts && pnpm format",
|
||||
"format": "biome format --write *.ts"
|
||||
},
|
||||
"author": "Lance Devs",
|
||||
"license": "Apache-2.0",
|
||||
"packageManager": "pnpm@11.1.1",
|
||||
"dependencies": {
|
||||
"@huggingface/transformers": "^3.0.2",
|
||||
"@huggingface/transformers": "3.0.2",
|
||||
"@lancedb/lancedb": "file:../dist",
|
||||
"openai": "^4.29.2",
|
||||
"sharp": "^0.33.5"
|
||||
"openai": "4.29.2",
|
||||
"sharp": "0.33.5"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@biomejs/biome": "^1.7.3",
|
||||
|
||||
3466
nodejs/examples/pnpm-lock.yaml
generated
Normal file
3466
nodejs/examples/pnpm-lock.yaml
generated
Normal file
File diff suppressed because it is too large
Load Diff
13
nodejs/examples/pnpm-workspace.yaml
Normal file
13
nodejs/examples/pnpm-workspace.yaml
Normal file
@@ -0,0 +1,13 @@
|
||||
# Block resolution of versions less than 24h old (Shai-Hulud window).
|
||||
# This is the pnpm 11 default but pinned here so it's visible to
|
||||
# reviewers and survives a future pnpm major flipping the default.
|
||||
minimumReleaseAge: 1440
|
||||
|
||||
# Fail install if a transitive dep tries to run an unapproved script.
|
||||
strictDepBuilds: true
|
||||
|
||||
allowBuilds:
|
||||
'@biomejs/biome': true
|
||||
onnxruntime-node: true
|
||||
protobufjs: true
|
||||
sharp: true
|
||||
10452
nodejs/package-lock.json
generated
10452
nodejs/package-lock.json
generated
File diff suppressed because it is too large
Load Diff
@@ -38,15 +38,15 @@
|
||||
"url": "https://github.com/lancedb/lancedb"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@aws-sdk/client-dynamodb": "^3.33.0",
|
||||
"@aws-sdk/client-kms": "^3.33.0",
|
||||
"@aws-sdk/client-s3": "^3.33.0",
|
||||
"@aws-sdk/client-dynamodb": "3.1003.0",
|
||||
"@aws-sdk/client-kms": "3.1003.0",
|
||||
"@aws-sdk/client-s3": "3.1003.0",
|
||||
"@biomejs/biome": "^1.7.3",
|
||||
"@jest/globals": "^29.7.0",
|
||||
"@napi-rs/cli": "^3.5.1",
|
||||
"@napi-rs/cli": "3.5.1",
|
||||
"@types/axios": "^0.14.0",
|
||||
"@types/jest": "^29.1.2",
|
||||
"@types/node": "^22.7.4",
|
||||
"@types/node": "22.7.4",
|
||||
"@types/tmp": "^0.2.6",
|
||||
"apache-arrow-15": "npm:apache-arrow@15.0.0",
|
||||
"apache-arrow-16": "npm:apache-arrow@16.0.0",
|
||||
@@ -57,9 +57,9 @@
|
||||
"shx": "^0.3.4",
|
||||
"tmp": "^0.2.3",
|
||||
"ts-jest": "^29.1.2",
|
||||
"typedoc": "^0.26.4",
|
||||
"typedoc-plugin-markdown": "^4.2.1",
|
||||
"typescript": "^5.5.4",
|
||||
"typedoc": "0.26.4",
|
||||
"typedoc-plugin-markdown": "4.2.1",
|
||||
"typescript": "5.5.4",
|
||||
"typescript-eslint": "^7.1.0"
|
||||
},
|
||||
"ava": {
|
||||
@@ -68,15 +68,16 @@
|
||||
"engines": {
|
||||
"node": ">= 18"
|
||||
},
|
||||
"packageManager": "pnpm@11.1.1",
|
||||
"cpu": ["x64", "arm64"],
|
||||
"os": ["darwin", "linux", "win32"],
|
||||
"scripts": {
|
||||
"artifacts": "napi artifacts",
|
||||
"build:debug": "napi build --platform --dts ../lancedb/native.d.ts --js ../lancedb/native.js --output-dir lancedb",
|
||||
"postbuild:debug": "shx mkdir -p dist && shx cp lancedb/*.node dist/",
|
||||
"postbuild:debug": "shx mkdir -p dist && shx cp lancedb/*.node dist/ && node -e \"require('fs').writeFileSync('dist/package.json', JSON.stringify({name:'@lancedb/lancedb',type:'commonjs'}))\"",
|
||||
"build:release": "napi build --platform --release --dts ../lancedb/native.d.ts --js ../lancedb/native.js --output-dir dist",
|
||||
"build": "npm run build:debug && npm run tsc",
|
||||
"build-release": "npm run build:release && npm run tsc",
|
||||
"build": "pnpm build:debug && pnpm tsc",
|
||||
"build-release": "pnpm build:release && pnpm tsc",
|
||||
"tsc": "tsc -b",
|
||||
"posttsc": "shx cp lancedb/native.d.ts dist/native.d.ts",
|
||||
"lint-ci": "biome ci .",
|
||||
@@ -86,7 +87,7 @@
|
||||
"lint-fix": "biome check --write . && biome format --write .",
|
||||
"prepublishOnly": "napi prepublish -t npm",
|
||||
"test": "jest --verbose",
|
||||
"integration": "S3_TEST=1 npm run test",
|
||||
"integration": "S3_TEST=1 pnpm test",
|
||||
"universal": "napi universalize",
|
||||
"version": "napi version"
|
||||
},
|
||||
@@ -94,8 +95,8 @@
|
||||
"reflect-metadata": "^0.2.2"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@huggingface/transformers": "^3.0.2",
|
||||
"openai": "^4.29.2"
|
||||
"@huggingface/transformers": "3.0.2",
|
||||
"openai": "4.29.2"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"apache-arrow": ">=15.0.0 <=18.1.0"
|
||||
|
||||
7317
nodejs/pnpm-lock.yaml
generated
Normal file
7317
nodejs/pnpm-lock.yaml
generated
Normal file
File diff suppressed because it is too large
Load Diff
18
nodejs/pnpm-workspace.yaml
Normal file
18
nodejs/pnpm-workspace.yaml
Normal file
@@ -0,0 +1,18 @@
|
||||
# Flat node_modules layout. The @napi-rs/cli build step fails to locate
|
||||
# the cdylib artifact under pnpm's isolated layout; the hoisted linker
|
||||
# mirrors npm's structure and unblocks the native build.
|
||||
nodeLinker: hoisted
|
||||
|
||||
# Block resolution of versions less than 24h old (Shai-Hulud window).
|
||||
# This is the pnpm 11 default but pinned here so it's visible to
|
||||
# reviewers and survives a future pnpm major flipping the default.
|
||||
minimumReleaseAge: 1440
|
||||
|
||||
# Fail install if a transitive dep tries to run an unapproved script.
|
||||
strictDepBuilds: true
|
||||
|
||||
allowBuilds:
|
||||
'@biomejs/biome': true
|
||||
onnxruntime-node: true
|
||||
protobufjs: true
|
||||
sharp: true
|
||||
Reference in New Issue
Block a user