Bilal Atique
0a1c1a04c7
Closes #4288. Blob reads in Node can now be cancelled. `BlobFile.read`, `readRange` and `readRanges`, and `Table.fetchBlobs` and `fetchBlobFiles` take an optional `{ signal?: AbortSignal }` (new `BlobReadOptions` type). When the signal aborts, the call rejects with `signal.reason` (an `AbortError` unless the caller gave a reason) and the native future is dropped, so the read stops instead of running to completion. A signal that is already aborted rejects before any read starts. How it works: - On the Rust side, these five methods are now sync `#[napi]` methods that return `env.spawn_future(...)`. The future is wrapped in `futures::future::abortable`, and the abort handle is registered with napi's `AbortSignal::on_abort`. napi's `AbortSignal` is `Rc`-based, so it can't be an argument of an `async fn`, which is why the methods changed shape. The TypeScript signatures are unchanged apart from the new optional argument. - napi-rs installs its own `onabort` on the signal it receives, which would overwrite a handler the caller set. So the TypeScript wrapper passes a private `AbortController` signal to the native call for each request, forwards the caller's abort to it with `addEventListener(..., { once: true })`, and removes the listener when the call settles. Tests in `table.test.ts`: - an already aborted signal rejects on all five methods, and the handle still reads afterwards - a custom abort reason is what the call rejects with - a read with a signal that never aborts returns normally and leaves no `abort` listener on the signal - aborting a 32 MiB `readRange` right after it starts rejects with `AbortError`, and the same handle reads again afterwards The Node docs under `docs/src/js` are regenerated. I didn't add a test for where the `read()` cursor ends up after a mid-flight abort on a local table. The remote `BlobFile` documents that a cancelled read leaves the cursor unchanged, but I haven't confirmed the same for the native Lance handle, so the docs don't claim it.
LanceDB Documentation
LanceDB docs are available at docs.lancedb.com.
The SDK docs are built and deployed automatically by Github Actions
whenever a commit is pushed to the main branch. So it is possible for the docs to show
unreleased features.
Building the docs
Setup
- Install LanceDB Python. See setup in Python contributing guide.
Run
make developto install the Python package. - Install documentation dependencies. From LanceDB repo root:
pip install -r docs/requirements.txt
Preview the docs
cd docs
mkdocs serve
If you want to just generate the HTML files:
PYTHONPATH=. mkdocs build -f docs/mkdocs.yml
If successful, you should see a docs/site directory that you can verify locally.
Adding examples
To make sure examples are correct, we put examples in test files so they can be run as part of our test suites.
You can see the tests are at:
- Python:
python/python/tests/docs - Typescript:
nodejs/examples/
Checking python examples
cd python
pytest -vv python/tests/docs
Checking typescript examples
The examples depend on @lancedb/lancedb at file:../dist, so the package must be
built before running the tests. This uses pnpm; see the
Typescript contributing guide for the toolchain setup.
pushd nodejs
pnpm install
pnpm build
popd
Then you can run the examples by going to the nodejs/examples directory, which is a
separate pnpm package with its own lockfile:
pushd nodejs/examples
pnpm install
pnpm test
popd
API documentation
Python
The Python API documentation is organized based on the file docs/src/python/python.md.
We manually add entries there so we can control the organization of the reference page.
However, this means any new types must be manually added to the file. No additional
steps are needed to generate the API documentation.
Typescript
The typescript API documentation is generated from the typescript source code using typedoc.
When new APIs are added, you must manually re-run the typedoc command to update the API documentation. The new files should be checked into the repository.
pushd nodejs
# `pnpm docs` would invoke pnpm's built-in `docs` command, not the script.
pnpm run docs
popd