Teach Python Function authoring to retain the compact V1 grammar for
existing types and emit canonical exact JSON for nested struct
signatures. Adds coverage for recursive struct/list schemas and exact
field properties.
The bindings are built, installed and published with pnpm everywhere,
but a parallel npm dependency graph was still being maintained beside
it. This removes it, raises the supported Node floor to the versions we
actually test, and gives Dependabot the npm coverage it was missing.
## Dropping npm
`nodejs/package-lock.json` was regenerated by `ci/update_lockfiles.sh`
on every release commit and read by nothing — no workflow runs `npm ci`
or `npm install` in `nodejs/`, and npm never publishes a lockfile in a
package tarball. It could not even agree with the real install, since
npm does not see pnpm's `overrides`. Because GitHub's dependency graph
parses `package-lock.json`, it was also reporting vulnerabilities for a
tree we neither install nor ship.
`docs/package.json`, `docs/package-lock.json` and `docs/tsconfig.json`
go too. They depend on `file:../node` and
`file:../node/node_modules/apache-arrow` — the `node/` directory was
removed long ago — the tsconfig compiles `src/*.ts` where no TypeScript
files exist, and nothing installs any of it. `docs.yml` only referenced
the lockfile to configure an npm cache for an install it never ran.
Two `workflow_dispatch` workflows for regenerating those lockfiles are
removed as well. Both were already broken: they `uses:` composite
actions at `.github/workflows/update_package_lock{,_nodejs}` that do not
exist, so dispatching either failed immediately.
The remaining `npx` calls become direct `node_modules/.bin/...`
invocations. These were already running locally installed binaries
rather than resolving anything, but naming the binary removes the npm
CLI from the loop and does not depend on which Node version is active.
`dev.yml`'s commitlint check was the last place doing real npm
dependency resolution — an unpinned `npm install
@commitlint/config-conventional` that also bypassed the
`minimumReleaseAge` hold configured for `nodejs/` — and is now a pinned
`pnpm dlx`.
## Node support
Node 18 and 20 both reached end-of-life, in April 2025 and April 2026.
The matrix moves to 22, 24 and 26, and `engines` rises from `>= 18` to
`>= 22` so the declared floor is one the matrix actually covers. Node 22
is LTS until April 2027; 24 is LTS; 26 is Current and becomes LTS in
October 2026.
This also removes the reason the workflows reached for `npx` in the
first place: pnpm 11 requires Node >= 22.13, which every matrix version
now satisfies.
The prebuilt-binary smoke test in `npm-publish.yml` moves from Node 20
to Node 22 — the floor, where a napi ABI problem would surface first —
rather than fanning out across all three, to keep the publish matrix
from tripling.
## Dependabot
There were no npm-ecosystem entries at all, which is why the advisories
behind #4073 went unnoticed. Both pnpm lockfiles are now watched —
`nodejs/` and `nodejs/examples/`, which is a separate install — using
the same `lockfile-only` strategy as the existing cargo and pip entries,
so version ranges in `package.json` are left alone.
## Pre-commit biome
The hook ran `npx @biomejs/biome@1.8.3` while `nodejs/package.json`
resolved 1.9.4. The two disagree about formatting, so the hook rejected
code that `pnpm lint` accepts, and failed on unmodified `main` for
anyone touching `nodejs/`. It now uses the pnpm-managed biome, which
fixes the drift with no source changes.
## Testing
`dev.yml`'s commitlint job does not check out the repo, so it runs in an
empty workspace, and I could not verify `pnpm/action-setup` there
locally. It triggers on `pull_request_target`, so this PR exercises it
directly — worth confirming green before merge. I did verify the `pnpm
dlx` invocation itself locally: it accepts a conventional title and
rejects a non-conventional one with exit 1.
Node 26 is new enough that the examples job may surface gaps in prebuilt
native binaries (`onnxruntime-node`, `sharp`) before their maintainers
publish for it.
## Not included
`nodejs/examples/` still pins `sharp: "0.33.5"` and has its own audit
findings. Raising the Node floor unblocks that work — sharp 0.35
requires Node >= 20.9, which the matrix now satisfies — but it is a
dependency bump rather than tooling cleanup, so it is left separate.
## Breaking changes
`@lancedb/lancedb` now requires Node >= 22; previously >= 18. The
`@types/node` peer range moves from `>=18` to `>=22` to match. Users on
Node 18 or 20 must upgrade their runtime; both have been end-of-life for
some time. Existing installs are unaffected, since `engines` is only
checked on install.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Computed-column planning currently sees Blob v2 storage descriptors, so
expressions cannot consume payload bytes or preserve Blob semantics in
their outputs.
A computed declaration now derives its output field from its expression.
A direct projection of a Blob v2 field inherits the source field's Blob
metadata; other expressions retain their ordinary Arrow-inferred type.
Declarations remain ordered, so the same rule applies across chained
projections.
Refresh materializes referenced Blob inputs as `LargeBinary` payload
bytes and publishes inherited Blob outputs through Lance's Blob
conversion path. Remote requests remain within the shared namespace
contract as `{name, computed}`; the server planner is being updated in
tandem to implement the same Blob-aware planning semantics, and remote
enablement must be aligned with that server rollout.
The existing null-as-unfilled contract remains unchanged. Row-level
freshness and cell flags remain follow-up work.
`add_columns().computed()` accepted several columns in one call but
bound each against the table's schema as it stood before the call, so
`a` and `b = a + 1` had to be two commits. A server staging declarations
behind other schema work has no atomic way to do that, and a caller
reading the builder's plural signature reasonably expects the batch to
be one.
Each accepted column now joins the schema the next one resolves against,
so the batch is planned and committed as one. Order is the dependency
order; reading ahead is still an unknown column. `validate_declarations`
exposes the schema-level checks -- the Function-binding guard and the
planning -- without a commit, for callers that must reject before
earlier work in the same request lands; LSM state is table state and
stays a commit-time check.
Refresh order matters for a dependent column: `b = coalesce(a, 0)`
refreshed before `a` would bake zeros from `a`'s placeholder null, and
the fill-once contract keeps them. Refresh now refuses, naming the
input, while a computed input still has rows a refresh of it would fill
-- the same probe refresh already uses to detect a no-op. Otherwise it
is one snapshot and one commit, as before; a concurrent append is not in
the commit and waits for the next refresh. Refreshing dependencies on
the caller's behalf was considered and rejected: it is not how
materialized views or our own backfill scheduler behave, and it needs
multi-commit fencing that an explicit per-row fill marker would make
unnecessary.
The nightly `NPM Publish` run has failed every night since at least Aug
23, always on the same two legs: `aarch64-unknown-linux-gnu` and
`aarch64-unknown-linux-musl`. The other five targets pass. rustc is
OOM-killed during the fat-LTO codegen of the cdylib — `signal: 9` with
no diagnostic, about 27 minutes in — and on the musl leg that takes the
whole runner down with `The runner has received a shutdown signal`.
Both legs now pass:
| leg | before | peak memory | wall time |
| --- | --- | --- | --- |
| `aarch64-unknown-linux-gnu` | OOM-killed at ~27 min | 31391 → 22851
MiB | 38m43s → 22m04s |
| `aarch64-unknown-linux-musl` | runner killed at ~28 min | >32 GiB →
16516 MiB | ~40 min → 20m50s |
**ThinLTO** is most of that. Fat LTO is single-threaded, and its peak is
consumed inside rustc's LLVM before any linker process is spawned —
which is why it is the whole fix on musl, and why lld alone left the gnu
leg still peaking at 31391 MiB against the runner's 32 GiB. Both legs
now use the `lto: thin` / `codegen_units: 16` settings that darwin and
both Windows legs already use, at a cost of a few percent runtime
performance.
**lld** covers the rest, on the gnu leg. arm64 Linux otherwise links
through GNU `ld` where x86_64 already defaults to `rust-lld`, which is
why only the arm64 legs hit this at all; on a comparable arm64 build
(`lancedb/sophon#7313`) it cut the largest single linker process from
7.0 to 4.0 GiB and wall time by 35%. The flags live in a small wrapper
script used as the linker rather than in `-C link-arg`, because the
per-target rustflags variable does not reach every unit that links:
dependency crates linking a dylib (`crc-fast`, `lance-arrow`) were
invoked as bare `clang`, which targets the x86_64 host and fails with
`Relocations in generic ELF (EM: 183)`.
Separately, and affecting five legs rather than two: the three ThinLTO
targets exported `CARGO_PROFILE_RELEASE_LTO` and
`CARGO_PROFILE_RELEASE_CODEGEN_UNITS` from `pre_build`, which runs
inside the build step — after the cache step. `Swatinem/rust-cache`
computes its key when the action runs, before any step, so step-local
values are invisible to it. The result is a loop that never converges:
the key never changes, so restores are exact hits, an exact hit makes
the post-run save a no-op, and cargo invalidates the restored artifacts
anyway because the flags differ. Those legs have been rebuilding cold on
every run. Both values move to job-level `env:` ahead of the cache step,
driven by new `lto:`/`codegen_units:` matrix fields, and are forwarded
into the containers with `-e` since `docker run` inherits nothing.
Every leg's cache key shifts once as a result, so expect one cold
rebuild.
A `Report peak memory` step is added so whether these legs fit is a
number rather than an inference from whether the runner survived. It
produced the figures above.
## Not included
Moving these legs to native arm64 runners. It would retire the zig cross
path, the `AT_HWCAP2` workaround and the `TARGET_CC` override, and arm64
runners are billed roughly 37% below x64 at equal core count — but the
`lts-debian-aarch64` image exists to link against the manylinux2014
sysroot's glibc 2.17, and building natively on ubuntu-24.04 would raise
the minimum glibc for every published aarch64 binary. That is a
user-facing decision, not a CI cleanup.
Dropping these legs to smaller runners, which is where the real cost
saving is — larger runners are billed even on public repos. On these
numbers it is not available yet: musl at 16516 MiB is about 130 MiB over
what a 16 GB standard runner has. Worth revisiting as a follow-up.
## Testing
Cargo's rustflags precedence was checked locally rather than taken from
the docs, since getting it wrong would silently change the published
binaries. With a throwaway crate carrying both a `target.'cfg(all())'`
and a per-target rustflags table: setting `RUSTFLAGS` discards both, and
setting it to the empty string discards them too. That rules out routing
the linker flag through a job-level `RUSTFLAGS`, because `env:` keys
cannot be conditionally omitted and every other leg would then silently
lose the `target-cpu`/`target-feature` settings in `.cargo/config.toml`
— `+avx2` on x86_64 and `-crt-static` on aarch64-musl.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
BREAKING CHANGE: list_tables now provides tables in arbitrary order and
the page token is now completely opaque. `table_names` retains the old
behavior of lexical ordering and `start-after` semantics.
Listing the tables in a directory database cost what the database held
rather than what the page held. `ListingDatabase::list_tables`
enumerated every child directory of the base path, sorted the names,
then discarded all but the requested page — on every request, for every
page. On object storage that is one full listing per page.
This PR pages the store instead. `list_tables` asks for one page at a
time through `ObjectStore::read_dir_page`, carrying the store's own
continuation token, so a page is one request. Non-table children can
leave a page short of its limit, so the walk continues until the page is
full or the store runs out.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- accept genuine Arrow metadata maps created in another JavaScript realm
- validate every metadata entry and clone it into a local Map
- cover an Arrow 15 VM-realm table through the public fromDataToBuffer
boundary
- retain structural typing for nested and dictionary Arrow data
## Root cause
The sanitizer used a local-realm instanceof Map check for schema and
field metadata. A genuine Map created in another JavaScript realm has
the required internal Map state but fails that identity check, so
fromDataToBuffer rejected the foreign table before serializing its rows.
## Scope
This fixes the distinct JavaScript-realm sanitizer failure identified
during review. It does not establish the cause of the S3/compaction
panic reported in #1525, so that issue remains open.
## Validation
- pnpm test --runInBand (707 passed, 5 skipped)
- pnpm test --runInBand __test__/arrow.test.ts (189 passed)
- pnpm build
- pnpm lint
- pnpm run docs
Related to #1525
<!-- lance-gatekeeper-fix:v1 agent=b522628ad3bae914eb7266ccd899d508
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- allow Python sync, async, and remote table updates to accept type-safe
`Expr` filters
- serialize expression filters before invoking the existing update
implementation
- cover numeric-looking text and apostrophe-containing text in sync and
async regression tests
## Root cause
`Table.update` was the remaining Python write path that required callers
to construct a raw SQL predicate. Dynamic text interpolated without SQL
literal encoding could therefore be parsed as an integer, float, or
unterminated string instead of Utf8. The expression API already encodes
literals safely for query and delete filters.
## Validation
- `cd python && .venv/bin/pytest
python/tests/test_table.py::test_update_async
python/tests/test_table.py::test_update_expr_filter_literals_async
python/tests/test_table.py::test_update
python/tests/test_table.py::test_update_expr_filter_literals -q`
- `cd python && .venv/bin/pytest python/tests/test_expr.py -q`
- `cd python && .venv/bin/ruff format --check .`
- `cd python && .venv/bin/ruff check .`
Fixes#1869
<!-- lance-gatekeeper-fix:v1 agent=01f1e7b69c65e8b6d3b3c1e1a7918179
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- add Python regression coverage for integer and double arithmetic
against the generated _distance column
- merge the current main base containing Lance v11.0.0-beta.3 from #3896
- verify both expressions retain the generated scoring field Float32
type and compute the expected values
## Root cause
Lance parsed dynamic projection expressions before vector search added
its generated Float32 _distance field. Without a typed provisional
field, expression discovery rejected mixed numeric arithmetic. Lance
upstream fixed discovery and final-schema replanning in
lance-format/lance#8163, and the current base consumes that fix through
Lance v11.0.0-beta.3.
## Validation
- uv run --extra tests pytest
python/tests/test_query.py::test_select_arithmetic_with_distance -vv
--maxfail=2 — 2 passed
- python/.venv/bin/ruff format --check python/python/tests/test_query.py
- python/.venv/bin/ruff check .
Fixes#2618
<!-- lance-gatekeeper-fix:v1 agent=816a060090517471edfb73652bb5c9fe
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- enrich local query field-not-found errors with recursively qualified
Arrow struct leaf paths
- preserve all other Lance and DataFusion errors unchanged
- add a regression test for the Python-visible filter error described in
the issue
## Root cause
DataFusion builds `FieldNotFound` candidates from the top-level Arrow
schema even though Lance supports dotted struct-field filters. As a
result, the error listed only the struct container and hid its valid
nested leaves.
## Validation
- `cargo test --quiet --features remote -p lancedb
table::query::tests::test_missing_filter_field_lists_nested_fields --
--exact`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples` (passes
with pre-existing unrelated warnings)
- `cargo fmt --all -- --check`
Fixes#951
<!-- lance-gatekeeper-fix:v1 agent=7893f7a181fd8bc1ad00acc62d1a85c2
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- normalize Python snake_case and TypeScript camelCase embedding
metadata
- use the normalized metadata for schema validation and embedding lookup
- cover appending through `Table.add()` with a Python-authored schema
fixture
## Root cause
Python writes embedding source and vector column names as
`source_column` and `vector_column`, but the TypeScript SDK only read
`sourceColumn` and `vectorColumn`. The missing source name reached the
add path as `undefined`, preventing JavaScript rows from being embedded
and appended.
## Validation
- `pnpm lint`
- `pnpm test __test__/embedding.test.ts __test__/arrow.test.ts
__test__/registry.test.ts --runInBand` (201 passed, 1 skipped)
- `pnpm build`
- `pnpm run docs`
Fixes#1289
<!-- lance-gatekeeper-fix:v1 agent=b71c18a5e33d26f4d138972e91d34e66
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- preserve Arrow field metadata when matching record data to a provided
schema
- retain metadata on partially reconstructed nested struct fields
- add a regression test for lance.json metadata through Arrow IPC
serialization
## Root cause
The TypeScript schema inferrer rebuilt fields selected from a provided
schema without copying their metadata. JSON columns therefore kept their
LargeBinary physical type but lost the lance.json extension marker
before insert, causing the schema mismatch reported in the issue.
## Validation
- pnpm lint
- pnpm build
- pnpm tsc
- pnpm run docs
- pnpm test --runInBand (18 suites and 798 tests passed; 5 tests
skipped)
Fixes#4062
<!-- lance-gatekeeper-fix:v1 agent=3ec52632b71563f53d199b22629f8c4f
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
<!-- lance-gatekeeper-fix:v1 agent=d30696bc46eb32f04c9927b0792e35d3
generation=1 -->
## Summary
- use the Lance native batch KNN path so fixed-size batch vector
searches share one flat table scan
- validate consistent query-vector dimensions and retain the per-vector
plan when offsets require its existing semantics
- add Rust and Python regressions and update Rust, Python, and
TypeScript API documentation
## Root cause
LanceDB expanded every vector in a batch into a separate scan plan and
joined the plans with `UnionExec`. For unindexed tables on S3, a batch
of ten vectors therefore ran ten concurrent full scans, amplifying CPU
and retained data enough to produce the reported memory spike.
The native Lance batch KNN path performs bounded-memory selection for
all query vectors over one flat scan. LanceDB now supplies the vectors
as a batch and avoids applying a global scanner limit to the combined
per-query results. Batch queries with a nonzero offset keep the previous
plan because the native batch API does not support per-query offsets.
## Validation
- targeted Rust batch-query plan and execution tests
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo fmt --all -- --check`
- targeted Python batch-vector regression after rebuilding the extension
- Ruff formatting/checks for the touched Python files
- Node.js build, lint, docs generation, and targeted batch-vector Jest
test
- `git diff --check`
Fixes#2468
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
In LanceDB Enterprise, we've adopted these conventions to give some
"canonical" metadata paths. This lets us display them in a certain way
in the UI or let agents standardize on them, to assume they'll find info
in a certain place. This PR (only comments/docs) just documents those
choices.
## Summary
LanceDB could not request Lance's list-element FTS document granularity
through Python or Remote APIs, and generic nested-field resolution
exposed Arrow's internal `item` segment instead of the public field
path.
This exposes typed `row | list_element` configuration for Python FTS
index creation and match/phrase queries, preserves `_doc_index`, and
keeps nested FTS paths public (for example, `docs.content`). Remote
list-element requests require server API version 0.6.0 so older servers
cannot silently execute them with row semantics; explicit row requests
remain compatible.
## Compatibility
Omitted index and query parameters retain row granularity. Remote row
index creation omits the new wire field.
## Tracking
[ENT-2342](https://linear.app/lancedb/issue/ENT-2342/expose-list-element-fts-document-granularity-end-to-end)
A Function's remote environment can now be conda instead of pip.
`@udf(conda=[...], conda_channels=[...])` registers one; pip and conda
are exclusive, channels are priority-ordered and require conda. The Rust
and Python `PythonEnvironmentSpec` models gain `channels`, dropped from
the canonical JSON when empty so existing pip registrations keep their
digests.
## Summary
- add regression coverage for the reported synchronous Python workload
with 32 simultaneous `open_table` calls
- verify every independently opened S3-backed table handle can read
through the connection's shared session and object-store client
## Root cause
In Python v0.13.0, each synchronous table handle lazily constructed its
own Lance dataset. Opening many handles in parallel therefore triggered
independent S3 client construction and bucket-region resolution, which
failed under thread pressure. The current Rust-backed connection path
owns a shared Lance session and retains its object-store handle, so
table opens reuse the existing S3 client; these tests lock in that
behavior through the public Python API and a causal Session-registry
invariant.
## Validation
- `uvx --from 'ruff==0.15.20' ruff format --check
python/tests/test_s3.py`
- `uvx --from 'ruff==0.15.20' ruff check .`
- `cargo fmt --all`
- `cargo test --quiet --features remote -p lancedb
test_concurrent_open_table_reuses_connection_object_store`
- `cargo check --quiet --features remote --tests --examples`
- equivalent 32-thread `open_table(...).count_rows()` workload against a
local database
- targeted S3 test collected successfully locally; execution requires
the CI LocalStack service, which is unavailable in this runner
Fixes#1786
<!-- lance-gatekeeper-fix:v1 agent=d311f3c7151f77ae22b4997702e7b7db
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Centralizes remote table freshness fencing and response-version tracking
in the default transport path.
Covers schema and blob bypass paths, keeps explicit time-travel and
cross-timeline operations unfenced, and advances freshness after refresh
and index job completion.
Updates the Rust workspace Lance crates and Java lance-core dependency
to
[v12.0.0-beta.2](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.2).
No compatibility fixes were required; full-workspace Clippy passes with
all features and warnings denied.
## Summary
- Resolve automatic string-search routing from the active table schema
whenever the query executes.
- Defer embedding-provider construction while leaving explicit vector
and FTS routes unchanged.
- Cover unrelated global registrations and metadata transitions across
repeated executions of one query builder.
## Root cause
LocalTable.search used the number of globally registered embedding
providers to choose between vector and full-text search. A provider
registered for any other table therefore sent a plain FTS table down the
vector path. A wrapper-lifetime metadata snapshot avoided that
contamination but became stale after time travel or read-consistency
refreshes. The query now records fluent builder operations and creates
the appropriate native vector or FTS query from the active schema on
each execution.
## Validation
- pnpm build
- pnpm tsc
- pnpm lint
- pnpm run docs
- pnpm test --runInBand (681 passed, 5 skipped)
Fixes#1557
<!-- lance-gatekeeper-fix:v1 agent=b6183df8296db4aabdc5d19a2256b029
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Updates the Lance Rust workspace dependencies and Java lance-core
dependency to
[v12.0.0-beta.1](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.1).
Includes compatibility updates for the renamed shard-manifest API and
paginated object-store wrappers.
## Summary
- lazily initialize built-in OpenAI and Hugging Face providers when
consumers call the public embedding registry API
- choose automatic vector versus FTS search from embedding metadata on a
fresh pinned table revision for every execution
- expose automatic string searches as an `AutoQuery` with only
operations common to both native query families
- keep the registry shared and built-in registration safe across
duplicated module graphs
## Root cause
Nitro treats dependency modules as side-effect-free and removes the bare
OpenAI provider import from its generated route. Registration therefore
never runs, so `getRegistry().get("openai")` remains undefined even when
the registry itself is shared globally. Bundlers may also duplicate the
provider and registry module graphs.
The public embedding entry point now initializes built-in providers only
when `getRegistry()` is explicitly called, keeping initialization on a
live path that Nitro retains. Each terminal automatic-search execution
pins the exact table revision visible at dispatch, reads embedding
metadata and computes an embedding from that snapshot, replays the
builder operations, and constructs and executes the selected native
query against the same snapshot. Pinned native snapshots execute locally
when namespace pushdown cannot carry their revision, while remote
snapshots are seeded directly from one version-and-schema response. The
public `AutoQuery` builder exposes only the operations shared by FTS and
vector search, so runtime class narrowing cannot expose invalid
vector-only methods. Repeated built-in registration replaces stale
constructors from duplicated module graphs while public `register()`
retains its duplicate-alias error.
## Validation
- `cargo fmt --all`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test --runInBand` (783 passed, 5 skipped)
- serial examples suite with a local OpenAI mock (11 passed), including
`sentence-transformers.test.ts`
- packaged Nitro 2.13.4 server route using the reported imports returned
`{"registered":true}`
- fresh-process FTS fixture initialized both public built-ins and
confirmed automatic string search still returned the indexed row
- schema-consistency regressions cover read-consistency refresh,
checkout, checkoutLatest, restore, runtime class narrowing, concurrent
overwrite during embedding computation, and reused automatic-search
builders
- focused regressions confirm pinned native snapshots bypass unversioned
namespace pushdown and remote snapshots use one describe request
Fixes#2429
<!-- lance-gatekeeper-fix:v1 agent=2adf0f21b8bfb634606ed8897a849e30
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- document that Azure Blob Storage credentials can be passed directly
through `storage_options`
- provide valid quoted `account_name` and `account_key` examples for
both sync and async Python connections
- execute the option dictionaries during doctests so the original
unquoted-key mistake is caught
## Root cause
The historical Python storage guide used `account_name` and
`account_key` as bare identifiers in dictionary literals. Following that
example either raised `NameError` or, when those names were predefined,
produced incorrect option keys. The runtime already accepts direct Azure
credentials, but the current Python API reference did not contain a
corrected Azure example.
## Validation
- `python/.venv/bin/ruff format --check
python/python/lancedb/__init__.py`
- `python/.venv/bin/ruff check .`
- `cd python && uv run --no-sync pytest --doctest-modules
python/lancedb/__init__.py -q`
- `cd python && uv run --no-sync pytest python/tests/test_import.py -q`
Fixes#2236
<!-- lance-gatekeeper-fix:v1 agent=7a7a9e009eb2ebe52ac9b1adf2e8afb2
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
#3528 added blob declarations and binary coercion. String values were
still rejected. They now coerce to the blob `uri` child.
```python
table.add([{"id": 1, "image": "s3://bucket/media/cat.jpg"}])
payload = table.fetch_blobs("image", table.search().to_arrow())
```
A URI under a registered base writes with no extra options. An
unregistered URI fails. `allow_external_blob_outside_bases` is a local
escape hatch that stores an absolute URI. It does not register a base.
Remote `add` rejects that flag before making a request. String input
still coerces and is sent as a `uri` struct.
`add_bases` is a follow-up. `merge_insert` does not coerce string blob
input.
### Testing
- `cargo test -p lancedb --test blob_integration`
- `cargo test -p lancedb blob_coerce`
- `cargo test -p lancedb --features remote --lib
add_rejects_external_blob_flag add_string_blob_becomes_uri_struct`
- `cd python && uv run --extra tests pytest python/tests/test_blob.py -k
uri -q`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Problem
`uv run ... maturin develop` synchronizes the project as an editable
package before running the command. Maturin's editable build otherwise
uses the release profile, which enables the repository's fat LTO
configuration during local bootstrap.
## Behavior
Editable Python builds now explicitly use Cargo's dev profile. The
minimum maturin version is raised to 1.10, where `editable-profile`
support was introduced.
## Problem
The unreleased First-Class Function authoring API exposed
`secrets=[...]` and serialized `required_secrets`, promising runtime
resolution and injection that Sophon does not implement.
## Behavior
Remove the secrets dimension from the public Python decorator, Python
and Rust registration/version models, and shared wire fixtures. Existing
successfully registered Functions retain stable identity: the field
could only be empty and empty values were already omitted from canonical
serialization.
A stable LanceDB release has not shipped this Function API, so this
contracts the surface before it becomes a published compatibility
commitment.
## Validation gap
The Rust shared-golden Function suites and Python formatting/lint checks
pass locally. Python pytest was not run because the local environment
lacks its runtime dependencies and the frozen native editable build did
not complete in practical time.
Function applications and bindings currently encode `group_id` and
binding `revision` even though `binding_id` already owns the complete
immutable binding lifecycle and `outputs` already defines the atomic
multi-output set.
Make `binding_id` the sole binding identity, remove the redundant fields
from the Rust and Python client contracts, and describe multi-output
declarations directly. This intentionally replaces the removed wire
fields without a compatibility path.
Registering a real (embedding) Function failed on the client for three
reasons:
- `_package_source` treated `inspect.getclosurevars().unbound` as
"unresolved globals"; CPython puts attribute names there, so any body
with `np.linalg.norm(...)` or `body.split()` was rejected. Module-scope
references now come from Python's own scope analysis (`symtable`) over
the function source, recursively, and each is resolved the way the
interpreter would: the function's globals first (a module global may
shadow a builtin), then builtins. Free variables of nested scopes stay
lexical; postponed annotations are not runtime loads. A genuinely
missing global still fails.
- `_canonical_arrow_type` emitted spellings the server's frozen grammar
rejects (`fixed_size_list<T>[n]`, `timestamp[us]`, `struct<...>`,
zero-sized lists). It now emits exactly the grammar, with the server's
`fixed_size_list<item, size>` form, and the Rust declaration planner
parses that form too.
A shared golden
(`tests/fixtures/first_class_functions/v1/arrow_types.json`)
enumerates every grammar type, nested forms and rejected spellings; the
Python emitter and Rust parser are tested against it, and the same file
is under test in sophon. Packaging tests execute the shipped artifact in
a fresh namespace.
Contract changes (hence `breaking-change`):
- `@udf` now rejects namespace acquisition structurally
(`globals()`/`eval`/... by name, plus `import
sys`/`builtins`/`importlib`/`inspect` inside the body), requires the
function's captured `__builtins__` to be the standard mapping itself
(identity, so neither lookups nor implicit hooks such as `__import__`
can differ), rejects module globals that are namespace-bearing modules
(`builtins`, `sys`, ...), and treats the function's own name as
recursion only when the module binds it to the function or to the exact
`UdfDefinition` the decorator produced; it resolves module globals
through the function's real namespace (a module global may shadow a
builtin) and ships importable classes/functions as imports.
- List outputs must declare a non-nullable, metadata-free child named
`item` (`pa.list_(pa.field("item", t, nullable=False))`); that is what
the grammar means, and pyarrow's default nullable child was being
silently collapsed into it.
Contract, stated in the `udf` docstring: the artifact is a snapshot of
the function source plus exactly the module names it references.
Reaching the module namespace by another route is rejected where a
static packager can see it and is otherwise unsupported; there is no
dynamic-access detection beyond that.
A permutation stores `_rowid`s, which are row addresses unless stable
row ids are enabled. Nothing in the data loader pinned a table version,
so a compaction between building a permutation and reading it can
resolve those ids to different rows.
The exposure differs by backend but exists on both:
- Remote never pins. `prepare_query_bodies` stamps `"version":
current_version()` on every request, but `current_version()` is `None`
unless `checkout` was called, so every request means "latest".
- Native pins implicitly by holding an `Arc<Dataset>` under
`ConsistencyMode::Lazy`, but `StreamingDataset.__setstate__` reopens the
table in each DataLoader worker, so each worker pins to whatever is
latest at fork time.
## Changes
`Table::at_version` returns an independent handle pinned to a version
without mutating the receiver. `checkout` cannot serve this: on remote
the version cell is an `Arc<RwLock<Option<u64>>>` shared across clones,
so pinning through it would silently pin the caller's table too.
`PermutationBuilder::build` pins for the whole build and records the
version in the permutation table's schema metadata, alongside the
existing split names. `PermutationReader` pins the base table to that
version before any take.
Because the reader pins on construction, the Python worker fork is
covered without touching the pickle format — `Permutation.__setstate__`
drops the reader and `_ensure_open` rebuilds it, which re-pins.
## Behaviour change
A permutation is now bound to the version it was built against, so rows
appended to the base table afterwards are not visible through an
existing permutation. That is the intended semantics — the permutation
only addresses rows that existed when it was built — but it is a change
worth flagging.
Permutations written before this carry no version key and read exactly
as they did before.
## Summary
- compare inferred Arrow types by their semantic representation across
records
- throw the schema inference error when a later record has an
incompatible type
- cover compatible and incompatible multi-record inference across
supported Arrow versions
## Root cause
Schema inference compared newly allocated Arrow DataType objects by
identity, so equivalent inferred types did not compare equal. The
mismatch path also constructed an Error without throwing it, which
silently accepted incompatible values.
## Validation
- pnpm test __test__/arrow.test.ts --runInBand (176 tests passed)
- pnpm lint
- pnpm build
- pnpm run docs
Fixes#3781
<!-- lance-gatekeeper-fix:v1 agent=00ec4f61a3fd82694fa4fb9bb2b37aa8
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`table_names` is being replaced by `list_tables` across the SDKs, but
TypeScript only had `tableNames`. This PR adds `listTables`, which
returns a page of table names together with the token that resumes after
it, and marks `tableNames` and `TableNamesOptions` deprecated in favor
of it.
It binds the `Connection::list_tables` that already exists, so nothing
in the Rust API changes and nothing existing breaks. `pageToken` is
documented as opaque rather than as a table name, since what resumes a
listing is the database's to decide — that keeps callers off a detail
that is going to change.
Stacked on #4040, which fixes a table being dropped at every page
boundary. The page-walking test here needs that fix to pass. Review the
last commit only until #4040 lands.
## Example
```ts
const names = [];
let pageToken = undefined;
do {
const page = await conn.listTables({ pageToken, limit: 100 });
names.push(...page.tables);
pageToken = page.pageToken;
} while (pageToken);
```
A namespace can be listed by passing its path first, mirroring
`tableNames`:
```ts
const page = await conn.listTables(["analytics"], { limit: 100 });
```
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes#4015
<!-- lance-gatekeeper-fix:v1 agent=1c6407eec60f0af2336ae2dee811e1be
generation=1 -->
## Root cause
PermutationBuilder issued count_rows and the projected row-ID scan
through an unpinned RemoteTable handle. Each request could independently
resolve latest, so a concurrent table update could make the count and
scanned rows come from different snapshots.
## Fix
- add a backend hook for obtaining an independent handle pinned to the
currently selected version
- resolve latest once for remote tables while preserving an explicit
checkout and leaving the caller handle unchanged
- build the count, filtered projection, and scan from that pinned handle
while retaining native-table behavior
- add a remote mock regression that advances latest between count and
scan and covers explicit checkout preservation
## Validation
- cargo test --quiet --features remote -p lancedb --lib
test_remote_permutation_builder_pins_snapshot
- cargo test --quiet --features remote -p lancedb --lib
dataloader::permutation::builder::tests
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples
- cargo fmt --all -- --check
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
A caller that queues a refresh and executes it later can only tell the
view it captured from a drop-and-recreate by comparing the definition
and
version. A recreated view with the same definition and an equal or
higher
version passes that check, and the check runs before the refresh reloads
the view, so it never sees the state it commits against.
This mints an `mv.incarnation` token in the schema metadata at each
physical creation of a view table.
`RefreshMaterializedViewBuilder::expect_incarnation` carries the
captured
token into the refresh, which compares it against the latest stored
manifest before planning and again immediately before each commit
(publish, fragment swap, watermark stamp), refusing to land in a
different
incarnation. A view with no token -- declared before tokens existed, or
its metadata replaced wholesale -- is refused under a bound refresh with
its own wording and is minted one by its next unbound refresh. The token
is exposed through `MaterializedView::incarnation`; refreshes without an
expectation are unchanged.
This is best effort: the token is not part of lance's commit condition,
so a recreation landing between the final pre-commit read and the commit
itself is not caught. Closing that window needs a base-manifest
precondition in lance's commit path.
`table_names` paginates using a `start_after` table name. This works for
the `/v1/table` endpoint, which guarantees table-order. But the
`/v1/namespace/{id}/table/list` does not. We change that caller to
instead collect all table names, sort, and apply the pagination locally.
We are deprecating this API, so this is just an interim fix. For good
performance, users should move to the `list_tables` API instead, which
uses opaque tokens that don't rely on lexical sorting.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- add `StreamingDataLoader`, which transports worker snapshots with
prefetched batches and commits them to the parent dataset only when the
trainer receives each batch
- preserve exact non-uniform per-split progress and resume lagging
splits without replaying already-consumed rows
- reject stale parent checkpoints after a standard multi-process
`DataLoader` has started, with guidance to use the consumer-aware loader
- document the new public loader and merge non-uniform state across
ranks
## Root cause
PyTorch runs `StreamingDataset.__iter__` in private worker-process
copies, while callers invoke `state_dict()` on the parent dataset.
Sharing producer counters would still be incorrect because DataLoader
prefetch can advance workers beyond batches returned to the trainer.
## Validation
- `uv run --extra tests pytest python/tests/test_elastic_dataloader.py
-q` (154 passed)
- focused non-uniform merge regression (1 passed)
- `uv run --project python --extra tests --extra dev ruff format .`
- `uv run --project python --extra tests --extra dev ruff check .`
- `cd docs && PYTHONPATH=. ../python/.venv/bin/mkdocs build`
Fixes#3967
<!-- lance-gatekeeper-fix:v1 agent=572be272619660b97e87fd5c85188341
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Listing tables a page at a time against a local database silently
skipped one table at every page boundary. `ListingDatabase::list_tables`
returned the first name of the *next* page as that page's token, but
resuming from a token drops every name at or before it — so the table
the token named was never handed to the caller. Walking `[a, b, c, d,
e]` with a limit of 2 returned `[a, b, d, e]`.
This PR returns the last name of the page as the token instead, which is
what resuming after the token expects.
This is reachable from Python today through
`db.list_tables(page_token=...)` on a local connection; it also affects
`len(db)` and `name in db`, which walk the pages. Remote and
namespace-backed connections page on the server and were never affected.
## Example
```python
db = lancedb.connect(tmp_path)
for name in ["a", "b", "c", "d", "e"]:
db.create_table(name, [{"id": 1}])
names, token = [], None
while True:
page = db.list_tables(page_token=token, limit=2)
names += page.tables
token = page.page_token
if not token:
break
# before: ['a', 'b', 'd', 'e']
# after: ['a', 'b', 'c', 'd', 'e']
```
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Problem
`refresh_column_async` returned a unit-result job even though durable
refresh jobs carry a canonical terminal result. Python callers could not
obtain row counts or source and published versions through the public
`Job` API, and local and remote refresh jobs exposed different result
semantics.
## Behavior
`refresh_column_async` now returns `Job[RefreshColumnResult]` for local
and remote tables. The general typed-job bridge binds each endpoint to
its public result model while preserving unit-result jobs and existing
status, wait, cancel, and timeout behavior. A local no-op refresh
reports no published version.
The Node.js API continues to resolve `wait()` as `void`; its binding
erases the Rust result type internally to preserve the existing public
contract.
## Ownership and integration boundary
LanceDB owns the language-neutral `Job<T>` contract and language-binding
decode. Sophon owns production and durable persistence of terminal
payloads. Sophon #7348 and #7378 now publish the canonical refresh
result for Function-backed and expression-backed refresh jobs,
respectively. The remote client fixture matches the merged server
schema; live deployment and end-to-end demo acceptance remain separate
rollout checks.
## How packing works
Consider four tokenized documents:
[1]
[2]
[10, 11, 12, 13, 14, 15, 16, 17]
[20]
With:
```
StreamingDataset(
table,
shuffle=False,
columns=["tokens"],
num_splits=2,
pack_sequences=5,
eos_id=9,
pad_id=0,
blocks_per_epoch=6,
)
```
the documents are assigned to two fixed logical splits. Each split
maintains an independent token buffer, appends eos_id after every
document, and emits blocks of five tokens.
Because blocks_per_epoch=6, each split emits exactly three blocks:
Cycl/e 1:
Split 0: [1, 9, 2, 9, 0] # 9 is eos, 0 is padding
Split 1: [10, 11, 12, 13, 14]
Cycle 2:
Split 0: [0, 0, 0, 0, 0]
Split 1: [15, 16, 17, 9, 20]
Cycle 3:
Split 0: [0, 0, 0, 0, 0]
Split 1: [9, 0, 0, 0, 0]
If a split runs out of tokens early, it emits padded blocks through the
fixed budget. This prevents one rank from finishing before another.
Logical splits are independent of rank and worker ownership. A
checkpoint records each split’s consumed-document count, emitted-block
count, remaining tokens, and document boundaries. Merging
those per-split states allows the same packed stream to resume after the
topology changes.
doc_ids identifies document segments, including continuations across
block boundaries. It is not a padding mask: padding retains the
preceding document ID, so callers must mask padding using a
reserved pad_id.
blocks_per_epoch="auto" is also available. It estimates the budget from
a deterministic bounded sample and warns that the result is approximate.
WIP pre-training tests:
```
┌────────────────────────────────────┬────────────────────────┬─────────────────────────────────────┐
│ │ GPT-2 124M │ GPT-2 medium 354M │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Corpus │ 2.4M docs / 12GB table │ 9.67M docs / 45GB table │
│ Tokens (Chinchilla) │ 2.43B │ 7.0B │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Data prep (ingest→curate→tokenize) │ ~12 min │ ~51 min │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Training wall time │ ~50 min │ 3h 06m │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Throughput / MFU │ 1.60M tok/s / 35% │ 684k tok/s / 42.0%, │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Final val loss │ 3.230 │ 2.840 │
└────────────────────────────────────┴────────────────────────┴─────────────────────────────────────┘
```
---------
Co-authored-by: OpenAI Codex <codex@openai.com>
Jack Ye
LanceDB Robot
Will Jones
Xuanwo
Lance Release
Wyatt Alt
Drew
lancedb-gatefixer[bot]
Dan Tasse
Ayush Chaurasia