Commit Graph
1396 Commits
Author SHA1 Message Date
Gatefixer 6366bab6a3 Merge main into gatekeeper/fix-4260-1
Keep main's Lance beta.22 update and align the patched Arrow crate metadata while preserving the sliced-child validity fix.

Validated workspace Clippy with warnings denied, 112 Arrow tests and 6 doctests, the nullable struct merge regression, 26 Python tests and the original reproducer, Node build/docs/lint with 518 tests, and source archive contents and dependency resolution. Root Ruff findings remain confined to the unchanged docs/web-tests directory.
2026-10-21 09:12:43 +00:00
Lance Release 46ccbe4e2c Bump version: 0.41.0-beta.10 → 0.41.0-beta.11 2026-10-21 08:21:13 +00:00
951ae51ef6 docs: add realtime engine (RTE) getting-started guide (#4548)
Adds a user-facing realtime engine (RTE) page at
`docs/web/tables/realtime-engine.mdx`, listed under **Management** after
Branches. It is adapted from the internal WAL field guide
(lancedb/sophon#6213), with the deployment and operations material left
out.

The page covers:
- what RTE is and how it works (memtable → SSTables → compaction; reads
merge the base table with the fresh tier)
- getting started: `set_unenforced_primary_key`, indexes, `enable_rte` /
`ALTER TABLE ... SET RTE`, `merge_insert` upserts and queries, partial
updates, deletes, and `checkpoint_rte` / `disable_rte`
- performance: write throughput and shard scaling, and read latency with
and without RTE
- a note on operations not yet supported (time travel, branching,
`count_rows`, row IDs, `TABLESAMPLE`)

Also aligns related docs with current RTE naming and behavior:
- `sql/ddl/table.mdx`: `SET MEMWAL` / `UNSET MEMWAL` → `SET RTE` /
`UNSET RTE`
- `maintenance/background-jobs.mdx`: job names are `memwal_compaction`
and `memwal_catchup_<type>`
- `feature-engineering/materialized-views.mdx`: MemWAL → realtime engine
- Python `AsyncTable.enable_rte` /
`RTEWriteSpec.with_maintained_indexes` docstrings: by default, indexes
created after enabling RTE are maintained too

The examples are written directly in the page rather than generated from
`docs/web-tests`, because RTE needs an Enterprise server to run.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: XY Zhan <zhaner08@hotmail.com>
2026-10-20 21:40:20 -05:00
Lance Release 9e54c8cf27 Bump version: 0.41.0-beta.9 → 0.41.0-beta.10 2026-10-20 17:06:59 +00:00
Dan RammerandClaude Opus 5.5 92825f6f13 refactor!: rename MemWAL APIs to RTE (realtime engine) (#4598)
## What

Renames the user-facing MemWAL surface to **RTE** (realtime engine)
across Rust, Python, TypeScript, and Java, including the REST wire
format. This follows the LSM → MemWAL rename in #4075.

| Before | After |
|---|---|
| `MemWalWriteSpec` / `MemWalStats` | `RTEWriteSpec` / `RTEStats` |
| `enable_memwal` / `disable_memwal` / `get_memwal_spec` | `enable_rte`
/ `disable_rte` / `get_rte_spec` |
| `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | `flush_rte`
/ `compact_rte` / `checkpoint_rte` |
| `get_memwal_stats` / `close_memwal_writers` | `get_rte_stats` /
`close_rte_writers` |
| `use_memwal` (query, merge insert) | `use_rte` |
| TS/Java `enableMemWal`, `useMemWal`, ... | `enableRTE`, `useRTE`, ...
(napi `js_name` set explicitly) |
| Java `LanceDbTableMemWal` | `LanceDbTableRTE` |
| `table/memwal_stats.rs` | `table/rte_stats.rs` |

Docs and comments now say "realtime engine (RTE)". Text about Lance's
underlying mechanism keeps "MemWAL" ("Lance's MemWAL", "MemWAL index").

**Left alone:** Lance-owned names (`MemWalIndexDetails`,
`DatasetMemWalExt`, `InitializeMemWal*`, `mem_wal` paths), internal
`Lsm*` names and modules, and the `does not support with_row_id` refusal
text that the hybrid fallback matches on.

## Compatibility

Breaking, for an experimental API. The wire format changes, so this must
roll out together with the matching sophon change (lancedb/sophon#8870):

- Routes: `/enable_memwal/` → `/enable_rte/`, `/disable_memwal/` →
`/disable_rte/`, `/get_memwal_spec/` → `/get_rte_spec/`,
`/get_memwal_stats/` → `/get_rte_stats/`, `/flush_memwal/` →
`/flush_rte/`, `/compact_memwal/` → `/compact_rte/`
- Response keys: `memwal_spec` → `rte_spec`, `memwal_stats` →
`rte_stats`
- Request field: `use_memwal` → `use_rte` on query and merge-insert
bodies

The SQL keyword becomes `ALTER TABLE ... SET RTE` server-side. The
guides under `docs/web` will be updated separately.

## Validation

- Rust: `cargo clippy --features remote --tests --examples` clean;
`cargo test --features remote -p lancedb --lib`: 1533 passed
- Java: `mvn test`: 37 passed (Spotless applied with JDK 11)
- Python: `ruff format` / `ruff check` clean; targeted tests
(`test_rte_spec`, `test_merge_insert_lsm`, `test_remote_db`,
`test_wal_hybrid`, `test_query`, `test_permutation`, `test_table`): 708
passed
- TypeScript: `pnpm build`; `table`, `query` tests: 606 passed; `biome
check` clean; `docs/src/js` regenerated with `pnpm run docs`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 11:58:57 -05:00
Lei XuandClaude Opus 5.5 1772f4ef83 fix: don't panic building permutations over tables with schema metadata (#4602)
## Problem

`permutation_builder(table).execute()` and `StreamingDataset(table,
...)` (which builds a permutation at construction) panic with
`RustPanic` on any table whose Arrow schema has schema-level metadata:

```python
schema = pa.schema([pa.field("x", pa.int64())], metadata={b"huggingface": b"{}"})
t = lancedb.connect(tmp).create_table("t", pa.table({"x": list(range(100))}, schema=schema))
permutation_builder(t).split_random(fixed=2, seed=1).execute()   # RustPanic
```

Every Hugging Face Lance dataset triggers this, e.g.
`hf://datasets/lance-format/lerobot-xvla-soft-fold/data` table `frames`
carries `source-column-name-map`.

## Root cause

`PermutationBuilder::add_config_metadata` replaced the stream schema's
metadata with the permutation config and then called
`batch.with_schema(schema).unwrap()`. The base-table scan's batches
carry the base schema metadata, and `RecordBatch::with_schema` requires
the target to be a superset of the current schema, metadata included. It
returned `SchemaError("target schema is not superset of current schema
...")`, and the `unwrap` turned that into a panic on a tokio worker.

## Fix

- The permutation table deliberately does **not** inherit base-table
schema metadata. That metadata describes columns the permutation table
(`row_id`, `split_id`) does not have, and its keys can collide with
config keys such as `split_names`. A merge would let a base table's
`split_names` leak into, or override, the permutation's split names.
Batches are rebuilt with `RecordBatch::try_new(schema, columns)`.
- Errors are propagated (`map_ok` → `and_then`) instead of `unwrap`.
- The same panic-on-worker pattern in the splitter's streaming closures
(`split.rs`: `try_new`/`filter(...).unwrap()`) now propagates errors
too. Nothing else in `dataloader/` used `with_schema(...).unwrap()`.

## Tests

- Rust: `test_permutation_over_table_with_schema_metadata` in
`builder.rs` builds a random split + shuffled permutation over a table
with `huggingface` and a colliding `split_names` key. It panics with the
original `SchemaError` on the old code and passes now.
- Python: `test_permutation_over_table_with_schema_metadata` in
`test_permutation.py` covers the repro and reads both splits through
`Permutations`/`Permutation.from_tables`.
`test_streaming_dataset_over_table_with_schema_metadata` in
`test_elastic_dataloader.py` covers `StreamingDataset` construction and
iteration.
- `cargo fmt`, `cargo clippy --features remote --tests --examples`, the
dataloader Rust tests (42), `test_permutation.py` (72), and
`test_elastic_dataloader.py` all pass.

`test_torch.py::test_remote_permutation_dataloader_multiprocessing`
fails locally with `take query result did not include ordering column
'__lancedb_take_row_offset'`. It uses `Permutation.identity`, which does
not go through `PermutationBuilder`, so this change does not cause it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 09:16:18 -07:00
lancedb-gatefixer[bot] 4a03859b2a fix!: normalize blob layouts and support external byte ranges (#4589) 2026-10-19 16:44:00 -07:00
Gatefixer eb4535398a Merge main into gatekeeper/fix-4260-1
Resolve Lance pins to beta.21 and refresh the vendored Arrow crate while preserving the sliced-child validity fix and regression tests.
2026-10-19 22:10:18 +00:00
Lance Release 6507615a6f Bump version: 0.41.0-beta.8 → 0.41.0-beta.9 2026-10-19 21:23:24 +00:00
Tobias 1ad97b148b build(python): publish an sdist alongside the wheels (#4522)
Closes #4246.

lancedb publishes no source distribution, so a platform without a
prebuilt wheel (#4104, #4239) has nothing to install. This adds an sdist
to the Python release.

## The sdist job

The release workflow builds an sdist with the maturin version of the
wheel jobs. On a tag, the publish job uploads it with the wheels: to
PyPI, and to Fury for beta tags. The job also runs on the pull request
dry run, which now starts on a change to `python/pyproject.toml`.

## Checks

- The job builds a wheel from the sdist with `--locked` and smoke-tests
it in a fresh venv.
- `ci/check_sdist.py` fails if a tracked file of the Python package is
missing from the tarball.
- `twine check --strict` checks that the metadata renders. `pydistcheck`
enforces PyPI's 100 MiB size limit.
- The unit tests of both check scripts run in the CI scripts workflow.

## Cargo.lock

maturin ships the repository lockfile, which pins crates that only the
dropped `nodejs` member uses, so `cargo --locked` fails
(PyO3/maturin#2609). The job first runs `cargo metadata --locked` on the
repository workspace, so a lockfile that has drifted from `Cargo.toml`
fails there. It then removes the nodejs-only pins and repacks the sdist.
`ci/check_cargo_lock.py` fails if the prune changes any package it
keeps, or any dependency of one. The published file is the repacked one.

## Behavior change

After the first release with an sdist, pip on a platform with no current
wheel builds the newest version from source, which needs a Rust
toolchain and `protoc`. Before, pip installed the last version with a
wheel for that platform.
2026-10-19 13:35:07 -07:00
Will JonesandClaude Opus 5.5 9020640538 fix(python)!: don't trust embedding settings from table metadata (#4550)
BREAKING CHANGE: `SentenceTransformerEmbeddings.trust_remote_code` now
defaults to `False` (was `True`). When an embedding function is restored
from table metadata, `trust_remote_code=True` and custom endpoints
(OpenAI `base_url`, Ollama `host`/proxy, watsonx `url`) are only honored
if the process approved them; otherwise computing embeddings raises
`UntrustedEmbeddingConfigError`. Functions created in the same process
are approved automatically. To reopen such a table in a new process,
call `get_registry().allow_remote_code(model, revision=...)` or
`get_registry().allow_endpoint(url)` first.

## Summary

Embedding functions are rebuilt from table metadata, which anyone who
can write the table controls. Since sentence-transformers defaulted
`trust_remote_code` to `True`, opening and querying a crafted table ran
arbitrary code from a model repository (GHSA-x8v2-vj73-gfwm). Metadata
could likewise point `base_url` at another server, sending it queries
and the `OPENAI_API_KEY` from the environment.

The table can no longer grant itself trust:

```python
registry = get_registry()
registry.allow_remote_code("nomic-ai/nomic-embed-text-v1.5", revision="<commit>")
registry.allow_endpoint("http://my-ollama-host:11434")
```

Trust is keyed by model name (optionally pinned to a revision), so
approving one model doesn't let a table swap in another. Restored
functions that aren't approved are replaced by a placeholder that raises
on use without constructing the real function (`huggingface` loads its
model in `__init__`), so non-embedding operations on the table keep
working. `sentence-transformers` and `huggingface` gain a `revision`
parameter. Custom embedding functions can opt in by overriding the new
`remote_code_model()` / `endpoints()` hooks.

Not covered: TypeScript (its built-ins have neither remote code nor
configurable endpoints), Bedrock `assumed_role`, and third-party
functions that don't implement the hooks.

## Evidence

`test_embeddings_trust.py` covers restore with and without approval,
revision pinning, swapped model names, endpoint gating, and that the
table stays usable. Opening a table with crafted metadata and searching
it now gives:

```
UntrustedEmbeddingConfigError: Embedding function 'sentence-transformers' was restored from table metadata with settings this process has not approved:
  - runs code from model 'attacker/model' at revision None. If you trust it, call get_registry().allow_remote_code('attacker/model', revision=None)
```

## Merge Risk

**Door:** One-way: changes a default and makes previously working
reopen-and-search flows require explicit approval.
**Blast Radius:** public-api (Python embeddings); users of remote-code
models or custom endpoints who reopen tables in a new process.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 12:11:25 -07:00
Weston PaceandClaude Sonnet 5 ec53c82ef8 fix(python): bound StreamingDataset's raw_batches queue against io_queue_depth (#4530)
## Summary
- `StreamingDataset`'s Stage 1 (I/O) queue (`raw_batches`) had no cap:
`_fill_io` only limited *in-flight* fetch futures (`io_pending`) to
`io_queue_depth`, but every completed fetch was unconditionally drained
into `raw_batches` and the freed slot immediately refilled, so I/O could
race arbitrarily far ahead of a slow transform stage with no bound on
buffered memory.
- `transform_queue_depth`'s admission check (`_try_submit_tx`) couldn't
compensate either, since it only counts `cooked` rows and in-flight
transform futures — it never looks at `raw_batches`.
- Fix: `_fill_io` now counts both `io_pending` and `raw_batches` against
the `io_queue_depth` budget, matching the parameter's documented
"batches to keep in flight per split" contract.

## Test plan
- [x] Added
`test_raw_queue_depth_bounded_when_transform_is_the_bottleneck`
(deterministic, via synchronous I/O) asserting `raw_queue_depth <=
io_queue_depth * read_batch_size` even when the transform stage is the
bottleneck.
- [x] Full `python/python/tests/test_elastic_dataloader.py` suite passes
(190/190), confirming no change to sample ordering, counts, or
checkpoint/resumability behavior.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-19 11:09:27 -07:00
lancedb-gatefixer[bot] a4f6e5fe37 fix(python): reject blob ranges unsupported by storage schemas (#4566) 2026-10-18 11:17:58 -07:00
Lance Release c54b4027ef Bump version: 0.41.0-beta.7 → 0.41.0-beta.8 2026-10-18 18:15:51 +00:00
Wyatt Alt 0220f3af17 feat(python): accept large_binary in Function signatures (#4577)
udf() rejected a large_binary input or output, and large_binary nested in a list or struct, outside a Blob v2 payload. It is now a grammar primitive like large_utf8, and the shared type golden lists it.
2026-10-18 11:13:18 -07:00
Gatefixer b604b425bb Merge main into repair branch and preserve beta.19 validity fix 2026-10-16 21:49:42 +00:00
lancedb-gatefixer[bot] 44ba9c6dee fix: support remote blob range reads (#4553) 2026-10-16 13:43:55 -07:00
XY Zhan ea1d2b172a fix: require a primary key to enable MemWAL (#4561)
## Why

Every MemWAL write and read resolves rows by the table's unenforced
primary key, but `enable_memwal` accepted a table without one. The spec
installed, and every MemWAL `merge_insert` then failed with "table has a
MemWAL index but no unenforced primary key" until MemWAL was disabled.
Lance's MemWAL initialization doesn't catch it: its key checks pass when
there is no key.

## What changes

`enable_memwal` refuses a spec on a table with no unenforced primary
key, for every sharding mode, before anything is committed. The check
runs against the dataset the spec is committed to; a key cannot be
removed once set, and an overwrite or restore of the table conflicts
with that commit. Servers that install specs through this method get the
check too.

The Rust doc on `Table::enable_memwal` now states the requirement; the
Python and TypeScript docs already did.

## Tests

- New `test_enable_memwal_requires_primary_key` (Rust and Python):
unsharded, bucket and identity specs are all refused without a key, a
refused spec leaves nothing installed, and the same table accepts a spec
once a key is set. The Rust test fails with the check disabled.
- Tests that installed a spec on a table without a key now declare one
first (6 Rust, 3 Python).
- `cargo nextest run -p lancedb --features remote --lib`: 1529 passed.
`cargo clippy -p lancedb --features remote --tests --examples -- -D
warnings` and `cargo fmt --check` are clean; no integration test or
example installs a spec.
- Python: `test_memwal_spec.py` and `test_merge_insert_lsm.py`, 43
passed; `ruff format` and `ruff check` are clean.
2026-10-16 14:27:53 -05:00
Dan RammerandClaude Opus 5 fe34eb3549 refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What

Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.

### 1. MemWAL APIs drop "LSM"

The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.

| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |

TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.

Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.

### 2. Stats use SSTable and table shard

Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.

**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.

**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.

| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |

Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.

`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.

## Compatibility

Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:

- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies

## Validation

- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-16 13:33:57 -05:00
Dan RammerandClaude Opus 5 0968c21f4c feat(remote): support set_unenforced_primary_key (#4158)
## Summary

`RemoteTable::set_unenforced_primary_key` returned `NotSupported`, so
the call failed against LanceDB Cloud and enterprise from every SDK —
Python and TypeScript both forward to it. That also blocked sharded LSM
writes, since the server rejects bucket/identity sharding on a table
that declares no unenforced primary key, and the SDK had no way to
declare one.

No server-side support was missing. The unenforced primary key is Lance
schema field metadata, and the existing `update_field_metadata` endpoint
writes exactly that, so the remote table now installs the key through
it. The commit layer behind that endpoint installs the position,
enforces immutability and runs `verify_primary_key()` — the same code a
native table reaches, so both paths agree on semantics and not just on
error messages.

## Changes

- `rust/lancedb/src/table/primary_key.rs`: the request validation
(single column, column exists, supported dtype, key not already set) and
the metadata edit (set the 1-indexed position, clear the legacy boolean
key) move into `validate()` and `install_edit()`, shared by both table
types so they cannot drift. Native behaviour is unchanged.
- `rust/lancedb/src/remote/table.rs`: real implementation —
`check_mutable()`, validate against the described schema, then one
`update_field_metadata` request.
- `python/python/lancedb/remote/table.py`: drop the "Not supported on
LanceDB Cloud." docstring.

No binding changes were needed; Python and TypeScript already forward to
the Rust `BaseTable` method.

## Tests

Three mocked-server tests in `rust/lancedb/src/remote/table.rs`:

- the happy path, asserting the request body carries
`lance-schema:unenforced-primary-key:position: "1"` and a null legacy
key on the right field path;
- the invalid requests a native table also rejects (empty, compound,
unknown column, unsupported dtype), each failing before any write
reaches the server;
- the already-set case, rejected from the schema the server already
reports.

`cargo test --features remote -p lancedb --lib`: 1186 passed, 0 failed.
`cargo clippy --features remote -p lancedb --lib --tests` is clean.

## Note for enterprise

Sophon pins the lancedb submodule at `v0.39.0-beta.6`, so this reaches
enterprise on the next bump.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011xV8EebEZDmDDV8hVBm6MD

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-16 11:41:27 -05:00
Gatefixer d46525e056 Merge main version bump to 0.41.0-beta.7 2026-10-15 23:13:47 +00:00
Gatefixer 21132eb251 Merge main and retain sliced Arrow validity fix 2026-10-15 22:56:09 +00:00
Lance Release 7b3ba4c437 Bump version: 0.41.0-beta.6 → 0.41.0-beta.7 2026-10-15 22:35:51 +00:00
Jack Ye 9f37f39267 fix(python): resolve the single vector column despite a query dimension mismatch (#4526)
\`inf_vector_column_query\`'s \`dim\` filter (added in #3423) is meant
to disambiguate between multiple vector columns of different sizes. When
a schema has exactly one vector column and the query's dimension doesn't
match it, the filter excluded that column too, raising "There is no
vector column in the data" instead of resolving the column and letting
the dimension mismatch surface at query execution.

This was latent until #4409 wired client-side vector-column inference
into remote search: a mismatched-dimension query against a
single-vector-column remote table now fails client-side with the wrong
error, before ever reaching the server's own dimension validation.

Fall back to the unfiltered candidate set when the dimension filter
leaves zero matches, and only raise when that is still ambiguous or
genuinely empty.
2026-10-14 17:40:04 -07:00
lancedb-gatefixer[bot] 4be1720c51 fix(python): support default blob conversion in pandas query results (#4308) 2026-10-14 12:23:10 -07:00
Drew 52da9ed934 feat(python): add blob_mode to to_arrow and to_list (#4529) 2026-10-14 10:57:19 -07:00
lancedb-gatefixer[bot] e485469e92 fix(python): coerce URI strings for blob v2 writes (#4294) 2026-10-14 10:53:03 -07:00
lancedb-gatefixer[bot] a1e4dc9f91 fix(python): align BlobFile seek and read behavior (#4290) 2026-10-14 10:41:54 -07:00
lancedb-gatefixer[bot] 3336bdfbec fix(remote): bound concurrent blob reads across handles (#4311) 2026-10-14 10:27:25 -07:00
lancedb-gatefixer[bot] 6487503cff fix: omit unfetchable list blobs from column discovery (#4312) 2026-10-14 09:50:31 -07:00
lancedb-gatefixer[bot] d2a42e9056 fix: coerce blob items in list columns on add (#4302) 2026-10-13 21:44:58 -07:00
Colin Patrick McCabe 6cc4919c3c feat: add an authorization API (#4363)
Add an authorization API to the LanceDB catalog. This is an API that can
be used to bind access control lists (ACLs) to individual catalog
objects. It can also be used to manage server-side roles, and introspect
principals and groups. The new API is implemented in both the Rust,
Python, and async Python styles and includes list_acls, add_acl,
delete_acl, list_role_bindings, add_role_binding, delete_role_binding,
get_principal, and get_group. This API implements the design principle
of forwards-compatibility. So unknown privilege, object, and subject
types do not cause errors, but are be supported as best as possible in
the library.
2026-10-13 19:59:25 -07:00
lancedb-gatefixer[bot] c6eeb17f62 fix(python): preserve blob v2 columns on table overwrite (#4289) 2026-10-13 15:03:15 -07:00
lancedb-gatefixer[bot] 0212cfd3d6 fix(python): normalize blob row ID inputs and errors (#4293) 2026-10-13 14:49:51 -07:00
SIDDARTHA REDDY 38a62db418 feat(python,rust): allow lancedb.blob() to set blob tier thresholds (#4323) 2026-10-13 14:36:24 -07:00
Gatefixer 433bd29429 Merge main and retain sliced Arrow validity fix 2026-10-13 17:08:34 +00:00
lancedb-gatefixer[bot] 792ffd1082 fix(python): materialize nested blob bytes in pandas (#4305) 2026-10-12 12:56:39 -07:00
Lance Release 0a988a40ac Bump version: 0.41.0-beta.5 → 0.41.0-beta.6 2026-10-12 11:38:01 +00:00
Yang Cen 0bceaaf3f0 feat(search): expose search effort (#4520)
Lance supports `search_effort`, but LanceDB cannot pass it through its
query APIs. Expose the parameter in Rust, Python sync/async and
TypeScript, including remote requests and hybrid query conversion, using
the [Lance
v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13)
dependency already on main.

The default is `None` (Auto, internally 0.5). Any explicit effort,
including 0.5, overrides fixed `nprobes` in either setter order;
explicit minimum/maximum bounds remain effective. Invalid values are
rejected in the Rust core. Hybrid conversion now retains the vector
probe parameters and handles an inferred FTS column without panicking.

Namespace queries with explicit effort execute locally because the
namespace wire model cannot carry it. MemWAL LSM queries reject it
explicitly because Lance's LSM scanner does not yet expose this option.
Maturin is included in the dev extra so the documented build command
works in the project environment.

BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now
includes `search_effort: Option<f64>`. Downstream code that initializes
every field explicitly must add `search_effort: None` or use
`..Default::default()`. Exhaustive destructuring patterns must include
the new field or `..`. Query-builder calls and struct literals that
already use `..Default::default()` require no source changes. The new
effort setters in Python and TypeScript are additive. This PR does not
change the persisted file format.

Using explicit effort with remote tables requires server support for
`search_effort` and its precedence over `nprobes`; the client wire tests
do not establish that a deployed server supports it. Server-side support
is tracked in lancedb/sophon#8607.

Validation on AWS after merging current main (Lance beta.13 / SDK
beta.5):

- 14 Rust regressions and the API doctest pass.
- 60 parameter regressions, the full 47-test namespace suite, and three
API doctests pass. The parameter selection includes some namespace
cases.
- 40 TypeScript query tests and 54 remote tests pass; build, lint and
generated docs pass.
- Rust Clippy, Python package lint/format, Pyright and lockfile checks
pass. Repository-wide Ruff reports 49 lint errors and 11 formatting
failures in unchanged `docs/web-tests` files; those files are
byte-identical to main and outside this change.

All 1,103 Git-tracked files match the AWS validation checkout. No
benchmark scripts or result artifacts are included, and no performance
improvement is claimed.

Uses lance-format/lance#9798. Server-side support is in
lancedb/sophon#8607.
2026-10-12 19:32:38 +08:00
Gatefixer 0c9e235bff Merge main into gatekeeper/fix-4260-1 2026-10-12 07:01:57 +00:00
Lance Release f32e9ab26a Bump version: 0.41.0-beta.4 → 0.41.0-beta.5 2026-10-12 06:04:07 +00:00
Gatefixer 3a66770e1a Merge main into gatekeeper/fix-4260-1
Align the vendored Arrow crate and JSONB pin with Lance 14.0.0-beta.11. Preserve the sliced-validity repair and carry the release's Null-child fix.
2026-10-11 07:30:44 +00:00
Lance Release 142d032cf1 Bump version: 0.41.0-beta.3 → 0.41.0-beta.4 2026-10-11 06:35:18 +00:00
Kevin Dong 1f378c1b49 fix(python): preserve Decimal precision in table updates (#4431)
Python `Decimal` values and Arrow decimal scalars raise
`NotImplementedError` in `table.update(values=...)`. Register a Decimal
converter that emits quoted fixed-point text so the native cast to the
target column preserves every digit, including decimal128 and decimal256
values. Non-finite Decimal values raise `ValueError` with a static
message.

Closes #4360.

Validation with the native extension built from the current source at
7f593359, on macOS arm64 and Python 3.12.14:

- All 33 new regression cases failed before the fix and passed
afterward. They cover Python and Arrow scalar updates, signs, zero,
exponents, full 38/76-digit precision, and a Decimal context precision
of 6.
- 16 existing converter, update, and Decimal expression tests passed.
- Repository Ruff formatting and lint passed; formatting left all 156
files unchanged.
- A native round-trip check confirmed that an unquoted SQL number rounds
a high-precision value while the quoted Decimal update preserves it
exactly.

The full suite, other Python versions, and remote API were not
exercised.

AI assistance: Codex helped implement and validate the Decimal
conversion and its regression tests.
2026-10-09 16:51:56 -07:00
Lance Release 55ca56d4e2 Bump version: 0.41.0-beta.2 → 0.41.0-beta.3 2026-10-09 21:13:42 +00:00
Gatefixer f75f7f2112 Merge main into gatekeeper/fix-4260-1
Preserve sliced child validity with Lance v14.0.0-beta.10 and its JSONB dependency.
2026-10-09 20:43:58 +00:00
Jack Ye f305a05fd4 fix: preserve adaptive IVF probe defaults (#4168)
Keep IVF probe fields unset by default so Lance supplies adaptive
behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes`
independently through Rust, Python, namespace, and remote request
construction, with compatibility fields serialized only when explicitly
configured.

The final Rust namespace and LSM propagation update depends on the Lance
release containing lance-format/lance#9184.
2026-10-09 12:45:02 -07:00
Lance Release f85c235d48 Bump version: 0.41.0-beta.1 → 0.41.0-beta.2 2026-10-09 18:21:31 +00:00
Jonathan HsiehandClaude Opus 5.5 ab2ae49200 feat!: carry a FunctionVersion's creation time as epoch millis (#4482)
Moves Function timestamps on the wire from RFC 3339 strings to epoch
milliseconds.

## What

- `FunctionVersion.created_at` (an RFC 3339 string) becomes
`created_at_millis` (milliseconds since the epoch, as an integer).
That's the shape Secrets, jobs and `FunctionErrorRecord` already use.
- Rust: `FunctionVersion::created_at() -> &str` becomes
`created_at_millis() -> i64`.
- Python: `FunctionVersion.created_at: str` becomes `created_at_millis:
int`.
- Fixtures and the mock server now send `1787270400000`, the old
`2026-08-21T00:00:00Z`.

Node.js doesn't expose `FunctionVersion`, so it doesn't change.

## Compatibility

Breaking, and it must ship together with the matching server change: a
server that still sends `created_at` fails to deserialize on this
client. Functions aren't deployed yet, so nothing is migrated.

## Tests

- `cargo test --features remote --test first_class_function_slice1`: 10
passed
- `cargo test --features remote --lib function`: 51 passed
- `pytest python/tests/test_first_class_function*.py`: 100 passed
- clippy, `cargo fmt --check`, ruff check and ruff format are clean

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01BY5xcEGEAa1E11VG55Kd1w

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:17:53 -07:00
Xuanwo 0aa842b17f feat: support FTS indexes on JSON fields (#4219)
`FTS()` rejects JSON columns because validation only considers their
`LargeBinary` storage type. Validate the complete Arrow field in
embedded and remote creation so `lance.json` columns reach Lance's
existing JSON-aware inverted index, while ordinary binary fields remain
rejected.

JSON searches use path-qualified triples such as `meta.tag,str,nature`;
the API documentation describes this syntax and its field-scoping
behavior. Remote creation preserves the supplied FTS configuration.

Two integration limits remain: malformed triples can still panic in the
pinned Lance dependency (tracked by lance-format/lance#8812 and the
pending fix in lance-format/lance#8813), and the opt-in server test has
not run against a real deployment because no remote database is
configured. That test covers index completion, statistics, search
results, and index use; HTTP mock coverage only verifies the client
request.

Refs #3891.
2026-10-09 12:48:29 +08:00