JobHandle/AsyncJobHandle now poll conn.get_job(id, table) -- one job -- instead
of list_jobs() + client-side filter over every active job. The job's table is
threaded in from refresh_column / MV refresh as an O(1) lookup hint. Plumbs
get_job through the Database trait (default not_supported), RemoteDatabase
(GET /v1/job/{id}?table=...), the Connection wrapper, and the pyo3 binding +
db.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
self._conn on a remote sync connection is an AsyncConnection (python), which
exposes `lineage` (parses the JSON), not the pyo3 `table_lineage`. The sync
wrapper was calling self._conn.table_lineage -> AttributeError. Drive
self._conn.lineage on the loop instead, mirroring create_materialized_view.
(table_lineage stays the pyo3 method the async path calls via self._inner.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
refresh_column returned the bare job-id str, so callers had to wrap it:
db.job(tbl.refresh_column("c")).wait(). Mirror MaterializedView.refresh() and
return a JobHandle directly, so tbl.refresh_column("c").wait() / .status() / .id
work without the wrapper. (db.job(job_id) stays for reconnecting by a stored id.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- new lineage.py: Lineage / Node / Edge / FunctionRef dataclasses that parse the
server's lineage JSON, with to_dict(), to_graphviz() (drift edges dashed+red),
and _repr_html_(); plus .functions() / .stale() helpers.
- Connection.lineage(table, column=, direction=, depth=) (sync + async) calls the
pyo3 table_lineage binding and deserializes into Lineage.
- Table.lineage(column=, ...) via the table's job connection; MaterializedView /
AsyncMaterializedView .lineage() delegate to the backing table (the server
already includes the view's sources + downstream dependents).
- export the new types.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A new Database::table_lineage(TableLineageRequest) -> Result<String> threaded
end to end: default not_supported in the trait; the remote impl issues
GET /v1/table/{name}/lineage with column/direction/depth query params and
returns the body verbatim; connection.rs exposes a pub wrapper; the pyo3
binding hands the JSON string to Python.
The lineage payload is carried as opaque JSON on purpose: the open-source
lancedb client must not depend on the sophon-internal derived_jobs crate that
defines the lineage schema, so the wire format is the contract and the Python
layer deserializes it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sync _refresh_materialized_view called self._conn.refresh_materialized_view
(no underscore); the async method is _refresh_materialized_view, so
MaterializedView.refresh() raised AttributeError. Add the underscore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sync RemoteDBConnection.create_materialized_view assembled the SELECT but
called the async create_materialized_view with the query as the 2nd positional
arg, which binds to `source=` (query= is keyword-only). Every call then failed
the "needs either query= or both source and select" validation. Pass query=query.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refresh is a submit-a-job verb, so its only public surface should be
MaterializedView.refresh() / AsyncMaterializedView.refresh() (which return a
job handle). Rename the connection methods to _refresh_materialized_view and
have the handles call that, so the raw by-name refresh is no longer advertised
on the connection. The pyo3 native binding is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- create_materialized_view now takes either query= or source+select (folds in
the old create_view builder) and returns a MaterializedView handle whose
.wait() blocks on initial population. create_view is removed -- it was
misnamed (it built a *materialized* view, while CREATE VIEW means the plain
non-materialized view the engine also supports).
- MaterializedView.refresh() and the remote Table.refresh_column() now return a
JobHandle directly, so tbl.refresh_column("c").wait() needs no db.job(...)
wrapper. db.job(id) is narrowed to reconnect-by-id (stored id / SQL / REST).
- rename View/AsyncView -> MaterializedView/AsyncMaterializedView (+ exports).
- tighten the replace path: only a not-found error on the pre-drop is benign;
real failures (perms/server) now surface instead of being swallowed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Geneva Table.load_columns() parity on the REST-only client. Fills existing
columns from an external Parquet/Lance/IPC source by primary-key join.
- BaseTable::load_columns default (NotSupported) + public Table::load_columns,
taking a LoadColumnsRequest (source uris/format/storage_options, target/source
key, (target, source?) column mappings, on_missing, worker/batch/commit knobs).
- Remote impl POSTs to /v1/table/{id}/load_columns with the matching body;
mock test asserts the request shape.
- PyO3 binding + Python remote Table.load_columns(source, pk, columns, *,
source_format, source_pk, on_missing, ...) accepting a column list or
{target: source} dict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
View.refresh(full=True) (sync + async) now works -- it previously raised
NotImplementedError. Thread the flag through the client: RefreshMaterialized-
ViewRequest.full -> the REST body (RemoteRefreshMaterializedViewRequest.full);
pyo3 refresh_materialized_view(full=...); Connection.refresh_materialized_view(
name, full=) sync + async. A full refresh forces a recompute-and-replace and
preserves the view's indexes (reindexed by the distributed indexer).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add View.create_index / create_scalar_index / create_fts_index / search
as pass-throughs to open_table(name). A materialized view is a real Lance
dataset; these let it be indexed and searched like any other table,
closing the parity gap with Geneva (whose create_materialized_view returns
a first-class Table).
The server-side create_index handler records indexes declared on a view so
they survive a full refresh (which overwrites the dataset, dropping its
indices); that re-apply is wired in the sophon engine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thread priority (Kueue tier) through refresh_column at every layer (Python sync+async
+ RemoteTable -> pyo3 -> Rust client trait/public/remote -> REST body), mirroring
num_workers/batch_size. The function keeps its priority as a default; the per-refresh
value overrides. Also adds the previously-missed batch_size to RemoteTable.refresh_column
(the REST sync path). cargo check (lancedb --features remote --tests, lancedb-python) +
ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
batch_size / num_workers / max_workers are invocation concerns (how to schedule THIS
refresh), so expose batch_size on refresh_column through every layer (Python sync+async
-> pyo3 -> Rust client -> the REST RefreshColumnRequest.batch_size, which the handler
already forwards into the backfill). num_workers/max_workers were already invocation-
placed; batch_size was the gap. The function may still carry a default; the refresh
override wins (extends the batch_size_override model). Both crates cargo-check clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A computed column is an expression over a registered function applied to input
columns, not a UDF coupled to a column. fn("data") already returned the expression
string "fn(data)"; make it a ColumnExpr (a str subclass) that also carries the
function's return type, so add_columns(computed={"vec": embed("data")}) declares the
column with no hand-written type. _normalize_computed handles the new form (and tuple
keys for STRUCT fan-out) and keeps the legacy {col: (sql_type, expression)} tuple.
add_computed_column is deprecated (delegates, with a DeprecationWarning). The function
stays decoupled from columns -- register once, apply anywhere.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thread an optional partition_by through the client: CreateMaterializedViewRequest
-> REST body -> pyo3 binding -> Python create_materialized_view/create_view
kwarg (sync + async). The server partitions the view's table function by the
named source column -- by IVF index clusters if the column is indexed
(image-dedup), else by distinct value. Unifies Geneva's partition_by +
partition_by_indexed_column into one knob.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirrors the sync ergonomics on the async surface: AsyncConnection
create_function(udf, replace=)/create_view/job; AsyncTable.add_computed_column;
AsyncView + AsyncJobHandle (await + asyncio.sleep; shared submission-prefix
matcher with the sync JobHandle). Decorator + REST routes are shared/already
validated; this is the async wrapper layer. Exported from the package root.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
db.job(id) gets the submission id the refresh/backfill endpoints return,
but list_jobs / cancel report the agent's manifest id
(<table>-<type>-<first 8 of submission id>). JobHandle now matches that
(exact id or submission prefix) so wait()/progress() truly track, and
cancel() cancels by the resolved canonical id instead of the unusable
submission uuid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings the @udf/@table_udf decorator + type inference into lancedb as
lancedb.udf (Apache-2.0), and adds the ergonomic glue to the existing
connection/table so there's no separate object model:
- create_function() accepts a Udf (and a replace= flag)
- Table.add_computed_column(column, udf)
- create_view(name, source, select, ...) -> View (assembles the SELECT)
- Connection.job(job_id) -> JobHandle
- View / JobHandle are thin references over a connection
Exports udf/table_udf/Udf/JobHandle/View from the package root. The
operations stay the existing remote-only methods (enterprise/cloud); the
decorator works locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Exposes the existing server-side CANCEL JOB (CoordinatorCatalog::cancel_job)
as a REST-backed SDK method: Database trait default NotSupported,
RemoteDatabase POSTs /v1/job/{id}/cancel, pyo3 binding, sync+async python
wrappers. Best-effort: a missing job returns false, not an error. Mock-HTTP
unit test in test_derived_compute_routes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per the interface design: computed columns are parameters on the
existing add_columns operation, not a separate method.
- BaseTable::add_computed_columns((name, sql_type) pairs + a f(args)
expression) -- default NotSupported; RemoteTable posts 'computed'
entries to the existing /v1/table/{id}/add_columns route.
- python add_columns gains computed= on LanceTable, RemoteTable, and
AsyncTable: tbl.add_columns(computed={'doubled': ('FLOAT',
'double_it(val)')}); grouped by expression so struct-returning
functions' columns land adjacently.
Adds the derived-compute interface to the SDK:
- Database trait: create/list/drop_function, create/refresh/alter/
drop/list_materialized_view, list_jobs -- default implementations
return Error::NotSupported (NotImplementedError in python), so
existing Database impls are unaffected; local single-node
implementations are planned. BaseTable gains refresh_column with
the same default.
- RemoteDatabase/RemoteTable implement them against the server REST
routes (/v1/function/*, /v1/materialized_view/*, /v1/job/list,
/v1/table/{id}/refresh_column), with mock-HTTP unit tests.
- Connection/Table public methods, pyo3 bindings (FunctionInfo,
MaterializedViewInfo, JobInfo pyclasses), and python wrappers:
sync on the DBConnection base (shared by local and remote
connections), async on AsyncConnection; refresh_column on
LanceTable, RemoteTable, and AsyncTable.
BREAKING CHANGE: splits generated by the permutation data loader will
not be the same, due to a change in hash function.
Updates the Lance dependencies and Java lance-core to
[v9.0.0-rc.1](https://github.com/lance-format/lance/releases/tag/v9.0.0-rc.1).
Includes the required DataFusion 54 and Lance file-reader compatibility
updates.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds client-side support for analyze_plan distributed metrics modes
across Rust, Python, and TypeScript clients. Defaults to aggregate for
backward compatibility and sends the remote distributed_metrics
parameter only when a non-default mode is requested.
## Summary
Fix `on_bad_vectors="fill"` so it replaces only invalid or missing
vector values instead of replacing the entire vector row.
Fixes#3026.
## Reasoning
The old Python sanitizer detected whether a vector row was bad at row
granularity. For `fill`, it then used that row-level flag to replace the
whole vector with `[fill_value] * dim`. That meant an input like `[1.0,
NaN, 3.0]` became `[0.0, 0.0, 0.0]`, even though the documented and more
useful behavior is to preserve valid values and fill only the bad
element.
I checked whether this should be a Rust-side fix so TypeScript users
would benefit too. Today, Rust core exposes `NaNVectorBehavior::{Error,
Keep}` for rejecting or keeping NaN vectors, while the Python
`on_bad_vectors` API (`error`, `drop`, `fill`, `null`) is implemented in
the Python ingestion sanitizer before data reaches Rust. TypeScript does
not expose the Python `on_bad_vectors="fill"` behavior today. Moving
this exact behavior to Rust would be a broader cross-language API
change, so this PR keeps the fix scoped to the currently affected Python
API.
## What changed
- Added a small helper that fills bad vector rows by preserving valid
elements, replacing NaN elements with `fill_value`, truncating vectors
longer than the expected dimension, and padding short vectors with
`fill_value`.
- Kept the existing fast path unchanged: the helper only runs after bad
vectors are detected and `on_bad_vectors="fill"` is selected.
- Updated sanitizer and table tests to assert element-wise NaN
replacement and short-vector padding for both `create_table` and `add`.
## Validation
- `uv run ruff format .`
- `uv run ruff check .`
- `cd python && uv run --no-sync pytest
python/tests/test_util.py::test_handle_bad_vectors_jagged
python/tests/test_util.py::test_handle_bad_vectors_nan
python/tests/test_table.py::test_create_with_nans
python/tests/test_table.py::test_add_with_nans -vv`
Targeted pytest result: `10 passed`.
## Why this fix is Python-side (and not Rust)
The problematic behavior lives in Python’s `on_bad_vectors` sanitizer,
before data is handed off to Rust. Rust currently only exposes
`NaNVectorBehavior::{Error, Keep}` for add operations, while Python has
the richer `on_bad_vectors={"error","drop","fill","null"}` API.
TypeScript does not currently expose the Python-style fill behavior, so
moving this exact fix into Rust would require designing a broader
cross-language bad-vector handling API.
This PR keeps the change scoped to the existing affected surface:
Python’s `on_bad_vectors="fill"` path. This way, Python users
immediately benefit.
## Summary
- add table-level FTS query tokenization returning token text and
position
- use the native index tokenizer for local tables and remote index
metadata for remote tables
- expose sync and async Python table wrappers with focused coverage
## Summary
- serialize sync phrase queries consistently for execution and query
plans
- restore the documented no-argument hybrid `phrase_query()` behavior
- keep reranker input as the original user text without mutating the
builder
Fixes#3653.
## Testing
- `python/.venv/bin/python -m pytest <8 focused test nodes> -q` (`8
passed`)
- `python/.venv/bin/python -m ruff format --check
python/python/lancedb/query.py python/python/tests/test_fts.py
python/python/tests/test_hybrid_query.py`
- `python/.venv/bin/python -m ruff check .`
- `git diff --check origin/main...HEAD`
The complete hybrid module and the real native FTS phrase test were not
completed
in the current PyO3 runtime environment: both stalled in the native
`lancedb.connect()` fixture and were interrupted without an assertion
failure.
Carrying over from #2915, this patch introduces:
* Single-API call batching support for Gemini embeddings (up to 100 at a
time, the API limit)
* A versioned user agent header for Gemini API calls
* Support for [variable embedding dimension
size](https://ai.google.dev/gemini-api/docs/embeddings#control-embedding-size)
(Gemini is MRL trained)
## Summary
- preserve explicit `0.0` distance bounds in synchronous hybrid search
- distinguish omitted `None` endpoints from zero-valued endpoints when
configuring the vector child query
- add a public end-to-end regression test for a zero upper bound
## Testing
- `cd python && uv run --extra tests pytest
python/tests/test_hybrid_query.py -q`
- `uv run --project python ruff format --check
python/python/lancedb/query.py python/python/tests/test_hybrid_query.py`
- `uv run --project python ruff check .`
Fixes#3651
Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.
## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.
## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.
## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.
Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.
## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps Lance to v9.0.0-beta.19, which includes lance-format/lance#7687
for side-effect-free DirectoryNamespace read paths.
This fixes root-level read-only table opens that previously could
trigger `__manifest` creation through directory namespace construction,
including Hugging Face bucket reads with read-only tokens. A LanceDB
regression test now covers root listing operations without creating
`__manifest`.
Fixes#3633.
### Summary
`flatten_columns` raises `ValueError` when called with `flatten=False`,
even though `False` should mean "do not flatten". This is reachable from
the public API — `Query.to_pandas(flatten=...)` and
`to_batches(flatten=...)` type their `flatten` param as
`Optional[Union[int, bool]]` and pass it straight to `flatten_columns`.
### Cause
`bool` is a subclass of `int`, so `isinstance(False, int)` is `True`.
`flatten=False` skips the `flatten is True` check, falls into the
integer branch, and `False <= 0` evaluates to `True`, raising:
```
ValueError: Please specify a positive integer for flatten or the boolean value `True`
```
### Reproduction
```python
import lancedb
db = lancedb.connect("/tmp/db")
t = db.create_table("t", data=[{"id": 1, "vector": [0.1, 0.2]}])
t.search([0.1, 0.2]).to_pandas(flatten=False) # -> ValueError
```
### Fix
Guard the integer branch with `not isinstance(flatten, bool)` so that
`flatten=False` (and `None`) mean "do not flatten". Behavior is
otherwise unchanged:
- `flatten=True` → flatten all nested levels
- positive `int` → flatten to that depth
- non-positive `int` (e.g. `0`) → still rejected with `ValueError`
Added a regression test in `tests/test_util.py` covering `None`,
`False`, `True`, a positive depth, and `0`.
This PR fixes a serialization error when using Ollama embeddings in
`create_table`.
The use of `@cached_property` for the Ollama client was causing issues
during serialization/pickling, which is required by certain LanceDB
operations (like when using multiprocessing or certain storage
backends). Switching to a standard `@property` ensures the client is
instantiated when needed without being stored in a way that breaks
serialization.
Verified with the following script:
```python
import lancedb
from lancedb.embeddings import get_registry
import pickle
registry = get_registry().get(\"ollama\")
model = registry(name=\"llama3\")
# This would fail before the fix
pickled = pickle.dumps(model)
unpickled = pickle.loads(pickled)
```
Fixes#2629 (or similar serialization issues reported).
---------
Co-authored-by: Unmilan Mukherjee <Missing-Identity@users.noreply.github.com>
## Summary
Closes#3525
This PR wires up two new optional object-store backends at the LanceDB
layer, exposing capabilities that already exist upstream in `lance` /
`lance-io`:
| Backend | Cargo feature | Default in Rust crate | Default in Python
wheel | Default in Node binding |
| --- | --- | --- | --- | --- |
| **Tencent COS** | `cos` | ❌ off | ✅ on | ❌ off |
| **GooseFS** | `goosefs` | ❌ off | ✅ on | ✅ on |
Both backends are additive and do not affect existing users who don't
opt in.
## Motivation
- **Tencent COS** is the dominant object storage in the China region.
Tencent Cloud users currently need an S3-compatible proxy or a private
fork to use LanceDB against COS buckets.
- **GooseFS** is Tencent Cloud's distributed cache acceleration layer
that sits in front of COS/S3, a common pattern for vector search / AI
training where the same hot dataset is read repeatedly.
- This brings COS / GooseFS to feature parity with the existing
first-class backends (`aws`, `gcs`, `azure`, `oss`, `huggingface`).
See the linked issue #3525 for the full discussion.
## Changes
### `rust/lancedb/Cargo.toml`
Add two new optional features that pull through the corresponding
upstream feature flags:
```toml
cos = ["lance/tencent", "lance-io/tencent"]
goosefs = [
"lance/goosefs",
"lance-io/goosefs",
"lance-namespace-impls/dir-goosefs",
]
```
### `python/Cargo.toml`
Enable both `cos` and `goosefs` by default for the Python wheels, so
`pip install lancedb` works against COS / GooseFS out of the box
(consistent with how `aws` / `gcs` / `azure` / `oss` are bundled today):
```diff
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/cos", "lancedb/goosefs"]
```
### `nodejs/Cargo.toml`
Enable `goosefs` by default for the Node binding (COS kept opt-in to
limit the default native binary size; can be revisited based on demand):
```diff
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/goosefs"]
```
### `Cargo.lock`
Regenerated to reflect the transitive dependencies brought in by the new
upstream features. No manual edits.
## Example Usage
### Rust
```toml
# Cargo.toml
lancedb = { version = "0.30", features = ["cos", "goosefs"] }
```
```rust
// Tencent COS
let db = lancedb::connect("cos://my-bucket/my-db").execute().await?;
// GooseFS
let db = lancedb::connect("goosefs://my-namespace/my-db").execute().await?;
```
### Python
```python
import lancedb
db = lancedb.connect(
"cos://my-bucket/my-db",
storage_options={
"secret_id": "...",
"secret_key": "...",
"region": "ap-guangzhou",
},
)
```
## Backwards Compatibility
- All new features are **opt-in** at the Rust crate level (`default =
[]` for `lancedb` itself is unchanged).
- The Python wheel gains both backends by default, increasing wheel size
slightly but matching the existing pattern of bundling all major cloud
backends.
- Node binding only adds `goosefs` to defaults; existing users see no
behavior change.
## Testing
- `cargo check --all-features` ✅
- `cargo check -p lancedb --features cos` ✅
- `cargo check -p lancedb --features goosefs` ✅
- End-to-end COS / GooseFS smoke tests require Tencent Cloud credentials
and are intentionally not added to CI in this PR (same approach used for
`s3-test`). Happy to add a gated test feature in a follow-up if
reviewers prefer.
## Checklist
- [x] Added `cos` and `goosefs` features to `rust/lancedb/Cargo.toml`
- [x] Updated `python/Cargo.toml` default features
- [x] Updated `nodejs/Cargo.toml` default features
- [x] Regenerated `Cargo.lock`
- [x] Verified build with `--all-features`
- [ ] Documentation update (can be done in a follow-up PR once API
stabilizes)
## Related
- Issue: #3525
- Upstream support:
[`lance/tencent`](https://github.com/lance-format/lance),
[`lance/goosefs`](https://github.com/lance-format/lance)
## Summary
Adds `Table::get_lsm_write_spec` returning `Option<LsmWriteSpec>` — the
read counterpart to the existing `set_lsm_write_spec` /
`unset_lsm_write_spec`. Returns `None` when the MemWAL LSM write path is
not enabled; otherwise reconstructs the spec (mode, shard column,
`num_buckets`, `maintained_indexes`, `writer_config_defaults`) exactly
as installed.
## Changes
- **Rust core (`NativeTable`)** — reconstructs the spec from
`mem_wal_index_details()`, resolving the shard column from its Lance
field id via the dataset schema. This is a raw metadata read, so it is
unaffected by `describe_indices` system-index filtering.
- **Remote (`RemoteTable`)** — reads the `__lance_mem_wal` system index
through `index/list` with `include_system: true` (so the curated
`list_indices` surface stays unchanged), then parses the index `details`
JSON. It matches the index by name and ignores `index_type`, so no
client `IndexType` variant is needed. It uses the **server-resolved
`column` name** from the details (Lance field ids do not travel to the
remote client).
- **Python + TypeScript bindings** — sync and async, mirroring
`set`/`unset`, with round-trip tests (bucket / identity / unsharded,
plus `None` when unset).
## Tests
- Rust: native round-trip unit test + remote mock-endpoint tests
(present + absent). All green (`cargo test --features remote -p
lancedb`).
- Python/TS: round-trip tests added; binding-runtime execution runs in
CI.
## Dependencies for the remote path
The remote path is complete on the client side but depends on two
out-of-repo pieces to work end-to-end:
1. **lance** — emit the server-resolved shard **`column`** name in the
MemWAL index `details` JSON (field ids can't reach the client). See
lance-format/lance#7667.
2. **server** — honor `include_system` on `index/list` so the
`__lance_mem_wal` entry is returned for this read.
Against an older server (no `include_system`), the remote getter
degrades gracefully to `Ok(None)` rather than erroring.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Closes#3245.
> **BREAKING CHANGE:** `with_format("torch")` no longer returns a list
of stacked row tensors. It now returns per-row dicts so PyTorch's
default `DataLoader` collate stacks them into `{col: tensor(B,)}`.
Switch to `with_format("torch_row")` to keep the old shape.
### What changed
`"torch"` now returns a list of per-row dicts (`[{col: tensor}, ...]`)
at every indexed access path. The default `DataLoader` collate stacks
them into a column-keyed batched dict, no custom `collate_fn` needed.
The old shape is preserved under a new `"torch_row"` literal.
`"torch_col"` is unchanged.
The unbatching lives inside the transform (`batch_to_tensor_dict`), not
`__getitems__`, so the shape survives pickling and works under
`DataLoader(num_workers>0, multiprocessing_context="spawn")`.
### Format comparison
| Format | `iter(batch_size=N)` | `__getitems__([0,1,2])` | `DataLoader`
default collate |
|---|---|---|---|
| `"torch"` (new) | `list[{col: tensor}]` length N | `list[{col:
tensor}]` length 3 | `{col: tensor(B,)}` |
| `"torch_row"` (old `"torch"` behavior) | `list[tensor(n_cols,)]`
length N | `list[tensor(n_cols,)]` length 3 | `tensor(B, n_cols)` |
| `"torch_col"` (unchanged) | `tensor(n_cols, N)` | `tensor(n_cols, 3)`
| needs `collate_fn=lambda x: x` |
Output matches HuggingFace `Dataset.set_format("torch")` on container
shape, keys, and values at every access path. The only divergence:
HuggingFace downcasts `float64` to `torch.float32` by default, LanceDB
preserves dtype. Verified by `scripts/verify_torch_format.py`.
### Migration
```python
# Old default — column names lost, shape was tensor(B, n_cols)
DataLoader(Permutation.identity(table).with_format("torch"))
# New default — column names preserved
DataLoader(Permutation.identity(table).with_format("torch")) # {col: tensor(B,)}
# Keep old behavior
DataLoader(Permutation.identity(table).with_format("torch_row")) # tensor(B, n_cols)
```
### **Summary**
Closes#3212
Extends the Python `lit()` helper to natively support three additional
types (`date`, `datetime`, and `Decimal`) and implements reflexive
operators for the `Expr` class.
This implementation specifically addresses the blocking feedback
regarding precision loss, CI discovery, and query engine limitations:
* **Logic Refactoring**: Simplified `lit()` by combining `date` and
`datetime` normalization into ISO-8601 strings, ensuring stable SQL
parsing across different engine locales.
* **Precision Preservation**: `decimal.Decimal` objects are now passed
as high-precision strings to the Rust bridge, bypassing intermediate
float conversions and preserving full 128-bit decimal precision for
DataFusion.
* **Averted CI Failures**: Temporarily deferred `bytes` literal support
to a future PR to resolve a known DataFusion `expr_to_sql` limitation
that was crashing the `Doctest` runner.
* **Reflexive Operators**: Added support for "literal-first" arithmetic
and logical operations (e.g., `10 + col('a')` or `True &
col('active')`). Redundant reflexive comparisons (e.g., `__rlt__`) were
pruned as Python's data model handles them automatically.
* **Integration Verification**: Added dedicated integration tests in the
official test directory to ensure the query engine correctly handles the
new types and preserves bit-perfect fidelity.
### **Changes**
####
[python/python/lancedb/expr.py](file:///c:/Users/Laksh/Documents/lancedb/python/python/lancedb/expr.py)
* Updated `lit()` to handle `date`, `datetime`, and `Decimal` natively.
* Implemented reflexive operators (`__radd__`, `__rand__`, `__rmul__`,
etc.) to support literals on the left-hand side.
* Removed the problematic `bytes` doctest example and `lit()` type
support to unblock CI.
####
[python/src/expr.rs](file:///c:/Users/Laksh/Documents/lancedb/python/src/expr.rs)
* Modified the Rust FFI bridge to extract `Decimal` objects as strings.
* Ensured the `expr_lit` handler is ready to receive normalized temporal
strings.
* Consolidated imports and added missing operator documentation.
####
[python/python/lancedb/_lancedb.pyi](file:///c:/Users/Laksh/Documents/lancedb/python/python/lancedb/_lancedb.pyi)
* Updated type stubs for `expr_lit` to include `Any` (allowing for
`Decimal`).
### **Testing**
Added several new advanced test cases in
[python/python/tests/test_expr.py](file:///c:/Users/Laksh/Documents/lancedb/python/python/tests/test_expr.py)
covering:
* **High-precision Decimal preservation**: Verified against 128-bit
boundaries with a "one point off" test case (`1.234567890123456789 <
1.234567890123456790`).
* **Reflexive operator positioning**: Verified successful query
construction with literals on the left.
* **Timezone-aware normalization**: Confirmed stable behavior for
`datetime` objects.
* **Integration Testing**: Confirmed Date32 and Decimal columns return
the correct Python types and values from the engine during `.to_arrow()`
calls.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
# Elastic Streaming Dataloader
## Motivation
Training large models on LanceDB tables today requires loading the
entire dataset
into memory or writing bespoke batching logic. This PR introduces
`StreamingDataset`, a PyTorch `IterableDataset` that streams directly
from a
LanceDB table with two hard guarantees that are difficult to achieve
together:
**elastic determinism** and **resumability**.
## Goals
### Elastic determinism
The dataset partitions the table into a fixed number of *splits*
(controlled by
`num_splits`, `shuffle_seed`, and `epoch`). Samples are yielded by
round-robining
over splits one sample per split per cycle. Because the split structure
is fixed,
the set of samples that makes up each global training step is identical
regardless
of `world_size` or `num_workers`. You can scale your cluster up or down
between
runs and the model sees the same data in the same order — no
re-sharding, no
gradient variance from topology changes.
### Resumability
`state_dict()` / `load_state_dict()` capture how many samples each split
has
consumed. Because all splits are the same size and the round-robin
design keeps
them in lockstep, the state reduces to a single scalar
(`samples_consumed_per_split`)
that is topology-independent. A checkpoint saved with 8 GPUs can resume
correctly
on 4 GPUs or 16 GPUs without any adjustment.
### PyTorch `IterableDataset` / streaming
`StreamingDataset` implements the standard PyTorch `IterableDataset`
interface, so
it drops into any existing `DataLoader` pipeline without modification.
Data is
fetched lazily from Lance in chunks — only the rows needed for the
current batch are
ever in memory.
Compared to the map dataset this takes more work from pytorch and puts
it into the dataset itself (e.g. shuffling, filtering, etc.). We do this
because we cannot achieve things like elastic determinism or
prefiltering otherwise.
### Multi-worker support
DataLoader workers are automatically assigned contiguous sub-blocks of
splits (the
rank's splits are divided evenly across workers). Each worker is
independent:
no shared state, no inter-process coordination. The only constraint is
that
`num_splits` must be divisible by `world_size * num_workers`.
That being said, multi-worker is highly discouraged as it relies on
multiprocessing which is inefficient. Still, we want to support it.
### Filters as prefilters
Filters are applied at *permutation-build time* via
`PermutationBuilder.filter()`,
not re-evaluated on every fetch. The filtered row IDs are stored in the
permutation
table so that subsequent reads see only the matching rows. This allows
us to avoid loading rows that don't match the filter (which is the
default pytorch behavior)
### Prefetching
Two parameters control the I/O pipeline:
- `read_batch_size` (default 64) — number of rows fetched per
`take_offsets` call.
Larger values amortise per-request overhead, which is critical on object
storage
where a single round-trip can cost ~100 ms.
- `prefetch_batches` (default 4) — number of batches prefetched in
parallel per
split via a `ThreadPoolExecutor`. While the model processes the current
batch,
the next several batches are already in flight, hiding storage latency
behind
compute.
If set correctly then you can get good performance even with
num_workers=0 (unless you are bottlenecked on transform).
### Transform parallelism
The underlying `Permutation` API supports a `with_transform()` callback
for
decoding, augmentation, and format conversion. Unfortunately, this is
not parallelized. Pytorch typically parallelizes this with num_workers
which is multiprocessing which is highly inefficient. For simple
transforms we should be able to utilize multithreading and Rust based
UDFs. For complex python UDFs we could have a dedicated multiprocessing
pipeline for just the transform. Or we could just utilize
multithreading. In both cases we would exclude the I/O stage from the
multiprocessing because that ends up being very memory hungry and
inefficient.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- When an embedding function returns an empty list (e.g. `[]`) for an
input row — as can happen when a model produces no output for a blank
string — `_append_vector_columns` crashed with `ArrowInvalid: Length of
item not correct: expected N but got array of size 0` because PyArrow
cannot fit a zero-length value into a fixed-size list element.
- The fix adds a validation step in `gen()`, inside
`_append_vector_columns`, that replaces any vector whose length does not
match the expected `ndims` (including empty lists and `None`) with
`None` before `pa.array()` is called.
- `None` is a valid null in a PyArrow fixed-size list array, so the bad
entry flows into `_handle_bad_vectors` and is handled according to the
caller-supplied `on_bad_vectors` policy (`error` / `drop` / `fill` /
`null`) instead of causing an unconditional crash.
## Test plan
- [ ] Added `test_embedding_with_empty_output_vectors` in
`python/python/tests/test_embeddings.py` that uses an embedding function
returning `[]` for empty-string inputs, calls `table.add(...,
on_bad_vectors="drop")`, and asserts no crash and that bad rows are
correctly dropped.
- [ ] Existing `test_embedding_with_bad_results` continues to pass (NaN
vectors still handled correctly).
- [ ] Verified manually that `pa.array([[1.,2.,3.,4.], []],
type=pa.list_(pa.float32(), 4))` raises `ArrowInvalid` without the fix,
and succeeds with `None` in place of `[]`.
Fixes#1672
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## What
`MRRReranker.rerank_multivector` averages each document's reciprocal
ranks over the wrong denominator. It divides by the number of rankings
the document *happens to appear in*, instead of the total number of
rankings being fused.
```python
# python/python/lancedb/rerankers/mrr.py
for result_id, reciprocal_ranks in mrr_score_map.items():
mean_rr = np.mean(reciprocal_ranks) # divides by len(present systems)
```
`mrr_score_map[doc]` only accumulates a reciprocal rank for the systems
in which the document was returned, so `np.mean` never accounts for the
systems that missed it.
## Why it's wrong
Mean Reciprocal Rank fusion treats a system that didn't return a
document as a reciprocal rank of `0` and averages across **all**
systems. That's the exact mechanism by which it rewards cross-system
consensus. Dividing by the appearance count removes that, so a document
liked by a single ranking can beat one ranked highly by every ranking.
Concretely, fusing 3 vector rankings:
| Doc | Ranks | Current score | Correct score |
|-----|-------|---------------|---------------|
| A | #1 in 1 system only | `mean([1.0]) = 1.000` | `1.0 / 3 = 0.333` |
| B | #1, #1, #2 across all 3 | `mean([1, 1, .5]) = 0.833` | `2.5 / 3 =
0.833` |
The current code ranks **A above B** - a document two of three rankings
ignored outranks one all three ranked at or near the top.
This also makes `rerank_multivector` inconsistent with `rerank_hybrid`
in the same file, which already treats a missing system as `0`
(`vector_rr = 0.0` / `fts_rr = 0.0`), and with the class docstring
("average of reciprocal ranks across different search results").
## Fix
Divide the summed reciprocal ranks by the total number of rankings:
```python
num_systems = len(vector_results)
...
mean_rr = float(np.sum(reciprocal_ranks)) / num_systems
```
## Tests
Adds `test_mrr_multivector_rewards_consensus`, which asserts the exact
MRR scores and that the consensus document ranks first. It fails on
`main` and passes with this change. Existing reranker tests are
unaffected.
Wyatt Alt
Jack Ye
LanceDB Robot
Lance Release
Prashanth Rao
kid
Mateusz Szewczyk
Mark McDonald
Drew Gallardo
Will Jones
Xuanwo
Pranav Achar
Unmilan Mukherjee
ForwardXu
Dan Rammer
Omkar Kabde
LAKSH JAIN
Weston Pace
Eric B
Armaan Sandhu
Neo-X7