BaseTable::create_index and IndexBuilder::execute now return
Option<String> -- the job id the server mints when an index build is
deferred to a background job (pending vector index on a remote table).
Native builds are synchronous and return None; older servers with empty
response bodies parse as None. The pyo3 binding forwards the id to
python; nodejs keeps its unit return.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JobHandle becomes Job (AsyncJobHandle -> AsyncJob), per the reference
naming convention (Job the reference vs JobInfo the snapshot), and the
implementation moves off the legacy inflight-listing poll onto the
platform jobs API:
- The reference holds the submission (manifest) id and lazily resolves
the platform job id (one /v1/jobs/list call with the manifest-id
filter), tolerating async dispatch with a pending grace window.
- status/progress/wait read /v1/jobs/describe: terminal states are
first-class (DONE / FAILED / CANCELLED) instead of inferred from
leaving the inflight listing, progress comes from the owner-written
payload, and a failed job raises JobFailedError with the server
error. A job that never registers raises instead of hanging.
- cancel() drives /v1/jobs/cancel (with a short resolve retry), which
the server now propagates to running workers.
Connection surfaces gain the three passthroughs (sync + async); every
Job-returning API (refresh_column, MV refresh/wait, load_columns)
hands out the new type. job_history()/errors() stay on their existing
routes (the per-row error store is outside the platform jobs API).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Expose describe_platform_job / resolve_platform_job_id /
cancel_platform_job on the connection, with PlatformJobDescription
carrying the registry lifecycle state and the owner-written status
payload as a JSON string.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wait() (sync + async) only stopped on finished/stale/committed, so a job the
server already reported as state=failed was polled until the (default 3600s)
timeout, then raised a misleading TimeoutError instead of the real cause. A
doomed backfill -- e.g. a multi-column REFRESH COLUMN of a scalar UDF -- hung
the client even though get_job surfaced the failure within ~3s.
Add a terminal failed branch that raises JobFailedError carrying the server
error, exported from the package. Verified end-to-end against the cluster:
raises in 3.6s instead of hanging. Unit-tested with a mock conn (sync+async,
failure + success + committed paths).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The client exposed list_jobs/get_job/cancel_job but not the durable job
history or the per-row UDF errors, so those SQL/REST surfaces had no SDK
equivalent. Add job_history(job_id=None) and errors(job_id=None, table=None)
through every layer:
- Database trait + Connection API (JobHistoryInfo, JobErrorInfo types).
- Remote REST impl: GET /v1/job/history (?job=) and GET /v1/job/errors
(?job=&table=), with serde response types + From mappings.
- pyo3 bindings + pyclasses JobHistoryEntry / JobErrorEntry, registered.
- Python sync + async db.py wrappers.
Mirrors the existing list_jobs plumbing exactly. Remote-handler test asserts
the GET paths, query filters, and response parsing for both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
JobHandle/AsyncJobHandle now poll conn.get_job(id, table) -- one job -- instead
of list_jobs() + client-side filter over every active job. The job's table is
threaded in from refresh_column / MV refresh as an O(1) lookup hint. Plumbs
get_job through the Database trait (default not_supported), RemoteDatabase
(GET /v1/job/{id}?table=...), the Connection wrapper, and the pyo3 binding +
db.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
self._conn on a remote sync connection is an AsyncConnection (python), which
exposes `lineage` (parses the JSON), not the pyo3 `table_lineage`. The sync
wrapper was calling self._conn.table_lineage -> AttributeError. Drive
self._conn.lineage on the loop instead, mirroring create_materialized_view.
(table_lineage stays the pyo3 method the async path calls via self._inner.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
refresh_column returned the bare job-id str, so callers had to wrap it:
db.job(tbl.refresh_column("c")).wait(). Mirror MaterializedView.refresh() and
return a JobHandle directly, so tbl.refresh_column("c").wait() / .status() / .id
work without the wrapper. (db.job(job_id) stays for reconnecting by a stored id.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- new lineage.py: Lineage / Node / Edge / FunctionRef dataclasses that parse the
server's lineage JSON, with to_dict(), to_graphviz() (drift edges dashed+red),
and _repr_html_(); plus .functions() / .stale() helpers.
- Connection.lineage(table, column=, direction=, depth=) (sync + async) calls the
pyo3 table_lineage binding and deserializes into Lineage.
- Table.lineage(column=, ...) via the table's job connection; MaterializedView /
AsyncMaterializedView .lineage() delegate to the backing table (the server
already includes the view's sources + downstream dependents).
- export the new types.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A new Database::table_lineage(TableLineageRequest) -> Result<String> threaded
end to end: default not_supported in the trait; the remote impl issues
GET /v1/table/{name}/lineage with column/direction/depth query params and
returns the body verbatim; connection.rs exposes a pub wrapper; the pyo3
binding hands the JSON string to Python.
The lineage payload is carried as opaque JSON on purpose: the open-source
lancedb client must not depend on the sophon-internal derived_jobs crate that
defines the lineage schema, so the wire format is the contract and the Python
layer deserializes it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sync _refresh_materialized_view called self._conn.refresh_materialized_view
(no underscore); the async method is _refresh_materialized_view, so
MaterializedView.refresh() raised AttributeError. Add the underscore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sync RemoteDBConnection.create_materialized_view assembled the SELECT but
called the async create_materialized_view with the query as the 2nd positional
arg, which binds to `source=` (query= is keyword-only). Every call then failed
the "needs either query= or both source and select" validation. Pass query=query.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refresh is a submit-a-job verb, so its only public surface should be
MaterializedView.refresh() / AsyncMaterializedView.refresh() (which return a
job handle). Rename the connection methods to _refresh_materialized_view and
have the handles call that, so the raw by-name refresh is no longer advertised
on the connection. The pyo3 native binding is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- create_materialized_view now takes either query= or source+select (folds in
the old create_view builder) and returns a MaterializedView handle whose
.wait() blocks on initial population. create_view is removed -- it was
misnamed (it built a *materialized* view, while CREATE VIEW means the plain
non-materialized view the engine also supports).
- MaterializedView.refresh() and the remote Table.refresh_column() now return a
JobHandle directly, so tbl.refresh_column("c").wait() needs no db.job(...)
wrapper. db.job(id) is narrowed to reconnect-by-id (stored id / SQL / REST).
- rename View/AsyncView -> MaterializedView/AsyncMaterializedView (+ exports).
- tighten the replace path: only a not-found error on the pre-drop is benign;
real failures (perms/server) now surface instead of being swallowed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Geneva Table.load_columns() parity on the REST-only client. Fills existing
columns from an external Parquet/Lance/IPC source by primary-key join.
- BaseTable::load_columns default (NotSupported) + public Table::load_columns,
taking a LoadColumnsRequest (source uris/format/storage_options, target/source
key, (target, source?) column mappings, on_missing, worker/batch/commit knobs).
- Remote impl POSTs to /v1/table/{id}/load_columns with the matching body;
mock test asserts the request shape.
- PyO3 binding + Python remote Table.load_columns(source, pk, columns, *,
source_format, source_pk, on_missing, ...) accepting a column list or
{target: source} dict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
View.refresh(full=True) (sync + async) now works -- it previously raised
NotImplementedError. Thread the flag through the client: RefreshMaterialized-
ViewRequest.full -> the REST body (RemoteRefreshMaterializedViewRequest.full);
pyo3 refresh_materialized_view(full=...); Connection.refresh_materialized_view(
name, full=) sync + async. A full refresh forces a recompute-and-replace and
preserves the view's indexes (reindexed by the distributed indexer).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add View.create_index / create_scalar_index / create_fts_index / search
as pass-throughs to open_table(name). A materialized view is a real Lance
dataset; these let it be indexed and searched like any other table,
closing the parity gap with Geneva (whose create_materialized_view returns
a first-class Table).
The server-side create_index handler records indexes declared on a view so
they survive a full refresh (which overwrites the dataset, dropping its
indices); that re-apply is wired in the sophon engine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thread priority (Kueue tier) through refresh_column at every layer (Python sync+async
+ RemoteTable -> pyo3 -> Rust client trait/public/remote -> REST body), mirroring
num_workers/batch_size. The function keeps its priority as a default; the per-refresh
value overrides. Also adds the previously-missed batch_size to RemoteTable.refresh_column
(the REST sync path). cargo check (lancedb --features remote --tests, lancedb-python) +
ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
batch_size / num_workers / max_workers are invocation concerns (how to schedule THIS
refresh), so expose batch_size on refresh_column through every layer (Python sync+async
-> pyo3 -> Rust client -> the REST RefreshColumnRequest.batch_size, which the handler
already forwards into the backfill). num_workers/max_workers were already invocation-
placed; batch_size was the gap. The function may still carry a default; the refresh
override wins (extends the batch_size_override model). Both crates cargo-check clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A computed column is an expression over a registered function applied to input
columns, not a UDF coupled to a column. fn("data") already returned the expression
string "fn(data)"; make it a ColumnExpr (a str subclass) that also carries the
function's return type, so add_columns(computed={"vec": embed("data")}) declares the
column with no hand-written type. _normalize_computed handles the new form (and tuple
keys for STRUCT fan-out) and keeps the legacy {col: (sql_type, expression)} tuple.
add_computed_column is deprecated (delegates, with a DeprecationWarning). The function
stays decoupled from columns -- register once, apply anywhere.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thread an optional partition_by through the client: CreateMaterializedViewRequest
-> REST body -> pyo3 binding -> Python create_materialized_view/create_view
kwarg (sync + async). The server partitions the view's table function by the
named source column -- by IVF index clusters if the column is indexed
(image-dedup), else by distinct value. Unifies Geneva's partition_by +
partition_by_indexed_column into one knob.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirrors the sync ergonomics on the async surface: AsyncConnection
create_function(udf, replace=)/create_view/job; AsyncTable.add_computed_column;
AsyncView + AsyncJobHandle (await + asyncio.sleep; shared submission-prefix
matcher with the sync JobHandle). Decorator + REST routes are shared/already
validated; this is the async wrapper layer. Exported from the package root.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
db.job(id) gets the submission id the refresh/backfill endpoints return,
but list_jobs / cancel report the agent's manifest id
(<table>-<type>-<first 8 of submission id>). JobHandle now matches that
(exact id or submission prefix) so wait()/progress() truly track, and
cancel() cancels by the resolved canonical id instead of the unusable
submission uuid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings the @udf/@table_udf decorator + type inference into lancedb as
lancedb.udf (Apache-2.0), and adds the ergonomic glue to the existing
connection/table so there's no separate object model:
- create_function() accepts a Udf (and a replace= flag)
- Table.add_computed_column(column, udf)
- create_view(name, source, select, ...) -> View (assembles the SELECT)
- Connection.job(job_id) -> JobHandle
- View / JobHandle are thin references over a connection
Exports udf/table_udf/Udf/JobHandle/View from the package root. The
operations stay the existing remote-only methods (enterprise/cloud); the
decorator works locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Exposes the existing server-side CANCEL JOB (CoordinatorCatalog::cancel_job)
as a REST-backed SDK method: Database trait default NotSupported,
RemoteDatabase POSTs /v1/job/{id}/cancel, pyo3 binding, sync+async python
wrappers. Best-effort: a missing job returns false, not an error. Mock-HTTP
unit test in test_derived_compute_routes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per the interface design: computed columns are parameters on the
existing add_columns operation, not a separate method.
- BaseTable::add_computed_columns((name, sql_type) pairs + a f(args)
expression) -- default NotSupported; RemoteTable posts 'computed'
entries to the existing /v1/table/{id}/add_columns route.
- python add_columns gains computed= on LanceTable, RemoteTable, and
AsyncTable: tbl.add_columns(computed={'doubled': ('FLOAT',
'double_it(val)')}); grouped by expression so struct-returning
functions' columns land adjacently.
Adds the derived-compute interface to the SDK:
- Database trait: create/list/drop_function, create/refresh/alter/
drop/list_materialized_view, list_jobs -- default implementations
return Error::NotSupported (NotImplementedError in python), so
existing Database impls are unaffected; local single-node
implementations are planned. BaseTable gains refresh_column with
the same default.
- RemoteDatabase/RemoteTable implement them against the server REST
routes (/v1/function/*, /v1/materialized_view/*, /v1/job/list,
/v1/table/{id}/refresh_column), with mock-HTTP unit tests.
- Connection/Table public methods, pyo3 bindings (FunctionInfo,
MaterializedViewInfo, JobInfo pyclasses), and python wrappers:
sync on the DBConnection base (shared by local and remote
connections), async on AsyncConnection; refresh_column on
LanceTable, RemoteTable, and AsyncTable.
BREAKING CHANGE: splits generated by the permutation data loader will
not be the same, due to a change in hash function.
Updates the Lance dependencies and Java lance-core to
[v9.0.0-rc.1](https://github.com/lance-format/lance/releases/tag/v9.0.0-rc.1).
Includes the required DataFusion 54 and Lance file-reader compatibility
updates.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds client-side support for analyze_plan distributed metrics modes
across Rust, Python, and TypeScript clients. Defaults to aggregate for
backward compatibility and sends the remote distributed_metrics
parameter only when a non-default mode is requested.
## Summary
Fix `on_bad_vectors="fill"` so it replaces only invalid or missing
vector values instead of replacing the entire vector row.
Fixes#3026.
## Reasoning
The old Python sanitizer detected whether a vector row was bad at row
granularity. For `fill`, it then used that row-level flag to replace the
whole vector with `[fill_value] * dim`. That meant an input like `[1.0,
NaN, 3.0]` became `[0.0, 0.0, 0.0]`, even though the documented and more
useful behavior is to preserve valid values and fill only the bad
element.
I checked whether this should be a Rust-side fix so TypeScript users
would benefit too. Today, Rust core exposes `NaNVectorBehavior::{Error,
Keep}` for rejecting or keeping NaN vectors, while the Python
`on_bad_vectors` API (`error`, `drop`, `fill`, `null`) is implemented in
the Python ingestion sanitizer before data reaches Rust. TypeScript does
not expose the Python `on_bad_vectors="fill"` behavior today. Moving
this exact behavior to Rust would be a broader cross-language API
change, so this PR keeps the fix scoped to the currently affected Python
API.
## What changed
- Added a small helper that fills bad vector rows by preserving valid
elements, replacing NaN elements with `fill_value`, truncating vectors
longer than the expected dimension, and padding short vectors with
`fill_value`.
- Kept the existing fast path unchanged: the helper only runs after bad
vectors are detected and `on_bad_vectors="fill"` is selected.
- Updated sanitizer and table tests to assert element-wise NaN
replacement and short-vector padding for both `create_table` and `add`.
## Validation
- `uv run ruff format .`
- `uv run ruff check .`
- `cd python && uv run --no-sync pytest
python/tests/test_util.py::test_handle_bad_vectors_jagged
python/tests/test_util.py::test_handle_bad_vectors_nan
python/tests/test_table.py::test_create_with_nans
python/tests/test_table.py::test_add_with_nans -vv`
Targeted pytest result: `10 passed`.
## Why this fix is Python-side (and not Rust)
The problematic behavior lives in Python’s `on_bad_vectors` sanitizer,
before data is handed off to Rust. Rust currently only exposes
`NaNVectorBehavior::{Error, Keep}` for add operations, while Python has
the richer `on_bad_vectors={"error","drop","fill","null"}` API.
TypeScript does not currently expose the Python-style fill behavior, so
moving this exact fix into Rust would require designing a broader
cross-language bad-vector handling API.
This PR keeps the change scoped to the existing affected surface:
Python’s `on_bad_vectors="fill"` path. This way, Python users
immediately benefit.
## Summary
- add table-level FTS query tokenization returning token text and
position
- use the native index tokenizer for local tables and remote index
metadata for remote tables
- expose sync and async Python table wrappers with focused coverage
## Summary
- serialize sync phrase queries consistently for execution and query
plans
- restore the documented no-argument hybrid `phrase_query()` behavior
- keep reranker input as the original user text without mutating the
builder
Fixes#3653.
## Testing
- `python/.venv/bin/python -m pytest <8 focused test nodes> -q` (`8
passed`)
- `python/.venv/bin/python -m ruff format --check
python/python/lancedb/query.py python/python/tests/test_fts.py
python/python/tests/test_hybrid_query.py`
- `python/.venv/bin/python -m ruff check .`
- `git diff --check origin/main...HEAD`
The complete hybrid module and the real native FTS phrase test were not
completed
in the current PyO3 runtime environment: both stalled in the native
`lancedb.connect()` fixture and were interrupted without an assertion
failure.
Carrying over from #2915, this patch introduces:
* Single-API call batching support for Gemini embeddings (up to 100 at a
time, the API limit)
* A versioned user agent header for Gemini API calls
* Support for [variable embedding dimension
size](https://ai.google.dev/gemini-api/docs/embeddings#control-embedding-size)
(Gemini is MRL trained)
## Summary
- preserve explicit `0.0` distance bounds in synchronous hybrid search
- distinguish omitted `None` endpoints from zero-valued endpoints when
configuring the vector child query
- add a public end-to-end regression test for a zero upper bound
## Testing
- `cd python && uv run --extra tests pytest
python/tests/test_hybrid_query.py -q`
- `uv run --project python ruff format --check
python/python/lancedb/query.py python/python/tests/test_hybrid_query.py`
- `uv run --project python ruff check .`
Fixes#3651
Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.
## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.
## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.
## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.
Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.
## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps Lance to v9.0.0-beta.19, which includes lance-format/lance#7687
for side-effect-free DirectoryNamespace read paths.
This fixes root-level read-only table opens that previously could
trigger `__manifest` creation through directory namespace construction,
including Hugging Face bucket reads with read-only tokens. A LanceDB
regression test now covers root listing operations without creating
`__manifest`.
Fixes#3633.
### Summary
`flatten_columns` raises `ValueError` when called with `flatten=False`,
even though `False` should mean "do not flatten". This is reachable from
the public API — `Query.to_pandas(flatten=...)` and
`to_batches(flatten=...)` type their `flatten` param as
`Optional[Union[int, bool]]` and pass it straight to `flatten_columns`.
### Cause
`bool` is a subclass of `int`, so `isinstance(False, int)` is `True`.
`flatten=False` skips the `flatten is True` check, falls into the
integer branch, and `False <= 0` evaluates to `True`, raising:
```
ValueError: Please specify a positive integer for flatten or the boolean value `True`
```
### Reproduction
```python
import lancedb
db = lancedb.connect("/tmp/db")
t = db.create_table("t", data=[{"id": 1, "vector": [0.1, 0.2]}])
t.search([0.1, 0.2]).to_pandas(flatten=False) # -> ValueError
```
### Fix
Guard the integer branch with `not isinstance(flatten, bool)` so that
`flatten=False` (and `None`) mean "do not flatten". Behavior is
otherwise unchanged:
- `flatten=True` → flatten all nested levels
- positive `int` → flatten to that depth
- non-positive `int` (e.g. `0`) → still rejected with `ValueError`
Added a regression test in `tests/test_util.py` covering `None`,
`False`, `True`, a positive depth, and `0`.
This PR fixes a serialization error when using Ollama embeddings in
`create_table`.
The use of `@cached_property` for the Ollama client was causing issues
during serialization/pickling, which is required by certain LanceDB
operations (like when using multiprocessing or certain storage
backends). Switching to a standard `@property` ensures the client is
instantiated when needed without being stored in a way that breaks
serialization.
Verified with the following script:
```python
import lancedb
from lancedb.embeddings import get_registry
import pickle
registry = get_registry().get(\"ollama\")
model = registry(name=\"llama3\")
# This would fail before the fix
pickled = pickle.dumps(model)
unpickled = pickle.loads(pickled)
```
Fixes#2629 (or similar serialization issues reported).
---------
Co-authored-by: Unmilan Mukherjee <Missing-Identity@users.noreply.github.com>
## Summary
Closes#3525
This PR wires up two new optional object-store backends at the LanceDB
layer, exposing capabilities that already exist upstream in `lance` /
`lance-io`:
| Backend | Cargo feature | Default in Rust crate | Default in Python
wheel | Default in Node binding |
| --- | --- | --- | --- | --- |
| **Tencent COS** | `cos` | ❌ off | ✅ on | ❌ off |
| **GooseFS** | `goosefs` | ❌ off | ✅ on | ✅ on |
Both backends are additive and do not affect existing users who don't
opt in.
## Motivation
- **Tencent COS** is the dominant object storage in the China region.
Tencent Cloud users currently need an S3-compatible proxy or a private
fork to use LanceDB against COS buckets.
- **GooseFS** is Tencent Cloud's distributed cache acceleration layer
that sits in front of COS/S3, a common pattern for vector search / AI
training where the same hot dataset is read repeatedly.
- This brings COS / GooseFS to feature parity with the existing
first-class backends (`aws`, `gcs`, `azure`, `oss`, `huggingface`).
See the linked issue #3525 for the full discussion.
## Changes
### `rust/lancedb/Cargo.toml`
Add two new optional features that pull through the corresponding
upstream feature flags:
```toml
cos = ["lance/tencent", "lance-io/tencent"]
goosefs = [
"lance/goosefs",
"lance-io/goosefs",
"lance-namespace-impls/dir-goosefs",
]
```
### `python/Cargo.toml`
Enable both `cos` and `goosefs` by default for the Python wheels, so
`pip install lancedb` works against COS / GooseFS out of the box
(consistent with how `aws` / `gcs` / `azure` / `oss` are bundled today):
```diff
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/cos", "lancedb/goosefs"]
```
### `nodejs/Cargo.toml`
Enable `goosefs` by default for the Node binding (COS kept opt-in to
limit the default native binary size; can be revisited based on demand):
```diff
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/goosefs"]
```
### `Cargo.lock`
Regenerated to reflect the transitive dependencies brought in by the new
upstream features. No manual edits.
## Example Usage
### Rust
```toml
# Cargo.toml
lancedb = { version = "0.30", features = ["cos", "goosefs"] }
```
```rust
// Tencent COS
let db = lancedb::connect("cos://my-bucket/my-db").execute().await?;
// GooseFS
let db = lancedb::connect("goosefs://my-namespace/my-db").execute().await?;
```
### Python
```python
import lancedb
db = lancedb.connect(
"cos://my-bucket/my-db",
storage_options={
"secret_id": "...",
"secret_key": "...",
"region": "ap-guangzhou",
},
)
```
## Backwards Compatibility
- All new features are **opt-in** at the Rust crate level (`default =
[]` for `lancedb` itself is unchanged).
- The Python wheel gains both backends by default, increasing wheel size
slightly but matching the existing pattern of bundling all major cloud
backends.
- Node binding only adds `goosefs` to defaults; existing users see no
behavior change.
## Testing
- `cargo check --all-features` ✅
- `cargo check -p lancedb --features cos` ✅
- `cargo check -p lancedb --features goosefs` ✅
- End-to-end COS / GooseFS smoke tests require Tencent Cloud credentials
and are intentionally not added to CI in this PR (same approach used for
`s3-test`). Happy to add a gated test feature in a follow-up if
reviewers prefer.
## Checklist
- [x] Added `cos` and `goosefs` features to `rust/lancedb/Cargo.toml`
- [x] Updated `python/Cargo.toml` default features
- [x] Updated `nodejs/Cargo.toml` default features
- [x] Regenerated `Cargo.lock`
- [x] Verified build with `--all-features`
- [ ] Documentation update (can be done in a follow-up PR once API
stabilizes)
## Related
- Issue: #3525
- Upstream support:
[`lance/tencent`](https://github.com/lance-format/lance),
[`lance/goosefs`](https://github.com/lance-format/lance)
## Summary
Adds `Table::get_lsm_write_spec` returning `Option<LsmWriteSpec>` — the
read counterpart to the existing `set_lsm_write_spec` /
`unset_lsm_write_spec`. Returns `None` when the MemWAL LSM write path is
not enabled; otherwise reconstructs the spec (mode, shard column,
`num_buckets`, `maintained_indexes`, `writer_config_defaults`) exactly
as installed.
## Changes
- **Rust core (`NativeTable`)** — reconstructs the spec from
`mem_wal_index_details()`, resolving the shard column from its Lance
field id via the dataset schema. This is a raw metadata read, so it is
unaffected by `describe_indices` system-index filtering.
- **Remote (`RemoteTable`)** — reads the `__lance_mem_wal` system index
through `index/list` with `include_system: true` (so the curated
`list_indices` surface stays unchanged), then parses the index `details`
JSON. It matches the index by name and ignores `index_type`, so no
client `IndexType` variant is needed. It uses the **server-resolved
`column` name** from the details (Lance field ids do not travel to the
remote client).
- **Python + TypeScript bindings** — sync and async, mirroring
`set`/`unset`, with round-trip tests (bucket / identity / unsharded,
plus `None` when unset).
## Tests
- Rust: native round-trip unit test + remote mock-endpoint tests
(present + absent). All green (`cargo test --features remote -p
lancedb`).
- Python/TS: round-trip tests added; binding-runtime execution runs in
CI.
## Dependencies for the remote path
The remote path is complete on the client side but depends on two
out-of-repo pieces to work end-to-end:
1. **lance** — emit the server-resolved shard **`column`** name in the
MemWAL index `details` JSON (field ids can't reach the client). See
lance-format/lance#7667.
2. **server** — honor `include_system` on `index/list` so the
`__lance_mem_wal` entry is returned for this read.
Against an older server (no `include_system`), the remote getter
degrades gracefully to `Ok(None)` rather than erroring.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Closes#3245.
> **BREAKING CHANGE:** `with_format("torch")` no longer returns a list
of stacked row tensors. It now returns per-row dicts so PyTorch's
default `DataLoader` collate stacks them into `{col: tensor(B,)}`.
Switch to `with_format("torch_row")` to keep the old shape.
### What changed
`"torch"` now returns a list of per-row dicts (`[{col: tensor}, ...]`)
at every indexed access path. The default `DataLoader` collate stacks
them into a column-keyed batched dict, no custom `collate_fn` needed.
The old shape is preserved under a new `"torch_row"` literal.
`"torch_col"` is unchanged.
The unbatching lives inside the transform (`batch_to_tensor_dict`), not
`__getitems__`, so the shape survives pickling and works under
`DataLoader(num_workers>0, multiprocessing_context="spawn")`.
### Format comparison
| Format | `iter(batch_size=N)` | `__getitems__([0,1,2])` | `DataLoader`
default collate |
|---|---|---|---|
| `"torch"` (new) | `list[{col: tensor}]` length N | `list[{col:
tensor}]` length 3 | `{col: tensor(B,)}` |
| `"torch_row"` (old `"torch"` behavior) | `list[tensor(n_cols,)]`
length N | `list[tensor(n_cols,)]` length 3 | `tensor(B, n_cols)` |
| `"torch_col"` (unchanged) | `tensor(n_cols, N)` | `tensor(n_cols, 3)`
| needs `collate_fn=lambda x: x` |
Output matches HuggingFace `Dataset.set_format("torch")` on container
shape, keys, and values at every access path. The only divergence:
HuggingFace downcasts `float64` to `torch.float32` by default, LanceDB
preserves dtype. Verified by `scripts/verify_torch_format.py`.
### Migration
```python
# Old default — column names lost, shape was tensor(B, n_cols)
DataLoader(Permutation.identity(table).with_format("torch"))
# New default — column names preserved
DataLoader(Permutation.identity(table).with_format("torch")) # {col: tensor(B,)}
# Keep old behavior
DataLoader(Permutation.identity(table).with_format("torch_row")) # tensor(B, n_cols)
```
Wyatt Alt
Jack Ye
LanceDB Robot
Lance Release
Prashanth Rao
kid
Mateusz Szewczyk
Mark McDonald
Drew Gallardo
Will Jones
Xuanwo
Pranav Achar
Unmilan Mukherjee
ForwardXu
Dan Rammer
Omkar Kabde