RustSec published four `zstd` advisories after #3819 merged, causing the
`deny` job to fail on `main`. `arrow-ipc 58.x` pins `zstd ^0.13`, while
the patched releases require `zstd >= 0.14.3`; moving to Arrow 59+
requires a coordinated DataFusion and Lance upgrade.
This records temporary exceptions with their reachability rationale.
LanceDB uses the `zstd::bulk` API through `arrow-ipc`; the affected
`with_context`, seekable, and streaming-decoder APIs are not used, and
the `zstd-sys` advisory is wasm32-only while wasm32 is not a shipped
target.
Validation: `cargo-deny 0.20.2` reports `advisories ok, bans ok,
licenses ok, sources ok`.
Keep FP16 vectors through IVF-PQ index creation and search in the
synchronous and asynchronous quickstarts. Add native and conditional
CUDA regression coverage using a float64 query against an explicitly
typed FP16 column. This is test and documentation-example coverage for
dtype preservation already present in Lance; it does not change
production code.
Related #2120.
Validation on `bbad20940632b21a9ee85bdcb46b4cc19dbd65e3` with Python
3.12.12, Lance 15.0.0-beta.1, PyArrow 24.0.0, and Torch 2.14.1:
- The native FP16 parameter, Arrow FP16 ingestion test, and both
quickstarts passed: 4 passed, 1 CUDA-only skip.
- The two affected test files passed: 386 passed, 1 CUDA-only skip (`-m
"not slow and not s3_test"`).
- Ruff 0.9.9 lint and formatting checks passed for the Python package.
Real CUDA execution has not been performed. Torch is installed, but
`torch.cuda.is_available()` is false; `test_create_f16_table[cuda]`
skips with `CUDA not available`. The native CPU result does not
establish that the original GPU failure is resolved. Issue #2120 remains
open for verification on an actual CUDA device.
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#4399.
Remote connections currently discard local-only settings, including
table write options that determine storage format. The synchronous
Python client also binds the fourth positional `create_table` argument
to `on_bad_vectors`, so an intended overwrite is sent as a create.
This change aligns the remote Python signature with
`DBConnection.create_table`, preserves the legacy remote timeout
keywords, and raises `ValueError` for other unknown connection keywords.
The shared Rust client rejects unsupported connection, create, and open
options before sending HTTP requests; open validation also runs before a
table-cache hit. Python reports these as `NotImplementedError`,
including explicit `enable_v2_manifest_paths=False`. Supported
Enterprise connection-level storage options remain available.
This implements option 2 from the issue: explicit rejection. It
deliberately does not add unverified server write parameters. Calls that
previously supplied ignored options must omit them to use server-managed
storage and caching. Database URI prefixes are already preserved on
current main and are unchanged here.
Regression tests cover Python sync/async, TypeScript create/open calls,
and the Rust remote client, including empty-table creation, overwrite
mode, and cached table handles. Python API documentation describes the
behavior and corrected positional order.
Reconciled with main at `0ea932da`, preserving upstream blob and
error-handling tests and updating the Python test that previously
expected open options to be ignored.
Validation on the merged source:
- Rebuilt the Python native extension; 304 tests passed across remote
options, database, remote database, and session suites.
- Rust remote database suite: 135 passed. Rebuilt the Node native
extension; TypeScript remote suite: 62 passed. TypeScript compilation
and documentation generation passed.
- Full Python SDK lint and formatting pass (159 files); full TypeScript
lint and formatting pass (66 files). Rust formatting and `git diff
--check` pass.
- Repository-wide Python lint/format reports existing issues in
unchanged `docs/web-tests` files (49 lint findings and 11 formatting
files).
Environment: Ubuntu under WSL2, Linux x86-64, CPython 3.12.13, Rust
1.97.0, and Node.js 22.16.0. Locked dependencies and the repository `ci`
build profile. Remote tests use HTTP fixtures; this does not claim
validation against the original ARM64 single-node deployment.
Documentation review follow-up: added a draft Enterprise migration note
in `docs/web/basics/sdk.mdx`. It explains that `db://` connections
reject local session/cache and per-table storage/write options, while
supported connection-level Enterprise storage options remain available.
Following the repository's release conventions, the note stays in an MDX
comment below the migration list until the behavior is released. Vale
3.24.0 with the pinned LanceDB style package reports no findings; `git
diff --check` passes. This follow-up changes documentation only.
BREAKING CHANGE: Remote connections now reject unsupported local
session/cache and per-table storage/write options instead of silently
ignoring them. Callers must omit these options and use server-managed
storage and caching; supported Enterprise connection-level storage
options remain available.
Integration with current main preserves its synchronous API signatures,
including the remote `create_table(mode=None)` default, and the table
compatibility defaults. The positional-order repair is already on main
via #4608; this PR retains the distinct unsupported-option rejection
behavior.
CI follow-up: the inherited IVF search-effort test occasionally measured
recall 0.375 against its unchanged 0.5 threshold after random index
training. A matching local reproducer observed this once in 30 builds.
The fixture now places the eight exact neighbors together, preserving
the 16-partition setup, two writes, all probe/precedence assertions, and
the recall threshold. The adjusted fixture returned recall 1.0 in 100
builds; the full query and remote suites passed (102 tests). This
stabilizes test data and does not change search behavior or promise
production recall.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#4400.
Sync remote connections fail on `uri`, `len(db)`, and `name in db`, and
remote tables lack `namespace`, `id`, and `replace_field_metadata`.
Share the paginated connection helpers and table identifier contract
through the base classes, and delegate remote metadata replacement to
the existing async implementation. Unsupported remote session and
read-consistency properties raise explicit `NotImplementedError`s.
Reconciled with main at `3dd57978`, including the synchronous method
signatures from #4498. The local session fix has already landed in
#4433, so this PR preserves its supplied-session-or-None contract and
adds no Rust/PyO3 session access. Namespace connections retain their
writable properties.
Existing custom `Table` subclasses remain instantiable: `namespace`
defaults to a fresh empty list, and deprecated `replace_field_metadata`
delegates to `update_field_metadata` with `replace=True` while retaining
its `None` return. Native and remote overrides are preserved. Regression
tests instantiate an old-style subclass through normal ABC class
creation and verify identifiers, independent namespace lists,
deprecation, and exact metadata delegation.
Historical validation at `03c43f7b` before the latest main integration
and subclass compatibility repair, Ubuntu x86-64, Python 3.12.13:
- Rebuilt the native extension with the repository `ci` profile and
locked Python dependencies.
- 273 tests passed across `test_sync_api_parity.py`, `test_db.py`,
`test_remote_db.py`, and `test_session.py`.
- 84 namespace and catalog compatibility tests passed (357 tests total).
- Python SDK lint and formatting pass (158 files); changed-file checks
and `git diff --check` pass.
- Repository-wide lint/format checks report existing issues in unchanged
`docs/web-tests` files (49 lint findings and 11 formatting files). These
are outside this patch.
Remote tests use the existing HTTP fixtures through the rebuilt native
client; the original aarch64 server deployment was not recreated.
Validation after merging main `3dd57978` and the compatibility repair
(macOS ARM64, uv-managed Python 3.12.12, rebuilt editable native
extension):
- 60 focused parity, signature, and error-contract tests passed.
- 381 database, remote, session, namespace, catalog, and sync parity
tests passed; 2 platform-dependent tests skipped.
- The two new subclass regressions fail against the pre-repair
implementation with the expected abstract-member instantiation error,
then pass with the repair.
- Python SDK Ruff lint/format passes (167 files); repository-wide
findings remain confined to unchanged docs/web-tests (49 lint findings,
11 formatting files).
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Closes#4396.
Calls written against `Table` can fail or bind different arguments when
used with `RemoteTable`. Align the affected sync signatures, positional
order, and defaults; preserve renamed remote keywords as deprecated
aliases. Share legacy vector configuration and tokenizer inference,
forward index tuning and timeout options, and accept `fast_search`
consistently.
Preserve current upstream conversion behavior: local
`to_polars(batch_size=...)` returns a LazyFrame; remote tables accept
the same argument signature but raise the documented
unsupported-operation error and direct callers to
`table.search().to_polars()`. The abstract `Table` method retains its
unsupported-operation error, with the implementation confined to
`LanceTable`.
Regression coverage checks argument binding, deprecated aliases, option
forwarding, native behavior through both wrappers, HTTP index requests,
and remote conversion rejection. The merge preserves current tokenizer
defaults, remote query defaults, HNSW-PQ support, and
unsupported-operation handling.
Runtime validation at `c4fb01f2` (Linux x86_64, uv-managed Python
3.12.15):
- Focused Ruff lint and formatting passed for all four changed Python
files.
- Rebuilt and installed the current native extension successfully with
the repository's `ci` Cargo profile.
- Seven regression suites passed: `test_errors.py`,
`test_table_signatures.py`, `test_remote_db.py`, `test_table.py`,
`test_index.py`, `test_fts.py`, and `test_query.py`: **874 passed, 1
deselected**, with slow and S3 tests excluded.
- CI caught an abstract-base conversion regression in the initial merge.
The existing error-contract test reproduced it locally (1 failed, 7
passed); restoring the implementation to `LanceTable` fixes it while
preserving signature parity.
- Full-root Ruff reports 49 lint errors and 11 files needing formatting,
all in unchanged upstream `docs/web-tests` files. The four changed
Python files pass both checks.
- Validated against upstream `197403e8`, using Lance 15.0.0-beta.1 and
LanceDB 0.41.0-beta.14 bindings.
Historical validation before this merge: 618 tests passed and one was
deselected. That result does not establish correctness of the current
merge. The original client-side reproduction produced nine
argument-binding errors without contacting a server; the initial
regression suite failed 32 cases before the fix and passed 39 after it.
Documentation review follow-up: restored the `batch_size` parameter
description on the abstract `Table.to_polars` method so it appears in
the public API reference. This adds six docstring lines and changes no
runtime behavior. After this edit, all 47 focused error-contract and
signature tests pass; changed-file Ruff lint/format and `git diff
--check` pass. Repository-wide Ruff still reports the same 49 findings
and 11 formatting files in unchanged `docs/web-tests`. The full
documentation site and seven-suite runtime run were not repeated for
this documentation-only follow-up.
The reported ARM64 production-server deployment has not been recreated.
HTTP integration tests use the repository's local mock server.
BREAKING CHANGE: Remote synchronous table methods now use the positional
argument order and defaults documented by `Table`. Migrate calls using
the former remote-only positional order to named arguments, particularly
`create_index` and `search`; pass `index_type` by keyword to
`create_scalar_index`. Remote scalar and vector index creation now
default to `replace=True`; specify `replace=False` to retain rejection
of an existing index. Renamed remote keywords remain supported as
deprecated aliases, as documented above.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
`IvfRq()` forced `num_bits=1` through Python's Rust binding, overriding
Lance's five-bit default. The shared native index creation path also
substituted one bit when the Rust or Node builder left the value unset.
Make Python's bit count optional and forward only explicit values. Use
`RQBuildParams::default()` for an unset native bit count, and omit unset
values when dispatching to pylance for accelerator builds. Update the
Python, Rust, Node, and vector-index documentation.
New indexes with an unset bit count now inherit Lance's default,
currently five bits, across Python, Rust, and Node. Five-bit vector
codes use roughly five times the storage of one-bit codes, plus other
index overhead. This also changes reader compatibility: indexes created
with `num_bits >= 2` use a newer on-disk layout that some older LanceDB
versions cannot read. Upgrade readers to versions that support multi-bit
IVF_RQ, or explicitly set `num_bits=1` (for example, `IvfRq(num_bits=1)`
in Python) when sharing indexes with readers limited to one-bit IVF_RQ.
Validation:
- Reproduced six regression failures before the fix: one-bit default
metadata, rejected `None`, and incorrect remote/accelerator forwarding.
- After merging `main` through `b762ec22`, rebuilt both native bindings
against Lance 15.0.0-beta.1. Python validation passed with 106 tests and
15 optional Ollama skips. Coverage includes the full index suite,
omitted and explicit bit counts, reading Lance-created IVF_RQ indexes,
accelerator forwarding, CUDA-unavailability handling, timeout
validation, remote requests, and table/namespace listing behavior.
- Rust formatting and Python package Ruff lint/format checks pass. Node
`pnpm lint`, `pnpm build` (including TypeScript checks), `pnpm run
docs`, the full connection suite, and the existing IVF_RQ regression
pass. Generated documentation adds no further changes.
The merge preserves upstream's explicit RaBitQ heading anchor alongside
the `IvfRq()` example and five-bit default/reader-compatibility
guidance.
Repository-wide `ruff check .` still reports 49 existing errors in
unchanged documentation examples under `docs/web-tests`; root formatting
also finds 11 unchanged documentation example files. The Python package
checks pass.
Fixes#4582
<!-- lance-gatekeeper-fix:v1 agent=6ef9406f0142b1ab66d53d9d56098223
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
`OpenClipEmbeddings` loads its model with
`open_clip.create_model_and_transforms()`, which hands the model back in
training mode (the open_clip README calls `model.eval()` right after it
for that reason). `OpenClipEmbeddings` never switches it out of training
mode, so layers that behave differently during training stay active
while we embed: stochastic depth in the timm-based image towers
(ViTamin, ConvNeXt, MobileCLIP2-L-14), dropout in the Hugging Face text
towers (`xlm-roberta-*`, `roberta-ViT-B-32`, `nllb-clip-*`), and batch
norm in the ResNet towers loaded with non-OpenAI weights. The
`torch.no_grad()` blocks turn off autograd but don't change how these
layers behave.
The default `ViT-B-32` has none of those layers, which is probably why
nobody has run into this. With dropout or stochastic depth active, the
same input gives a different vector on every call, so the vector stored
at ingestion and the one computed for an identical query don't match.
With batch norm in training mode, an image's vector depends on the other
images in its batch:
```python
import numpy as np
from PIL import Image
from lancedb.embeddings import get_registry
func = get_registry().get("open-clip").create(name="ViTamin-S", pretrained="datacomp1b")
image = Image.new("RGB", (256, 256), "red")
a = func.generate_image_embedding(image)
b = func.generate_image_embedding(image)
print(func._model.training, np.array_equal(a, b))
# before: True False
# after: False True
```
The fix calls `model.eval()` once after loading, same as the SigLIP,
ColPali and ImageBind embedding functions already do.
The new test swaps in a stand-in for the `open_clip` module, so it runs
without torch or a model download, and checks that the model handed back
by `create_model_and_transforms` ends up in eval mode.
BREAKING CHANGE: Embeddings change for OpenCLIP models with
training-sensitive layers, including ViTamin, ConvNeXt, MobileCLIP2,
Hugging Face RoBERTa/XLM-RoBERTa/NLLB text towers, and ResNet models
loaded with non-OpenAI weights. For tables embedded with affected
models, regenerate the stored embeddings in eval mode and rebuild their
vector indexes before querying with the new embeddings. Rebuilding an
index alone does not regenerate embeddings. The default ViT-B-32 model
is unaffected.
Co-authored-by: devjoinedthechat <235875317+devjoinedthechat@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Pandas converts NaN vector components to Arrow nulls. Both Python's NaN
mask and Rust's default append validator ignored null children, so
`add()` silently stored incomplete vectors despite
`on_bad_vectors="error"`.
Reject null components in the existing Rust ingestion validator, and
include null components in Python's bad-vector detection and filling.
`drop` removes affected rows, `fill` repairs missing components while
preserving valid ones, and `null` replaces affected vectors. Explicit
null components in Arrow and list inputs follow the same policies. Rust
allows whole null vectors and retains the explicit
`NaNVectorBehavior::Keep` opt-out.
The ingestion guide and synchronous/asynchronous `create_table()` option
docs describe null components, all four policies, and filling that
preserves valid components.
Validation:
- Reproduced the default pandas append failure with the regression test
before changing the Rust validator.
- Rebuilt the native Python extension from the merged sources. The
Python table and utility suites passed: 481 passed, 1 skipped. Coverage
includes create/append/overwrite, pandas/rows/Arrow inputs, every
bad-vector policy, float16/32/64, sliced and chunked arrays, all-null
components, and hidden children of null lists.
- All 43 focused Rust tests passed: 7 validator tests and 36 ingestion
tests.
- Rust Clippy with `remote`, tests, and examples passed. Ruff lint
passes for the Python package, formatting checks pass for modified
Python files, and `cargo fmt --all` is clean.
- Documentation follow-up: all 56 focused vector-policy tests passed;
Python package lint and creation-docstring formatting checks passed.
Repository-wide Ruff reports 49 existing errors in untouched
documentation examples. Create/overwrite retain their existing
`RuntimeError` wrapping for errors raised while consuming lazy Arrow
readers.
Fixes#4435
<!-- lance-gatekeeper-fix:v1 agent=b5108e27d28ef613a78c1e4354c03c48
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
`RemoteDBConnection.create_table` orders its parameters `(name, data,
schema, on_bad_vectors, fill_value, mode, exist_ok, ...)`, while
`DBConnection`, `LanceDBConnection` and `LanceNamespaceDBConnection` all
use `(name, data, schema, mode, exist_ok, on_bad_vectors, fill_value,
...)`. Since `RemoteDBConnection` subclasses `DBConnection`, identical
positional calls bind to different parameters: switching a working
script from `lancedb.connect(path)` to `lancedb.connect("db://...")`
silently turns `mode="overwrite"` into `on_bad_vectors="overwrite"` and
`exist_ok=True` into `fill_value=True`, and nothing client-side
validates either. `inspect.signature(...).bind(None, "t", data, None,
"overwrite", True)` shows it deterministically, no cloud needed.
This reorders the remote signature to match the other three; every
parameter keeps its own existing default (including the remote's
`mode=None`, which its body relies on), and the body already forwards by
keyword so nothing else changes. The new test asserts positional parity
with `DBConnection` through `embedding_functions` and fails on `main`.
Blast radius, stated plainly: only remote callers passing four or more
positional arguments are affected — positions 4–7 move; positions 1–3
and 8 are unchanged and keyword callers are unaffected. No in-repo
caller passes more than two positionals. Anyone relying on the current
order was relying on behaviour that contradicts the documented base
contract and diverges from local connections. Not added here: the
`storage_options`/`data_storage_version`/`enable_v2_manifest_paths`
parameters the remote signature still lacks.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
DataFusion plans SQL against the adapter's declared schema, which
omitted `_rowid` and `_rowaddr` even though Lance can project both. SQL
therefore rejected these columns before reaching the scan.
Add `BaseTableAdapter::with_system_columns` to opt into either column
using Lance's nullable `UInt64` fields. Scans without a projection now
select the full declared column list so their output matches the
provider schema. Default adapters retain data-only `SELECT *` and remain
compatible with `INSERT INTO ... SELECT *`; opted-in adapters include
the requested system columns in `SELECT *`.
Regression tests compare SQL results with native queries and cover
projection order, system-only projections, row-ID predicates, scans
without a projection, duplicate/unsupported names, and default SQL
inserts.
Validation:
- DataFusion unit suite: 436 tests passed with `cargo test --locked
--quiet --profile ci -p lancedb --features remote --lib
table::datafusion`.
- `cargo clippy --locked --quiet --profile ci -p lancedb --features
remote --tests --examples -- -D warnings` passed.
- The new builder's documentation example passed its doctest.
- `cargo fmt --all -- --check` and `git diff --check` passed.
Fixes#4666
<!-- lance-gatekeeper-fix:v1 agent=58d267206580227b3820df65af15b0a7
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Consumers that provide a local Lance checkout cannot resolve the
dependency graph because the SDK directly selects a vendored
`lance-arrow`, producing two path packages with the same name and
version. The Rust core now declares its upstream Git dependency
separately so the consuming workspace can override that source
consistently. Python continues inheriting the vendored workspace
dependency, keeping the sliced-struct validity fix in Maturin source
distributions. Standalone SDK builds still select that same fix through
the existing root patch.
Adds a downstream consumer CI regression that resolves a local Lance
override and checks that exactly one canonical `lance-arrow` package is
selected. The Lance version updater also updates the core declaration in
stable, preview and local modes. Public APIs and query behavior are
unchanged.
The workspace declaration lint excludes path dependencies because the
Python binding intentionally uses the bundled source. Duplicate and
unused external declarations remain errors; advisory, license,
banned-package and source checks continue covering the full dependency
graph.
---------
Co-authored-by: Lance Release <lance-dev@lancedb.com>
SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.
### Python experience
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.
The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.
For an async table:
```python
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
```
### Execution
The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.
Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.
Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.
Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
Remote materialized-view creation currently expects a REST response
containing a registry job ID, while refresh already submits SQL. Route
both through the SQL query backend and preserve Python's synchronous and
asynchronous return types.
### Query to Job mapping
`SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through
the same backend as `execute_query_async()`. Once submission succeeds,
it retains the returned `Query` and starts one client-side task that
consumes its result reader to completion. `Job::from_query_task()` wraps
that task, its cached outcome, a single ID and a cancellation callback.
Python exposes this as `Job[T]` or `AsyncJob[T]`.
The existing `job_type` property maps SQL-backed handles to the new
`"query"` kind. The shared `id` contains that query's UUID string, so
Python callers can inspect it with `db.describe_query(job.id)` on the
submitting connection. No separate `query_id`, backend enum or `type()`
API is added.
| Handle | `job_type` | `id` | Available lookup |
| --- | --- | --- | --- |
| SQL-backed | `"query"`, known at submission | Connection-scoped query
UUID string | `describe_query(job.id)` |
| Registry-backed | Existing server-reported kind, after `status()` or
`refresh()` | Registry job ID string | Registry job APIs, descriptions
and events |
| Local | `None` | `None` | The current in-process handle |
| Job concept | SQL-backed behavior |
| --- | --- |
| Submission | Await initial SQL submission before returning the handle.
Submission errors propagate immediately. |
| `status()` | Tracks the adapter task: running during result
consumption/finalization, finished after both succeed, failed if either
fails, or cancelled after cancellation. |
| `wait()` | Returns the operation's typed terminal result. One task
consumes the query reader once; repeated waits reuse its cached outcome
without resubmitting SQL or reopening the reader. |
| `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported
cancellation leaves the task running; an accepted request stops the
task. Cancelling a finished operation is a no-op. |
| Registry details | Query-backed handles expose no registry
description, event history, pause/resume or reattachment after a client
restart. `refresh()` updates cached lifecycle state. |
The Job tracks command completion rather than exposing SQL batches. A
wait timeout does not cancel the query. Query IDs remain scoped to the
submitting connection and cannot be used with registry APIs or as
durable resume tokens. Python `describe_query()` accepts both UUID
objects and the string `id` exposed by query-backed jobs. Rust, Python
and TypeScript expose the same classification through their existing
job-type accessors.
### Materialized-view operations
- **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace
and optional auto-refresh settings. After command completion, resolve
the job to `None`. Synchronous creation waits on the job, then opens and
returns the materialized view.
- **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same
adapter. After command completion, read the refreshed table's version
and row count to construct the existing typed refresh result.
Synchronous refresh waits on the same path.
Remote creation requires the SQL service. Execution placement remains a
server decision. REST endpoints are unchanged. The job guide documents
the ID/job-type contract and query-backed limitations.
Regression coverage exercises SQL submission, namespaces, refresh
options, job-type/ID preservation through typed result mapping, repeated
waits, failures and cancellation rejection. No local builds or tests
were run.
SQL execution now starts with one Flight call. The server can stream
rows immediately or return a polling continuation, which the SDK follows
through the existing query handle and reader APIs. The same behavior
applies with and without parameters.
Both initial transports share direct-stream lifecycle handling. Handoffs
are explicitly negotiated; existing servers can continue returning
ordinary streams. Add coverage for streaming, deferred results,
cancellation, malformed decisions, and legacy responses.
```mermaid
flowchart TD
A[execute_query or execute_query_async] --> B{Parameters supplied?}
B -->|No| C[DoGet: SQL command ticket]
B -->|Yes| D[DoExchange: SQL and Arrow parameter row]
C --> E[Server chooses execution path]
D --> E
E -->|Stream| F[Initial response: streaming decision and Arrow batches]
E -->|Defer| G[Initial response: PollInfo continuation]
G --> H[SDK follows PollFlightInfo]
H -->|Still running| H
H -->|Result endpoints available| I[SDK fetches tickets with DoGet]
F --> J[Query reader yields Arrow batches]
I --> J
```
`execute_query` opens the reader; `execute_query_async` returns its
query handle. The caller does not select the transport or execution
path.
Infer Blob V2 input access from the Python Function signature. `video:
BlobReader` requests a lazy, seekable reader; `video: bytes` requests
the complete payload. Users do not need a separate `blob_inputs`
registration option.
## Python experience
For a video table joined to a table of frame timestamps, the Function
can pass its reader directly to a media decoder:
```python
import io
import av
from lancedb import BlobReader, udf
@udf(pip=["av", "Pillow"])
def extract_frame(video: BlobReader, timestamp_s: float) -> bytes:
with av.open(video) as container:
stream = container.streams.video[0]
container.seek(int(timestamp_s / stream.time_base), stream=stream)
for frame in container.decode(stream):
if frame.time is not None and frame.time >= timestamp_s:
output = io.BytesIO()
frame.to_image().save(output, format="JPEG")
return output.getvalue()
raise ValueError("No frame at or after the requested timestamp")
```
After registering `extract_frame`, the query keeps the usual Function
call:
```sql
SELECT f.video_id, f.timestamp_s,
extract_frame(v.video, f.timestamp_s) AS image
FROM videos AS v
JOIN frames AS f ON v.id = f.video_id
```
The returned JPEG can feed a subsequent image-model step. The reader
provides `read`, `readinto`, `seek`, `tell`, `size()`, and
`read_range(offset, length)`; range reads leave the sequential cursor
unchanged. Each row invocation owns its reader, which closes after the
call. Decoder seeks can request the parts of the MP4 needed to locate
and decode the requested frame; the amount read depends on the file
layout and codec.
Functions that need the whole value can keep a bytes signature:
```python
@udf
def video_length(video: bytes) -> int:
return len(video)
```
## Execution contract
- Signature inference maps `BlobReader` to a Blob V2 field and records
`blob_access="reader"`. `bytes` keeps its binary Arrow type and records
`blob_access="bytes"`; it can also bind to a Blob V2 source. Explicit
schemas support the same choice, including a Blob V2 field annotated as
`bytes`.
- Reader execution carries a compact blob reference into the Function
worker. Payload reads happen when Python calls the reader, with requests
split into at most 1 MiB ranges. The full video does not need to enter
the Arrow input batch just to invoke the Function.
- Bytes execution materializes the full source before the Function runs
and remains subject to eager-input memory admission. Calling
`reader.read()` without a size also materializes the remaining contents
in Python; lazy access depends on how the Function consumes it.
- Input access metadata survives Python/Rust Function serialization and
registration. Packaged Functions import the worker-provided reader
implementation. This PR defines the SDK contract; execution requires
corresponding server support.
- This changes blob inputs. The example still returns ordinary image
bytes; it does not add streamed outputs or run an image model.
Adds coverage for signature inference, explicit schemas, nullable
readers, signature round trips, seeking, bounded range reads, buffering,
and invalid responses.
## What
Move typed job decode error construction behind a feature-aware helper
so builds without the `remote` feature do not reference `Error::Http`.
Local decode failures continue to surface as `Error::Runtime`, while
remote builds retain HTTP errors with request IDs.
Add a regression test for local typed-result decode failures.
Closes#4096.
## Testing
- `cargo fmt --all -- --check`
- `cargo check -p lancedb --no-default-features`
- `cargo test -p lancedb --no-default-features
job::tests::local_typed_result_decode_errors_are_runtime_errors`
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#4250
`list_indices()` loads each FTS index's segment params from the index
files to fill in `index_details`. That call used `?`, so a single FTS
index with missing files made the whole listing fail. A BTree or Bitmap
index with missing files still lists fine, because it never takes that
path.
Now, if the params can't be loaded, it logs a warning and keeps the
entry with the details from the manifest. Callers can still see the
damaged index and rebuild or drop it.
Added a Python test based on the repro in the issue. It fails on 0.39.0
with the same RuntimeError.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
Fixes#4401.
- Validate cached remote table entries with the existing `describe`
endpoint before returning a handle.
- Preserve cached server-version behavior after a successful validation.
- Add a regression test for a table dropped by another connection after
its entry has been cached.
A cached entry can otherwise outlive an external drop, allowing
`open_table()` to return a handle that fails only on its first later
operation.
## Verification
[Linux
validation](https://github.com/Ganesh1729-ui/lancedb/actions/runs/39420560974)
passed against the exact runtime and test sources at PR head
`d39092b01f6b49921b5d66ccedf29709a7ddc23d`, using the repository's Rust
toolchain and `ci` profile:
- `cargo fmt --all -- --check`
- `cargo test --profile ci --locked -p lancedb --features remote --lib
remote::db::tests::test_open_table -- --nocapture`: **8 passed**,
including cached-table deletion, independent handle checkout,
branch/version handling, schema seeding and describe-response handling.
- Removing only the cached-hit existence guard makes
`test_open_table_checks_that_a_cached_table_still_exists` fail at its
`TableNotFound` assertion; restoring the fix makes the same test pass
again.
- `git diff --check`
The validation workflow is on a separate fork branch and is excluded
from this PR. These are focused, mock-backed remote unit tests; the full
upstream matrix and live Cloud integration tests still need upstream CI
approval. Local Windows compilation was unavailable because the MSVC
environment has no `link.exe`; the linked Linux run completed
successfully.
---------
Co-authored-by: Ganesh172919 <ganeshreddy17291122@gmail.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#2211.
Local queries converted `limit`/`offset` with `as i64`, so
`limit(usize::MAX)` wrapped to `-1` and Lance failed with `Limit must be
non-negative`. The remote client already worked around this (#2354);
this fixes the local path.
- Saturate scanner offsets to `i64::MAX` and cap limits at `i64::MAX -
offset`, keeping both conversion and full-text pagination addition safe.
- Use `saturating_add` for vector `top_k = limit + offset`, then cap
candidates at the dataset row count to avoid indexed-search capacity and
refinement overflows. Keep a positive candidate bound for empty tables
because Lance rejects `k = 0`.
- Leave public APIs and remote query handling unchanged.
### Testing
- Regression coverage includes plain scans, indexed IVF search,
refinement, index bypass, empty vector tables, and indexed full-text
pagination with finite and extreme limits/offsets.
- Reproduced the indexed-search and FTS overflows before applying the
follow-up fix.
- `cargo test --quiet --features remote -p lancedb --lib query::` — 81
passed.
- `cargo clippy --quiet --features remote -p lancedb --tests --examples`
— clean.
- `cargo fmt --all` and `git diff --check` — clean.
Vector planning now reads the dataset row count (metadata fast path) to
bound candidate allocation; this was not performance-benchmarked.
---------
Co-authored-by: Romin Parekh <roparekh@adobe.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- allow `db://` to select the remote storage root when a host override
supplies the endpoint
- preserve the empty `x-lancedb-database` header used by the legacy
clients for root storage
- reject root-storage connections without a host override before
constructing an invalid Cloud URL
- document the supported URI and cover both accepted and rejected
configurations
## Root cause
The shared Rust remote URL parser required every `db://` URI to contain
a host. That validation was introduced when the Node and Python clients
moved to the Rust implementation, but the legacy Node client
intentionally treated the empty host in `db://` as the storage-root
selector.
## Validation
- `cargo fmt --all`
- `cargo test --quiet --features remote -p lancedb
test_connect_remote_storage_root`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
Fixes#1944
<!-- lance-gatekeeper-fix:v1 agent=c71ca188791ad36fe5644c12960f758c
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Sync remote table and connection methods inherited bare base-class
stubs, so unsupported conversion and bulk-drop operations raised
`NotImplementedError` with an empty message and differed from the async
API.
Delegate `to_lance()`, `drop_all_tables()`, and `drop_database()` to the
async implementation. Sync and async remote Lance conversion share
#4306's native-table guard and reject before requesting a URI or opening
client storage. Conversion documentation directs remote reads through
server queries.
Add an explicit Polars error, suggest query conversions in
Arrow/Pandas/Polars errors, and give the remaining base-class errors an
operation name. Preserve upstream `blob_mode` validation for remote
Arrow and Pandas conversion. The shared remote bulk-drop error suggests
dropping tables individually.
Regression tests cover sync/async rejection parity, accessible and
missing advertised locations, pinned versions, and bulk-drop rejection
without requests. Sync blob query tests verify that descriptions come
from the server and unsupported modes cannot open client storage. Keep
the remote closed-blob-handle assertion consistent with the upstream
`ValueError` contract.
Validation:
- All 30 focused regression cases passed.
- 222 Python tests passed with rebuilt bindings in the locked uv
environment: the full remote and error suites, plus selected local
conversion, blob, and deletion tests.
- Python lint and formatting, Rust formatting, `cargo check --quiet
--features remote --tests --examples`, and `cargo clippy --quiet
--features remote --tests --examples` passed.
- Root Ruff checking reports 49 existing errors in upstream
`docs/web-tests`; those files are unchanged relative to `main`.
Fixes#4392
<!-- lance-gatekeeper-fix:v1 agent=95db5dd7002043a2c48924ded18b51b2
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#4514
## Summary
Honor an explicitly supplied `OpenaiReranker(api_key=...)` before
falling back to `OPENAI_API_KEY`, matching the constructor
documentation. Previously the environment key always won when both were
set, so an instance configured for a different account still used the
process-wide account.
## Tests
Added four cases using real OpenAI client construction without sending
requests: explicit/environment conflict, explicit key alone, environment
fallback, and missing credentials. The conflict case fails before the
change; all four pass afterward. Clients are closed after each
assertion.
- Targeted source-module tests: 4 passed.
- Ruff check and format check: passed for both changed files.
- Full native source build was attempted but failed because the
environment has no Rust toolchain; bootstrap then attempted a cache path
outside the writable workspace. The targeted tests load the current
Python reranker module against installed dependencies and the released
LanceDB package; no database/native behavior is exercised. Full
native/integration suite not run.
This change and its tests were prepared with an AI coding agent. The
diff was reviewed and the failure/pass behavior verified locally.
Co-authored-by: Xuanwo <github@xuanwo.io>
OpenAI shuts down `gpt-4-turbo` and its snapshots on 2026-10-23
(https://developers.openai.com/api/docs/deprecations), so
`OpenaiReranker` fails for every user who relies on the default
`gpt-4-turbo-preview` after that date.
This changes the default to `gpt-5.6-sol`, the replacement OpenAI names
for `gpt-4-turbo` on that page, in the class, its docstring and the
integration docs page. `_rerank` also stops sending `temperature=0`: the
GPT-5 models only accept the default temperature, so keeping it would
fail the new default on its first call. The reranker still asks for
`json_object` output and the rest of the request is unchanged.
The new test calls `_rerank` with a mocked client and checks that no
`temperature` is sent. It fails on main and passes with the change. No
request was made to OpenAI.
This change was made with Claude Code.
Co-authored-by: Xuanwo <github@xuanwo.io>
`Table.take_offsets` and `Table.take_row_ids` declare a keyword-only
`with_row_id: bool = False`, but none of the three implementations
accept it — `LanceTable`, `AsyncTable` and `RemoteTable` all raise
`TypeError: ... got an unexpected keyword argument 'with_row_id'` on the
documented call. The builders they return already support it
(`LanceTakeQueryBuilder.with_row_id()` /
`AsyncTakeQuery.with_row_id()`), so this is a signature that never
reached the implementations. The base docstrings also don't mention the
parameter and still say `take_offsets` returns a `pa.RecordBatch` (it
returns a `LanceTakeQueryBuilder`), so those are corrected too.
Each implementation now accepts the kwarg and applies `.with_row_id()`
to the builder it returns. Tests: sync and async assert `_rowid` appears
in the result columns; the remote test uses the suite's
request-capturing fixture and asserts `with_row_id: true` reaches the
wire (and is absent by default). All three fail on `main` with the
`TypeError`. ruff format/check clean.
A related mismatch I left alone: the deprecated
`LanceTable.create_scalar_index`/`create_fts_index` also drop a
`wait_timeout` the base declares — deprecated since 0.25, so not worth
touching here.
Co-authored-by: Xuanwo <github@xuanwo.io>
Arrow map scalars returned by a table query cannot currently be passed
back to `update`: the generic Arrow scalar conversion turns them into a
list of tuples, which the SQL literal encoder cannot handle. Empty map
scalars become an untyped SQL list and also fail to update a map column.
Add a dedicated `MapScalar` conversion that renders its keys and values
as a SQL map while preserving the existing null conversion. The
regression tests exercise real synchronous and asynchronous table
updates, quoted keys, nullable values, empty typed maps, and nested
maps.
Validation: the complete documented unit-test scope passed locally
(1,515 passed, 37 skipped; unchanged baseline: 1,507 passed, 37
skipped). Full doctests passed (71 passed, 6 skipped), and
repository-wide Python formatting/lint, Rust formatting/clippy, and
Python 3.13 type checking passed. The locally built native wheel also
passed all eight regression cases in a separate Python 3.13 environment.
Full Linux/S3 integration and all platform matrices remain for CI.
The SQL map constructor's existing backend constraints still apply:
duplicate keys are rejected, and a nested map containing both nonempty
and empty inner maps can fail SQL type unification. This change does not
claim arbitrary Arrow map roundtrip support.
AI assistance: implementation and local verification used Codex with
gpt-6.1-sol agents; the diff and evidence were reviewed before
submission.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Upgrade the locked Tokio version from 1.53.1 to 1.53.3 to fix
[RUSTSEC-2026-0386](https://rustsec.org/advisories/RUSTSEC-2026-0386)
and restore the Rust workflow's `deny` job. The diff changes only
Tokio's version and checksum.
Validation:
- `cargo deny check advisories` — passed.
- `cargo deny check advisories bans licenses sources` — passed with the
repository's `deny.toml`.
- `cargo check --quiet -p lancedb` — passed (dead-code warnings).
- PR `deny` job —
[passed](https://github.com/lancedb/lancedb/actions/runs/40000920646/job/120576455117).
Fixes#4248.
The preview index still contains pylance 1.0.0b6. uv's default
first-index resolution can select it, or backtrack to an old LanceDB
release when installing the extra. Meanwhile, the extra still permits
pylance 5 even though current main bundles Lance 14.
Raise the optional dependency to `pylance>=14.0.0b12`, the version
already pinned in the test extra, and keep the lockfile metadata
consistent. Document both Fury indexes and uv's `unsafe-best-match`
strategy in the Python README and the source-owned quickstart
(`docs/web/quickstart.mdx`). Explain the index trust requirement and
recommend installing the extra to enforce the version floor.
Add CI checks that reject older Lance majors and ensure the pylance and
tests extras remain compatible. Run these checks when either dependency
manifest changes, so a future Lance major bump cannot silently leave the
Python minimum behind.
### Validation
- Recreated the reported install on Ubuntu 26.04 / CPython 3.12.15 with
uv 0.12.23: `lancedb==0.40.0b5` plus the old index selects
`pylance==1.0.0b6`. Reproduced both the missing `blob_array` import and
the unexpected `blob_handling` scanner argument using a real blob table.
- Resolved the modified `python/pyproject.toml` with the pylance extra
against both indexes: selects `pylance==14.0.0b18`, matching the current
Rust pin.
- Ran both documented pip and uv installations in isolated environments:
`lancedb==0.41.0b7`, `pylance==14.0.0b18`; blob-array construction and a
blob-bytes round trip through `to_pandas(blob_mode="bytes")` pass. Both
package dependency checks pass.
- New regression fails on the original dependency bound (27
rejected-version cases incorrectly accepted), then passes with the fix.
Full CI-script suite: 3 tests pass.
- `uv lock --check` with both indexes passes; focused Ruff checks pass;
`git diff --check` passes.
- Repository-wide Ruff reports 49 existing errors in untouched
`docs/web-tests` files. Unrelated formatting changes were discarded.
Runtime smoke checks used released native wheels; source dependency
resolution was tested directly from the modified pyproject. No Rust or
Python runtime code changes.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Run the aarch64 Windows build/test cycle only on pushes to main so PRs
wait
for a single Windows platform (x86_64). Main keeps the same native
aarch64
coverage as before.
GitHub evaluates job-level `if` before the matrix strategy is applied,
so
the aarch64 target is split into its own `windows-arm` job instead of
being
filtered inside the existing matrix. The job keeps the historical check
name.
Adds the Enterprise reference page at
`docs/web/enterprise/distributed-search.mdx`, following the same
arrangement as Enterprise architecture. The page describes distributed
search and the embedded execution model, with links to shared search
guides. Enterprise architecture links to it, and the official sidebar
lists it immediately after Architecture in the existing Deployment
group.
Adds stable anchors for the RaBitQ and `approx_mode` sections linked by
the Enterprise guide.
Companion: https://github.com/lancedb/sophon/pull/8986 supplies the
detailed Enterprise overlay at the same path. Merge this reference-page
PR first so the overlay has an existing route to replace.
Validation:
- Combined-site assembly with both PR branches passed (309 files).
- Mintlify build validation and link, anchor, and redirect checks
passed.
Fixes RUSTSEC-2026-0385: `ctutils` < 0.4.4 generates masks with branches
on secret values (`u128` mask generation / comparisons are not
constant-time on some targets).
- Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0385
- Affected: `ctutils 0.4.2` pinned in `Cargo.lock`, reached transitively
via `digest 0.11` (RustCrypto)
- Change: `cargo update -p ctutils --precise 0.4.4`; single
lockfile-only change (patch bump, no API impact)
The `deny` CI job started failing for every PR head after the advisory
was published at 06:45Z today; `main`'s last `deny` run (05:51Z)
predates it and does not yet cover this advisory.
job.wait() polled on a backoff that doubled up to 5 s, so a job's
completion was seen 2.5 s late on average, as long as a small refresh
itself takes. Cap the interval at 1 s; the first polls still back off
from 200 ms, so a quick job is seen quickly and a long one costs one
request a second.
Linear: OSS-2704 https://linear.app/lancedb/issue/OSS-2704
The docs deploy workflow installs protobuf-compiler without refreshing
apt
lists first. Ubuntu noble-updates replaced `protobuf
3.21.12-8.2ubuntu0.3`
with `.0.4`, so stale package lists in older runner images make `apt
install`
fail with 404 and exit code 100. Run `apt update` before installing,
matching
the other workflows.
Make table and namespace listings consistent with other resource
listings across Rust, Python, and TypeScript. Callers can iterate over
individual names across all pages, with configurable page sizes and
continuation tokens.
In order to avoid breaking existing Python users, this change adds some
compatibility methods to the returned Python iterators. Code such as:
```
for t in db.list_tables().tables:
print(t)
```
will continue to work as expected because the iterator has a `tables`
field and a `page_token` field, which function as expected to reveal the
first page of results. In other words, it is never necessary to use the
iterator __as an iterator__. It may be used as if it were an old-style
response object. (As the iterator progresses, the `tables` list may
switch to the second, third, etc. page of results.) Similar
compatibility code allows the async python API to keep working as well.
The `Permutation.rename_column` docstring says it "only changes the name
of the column that is visible from this permutation". But
`permutation.py:868-870` adds the new name at the end of the selection
and then deletes the old key, so the renamed column also moves to the
last position. Hugging Face `datasets`, which `Permutation` mirrors
(#2725), keeps it in place. Positional formats feel it most:
`with_format("numpy")` swaps the feature order of every row.
Before, on `d97df0c`:
```
$ python repro.py <tree>/python/python # lancedb 0.41.0b12 extension, Python sources from the tree
['id', 'value']
['value', 'row_id']
[{'value': 0, 'row_id': 0}]
[[10 1]]
```
After:
```
$ python repro.py <tree>/python/python
['id', 'value']
['row_id', 'value']
[{'row_id': 0, 'value': 0}]
[[ 1 10]]
```
<details><summary>repro.py (its first output line,
<code>lancedb.__file__</code>, is cut above)</summary>
```python
import sys; sys.path.insert(0, sys.argv[1])
import tempfile
import lancedb
from lancedb.permutation import Permutation
print(lancedb.__file__)
db = lancedb.connect(tempfile.mkdtemp())
tbl = db.create_table("t", [{"id": i, "value": i * 10} for i in range(3)])
perm = Permutation.identity(tbl)
print(perm.column_names)
renamed = perm.rename_column("id", "row_id")
print(renamed.column_names)
print(renamed.__getitems__([0]))
print(renamed.with_format("numpy").__getitems__([1]))
```
</details>
For comparison, `datasets` 5.1.0 gives `['new_id', 'value']` for
`Dataset.from_dict({'id': [1, 2], 'value': [3, 4]}).rename_column('id',
'new_id').column_names`.
No issue exists for this. The fix rebuilds the selection dict in order.
The cost: code that renamed a column and then read it by position in
numpy batches (say `[:, -1]`) gets a different column after this, with
no error. `rename_columns` calls `rename_column` once per name, so it
gets the fix too. The tests that check schema order only rename `value`,
the last column, and `test_rename_rowid` checks membership but not
order. The new test renames `id`.
### Verification
```
# Python 3.12.6; lancedb 0.41.0b12 wheel from pypi.fury.io, its _lancedb.abi3.so copied into the tree; pylance 14.0.0-rc.1
$ PYTHONPATH=python python -m pytest python/tests/test_permutation.py -q -p no:cacheprovider
d97df0c: 72 passed in 2.53s
this branch: 73 passed in 2.36s
permutation.py reverted, test kept:
FAILED python/tests/test_permutation.py::test_rename_column_keeps_position
assert value: int64\nnew_id: int64 == new_id: int64\nvalue: int64
1 failed, 72 passed in 2.41s
$ ruff format --check . && ruff check . # ruff 0.9.9, as python.yml pins it
152 files already formatted
All checks passed!
# not run: pyright (its include list covers neither permutation.py nor tests/), the Rust build
```
@westonpace, this is your class from #2725. Do you want the docstring to
say outright that the column keeps its position? If so, I'll add that
line to both rename methods in this PR.
Signed-off-by: Christopher Krah <48338417+chrikrah@users.noreply.github.com>
`blobMode: "bytes"` left blobs inside structs as descriptors because
projection matching ignored parent structs and byte replacement only
rebuilt top-level fields.
Expand direct struct projections and aliases to their blob leaves, then
recursively replace the corresponding Arrow fields and data with
`LargeBinary`. Track literal output names separately from descendant
paths so scalar aliases stay intact and every overlapping blob
projection is converted. Preserve null structs, null and empty blobs,
sibling fields, schema metadata, and the existing row ID and snapshot
behavior. Blobs inside lists retain their descriptors. Update the API
docs to describe struct support.
Regression coverage includes full and parent-struct projections, leaf
and struct aliases, dotted output names, overlapping projections in both
orders, nested structs, nulls, empty blobs and results, multiple
batches, list siblings, query reuse, row IDs, and concurrent writes.
Regression tests reproduce the nested-blob and projection-collision
failures before the corresponding changes.
Validation:
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test __test__/nested_blob.test.ts __test__/blob.test.ts
__test__/query.test.ts --runInBand --verbose=false` — 142 passed
- `pnpm test __test__/table.test.ts --runInBand --verbose=false -t 'when
dealing with blob columns'` — 60 passed
Fixes#4609
<!-- lance-gatekeeper-fix:v1 agent=76edaac77d044c468d569b58efa2104a
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Xuanwo
Drew
lancedb-gatefixer[bot]andXuanwo
KarthikSandXuanwo
Joaquin Hui GomezandXuanwo
Jack YeandLance Release
Shiduo LiandXuanwo
Yeluru NitishandXuanwo
Kerman LauandXuanwo
alexcastgonzandXuanwo
LeonSGPandXuanwo
LuQQiu
Wyatt Alt
Colin Patrick McCabe
Christopher Krah