The sync RemoteDBConnection.create_materialized_view assembled the SELECT but
called the async create_materialized_view with the query as the 2nd positional
arg, which binds to `source=` (query= is keyword-only). Every call then failed
the "needs either query= or both source and select" validation. Pass query=query.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refresh is a submit-a-job verb, so its only public surface should be
MaterializedView.refresh() / AsyncMaterializedView.refresh() (which return a
job handle). Rename the connection methods to _refresh_materialized_view and
have the handles call that, so the raw by-name refresh is no longer advertised
on the connection. The pyo3 native binding is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- create_materialized_view now takes either query= or source+select (folds in
the old create_view builder) and returns a MaterializedView handle whose
.wait() blocks on initial population. create_view is removed -- it was
misnamed (it built a *materialized* view, while CREATE VIEW means the plain
non-materialized view the engine also supports).
- MaterializedView.refresh() and the remote Table.refresh_column() now return a
JobHandle directly, so tbl.refresh_column("c").wait() needs no db.job(...)
wrapper. db.job(id) is narrowed to reconnect-by-id (stored id / SQL / REST).
- rename View/AsyncView -> MaterializedView/AsyncMaterializedView (+ exports).
- tighten the replace path: only a not-found error on the pre-drop is benign;
real failures (perms/server) now surface instead of being swallowed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Geneva Table.load_columns() parity on the REST-only client. Fills existing
columns from an external Parquet/Lance/IPC source by primary-key join.
- BaseTable::load_columns default (NotSupported) + public Table::load_columns,
taking a LoadColumnsRequest (source uris/format/storage_options, target/source
key, (target, source?) column mappings, on_missing, worker/batch/commit knobs).
- Remote impl POSTs to /v1/table/{id}/load_columns with the matching body;
mock test asserts the request shape.
- PyO3 binding + Python remote Table.load_columns(source, pk, columns, *,
source_format, source_pk, on_missing, ...) accepting a column list or
{target: source} dict.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
View.refresh(full=True) (sync + async) now works -- it previously raised
NotImplementedError. Thread the flag through the client: RefreshMaterialized-
ViewRequest.full -> the REST body (RemoteRefreshMaterializedViewRequest.full);
pyo3 refresh_materialized_view(full=...); Connection.refresh_materialized_view(
name, full=) sync + async. A full refresh forces a recompute-and-replace and
preserves the view's indexes (reindexed by the distributed indexer).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add View.create_index / create_scalar_index / create_fts_index / search
as pass-throughs to open_table(name). A materialized view is a real Lance
dataset; these let it be indexed and searched like any other table,
closing the parity gap with Geneva (whose create_materialized_view returns
a first-class Table).
The server-side create_index handler records indexes declared on a view so
they survive a full refresh (which overwrites the dataset, dropping its
indices); that re-apply is wired in the sophon engine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thread priority (Kueue tier) through refresh_column at every layer (Python sync+async
+ RemoteTable -> pyo3 -> Rust client trait/public/remote -> REST body), mirroring
num_workers/batch_size. The function keeps its priority as a default; the per-refresh
value overrides. Also adds the previously-missed batch_size to RemoteTable.refresh_column
(the REST sync path). cargo check (lancedb --features remote --tests, lancedb-python) +
ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
batch_size / num_workers / max_workers are invocation concerns (how to schedule THIS
refresh), so expose batch_size on refresh_column through every layer (Python sync+async
-> pyo3 -> Rust client -> the REST RefreshColumnRequest.batch_size, which the handler
already forwards into the backfill). num_workers/max_workers were already invocation-
placed; batch_size was the gap. The function may still carry a default; the refresh
override wins (extends the batch_size_override model). Both crates cargo-check clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A computed column is an expression over a registered function applied to input
columns, not a UDF coupled to a column. fn("data") already returned the expression
string "fn(data)"; make it a ColumnExpr (a str subclass) that also carries the
function's return type, so add_columns(computed={"vec": embed("data")}) declares the
column with no hand-written type. _normalize_computed handles the new form (and tuple
keys for STRUCT fan-out) and keeps the legacy {col: (sql_type, expression)} tuple.
add_computed_column is deprecated (delegates, with a DeprecationWarning). The function
stays decoupled from columns -- register once, apply anywhere.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thread an optional partition_by through the client: CreateMaterializedViewRequest
-> REST body -> pyo3 binding -> Python create_materialized_view/create_view
kwarg (sync + async). The server partitions the view's table function by the
named source column -- by IVF index clusters if the column is indexed
(image-dedup), else by distinct value. Unifies Geneva's partition_by +
partition_by_indexed_column into one knob.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirrors the sync ergonomics on the async surface: AsyncConnection
create_function(udf, replace=)/create_view/job; AsyncTable.add_computed_column;
AsyncView + AsyncJobHandle (await + asyncio.sleep; shared submission-prefix
matcher with the sync JobHandle). Decorator + REST routes are shared/already
validated; this is the async wrapper layer. Exported from the package root.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
db.job(id) gets the submission id the refresh/backfill endpoints return,
but list_jobs / cancel report the agent's manifest id
(<table>-<type>-<first 8 of submission id>). JobHandle now matches that
(exact id or submission prefix) so wait()/progress() truly track, and
cancel() cancels by the resolved canonical id instead of the unusable
submission uuid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings the @udf/@table_udf decorator + type inference into lancedb as
lancedb.udf (Apache-2.0), and adds the ergonomic glue to the existing
connection/table so there's no separate object model:
- create_function() accepts a Udf (and a replace= flag)
- Table.add_computed_column(column, udf)
- create_view(name, source, select, ...) -> View (assembles the SELECT)
- Connection.job(job_id) -> JobHandle
- View / JobHandle are thin references over a connection
Exports udf/table_udf/Udf/JobHandle/View from the package root. The
operations stay the existing remote-only methods (enterprise/cloud); the
decorator works locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Exposes the existing server-side CANCEL JOB (CoordinatorCatalog::cancel_job)
as a REST-backed SDK method: Database trait default NotSupported,
RemoteDatabase POSTs /v1/job/{id}/cancel, pyo3 binding, sync+async python
wrappers. Best-effort: a missing job returns false, not an error. Mock-HTTP
unit test in test_derived_compute_routes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per the interface design: computed columns are parameters on the
existing add_columns operation, not a separate method.
- BaseTable::add_computed_columns((name, sql_type) pairs + a f(args)
expression) -- default NotSupported; RemoteTable posts 'computed'
entries to the existing /v1/table/{id}/add_columns route.
- python add_columns gains computed= on LanceTable, RemoteTable, and
AsyncTable: tbl.add_columns(computed={'doubled': ('FLOAT',
'double_it(val)')}); grouped by expression so struct-returning
functions' columns land adjacently.
Adds the derived-compute interface to the SDK:
- Database trait: create/list/drop_function, create/refresh/alter/
drop/list_materialized_view, list_jobs -- default implementations
return Error::NotSupported (NotImplementedError in python), so
existing Database impls are unaffected; local single-node
implementations are planned. BaseTable gains refresh_column with
the same default.
- RemoteDatabase/RemoteTable implement them against the server REST
routes (/v1/function/*, /v1/materialized_view/*, /v1/job/list,
/v1/table/{id}/refresh_column), with mock-HTTP unit tests.
- Connection/Table public methods, pyo3 bindings (FunctionInfo,
MaterializedViewInfo, JobInfo pyclasses), and python wrappers:
sync on the DBConnection base (shared by local and remote
connections), async on AsyncConnection; refresh_column on
LanceTable, RemoteTable, and AsyncTable.
Expose the merged Rust OAuth header provider through the Node/TypeScript
connection path.
Includes:
- Native OAuthConfig conversion for napi-rs
- ConnectionOptions.oauthConfig plumbing
- Public TypeScript OAuthConfig and OAuthFlowType exports
- Generated TypeScript API docs for the new config surface
- input-validation and debug-redaction coverage in the Rust binding
layer
Local validation: cargo fmt --all; git diff --check.
Fixes#3589
## Problem
Multiple `warnings.warn()` calls across the Python client are missing
the `stacklevel=2` parameter. This causes warning messages to point to
lancedb internal code instead of the user's code that triggered the
warning, making debugging difficult.
## Solution
Add `stacklevel=2` to 7 `warnings.warn()` calls across 4 files:
| File | Warnings Fixed |
|------|---------------|
| `remote/db.py` | `request_thread_pool`, `connection_timeout`,
`read_timeout` deprecation warnings |
| `remote/table.py` | `cleanup_old_versions`, `compact_files`,
`optimize` no-op warnings |
| `table.py` | `data_storage_version`, `enable_v2_manifest_paths`,
`retrain` deprecation warnings |
| `embeddings/colpali.py` | `use_token_pooling` deprecation warning |
## Verification
- All 4 modified files pass `ast.parse()` syntax check
- Only `stacklevel=2` added — no other changes
## Changelog
| Date | Change | Author |
|------|--------|--------|
| 2026-06-27 | Add missing stacklevel=2 to warnings.warn() calls |
rtmalikian |
### Files Changed
- `python/python/lancedb/remote/db.py` — Add stacklevel=2 to 3
deprecation warnings
- `python/python/lancedb/remote/table.py` — Add stacklevel=2 to 3 no-op
warnings
- `python/python/lancedb/table.py` — Add stacklevel=2 to 3 deprecation
warnings
- `python/python/lancedb/embeddings/colpali.py` — Add stacklevel=2 to 1
deprecation warning
### Verification
- Syntax check passed on all modified files
---
**About the Author:** Raphael Malikian — Clinical AI Solutions
Architect. I specialise in building and fixing AI/ML systems for
healthcare, including vector databases, RAG pipelines, and clinical NLP.
If you need help with your project or think I can add value to your
organisation, feel free to reach out — I'd love to connect.
📧rtmalikian@gmail.com🔗 GitHub: https://github.com/rtmalikian🔗 LinkedIn:
http://www.linkedin.com/in/raphael-t-malikian-mbbs-bsc-hons-71075436a
---
**Disclosure:** This code was developed with assistance from
DeepSeek-V4-Pro (DeepSeek) via Hermes Agent (Nous Research). All changes
were reviewed, tested against the actual codebase, and verified for
correctness.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Fixes#2934
## Problem
Passing a `RemoteTable` to `permutation_builder()` raises a cryptic
`AttributeError`:
```
AttributeError: 'RemoteTable' object has no attribute '_inner'
```
This leaves users confused about what went wrong and why.
## Root Cause
`PermutationBuilder.__init__()` calls `async_permutation_builder(table)`
which accesses `table._inner` — the underlying Rust Lance table object.
`RemoteTable` connects to LanceDB Cloud/Enterprise and does not have a
local `_inner` attribute, making permutations fundamentally unsupported
on remote tables.
## Solution
Added an early check in `PermutationBuilder.__init__()` that verifies
the table has `_inner` before calling the Rust function, raising a clear
`TypeError` with an explanation of why permutations don't work on remote
tables.
## Verification
- Syntax validated with `ast.parse()`
- Structural verification: single call site (`permutation_builder()`),
guard placed before Rust FFI call
- Error message tested with mock: `MockRemoteTable()` correctly triggers
`TypeError`
## Changelog
| Date | Change | Author |
|------|--------|--------|
| 2026-06-28 | Added remote table guard in PermutationBuilder.__init__ |
rtmalikian |
### Files Changed
- python/python/lancedb/permutation.py — Added `hasattr(table,
"_inner")` check with clear error
---
**About the Author:** Raphael Malikian — Clinical AI Solutions
Architect. I specialise in building and fixing AI/ML systems for
healthcare, including vector databases, RAG pipelines, and clinical NLP.
If you need help with your project or think I can add value to your
organisation, feel free to reach out — I'd love to connect.
📧rtmalikian@gmail.com🔗 GitHub: https://github.com/rtmalikian🔗 LinkedIn:
http://www.linkedin.com/in/raphael-t-malikian-mbbs-bsc-hons-71075436a
---
**Disclosure:** This code was developed with assistance from
deepseek-v4-pro (DeepSeek) via Hermes Agent (Nous Research). All changes
were reviewed, tested against the actual codebase, and verified for
correctness.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Updates Lance Rust workspace dependencies and Java lance-core to
v9.0.0-beta.10.
No compatibility code changes were required; clippy and rustfmt passed
after installing the missing runner components.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.10
Expose the merged Rust OAuth header provider through the Python async
connection path.
Includes:
- Python OAuthConfig and OAuthFlowType public config objects
- PyO3 conversion into the Rust OAuthConfig
- connect_async(oauth_config=...) plumbing
- repr redaction coverage for client_secret
Local validation: cargo fmt --all; ruff format/check on touched Python
files.
## Summary
Add the Rust OAuth header provider for remote LanceDB connections.
This supports client credentials and Azure managed identity flows,
handles token caching and refresh, redacts secrets in Debug output, and
wires `ConnectBuilder::oauth_config()` into the remote client while
rejecting ambiguous API-key/header-provider combinations.
By default the read freshness provider was not included in the namespace
client, preventing the read freshness headers from being included in the
request. This prevents checkout_latest() from working as expected when
using the namespace client.
This fix ensures the provided is built into the client when the
namespace impl and properties are provided.
## Summary
Skip inserting the x-api-key header when the configured API key is
empty.
This lets bearer-token or other dynamic-header authentication avoid
sending an empty static API key header alongside the real auth header.
Fixes#3563
## Summary
- Add `stacklevel=2` to 10 `warnings.warn()` calls across 4 files
- Fix broken message concatenation in `table.py` where the second string
was incorrectly passed as the `category` parameter
## Problem
Multiple `warnings.warn()` calls in the `python/lancedb/` codebase were
missing the `stacklevel` parameter. Without `stacklevel=2`, warnings
point to library internals instead of the caller's code, making it
impossible for users to identify which of their function calls triggered
the warning.
Additionally, two calls in `table.py` (lines 3411 and 3420) had a more
serious bug: the deprecation message was split across two separate
string arguments, causing the second string to be passed as the
`category` parameter instead of being concatenated with the first
string. This would cause `TypeError` when the warning was triggered.
## Changes
| File | Fixes | Description |
|------|-------|-------------|
| `embeddings/colpali.py` | 1 | Add `stacklevel=2` to
`use_token_pooling` deprecation warning |
| `remote/db.py` | 3 | Add `stacklevel=2` to `request_thread_pool`,
`connection_timeout`, `read_timeout` deprecation warnings |
| `remote/table.py` | 3 | Add `stacklevel=2` to `cleanup_old_versions`,
`compact_files`, `optimize` no-op warnings |
| `table.py` | 3 | Fix broken message concatenation for
`data_storage_version` and `enable_v2_manifest_paths` deprecation
warnings + add `stacklevel=2` to `retrain` deprecation warning |
## Verification
```python
# All warnings.warn() calls now have stacklevel
python3 -c "import ast, os; ..."
# Result: All warnings.warn() calls now have stacklevel!
```
## Changelog
| Date | Change | Author |
|------|--------|--------|
| 2026-06-20 | Fix missing stacklevel=2 in 10 warnings.warn() calls +
fix broken message concatenation | rtmalikian |
### Files Changed
- `python/python/lancedb/embeddings/colpali.py` — Add stacklevel=2
- `python/python/lancedb/remote/db.py` — Add stacklevel=2 to 3
deprecation warnings
- `python/python/lancedb/remote/table.py` — Add stacklevel=2 to 3 no-op
warnings
- `python/python/lancedb/table.py` — Fix broken message concatenation +
add stacklevel=2
### Verification
- AST-based audit confirms all `warnings.warn()` calls now include
`stacklevel=2`
- Syntax check passes for all 4 modified files
---
**About the Author:** Raphael Malikian — Clinical AI Solutions
Architect. I specialise in building and fixing AI/ML systems for
healthcare, including vector databases, RAG pipelines, and clinical NLP.
If you need help with your project or think I can add value to your
organisation, feel free to reach out — I'd love to connect.
📧rtmalikian@gmail.com🔗 GitHub: https://github.com/rtmalikian🔗 LinkedIn:
http://www.linkedin.com/in/raphael-t-malikian-mbbs-bsc-hons-71075436a
---
**Disclosure:** This code was developed with assistance from **Hermes
Agent** (Nous Research). All changes were reviewed, tested against the
actual codebase, and verified for correctness.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Updates LanceDB's Lance dependencies to v9.0.0-beta.2 across the Rust
workspace and Java lance-core dependency.\n\nNo compatibility fixes were
required; clippy and formatting pass after installing the missing
toolchain components on the runner. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.2
This PR is for the Read path against blob v2. #3528 handles declare +
write, and this this adds materialization on local tables.
- blob_columns()
- fetch_blobs(column, row_ids) → bytes
- fetch_blob_files(column, row_ids) → lazy handles
- Pass _rowid from query().with_row_id(). Remote returns NotSupported.
(for now)
### Use cases
search, grab row ids, materialize images:
```rust
let row_ids = /* _rowid from hits */;
let images = table.fetch_blobs("image", &row_ids).await?;
```
Large blobs: open handles, read only what you need:
```rust
let handles = table.fetch_blob_files("image", &row_ids).await?;
let bytes = handles[0].as_ref().unwrap().read().await?;
```
Filter then batch fetch: collect ids from a filter, one call.
Multiple blob columns: image and thumbnail independently.
Row ids from before compact: still resolve.
### Alignment note
Lance `read_blobs` drops null rows. We descriptor-take first, read
non-null ids, re-expand to match input order. Null and zero-length blobs
come back null/None. Bytes path sets `preserve_order(true)`. So I added:
```
TODO(lance): expose selection_index or an aligned execute so we can drop the pre-read.
```
### Tests
`cargo test -p lancedb --test blob_integration`
- 30 tests covering nulls, reorder, dups, cross-fragment bytes + files,
compact, delete, legacy v1 errors.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The server now serializes an index's `created_at` as an RFC 3339 string
(e.g. `"2026-06-18T21:37:36.637Z"`), but the client deserializer only
accepted a unix timestamp in milliseconds. This caused `list_indices` to
fail with:
```
Failed to parse list_indices response: invalid type: string "2026-06-18T21:37:36.637Z", expected a unix timestamp in milliseconds
```
This PR replaces the fixed millisecond deserializer with a custom one
that accepts both an RFC 3339 string (current server) and a
unix-millisecond integer (legacy deployments), so the client works
against any server version.
It also improves the `IndexConfig` repr in the Python bindings.
Previously it printed only three fields (`Index(FTS, columns=["text"],
name="text_idx")`), hiding the metadata that `list_indices` returns. It
now renders every populated field, omitting any that are `None`. Each
value is valid Python — integer counts use `_` thousands separators and
`created_at` uses the `datetime` repr — so values round-trip. The real
repr is a single line; it's wrapped here for readability:
```python
>>> table.list_indices()
[IndexConfig(
name="text_idx",
index_type="FTS",
columns=["text"],
index_uuid="aefd3e00-2f95-4bdc-92ac-06de84442bf1",
type_url="/lance.table.InvertedIndexDetails",
created_at=datetime.datetime(2026, 6, 18, 21, 37, 36, 637000, tzinfo=datetime.timezone.utc),
num_indexed_rows=2,
size_bytes=3_669,
num_segments=1,
index_version=1,
index_details={
'lance_tokenizer': None,
'base_tokenizer': 'simple',
'language': 'English',
'with_position': False,
'max_token_length': 40,
'lower_case': True,
'stem': True,
'remove_stop_words': True,
'custom_stop_words': None,
'ascii_folding': True,
'min_ngram_length': 3,
'max_ngram_length': 3,
'prefix_only': False,
},
)]
```
Fixes#3556🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updates LanceDB's Lance dependencies from v8.0.0-beta.17 to
v8.0.0-beta.19.
This includes the Rust workspace Lance crates, Cargo.lock refresh, and
Java lance-core version bump. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.19
Updates the Lance Rust workspace dependencies and Java lance-core
dependency to v8.0.0-beta.17.
No LanceDB compatibility code changes were required; validation passed
with cargo clippy and cargo fmt. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.17
The "Create release commit" workflow (`make-release-commit.yml`) has
failed on its last two runs; no release tags have been created since
June 4. Since this workflow creates the tag that the cargo/npm/pypi/java
publish workflows trigger off of, all recent releases are effectively
blocked.
The workflow installs `bump-my-version` unpinned. Version `1.4.0` added
a check that refuses to run `pre_commit_hooks` containing shell syntax
(pipes, `&&`, `if`, variable expansion) unless `allow_shell_hooks =
true` is set. Both bumpversion configs use such hooks:
- `python/.bumpversion.toml` — updates `Cargo.lock` after the bump
(fails first)
- `.bumpversion.toml` — runs `mvn versions:set` for the Java packages
The job dies at the version-bump step with:
> Hook '…' contains shell syntax (pipes, redirects, or variable
expansion). Set `allow_shell_hooks = true` in your configuration to
enable shell execution…
This sets `allow_shell_hooks = true` in both configs to restore the
previous behavior.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
