mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-14 15:52:26 +00:00
549a41e6483ab96ff0d8063b2a983ec18f043734
497
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Gatefixer
|
549a41e648 | Merge main into blob concurrency repair | ||
|
lancedb-gatefixer[bot]
|
629f39d4a5 |
fix(python): preserve mixed blob values before Arrow conversion (#4304)
## Cause Python list and pandas rows were converted to Arrow before the target blob v2 schema was applied. In a mixed bytes and URI batch, Arrow inferred a binary column and encoded the URI as inline bytes; dict and `lance.Blob` values could fail conversion. ## Fix Build blob v2 columns from the original Python values using the target field's storage type before converting the rest of each row. This preserves inline bytes, external URIs, descriptor dictionaries, blob objects, and nulls for `add`, `merge_insert`, and `create_table`. ## Validation - `python/tests/test_blob.py`: 76 passed, 1 skipped. - Python project Ruff check passed. - Repository-wide Ruff check still reports 29 existing findings in `ci/` files outside this change. Fixes #4271 <!-- lance-gatekeeper-fix:v1 agent=54c95f06f11c88187b07a430aac6be2f generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
51ea02ea8c |
fix: pin remote BlobFile reads to the opened version (#4295)
## Cause On latest-tracking remote tables, `fetch_blob_files` probed a concrete table version but retained `None` on each handle. Later range reads resolved the latest version again, so an update or delete could make an opened handle fail. ## Fix For nonempty blobs, capture `x-lancedb-version` from the 206 size probe and include it on every later range request. Reject a nonempty probe without a valid version. Accept an empty-blob 416 probe with or without that header, because an empty handle makes no later data range request. The Python remote mock now covers both response shapes. ## Limitation A table dropped and recreated with the same name can reuse version numbers and row addresses. Version pinning does not identify the table itself, so an old handle can still return data from the recreated table. That identity case needs a separate fix. Callers that replace a table should discard its old blob handles. ## Validation - `cargo test --quiet --features remote -p lancedb --lib remote::table::blobs::tests` (24 passed) - Rebuilt the Python extension; `uv run --extra tests pytest python/tests/test_remote_db.py -q -k remote_blob` (6 passed) - `cargo clippy --quiet --features remote -p lancedb --lib --tests --examples` - `cargo fmt --all -- --check`, `ruff check .`, and `ruff format .` Fixes #4276 <!-- lance-gatekeeper-fix:v1 agent=e309bdf75b1a87f6492a6dc3ee1bfd81 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
  brekenandbreken-ai
|
c119f0c8e5 |
fix(python): rank each result list separately in RRF multivector rerank (#4325)
## What's wrong
`RRFReranker.rerank_multivector` concatenates every vector result list
and passes the concatenation to `rerank_hybrid` as a single list.
`rerank_hybrid` then scores each row by its position in that
concatenation, so a row's rank depends on how many rows came before it
in *earlier* lists.
With two searches of 3 rows each (`K=60`):
| row | rank in its own list | score today | RRF score |
|---|---|---|---|
| 1 (top of list 1) | 1 | 1/61 | 1/61 |
| 3 (third of list 1) | 3 | 1/63 | 1/63 |
| 4 (top of list 2) | 1 | **1/64** | 1/61 |
| 5 (second of list 2) | 2 | **1/65** | 1/62 |
So the best hit of the second search ranks below the worst hit of the
first, and the order of the lists passed in decides the fused ranking.
`MRRReranker.rerank_multivector` already scores each list on its own
("MRR semantics require treating each vector result as a separate
ranking system"). RRF is defined the same way.
## Fix
Score each result list with its own ranks (`1 / (rank + K)`, summed
across lists for rows found by more than one), then dedupe the
concatenation by `_rowid`, attach `_relevance_score`, and sort. The
`return_score` handling is unchanged.
## Tests
`test_rrf_multivector_ranks_each_result_list` in
`python/python/tests/test_rerankers.py`:
- On `main`, it fails with `assert 0.015625 == 0.01639344262295082` (the
top hit of list 2 got `1/64` instead of `1/61`).
- With the fix, it passes, and so do the other tests in
`test_rerankers.py` (16 passed, 12 skipped for missing API keys or
models).
- `ruff format --check` and `ruff check` are clean.
The fix was found and written by an AI agent (Breken) and checked
against the tests above.
Co-authored-by: breken-ai <312387581+breken-ai@users.noreply.github.com>
|
||
|
brekenandbreken-ai
|
70a1123404 |
fix(python): write NaN and infinity as SQL literals in table.update (#4326)
## What's wrong
`value_to_sql` renders floats with `str()`, so `float("nan")` becomes
`nan` and `float("inf")` becomes `inf`. SQL reads those as column names:
```python
table.update(where="id = 1", values={"price": float("nan")})
# ValueError: Invalid input, Schema error: No field named nan. Valid fields are id, price.
```
The same happens for `math.inf`, `-math.inf`, `np.nan` and
`np.float32("inf")`, which go through the same float path. The only
workaround today is `values_sql`.
## Fix
Emit `CAST('NaN' AS DOUBLE)`, `CAST('Infinity' AS DOUBLE)` and
`CAST('-Infinity' AS DOUBLE)` for these values. Finite floats render as
before. The cast also lands correctly in `float32` columns.
## Tests
`test_value_to_sql_nan_and_infinity` in
`python/python/tests/test_util.py` updates `float64` and `float32`
columns to NaN, +inf and -inf, then reads them back:
- On `main`, it fails with `ValueError: Invalid input, Schema error: No
field named nan`.
- With the fix, it passes. All of `test_util.py` passes (75 passed, 1
skipped), and so do the update tests in `test_table.py` (`-k update`, 8
passed).
- `ruff format --check` and `ruff check` are clean.
NaN inside a list value (for example a vector) still fails, because
Lance only accepts literals inside array values. This PR covers scalar
columns only.
The fix was found and written by an AI agent (Breken) and checked
against the tests above.
Co-authored-by: breken-ai <312387581+breken-ai@users.noreply.github.com>
|
||
 Mohammad Hijjawi
|
580240278f |
fix(python): apply the default limit of 10 to sync hybrid queries (#4344)
## Summary
A synchronous hybrid query without an explicit `.limit()` can return up
to 20 rows. The documented default is 10, and `AsyncHybridQuery` already
falls back to `DEFAULT_HYBRID_LIMIT` (10).
In `LanceHybridQueryBuilder`, each sub-query (vector and FTS) falls back
to its own default of 10 rows. The fused result was then sliced with
`limit=None`, i.e. not truncated at all. When the two legs match
different rows, the caller gets the union.
t = db.create_table("t", [{"text": ("dog" if i >= 20 else "cat") + f"
{i}", "vector": [float(i), 0.0]} for i in range(40)])
t.create_index("text", config=FTS())
len(t.search(query_type="hybrid").vector([0.0,
0.0]).text("dog").to_arrow())
# before: 20, after: 10
For the same reason, `.offset(n)` without `.limit()` did not raise the
sub-query limits to cover the skipped prefix; that only happened inside
`if self._limit:`.
## Changes
- Use `DEFAULT_HYBRID_LIMIT` when no limit is set, both for the
sub-query limits (`limit + offset`) and for the final slice of the
reranked results, matching `AsyncHybridQuery`.
## Test plan
- [x] New `test_hybrid_query_default_limit` in `test_hybrid_query.py`.
It is the sync counterpart of `test_async_hybrid_query_default_limit`,
plus an `offset()`-without-`limit()` check. It fails before the fix (20
rows) and passes after.
- [x] `pytest python/tests/test_hybrid_query.py
python/tests/test_rerankers.py`
- [x] `ruff format --check` / `ruff check`
I have no local Rust toolchain, so I ran the tests against the released
0.39.0 native module with the changed Python files overlaid. The changed
functions are identical between 0.39.0 and main.
|
||
  
|
1a463f3633 |
fix(python): warn when optimize cleans up all old versions (#4337)
## Why `optimize(cleanup_older_than=timedelta(0))` deletes the data files of every version except the latest. Any other handle still on an older version then fails with a missing-file error, and nothing tells the caller this can happen. #2470 asks for a docs note and a runtime warning. Fixes #2470 ## Scope - `AsyncTable.optimize` emits a `UserWarning` when `cleanup_older_than` rounds to 0 ms or less. `LanceTable.optimize` runs through it, so sync callers get the warning too. - The `cleanup_older_than` docstring on `Table.optimize`, `LanceTable.optimize`, and `AsyncTable.optimize` gets a `.. warning::` block, in the same style as the existing `delete_unverified` one. - Removed the function-local `import warnings` in `AsyncTable.optimize`. `warnings` is already imported at module level, and the local import would shadow it for the new check (`UnboundLocalError`). - `test_optimize` and `test_optimize_delete_unverified` intentionally use a 0 cleanup, so they now assert the warning with `pytest.warns`. Nodejs `cleanupOlderThan` is out of scope, since the issue is filed against the Python SDK. ## Tradeoffs Like the existing `retrain` deprecation warning, the warning is raised inside the async coroutine. For sync callers it is attributed to `asyncio/events.py` rather than the caller's line. Warning from both the sync and async entry points would duplicate the check for a small gain. ## Verification - Repro: a second handle checked out at an older version, then `optimize(cleanup_older_than=timedelta(0))` on the first. The second handle's read fails with `LanceError(IO) ... Not found: .../data/<file>.lance`. Before this change no warning is emitted. After it, the `UserWarning` fires. - New tests `test_optimize_warns_on_zero_cleanup` (async, and checks that `timedelta(days=1)` and the default don't warn) and `test_optimize_warns_on_zero_cleanup_sync` fail on main (`DID NOT WARN`) and pass with the fix. The failing tests are in their own commit before the fix. - `pytest python/tests/test_table.py -m "not slow and not s3_test"`: all pass except `test_merge_insert`. That test's `timeout=timedelta(0)` assertion fails the same way on unmodified main on my machine. - `ruff format --check` and `ruff check` are clean on the changed files. Tests ran against the source tree with the `_lancedb` extension from the matching `0.40.0b11` wheel. The change touches no Rust. This PR was written by an AI coding agent (Claude Code) and reviewed before submission. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Jay Zhou <190699450+jayzhou2309@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
  
|
0327618a62 |
feat(sql): add native vector dedup and partitioned mv refresh (#4303)
## Summary
Materialize a source-preserving dedup result from an indexed, pinned
snapshot with a short declaration. The default greedy policy keeps the
smallest unassigned snapshot row ID and removes only its directly
matching neighbors: For A < B < C, an A–B–C chain without an A–C edge
keeps A and C.
```python
from lancedb import vector_dedup
view = db.create_materialized_view(
"images_clean",
vector_dedup("images", column="phash", distance_threshold=4),
)
cleaned = view.table
```
The Python source expression captures the source version once at
creation and
waits for the materialized result by default. Use `dataset_version=...`
for an
explicit snapshot, `with_no_data=True` for declaration only, or
`create_materialized_view_async(...)` for a job handle. AsyncConnection
supports
the same expression. It uses the existing MV REST endpoints and registry
jobs,
without opening a SQL query connection. Source names are exact
root-namespace
table names. Additional select/where/limit options are rejected for this
source.
The SQL declaration remains available:
```sql
CREATE MATERIALIZED VIEW images_clean AS
SELECT * FROM vector_dedup('images', 7, 'phash', 4);
REFRESH MATERIALIZED VIEW images_clean;
```
This is the host-facing SQL entry point. The MV builder now accepts a
typed native vector source; the Rust SDK also supports the same
definition through `MaterializedViewDefinition::from_sql` and
`prepare_definition`; `view.refresh().execute()` executes it locally.
Hosts distribute its units using their existing job infrastructure. The
raw `vector_duplicate_pairs` source remains available separately when
callers want the pair relation.
## Implementation and contract
- Use actual Rust Lance `13.0.0-beta.15` (`db53dcf367`); the native
partition primitive computes pairs from index representations without
source-vector aggregation or a Python pair loop.
- Extend partitioned MV refresh with two dependent waves. Physical
segment/partition tasks stream pairs through a bounded external sort,
select direct representatives and write per-source-fragment bitmap
artifacts. Source-fragment tasks start after every selection receipt
completes and stream retained original rows, including blob payloads,
into uncommitted output fragments.
- Reuse normal blob write coercion and row-mask scans, including stable
row-ID translation after compaction. Null vectors, isolated rows and
source deletions are preserved correctly. Conflicting representative
decisions across overlapping index scopes fail before publication.
- Expose `dependency_units()` and `write_refresh_partition_with_inputs`
alongside the existing plan/commit API. Receipts bind
snapshot/configuration/policy/destination generation. Only a complete
accepted set can publish; unfinished units restart, completed units can
be reused. There is no pair-position checkpoint.
- Use MV format 4 for dedup results and format 3 for native pairs;
Python and TypeScript metadata readers recognize both. No source
deletion or source row-ID migration occurs.
Only same-segment/same-partition edges are covered. Quantized scores use
symmetric represented-vector distances. Result materialization requires
V2 data storage and is initially a native MV source, not an ordinary
SELECT table function. Compressed vertex masks stay in memory; edge
sorting spills to local disk. Unreferenced artifacts use normal Lance
retention/cleanup and must be retained for the job's recovery window.
## Validation
AWS Linux validation:
- Native pair/result integration matrix passed: multiple
segments/partitions, exact thresholds, empty output, cross-batch data,
single-process pair parity, source preservation, lost receipts and
publication guards.
- Blob/null-vector/pinned-deletion tests passed with both row-ID modes,
including compaction that changes physical row addresses.
- Representative policy, forced external-sort spill and
conflicting/corrupt artifact tests passed.
- Full MV regression suite: 153 passed, 2 pre-existing ignored. Rust
formatting and Clippy passed.
- Python MV suite: 47 passed; TypeScript MV suite: 11 passed. Native
bindings were rebuilt in project-local environments; formatting, lint
and TypeScript docs passed.
- Rust declaration doctests: 3 passed, including the complete local
dedup example.
- Companion host validation: 41 MV job tests passed (1 pre-existing
ignored), full workspace Clippy passed, and the complete process-level
CLI E2E suite passed (3 tests). Recovery exercises both selection and
materialization waves.
- A real CIFAR-10 pHash run used five actual registry-worker owners,
completed 32 selection + 24 materialization tasks, and retained 49,846
of 50,000 rows with identical original values and image payloads.
Host workflow performance measurements are recorded in the companion
host PR. Cross-platform CI is still running. The previous remote-test
bootstrap failed before Cargo tests with `401 Bad credentials`; the
workflow fix described below is being validated by CI. No deployment or
merge is performed.
## Python source validation
AWS tests cover automatic and explicit snapshots, physical/stable row
IDs, local
and remote creation, sync/job/async-connection APIs, delayed refresh,
strict
options and identifier quoting. The native Python extension was rebuilt
through
the project's uv/maturin backend. Ruff, Pyright and the full MkDocs
API-reference
build passed; both new public symbols are rendered.
A real 50,000-image FEv2 run disabled both Python SQL entry points and
supplied an
unreachable SQL endpoint. The new API still materialized the same 49,846
retained
rows and original image payloads. Three recorded creation/refresh jobs
completed
56 tasks each on all five registry-worker owners, and remained on source
snapshot
3 after source latest advanced to 4. No worker or server wire-protocol
changes
were needed for this client entry point.
Previous CI authentication failure:
https://github.com/lancedb/lancedb/actions/runs/36334679276/job/108663151265.
## Review and CI fixes
- Validate all JSON merge source fields before projection, rejecting
unknown and duplicate names at every struct level. Missing/reordered
fields and ordinary JSON merges remain supported. A checked-in 12-case
integration matrix covers root fields, structs, lists, large lists,
fixed-size lists and map values; rejected writes leave the table version
and row count unchanged.
- Match namespace connection MV method signatures to
`MaterializedViewSource` in all four sync/async methods, fixing the
Python 3.10 import failure from runtime override checks.
- Restore the remote-test script's encrypted-CLI path: use the Actions
token for artifact download and pass `SOPHON_READ_TOKEN` under the
variable the decryption script consumes. The previous workflow omitted
that variable and attempted the private release path, failing with HTTP
401. On the final head, remote CI passes: 1,464 Rust unit tests, the
integration targets and doctests pass (3 unit tests ignored).
AWS validation for these fixes: both review reproducers failed before
their fixes and pass afterward; 55 merge tests, 29 cast tests and the
standalone 7-case admission regression pass. Python 3.10.21 imports
successfully; 90 MV/namespace tests, 14 JSON tests and four real
namespace dedup cases pass. After `794b97a0`, the native binding was
rebuilt again and the JSON suite rerun. Rust formatting/Clippy, Ruff and
Pyright pass. Sophon's locked test compilation passes against the final
SDK source for all seven affected crates. Extended runners, logs and
benchmark artifacts remain local.
Gatekeeper recommends approval on `794b97a0`, and the `breaking-change`
label is set. Final CI: **49 checks passed, 3 expected skips, no
failures or pending checks**, including remote tests, Python 3.10,
minimum dependencies and Windows ARM Rust tests. All wheel builds pass
after retrying the Linux matrix affected by transient AlmaLinux
repository-metadata 404s. All code-review threads are resolved. The PR
remains open for maintainer review and merge.
---------
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
Co-authored-by: Lance Release <lance-dev@lancedb.com>
|
||
 Ayush Chaurasia
|
139382b5f7 | feat(python): pass batch_size and other CrossEncoder kwargs (#4328) | ||
 Dan RammerandClaude Opus 5
|
2216586023 |
feat(query): dedup hybrid search by primary key on MemWAL tables (#4268)
Implementing a PK-based dedup to support WAL in hybrid search. Basically, the existing code just calls a vector query + FTS query and dedups locally using the row ID. Since row ID is unsupported in WAL (currently), we update the algorithm to use the tables PK (which is required in WAL, and uniquely linked to a PK on updates). To enable PK-dedup branch, we've chosen to fallback on error rather than fire an exploratory query. Specifically, we issue the regular hybrid query, using row IDs, and when that fails with "WAL does not support row IDs" we fallback to the PK-based variant, and then cache the fallback for future calls from the same client. The alternative is that we chec if WAL is enabled before issuing the query. This would require a `get_lsm_write_spec` call, slowing down all the existing non-WAL paths unnecessarily. The ultimate goal is to get row ID supported in WAL, but that is pending stable row ID GA and subsequent integration into the enterprise stack. Once that lands, we will rip out the PK-based dedup that is only for WAL-enabled tables and use the existing row ID code. This is the main reason for building this adaptively. I had spec-ed out a refactor so that rather than duplicating code between python and rust we build this in rust and have python call that through PyO3. However, that made the reverse path (ie. ) more difficult. So I think the tradeoff of maintaining duplicate code in the short term for the ease of reverting this in the medium term (hopefully 2-3 months). --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
 Wyatt Alt
|
604b1b583b |
feat: pause and resume jobs from the Python SDK (#4112)
Cancellation is the only job control the SDK exposes, and it is terminal, so a long-running server-side job cannot be parked and picked up again. This adds `pause_job` and `resume_job` to the Rust core connection and the Python bindings (sync and async), posting to the server's `/v1/jobs/pause` and `/v1/jobs/resume` endpoints. A pause parks the job until it is resumed: its workers drain and stop. The outcome strings mirror the server's answers — a job finalizing its results reports `"committing"` and cannot be parked, and a resume before the drain is confirmed reports `"still_pausing"`; both are retried rather than failed. Resuming re-queues the job and its workers pick their work back up from checkpoints. Local connections report the operations as unsupported, like the rest of the jobs API. |
||
 Xuanwo
|
0b18dcdaec |
fix(python): warn instead of failing on local imports in @udf source (#4318)
Since 0.40.0b8 (#4254), `@udf` raises when the packaged source imports a module from a local source tree that is neither shipped with `code=` nor named by a declared pip/conda package. That turned registrations that used to succeed into errors: - Callers that rewrite the packaged source before registering it now fail at decoration. Sophon's datagen integration suite inlines its test-module helpers this way, and `test_registered_udf_source_stands_alone_on_a_worker` fails on 0.40.0b8. - The check cannot know which modules a distribution installs. A declared package whose distribution name differs from its module (for example an internal library installed editable) is reported as local, and there was no way to proceed. This keeps the diagnosis but makes it a `UserWarning`, so registration behaves as it did before #4254 and still tells the author, at registration time, why the worker is likely to fail. |
||
 Taylor
|
a6dcbba49e |
feat(python): add opt-in request batching to TypeSafeReranker (#4316)
Adds opt-in `batch_size=40`: 80 non-null candidates use **2 requests instead of 80**. Default `batch_size=1` preserves the existing payload. Each batched question sees only its document and the shared query; instructions/criteria stay unchanged. Prompts referencing `state.document` need adaptation. Concurrency still limits requests; retries remain SDK-managed. Batch-local IDs and ordered results preserve duplicate documents; invalid IDs/probabilities raise without fallback. Validation: **124 automated tests passed**, plus live search integrations in both modes. A **100-query live subset test of the initial implementation** (6,851 candidate pairs; `jev-1.13.0`, SDK 0.7.1, concurrency 32) passed: unbatched/batched calls **6,851/199**, median **877/351 ms**, p95 **9,446/598 ms**. Hybrid Hit@5/Hit@10 was **83%/89% vs 84%/90%**; vector/FTS results and exact configuration are in the [test report](https://github.com/lancedb/lancedb/blob/typesafe-request-batching/python/benchmarks/typesafe_batching.md). Ruff and MkDocs passed. The simplified batching flow also passed a fresh four-query live check: 292 candidate pairs, with 292 unbatched versus 8 batched calls. Builds on #4209; motivated by [research #7](https://github.com/lancedb/research/pull/7). |
||
 
|
53a2afdffa |
chore: update lance dependency to v13.0.0-beta.12 (#4265)
Update the Rust workspace Lance dependencies and Java lance-core from v13.0.0-beta.8 to [v13.0.0-beta.12](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.12), refreshing Cargo.lock. No compatibility fixes were required; validation passed with `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check`. --------- Co-authored-by: Lu Qiu <luqiujob@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
lancedb-gatefixer[bot]
|
75e71176cb |
fix(python): prevent BlobFile finalizer panics on Tokio workers (#4291)
## Cause `io.RawIOBase.__del__` checks `closed` and calls `close()` during GC. Both PyO3 methods entered `block_on`, which panics when GC runs on a LanceDB Tokio worker. ## Fix Track Python blob handle closure atomically so `closed` is synchronous. `close()` marks the handle closed immediately and, when called on a Tokio thread, schedules the underlying async cleanup without blocking that thread. Reads and seeks after close fail immediately; ordinary callers still wait for cleanup. ## Validation - Rebuilt the Python extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`. - Ran the blob test file and remote blob handle test: 69 passed, 1 skipped. - Ran `cargo fmt --all`, `ruff format .`, and `ruff check .`. Fixes #4283 <!-- lance-gatekeeper-fix:v1 agent=15a7bd545e63dea4b69faa0429f0a684 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
Gatefixer
|
4c7c0a85d9 | fix(remote): bound concurrent blob reads across handles | ||
|
Xuanwo
|
b944055b63 |
feat(python): class Functions with initialization and shipped helper code (#4254)
Stacked on #4253. Remote Python Functions could not share helper code, take typed parameters, or keep per-process state: helpers became `from <module> import ...` lines a worker cannot resolve, every parameter was an `env=` string, and state had to be stashed on imported modules. MMLB's OpenAI preset shows the cost — 24 env vars, a 302-line callable, and a rate limiter duplicated between the scalar and batch variants, one of which never constructed it (ENT-2516). **Class Functions.** `@udf` accepts a class. Each remote instance runs `__init__` once, calls `__call__` for every row or batch it processes (row or batch mode is inferred from annotations as before), and calls `close()` once if defined. Calling the definition locally constructs the class, so unit tests stay ordinary. **Initialization.** The annotated `__init__` parameters become the Function's initialization fields. A binding passes their values beside its column inputs: `fn(text=col("body"), model="small", dimensions=512)`. Values are constants of that binding; another column can bind the same Function version with other values. Types are limited to booleans, integers, floats, strings, and lists or structs of them, which have one unambiguous JSON and SQL spelling. A parameter with a default may be omitted; a null value then takes the default. Initialization and input names must be disjoint because both are keyword arguments of one call. Secrets stay on `EnvVarSecret`. **Helper code.** `code=[package, ...]` ships top-level modules or packages as Python source in a `python_bundle` artifact (a canonical JSON object of path → source), imported normally on the worker. Changing a helper changes the artifact digest and so the Function version; the environment from `pip`/`conda` is reused. Functions and classes defined in `__main__` (notebook cells, scripts) are packaged by source, recursively and dependencies first. An import of a module that lives in a local source tree, is not shipped with `code=`, and names no declared package is now rejected at registration instead of failing on the worker. Closures, lambdas, and nested definitions are still rejected, with the reason. A Function without these features packages byte-identically to before (`python_callable`, same digest, no `initialization` on the wire). Class sources are located through a method's code object, because `inspect.getsource` cannot find a class defined in a notebook cell or doctest. The packaging contract is documented on `udf`. Sophon changes that build and execute these artifacts are in lancedb/sophon; they were verified end to end on a Linux local server (registration → REST, SQL, and materialized-view bindings with different initialization → refresh → query, one construction per instance). |
||
|
Xuanwo
|
3728d41f02 |
feat: carry Function initialization in the canonical model (#4253)
A Function instance is created once with an initialization row (`create(initialization, context)` in the Function Format), but nothing in the client model could say what that row holds or where its values come from, so every source-built Function ran with an empty row and users pushed configuration through `env=` strings instead (ENT-2516). This adds initialization to the canonical wire model without changing a Function's identity: - `FunctionSignature.initialization` lists the fields of the row, in order. It is omitted when empty, so existing signatures and FunctionVersion hashes are unchanged. - `FunctionApplication.initialization` carries a binding's values as JSON. The service validates them against the Function's fields and re-encodes them as Arrow, so this JSON is transport only and never hashed; that is why floats are accepted here while column-input literals keep the Slice 1 domain. - `FunctionBinding.initialization` is the validated one-row Arrow IPC stream (base64) that every instance of the binding is created with. Values belong to the binding, so one Function version can back several columns with different values. - The binding-shape check that guards schema mutations accepts the new field; without it a table holding an initialized binding refused further declarations. Rust and Python share new canonical goldens for an initialized application and binding. The authoring side (class `@udf`, initialization arguments, shipped helper code) is the stacked PR on top of this one. |
||
 Jack Ye
|
8541d6d9df |
feat: add view CRUD APIs (#4236)
A view is a named query a database stores and plans on every read. It
holds no rows, which is the whole difference from a materialized view.
## API
| Verb | Route |
| --- | --- |
| `create_view(name, query, namespace_path)` | `POST
/v1/view/{id}/create` |
| `describe_view(name, namespace_path)` | `POST /v1/view/{id}/describe`
|
| `drop_view(name, namespace_path)` | `POST /v1/view/{id}/drop` |
| `list_views(namespace_path)` | `GET /v1/namespace/{id}/view/list` |
On `Connection` and the `Database` trait, with the remote client, Python
(sync and async) and Node bindings. Local databases return
`NotSupported`: the server side is Sophon's, where a view is an object
of the database manifest.
`ViewDescription` carries the defining query, the database *and
namespace path* unqualified names in it resolve against, and the schema
the query resolved to. `create_view` returns one, so a caller has the
schema without a second call.
Both defaults travel with the view because it outlives the session that
declared it: the server re-plans the stored query on every read, so a
reader resolving an unqualified name against its own defaults would read
a different table. `default_namespace_path` crosses the wire as
`default_namespace`, a path like `namespace`, absent for the root.
There is no replace: a name already taken is an error, and changing a
view is a drop followed by a create, each authorized against what it
actually touches.
Querying a view stays SQL's job. There are no rows behind a view, so
there is no `open_view` returning a `Table`.
|
||
 Rudra Prasad Bhuyan
|
dd2539c2ca |
fix(python): convert objects for JSON fields (#4211)
## Summary Closes: #4060 - Convert Python dict/list objects to JSON strings during ingestion. - Support JSON fields nested inside structs and lists. - Cover `add` and `merge_insert` JSON ingestion paths. - Add regression coverage for nested struct and list JSON fields. ## Testing - `git diff --check` - Python compilation passed. - Focused pytest was attempted but could not complete because the native extension build stalled during `uv` bootstrap. |
||
|
Wyatt Alt
|
254688df96 |
feat: group a materialized view by GROUP BY (#4222)
A view could only map source rows one to one, or one to many through a FROM-position item, so nothing could see all the rows sharing a key. A view query now takes `GROUP BY expr, ...` with aggregate projections, planned and executed by DataFusion over the lance scan. A group spans fragments, so a grouped view is recomputed in full whenever its source changes; each row's provenance is its group's smallest source row id. Such a query is stored as definition format 2, so a reader that predates grouping reports the view as unrefreshable instead of failing to parse it. Stacked on #4190. |
||
|
Wyatt Alt
|
01ee01dbc8 |
feat: define a materialized view by its query, with Functions in FROM position (#4190)
A view definition was a structured record under a `kind` tag, one kind per query shape, and a Function returning `list<struct>` was about to add a third. That names shapes instead of describing a relation. A materialized view is now a relation defined by a query, stored as one canonical SQL string under a format number: ```sql SELECT columns FROM [ns.]table [, function(args) AS alias | , UNNEST(column) AS alias] [WHERE predicate] [LIMIT n] ``` Any other clause is refused at parse time. Older readers report the view as unrefreshable, the pre-format layouts still read, and a legacy view is rewritten on its next refresh that commits, rebuilt only where its raw text meant something else under lance's parser. A Function in FROM position yields one row per element it returns, as a table function does in any dialect. The server stages its list output in a hidden table and records that binding beside the query, which stays as the user wrote it; refresh scans the staging and unnests the column, the same operator as UNNEST over a list column the table already holds. Row ids repeat per element, so eviction and incremental append are unchanged. A local database refuses a Function in FROM position, since it has no executor. |
||
 Hongzhu YiandXuanwo
|
7955c50929 |
fix(python): decode file URIs in OpenCLIP (#4131)
## What Decode the path component of `file://` image URIs before passing it to Pillow. ## Why `Path.as_uri()` percent-encodes characters such as spaces. Passing `parsed.path` directly to Pillow therefore tries to open a literal `%20` path and fails. ## Testing - Added a regression test that opens an image whose local filename contains a space. - Verified the focused URI conversion behavior against the changed method. - Ruff check, formatting check, and `compileall` on both changed files. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Wyatt Alt
|
5231d37f8d |
feat: list a table's per-row Function errors from the client (#4208)
A refresh running under a skip policy records each row it skipped, with the failing input and the error, but the client could not read that store: the server exposes it over SQL and, since recently, a REST route. A user who hit per-row failures still had to open a SQL session. `Table::function_errors` calls the route. The listing is table-addressed with optional job and column filters, the same addressing the SQL surface uses, so the two cannot disagree about what a table's errors are. The two non-record signals come back as their own fields rather than as rows: capped-fragment summaries, and whether the listing stopped at its limit. Local tables refuse rather than answer with an empty list. Python and Node expose the same call, with the same optional filters. |
||
|
Xuanwo
|
c72931b30f |
feat(python): add TypeSafe reranker (#4209)
Adds `TypeSafeReranker`, which reranks vector, FTS, and hybrid results with the [TypeSafe System One API](https://docs.typesafe.ai/introduction). Each result is scored independently: TypeSafe reads `{"query", "document"}` and answers a yes/no (noul) question, and the probability of yes becomes `_relevance_score`. Unlike listwise LLM rerankers, the score is an absolute probability, so it is comparable across queries and can be thresholded. The question's `instructions` and `true`/`false` `criteria` are configurable, since domain-specific criteria are what make this kind of scoring work well ([TypeSafe's re-ranking cookbook](https://docs.typesafe.ai/cookbooks/rerank_typesafe)). The API takes one state per request, so the reranker sends one request per result on a thread pool bounded by `max_concurrency`. It deliberately does not use the background event loop: rerankers are called synchronously from inside the async query APIs, where `LOOP.run` would deadlock. TypeSafe scores for the same pair vary slightly between calls, so results with close scores can swap places when a search is repeated. The shared reranker test helper now takes `deterministic=False` for this case: it still checks result sizes and descending scores, but not that two identical searches return the same order. The SDK is imported only when the client is created and questions are sent as plain dicts, so the new tests run in CI with a fake client and without `typesafe-sdk` installed. The live-API test is skipped without `TYPESAFE_API_KEY`. Ranking quality has not been compared with other API rerankers. |
||
 Jonathan HsiehandClaude Opus 5
|
60a1b4c219 |
feat(secrets): named Secrets, bindings, and namespace addressing (#4150)
Adds the client half of database-scoped named Secrets: a Secret is a
name and
an opaque value stored by the service, and a Function binds one to the
environment variable its library already reads. Secrets are addressed by
a
namespace path plus a name.
The UDF body is unchanged and stays portable — it reads `OPENAI_API_KEY`
the
way it always did, and the binding is what puts a value there:
```python
db.create_secret("openai-prod", os.environ["OPENAI_API_KEY"])
function = db.create_function(
analyze_caption,
secrets=[
EnvVarSecret(secret_name="openai-prod", env_variable="OPENAI_API_KEY")
],
)
function.secret_bindings # the Secret's name, never its value
```
- `create_secret` / `alter_secret` / `list_secrets` / `describe_secret`
/
`drop_secret` on sync, async and remote connections, with the pyo3
binding
and the Rust client behind them. Each takes `namespace_path`
keyword-only,
defaulting to the root.
- **There is no read API, by construction rather than by policy** — no
code
path returns a stored credential, and `describe_secret` answers with
metadata only.
- `EnvVarSecret` is a pure local constructor: it contacts no server, so
it
cannot fail on a Secret that does not exist. It exists so that a bare
string
in that position — which would be a credential — is a `TypeError` rather
than a plausible-looking mistake that reads identically in a diff.
- `create_function(..., secrets=[...])` carries the bindings as
`secret_bindings`: a list of `SecretBinding` tagged by `kind`, so a
later
delivery mode is a variant rather than a sibling field. The value never
travels — it is resolved by the service when the Function runs, which is
what
lets a rotation reach columns already pinned to an older
FunctionVersion.
- A binding names its Secret as a `SecretReference` of `{name,
namespace_path}`
rather than one joined string, so no delimiter has to be excluded from
every
name and segment forever, and `ClientConfig.id_delimiter` cannot
contradict
an identity built on a fixed separator.
- A root namespace is omitted from the request body rather than sent
empty, so
a root request is byte-identical to one from a client that predates
namespaces. Tests pin it.
This is the client surface the design's §4 describes; the service side
lives in
sophon.
**Previously split across two PRs.** Namespace addressing was #4151,
stacked on
this one; it is folded in here so the Secret identity contract — name,
namespace path, and the binding that carries both — is reviewable as one
piece
rather than as a shape introduced and then replaced.
## Identifier safety, merged from #4189
**#4189 is merged into this branch**, so the client half of Secrets and
the
guards on the identity it puts in the URL are one PR. What it added:
- Components are checked where the identifier is built, before a request
is
constructed. `create_secret("../jobs", value)` no longer resolves to
`/v1/jobs/create` and delivers a credential-bearing body to a route with
none
of this one's body suppression.
- Each component is percent-encoded and joined by the delimiter, so
nothing
inside a component can end the path segment or add one.
- A component may not be empty, a relative segment (`.`, `..`, and their
`%2e`
spellings), or the delimiter itself — the three ways a component erases
a
boundary the split has to recover. `["prod", ""]` joined to `prod$`,
which
reads back as `["prod"]`.
- `$` is the only accepted `id_delimiter`, refused at client
construction.
`ClientConfig.id_delimiter` remains, since the identifier grammar comes
from
the Lance REST catalog standard, but a value that would produce
identifiers no
service splits the caller's way is now an error where it was written.
- One `build_object_identifier` and one character set serve tables,
namespaces,
Secrets, Functions and materialized views.
Components are checked for *addressability*, not a character set: the
name's
own grammar stays each object's own, so a catalog database keeps the `/`
that
`RemoteCatalog::validate_name` allows.
## Known shortcoming
`secret_bindings` is omitted from a registration body when empty, so a
client
that binds nothing sends what a client without bindings sends. When a
client
does bind a Secret and the service does not know the field, the field is
ignored: registration succeeds, the returned version carries no
bindings, and
the Function fails at execution with the variable unset, far from the
call that
asked for it.
`ServerVersion` is how this codebase refuses a feature the service is
too old
for, and it gates five features already. It does not gate this one: it
is held
per table, and registering a Function is a database-level call. Noted at
the
field in `remote/db.rs`; wiring the gate is follow-up work.
**Tests:** lancedb lib 1340 passed, `first_class_function_slice1` 9,
`first_class_function_slice2` 3, plus Python tests across both slices.
Rebased onto `main` after #4176 (OCI Function identity), #4191 (`.`/`..`
table
names) and #4195 (remote catalogs).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01UfmeJ533rQDnPBkMtjerV6
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Jack YeandXuanwo
|
6a07f88980 |
feat: add remote catalogs and Python and TypeScript bindings (#4195)
Add a `Catalog` trait and `RemoteCatalog` for managing databases,
exposed through Rust, synchronous/asynchronous Python, and TypeScript. A
remote catalog represents the server's root namespace, and each database
is one child namespace. Create/connect return ordinary LanceDB
connections, so existing table APIs work unchanged.
## Rust API
`Catalog` is an object-safe async trait with `create_database`,
`connect_database`, `list_databases`, and `drop_database`. Backend
create/connect methods return `Arc<dyn Database>`; the public
`CatalogConnection` wraps them as `Connection` values and shares its
embedding registry with those connections. `RemoteCatalog` implements
the trait; `connect_catalog` is the convenience builder, available with
the `remote` feature.
```rust
use lancedb::catalog::{
CreateDatabaseRequest, DropDatabaseRequest, ListDatabasesRequest,
};
let catalog = lancedb::connect_catalog("https://my-server.example")
.api_key("my-api-key")
.execute()
.await?;
let db = catalog.create_database(
CreateDatabaseRequest::new("analytics").exist_ok(true),
).await?;
let connected = catalog.connect_database("analytics").await?;
let page = catalog.list_databases(
ListDatabasesRequest::default().limit(20),
).await?;
// page.databases: Vec<String>; page.page_token: Option<String>
catalog.drop_database(
DropDatabaseRequest::new("analytics").ignore_missing(true),
).await?;
```
Create/drop also accept a plain name for default behavior, e.g.
`catalog.create_database("analytics").await?`. Existing names fail
creation unless `exist_ok` is enabled; missing names fail drop unless
`ignore_missing` is enabled. Drop always requires an empty database.
## Python API
```python
import lancedb
catalog = lancedb.connect_catalog(
"https://my-server.example", api_key="my-api-key"
)
db = catalog.create_database("analytics", exist_ok=True)
connected = catalog.connect_database("analytics")
page = catalog.list_databases(limit=20)
# page.databases: list[str]; page.page_token: Optional[str]
if page.page_token is not None:
next_page = catalog.list_databases(limit=20, page_token=page.page_token)
catalog.drop_database("analytics", ignore_missing=True)
```
`connect_catalog` returns `Catalog`; create/connect return the existing
`DBConnection` API. The async equivalent is `catalog = await
lancedb.connect_catalog_async(...)`, returning `AsyncCatalog`; await
each of the same four methods, with create/connect returning
`AsyncConnection`.
## TypeScript API
```typescript
import { connectCatalog } from "@lancedb/lancedb";
const catalog = await connectCatalog("https://my-server.example", {
apiKey: "my-api-key",
});
const db = await catalog.createDatabase("analytics", { existOk: true });
const connected = await catalog.connectDatabase("analytics");
const page = await catalog.listDatabases({ limit: 20 });
// page.databases: string[]; page.pageToken?: string
if (page.pageToken !== undefined) {
const nextPage = await catalog.listDatabases({
limit: 20, pageToken: page.pageToken,
});
}
await catalog.dropDatabase("analytics", { ignoreMissing: true });
```
Create/connect return the existing `Connection` API. All four methods
are asynchronous.
## REST mapping
All paths below are relative to the catalog endpoint. `{name}` is the
logical database name encoded as one URL path component. The default
namespace delimiter is `$`, so the root identifier is encoded as `%24`.
| Catalog operation | Existing REST route | Request |
| --- | --- | --- |
| `create_database(name)` | `POST /v1/namespace/{name}/create` |
`{"mode":"Create"}`; `exist_ok=true` sends `{"mode":"ExistOk"}` |
| `connect_database(name)` | `POST /v1/namespace/{name}/describe` |
`{}`; verifies existence before returning a scoped connection |
| `list_databases(...)` | `GET /v1/namespace/%24/list` | Optional
`limit` and `page_token` query parameters |
| `drop_database(name)` | `POST /v1/namespace/{name}/drop` |
`{"mode":"Fail","behavior":"Restrict"}`; `ignore_missing=true` changes
mode to `"Skip"` |
For example, database `team/search` uses
`/v1/namespace/team%2Fsearch/create`. A paginated root listing can use
`/v1/namespace/%24/list?limit=20&page_token=a%2Fb`. The list response
retains the existing namespace wire shape,
`{"namespaces":["analytics"],"page_token":"next"}`; the SDK exposes
`namespaces` as `databases` and preserves the opaque continuation token.
An absent or empty token ends pagination. Page limits must be between 1
and 2147483647. Create/drop accept a namespace JSON response or HTTP
204.
Catalog management requests omit both `x-lancedb-database` and
`x-lancedb-database-prefix`, including values supplied through static or
dynamic headers. Returned database connections set `x-lancedb-database`
to the exact logical name and keep independent scope. API keys, OAuth or
dynamic authentication, client settings, table read consistency
settings, and an optional SQL endpoint override carry over to those
connections. OAuth cannot be combined with an API key or a custom header
provider.
For SQL through an HTTPS catalog, configure the existing SQL endpoint
contract with Rust
`.sql_host_override("grpc+tls://sql.example.com:10026")` or Python
`sql_host_override="grpc+tls://sql.example.com:10026"`. TypeScript
catalog options expose the same setting as `sqlHostOverride`. It is
inherited by created/connected databases, retained by Python connection
serialization, and initialized lazily when SQL is executed.
Create HTTP 409 maps to `DatabaseAlreadyExists`; connect/drop HTTP 404
maps to `DatabaseNotFound`, except that `ignore_missing` suppresses a
missing-database drop. Other server errors propagate. The server
enforces restricted deletion; the client never requests cascading
deletion.
Database names preserve literal slashes as part of one name. They must
be nonempty ASCII, with no control characters, surrounding whitespace,
or configured namespace delimiter, and cannot be `.` or `..`. Endpoints
must be HTTP(S) URLs without embedded credentials, query parameters, or
fragments.
## Scope
This PR adds the client API and reuses existing namespace endpoints.
Local catalogs, `__catalog` storage, location generation/sanitization,
and `__manifest` lifecycle support remain deferred; the Lance dependency
is unchanged.
The PR also runs macOS Node tests serially to avoid existing
resource-contention timeouts reproduced across recent main runs.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
|
3b37ea2c7a |
feat: represent registered Functions by OCI image identity (#4176)
Function versions identify independent Function objects and their numeric revisions. Rust and Python expose the object ID, location, canonical decimal version, metadata, and availability separately from the OCI image digest. Computed-column applications carry the complete object reference, so existing bindings retain their identity after a name is removed and reused. Source authoring keeps the existing create_function/create_function_async, job wait, and column-binding APIs. The server coordinates baking followed by registration; users do not have to manage manifest digests to create a Function. The previous stored Function representation is intentionally unsupported. Existing contract tests and shared wire fixtures are migrated to the new model. This SDK change accompanies the final integration layer https://github.com/lancedb/sophon/pull/7887 in the Sophon stack: https://github.com/lancedb/sophon/pull/7885 → https://github.com/lancedb/sophon/pull/7886 → https://github.com/lancedb/sophon/pull/7887. A metadata-only revision can retain the same executable image; Function version numbers must not be used as image cache keys. --------- Co-authored-by: lancedb automation <robot@lancedb.com> Co-authored-by: Yang Cen <bubble-cal@outlook.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
 Will JonesandClaude Opus 5
|
1d2a5d084b |
fix: accept all-null batches and plain JSON strings for json columns (#4067)
Two ways of writing to a `json` column failed or silently corrupted data. **All-null batches were rejected.** `add()` refused a batch whose values for a `json` column were all null, while every plain Arrow type accepted the same batch. This bites row-at-a-time inserts hardest: a one-row batch with no value for an optional column is trivially all-null, so most such writes failed. pyarrow infers `null` as the column's type, and the write path had no handling for it — casting to the table's type dropped the field metadata that identifies the column as `lance.json`, so lance rejected the batch (`` `val` should have type json but type was large_binary ``). A null-typed input column now becomes typed nulls matching the table's field exactly, metadata included. **Unlabelled JSON text was stored raw.** JSON supplied as plain strings (what pyarrow infers for a column of `str`) was cast to the column's `LargeBinary` storage type and relabelled `lance.json`, putting unparsed text where JSONB was expected. Reads returned the text unnormalized and `json_extract` failed with `InvalidJsonb`. Lance-core does the JSONB encoding, but only for input labelled `arrow.json`, so string input is now labelled rather than cast — at the top level and inside structs. Both fixes are in the shared Rust write path, so they apply to any binding, including hand-built Arrow tables that never pass through Python's list-of-dicts type inference. `_align_field` gets the same JSON-string fix for the legacy Python `_sanitize_data` path, which `on_bad_vectors` and embedding functions still route through. The blob v2 half of the issue landed separately in #4065, which added a `DataType::Null` arm to blob coercion. This PR keeps that implementation and adds end-to-end add-path coverage for it. The tests from #4066 are included here and pass, so that PR's Python-layer inference changes are no longer needed to close the issue. Fixes #3759 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jack Ye
|
f12996557f |
feat(remote): support materialized view APIs (#4180)
## Summary
Align the experimental materialized-view HTTP transport with the
equivalent Table API shape and add remote materialized-view support
across Rust, Python, and TypeScript. This is an intentional breaking
change to the experimental materialized-view surface.
Materialized-view creation performs an initial refresh by default. The
create endpoint returns `202 Accepted` with `{ "job_id": "..." }`;
blocking SDK creation waits for that job before returning a populated
view. `with_no_data` / `withNoData` explicitly creates only the
definition and empty backing table.
## Route comparison
| Operation | Materialized-view API | Equivalent Table API |
| --- | --- | --- |
| Create | `POST /v1/materialized_view/{id}/create` | `POST
/v1/table/{id}/create` |
| Describe/open | `POST /v1/materialized_view/{id}/describe` | `POST
/v1/table/{id}/describe` |
| List | `GET /v1/namespace/{id}/materialized_view/list` | `GET
/v1/namespace/{id}/table/list` |
| Refresh | `POST /v1/materialized_view/{id}/refresh` | asynchronous
Table mutation pattern |
| Drop | `POST /v1/materialized_view/{id}/drop` | `POST
/v1/table/{id}/drop` |
Create, describe, refresh, and drop identify the target in the singular
item path instead of duplicating it in the request body. Create and drop
require `202 Accepted` with a valid job ID. List is a namespace-scoped
GET with opaque pagination tokens. The Rust list API now returns view
names, matching Table listing and the existing Python and TypeScript
APIs.
## Python API changes
| Operation | Synchronous API | Asynchronous API | Table/job pattern |
| --- | --- | --- | --- |
| Create and wait | `DBConnection.create_materialized_view(...)` |
`await AsyncConnection.create_materialized_view(...)` | Returns a
materialized-view handle after its initial-population job finishes |
| Submit create | `DBConnection.create_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.create_materialized_view_async(...)
-> AsyncJob[None]` | Matches job-returning Table mutations such as
`create_index_async` |
| Open | `DBConnection.open_materialized_view(...)` | `await
AsyncConnection.open_materialized_view(...)` | Opens the backing Table
plus its definition |
| List | `DBConnection.list_materialized_views()` | `await
AsyncConnection.list_materialized_views()` | Returns names like Table
listing |
| Refresh and wait | `MaterializedView.refresh(...)` | `await
AsyncMaterializedView.refresh(...)` | Returns the typed refresh result
after the job finishes |
| Submit refresh | `MaterializedView.refresh_async(...) ->
Job[RefreshMaterializedViewResult]` | `await
AsyncMaterializedView.refresh_async(...) ->
AsyncJob[RefreshMaterializedViewResult]` | Matches
`Table.refresh_column_async`; remote job handles expose the server job
ID |
| Drop | `DBConnection.drop_materialized_view(...)` | `await
AsyncConnection.drop_materialized_view(...)` | Matches blocking
`drop_table` |
| Submit drop | `DBConnection.drop_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.drop_materialized_view_async(...) ->
AsyncJob[None]` | Matches `drop_table_async`; remote handles expose the
server cleanup job ID |
The materialized-view handle exposes its backing Table through `.table`,
so normal Table query, search, and index APIs apply. Definition lookup
and refresh are backend-aware rather than depending on local schema
metadata. TypeScript exposes the equivalent blocking/job drop pair as
`dropMaterializedView` and `dropMaterializedViewAsync`.
|
||
|
Jack YeandXuanwo
|
56bbd19ff4 |
test(python): cover namespace operations without pylance (#4174)
The "Test without pylance or pandas" CI job only ran `test_table.py`, so the regression fixed in #3606 had no guard: sync namespace operations used to route through the Python `lance_namespace` client, whose `dir` implementation ships in the optional `pylance` extra, so `lancedb.connect(path).list_namespaces()` failed with `No module named 'lance'` while the async API worked. Adds `python/python/tests/test_namespace_no_pylance.py` and runs it in that job. Its `without_pylance` fixture blocks `lance` imports, so the guard also fires in environments that do have `pylance` installed. Coverage: the original reproducer, nested namespace lifecycle, namespaced and root table lifecycle, the async path, and the one API that legitimately still needs `pylance` (`namespace_client()`). Verified the guard actually catches the regression: against `lancedb==0.33.0`, three of these tests fail with the original error; against a fixed build all pass. Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Wyatt Alt
|
0665575a76 |
feat: recompute computed column rows whose inputs changed (#4161)
refresh_column fills nulls, so once a row has a value nothing revisits it: an update to one of its inputs, or a definition change, leaves the computed value stale for good. This stamps the column's field metadata with the definition it was computed under and a per-fragment signature of the input storage it was read from (input data files and overlays; not the deletion file, since a delete changes no surviving value). A refresh recomputes every live row of a fragment whose stamp disagrees with the manifest, then records what it computed from in a second commit after the fill. A compacted fragment inherits freshness through the Rewrite lineage when every fragment it was built from was signed, or was appended since the stamp, never had an input moved, and left its rows of the product unfilled (a raw append may supply a value; the product's data is the evidence, and the null fill covers those rows); otherwise it recomputes. A column declared before the stamps existed keeps the null-fill contract on its first refresh, which enrolls it as it stood. The map is one entry per fragment per column, so it is kept out of the manifest: each stamp writes an immutable sidecar under `_computed/`, named by its content digest, and the field metadata holds the digest. Pruning old versions also drops the sidecars no remaining version references, keeping any younger than seven days as lance keeps unverified files, since a sidecar is put before the commit that references it. The stamp commit is metadata-only, so a materialized view's drift check treats it like the fill. The core lives in `table::freshness` so a remote refresh can share the contract. |
||
|
Xuanwo
|
255da8a854 |
fix(python): resolve native job types in API docs (#4170)
Native job metadata types report `builtins` as their module, so Griffe cannot resolve the public `lancedb.job` re-exports and the Python API reference build fails. Set the PyO3 module metadata for `JobInfo`, `JobDescription`, and `JobFailureInfo`, and cover import resolution in the existing package metadata tests. Reproduced the failure and validated the fix with the docs CI toolchain (`griffe==0.49.0`, `mkdocstrings==0.25.2`, and `mkdocstrings-python==1.10.9`). After rebuilding the native extension, the full `PYTHONPATH=. mkdocs build` succeeds and all three classes and their public members appear in the generated reference. |
||
|
Jack Ye
|
9fe10c7362 |
fix(remote): align Function CRUD routes (#4166)
Align the experimental Function HTTP transport with the equivalent Table
CRUD API shape. This is an intentional breaking change to the
experimental Function routes; public Rust and Python APIs remain
unchanged.
## Route comparison
| Operation | Function before | Function after | Equivalent Table API |
| --- | --- | --- | --- |
| Create | `POST /v1/functions/create` | `POST /v1/function/{id}/create`
| `POST /v1/table/{id}/create` |
| Describe | `POST /v1/functions/describe` | `POST
/v1/function/{id}/describe` | `POST /v1/table/{id}/describe` |
| List | `POST /v1/functions/list` | `GET
/v1/namespace/{id}/function/list` | `GET /v1/namespace/{id}/table/list`
|
| Drop | `POST /v1/functions/drop` | `POST /v1/function/{id}/drop` |
`POST /v1/table/{id}/drop` |
## Contract details
- Create, describe, and drop use a singular resource path. Their `{id}`
path parameter is the URL-encoded Function name, and the duplicate
Function identifier is removed from each request body.
- Create continues to accept `202 Accepted`.
- List changes from a POST with a JSON body to a namespace-scoped GET.
Its `{id}` path parameter is the namespace identifier rather than a
Function name.
- Functions do not support nested namespaces yet, so the client lists
against the root namespace identifier (`$` with the default delimiter).
A non-root namespace is rejected.
- The optional list filter is named `name`. `limit`, `page_token`, and
`include_definition` remain available as query parameters.
- The paginated list response shape is unchanged.
|
||
|
Will JonesandClaude Sonnet 5
|
1da5876870 |
ci: add spell checking (#4148)
Adds [typos](https://github.com/crate-ci/typos) as a CI check and pre-commit hook, the same way Lance does it, so misspellings like the ones fixed in #4146 get caught automatically going forward. This also fixes the misspellings `typos` found across the repo (Rust, Python, TypeScript source, comments, and generated docs), and adds a small `.typos.toml` with `extend-words` entries for terms that are correct but look like typos: `AKS` (Azure Kubernetes Service), `RabitQ` (a real quantization algorithm name), `mmaped` (the actual name of a `candle-core` API we call), and `Writeable` (from Python's `_typeshed.WriteableBuffer`). Third-party license files are excluded. Fixes #4147 Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
 Madan Kumar
|
577fb48376 |
fix(python): apply offset when combining async hybrid results (#4028)
Fixes #4027 ## Summary `AsyncHybridQuery` (`table.query().nearest_to(...).nearest_to_text(...)`) paginates incorrectly when `.offset()` is used: the second page repeats rows from the first page and silently drops others. `offset()` on a hybrid query pushes the offset down into *both* sub-queries (`HybridQuery::offset` in `python/src/query.rs` forwards to `inner_vec` and `inner_fts`), so each sub-query independently skips its own first `offset` rows before the results are fused. `AsyncHybridQuery.to_batches` then called `_combine_hybrid_results(..., limit=self._inner.get_limit())` without an `offset`, so the reranked table was sliced starting at position 0 and the sub-query limits were never raised to cover the skipped prefix. On the 4-row fixture in `test_hybrid_query.py`, with `_rowid` ordering `[3, 0, 2, 1]`: | query | before | after | | --- | --- | --- | | `.limit(2)` | `[0, 3]` | `[0, 3]` | | `.offset(2).limit(2)` | `[3, 1]` | `[2, 1]` | Row `3` was returned on both pages and row `2` was never returned at all. This is the async counterpart of #3769 (`Fixes #3765`), which fixed the same bug in the synchronous `LanceHybridQueryBuilder`. #3765 explicitly deferred the async path; this PR closes that gap and reuses the `offset` parameter that #3769 already added to `_combine_hybrid_results`. The synchronous path is unaffected — it was fixed in #3769. ## Changes `python/python/lancedb/query.py`, `AsyncHybridQuery.to_batches`: - Each sub-query now fetches `limit + offset` rows and its own offset is reset to 0, so the fused result contains the full prefix the window is sliced out of. - The combined, reranked table is sliced with `offset=` instead of always starting at 0. Both halves are needed: raising the sub-query limits without the final slice still returns page 1, and slicing without raising the limits still misses rows. `nodejs` has no equivalent hybrid combine path, so there is no SDK parity gap here. ## Test plan - [x] New regression test `test_async_hybrid_query_offset` in `python/python/tests/test_hybrid_query.py`, mirroring the sync `test_hybrid_query_offset`. It asserts the offset window is a suffix of the un-offset result *and* that page 1 + page 2 together cover every row exactly once (a row-count-only assertion would pass even with duplicates). - [x] `pytest python/tests/test_hybrid_query.py` — 16 passed - [x] `pytest python/tests/test_rerankers.py` — 9 passed, 11 skipped - [x] `pytest python/tests/test_query.py` — 86 passed - [x] `pytest --doctest-modules python/lancedb/query.py` — 13 passed - [x] `ruff format --check` / `ruff check` — clean --- ## Scope, after review @lancedb-gatekeeper raised three points. Two were mine and are fixed in `04d07c2`; the third is deliberately left alone and I'd like a maintainer's call on it. **Fixed — effective limit was read from the FTS child only.** `HybridQuery::get_limit()` (`python/src/query.rs:1159`) returns `self.inner_fts.inner.current_request().limit`, so an FTS-first hybrid with no explicit `.limit()` yielded `None`, skipped the widening branch and passed `limit=None` to the combiner — returning the union of both candidate lists instead of the documented default of 10. The limit is now derived from both children with a `DEFAULT_HYBRID_LIMIT = 10` fallback, so construction order no longer matters. **Fixed — `explain_plan()` / `analyze_plan()` described a different query than the one that ran.** Both built their children straight from `self._inner`, bypassing the limit/offset rewrite in `to_batches`, and reported `skip=2, fetch=2` while execution used `skip=0, fetch=4`. Child preparation now lives in one `_create_child_queries()` helper used by all three. > **Visible change to `explain_plan()` output:** because the plan is now built from the real execution children, which carry `with_row_id()`, the two `ProjectionExec` lines gain a `_rowid` column. The doctest is updated to match. This is the diagnostic becoming truthful rather than the assertion being weakened — it is still an exact-match comparison. **Not fixed here — RRF candidate-pool invariance.** Widening each sub-query to `limit + offset` does change the candidate pool between page requests, so the fused ranking can shift and pagination can still repeat rows. That's a real problem, but it is exactly what the merged sync path does today: ```python # LanceHybridQueryBuilder (sync), merged in #3769 sub_query_limit = self._limit + (self._offset or 0) ``` Making the pool invariant means choosing a contract — a fixed candidate pool, or an explicit cursor — and that ought to apply to sync and async together rather than letting the two paths diverge. I've asked in the review thread which way you'd prefer, and I'm happy to do it here or in a follow-up covering both paths. So, to be precise about what this PR delivers: it makes `.offset()` take effect on the async hybrid path and makes the diagnostics honest. It does not make hybrid pagination stable across pages under reranking — that needs the contract decision above. |
||
|
LanceDB RobotandJack Ye
|
3e3878b223 |
chore: update lance dependency to v12.0.0-beta.15 (#4143)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java lance-core from v12.0.0-beta.14 to [v12.0.0-beta.15](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.15). No compatibility fixes were required; `cargo clippy --quiet --workspace --tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed. --------- Co-authored-by: Jack Ye <yezhaoqin@gmail.com> |
||
|
Jack Ye
|
21f11b4463 |
feat!: replace get_job/job_history with describe_job/query_job_events (#4130)
A 1M-row column refresh over 200 fragments produced no visible result,
and the client could only ever say `"running"`. Everything needed to
diagnose it already existed server-side — the job registry records a
`claim`/`claim_complete` pair per fragment carrying `rows_processed` —
but none of it was reachable.
## Before
Four ways to ask about a job, none of which told you much.
```python
job = table.refresh_column_async("embedding")
job.status() # "running". That was the entire debug surface.
db.get_job(job_id) # state, and a spec. No result, no progress.
db.job_history(job_id) # raw record batches, no limit, no filter
db.job(job_id) # a handle that knew nothing
```
## After
Open a job the way you open a table; the handle answers everything.
```python
job = db.open_job(job_id) # raises JobNotFoundError if there is no such job
```
```python
>>> print(job)
Job(
id='job-1',
state='failed',
job_type='refresh_column',
creation_ms=1757000000000,
spec={
"column": "embedding",
"num_workers": 4
},
failure=JobFailureInfo(phase='execute', message='worker died', retryable=True),
)
```
Individual fields are there too — `job.state`, `job.job_type`,
`job.creation_ms`, `job.spec`, `job.result`, `job.failure` — and
`job.result` carries `rows_assigned` / `rows_failed` as soon as the job
succeeds, with no `wait()` required.
Per-fragment progress *while it is still running*:
```python
done = job.events(filter="state = 'claim_complete'", limit=10_000)
done.column("rows_processed").to_pylist() # [5000, 5000, ...]
```
The handle an async action returns is the same object, one `refresh()`
away:
```python
job = table.refresh_column_async("embedding")
job.refresh()
job.state, job.result
```
TypeScript is the same experience, down to `console.log`:
```ts
const job = await db.openJob(jobId); // rejects if there is no such job
console.log(job); // same multi-line layout
job.state; job.jobType; job.spec; job.result; job.failure;
const done = await job.events({ filter: "state = 'claim_complete'", limit: 10_000 });
```
## Why each piece matters
- **A result without waiting.** `rows_assigned` / `rows_failed` used to
live only on the terminal result, so a job that never terminated
reported nothing at all.
- **`limit`.** The server caps event rows at 1000 and truncates without
saying so, which silently hid most of a 200-fragment job's history.
- **`filter`.** `claim_complete` rows carry per-claim `rows_processed` —
the only progress signal that exists mid-flight.
- **Events outlive the worker.** They live in the job registry, not in
pod logs that vanish with the pod.
- **One place to ask.** `open_job` replaces `describe_job`,
`query_job_events` and `job`, so a question about a job has one answer
instead of one per calling location.
- **A missing job is an error, not a `None`.** The common case is a job
id copied out of a log, where absence is the surprise worth raising —
and it matches `open_table`.
- **Printing is the debug surface.** Every field on its own line, JSON
payloads keeping their structure. An unrefreshed handle stays on one
line, because there is nothing to lay out.
- **In-process jobs say so.** A local refresh reports `state` and leaves
the rest null rather than inventing fields it has no record for.
`list_jobs` and `cancel_job` stay as they were: one lists, the other is
a one-shot action that should not need a describe first.
## Breaking
All shipped in 0.38.0. No deprecated aliases.
| Was | Now |
| --- | --- |
| `Connection.get_job` → `describe_job` | `Connection.open_job` returns
a populated `Job`, or raises |
| `Connection.job_history` → `query_job_events` | `job.events(...)` |
| `Connection.job` | `Connection.open_job` |
| Python events → `List[pa.RecordBatch]` | `pa.Table` |
| `JobDescription.spec_json` / `.result_json` | internal; use `job.spec`
/ `job.result` |
Node's `Job` is now a TypeScript class wrapping the native handle, so it
returns an Arrow table and parsed values like Python does. New
`Error::JobNotFound` / `JobNotFoundError`; the three job exceptions are
now in the Python API reference.
|
||
 Drew
|
8c9c5c5a5f |
fix: stop enabling stable row ids on blob table create (#4126)
This PR stops blob table create from implicitly enabling stable row ids. A blob schema still selects Lance file format 2.2, but row id behavior stays with the table config. Compact then fetch with a `_rowid` captured before compaction is still not supported on a default table. That needs `take` to remap row addresses through blob reuse rather than making stable row ids a blob-table default. BREAKING CHANGE: blob create no longer enables stable row ids. A blob schemastill selects Lance file format 2.2. Fetch uses `_rowid` on HEAD. Held ids survive compact only when the table has stable row ids. ## Testing * `cargo fmt --all` * `ruff format .` * `ruff check .` * `cargo clippy --quiet --features remote --tests --examples -p lancedb` * `cargo test --quiet --features remote -p lancedb --test blob_integration` * `python/.venv/bin/pytest python/python/tests/test_blob.py -q` |
||
|
Jack Ye
|
aab23eb39e |
feat: support nullable named function outputs (#4123)
Supports fully nullable named Function outputs while preserving the
distinction between a valid all-null struct and a null/unassigned
result.
## Concrete example
This UDF contract is now valid:
```python
@udf(
input_schema=pa.schema([
pa.field("text", pa.string(), nullable=False),
]),
output_schema=pa.schema([
pa.field(
"embedding",
pa.list_(pa.float32(), list_size=1024),
nullable=True,
),
pa.field("embedding_failure_reason", pa.string(), nullable=True),
pa.field("embedding_failure_code", pa.int32(), nullable=True),
]),
)
def embed(text):
...
```
A successful row can return:
```text
embedding = [0.12, ...]
embedding_failure_reason = NULL
embedding_failure_code = NULL
```
If remote inference still fails after retries, it can return:
```text
embedding = NULL
embedding_failure_reason = "HTTP 429: rate limited"
embedding_failure_code = 429
```
An all-null but valid result struct is also assigned; it is not mistaken
for unfinished work.
## Binding shapes
- Mapping the result to one output column stores the `StructArray`
directly, including its parent validity bitmap.
- Flattening the result into top-level columns stores the parent
validity in a reserved internal nullable Boolean assignment column that
is not part of the UDF result mapping.
- An outer null struct remains unassigned/skipped. A valid struct
remains assigned regardless of which child fields are null.
- Scalar Function outputs remain non-nullable.
The contract is preserved through Python registration, Rust application
planning, persisted `FunctionBinding` metadata, schema revalidation, and
Enterprise execution.
|
||
|
Jack Ye
|
e639b1b650 |
feat: add asynchronous remote SQL queries (#4070)
## Summary
Add SQL execution to remote LanceDB connections. On the standard
synchronous connection, `execute_query` waits for the initial result
stream and returns its Arrow reader. `execute_query_async` is called
without Python `await` and immediately returns a query handle for status
inspection, streaming, or cancellation. Local databases report that SQL
is not supported.
The transport and query lifecycle live in Rust. Python exposes
native-backed synchronous and asynchronous connection methods and query
wrappers; it does not use PyArrow's Flight client.
## User experience
The standard synchronous connection supports both direct reads and
background query execution:
```python
db = lancedb.connect(
"db://analytics",
api_key="ldb_...",
sql_host_override="grpc+tls://sql.example.com:10026",
)
# Direct execution waits only until the initial result stream is available.
# Later batches continue streaming as the query progresses.
reader = db.execute_query(
"SELECT * FROM events",
default_namespace_path=["production"],
)
for batch in reader:
print(batch.num_rows)
# Background execution returns a query handle immediately. Despite the
# `_async` suffix, no Python `await` is needed on a synchronous connection.
query = db.execute_query_async("SELECT * FROM events")
print(query.id)
description = db.describe_query(query.id)
print(description.status)
print(description.progress)
print(description.expires_at)
# Start reading as soon as the service advertises partial results. The reader
# continues polling and yields newly available record batches until the query
# and all result endpoints are complete.
reader = query.reader()
for batch in reader:
print(batch.num_rows)
# Or cancel a different still-running query. Its status becomes "cancelling"
# while the server is still working, then "cancelled" once confirmed.
cancelled_query = db.execute_query_async("SELECT * FROM large_events")
cancelled_query.cancel()
```
The less commonly used asynchronous connection exposes the same
operations as coroutines:
```python
async_db = await lancedb.connect_async(
"db://analytics",
api_key="ldb_...",
sql_host_override="grpc+tls://sql.example.com:10026",
)
query = await async_db.execute_query_async("SELECT * FROM events")
async for batch in await query.reader():
print(batch.num_rows)
```
The UUIDv7 query id is scoped to the connection that submitted it. The
connection retains lightweight shared query state used by
`query.describe()` and `db.describe_query(query.id)`; the id does not
encode SQL or a Flight continuation token and is not a cross-connection
resume token. Abandoned state has bounded retention, and terminal state
remains available briefly.
Unqualified table names use the connected database and the `public`
namespace by default. `default_namespace_path` accepts a list such as
`["production", "events"]`. SQL can still use qualified names to
reference other databases and namespaces available to the deployment.
## Design
- Uses Arrow Flight `PollFlightInfo` for submission and long polling,
`DoGet` for results, and `CancelFlightInfo` for cancellation. Each
`PollInfo.info` is treated as the cumulative set of currently available
endpoints, so advertised tickets are consumed once and batches can be
delivered before execution is complete.
- Serializes result completion and cancellation into one lifecycle. A
server-accepted request reports `cancelling` and wakes blocked
status/result work; a later retry can confirm `cancelled`. Result
retrieval is rejected after cancellation is accepted, while cancellation
after a result was already delivered is a no-op.
- Assigns a time-ordered UUIDv7 connection-scoped query id and retains
only shared evolving lifecycle state, keeping SQL, Flight continuation
tokens, and Arrow result data out of public ids and the registry.
- Leaves admission control to the server while honoring server
expiration and a local fallback retention window for abandoned entries.
- Retains terminal ids for five minutes so they remain available for
connection-level description.
- Keeps one lazily initialized SQL client on each remote database
connection and attaches fresh authentication, routing, namespace, and
request metadata to every operation.
- Applies the configured overall timeout to each execution, description,
reader, and cancellation operation. A result reader carries one absolute
deadline from `reader()` through the end of streaming; connect and read
timeouts continue to bound their individual phases.
- Returns a bounded, backpressured, single-consumer Arrow stream rather
than collecting the full result in memory. Dropping the reader stops
downloading but does not implicitly cancel the server query.
- Preserves typed schemas for empty result sets through the stream
schema.
- Accepts Flight result messages up to 1 GiB so a valid row containing a
large blob, string, or vector is not rejected by tonic's 4 MiB default
receive limit.
- Supports the Python client first while keeping the authoritative
implementation in the Rust core.
|
||
|
Will JonesandClaude Opus 5
|
d2ca0ce0ab |
feat: accept multiple on columns for merge insert on remote tables (#4102)
Merge insert has always taken a list of columns to match on, and local tables have always joined on all of them. Remote tables did not: any list longer than one was rejected with `MergeInsertBuilder only supports a single 'on' column`, so a composite-key upsert was impossible against LanceDB Cloud and Enterprise from Rust, Python or TypeScript. The remote request now carries `on` as a list and sends it as one repeated query parameter per column — `?on=shard_key&on=id`. That is how the lance-namespace spec encodes an array-valued `on`, so the server receives a composite key in the shape it expects. A single column still serializes to `?on=id`, exactly what clients sent before, so existing callers are unaffected. A column repeated within `on` is now rejected client-side rather than sent for the server to reject with a 400. No binding changes were needed: `Table.merge_insert` in Python and `Table.mergeInsert` in TypeScript already accepted a list, it just could not reach a remote table. Both gain a test for composite keys, and the doc comments now say what passing several columns means. Part of [ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the). ## Example ```python table.merge_insert(["shard_key", "id"]) \ .when_matched_update_all() \ .when_not_matched_insert_all() \ .execute(new_data) ``` A row whose `id` matches an existing row but whose `shard_key` differs is an insert, not an update. ## Not included Java. Java callers reach merge insert through `org.lance.namespace.LanceNamespace`, whose `MergeInsertIntoTableRequest.on` is a single string until lance-namespace 0.12 ([lance-namespace#363](https://github.com/lance-format/lance-namespace/pull/363), [lance#8915](https://github.com/lance-format/lance/pull/8915)). There is nothing in this repo's Java SDK to change until the `lance-core` pin can move. Sending more than one column requires a server that accepts the repeated parameter ([sophon#7571](https://github.com/lancedb/sophon/pull/7571)); an older server returns a 400 rather than silently merging on one column. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 Bruno Ramirez
|
9a1ffb9e02 |
fix(remote): forward create index replace flag (#4115)
Remote create-index requests already expose `replace` on the builder, but the remote client did not consistently forward an explicit `replace=false` over REST. That meant create-only intent could be lost before it reached a remote server, even though local builders and Python APIs can express it. This PR forwards `replace=false` on the existing `create_index` endpoint and keeps the current default behavior unchanged for compatibility. This was accomplished with the following changes: - Serialize `replace: false` into the existing remote create-index request body when the builder is configured with `.replace(false)`. - Forward `replace` through the synchronous Python remote `create_index` wrapper so `RemoteTable.create_index(..., replace=False)` reaches the repaired path. - Continue omitting `replace` for the default path so existing remote create-index requests keep their current semantics. - Document `name` and `replace` on the existing OpenAPI create-index request schema. - Add coverage that verifies the remote client uses the existing `/create_index/` route and forwards `replace=false`, including the synchronous Python unified API. ### Testing - `cargo fmt --all --check` - `cargo test -p lancedb --features remote test_create_index_forwards_replace_false_on_existing_route --locked` - `uv tool run maturin develop --extras tests,dev,embeddings` - `uv run --frozen pytest python/tests/test_remote_db.py::test_remote_create_index_new_api` - `uv run ruff format --check python/lancedb/remote/table.py python/tests/test_remote_db.py` - `cargo build -p lancedb --features remote --locked` - `cargo clippy -p lancedb --features remote --all-targets --locked -- -D warnings` |
||
|
Xuanwo
|
e6867f7d04 |
feat: support nested blob function signatures (#4109)
Function signatures currently reject Blob v2 fields nested inside structs, preventing UDFs from accepting or returning structured values that contain blobs. Accept canonical Blob v2 fields as direct or recursive struct children while preserving exact field metadata and nullability. Blob fields under list, large-list, fixed-size-list, or map ancestors remain rejected because collection runtime adaptation is outside the supported Function ABI. A whole named struct result can bind directly to one destination column without introducing an extra wrapper level. |
||
|
Xuanwo
|
193c5e3458 |
feat: add list_functions client APIs (#4108)
Function registration and exact lookup are exposed through the SDK, but clients cannot discover published versions even though the server provides `POST /v1/functions/list`. Add Rust and Python sync/async `list_functions()` APIs that return typed `FunctionVersion` values. The remote client requests canonical definitions and follows opaque page tokens until the listing is complete, including empty intermediate pages, while preserving the server's name/version ordering. Local databases retain the existing Function-catalog unsupported error. The SDK consumes protocol pagination internally so callers receive the complete catalog rather than handling server-specific page tokens. |
||
|
Wyatt Alt
|
d118ef168b |
feat: record the source namespace in a materialized view definition (#4098)
A view definition recorded its source by bare name and refresh resolved that name at the root, so declaring a view over a namespaced source was refused outright -- materialized views were root-only for every caller. The definition now carries `source_namespace`, and refresh opens the source at that coordinate. `plan` takes the namespace too: refresh re-plans the stored definition and persists the result when it migrates, so defaulting it there would strand the view on its next rebuild. The stored kind is the version boundary. Root definitions keep the `select` form byte-for-byte, so everything written before this change reads exactly as it always did. A namespaced source is stored as `namespaced_select`: released readers drop unknown fields and resolve a `select` source at the root, so keeping the old kind would let a rolled-back worker refresh a view from a same-name root table -- the new kind routes them to their existing unrecognized-kind refusal instead. The Python and Node definition parsers learn the new kind alongside the Rust core. |
||
|
lancedb-gatefixer[bot]andXuanwo
|
19232f9c50 |
fix: preserve duplicate take offsets (#4024)
## Summary - preserve repeated table offsets without adding a public ordering guarantee - retain exact requested ordering in identity and persisted permutations - cover local, projected, multi-batch, and mocked-remote query paths ## Root cause Take queries lowered offsets to a set-like IN predicate and discarded repeated occurrences. Persisted permutation loading also compared the distinct base-table result count with the requested occurrence count, rejecting repeated row IDs before its existing reordering step could expand them. ## Fix The shared take-query path now deduplicates the predicate for efficient lookup, requests row-offset metadata internally, and expands each matching row to the requested multiplicity in backend result order. An internal opt-in keeps exact requested order for identity PermutationReader reads, while persisted permutations continue using their existing ordering map. ## Validation - cargo test --quiet --features remote --tests - cargo check --quiet --features remote --tests --examples - cargo clippy --quiet --features remote --tests --examples - targeted Python local and mocked-remote regression tests - exact issue reproduction Fixes #2820 <!-- lance-gatekeeper-fix:v1 agent=75acf840afa6f4be4bff98b567b504bd generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Yang Cen
|
c8fd3e97d1 |
test(python): cover stable main udf registration identity (#4094)
## Other changes ### What changed? - Add a subprocess regression harness for an ordinary `@udf` function defined in `__main__`. - Verify the full registration request, artifact digest, and Function signature stay identical across independent Python processes and renamed/moved script paths. - Verify body, referenced-global, and annotation changes still produce distinct artifact identities, with annotation changes also producing a distinct Function signature. ### Why is the change needed? [ENT-2441](https://linear.app/lancedb/issue/ENT-2441/make-sure-function-defined-in-main-gets-stable-signature) tracks the stability guarantee. Investigation on the exact `840e1d73` main base found that LanceDB already packages canonical source instead of cloudpickle bytes, so the unchanged `__main__` function is stable and no production-code fix is needed. This change closes the missing regression-test coverage. [GEN-950](https://linear.app/lancedb/issue/GEN-950/class-based-udfs-defined-in-main-get-a-new-auto-version-on-every-run) remains a separate Geneva checkpoint-version issue for class-based callables. LanceDB's Function API continues to accept synchronous Python functions only. ## Validation - `cd python && uv run --extra tests pytest python/tests/test_first_class_function_slice2.py -q` (`40 passed`) - `uv run --project python --extra dev ruff format .` - `uv run --project python --extra dev ruff check .` (`All checks passed!`) |
||
|
Xuanwo
|
c196d033e9 | feat: add drop_function client APIs (#4097) | ||
|
Jack Ye
|
c6dfe830d9 |
feat(python): support large_utf8 function signatures (#4092)
Teach the Python Function signature emitter to serialize PyArrow large strings as the canonical Arrow type name `large_utf8`. Extend the shared Function Arrow type fixture and explicit-schema coverage for scalar, nested, list, and large-list compositions. |