Compare commits

...

6 Commits

Author SHA1 Message Date
Lance Release d3a7d27d36 Bump version: 0.38.0-beta.1 → 0.38.0-beta.2 2026-08-19 01:58:48 +00:00
Dan Rammer cdebea118d feat(python): expose LSM checkpoint and stats on sync RemoteTable (#3961)
## Summary

The sync `RemoteTable` carried `set_lsm_write_spec`,
`unset_lsm_write_spec`, `get_lsm_write_spec`, and `close_lsm_writers`,
but not `checkpoint_lsm`, `flush_lsm`, `compact_lsm`, or
`get_lsm_stats`.

That left the four LSM control methods reachable from `AsyncTable` only.
They are also the four that *only* work against a remote table —
`NativeTable` does not override the `BaseTable` defaults, so on a local
table they return `NotSupported` (`rust/lancedb/src/table.rs:679-701`).
The net effect for sync users:

| | `checkpoint_lsm` / `get_lsm_stats` |
|---|---|
| `LanceTable` (sync, local) | present, but always `NotSupported` |
| `RemoteTable` (sync, remote) | `AttributeError` — method absent |
| `AsyncTable` (remote) | works |

So there was no working sync path at all, despite the Rust `RemoteTable`
implementing every one of these against real endpoints.

## Changes

* Add `checkpoint_lsm`, `flush_lsm`, `compact_lsm`, and `get_lsm_stats`
to `lancedb.remote.table.RemoteTable`, mirroring the delegation style of
their neighbours.
* Correct the docstrings on `set_lsm_write_spec` /
`unset_lsm_write_spec`, which read `"""Not supported on LanceDB
Cloud."""` although `rust/lancedb/src/remote/table.rs:2549-2601`
implements both against `/v1/table/{}/set_lsm_write_spec/` and
`/unset_lsm_write_spec/`. They appear to have been copy-pasted from
`set_unenforced_primary_key` directly above.

No Rust or PyO3 changes — the bindings and the `AsyncTable` methods
already existed. The `Table` ABC is left alone, matching how the
existing `*_lsm_write_spec` methods are declared on the concrete classes
only.

## Tests

Four new tests in `python/python/tests/test_remote_db.py`, against the
existing mock HTTP server:

* `test_get_lsm_stats_sync` — the server payload round-trips into the
dict, and `include_generation_rows` defaults to `False` and is forwarded
when set.
* `test_get_lsm_stats_sync_returns_none_when_lsm_disabled` — a
`{"lsm_stats": null}` envelope yields `None` rather than an error.
* `test_flush_and_compact_lsm_sync` — both are one-shot POSTs answered
`202` with no body.
* `test_checkpoint_lsm_sync` — pins the binding to the endpoints it
drives (`flush_lsm` then `get_lsm_stats`); the convergence loop itself
is already covered in Rust.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 17:25:08 -05:00
Xuanwo 76942306b7 docs(java): add vended credentials example (#3958)
## Context

Java users opening catalog-backed tables with vended credentials
currently lack a documented workflow. Opening the catalog-returned URI
directly drops the namespace-provided storage options and automatic
credential refresh.

Document the namespace-backed `Dataset.open()` path so temporary object
store credentials are applied and refreshed transparently.
2026-08-18 20:03:20 +08:00
Adityaj0 d742b174c4 fix: hybrid search silently ignores .offset() (#3769)
## Summary

`LanceHybridQueryBuilder` (sync hybrid search,
`table.search(query_type="hybrid")`) silently ignored `.offset()`.
`self._offset` was never forwarded to the vector/FTS sub-queries and
never applied when slicing the final combined/reranked result, so
`.offset(N)` behaved identically to `.offset(0)` — no error, just wrong
pagination.

Fixes #3765

## Changes

- `_create_query_builders()`: each sub-query now fetches `limit +
offset` rows so there's enough data to slice the correct window out of
after combining/reranking.
- `_combine_hybrid_results()` / `to_arrow()`: the final table is sliced
with `offset=self._offset` instead of always starting at 0.

## Test plan

- [x] New regression test `test_hybrid_query_offset` in
`python/python/tests/test_hybrid_query.py`
- [x] `uv run --extra tests pytest python/tests/test_hybrid_query.py
-vv` — 13 passed
- [x] `uv run --extra dev ruff format` / `ruff check` — clean

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
2026-08-17 11:38:38 -07:00
Igor Ganapolsky a075aa62f8 fix(python): treat naive lit(datetime) as UTC wall clock (#3262) (#3775)
## Summary

Fixes naive `lit(datetime)` equality filters against table timestamp
columns on non-UTC hosts, and adds the integration matrix from #3262.

## Failure (before)

On a machine in US Eastern (UTC−4 / EDT), with PyPI `lancedb==0.36.0`:

```python
from datetime import datetime
import lancedb
from lancedb.expr import col, lit

db = lancedb.connect("memory://")
ts = datetime(2024, 7, 1, 10, 0, 0)  # naive
table = db.create_table("t", [{"id": 1, "ts": ts}])
rows = table.search().where(col("ts") == lit(ts)).to_list()
# actual: []  (0 rows)
# expected: 1 row
```

### Root cause

In `python/src/expr.rs`, `expr_lit` converted every `datetime` via
Python's `.timestamp()`:

- **naive** `.timestamp()` = local wall → UTC epoch (shifted by host
offset)
- **PyArrow naive** storage = UTC wall-clock microseconds (no local
shift)

So `lit(naive)` became `CAST('2024-07-01 14:00:00' AS TIMESTAMP)` on EDT
while the table held `10:00:00`.

## After

Naive datetimes are interpreted as UTC wall clock
(`replace(tzinfo=timezone.utc).timestamp()`), matching Arrow storage.
Aware datetimes still use `.timestamp()` (correct epoch).

Same repro on this branch: **1 matching row**.

## Tests

Added `TestExprDatetimeTimezoneIntegration` covering:

| Case | Result |
|------|--------|
| both naive | match |
| both same TZ (UTC) | match |
| different TZs, same instant | match |
| table TZ + naive lit | match (wall clock) |
| table naive + aware lit | match |
| naive lit SQL is wall clock, not local-shifted | asserts `10:00:00` in
SQL |

### Verification

```bash
cd python
maturin develop
pytest python/tests/test_expr.py -v
```

**102 passed** (full `test_expr.py`, including the 6 new cases).

Closes #3262

---------

Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:48:02 -07:00
Lance Release 040a4120c8 Bump version: 0.38.0-beta.0 → 0.38.0-beta.1 2026-08-17 16:56:54 +00:00
24 changed files with 356 additions and 29 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
[tool.bumpversion]
current_version = "0.38.0-beta.0"
current_version = "0.38.0-beta.2"
parse = """(?x)
(?P<major>0|[1-9]\\d*)\\.
(?P<minor>0|[1-9]\\d*)\\.
Generated
+3 -3
View File
@@ -5398,7 +5398,7 @@ dependencies = [
[[package]]
name = "lancedb"
version = "0.38.0-beta.0"
version = "0.38.0-beta.1"
dependencies = [
"ahash",
"anyhow",
@@ -5486,7 +5486,7 @@ dependencies = [
[[package]]
name = "lancedb-nodejs"
version = "0.38.0-beta.0"
version = "0.38.0-beta.1"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -5511,7 +5511,7 @@ dependencies = [
[[package]]
name = "lancedb-python"
version = "0.38.0-beta.0"
version = "0.38.0-beta.1"
dependencies = [
"arrow",
"async-trait",
+33 -1
View File
@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
<dependency>
<groupId>com.lancedb</groupId>
<artifactId>lancedb-core</artifactId>
<version>0.38.0-beta.0</version>
<version>0.38.0-beta.2</version>
</dependency>
```
@@ -55,6 +55,38 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
| `region(String)` | AWS region (default: "us-east-1") | No |
| `config(String, String)` | Additional configuration parameters | No |
### Opening a Table with Vended Credentials
When the catalog vends temporary object store credentials, open the table through the
namespace client. The Lance dataset builder fetches the table location and storage options
from the catalog and refreshes the credentials when they expire.
```java
import com.lancedb.LanceDbNamespaceClientBuilder;
import org.lance.Dataset;
import org.lance.namespace.LanceNamespace;
import java.util.Arrays;
LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
.apiKey(System.getenv("LANCEDB_API_KEY"))
.database(System.getenv("LANCEDB_DATABASE"))
// Set the endpoint for a LanceDB Enterprise deployment.
// .endpoint("https://your-enterprise-endpoint")
.build();
try (Dataset dataset = Dataset.open()
.namespaceClient(namespaceClient)
.tableId(Arrays.asList("my_namespace", "my_table"))
.build()) {
System.out.println("Rows: " + dataset.countRows());
}
```
Do not call `describeTable()` and then open the returned location with `Dataset.open(uri)`.
Opening through `namespaceClient()` is what applies the vended storage options and enables
automatic credential refresh. No object store credentials need to be passed by the application.
## Metadata Operations
### Creating a Namespace Path
+1 -1
View File
@@ -8,7 +8,7 @@
<parent>
<groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId>
<version>0.38.0-beta.0</version>
<version>0.38.0-beta.2</version>
<relativePath>../pom.xml</relativePath>
</parent>
+1 -1
View File
@@ -6,7 +6,7 @@
<groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId>
<version>0.38.0-beta.0</version>
<version>0.38.0-beta.2</version>
<packaging>pom</packaging>
<name>${project.artifactId}</name>
<description>LanceDB Java SDK Parent POM</description>
+1 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "lancedb-nodejs"
edition.workspace = true
version = "0.38.0-beta.0"
version = "0.38.0-beta.2"
publish = false
license.workspace = true
description.workspace = true
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-darwin-arm64",
"version": "0.38.0-beta.0",
"version": "0.38.0-beta.2",
"os": ["darwin"],
"cpu": ["arm64"],
"main": "lancedb.darwin-arm64.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-arm64-gnu",
"version": "0.38.0-beta.0",
"version": "0.38.0-beta.2",
"os": ["linux"],
"cpu": ["arm64"],
"main": "lancedb.linux-arm64-gnu.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-arm64-musl",
"version": "0.38.0-beta.0",
"version": "0.38.0-beta.2",
"os": ["linux"],
"cpu": ["arm64"],
"main": "lancedb.linux-arm64-musl.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-x64-gnu",
"version": "0.38.0-beta.0",
"version": "0.38.0-beta.2",
"os": ["linux"],
"cpu": ["x64"],
"main": "lancedb.linux-x64-gnu.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-x64-musl",
"version": "0.38.0-beta.0",
"version": "0.38.0-beta.2",
"os": ["linux"],
"cpu": ["x64"],
"main": "lancedb.linux-x64-musl.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-win32-arm64-msvc",
"version": "0.38.0-beta.0",
"version": "0.38.0-beta.2",
"os": [
"win32"
],
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-win32-x64-msvc",
"version": "0.38.0-beta.0",
"version": "0.38.0-beta.2",
"os": ["win32"],
"cpu": ["x64"],
"main": "lancedb.win32-x64-msvc.node",
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@lancedb/lancedb",
"version": "0.38.0-beta.0",
"version": "0.38.0-beta.1",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@lancedb/lancedb",
"version": "0.38.0-beta.0",
"version": "0.38.0-beta.1",
"cpu": [
"x64",
"arm64"
+1 -1
View File
@@ -11,7 +11,7 @@
"ann"
],
"private": false,
"version": "0.38.0-beta.0",
"version": "0.38.0-beta.2",
"main": "dist/index.js",
"exports": {
".": "./dist/index.js",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "lancedb-python"
version = "0.38.0-beta.0"
version = "0.38.0-beta.2"
publish = false
edition.workspace = true
description = "Python bindings for LanceDB"
+9 -3
View File
@@ -2235,6 +2235,7 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
reranker=self._reranker,
limit=self._limit,
with_row_ids=True,
offset=self._offset,
)
return self._finish_hybrid_results(results)
@@ -2256,6 +2257,7 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
reranker,
limit: int,
with_row_ids: bool,
offset: Optional[int] = None,
) -> pa.Table:
if norm == "rank":
vector_results = LanceHybridQueryBuilder._rank(vector_results, "_distance")
@@ -2332,7 +2334,7 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
score_i = results.column_names.index("_score")
results = results.set_column(score_i, "_score", original_scores)
results = results.slice(length=limit)
results = results.slice(offset=offset or 0, length=limit)
if not with_row_ids:
results = results.drop(["_rowid"])
@@ -2679,8 +2681,12 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
# Apply common configurations
if self._limit:
self._vector_query.limit(self._limit)
self._fts_query.limit(self._limit)
# The final offset/limit window is sliced out of the combined,
# reranked results, so each sub-query must fetch enough rows to
# cover the skipped prefix as well as the window itself.
sub_query_limit = self._limit + (self._offset or 0)
self._vector_query.limit(sub_query_limit)
self._fts_query.limit(sub_query_limit)
if self._columns:
self._vector_query.select(self._columns)
self._fts_query.select(self._columns)
+24 -2
View File
@@ -990,17 +990,39 @@ class RemoteTable(Table):
return LOOP.run(self._table.set_unenforced_primary_key(columns))
def set_lsm_write_spec(self, spec: "LsmWriteSpec") -> None:
"""Not supported on LanceDB Cloud."""
"""Install an LsmWriteSpec."""
return LOOP.run(self._table.set_lsm_write_spec(spec))
def unset_lsm_write_spec(self) -> None:
"""Not supported on LanceDB Cloud."""
"""Remove the LsmWriteSpec."""
return LOOP.run(self._table.unset_lsm_write_spec())
def get_lsm_write_spec(self) -> Optional["LsmWriteSpec"]:
"""Read the installed LsmWriteSpec, or ``None``."""
return LOOP.run(self._table.get_lsm_write_spec())
def checkpoint_lsm(self) -> None:
"""Synchronous version of
[`AsyncTable.checkpoint_lsm`][lancedb.AsyncTable.checkpoint_lsm]."""
return LOOP.run(self._table.checkpoint_lsm())
def flush_lsm(self) -> None:
"""Synchronous version of
[`AsyncTable.flush_lsm`][lancedb.AsyncTable.flush_lsm]."""
return LOOP.run(self._table.flush_lsm())
def compact_lsm(self) -> None:
"""Synchronous version of
[`AsyncTable.compact_lsm`][lancedb.AsyncTable.compact_lsm]."""
return LOOP.run(self._table.compact_lsm())
def get_lsm_stats(self, *, include_generation_rows: bool = False) -> Optional[dict]:
"""Synchronous version of
[`AsyncTable.get_lsm_stats`][lancedb.AsyncTable.get_lsm_stats]."""
return LOOP.run(
self._table.get_lsm_stats(include_generation_rows=include_generation_rows)
)
def close_lsm_writers(self) -> None:
"""No-op on LanceDB Cloud (no local shard writers)."""
return LOOP.run(self._table.close_lsm_writers())
+3 -3
View File
@@ -4846,7 +4846,7 @@ class AsyncTable:
``asyncio.wait_for`` for a wall-clock bound; abandoning it partway
costs nothing.
"""
return await self._inner.checkpoint_lsm()
await self._inner.checkpoint_lsm()
async def flush_lsm(self) -> None:
"""Seal every bucket's active memtable into L0.
@@ -4855,7 +4855,7 @@ class AsyncTable:
`compact_lsm`. On a node that has not claimed this table, this claims
it and replays its WAL log first.
"""
return await self._inner.flush_lsm()
await self._inner.flush_lsm()
async def compact_lsm(self) -> None:
"""Trigger a background L0 to base compaction pass per bucket.
@@ -4864,7 +4864,7 @@ class AsyncTable:
``get_lsm_stats`` for progress, or use ``checkpoint_lsm`` to loop
until the current L0 has reached base.
"""
return await self._inner.compact_lsm()
await self._inner.compact_lsm()
async def get_lsm_stats(
self, *, include_generation_rows: bool = False
+98
View File
@@ -632,3 +632,101 @@ class TestExprBytesIntegration:
.to_arrow()
)
assert result.num_rows == 2
# ── datetime / timezone integration for lit() (issue #3262) ──────────────────
class TestExprDatetimeTimezoneIntegration:
"""Integration coverage for lit(datetime) against table timestamp columns.
PyArrow stores naive timestamps as UTC wall-clock microseconds. Python's
datetime.timestamp() treats naive values as *local* time, which used to
shift lit(naive) by the host UTC offset and break equality filters on
non-UTC machines. These cases lock the expected semantics.
"""
def test_both_naive_match(self, tmp_path):
"""Table naive + lit naive with the same wall clock must match."""
db = lancedb.connect(str(tmp_path / "naive"))
ts = datetime(2024, 7, 1, 10, 0, 0)
table = db.create_table(
"t", [{"id": 1, "ts": ts}, {"id": 2, "ts": datetime(2024, 7, 2, 10, 0, 0)}]
)
result = table.search().where(col("ts") == lit(ts)).to_list()
assert len(result) == 1
assert result[0]["id"] == 1
def test_both_same_timezone_match(self, tmp_path):
"""Table UTC + lit UTC for the same instant must match."""
db = lancedb.connect(str(tmp_path / "utc"))
ts = datetime(2024, 7, 1, 10, 0, 0, tzinfo=timezone.utc)
table = db.create_table(
"t",
pa.table(
{
"id": [1, 2],
"ts": pa.array(
[ts, datetime(2024, 7, 2, 10, 0, 0, tzinfo=timezone.utc)],
type=pa.timestamp("us", tz="UTC"),
),
}
),
)
result = table.search().where(col("ts") == lit(ts)).to_list()
assert len(result) == 1
assert result[0]["id"] == 1
def test_different_timezones_same_instant(self, tmp_path):
"""UTC table row equals lit of the same instant in a different zone."""
db = lancedb.connect(str(tmp_path / "diff_tz"))
ts_utc = datetime(2024, 7, 1, 10, 0, 0, tzinfo=timezone.utc)
# Same instant as 06:00 in UTC-4
ts_est = datetime(2024, 7, 1, 6, 0, 0, tzinfo=timezone(timedelta(hours=-4)))
table = db.create_table(
"t",
pa.table(
{
"id": [1],
"ts": pa.array([ts_utc], type=pa.timestamp("us", tz="UTC")),
}
),
)
result = table.search().where(col("ts") == lit(ts_est)).to_list()
assert len(result) == 1
assert result[0]["id"] == 1
def test_table_tz_literal_naive(self, tmp_path):
"""UTC table + naive lit uses wall-clock equality (10:00 == 10:00 UTC)."""
db = lancedb.connect(str(tmp_path / "tz_naive"))
ts_utc = datetime(2024, 7, 1, 10, 0, 0, tzinfo=timezone.utc)
ts_naive = datetime(2024, 7, 1, 10, 0, 0)
table = db.create_table(
"t",
pa.table(
{
"id": [1],
"ts": pa.array([ts_utc], type=pa.timestamp("us", tz="UTC")),
}
),
)
result = table.search().where(col("ts") == lit(ts_naive)).to_list()
assert len(result) == 1
assert result[0]["id"] == 1
def test_table_naive_literal_aware(self, tmp_path):
"""Naive table + UTC lit with the same wall clock must match."""
db = lancedb.connect(str(tmp_path / "naive_aware"))
ts_naive = datetime(2024, 7, 1, 10, 0, 0)
ts_utc = datetime(2024, 7, 1, 10, 0, 0, tzinfo=timezone.utc)
table = db.create_table("t", [{"id": 1, "ts": ts_naive}])
result = table.search().where(col("ts") == lit(ts_utc)).to_list()
assert len(result) == 1
assert result[0]["id"] == 1
def test_naive_lit_sql_is_wall_clock_not_local_shifted(self):
"""Regression: naive lit must not apply the host local UTC offset."""
ts = datetime(2024, 7, 1, 10, 0, 0)
sql = lit(ts).to_sql()
# Must encode 10:00 wall clock, not 10:00+local_offset.
assert "2024-07-01 10:00:00" in sql
+25
View File
@@ -203,6 +203,31 @@ async def test_async_hybrid_query_default_limit(table: AsyncTable):
assert texts.count("a") == 1
def test_hybrid_query_offset(sync_table: Table):
# The offset window of a hybrid query must be a suffix of the same query
# run without an offset -- it must not be silently ignored.
full = (
sync_table.search(query_type="hybrid")
.vector([0.0, 0.4])
.text("dog")
.limit(4)
.with_row_id(True)
.to_arrow()
)
assert len(full) == 4
offset_result = (
sync_table.search(query_type="hybrid")
.vector([0.0, 0.4])
.text("dog")
.offset(2)
.limit(2)
.with_row_id(True)
.to_arrow()
)
assert offset_result["_rowid"].to_pylist() == full["_rowid"].to_pylist()[2:]
def test_hybrid_query_minimum_nprobes_zero_raises(sync_table: Table):
# minimum_nprobes(0) must raise the same validation error a plain vector
# query raises, not silently no-op because 0 is falsy.
+125
View File
@@ -1133,6 +1133,131 @@ def test_stats():
assert res == stats
@contextlib.contextmanager
def lsm_test_table(lsm_handler):
"""A remote table whose LSM routes are served by ``lsm_handler``.
``lsm_handler(request, route)`` is called for ``/v1/table/test/<route>/``
where route is one of flush_lsm, compact_lsm, get_lsm_stats, and is
responsible for writing the response.
"""
routes = ("flush_lsm", "compact_lsm", "get_lsm_stats")
def handler(request):
match = re.fullmatch(r"/v1/table/test/(\w+)/", request.path)
route = match.group(1) if match else None
if route in routes:
lsm_handler(request, route)
elif route == "describe":
request.send_response(200)
request.send_header("Content-Type", "application/json")
request.end_headers()
request.wfile.write(b'{"version": 1, "schema": {"fields": []}}')
else:
request.send_response(404)
request.end_headers()
with mock_lancedb_connection(handler) as db:
yield db.open_table("test")
def read_json_body(request):
content_len = int(request.headers.get("Content-Length"))
return json.loads(request.rfile.read(content_len))
def send_json(request, payload, status=200):
request.send_response(status)
request.send_header("Content-Type", "application/json")
request.end_headers()
request.wfile.write(json.dumps(payload).encode())
def test_get_lsm_stats_sync():
"""The sync wrapper round-trips the server payload into a dict."""
bucket = {
"shard_id": "b0",
"status": "Active",
"writer_epoch": 3,
"manifest_version": 12,
"current_generation": 6,
"replay_after_wal_entry_position": 40,
"wal_entry_position_last_seen": 42,
"generations": [{"generation": 5, "bytes": 1024, "rows": 7}],
"compacting": False,
"memtables": [
{
"generation": 6,
"rows": 2,
"bytes": 64,
"batches": 1,
"indexes": ["vec_idx"],
}
],
}
seen_bodies = []
def lsm_handler(request, route):
assert route == "get_lsm_stats"
seen_bodies.append(read_json_body(request))
send_json(request, {"lsm_stats": {"buckets": [bucket]}})
with lsm_test_table(lsm_handler) as table:
assert table.get_lsm_stats() == {"buckets": [bucket]}
# Off by default, and forwarded when asked for.
assert seen_bodies == [{"include_generation_rows": False}]
table.get_lsm_stats(include_generation_rows=True)
assert seen_bodies[-1] == {"include_generation_rows": True}
def test_get_lsm_stats_sync_returns_none_when_lsm_disabled():
"""A null envelope means the LSM write path is not enabled, not an error."""
def lsm_handler(request, route):
send_json(request, {"lsm_stats": None})
with lsm_test_table(lsm_handler) as table:
assert table.get_lsm_stats() is None
def test_flush_and_compact_lsm_sync():
"""Both are one-shot POSTs answered 202 with no body."""
called = []
def lsm_handler(request, route):
called.append(route)
request.send_response(202)
request.end_headers()
with lsm_test_table(lsm_handler) as table:
assert table.flush_lsm() is None
assert table.compact_lsm() is None
assert called == ["flush_lsm", "compact_lsm"]
def test_checkpoint_lsm_sync():
"""Seal, read the watermark, and return once L0 holds nothing.
The convergence loop itself is covered in Rust; this pins the sync
binding to the endpoints it drives.
"""
called = []
def lsm_handler(request, route):
called.append(route)
if route == "get_lsm_stats":
# An empty L0 yields no target watermark, so the loop is done
# after the seal without ever polling compaction.
send_json(request, {"lsm_stats": {"buckets": []}})
else:
request.send_response(202)
request.end_headers()
with lsm_test_table(lsm_handler) as table:
assert table.checkpoint_lsm() is None
assert called == ["flush_lsm", "get_lsm_stats"]
@contextlib.contextmanager
def query_test_table(query_handler, *, server_version=Version("0.1.0")):
def handler(request):
+20 -1
View File
@@ -191,8 +191,27 @@ pub fn expr_lit(value: Bound<'_, PyAny>) -> PyResult<PyExpr> {
}
// datetime.datetime is a subclass of datetime.date, so it must be checked first.
//
// Python's datetime.timestamp() treats *naive* datetimes as local wall time.
// PyArrow (and therefore Lance table storage) encodes naive timestamps as
// UTC wall-clock microseconds. Using .timestamp() for naive values therefore
// shifts the literal by the local UTC offset on non-UTC machines, so
// `col("ts") == lit(naive_dt)` fails against a table that holds the same
// naive value. Fix: treat naive datetimes as UTC wall clock (match Arrow);
// keep aware datetimes on the real .timestamp() path (correct epoch).
if let Ok(dt) = value.cast::<PyDateTime>() {
let ts: f64 = dt.call_method0("timestamp")?.extract()?;
let ts: f64 = if dt.getattr("tzinfo")?.is_none() {
// Force UTC interpretation of the naive wall clock.
let utc = pyo3::types::PyModule::import(value.py(), "datetime")?
.getattr("timezone")?
.getattr("utc")?;
let kwargs = pyo3::types::PyDict::new(value.py());
kwargs.set_item("tzinfo", utc)?;
let aware = dt.call_method("replace", (), Some(&kwargs))?;
aware.call_method0("timestamp")?.extract()?
} else {
dt.call_method0("timestamp")?.extract()?
};
let micros = (ts * 1_000_000.0).round() as i64;
return Ok(PyExpr(df_lit(ScalarValue::TimestampMicrosecond(
Some(micros),
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "lancedb"
version = "0.38.0-beta.0"
version = "0.38.0-beta.2"
edition.workspace = true
description = "LanceDB: A serverless, low-latency vector database for AI applications"
license.workspace = true