mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 20:18:37 +00:00
Igor Ganapolsky
a075aa62f8
## Summary Fixes naive `lit(datetime)` equality filters against table timestamp columns on non-UTC hosts, and adds the integration matrix from #3262. ## Failure (before) On a machine in US Eastern (UTC−4 / EDT), with PyPI `lancedb==0.36.0`: ```python from datetime import datetime import lancedb from lancedb.expr import col, lit db = lancedb.connect("memory://") ts = datetime(2024, 7, 1, 10, 0, 0) # naive table = db.create_table("t", [{"id": 1, "ts": ts}]) rows = table.search().where(col("ts") == lit(ts)).to_list() # actual: [] (0 rows) # expected: 1 row ``` ### Root cause In `python/src/expr.rs`, `expr_lit` converted every `datetime` via Python's `.timestamp()`: - **naive** `.timestamp()` = local wall → UTC epoch (shifted by host offset) - **PyArrow naive** storage = UTC wall-clock microseconds (no local shift) So `lit(naive)` became `CAST('2024-07-01 14:00:00' AS TIMESTAMP)` on EDT while the table held `10:00:00`. ## After Naive datetimes are interpreted as UTC wall clock (`replace(tzinfo=timezone.utc).timestamp()`), matching Arrow storage. Aware datetimes still use `.timestamp()` (correct epoch). Same repro on this branch: **1 matching row**. ## Tests Added `TestExprDatetimeTimezoneIntegration` covering: | Case | Result | |------|--------| | both naive | match | | both same TZ (UTC) | match | | different TZs, same instant | match | | table TZ + naive lit | match (wall clock) | | table naive + aware lit | match | | naive lit SQL is wall clock, not local-shifted | asserts `10:00:00` in SQL | ### Verification ```bash cd python maturin develop pytest python/tests/test_expr.py -v ``` **102 passed** (full `test_expr.py`, including the 6 new cases). Closes #3262 --------- Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>