mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 20:18:37 +00:00
## Summary Fixes naive `lit(datetime)` equality filters against table timestamp columns on non-UTC hosts, and adds the integration matrix from #3262. ## Failure (before) On a machine in US Eastern (UTC−4 / EDT), with PyPI `lancedb==0.36.0`: ```python from datetime import datetime import lancedb from lancedb.expr import col, lit db = lancedb.connect("memory://") ts = datetime(2024, 7, 1, 10, 0, 0) # naive table = db.create_table("t", [{"id": 1, "ts": ts}]) rows = table.search().where(col("ts") == lit(ts)).to_list() # actual: [] (0 rows) # expected: 1 row ``` ### Root cause In `python/src/expr.rs`, `expr_lit` converted every `datetime` via Python's `.timestamp()`: - **naive** `.timestamp()` = local wall → UTC epoch (shifted by host offset) - **PyArrow naive** storage = UTC wall-clock microseconds (no local shift) So `lit(naive)` became `CAST('2024-07-01 14:00:00' AS TIMESTAMP)` on EDT while the table held `10:00:00`. ## After Naive datetimes are interpreted as UTC wall clock (`replace(tzinfo=timezone.utc).timestamp()`), matching Arrow storage. Aware datetimes still use `.timestamp()` (correct epoch). Same repro on this branch: **1 matching row**. ## Tests Added `TestExprDatetimeTimezoneIntegration` covering: | Case | Result | |------|--------| | both naive | match | | both same TZ (UTC) | match | | different TZs, same instant | match | | table TZ + naive lit | match (wall clock) | | table naive + aware lit | match | | naive lit SQL is wall clock, not local-shifted | asserts `10:00:00` in SQL | ### Verification ```bash cd python maturin develop pytest python/tests/test_expr.py -v ``` **102 passed** (full `test_expr.py`, including the 6 new cases). Closes #3262 --------- Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+20
-1
@@ -191,8 +191,27 @@ pub fn expr_lit(value: Bound<'_, PyAny>) -> PyResult<PyExpr> {
|
||||
}
|
||||
|
||||
// datetime.datetime is a subclass of datetime.date, so it must be checked first.
|
||||
//
|
||||
// Python's datetime.timestamp() treats *naive* datetimes as local wall time.
|
||||
// PyArrow (and therefore Lance table storage) encodes naive timestamps as
|
||||
// UTC wall-clock microseconds. Using .timestamp() for naive values therefore
|
||||
// shifts the literal by the local UTC offset on non-UTC machines, so
|
||||
// `col("ts") == lit(naive_dt)` fails against a table that holds the same
|
||||
// naive value. Fix: treat naive datetimes as UTC wall clock (match Arrow);
|
||||
// keep aware datetimes on the real .timestamp() path (correct epoch).
|
||||
if let Ok(dt) = value.cast::<PyDateTime>() {
|
||||
let ts: f64 = dt.call_method0("timestamp")?.extract()?;
|
||||
let ts: f64 = if dt.getattr("tzinfo")?.is_none() {
|
||||
// Force UTC interpretation of the naive wall clock.
|
||||
let utc = pyo3::types::PyModule::import(value.py(), "datetime")?
|
||||
.getattr("timezone")?
|
||||
.getattr("utc")?;
|
||||
let kwargs = pyo3::types::PyDict::new(value.py());
|
||||
kwargs.set_item("tzinfo", utc)?;
|
||||
let aware = dt.call_method("replace", (), Some(&kwargs))?;
|
||||
aware.call_method0("timestamp")?.extract()?
|
||||
} else {
|
||||
dt.call_method0("timestamp")?.extract()?
|
||||
};
|
||||
let micros = (ts * 1_000_000.0).round() as i64;
|
||||
return Ok(PyExpr(df_lit(ScalarValue::TimestampMicrosecond(
|
||||
Some(micros),
|
||||
|
||||
Reference in New Issue
Block a user