fix(python): treat naive lit(datetime) as UTC wall clock (#3262) (#3775)

## Summary

Fixes naive `lit(datetime)` equality filters against table timestamp
columns on non-UTC hosts, and adds the integration matrix from #3262.

## Failure (before)

On a machine in US Eastern (UTC−4 / EDT), with PyPI `lancedb==0.36.0`:

```python
from datetime import datetime
import lancedb
from lancedb.expr import col, lit

db = lancedb.connect("memory://")
ts = datetime(2024, 7, 1, 10, 0, 0)  # naive
table = db.create_table("t", [{"id": 1, "ts": ts}])
rows = table.search().where(col("ts") == lit(ts)).to_list()
# actual: []  (0 rows)
# expected: 1 row
```

### Root cause

In `python/src/expr.rs`, `expr_lit` converted every `datetime` via
Python's `.timestamp()`:

- **naive** `.timestamp()` = local wall → UTC epoch (shifted by host
offset)
- **PyArrow naive** storage = UTC wall-clock microseconds (no local
shift)

So `lit(naive)` became `CAST('2024-07-01 14:00:00' AS TIMESTAMP)` on EDT
while the table held `10:00:00`.

## After

Naive datetimes are interpreted as UTC wall clock
(`replace(tzinfo=timezone.utc).timestamp()`), matching Arrow storage.
Aware datetimes still use `.timestamp()` (correct epoch).

Same repro on this branch: **1 matching row**.

## Tests

Added `TestExprDatetimeTimezoneIntegration` covering:

| Case | Result |
|------|--------|
| both naive | match |
| both same TZ (UTC) | match |
| different TZs, same instant | match |
| table TZ + naive lit | match (wall clock) |
| table naive + aware lit | match |
| naive lit SQL is wall clock, not local-shifted | asserts `10:00:00` in
SQL |

### Verification

```bash
cd python
maturin develop
pytest python/tests/test_expr.py -v
```

**102 passed** (full `test_expr.py`, including the 6 new cases).

Closes #3262

---------

Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Igor Ganapolsky
2026-08-17 10:48:02 -07:00
committed by GitHub
parent 040a4120c8
commit a075aa62f8
2 changed files with 118 additions and 1 deletions
+20 -1
View File
@@ -191,8 +191,27 @@ pub fn expr_lit(value: Bound<'_, PyAny>) -> PyResult<PyExpr> {
}
// datetime.datetime is a subclass of datetime.date, so it must be checked first.
//
// Python's datetime.timestamp() treats *naive* datetimes as local wall time.
// PyArrow (and therefore Lance table storage) encodes naive timestamps as
// UTC wall-clock microseconds. Using .timestamp() for naive values therefore
// shifts the literal by the local UTC offset on non-UTC machines, so
// `col("ts") == lit(naive_dt)` fails against a table that holds the same
// naive value. Fix: treat naive datetimes as UTC wall clock (match Arrow);
// keep aware datetimes on the real .timestamp() path (correct epoch).
if let Ok(dt) = value.cast::<PyDateTime>() {
let ts: f64 = dt.call_method0("timestamp")?.extract()?;
let ts: f64 = if dt.getattr("tzinfo")?.is_none() {
// Force UTC interpretation of the naive wall clock.
let utc = pyo3::types::PyModule::import(value.py(), "datetime")?
.getattr("timezone")?
.getattr("utc")?;
let kwargs = pyo3::types::PyDict::new(value.py());
kwargs.set_item("tzinfo", utc)?;
let aware = dt.call_method("replace", (), Some(&kwargs))?;
aware.call_method0("timestamp")?.extract()?
} else {
dt.call_method0("timestamp")?.extract()?
};
let micros = (ts * 1_000_000.0).round() as i64;
return Ok(PyExpr(df_lit(ScalarValue::TimestampMicrosecond(
Some(micros),