mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Yang Cen
69ac2158a1
`take_row_ids` used an `IN` filter, so the backend could reorder rows and collapse duplicate IDs. That can silently misalign training rows with labels or sample weights. Restore the requested order and duplicate occurrences in the shared Rust take path for native and remote tables, and retain that information through Python query serialization. Missing or deleted IDs are omitted. Explicit Rust `order_by` overrides request order, and pagination applies after restoring occurrences. Empty projections and `_rowid` projection aliases retain their output schema. RTE tables reject unsupported takes with the existing base-table guidance before checking the internal row-ID projection used for ordering. An explicit `use_rte(False)` reads the base table with the same requested order and occurrences. A pre-existing fixture defect also blocked CI: `test_table_add_in_threadpool` responded without reading uploads, allowing a connection reset during response decoding. This PR includes a fix confined to that test's handler: consume each request body before responding and verify one create row plus ten inserted rows. This was not a regression in the take implementation. Production write retries, concurrency and HTTP client code are unchanged. The selective sampling guide scopes the row-ID guarantees to SDK releases containing this fix, preserves the distinct offset-take semantics, and explains that ordered takes collect and combine matching rows before output batching. It recommends smaller ID requests and narrow projections because `max_batch_length` does not cap working memory. This uses the existing REST query protocol and requires a new SDK release; no deployed service was changed. Validation: - Docs follow-up: the changed page compiles with the Mint 4.2.888 MDX compiler; Typos 1.26.0 passes and Vale 3.24.0 has no findings on changed lines. Full-site Mint validation reports four existing snippet-import warnings on the reranking/vector-search pages, identical with the original page restored in a baseline comparison. - The applied threadpool test passed locally after the fixture change. The identical candidate previously passed 10 trials with the ordinary reader and 10 with an unbuffered reader; each trial verifies all uploaded rows. Those checks used macOS arm64/Python 3.14/default features, not the full failing CI environment. The normal new-head macOS Python 3.13/experimental-functions job remains the required platform check. - Python package Ruff format/check passed. Repository-root Ruff still reports 59 existing errors and six formatting differences in unchanged `docs/web-tests` files; none are in changed files. - At `a569d3f0`, the RTE diagnostic regression was reproduced and fixed; 5 targeted cases and strict workspace Clippy passed. The fixture-only follow-up changes no production Rust/Python code, so that evidence is reused. - Earlier validation at `6bcc0878`: 439 Rust query/remote-table tests and 24 Python take/getitems tests passed; the focused 49-test suite also covered offset dataloaders. A local 9,216-row, four-fragment reproduction returned 200 seeded random batches of 64 rows with zero ordering or label-alignment mismatches. The ordering and projection implementation is unchanged by the follow-ups. - No live Enterprise run was performed. Fixes [ENT-3469](https://linear.app/lancedb/issue/ENT-3469).