mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
feat: expose exact Function column freshness (#4958)
## Problem Function-backed columns need an exact `fresh`, `stale`, `never_computed`, and `failed` partition for the table version a caller is reading. The existing error listing is capped, and composing it with job and schema listings cannot prove which rows currently failed. ## Behavior - Adds a strict freshness reader beside the existing permissive planner. It distinguishes an absent certificate from legacy evidence that omits failures, and from a current certificate carrying failure coverage. - Extends the immutable freshness sidecar with per-fragment known and failed row bitmaps. Partial or `WHERE` publications cover only observed rows; untouched rows stay unknown. Deleted rows do not block classification. - Adds failure-aware whole-fragment and row-repair writers. A rejected older attempt cannot overwrite failure evidence already recorded for a newer input signature. - Adds `column_status`, `column_statuses`, and `wait_fresh` to Python and the corresponding Node methods. The client calls the server's exact status endpoint, validates the four-way partition and one-version guarantee, and preserves the complete Function version, last-job, and automatic-refresh metadata. - Adds an MDX-commented draft usage section to the Enterprise computed-column refresh guide, covering Python, Rust, and TypeScript and remaining hidden until the server integration ships. - Sends `column`, `branch`, and exact `version` selectors to the status route. A pinned handle rejects a nonempty response from any other table version; latest handles keep the existing read-freshness headers. - Retries `wait_fresh` only for HTTP 503 with `function_column_status_finalization_pending`. Existing failed rows remain visible and do not keep the wait pending; other errors propagate. - Models the shared scheduler projection, including `up_to_date`, real `blocked` states, their reason, failure streak, and effective cooldown. ## Validation This is the SDK side of ENT-3563. Sophon's failure-coverage publication and status endpoint are a separate integration; deployed end-to-end and whole-ticket acceptance remain pending. - Rust: 9 exact freshness/status tests passed, plus 1 exact latest-handle freshness-header test. The pinned transport test was extended with a wrong-version response and rerun at `ebd6649a` (1 passed). Coverage includes untouched partial rows, selected-row failure clearing, cross-generation evidence, typed and non-retried errors, `up_to_date`, blocked scheduler fields, and exact-version requests. - Python: 2 native PyO3 remote-transport tests passed together, including typed retry; one local facade test reaches the existing unsupported boundary. - Node: the NAPI addon built successfully, TypeScript and Biome passed, and the selected native remote-transport test passed 1/0, including exact-version selection and blocked scheduler metadata. - `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile, Biome, and TypeScript checks passed. Validation is mapped by the relevant unchanged source rather than treated as one final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header and Python remote tests at `31e68b7d`, the local Python test and Node build at `6f213584`, and the final TypeScript/Jest delta at `4f2d5c27`. The final product tree excludes those five private EC2 runner files and adds six generated TypeDoc files plus one hidden draft usage section in the computed-column refresh guide. The Node delta corrected a private TypeScript command and reused the unchanged native addon.
This commit is contained in:
+8
-3
@@ -16,9 +16,10 @@ use query::{FTSQuery, HybridQuery, Query, VectorQuery};
|
||||
use session::Session;
|
||||
use table::{
|
||||
AddColumnsResult, AddResult, AlterColumnsResult, DeleteResult, DropColumnsResult, FtsToken,
|
||||
FunctionErrorFragment, FunctionErrorRecord, FunctionErrors, MergeResult, PyBlobFile,
|
||||
RTEWriteSpec, RefreshColumnResult, RefreshMaterializedViewResult, Table,
|
||||
UpdateFieldMetadataResult, UpdateResult,
|
||||
FunctionColumnAutoRefreshStatus, FunctionColumnLastRefreshJob, FunctionColumnStatus,
|
||||
FunctionColumnVersionStatus, FunctionErrorFragment, FunctionErrorRecord, FunctionErrors,
|
||||
MergeResult, PyBlobFile, RTEWriteSpec, RefreshColumnResult, RefreshMaterializedViewResult,
|
||||
Table, UpdateFieldMetadataResult, UpdateResult,
|
||||
};
|
||||
|
||||
pub mod arrow;
|
||||
@@ -117,6 +118,10 @@ pub fn _lancedb(py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> {
|
||||
m.add_class::<FunctionErrors>()?;
|
||||
m.add_class::<FunctionErrorRecord>()?;
|
||||
m.add_class::<FunctionErrorFragment>()?;
|
||||
m.add_class::<FunctionColumnStatus>()?;
|
||||
m.add_class::<FunctionColumnVersionStatus>()?;
|
||||
m.add_class::<FunctionColumnLastRefreshJob>()?;
|
||||
m.add_class::<FunctionColumnAutoRefreshStatus>()?;
|
||||
m.add_class::<RefreshMaterializedViewResult>()?;
|
||||
m.add_class::<AlterColumnsResult>()?;
|
||||
m.add_class::<UpdateFieldMetadataResult>()?;
|
||||
|
||||
Reference in New Issue
Block a user