mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
feat: expose exact Function column freshness (#4958)
## Problem Function-backed columns need an exact `fresh`, `stale`, `never_computed`, and `failed` partition for the table version a caller is reading. The existing error listing is capped, and composing it with job and schema listings cannot prove which rows currently failed. ## Behavior - Adds a strict freshness reader beside the existing permissive planner. It distinguishes an absent certificate from legacy evidence that omits failures, and from a current certificate carrying failure coverage. - Extends the immutable freshness sidecar with per-fragment known and failed row bitmaps. Partial or `WHERE` publications cover only observed rows; untouched rows stay unknown. Deleted rows do not block classification. - Adds failure-aware whole-fragment and row-repair writers. A rejected older attempt cannot overwrite failure evidence already recorded for a newer input signature. - Adds `column_status`, `column_statuses`, and `wait_fresh` to Python and the corresponding Node methods. The client calls the server's exact status endpoint, validates the four-way partition and one-version guarantee, and preserves the complete Function version, last-job, and automatic-refresh metadata. - Adds an MDX-commented draft usage section to the Enterprise computed-column refresh guide, covering Python, Rust, and TypeScript and remaining hidden until the server integration ships. - Sends `column`, `branch`, and exact `version` selectors to the status route. A pinned handle rejects a nonempty response from any other table version; latest handles keep the existing read-freshness headers. - Retries `wait_fresh` only for HTTP 503 with `function_column_status_finalization_pending`. Existing failed rows remain visible and do not keep the wait pending; other errors propagate. - Models the shared scheduler projection, including `up_to_date`, real `blocked` states, their reason, failure streak, and effective cooldown. ## Validation This is the SDK side of ENT-3563. Sophon's failure-coverage publication and status endpoint are a separate integration; deployed end-to-end and whole-ticket acceptance remain pending. - Rust: 9 exact freshness/status tests passed, plus 1 exact latest-handle freshness-header test. The pinned transport test was extended with a wrong-version response and rerun at `ebd6649a` (1 passed). Coverage includes untouched partial rows, selected-row failure clearing, cross-generation evidence, typed and non-retried errors, `up_to_date`, blocked scheduler fields, and exact-version requests. - Python: 2 native PyO3 remote-transport tests passed together, including typed retry; one local facade test reaches the existing unsupported boundary. - Node: the NAPI addon built successfully, TypeScript and Biome passed, and the selected native remote-transport test passed 1/0, including exact-version selection and blocked scheduler metadata. - `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile, Biome, and TypeScript checks passed. Validation is mapped by the relevant unchanged source rather than treated as one final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header and Python remote tests at `31e68b7d`, the local Python test and Node build at `6f213584`, and the final TypeScript/Jest delta at `4f2d5c27`. The final product tree excludes those five private EC2 runner files and adds six generated TypeDoc files plus one hidden draft usage section in the computed-column refresh guide. The Node delta corrected a private TypeScript command and reused the unchanged native addon.
This commit is contained in:
@@ -534,6 +534,55 @@ impl Table {
|
||||
Ok(errors.into())
|
||||
}
|
||||
|
||||
#[napi(catch_unwind)]
|
||||
pub async fn function_column_statuses(
|
||||
&self,
|
||||
column: Option<String>,
|
||||
) -> napi::Result<Vec<FunctionColumnStatus>> {
|
||||
let statuses = self
|
||||
.inner_ref()?
|
||||
.function_column_statuses(column.as_deref())
|
||||
.await
|
||||
.default_error()?;
|
||||
Ok(statuses.into_iter().map(Into::into).collect())
|
||||
}
|
||||
|
||||
#[napi(catch_unwind)]
|
||||
pub async fn function_column_status(
|
||||
&self,
|
||||
column: String,
|
||||
) -> napi::Result<FunctionColumnStatus> {
|
||||
let status = self
|
||||
.inner_ref()?
|
||||
.function_column_status(&column)
|
||||
.await
|
||||
.default_error()?;
|
||||
Ok(status.into())
|
||||
}
|
||||
|
||||
#[napi(catch_unwind)]
|
||||
pub async fn wait_for_function_column(
|
||||
&self,
|
||||
column: String,
|
||||
timeout_seconds: Option<i64>,
|
||||
) -> napi::Result<FunctionColumnStatus> {
|
||||
let timeout = timeout_seconds
|
||||
.map(|seconds| {
|
||||
u64::try_from(seconds)
|
||||
.map(std::time::Duration::from_secs)
|
||||
.map_err(|_| {
|
||||
napi::Error::from_reason("timeoutSeconds must be a non-negative integer")
|
||||
})
|
||||
})
|
||||
.transpose()?;
|
||||
let status = self
|
||||
.inner_ref()?
|
||||
.wait_for_function_column(&column, timeout)
|
||||
.await
|
||||
.default_error()?;
|
||||
Ok(status.into())
|
||||
}
|
||||
|
||||
#[napi(catch_unwind)]
|
||||
pub async fn refresh_materialized_view(
|
||||
&self,
|
||||
@@ -1799,6 +1848,108 @@ pub struct FunctionErrors {
|
||||
pub truncated: bool,
|
||||
}
|
||||
|
||||
/// One exact immutable Function revision attached to a table output.
|
||||
#[napi(object)]
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct FunctionColumnVersionStatus {
|
||||
pub name: String,
|
||||
pub namespace_path: Vec<String>,
|
||||
pub object_id: String,
|
||||
pub location: String,
|
||||
pub version: String,
|
||||
pub manifest_digest: String,
|
||||
}
|
||||
|
||||
/// Latest matching refresh Job projected for a Function output.
|
||||
#[napi(object)]
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct FunctionColumnLastRefreshJob {
|
||||
pub job_id: String,
|
||||
pub state: String,
|
||||
pub automatic: Option<bool>,
|
||||
pub completed_at_millis: Option<BigInt>,
|
||||
}
|
||||
|
||||
/// Automatic-refresh projection for a Function output.
|
||||
#[napi(object)]
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct FunctionColumnAutoRefreshStatus {
|
||||
pub enabled: bool,
|
||||
pub interval_seconds: Option<BigInt>,
|
||||
pub active_job_id: Option<String>,
|
||||
pub state: String,
|
||||
pub failure_streak: u32,
|
||||
pub effective_cooldown_millis: Option<BigInt>,
|
||||
pub blocked_reason: Option<String>,
|
||||
pub next_run_at_millis: Option<BigInt>,
|
||||
pub next_run_at_reason: Option<String>,
|
||||
}
|
||||
|
||||
/// Exact four-way status of one Function output at one table version.
|
||||
#[napi(object)]
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct FunctionColumnStatus {
|
||||
pub column: String,
|
||||
pub binding_id: String,
|
||||
pub function: FunctionColumnVersionStatus,
|
||||
pub table_version: BigInt,
|
||||
pub rows: BigInt,
|
||||
pub fresh: BigInt,
|
||||
pub stale: BigInt,
|
||||
pub never_computed: BigInt,
|
||||
pub failed: BigInt,
|
||||
pub last_refresh_job: Option<FunctionColumnLastRefreshJob>,
|
||||
pub auto_refresh: FunctionColumnAutoRefreshStatus,
|
||||
}
|
||||
|
||||
impl From<lancedb::table::FunctionColumnStatus> for FunctionColumnStatus {
|
||||
fn from(status: lancedb::table::FunctionColumnStatus) -> Self {
|
||||
Self {
|
||||
column: status.column,
|
||||
binding_id: status.binding_id,
|
||||
function: FunctionColumnVersionStatus {
|
||||
name: status.function.name,
|
||||
namespace_path: status.function.namespace_path,
|
||||
object_id: status.function.object_id,
|
||||
location: status.function.location,
|
||||
version: status.function.version,
|
||||
manifest_digest: status.function.manifest_digest,
|
||||
},
|
||||
table_version: BigInt::from(status.table_version),
|
||||
rows: BigInt::from(status.rows),
|
||||
fresh: BigInt::from(status.fresh),
|
||||
stale: BigInt::from(status.stale),
|
||||
never_computed: BigInt::from(status.never_computed),
|
||||
failed: BigInt::from(status.failed),
|
||||
last_refresh_job: status
|
||||
.last_refresh_job
|
||||
.map(|job| FunctionColumnLastRefreshJob {
|
||||
job_id: job.job_id,
|
||||
state: job.state,
|
||||
automatic: job.automatic,
|
||||
completed_at_millis: job.completed_at_millis.map(BigInt::from),
|
||||
}),
|
||||
auto_refresh: FunctionColumnAutoRefreshStatus {
|
||||
enabled: status.auto_refresh.enabled,
|
||||
interval_seconds: status.auto_refresh.interval_seconds.map(BigInt::from),
|
||||
active_job_id: status.auto_refresh.active_job_id,
|
||||
state: status.auto_refresh.state.as_str().to_string(),
|
||||
failure_streak: status.auto_refresh.failure_streak,
|
||||
effective_cooldown_millis: status
|
||||
.auto_refresh
|
||||
.effective_cooldown_millis
|
||||
.map(BigInt::from),
|
||||
blocked_reason: status
|
||||
.auto_refresh
|
||||
.blocked_reason
|
||||
.map(|reason| reason.as_str().to_string()),
|
||||
next_run_at_millis: status.auto_refresh.next_run_at_millis.map(BigInt::from),
|
||||
next_run_at_reason: status.auto_refresh.next_run_at_reason,
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<lancedb::function::FunctionErrors> for FunctionErrors {
|
||||
fn from(errors: lancedb::function::FunctionErrors) -> Self {
|
||||
Self {
|
||||
|
||||
Reference in New Issue
Block a user