feat: expose exact Function column freshness (#4958)

## Problem

Function-backed columns need an exact `fresh`, `stale`,
`never_computed`, and
`failed` partition for the table version a caller is reading. The
existing
error listing is capped, and composing it with job and schema listings
cannot
prove which rows currently failed.

## Behavior

- Adds a strict freshness reader beside the existing permissive planner.
It
  distinguishes an absent certificate from legacy evidence that omits
  failures, and from a current certificate carrying failure coverage.
- Extends the immutable freshness sidecar with per-fragment known and
failed
  row bitmaps. Partial or `WHERE` publications cover only observed rows;
  untouched rows stay unknown. Deleted rows do not block classification.
- Adds failure-aware whole-fragment and row-repair writers. A rejected
older
attempt cannot overwrite failure evidence already recorded for a newer
input
  signature.
- Adds `column_status`, `column_statuses`, and `wait_fresh` to Python
and the
  corresponding Node methods. The client calls the server's exact status
endpoint, validates the four-way partition and one-version guarantee,
and
preserves the complete Function version, last-job, and automatic-refresh
  metadata.
- Adds an MDX-commented draft usage section to the Enterprise
computed-column
refresh guide, covering Python, Rust, and TypeScript and remaining
hidden
  until the server integration ships.
- Sends `column`, `branch`, and exact `version` selectors to the status
route.
A pinned handle rejects a nonempty response from any other table
version;
  latest handles keep the existing read-freshness headers.
- Retries `wait_fresh` only for HTTP 503 with
`function_column_status_finalization_pending`. Existing failed rows
remain
  visible and do not keep the wait pending; other errors propagate.
- Models the shared scheduler projection, including `up_to_date`, real
`blocked` states, their reason, failure streak, and effective cooldown.

## Validation

This is the SDK side of ENT-3563. Sophon's failure-coverage publication
and
status endpoint are a separate integration; deployed end-to-end and
whole-ticket
acceptance remain pending.

- Rust: 9 exact freshness/status tests passed, plus 1 exact
latest-handle
  freshness-header test. The pinned transport test was extended with a
wrong-version response and rerun at `ebd6649a` (1 passed). Coverage
includes untouched partial rows,
  selected-row failure clearing, cross-generation evidence, typed and
non-retried errors, `up_to_date`, blocked scheduler fields, and
exact-version
  requests.
- Python: 2 native PyO3 remote-transport tests passed together,
including
typed retry; one local facade test reaches the existing unsupported
boundary.
- Node: the NAPI addon built successfully, TypeScript and Biome passed,
and the
selected native remote-transport test passed 1/0, including
exact-version
  selection and blocked scheduler metadata.
- `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile,
Biome, and
  TypeScript checks passed.

Validation is mapped by the relevant unchanged source rather than
treated as one
final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header
and Python
remote tests at `31e68b7d`, the local Python test and Node build at
`6f213584`, and
the final TypeScript/Jest delta at `4f2d5c27`. The final product tree
excludes those five private EC2 runner files and adds
six generated TypeDoc files plus one hidden draft usage section in the
computed-column refresh guide. The Node delta corrected a private
TypeScript
command and reused the unchanged native addon.
This commit is contained in:
Xuanwo
2026-11-11 19:12:06 +08:00
committed by GitHub
parent 34e8b8087b
commit 7cc3904a7e
25 changed files with 3055 additions and 41 deletions
+64
View File
@@ -704,6 +704,47 @@ so this is safe to call repeatedly.
***
### functionColumnStatus()
```ts
abstract functionColumnStatus(column): Promise<FunctionColumnStatus>
```
Return exact freshness status for one Function-backed output column.
#### Parameters
* **column**: `string`
#### Returns
`Promise`&lt;[`FunctionColumnStatus`](../interfaces/FunctionColumnStatus.md)&gt;
***
### functionColumnStatuses()
```ts
abstract functionColumnStatuses(column?): Promise<FunctionColumnStatus[]>
```
Return exact freshness status for Function-backed output columns.
All results describe one server table version. Omit `column` for every
Function output. Rows already failed remain visible and do not prevent
`waitFresh` from completing. A handle pinned to an exact version requests
that version; a latest handle retains normal read-freshness behavior.
#### Parameters
* **column?**: `string`
#### Returns
`Promise`&lt;[`FunctionColumnStatus`](../interfaces/FunctionColumnStatus.md)[]&gt;
***
### functionErrors()
```ts
@@ -1598,3 +1639,26 @@ Waits for asynchronous indexing to complete on the table.
#### Returns
`Promise`&lt;`void`&gt;
***
### waitFresh()
```ts
abstract waitFresh(column, timeoutSeconds?): Promise<FunctionColumnStatus>
```
Poll until no stale or never-computed rows remain.
Only HTTP 503 with the exact finalization-pending error code is retried;
every other error propagates. Failed rows do not block completion.
#### Parameters
* **column**: `string`
* **timeoutSeconds?**: `number`
#### Returns
`Promise`&lt;[`FunctionColumnStatus`](../interfaces/FunctionColumnStatus.md)&gt;
+4
View File
@@ -95,6 +95,10 @@
- [FtsToken](interfaces/FtsToken.md)
- [FullTextQuery](interfaces/FullTextQuery.md)
- [FullTextSearchOptions](interfaces/FullTextSearchOptions.md)
- [FunctionColumnAutoRefreshStatus](interfaces/FunctionColumnAutoRefreshStatus.md)
- [FunctionColumnLastRefreshJob](interfaces/FunctionColumnLastRefreshJob.md)
- [FunctionColumnStatus](interfaces/FunctionColumnStatus.md)
- [FunctionColumnVersionStatus](interfaces/FunctionColumnVersionStatus.md)
- [FunctionErrorFragment](interfaces/FunctionErrorFragment.md)
- [FunctionErrorRecord](interfaces/FunctionErrorRecord.md)
- [FunctionErrors](interfaces/FunctionErrors.md)
@@ -0,0 +1,95 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / FunctionColumnAutoRefreshStatus
# Interface: FunctionColumnAutoRefreshStatus
Automatic-refresh projection for a Function output.
## Properties
### activeJobId?
```ts
optional activeJobId: string;
```
***
### blockedReason?
```ts
optional blockedReason:
| "table_has_mem_wal"
| "inputs_not_settled"
| "dependency_in_flight"
| "table_capacity"
| "cancelled"
| "preparation_backoff";
```
***
### effectiveCooldownMillis?
```ts
optional effectiveCooldownMillis: bigint;
```
***
### enabled
```ts
enabled: boolean;
```
***
### failureStreak
```ts
failureStreak: number;
```
***
### intervalSeconds?
```ts
optional intervalSeconds: bigint;
```
***
### nextRunAtMillis
```ts
nextRunAtMillis: null | bigint;
```
***
### nextRunAtReason?
```ts
optional nextRunAtReason: string;
```
***
### state
```ts
state:
| "running"
| "disabled"
| "ready"
| "up_to_date"
| "waiting_interval"
| "waiting_backoff"
| "blocked"
| "unobservable";
```
@@ -0,0 +1,41 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / FunctionColumnLastRefreshJob
# Interface: FunctionColumnLastRefreshJob
Latest matching refresh Job projected for a Function output.
## Properties
### automatic?
```ts
optional automatic: boolean;
```
***
### completedAtMillis?
```ts
optional completedAtMillis: bigint;
```
***
### jobId
```ts
jobId: string;
```
***
### state
```ts
state: string;
```
@@ -0,0 +1,97 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / FunctionColumnStatus
# Interface: FunctionColumnStatus
Exact four-way status of one Function output at one table version.
## Properties
### autoRefresh
```ts
autoRefresh: FunctionColumnAutoRefreshStatus;
```
***
### bindingId
```ts
bindingId: string;
```
***
### column
```ts
column: string;
```
***
### failed
```ts
failed: bigint;
```
***
### fresh
```ts
fresh: bigint;
```
***
### function
```ts
function: FunctionColumnVersionStatus;
```
***
### lastRefreshJob?
```ts
optional lastRefreshJob: FunctionColumnLastRefreshJob;
```
***
### neverComputed
```ts
neverComputed: bigint;
```
***
### rows
```ts
rows: bigint;
```
***
### stale
```ts
stale: bigint;
```
***
### tableVersion
```ts
tableVersion: bigint;
```
@@ -0,0 +1,57 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / FunctionColumnVersionStatus
# Interface: FunctionColumnVersionStatus
Exact immutable Function revision attached to a table output.
## Properties
### location
```ts
location: string;
```
***
### manifestDigest
```ts
manifestDigest: string;
```
***
### name
```ts
name: string;
```
***
### namespacePath
```ts
namespacePath: string[];
```
***
### objectId
```ts
objectId: string;
```
***
### version
```ts
version: string;
```
+8
View File
@@ -523,6 +523,14 @@ with BlobReader(read_range, len(payload)) as reader:
::: lancedb.FunctionErrorFragment
::: lancedb.FunctionColumnStatus
::: lancedb.FunctionColumnVersionStatus
::: lancedb.FunctionColumnLastRefreshJob
::: lancedb.FunctionColumnAutoRefreshStatus
::: lancedb.job.Job
::: lancedb.job.AsyncJob