mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-21 10:03:58 +00:00
feat: expose exact Function column freshness (#4958)
## Problem Function-backed columns need an exact `fresh`, `stale`, `never_computed`, and `failed` partition for the table version a caller is reading. The existing error listing is capped, and composing it with job and schema listings cannot prove which rows currently failed. ## Behavior - Adds a strict freshness reader beside the existing permissive planner. It distinguishes an absent certificate from legacy evidence that omits failures, and from a current certificate carrying failure coverage. - Extends the immutable freshness sidecar with per-fragment known and failed row bitmaps. Partial or `WHERE` publications cover only observed rows; untouched rows stay unknown. Deleted rows do not block classification. - Adds failure-aware whole-fragment and row-repair writers. A rejected older attempt cannot overwrite failure evidence already recorded for a newer input signature. - Adds `column_status`, `column_statuses`, and `wait_fresh` to Python and the corresponding Node methods. The client calls the server's exact status endpoint, validates the four-way partition and one-version guarantee, and preserves the complete Function version, last-job, and automatic-refresh metadata. - Adds an MDX-commented draft usage section to the Enterprise computed-column refresh guide, covering Python, Rust, and TypeScript and remaining hidden until the server integration ships. - Sends `column`, `branch`, and exact `version` selectors to the status route. A pinned handle rejects a nonempty response from any other table version; latest handles keep the existing read-freshness headers. - Retries `wait_fresh` only for HTTP 503 with `function_column_status_finalization_pending`. Existing failed rows remain visible and do not keep the wait pending; other errors propagate. - Models the shared scheduler projection, including `up_to_date`, real `blocked` states, their reason, failure streak, and effective cooldown. ## Validation This is the SDK side of ENT-3563. Sophon's failure-coverage publication and status endpoint are a separate integration; deployed end-to-end and whole-ticket acceptance remain pending. - Rust: 9 exact freshness/status tests passed, plus 1 exact latest-handle freshness-header test. The pinned transport test was extended with a wrong-version response and rerun at `ebd6649a` (1 passed). Coverage includes untouched partial rows, selected-row failure clearing, cross-generation evidence, typed and non-retried errors, `up_to_date`, blocked scheduler fields, and exact-version requests. - Python: 2 native PyO3 remote-transport tests passed together, including typed retry; one local facade test reaches the existing unsupported boundary. - Node: the NAPI addon built successfully, TypeScript and Biome passed, and the selected native remote-transport test passed 1/0, including exact-version selection and blocked scheduler metadata. - `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile, Biome, and TypeScript checks passed. Validation is mapped by the relevant unchanged source rather than treated as one final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header and Python remote tests at `31e68b7d`, the local Python test and Node build at `6f213584`, and the final TypeScript/Jest delta at `4f2d5c27`. The final product tree excludes those five private EC2 runner files and adds six generated TypeDoc files plus one hidden draft usage section in the computed-column refresh guide. The Node delta corrected a private TypeScript command and reused the unchanged native addon.
This commit is contained in:
@@ -704,6 +704,47 @@ so this is safe to call repeatedly.
|
||||
|
||||
***
|
||||
|
||||
### functionColumnStatus()
|
||||
|
||||
```ts
|
||||
abstract functionColumnStatus(column): Promise<FunctionColumnStatus>
|
||||
```
|
||||
|
||||
Return exact freshness status for one Function-backed output column.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **column**: `string`
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<[`FunctionColumnStatus`](../interfaces/FunctionColumnStatus.md)>
|
||||
|
||||
***
|
||||
|
||||
### functionColumnStatuses()
|
||||
|
||||
```ts
|
||||
abstract functionColumnStatuses(column?): Promise<FunctionColumnStatus[]>
|
||||
```
|
||||
|
||||
Return exact freshness status for Function-backed output columns.
|
||||
|
||||
All results describe one server table version. Omit `column` for every
|
||||
Function output. Rows already failed remain visible and do not prevent
|
||||
`waitFresh` from completing. A handle pinned to an exact version requests
|
||||
that version; a latest handle retains normal read-freshness behavior.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **column?**: `string`
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<[`FunctionColumnStatus`](../interfaces/FunctionColumnStatus.md)[]>
|
||||
|
||||
***
|
||||
|
||||
### functionErrors()
|
||||
|
||||
```ts
|
||||
@@ -1598,3 +1639,26 @@ Waits for asynchronous indexing to complete on the table.
|
||||
#### Returns
|
||||
|
||||
`Promise`<`void`>
|
||||
|
||||
***
|
||||
|
||||
### waitFresh()
|
||||
|
||||
```ts
|
||||
abstract waitFresh(column, timeoutSeconds?): Promise<FunctionColumnStatus>
|
||||
```
|
||||
|
||||
Poll until no stale or never-computed rows remain.
|
||||
|
||||
Only HTTP 503 with the exact finalization-pending error code is retried;
|
||||
every other error propagates. Failed rows do not block completion.
|
||||
|
||||
#### Parameters
|
||||
|
||||
* **column**: `string`
|
||||
|
||||
* **timeoutSeconds?**: `number`
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<[`FunctionColumnStatus`](../interfaces/FunctionColumnStatus.md)>
|
||||
|
||||
@@ -95,6 +95,10 @@
|
||||
- [FtsToken](interfaces/FtsToken.md)
|
||||
- [FullTextQuery](interfaces/FullTextQuery.md)
|
||||
- [FullTextSearchOptions](interfaces/FullTextSearchOptions.md)
|
||||
- [FunctionColumnAutoRefreshStatus](interfaces/FunctionColumnAutoRefreshStatus.md)
|
||||
- [FunctionColumnLastRefreshJob](interfaces/FunctionColumnLastRefreshJob.md)
|
||||
- [FunctionColumnStatus](interfaces/FunctionColumnStatus.md)
|
||||
- [FunctionColumnVersionStatus](interfaces/FunctionColumnVersionStatus.md)
|
||||
- [FunctionErrorFragment](interfaces/FunctionErrorFragment.md)
|
||||
- [FunctionErrorRecord](interfaces/FunctionErrorRecord.md)
|
||||
- [FunctionErrors](interfaces/FunctionErrors.md)
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / FunctionColumnAutoRefreshStatus
|
||||
|
||||
# Interface: FunctionColumnAutoRefreshStatus
|
||||
|
||||
Automatic-refresh projection for a Function output.
|
||||
|
||||
## Properties
|
||||
|
||||
### activeJobId?
|
||||
|
||||
```ts
|
||||
optional activeJobId: string;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### blockedReason?
|
||||
|
||||
```ts
|
||||
optional blockedReason:
|
||||
| "table_has_mem_wal"
|
||||
| "inputs_not_settled"
|
||||
| "dependency_in_flight"
|
||||
| "table_capacity"
|
||||
| "cancelled"
|
||||
| "preparation_backoff";
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### effectiveCooldownMillis?
|
||||
|
||||
```ts
|
||||
optional effectiveCooldownMillis: bigint;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### enabled
|
||||
|
||||
```ts
|
||||
enabled: boolean;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### failureStreak
|
||||
|
||||
```ts
|
||||
failureStreak: number;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### intervalSeconds?
|
||||
|
||||
```ts
|
||||
optional intervalSeconds: bigint;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### nextRunAtMillis
|
||||
|
||||
```ts
|
||||
nextRunAtMillis: null | bigint;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### nextRunAtReason?
|
||||
|
||||
```ts
|
||||
optional nextRunAtReason: string;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### state
|
||||
|
||||
```ts
|
||||
state:
|
||||
| "running"
|
||||
| "disabled"
|
||||
| "ready"
|
||||
| "up_to_date"
|
||||
| "waiting_interval"
|
||||
| "waiting_backoff"
|
||||
| "blocked"
|
||||
| "unobservable";
|
||||
```
|
||||
@@ -0,0 +1,41 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / FunctionColumnLastRefreshJob
|
||||
|
||||
# Interface: FunctionColumnLastRefreshJob
|
||||
|
||||
Latest matching refresh Job projected for a Function output.
|
||||
|
||||
## Properties
|
||||
|
||||
### automatic?
|
||||
|
||||
```ts
|
||||
optional automatic: boolean;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### completedAtMillis?
|
||||
|
||||
```ts
|
||||
optional completedAtMillis: bigint;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### jobId
|
||||
|
||||
```ts
|
||||
jobId: string;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### state
|
||||
|
||||
```ts
|
||||
state: string;
|
||||
```
|
||||
@@ -0,0 +1,97 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / FunctionColumnStatus
|
||||
|
||||
# Interface: FunctionColumnStatus
|
||||
|
||||
Exact four-way status of one Function output at one table version.
|
||||
|
||||
## Properties
|
||||
|
||||
### autoRefresh
|
||||
|
||||
```ts
|
||||
autoRefresh: FunctionColumnAutoRefreshStatus;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### bindingId
|
||||
|
||||
```ts
|
||||
bindingId: string;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### column
|
||||
|
||||
```ts
|
||||
column: string;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### failed
|
||||
|
||||
```ts
|
||||
failed: bigint;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### fresh
|
||||
|
||||
```ts
|
||||
fresh: bigint;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### function
|
||||
|
||||
```ts
|
||||
function: FunctionColumnVersionStatus;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### lastRefreshJob?
|
||||
|
||||
```ts
|
||||
optional lastRefreshJob: FunctionColumnLastRefreshJob;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### neverComputed
|
||||
|
||||
```ts
|
||||
neverComputed: bigint;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### rows
|
||||
|
||||
```ts
|
||||
rows: bigint;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### stale
|
||||
|
||||
```ts
|
||||
stale: bigint;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### tableVersion
|
||||
|
||||
```ts
|
||||
tableVersion: bigint;
|
||||
```
|
||||
@@ -0,0 +1,57 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / FunctionColumnVersionStatus
|
||||
|
||||
# Interface: FunctionColumnVersionStatus
|
||||
|
||||
Exact immutable Function revision attached to a table output.
|
||||
|
||||
## Properties
|
||||
|
||||
### location
|
||||
|
||||
```ts
|
||||
location: string;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### manifestDigest
|
||||
|
||||
```ts
|
||||
manifestDigest: string;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### name
|
||||
|
||||
```ts
|
||||
name: string;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### namespacePath
|
||||
|
||||
```ts
|
||||
namespacePath: string[];
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### objectId
|
||||
|
||||
```ts
|
||||
objectId: string;
|
||||
```
|
||||
|
||||
***
|
||||
|
||||
### version
|
||||
|
||||
```ts
|
||||
version: string;
|
||||
```
|
||||
@@ -523,6 +523,14 @@ with BlobReader(read_range, len(payload)) as reader:
|
||||
|
||||
::: lancedb.FunctionErrorFragment
|
||||
|
||||
::: lancedb.FunctionColumnStatus
|
||||
|
||||
::: lancedb.FunctionColumnVersionStatus
|
||||
|
||||
::: lancedb.FunctionColumnLastRefreshJob
|
||||
|
||||
::: lancedb.FunctionColumnAutoRefreshStatus
|
||||
|
||||
::: lancedb.job.Job
|
||||
|
||||
::: lancedb.job.AsyncJob
|
||||
|
||||
Reference in New Issue
Block a user