mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-21 10:03:58 +00:00
feat: expose the remaining refresh options on the SDKs (#4890)
The server's backfill request takes a row filter, failure limits, per-call timeouts and worker sizing, and SQL exposes all of them through `REFRESH COLUMN ... WHERE ... WITH (...)`, but the typed SDKs could send only batch_size, GPU placement and the retry and error policies. Anyone needing a filtered refresh or a bounded skip policy had to drop to SQL. This adds `where`, `max_failed_fraction`, `max_failed_rows`, `row_timeout_seconds`, `init_timeout_seconds`, `num_workers` and `max_executor_processes_per_worker` to Python, Rust and Node, passed through unchanged. `RefreshColumnOptions` now serializes straight into the request body, so the three SDKs share one producer. Node gets a `RefreshColumnOptions` object, which also carries the options it lacked before. A LanceDB Embedded refresh rejects every option other than batch_size and GPU placement, naming the option.
This commit is contained in:
@@ -1082,7 +1082,7 @@ for await (const batch of table.query()) {
|
||||
### refreshColumn()
|
||||
|
||||
```ts
|
||||
abstract refreshColumn(column): Promise<RefreshColumnResult>
|
||||
abstract refreshColumn(column, options?): Promise<RefreshColumnResult>
|
||||
```
|
||||
|
||||
Fill the rows of a computed column that hold no value yet.
|
||||
@@ -1098,6 +1098,10 @@ Enterprise a refresh runs as a server job, through [Table#refreshColumnAsync](Ta
|
||||
* **column**: `string`
|
||||
The name of the computed column to fill.
|
||||
|
||||
* **options?**: [`RefreshColumnOptions`](../interfaces/RefreshColumnOptions.md)
|
||||
Refresh options, all LanceDB
|
||||
Enterprise only; see [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<[`RefreshColumnResult`](../interfaces/RefreshColumnResult.md)>
|
||||
@@ -1110,7 +1114,7 @@ number of rows filled and the new version number of the table.
|
||||
### refreshColumnAsync()
|
||||
|
||||
```ts
|
||||
abstract refreshColumnAsync(column): Promise<Job>
|
||||
abstract refreshColumnAsync(column, options?): Promise<Job>
|
||||
```
|
||||
|
||||
Like [Table#refreshColumn](Table.md#refreshcolumn), but returns a handle to the refresh
|
||||
@@ -1130,6 +1134,16 @@ LanceDB Enterprise it is the server's backfill job.
|
||||
* **column**: `string`
|
||||
The name of the computed column to fill.
|
||||
|
||||
* **options?**: [`RefreshColumnOptions`](../interfaces/RefreshColumnOptions.md)
|
||||
Refresh options, mirroring the SQL
|
||||
`REFRESH COLUMN ... WHERE ... WITH (...)` form: `where` narrows the rows,
|
||||
`errorPolicy`, `maxFailedFraction`, `maxFailedRows`, `rowTimeoutSeconds`
|
||||
and `initTimeoutSeconds` govern failing rows, and `batchSize`,
|
||||
`gpuClass`, `gpuCount`, `numWorkers` and `maxExecutorProcessesPerWorker`
|
||||
size the workers. All are LanceDB Enterprise only; a LanceDB Embedded
|
||||
refresh rejects every option other than `batchSize`, `gpuClass` and
|
||||
`gpuCount`.
|
||||
|
||||
#### Returns
|
||||
|
||||
`Promise`<[`Job`](Job.md)>
|
||||
@@ -1137,7 +1151,12 @@ LanceDB Enterprise it is the server's backfill job.
|
||||
#### Example
|
||||
|
||||
```ts
|
||||
const job = await table.refreshColumnAsync("doubled");
|
||||
const job = await table.refreshColumnAsync("embedding", {
|
||||
where: "published_at > '2026-01-01'",
|
||||
errorPolicy: "skip",
|
||||
maxFailedFraction: 0.01,
|
||||
rowTimeoutSeconds: 30,
|
||||
});
|
||||
await job.wait();
|
||||
console.log(await job.status()); // "finished"
|
||||
```
|
||||
|
||||
@@ -122,6 +122,7 @@
|
||||
- [QueryExecutionOptions](interfaces/QueryExecutionOptions.md)
|
||||
- [RTEStats](interfaces/RTEStats.md)
|
||||
- [RTEWriteSpec](interfaces/RTEWriteSpec.md)
|
||||
- [RefreshColumnOptions](interfaces/RefreshColumnOptions.md)
|
||||
- [RefreshColumnResult](interfaces/RefreshColumnResult.md)
|
||||
- [RefreshMaterializedViewResult](interfaces/RefreshMaterializedViewResult.md)
|
||||
- [RemovalStats](interfaces/RemovalStats.md)
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
[**@lancedb/lancedb**](../README.md) • **Docs**
|
||||
|
||||
***
|
||||
|
||||
[@lancedb/lancedb](../globals.md) / RefreshColumnOptions
|
||||
|
||||
# Interface: RefreshColumnOptions
|
||||
|
||||
Options for `Table.refreshColumn` and `Table.refreshColumnAsync`. Every
|
||||
option is LanceDB Enterprise only; a LanceDB Embedded refresh ignores
|
||||
`batchSize`, `gpuClass` and `gpuCount` and rejects the rest.
|
||||
|
||||
## Properties
|
||||
|
||||
### batchSize?
|
||||
|
||||
```ts
|
||||
optional batchSize: number;
|
||||
```
|
||||
|
||||
Maximum rows per Function call; must be greater than zero.
|
||||
|
||||
***
|
||||
|
||||
### errorPolicy?
|
||||
|
||||
```ts
|
||||
optional errorPolicy: string;
|
||||
```
|
||||
|
||||
`"fail"` (default) stops on a row error; `"skip"` writes null and
|
||||
records the error, subject to the failure limits.
|
||||
|
||||
***
|
||||
|
||||
### gpuClass?
|
||||
|
||||
```ts
|
||||
optional gpuClass: string;
|
||||
```
|
||||
|
||||
GPU class for Function workers.
|
||||
|
||||
***
|
||||
|
||||
### gpuCount?
|
||||
|
||||
```ts
|
||||
optional gpuCount: number;
|
||||
```
|
||||
|
||||
Positive aggregate GPU budget for Function workers.
|
||||
|
||||
***
|
||||
|
||||
### initTimeoutSeconds?
|
||||
|
||||
```ts
|
||||
optional initTimeoutSeconds: number;
|
||||
```
|
||||
|
||||
Deadline for Function initialization, in seconds.
|
||||
|
||||
***
|
||||
|
||||
### maxExecutorProcessesPerWorker?
|
||||
|
||||
```ts
|
||||
optional maxExecutorProcessesPerWorker: number;
|
||||
```
|
||||
|
||||
Cap on executor processes per Function worker.
|
||||
|
||||
***
|
||||
|
||||
### maxFailedFraction?
|
||||
|
||||
```ts
|
||||
optional maxFailedFraction: number;
|
||||
```
|
||||
|
||||
Under a skip policy, fail the job once more than this share of rows
|
||||
(0 to 1) has failed.
|
||||
|
||||
***
|
||||
|
||||
### maxFailedRows?
|
||||
|
||||
```ts
|
||||
optional maxFailedRows: number;
|
||||
```
|
||||
|
||||
Under a skip policy, fail the job once more than this many rows have
|
||||
failed.
|
||||
|
||||
***
|
||||
|
||||
### numWorkers?
|
||||
|
||||
```ts
|
||||
optional numWorkers: number;
|
||||
```
|
||||
|
||||
Worker count for an expression column; Function columns reject it.
|
||||
|
||||
***
|
||||
|
||||
### rowTimeoutSeconds?
|
||||
|
||||
```ts
|
||||
optional rowTimeoutSeconds: number;
|
||||
```
|
||||
|
||||
Deadline for one Function call, in seconds.
|
||||
|
||||
***
|
||||
|
||||
### where?
|
||||
|
||||
```ts
|
||||
optional where: string;
|
||||
```
|
||||
|
||||
Only refresh the rows matching this SQL filter.
|
||||
@@ -241,10 +241,59 @@ GPU class discovery.
|
||||
|
||||
<Note>
|
||||
The Rust GPU discovery and refresh controls are not in the latest Rust
|
||||
release. TypeScript does not expose GPU class discovery or placement; a
|
||||
tracking issue is needed before adding equivalent controls.
|
||||
release. TypeScript does not expose GPU class discovery; pass `gpuClass`
|
||||
and `gpuCount` in the `refreshColumnAsync` options.
|
||||
</Note>
|
||||
|
||||
The job APIs also take the row filter and failure handling of the SQL
|
||||
`REFRESH COLUMN ... WHERE ... WITH (...)` form, so a refresh can be narrowed
|
||||
to new rows and told to skip rows whose Function call fails:
|
||||
|
||||
<CodeGroup>
|
||||
```python
|
||||
job = table.refresh_column_async(
|
||||
"embedding",
|
||||
where="published_at > '2026-01-01'",
|
||||
error_policy="skip",
|
||||
max_failed_fraction=0.01,
|
||||
row_timeout_seconds=30,
|
||||
)
|
||||
```
|
||||
|
||||
```typescript
|
||||
const job = await table.refreshColumnAsync("embedding", {
|
||||
where: "published_at > '2026-01-01'",
|
||||
errorPolicy: "skip",
|
||||
maxFailedFraction: 0.01,
|
||||
rowTimeoutSeconds: 30,
|
||||
});
|
||||
```
|
||||
|
||||
```rust
|
||||
use lancedb::{function::FunctionErrorPolicy, table::RefreshColumnOptions};
|
||||
|
||||
let _job = table
|
||||
.refresh_column_async_with_options(
|
||||
"embedding",
|
||||
RefreshColumnOptions::default()
|
||||
.where_clause("published_at > '2026-01-01'")
|
||||
.error_policy(FunctionErrorPolicy::Skip)
|
||||
.max_failed_fraction(0.01)
|
||||
.row_timeout_seconds(30.0),
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
`max_failed_rows`, `init_timeout_seconds`, `num_workers` and
|
||||
`max_executor_processes_per_worker` are accepted the same way; the
|
||||
[SQL refresh options](/sql/dml/refresh-column#options) describe each one.
|
||||
The SDK rejects a blank `where`, a `max_failed_fraction` outside 0 to 1, a
|
||||
timeout that is not a positive number and a zero worker or process count
|
||||
before sending the request; the server validates everything else. A LanceDB
|
||||
Embedded refresh rejects every option other than `batch_size`, `gpu_class`
|
||||
and `gpu_count`.
|
||||
|
||||
## When a refresh fails {#failures}
|
||||
|
||||
What a failure keeps depends on where the refresh ran. The local, blocking
|
||||
|
||||
Reference in New Issue
Block a user