mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-15 15:16:53 +00:00
feat: expose Function publication intervals in refresh options (#4928)
Python refresh APIs currently reject `publication_interval_rows`, so callers cannot request incremental Function output publication through the SDK. Add the optional typed parameter through the sync/async Python APIs, PyO3, and Rust `RefreshColumnOptions`, and serialize it as a top-level Enterprise refresh option. An omitted value preserves the existing wire request and whole-fragment default. Positive values set a publication threshold; zero and negative Python values fail before a request is sent. Embedded refreshes reject the Enterprise-only option. Documentation describes a threshold rather than an exact number of rows per publication. Validation on the designated development host: - A matching baseline build reproduced the missing public keyword argument in both sync and async submission tests. - The updated PyO3 build passed 54 focused Python tests; Rust payload, invalid-option, and default-wire tests each passed. Rust formatting and Python Ruff checks passed. - The single signed delivery commit differs from that tested implementation only in a Rustdoc correction describing Embedded rejection; formatting passed again. No dependency pins or lockfiles changed. The matching SDK is prepared for the separate server validation of failure/cancellation retention and missing-only refill. Those end-to-end results are still pending and are not implied by these SDK tests. Refs [ENT-3733](https://linear.app/lancedb/issue/ENT-3733).
This commit is contained in:
@@ -261,6 +261,38 @@ release. TypeScript does not expose GPU class discovery; pass `gpuClass`
|
||||
and `gpuCount` in the `refreshColumnAsync` options.
|
||||
</Note>
|
||||
|
||||
Completed Function outputs publish with their fragment by default. To opt into
|
||||
a row-count publication threshold, set `publication_interval_rows` in Python or
|
||||
`publication_interval_rows()` in Rust:
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
job = table.refresh_column_async(
|
||||
"embedding",
|
||||
publication_interval_rows=60,
|
||||
)
|
||||
job.wait()
|
||||
```
|
||||
|
||||
```rust Rust icon="rust"
|
||||
use lancedb::table::RefreshColumnOptions;
|
||||
|
||||
let _job = table
|
||||
.refresh_column_async_with_options(
|
||||
"embedding",
|
||||
RefreshColumnOptions::default().publication_interval_rows(60),
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The threshold is opt-in and must be greater than zero. The number of rows in an
|
||||
individual publication can differ from this value depending on completed work
|
||||
and Function batching; this is not an exact-N contract. Omit it to keep
|
||||
whole-fragment publication.
|
||||
|
||||
The TypeScript SDK does not expose the publication threshold option yet.
|
||||
|
||||
The job APIs also take the row filter and failure handling of the SQL
|
||||
`REFRESH COLUMN ... WHERE ... WITH (...)` form, so a refresh can be narrowed
|
||||
to new rows (see [what a filter recomputes](#partial)) and told to skip rows
|
||||
|
||||
Reference in New Issue
Block a user