feat: expose Function publication intervals in refresh options (#4928)

Python refresh APIs currently reject `publication_interval_rows`, so
callers cannot request incremental Function output publication through
the SDK. Add the optional typed parameter through the sync/async Python
APIs, PyO3, and Rust `RefreshColumnOptions`, and serialize it as a
top-level Enterprise refresh option.

An omitted value preserves the existing wire request and whole-fragment
default. Positive values set a publication threshold; zero and negative
Python values fail before a request is sent. Embedded refreshes reject
the Enterprise-only option. Documentation describes a threshold rather
than an exact number of rows per publication.

Validation on the designated development host:
- A matching baseline build reproduced the missing public keyword
argument in both sync and async submission tests.
- The updated PyO3 build passed 54 focused Python tests; Rust payload,
invalid-option, and default-wire tests each passed. Rust formatting and
Python Ruff checks passed.
- The single signed delivery commit differs from that tested
implementation only in a Rustdoc correction describing Embedded
rejection; formatting passed again. No dependency pins or lockfiles
changed.

The matching SDK is prepared for the separate server validation of
failure/cancellation retention and missing-only refill. Those end-to-end
results are still pending and are not implied by these SDK tests.

Refs [ENT-3733](https://linear.app/lancedb/issue/ENT-3733).
This commit is contained in:
Xuanwo
2026-11-10 15:22:56 +08:00
committed by GitHub
parent 0a5c36c1e2
commit 13e4cf7e58
10 changed files with 128 additions and 5 deletions
@@ -261,6 +261,38 @@ release. TypeScript does not expose GPU class discovery; pass `gpuClass`
and `gpuCount` in the `refreshColumnAsync` options.
</Note>
Completed Function outputs publish with their fragment by default. To opt into
a row-count publication threshold, set `publication_interval_rows` in Python or
`publication_interval_rows()` in Rust:
<CodeGroup>
```python Python icon="python"
job = table.refresh_column_async(
"embedding",
publication_interval_rows=60,
)
job.wait()
```
```rust Rust icon="rust"
use lancedb::table::RefreshColumnOptions;
let _job = table
.refresh_column_async_with_options(
"embedding",
RefreshColumnOptions::default().publication_interval_rows(60),
)
.await?;
```
</CodeGroup>
The threshold is opt-in and must be greater than zero. The number of rows in an
individual publication can differ from this value depending on completed work
and Function batching; this is not an exact-N contract. Omit it to keep
whole-fragment publication.
The TypeScript SDK does not expose the publication threshold option yet.
The job APIs also take the row filter and failure handling of the SQL
`REFRESH COLUMN ... WHERE ... WITH (...)` form, so a refresh can be narrowed
to new rows (see [what a filter recomputes](#partial)) and told to skip rows