mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-28 22:46:14 +00:00
fix: gate experimental local functions behind an opt-in feature (#4688)
Published SDK builds currently expose local SQL computed columns and materialized-view execution that are not ready for release. Add the opt-in `experimental-functions` Cargo feature to the Rust core and forward it through the Python and Node bindings; published packages leave it disabled. Local declarations and refreshes now return `NotSupported` before mutating tables or spawning jobs, including prepared materialized-view creation. Remote Cloud/Enterprise client operations, Python UDF packaging, ordinary eager `add_columns`, and access to existing data remain available. Stored metadata and write protections stay active, and shared server commit helpers remain usable. CI covers both feature configurations. Existing local execution tests run with the feature enabled; default-build regressions cover early rejection, unchanged table versions, existing metadata protections, and normal eager transforms. API and source-build documentation explain the opt-in. Validation: Rust library tests pass with the feature disabled (1,869 tests) and enabled (1,947 tests), plus three default-build integration regressions. Python and TypeScript bindings were rebuilt and tested in both configurations, including mocked remote requests; Rust formatting, Python package lint, TypeScript lint/build, and generated docs were checked. Repository-wide Ruff still reports 48 pre-existing errors in unchanged `docs/web-tests` files.
This commit is contained in:
Xuanwo
@@ -50,6 +50,19 @@ pnpm install
|
||||
pnpm build
|
||||
```
|
||||
|
||||
Local SQL computed columns and materialized-view creation/refresh are experimental
|
||||
and disabled in published packages. To opt in when building from this directory:
|
||||
|
||||
```shell
|
||||
pnpm build:debug --features experimental-functions
|
||||
pnpm tsc
|
||||
pnpm test --runInBand table.test.ts materialized_view.test.ts
|
||||
```
|
||||
|
||||
Rebuild without `--features experimental-functions` to test the default package.
|
||||
The tests detect the compiled feature; remote operations, metadata protections,
|
||||
and ordinary eager `addColumns` transforms remain available in both builds.
|
||||
|
||||
Lint:
|
||||
|
||||
```shell
|
||||
|
||||
@@ -180,6 +180,10 @@ abstract createMaterializedView(
|
||||
|
||||
Define a materialized view named `name` over the table `source`.
|
||||
|
||||
Local creation and refresh require a source build with the
|
||||
`experimental-functions` Cargo feature, disabled in published packages.
|
||||
Remote Cloud/Enterprise operations do not require this feature.
|
||||
|
||||
The view is populated before creation returns. Set `withNoData` to create
|
||||
only its definition and empty backing table. The view is a normal table:
|
||||
it can be queried, indexed and searched, and it appears in `tableNames`.
|
||||
|
||||
@@ -64,6 +64,7 @@ refresh(options?): Promise<RefreshMaterializedViewResult>
|
||||
```
|
||||
|
||||
Recompute the view from its source.
|
||||
Local refresh requires a source build with `experimental-functions`.
|
||||
|
||||
Every refresh rebuilds the complete view. For local views,
|
||||
`sourceVersion` refreshes to that source version instead of the latest;
|
||||
|
||||
@@ -69,6 +69,10 @@ abstract addColumns(newColumnTransforms): Promise<AddColumnsResult>
|
||||
|
||||
Add new columns with defined values.
|
||||
|
||||
Local computed-column declaration and refresh require a source build with
|
||||
the `experimental-functions` Cargo feature. Published packages disable it;
|
||||
remote Cloud/Enterprise operations remain available.
|
||||
|
||||
The `{ computed }` form stores the expression rather than evaluating it
|
||||
now: the column is committed with no values, and rows get them from
|
||||
[Table#refreshColumn](Table.md#refreshcolumn). Declaring one therefore costs the same on a
|
||||
@@ -1035,6 +1039,7 @@ abstract refreshColumn(column): Promise<RefreshColumnResult>
|
||||
```
|
||||
|
||||
Fill the rows of a computed column that hold no value yet.
|
||||
Local refresh requires a source build with `experimental-functions`.
|
||||
|
||||
Rows appended since the last refresh are filled by the next one, and
|
||||
rows whose inputs changed since they were computed are recomputed;
|
||||
@@ -1063,6 +1068,8 @@ abstract refreshColumnAsync(column): Promise<Job>
|
||||
|
||||
Like [Table#refreshColumn](Table.md#refreshcolumn), but returns a handle to the refresh
|
||||
job instead of blocking until it completes.
|
||||
Local refresh requires a source build with `experimental-functions`.
|
||||
Remote Cloud/Enterprise jobs do not require this feature.
|
||||
|
||||
The job may already be complete when returned; callers must not assume
|
||||
the column is filled until [Job.wait](Job.md#wait) resolves. Invalid input --
|
||||
|
||||
@@ -429,6 +429,13 @@ scoped to each invocation and support `read`, `readinto`, `seek`, `tell`,
|
||||
|
||||
## Materialized Views (Synchronous)
|
||||
|
||||
In the 0.41 SDK release, local materialized-view creation and refresh require a
|
||||
source build with the `experimental-functions` Cargo feature. Published Python
|
||||
packages leave it disabled. Opening, reading, listing and dropping existing views
|
||||
remain available, as do remote Cloud and Enterprise operations. The same feature
|
||||
requirement applies to local SQL computed-column declaration and refresh; ordinary
|
||||
eager `add_columns` transforms remain available.
|
||||
|
||||
On Cloud and Enterprise, creation accepts `auto_refresh_enabled` and
|
||||
`auto_refresh_interval_seconds`:
|
||||
|
||||
@@ -443,7 +450,8 @@ view = db.create_materialized_view(
|
||||
|
||||
Omit `auto_refresh_interval_seconds` to let the server determine when to refresh.
|
||||
These options also apply to asynchronous creation and creation jobs. Embedded
|
||||
connections support manual refresh and reject automatic scheduling requests.
|
||||
connections built with `experimental-functions` support manual refresh and reject
|
||||
automatic scheduling requests.
|
||||
|
||||
::: lancedb.materialized_view.MaterializedView
|
||||
|
||||
|
||||
@@ -8,13 +8,20 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Both offerings support computed columns. Enterprise requires the deployment to enable them; the embedded library evaluates them in your process.
|
||||
Local computed columns require a source build with the experimental-functions Cargo feature. Enterprise requires the deployment to enable them.
|
||||
---
|
||||
|
||||
A computed column is declared with an expression and starts with no values.
|
||||
Declaring it is cheap on any table size, because nothing is evaluated until
|
||||
[a refresh](/feature-engineering/refresh-column) asks for it.
|
||||
|
||||
<Note>
|
||||
In the 0.41 SDK release, local computed-column declaration and refresh require a
|
||||
source build with the `experimental-functions` Cargo feature. Published Python
|
||||
and TypeScript packages leave it disabled. Remote Cloud and Enterprise operations
|
||||
do not require this feature. Ordinary eager `add_columns` transforms remain available.
|
||||
</Note>
|
||||
|
||||
## Add a computed column {#declare}
|
||||
|
||||
```python
|
||||
|
||||
@@ -8,7 +8,7 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Both offerings support materialized views. OSS refreshes run in your process; Enterprise runs the work on remote compute. See [Enterprise refreshes](#availability) for server compatibility requirements.
|
||||
Local creation and refresh require a source build with the experimental-functions Cargo feature. Enterprise runs the work on remote compute. See [Enterprise refreshes](#availability) for server compatibility requirements.
|
||||
---
|
||||
|
||||
import {
|
||||
@@ -20,6 +20,13 @@ A materialized view stores the result of a query as a table. Use it to keep a
|
||||
filtered or transformed dataset that you can query, index and search repeatedly.
|
||||
Refresh the view after its source data changes.
|
||||
|
||||
<Note>
|
||||
In the 0.41 SDK release, local materialized-view creation and refresh require a
|
||||
source build with the `experimental-functions` Cargo feature. Published Python
|
||||
and TypeScript packages leave it disabled. Opening, reading, listing and dropping
|
||||
existing views remain available, as do remote Cloud and Enterprise operations.
|
||||
</Note>
|
||||
|
||||
<Note>
|
||||
In SQL this is [`REFRESH MATERIALIZED VIEW`](/sql/refresh#refresh-materialized-view).
|
||||
</Note>
|
||||
|
||||
@@ -8,13 +8,20 @@ availability:
|
||||
oss: available
|
||||
enterprise: available
|
||||
summary: >-
|
||||
Both offerings support column refresh. The embedded library runs it in your process; Enterprise runs it as a server job that must be enabled for your Enterprise cluster.
|
||||
Local column refresh requires a source build with the experimental-functions Cargo feature. Enterprise runs it as a server job that must be enabled for your Enterprise cluster.
|
||||
---
|
||||
|
||||
Refresh a [computed column](/feature-engineering/add-computed-column) to fill
|
||||
new rows and recompute values whose inputs changed. The example below continues
|
||||
with the `doubled` column from the computed-column guide.
|
||||
|
||||
<Note>
|
||||
In the 0.41 SDK release, local declaration and refresh require a source build
|
||||
with the `experimental-functions` Cargo feature. Published Python and TypeScript
|
||||
packages leave it disabled. Remote Cloud and Enterprise operations do not require
|
||||
this feature; existing computed-column data remains readable in default builds.
|
||||
</Note>
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
table.refresh_column("doubled")
|
||||
|
||||
Reference in New Issue
Block a user