mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-29 14:56:51 +00:00
Xuanwo
8a4effa06a
Published SDK builds currently expose local SQL computed columns and materialized-view execution that are not ready for release. Add the opt-in `experimental-functions` Cargo feature to the Rust core and forward it through the Python and Node bindings; published packages leave it disabled. Local declarations and refreshes now return `NotSupported` before mutating tables or spawning jobs, including prepared materialized-view creation. Remote Cloud/Enterprise client operations, Python UDF packaging, ordinary eager `add_columns`, and access to existing data remain available. Stored metadata and write protections stay active, and shared server commit helpers remain usable. CI covers both feature configurations. Existing local execution tests run with the feature enabled; default-build regressions cover early rejection, unchanged table versions, existing metadata protections, and normal eager transforms. API and source-build documentation explain the opt-in. Validation: Rust library tests pass with the feature disabled (1,869 tests) and enabled (1,947 tests), plus three default-build integration regressions. Python and TypeScript bindings were rebuilt and tested in both configurations, including mocked remote requests; Rust formatting, Python package lint, TypeScript lint/build, and generated docs were checked. Repository-wide Ruff still reports 48 pre-existing errors in unchanged `docs/web-tests` files.
132 lines
4.8 KiB
Plaintext
132 lines
4.8 KiB
Plaintext
---
|
|
title: "Add a computed column"
|
|
sidebarTitle: "Computed Columns"
|
|
description: "Declare a column from a SQL expression. Nothing is evaluated until you refresh it."
|
|
icon: table-columns
|
|
keywords: ["computed columns", "add_columns", "transforms", "schema"]
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Local computed columns require a source build with the experimental-functions Cargo feature. Enterprise requires the deployment to enable them.
|
|
---
|
|
|
|
A computed column is declared with an expression and starts with no values.
|
|
Declaring it is cheap on any table size, because nothing is evaluated until
|
|
[a refresh](/feature-engineering/refresh-column) asks for it.
|
|
|
|
<Note>
|
|
In the 0.41 SDK release, local computed-column declaration and refresh require a
|
|
source build with the `experimental-functions` Cargo feature. Published Python
|
|
and TypeScript packages leave it disabled. Remote Cloud and Enterprise operations
|
|
do not require this feature. Ordinary eager `add_columns` transforms remain available.
|
|
</Note>
|
|
|
|
## Add a computed column {#declare}
|
|
|
|
```python
|
|
import lancedb
|
|
|
|
db = lancedb.connect("./.lancedb")
|
|
table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}])
|
|
|
|
table.add_columns(computed={"doubled": "x * 2"})
|
|
# AddColumnsResult(version=2)
|
|
```
|
|
|
|
That commits a new table version with the column present and empty. It is an
|
|
ordinary table version, so the usual [versioning](/management/table#versioning) applies,
|
|
including that you can travel back to before the column existed.
|
|
|
|
## Compute now or on refresh {#computed-vs-transforms}
|
|
|
|
`add_columns` takes either, and the difference matters:
|
|
|
|
- **SQL expressions in `transforms`** are evaluated *now*, for every row, as
|
|
part of the call. Good for a small table, or a column you want populated
|
|
immediately.
|
|
{/* Draft for the Functions release:
|
|
A `FunctionApplication` passed through `transforms` instead
|
|
declares empty Function-bound columns, which a refresh fills; that needs a
|
|
deployment with Functions enabled, and the embedded library refuses it. */}
|
|
- **`computed`** stores the expression and evaluates nothing. The column is
|
|
committed empty and filled later by
|
|
[`refresh_column`](/feature-engineering/refresh-column).
|
|
|
|
`computed` and `transforms` cannot be combined in one call. Reach for `computed` when the table is
|
|
large enough that you would rather control when the work happens.
|
|
|
|
## The expression {#expressions}
|
|
|
|
The expression is SQL, evaluated over the row. It can reference any column in
|
|
the table, including another computed column:
|
|
|
|
```python
|
|
table.add_columns(computed={
|
|
"power_score": "cast(x as double) / 2.0",
|
|
"is_strong": "power_score > 0.5",
|
|
})
|
|
```
|
|
|
|
Declaration order is dependency order: a computed column may reference one
|
|
declared earlier in the same mapping, but not one declared after it.
|
|
|
|
The functions available are the ones the
|
|
[SQL dialect](/sql/functions) documents.
|
|
|
|
A literal in the expression has to fit the column it meets: `x / 2.0` is
|
|
refused when `x` is an integer column, so write `cast(x as double) / 2.0`.
|
|
|
|
<Warning>
|
|
In LanceDB Enterprise, keep the expression to top-level columns. A column
|
|
refresh can fail on an expression that reads a field inside a struct, such as
|
|
`stats.magic + stats.strength`, which the embedded library computes. Try a new
|
|
expression on a small table first.
|
|
</Warning>
|
|
|
|
## Change columns used by an expression {#input-constraints}
|
|
|
|
<Warning>
|
|
While a computed column's declaration references another column, that column
|
|
cannot be renamed, retyped, or dropped. Drop the computed column first
|
|
if you need to change one of its inputs.
|
|
</Warning>
|
|
|
|
This is worth knowing before deriving a column from something you expect to
|
|
reshape later. Dropping the computed column releases the constraint.
|
|
|
|
## Filling with Python instead of SQL {#functions}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
When the value cannot be expressed in SQL (an embedding, a model call, an
|
|
image decode), [Geneva](/geneva), the feature engineering package for LanceDB
|
|
Enterprise, fills the column with a Python function instead.
|
|
|
|
{/*
|
|
Draft for the Functions release. Replaces the paragraph above:
|
|
|
|
When the value cannot be expressed in SQL -- an embedding, a model call, an
|
|
image decode -- the column is filled by a registered
|
|
[function](/management/function) instead of an expression. Declaring it looks
|
|
the same; what differs is what fills it. It needs a deployment with Functions
|
|
enabled.
|
|
*/}
|
|
|
|
## Related guides {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
Fill the rows that lack values.
|
|
</Card>
|
|
<Card title="Geneva" icon="layer-group" href="/geneva">
|
|
Fill a column with Python on LanceDB Enterprise.
|
|
</Card>
|
|
{/*
|
|
Draft for the Functions release. Replaces the Geneva card above:
|
|
<Card title="Functions" icon="code" href="/management/function">
|
|
Fill a column with Python.
|
|
</Card>
|
|
*/}
|
|
</CardGroup>
|