mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-16 08:42:29 +00:00
Brendan Clement
06aa526ae1
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The curation, SQL and operations pages contain outdated claims and examples that do not reliably work as written. This corrects those pages against the reviewed SDK and local server evidence, including prepared statements, background jobs, cleanup and monitoring. Integration guides are grouped with their examples in #4533. Function and Secret-object documentation stays unpublished until its release. The two Function pages are listed in `.mintignore` and left out of the navigation, and the sections about them on other pages are MDX comments. Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after #4533 is squash-merged. Local assembly, Mint validation and link checks pass, and the assembled site shows no Function or Secret-object content. A local release rehearsal that uncomments the drafts and restores the navigation also passes. Hold publication until this PR and lancedb/sophon#8688 are on `main`.
125 lines
4.5 KiB
Plaintext
125 lines
4.5 KiB
Plaintext
---
|
|
title: "Add a computed column"
|
|
sidebarTitle: "Computed Columns"
|
|
description: "Declare a column from a SQL expression. Nothing is evaluated until you refresh it."
|
|
icon: table-columns
|
|
keywords: ["computed columns", "add_columns", "transforms", "schema"]
|
|
availability:
|
|
oss: available
|
|
enterprise: available
|
|
summary: >-
|
|
Both offerings support computed columns. Enterprise requires the deployment to enable them; the embedded library evaluates them in your process.
|
|
---
|
|
|
|
A computed column is declared with an expression and starts with no values.
|
|
Declaring it is cheap on any table size, because nothing is evaluated until
|
|
[a refresh](/feature-engineering/refresh-column) asks for it.
|
|
|
|
## Add a computed column {#declare}
|
|
|
|
```python
|
|
import lancedb
|
|
|
|
db = lancedb.connect("./.lancedb")
|
|
table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}])
|
|
|
|
table.add_columns(computed={"doubled": "x * 2"})
|
|
# AddColumnsResult(version=2)
|
|
```
|
|
|
|
That commits a new table version with the column present and empty. It is an
|
|
ordinary table version, so the usual [versioning](/management/table#versioning) applies --
|
|
including that you can travel back to before the column existed.
|
|
|
|
## Compute now or on refresh {#computed-vs-transforms}
|
|
|
|
`add_columns` takes either, and the difference matters:
|
|
|
|
- **SQL expressions in `transforms`** are evaluated *now*, for every row, as
|
|
part of the call. Good for a small table, or a column you want populated
|
|
immediately.
|
|
{/* Draft for the Functions release:
|
|
A `FunctionApplication` passed through `transforms` instead
|
|
declares empty Function-bound columns, which a refresh fills; that needs a
|
|
deployment with Functions enabled, and the embedded library refuses it. */}
|
|
- **`computed`** stores the expression and evaluates nothing. The column is
|
|
committed empty and filled later by
|
|
[`refresh_column`](/feature-engineering/refresh-column).
|
|
|
|
`computed` and `transforms` cannot be combined in one call. Reach for `computed` when the table is
|
|
large enough that you would rather control when the work happens.
|
|
|
|
## The expression {#expressions}
|
|
|
|
The expression is SQL, evaluated over the row. It can reference any column in
|
|
the table, including another computed column:
|
|
|
|
```python
|
|
table.add_columns(computed={
|
|
"power_score": "cast(x as double) / 2.0",
|
|
"is_strong": "power_score > 0.5",
|
|
})
|
|
```
|
|
|
|
Declaration order is dependency order -- a computed column may reference one
|
|
declared earlier in the same mapping, but not one declared after it.
|
|
|
|
The functions available are the ones the
|
|
[SQL dialect](/sql/functions) documents.
|
|
|
|
A literal in the expression has to fit the column it meets: `x / 2.0` is
|
|
refused when `x` is an integer column, so write `cast(x as double) / 2.0`.
|
|
|
|
<Warning>
|
|
In LanceDB Enterprise, keep the expression to top-level columns. A column
|
|
refresh can fail on an expression that reads a field inside a struct, such as
|
|
`stats.magic + stats.strength`, which the embedded library computes. Try a new
|
|
expression on a small table first.
|
|
</Warning>
|
|
|
|
## Change columns used by an expression {#input-constraints}
|
|
|
|
<Warning>
|
|
While a computed column's declaration references another column, that column
|
|
cannot be renamed, retyped, or dropped. Drop the computed column first
|
|
if you need to change one of its inputs.
|
|
</Warning>
|
|
|
|
This is worth knowing before deriving a column from something you expect to
|
|
reshape later. Dropping the computed column releases the constraint.
|
|
|
|
## Filling with Python instead of SQL {#functions}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
When the value cannot be expressed in SQL -- an embedding, a model call, an
|
|
image decode -- [Geneva](/geneva), the feature engineering package for LanceDB
|
|
Enterprise, fills the column with a Python function instead.
|
|
|
|
{/*
|
|
Draft for the Functions release. Replaces the paragraph above:
|
|
|
|
When the value cannot be expressed in SQL -- an embedding, a model call, an
|
|
image decode -- the column is filled by a registered
|
|
[function](/management/function) instead of an expression. Declaring it looks
|
|
the same; what differs is what fills it. It needs a deployment with Functions
|
|
enabled.
|
|
*/}
|
|
|
|
## Related guides {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
Fill the rows that lack values.
|
|
</Card>
|
|
<Card title="Geneva" icon="layer-group" href="/geneva">
|
|
Fill a column with Python on LanceDB Enterprise.
|
|
</Card>
|
|
{/*
|
|
Draft for the Functions release. Replaces the Geneva card above:
|
|
<Card title="Functions" icon="code" href="/management/function">
|
|
Fill a column with Python.
|
|
</Card>
|
|
*/}
|
|
</CardGroup>
|