mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
Brendan Clement
e920dad55b
Two pages said list_materialized_views reads every table's schema and that SHOW MATERIALIZED VIEWS is cheaper. That is the local implementation; on a db:// connection the SDK asks the deployment for its views, as SHOW MATERIALIZED VIEWS does. The computed-column page now says that a literal must fit the column it meets, and that a deployment's refresh can fail on an expression reading a struct field, which the embedded library computes, so an expression for a deployment should use top-level columns.
127 lines
4.6 KiB
Plaintext
127 lines
4.6 KiB
Plaintext
---
|
|
title: "Add a computed column"
|
|
sidebarTitle: "Computed Columns"
|
|
description: "Declare a column from a SQL expression. Nothing is evaluated until you refresh it."
|
|
icon: table-columns
|
|
keywords: ["computed columns", "add_columns", "transforms", "schema"]
|
|
availability:
|
|
oss: available
|
|
enterprise: varies
|
|
summary: >-
|
|
Computed columns work in the embedded library. On a deployment they are off
|
|
by default and have to be enabled there.
|
|
---
|
|
|
|
A computed column is declared with an expression and starts with no values.
|
|
Declaring it is cheap on any table size, because nothing is evaluated until
|
|
[a refresh](/feature-engineering/refresh-column) asks for it.
|
|
|
|
## Declaring one {#declare}
|
|
|
|
```python
|
|
import lancedb
|
|
|
|
db = lancedb.connect("./.lancedb")
|
|
table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}])
|
|
|
|
table.add_columns(computed={"doubled": "x * 2"})
|
|
# AddColumnsResult(version=2)
|
|
```
|
|
|
|
That commits a new table version with the column present and empty. It is an
|
|
ordinary table version, so the usual [versioning](/management/table#versioning) applies --
|
|
including that you can travel back to before the column existed.
|
|
|
|
## `computed` and `transforms` are different {#computed-vs-transforms}
|
|
|
|
`add_columns` takes either, and the difference matters:
|
|
|
|
- **SQL expressions in `transforms`** are evaluated *now*, for every row, as
|
|
part of the call. Good for a small table, or a column you want populated
|
|
immediately.
|
|
{/* if functions launched */}
|
|
A `FunctionApplication` passed through `transforms` instead
|
|
declares empty Function-bound columns, which a refresh fills; that needs a
|
|
deployment with Functions enabled, and the embedded library refuses it.
|
|
{/* end if */}
|
|
- **`computed`** stores the expression and evaluates nothing. The column is
|
|
committed empty and filled later by
|
|
[`refresh_column`](/feature-engineering/refresh-column).
|
|
|
|
`computed` and `transforms` cannot be combined in one call. Reach for `computed` when the table is
|
|
large enough that you would rather control when the work happens.
|
|
|
|
## The expression {#expressions}
|
|
|
|
The expression is SQL, evaluated over the row. It can reference any column in
|
|
the table, including another computed column:
|
|
|
|
```python
|
|
table.add_columns(computed={
|
|
"power_score": "(stats.magic + stats.strength) / 2.0",
|
|
"is_strong": "power_score > 3.0",
|
|
})
|
|
```
|
|
|
|
Declaration order is dependency order -- a computed column may reference one
|
|
declared earlier in the same mapping, but not one declared after it.
|
|
|
|
The functions available are the ones the
|
|
[SQL dialect](/sql/functions) documents.
|
|
|
|
A literal in the expression has to fit the column it meets: `x / 2.0` is
|
|
refused when `x` is an integer column, so write `cast(x as double) / 2.0`.
|
|
|
|
<Warning>
|
|
On a deployment, keep the expression to top-level columns. A deployment's
|
|
refresh can fail on an expression that reads a field inside a struct, such as
|
|
`stats.magic + stats.strength`, which the embedded library computes. Try a new
|
|
expression on a small table first.
|
|
</Warning>
|
|
|
|
## Its inputs become load-bearing {#input-constraints}
|
|
|
|
<Warning>
|
|
While a computed column's declaration references another column, that column
|
|
cannot be renamed, retyped, or dropped. The declaration is part of the schema,
|
|
and LanceDB will not let you invalidate it out from under itself.
|
|
</Warning>
|
|
|
|
This is worth knowing before deriving a column from something you expect to
|
|
reshape later. Dropping the computed column releases the constraint.
|
|
|
|
## Filling with Python instead of SQL {#functions}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
{/* if functions launched */}
|
|
When the value cannot be expressed in SQL -- an embedding, a model call, an
|
|
image decode -- the column is filled by a registered
|
|
[function](/management/function) instead of an expression. Declaring it looks
|
|
the same; what differs is what fills it. It needs a deployment with Functions
|
|
enabled.
|
|
{/* end if */}
|
|
{/* if functions not launched */}
|
|
When the value cannot be expressed in SQL -- an embedding, a model call, an
|
|
image decode -- [Geneva](/geneva), the feature engineering package for LanceDB
|
|
Enterprise, fills the column with a Python function instead.
|
|
{/* end if */}
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
Fill the rows that lack values.
|
|
</Card>
|
|
{/* if functions launched */}
|
|
<Card title="Functions" icon="code" href="/management/function">
|
|
Fill a column with Python.
|
|
</Card>
|
|
{/* end if */}
|
|
{/* if functions not launched */}
|
|
<Card title="Geneva" icon="layer-group" href="/geneva">
|
|
Fill a column with Python on LanceDB Enterprise.
|
|
</Card>
|
|
{/* end if */}
|
|
</CardGroup>
|