lancedb/docs/web/feature-engineering/data-evolution.mdx

---
title: "Data evolution"
sidebarTitle: "Data Evolution"
description: "Derive new columns from data already in the table, without rewriting what is already there."
icon: wand-magic-sparkles
keywords: ["data evolution", "derived columns", "add_columns", "zero copy", "feature engineering"]
availability:
  oss: available
  enterprise: available
  summary: >-
    Adding columns computed from SQL expressions works in the embedded library
    and in LanceDB Enterprise.
---

import {
    PySchemaAddSetup,
    TsSchemaAddSetup as TsSchemaAddSetup,
    RsSchemaAddSetup as RsSchemaAddSetup,
    PyAddColumnsCalculated as AddColumnsCalculated,
    TsAddColumnsCalculated as TsAddColumnsCalculated,
    RsAddColumnsCalculated as RsAddColumnsCalculated,
    PyAddFeatureColumnsSql as AddFeatureColumnsSql,
    TsAddFeatureColumnsSql as TsAddFeatureColumnsSql,
    RsAddFeatureColumnsSql as RsAddFeatureColumnsSql,
} from '/snippets/tables.mdx';

Add derived columns to calculate features from data already in a table.
For example, compute a product score from existing fields without reading
and rewriting the table in your application.

Lance can write the new columns without rewriting the existing columns.
This is useful when the table also contains large images, video or embeddings.
See [Lance format](/basics/lance) for the storage model.

<Note>
In SQL, on an Enterprise cluster that enables computed columns, this is [`ALTER TABLE ... ADD COLUMN ... AS`](/sql/ddl/table#computed-column).
</Note>

## Set up the example table {#set-up-the-example-table}

The examples use `db`, a connection opened as in
[Connect to LanceDB](/basics/connect), and create a small product table in it:

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {PySchemaAddSetup}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsSchemaAddSetup}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsSchemaAddSetup}
    </CodeBlock>
</CodeGroup>

## Add derived columns {#add-derived-columns}

You can add new columns that are derived from existing data using SQL expressions.
For feature engineering on large existing tables, group related derived features into
one `add_columns` operation instead of running many separate writes. This creates one
new table version for the schema change and computes the new columns from the existing
rows, which avoids growing the table's version history with many small updates.

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {AddColumnsCalculated}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsAddColumnsCalculated}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsAddColumnsCalculated}
    </CodeBlock>
</CodeGroup>

The same call can add multiple derived columns at once. For example, if you are
building several lightweight features from existing product fields, pass all of the
new column expressions together:

<CodeGroup>
    <CodeBlock filename="Python" language="Python" icon="python">
    {AddFeatureColumnsSql}
    </CodeBlock>

    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
    {TsAddFeatureColumnsSql}
    </CodeBlock>

    <CodeBlock filename="Rust" language="Rust" icon="rust">
    {RsAddFeatureColumnsSql}
    </CodeBlock>
</CodeGroup>

<Info>
LanceDB `add_columns` does not currently accept Python callables, batch UDFs, or
PyArrow `RecordBatch` iterators for populating new columns. New column values must be
defined with SQL expressions, or added as NULL columns from an Arrow field or schema.
If your transformation cannot be expressed in SQL, compute the values outside
`add_columns` before writing them back through another workflow.
</Info>

## When SQL is not enough {#beyond-sql}

The expressions above are evaluated once, as part of the call. For other workflows:

- A [computed column](/feature-engineering/add-computed-column) declares the
  expression and leaves the column empty, so you choose when the work happens.
  Use it when you want to schedule the computation separately.
- On LanceDB Enterprise, [Geneva](/geneva) fills a column with Python, for
  values SQL cannot express -- an embedding, a model call, an image decode.

{/*
Draft for the Functions release. Replaces the Geneva item in the list above:
- A [function](/management/function) fills a column with Python, for values SQL
  cannot express -- an embedding, a model call, an image decode.
*/}

A deployment has to enable computed columns before you use them.
{/* Draft for the Functions release: Functions need enabling there too. */}

## Related guides {#next}

<CardGroup cols={2}>
  <Card title="Add a computed column" icon="table-columns" href="/feature-engineering/add-computed-column">
    Declare now, fill later.
  </Card>
  <Card title="Schema evolution" icon="boxes-stacked" href="/management/table#schema">
    Change the table's shape instead of its contents.
  </Card>
</CardGroup>