mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-06 02:36:52 +00:00
Jack Ye
ddb318567a
## Summary - Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause, and `dedupe()`/`duplicate_pairs()` (verified against current sophon source; these were recently renamed from `vector_dedup`/`vector_duplicate_pairs`). Fix the blob-column default-projection claim and add the previously undocumented `blob_descriptor()`. - Enrich the gated Functions draft (still `.mintignore`'d, pre-release): chained Functions over blob outputs, GPU-produced blobs, Secrets, multi-input/output binding, `code=[module]` packaging. - Upgrade the materialized-view and computed-column auto-refresh drafts with precise, verified catch-up semantics, and add the missing `ALTER MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the automatic-refresh scheduler is still dev/staging-only). - Thread one running example (video frames → GPU extract/caption → dedup → auto-refreshing materialized view) through Functions, materialized views, and the LeRobotDataset integration guide instead of a separate tutorial. - Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout (52 files); activate the existing availability-comparison mechanism on `basics/offerings.mdx` for a full feature-by-feature comparison table.
182 lines
8.4 KiB
Plaintext
182 lines
8.4 KiB
Plaintext
---
|
|
title: "Feature engineering"
|
|
sidebarTitle: "Overview"
|
|
description: "Derive columns and tables from data already in the lakehouse, without moving it somewhere else to compute."
|
|
icon: layer-group
|
|
keywords: ["feature engineering", "computed columns", "materialized views", "lakehouse", "jobs"]
|
|
---
|
|
|
|
Use feature engineering to turn raw data into features for search, analysis
|
|
and model training. LanceDB stores derived columns and materialized views
|
|
alongside the source data, so you can reuse the results across those workflows.
|
|
|
|
[Materialized views](/feature-engineering/materialized-views) are ordinary
|
|
[catalog objects](/basics/catalog). Create them through the SDK or SQL,
|
|
then refresh the derived data when its inputs change.
|
|
|
|
{/*
|
|
Draft for the Functions release. Replaces the paragraph above:
|
|
|
|
[Functions](/management/function) and
|
|
[materialized views](/feature-engineering/materialized-views) are ordinary
|
|
[catalog objects](/basics/catalog). Create them through the SDK or SQL,
|
|
then refresh the derived data when its inputs change.
|
|
*/}
|
|
|
|
<Note>
|
|
In LanceDB Enterprise, every operation here also has a SQL form. See [DDL](/sql/ddl/table#alter) to declare a column and [REFRESH COLUMN](/sql/dml/refresh-column) to fill it.
|
|
</Note>
|
|
|
|
## Computed columns and materialized views {#the-shape}
|
|
|
|
Computed columns and materialized views both keep derived data in the
|
|
lakehouse, but they fill it differently, and the difference decides what a
|
|
refresh costs:
|
|
|
|
| | Computed column | Materialized view |
|
|
| --- | --- | --- |
|
|
| Declaring or creating it | records the expression; no values are written | runs the query and stores its rows, unless created empty |
|
|
| A refresh | fills rows with no value and recomputes rows whose inputs changed | rebuilds the view completely from its source query |
|
|
| A refresh when nothing changed | fills nothing | still rebuilds the view and commits a new version |
|
|
| In LanceDB Enterprise | runs as a [job](/basics/jobs) | runs as a server-side materialization job |
|
|
|
|
### Computed columns fill incrementally {#incremental}
|
|
|
|
A [computed column](/feature-engineering/add-computed-column) separates
|
|
**declaring** from **filling**:
|
|
|
|
- Declaring is cheap and immediate. It records the expression in the schema
|
|
and commits a new table version, with no data written. It does not evaluate the expression over existing rows.
|
|
- Fill the column with a separate refresh call that you can retry. In LanceDB
|
|
Enterprise, submit the refresh as a [job](/basics/jobs) and track its progress.
|
|
|
|
**Refresh new and changed rows.** A
|
|
[refresh](/feature-engineering/refresh-column) gives values to rows that lack
|
|
them and recomputes rows whose inputs changed since they were computed. It
|
|
leaves the rest alone, so calling it twice is not twice the work, and appending
|
|
rows means the next refresh handles only the new ones.
|
|
|
|
**Computed values update when you refresh.** If a column derives from `price` and a row's
|
|
`price` is later updated, the derived value keeps its old value until the next
|
|
refresh recomputes it. Schedule refreshes after changing the source data when consumers need
|
|
up-to-date features.
|
|
|
|
### Materialized views store query results {#views}
|
|
|
|
A [materialized view](/feature-engineering/materialized-views) has no separate
|
|
declaration step: creating one runs its query and stores the result, so the view
|
|
is ready to read. A refresh rebuilds the stored result completely from the
|
|
source query; see [Refresh a materialized view](/feature-engineering/refresh-materialized-view)
|
|
for the full behavior, including automatic refresh.
|
|
|
|
Enterprise runs materialization on remote compute, as a server-side job.
|
|
|
|
## Choose a feature engineering workflow {#which-one}
|
|
|
|
| | What it produces | Filled by | Available |
|
|
| --- | --- | --- | --- |
|
|
| [View](/feature-engineering/view) | a saved query, no stored rows of its own | nothing -- it runs on every read | Enterprise |
|
|
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | Embedded and Enterprise |
|
|
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | Enterprise |
|
|
| [Geneva](/geneva) | columns filled by Python | the `geneva` package | <Badge color="red">Enterprise</Badge> |
|
|
|
|
{/*
|
|
Draft for the Functions release. Replace the Geneva row of the table above with:
|
|
| [Function](/management/function) | a computed column filled by Python | registered code | <Badge color="red">Enterprise</Badge> |
|
|
| [Geneva](/geneva) | columns filled by Python, for existing pipelines | the `geneva` package | <Badge color="red">Enterprise</Badge> |
|
|
*/}
|
|
|
|
**A view** has no refresh and no staleness window, because it stores the
|
|
query rather than its result. Reach for one first, before a computed column
|
|
or a materialized view, when the query is cheap enough to simply re-run on
|
|
every read.
|
|
|
|
**Computed columns** are the common case: a column declared with a SQL
|
|
expression, filled by [a refresh](/feature-engineering/refresh-column).
|
|
|
|
**Materialized views** derive a whole table (a filtered, projected, or
|
|
aggregated view of a source) and bring it up to date on each refresh. The
|
|
result is an ordinary table: queryable, indexable, searchable.
|
|
|
|
**Geneva** is LanceDB Enterprise's Python package for work SQL cannot express:
|
|
embedding a column, calling a model, decoding an image. It defines features as
|
|
Python functions and backfills them as distributed jobs.
|
|
|
|
{/*
|
|
Draft for the Functions release. Replaces the Geneva paragraph above:
|
|
|
|
**Functions** run Python where the data is, for work SQL cannot express:
|
|
embedding a column, calling a model, decoding an image. Registered once,
|
|
versioned, then applied as a column. They are the way to do that on a
|
|
deployment; [Geneva](/geneva) remains available for existing pipelines.
|
|
*/}
|
|
|
|
A materialized view can select computed columns.
|
|
{/* Draft for the Functions release: A computed column can be filled by a registered function. */}
|
|
|
|
## Where the work runs {#where}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
Locally, a refresh runs in your process. With LanceDB Enterprise it runs across the
|
|
cluster, and the API reflects that: `refresh_column` raises, and
|
|
`refresh_column_async` returns a [job](/basics/jobs) handle instead. Computed
|
|
columns are off by default in LanceDB Enterprise, and its administrator has to enable
|
|
them before a column can be declared there.
|
|
|
|
```python
|
|
job = table.refresh_column_async("power_score")
|
|
result = job.wait()
|
|
result.rows_assigned
|
|
```
|
|
|
|
A backfill over a large table can therefore be started, left, and checked on
|
|
from a different process: the connection does not have to stay open.
|
|
|
|
Enterprise manages the compute for the refresh. You still control when to
|
|
submit it; declaring a computed column does not schedule recurring refreshes.
|
|
{/* Draft for the Functions release: That is also why a function must declare the environment it needs. */}
|
|
|
|
## Recovering from a failure {#failures}
|
|
|
|
{/*
|
|
Draft for the Functions release.
|
|
|
|
A function that raises fails the job. There is no per-row retry or skip policy
|
|
in the function definition, so a function calling a flaky external service
|
|
should handle that itself, inside the function body.
|
|
*/}
|
|
|
|
When a refresh fails, fix the cause and run the refresh
|
|
again. The new refresh fills what is still empty or stale. How much of the
|
|
failed run was kept depends on where it ran (see
|
|
[Refresh a computed column](/feature-engineering/refresh-column#failures)), and a retry
|
|
can recompute values that were already written, so do not count on it costing
|
|
only the rows that did not complete.
|
|
|
|
## Related guides {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Create a view" icon="eye" href="/feature-engineering/view">
|
|
A saved query, with no storage or refresh of its own.
|
|
</Card>
|
|
<Card title="Create a computed column" icon="table-columns" href="/feature-engineering/add-computed-column">
|
|
Declare a column from a SQL expression.
|
|
</Card>
|
|
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
Fill the rows that lack values, locally or as a job.
|
|
</Card>
|
|
<Card title="Create a materialized view" icon="table" href="/feature-engineering/materialized-views">
|
|
Store a query's result as a table.
|
|
</Card>
|
|
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/refresh-materialized-view">
|
|
Bring a derived table up to date with its source.
|
|
</Card>
|
|
{/*
|
|
Draft for the Functions release:
|
|
<Card title="Functions" icon="code" href="/management/function">
|
|
Register Python to fill a column with.
|
|
</Card>
|
|
*/}
|
|
</CardGroup>
|