mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
Brendan ClementandClaude Opus 5.5
ca343a5c06
## Summary Fixes the problems a page-by-page review of the docs found before launch, after checking each one against the current source and, where noted, the code. The matching fixes for the dataset and Geneva pages, which live in `lancedb/docs`, are in lancedb/docs#402. **Tables** - `integrations/embedding/cohere`: each input-type description was split across two table rows and wrapped in literal quotes (`"`search_document`"`). Now one row per type. - `integrations/embedding/gemini`: removes the same stray quotes from the task-type table. - `integrations/ai/langchain`: the MMR search table's `kwargs` row was missing a cell, so its description sat under Type. - Six reranker pages (answerdotai, cohere, colbert, jina, openai, voyageai): closes the backtick before `_relevance_score`. - `indexing/vector-index`: finishes a cell cut off at "with scalar". - `sql/dql/dedupe`: labels the empty header cell in the resolution table. **Corrections** - `curation/sampling`: sampling works in LanceDB Embedded through the Python SDK, so the page is available in both, with the SQL sections marked Enterprise. - `feature-engineering/index`, `add-computed-column`: automatic refresh is on by default for each computed column and takes effect once a deployment enables its scheduler. `refresh-column` already said so; these two implied it was off. - `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates `array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also merges scalar-index's two "nested fields" sections into one. - `integrations/embedding/ibm`: lists the four current models from `CURRENT_MODELS` (it listed a deprecated one) and explains that the default stays a legacy ID so existing tables reload with their model. - `integrations/reranking/openai`: the class is `OpenaiReranker`, not `OpenAI()`. - `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit` throughout, each step runs one command, each step passes the file it writes to the next, and `save-as` uses `-f`/`-o`. - `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the pipeline reads, and the run command runs the file the page creates. - `search/vector-search`: the multivector walkthrough connected to Enterprise in step 1 and a local path in step 2. It now runs locally throughout, with a note on connecting to Enterprise, and is renamed so it doesn't read as a duplicate of the short section above. **Structure and wording** - `integrations/embedding/nvidia`: adds headings, and drops test-suite details aimed at contributors. - `integrations/ai/genkit`: makes the retrieval test a real heading and describes the query step. - `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a `<Note>`. - `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading, not bold text. - `curation/windowing`: sidebar title matches the curation overview ("Rank and Deduplicate"). - `deployment/storage`, `basics/agentic-skills`: fixes a garbled sentence and a stale reference to the quickstart. **Checked:** no new Vale alerts on the changed pages. Assembled together with lancedb/docs#402, the site passes `mint validate` and `mint broken-links --check-anchors --check-redirects`. Every changed page was reviewed in a local preview. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
183 lines
8.5 KiB
Plaintext
183 lines
8.5 KiB
Plaintext
---
|
|
title: "Feature engineering"
|
|
sidebarTitle: "Overview"
|
|
description: "Derive columns and tables from data already in the lakehouse, without moving it somewhere else to compute."
|
|
icon: layer-group
|
|
keywords: ["feature engineering", "computed columns", "materialized views", "lakehouse", "jobs"]
|
|
---
|
|
|
|
Use feature engineering to turn raw data into features for search, analysis
|
|
and model training. LanceDB stores derived columns and materialized views
|
|
alongside the source data, so you can reuse the results across those workflows.
|
|
|
|
[Materialized views](/feature-engineering/materialized-views) are ordinary
|
|
[catalog objects](/basics/catalog). Create them through the SDK or SQL,
|
|
then refresh the derived data when its inputs change.
|
|
|
|
{/*
|
|
Draft for the Functions release. Replaces the paragraph above:
|
|
|
|
[Functions](/management/function) and
|
|
[materialized views](/feature-engineering/materialized-views) are ordinary
|
|
[catalog objects](/basics/catalog). Create them through the SDK or SQL,
|
|
then refresh the derived data when its inputs change.
|
|
*/}
|
|
|
|
<Note>
|
|
In LanceDB Enterprise, every operation here also has a SQL form. See [DDL](/sql/ddl/table#alter) to declare a column and [REFRESH COLUMN](/sql/dml/refresh-column) to fill it.
|
|
</Note>
|
|
|
|
## Computed columns and materialized views {#the-shape}
|
|
|
|
Computed columns and materialized views both keep derived data in the
|
|
lakehouse, but they fill it differently, and the difference decides what a
|
|
refresh costs:
|
|
|
|
| | Computed column | Materialized view |
|
|
| --- | --- | --- |
|
|
| Declaring or creating it | records the expression; no values are written | runs the query and stores its rows, unless created empty |
|
|
| A refresh | fills rows with no value and recomputes rows whose inputs changed | rebuilds the view completely from its source query |
|
|
| A refresh when nothing changed | fills nothing | still rebuilds the view and commits a new version |
|
|
| In LanceDB Enterprise | runs as a [job](/basics/jobs) | runs as a server-side materialization job |
|
|
|
|
### Computed columns fill incrementally {#incremental}
|
|
|
|
A [computed column](/feature-engineering/add-computed-column) separates
|
|
**declaring** from **filling**:
|
|
|
|
- Declaring is cheap and immediate. It records the expression in the schema
|
|
and commits a new table version, with no data written. It does not evaluate the expression over existing rows.
|
|
- Fill the column with a separate refresh call that you can retry. In LanceDB
|
|
Enterprise, submit the refresh as a [job](/basics/jobs) and track its progress.
|
|
|
|
**Refresh new and changed rows.** A
|
|
[refresh](/feature-engineering/refresh-column) gives values to rows that lack
|
|
them and recomputes rows whose inputs changed since they were computed. It
|
|
leaves the rest alone, so calling it twice is not twice the work, and appending
|
|
rows means the next refresh handles only the new ones.
|
|
|
|
**Computed values update when you refresh.** If a column derives from `price` and a row's
|
|
`price` is later updated, the derived value keeps its old value until the next
|
|
refresh recomputes it. Schedule refreshes after changing the source data when consumers need
|
|
up-to-date features.
|
|
|
|
### Materialized views store query results {#views}
|
|
|
|
A [materialized view](/feature-engineering/materialized-views) has no separate
|
|
declaration step: creating one runs its query and stores the result, so the view
|
|
is ready to read. A refresh rebuilds the stored result completely from the
|
|
source query; see [Refresh a materialized view](/feature-engineering/refresh-materialized-view)
|
|
for the full behavior, including automatic refresh.
|
|
|
|
Enterprise runs materialization on remote compute, as a server-side job.
|
|
|
|
## Choose a feature engineering workflow {#which-one}
|
|
|
|
| | What it produces | Filled by | Available |
|
|
| --- | --- | --- | --- |
|
|
| [View](/feature-engineering/view) | a saved query, no stored rows of its own | nothing -- it runs on every read | Enterprise |
|
|
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | Embedded and Enterprise |
|
|
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | Enterprise |
|
|
| [Geneva](/geneva) | columns filled by Python | the `geneva` package | <Badge color="red">Enterprise</Badge> |
|
|
|
|
{/*
|
|
Draft for the Functions release. Replace the Geneva row of the table above with:
|
|
| [Function](/management/function) | a computed column filled by Python | registered code | <Badge color="red">Enterprise</Badge> |
|
|
| [Geneva](/geneva) | columns filled by Python, for existing pipelines | the `geneva` package | <Badge color="red">Enterprise</Badge> |
|
|
*/}
|
|
|
|
**A view** has no refresh and no staleness window, because it stores the
|
|
query rather than its result. Reach for one first, before a computed column
|
|
or a materialized view, when the query is cheap enough to simply re-run on
|
|
every read.
|
|
|
|
**Computed columns** are the common case: a column declared with a SQL
|
|
expression, filled by [a refresh](/feature-engineering/refresh-column).
|
|
|
|
**Materialized views** derive a whole table (a filtered, projected, or
|
|
aggregated view of a source) and bring it up to date on each refresh. The
|
|
result is an ordinary table: queryable, indexable, searchable.
|
|
|
|
**Geneva** is LanceDB Enterprise's Python package for work SQL cannot express:
|
|
embedding a column, calling a model, decoding an image. It defines features as
|
|
Python functions and backfills them as distributed jobs.
|
|
|
|
{/*
|
|
Draft for the Functions release. Replaces the Geneva paragraph above:
|
|
|
|
**Functions** run Python where the data is, for work SQL cannot express:
|
|
embedding a column, calling a model, decoding an image. Registered once,
|
|
versioned, then applied as a column. They are the way to do that on a
|
|
deployment; [Geneva](/geneva) remains available for existing pipelines.
|
|
*/}
|
|
|
|
A materialized view can select computed columns.
|
|
{/* Draft for the Functions release: A computed column can be filled by a registered function. */}
|
|
|
|
## Where the work runs {#where}
|
|
|
|
<Badge color="red">Enterprise</Badge>
|
|
|
|
Locally, a refresh runs in your process. With LanceDB Enterprise it runs across the
|
|
cluster, and the API reflects that: `refresh_column` raises, and
|
|
`refresh_column_async` returns a [job](/basics/jobs) handle instead. Computed
|
|
columns are off by default in LanceDB Enterprise, and its administrator has to enable
|
|
them before a column can be declared there.
|
|
|
|
```python
|
|
job = table.refresh_column_async("power_score")
|
|
result = job.wait()
|
|
result.rows_assigned
|
|
```
|
|
|
|
A backfill over a large table can therefore be started, left, and checked on
|
|
from a different process: the connection does not have to stay open.
|
|
|
|
Enterprise manages the compute for the refresh. Declaring a computed column
|
|
doesn't refresh it on a schedule unless your deployment enables automatic
|
|
refresh; see [Automatic refresh](/feature-engineering/refresh-column#automatic).
|
|
{/* Draft for the Functions release: That is also why a function must declare the environment it needs. */}
|
|
|
|
## Recovering from a failure {#failures}
|
|
|
|
{/*
|
|
Draft for the Functions release.
|
|
|
|
A function that raises fails the job. There is no per-row retry or skip policy
|
|
in the function definition, so a function calling a flaky external service
|
|
should handle that itself, inside the function body.
|
|
*/}
|
|
|
|
When a refresh fails, fix the cause and run the refresh
|
|
again. The new refresh fills what is still empty or stale. How much of the
|
|
failed run was kept depends on where it ran (see
|
|
[Refresh a computed column](/feature-engineering/refresh-column#failures)), and a retry
|
|
can recompute values that were already written, so do not count on it costing
|
|
only the rows that did not complete.
|
|
|
|
## Related guides {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Create a view" icon="eye" href="/feature-engineering/view">
|
|
A saved query, with no storage or refresh of its own.
|
|
</Card>
|
|
<Card title="Create a computed column" icon="table-columns" href="/feature-engineering/add-computed-column">
|
|
Declare a column from a SQL expression.
|
|
</Card>
|
|
<Card title="Refresh a computed column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
|
Fill the rows that lack values, locally or as a job.
|
|
</Card>
|
|
<Card title="Create a materialized view" icon="table" href="/feature-engineering/materialized-views">
|
|
Store a query's result as a table.
|
|
</Card>
|
|
<Card title="Refresh a materialized view" icon="arrows-rotate" href="/feature-engineering/refresh-materialized-view">
|
|
Bring a derived table up to date with its source.
|
|
</Card>
|
|
{/*
|
|
Draft for the Functions release:
|
|
<Card title="Functions" icon="code" href="/management/function">
|
|
Register Python to fill a column with.
|
|
</Card>
|
|
*/}
|
|
</CardGroup>
|