mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-17 09:12:28 +00:00
Jack Ye
4015560b83
The Documentation tab now reads as a manual over Python, TypeScript and SQL rather than a Python guide with SQL bolted on. Basics replaces the Catalog section: what the catalog holds, what each API surface can do, jobs, agentic skills, and what Lance is. Management is new and is the CRUD and listing reference for every catalog object, with a column per language so the gaps are visible -- TypeScript has no function or SQL methods, SQL needs a deployment. Search becomes Query and gains a SELECT reference, since an analytical query is a query whether it came from a builder or a statement. Jobs is new. Every long-running operation already returned the same handle from an _async method; nothing documented it. The SQL function reference gains what Lance registers on top of DataFusion: contains_tokens, backed by the FTS index, and the JSON accessors, backed by a JSON scalar index. Geospatial st_* functions are behind a build feature LanceDB does not ship, which is now stated rather than left to a confusing error. Removes the Geneva pages' orphaned snippets, merges FTS Examples into Full-Text Search, and splits Query Optimization into Explain and Explain Analyze.
77 lines
3.8 KiB
Plaintext
77 lines
3.8 KiB
Plaintext
---
|
|
title: "LanceDB Feature Engineering"
|
|
sidebarTitle: "Overview"
|
|
description: "Derive columns and tables from data already in the lakehouse, without moving it somewhere else to compute."
|
|
icon: layer-group
|
|
keywords: ["feature engineering", "computed columns", "materialized views", "functions", "lakehouse"]
|
|
---
|
|
|
|
Feature engineering happens where the data already is. Rather than exporting a
|
|
table, computing somewhere else and writing the result back, you declare what a
|
|
column or a table should contain and let LanceDB fill it.
|
|
|
|
**Functions and materialized views are objects in the Lance Multimodal
|
|
Lakehouse**, alongside tables and indexes. They are created, versioned, and
|
|
refreshed like anything else in it — not a separate system bolted alongside, and
|
|
not a pipeline you assemble out of external orchestration.
|
|
|
|
Three mechanisms cover different shapes of the problem. They compose — a
|
|
materialized view can select computed columns, and a computed column can be
|
|
filled by a registered function — but each is worth understanding on its own.
|
|
|
|
## Which one to reach for {#which-one}
|
|
|
|
| | What it produces | Filled by | Available |
|
|
| --- | --- | --- | --- |
|
|
| [Computed columns](/feature-engineering/computed-columns) | a new column on an existing table | `refresh_column()` | everywhere |
|
|
| [Materialized views](/feature-engineering/materialized-views) | a new table derived by a query | `refresh()` | local databases |
|
|
| [Functions](/feature-engineering/functions) | a computed column filled by Python | a registered function | Enterprise |
|
|
|
|
**Computed columns** are the common case. A column is declared with a SQL
|
|
expression and starts empty; a refresh gives values to the rows that lack them.
|
|
Declaring one costs the same on a billion-row table as on an empty one, because
|
|
nothing is evaluated until you ask.
|
|
|
|
**Materialized views** derive a whole table — a filtered, projected, or
|
|
aggregated view of a source — and keep it up to date incrementally as the source
|
|
changes. The result is an ordinary table: queryable, indexable, searchable.
|
|
|
|
**Functions** run Python where the data is, for work SQL cannot express:
|
|
embedding a column, calling a model, decoding an image. The function is
|
|
registered once, versioned, and then applied as a column.
|
|
|
|
## The shape they share {#the-shape}
|
|
|
|
All three separate **declaring** from **filling**, and that separation is the
|
|
point:
|
|
|
|
- Declaring is cheap and immediate. It records intent in the schema and commits
|
|
a new table version, with no data written.
|
|
- Filling is a separate call you control, can retry, and — on Cloud and
|
|
Enterprise — can hand to the server as a job rather than waiting on.
|
|
|
|
Filling never revisits work it has already done. A refresh gives values to rows
|
|
that lack them and leaves the rest alone, so calling it twice is not twice the
|
|
work, and appending rows means the next refresh handles only the new ones.
|
|
|
|
The corollary is worth stating plainly: **a filled value is not recomputed when
|
|
its inputs change.** If a column derives from `price` and a row's `price` is
|
|
later updated, the derived value stays as it was computed. Recomputing means
|
|
dropping the column and declaring it again. That is a deliberate trade — it is
|
|
what makes refreshes idempotent and cheap — but it will surprise you if you
|
|
expect a spreadsheet.
|
|
|
|
## Where to go next {#next}
|
|
|
|
<CardGroup cols={3}>
|
|
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/computed-columns">
|
|
Declare a column from a SQL expression and fill it.
|
|
</Card>
|
|
<Card title="Materialized views" icon="table" href="/feature-engineering/materialized-views">
|
|
Derive a table from a query and refresh it incrementally.
|
|
</Card>
|
|
<Card title="Functions" icon="code" href="/feature-engineering/functions">
|
|
Run Python over your data to produce a column.
|
|
</Card>
|
|
</CardGroup>
|