rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-12 23:02:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: document feature engineering against the LanceDB API

Browse Source 
The feature engineering documentation describes the geneva package, a separate
SDK. The capability it covers now lives in this client — materialized views,
computed columns, and registered functions are all shipped API with no pages
describing them.

Four pages cover the three mechanisms and, more usefully, when each applies.
The overview leads with that choice rather than with concepts, because the
mechanisms overlap and picking the wrong one is the expensive mistake.

Two things they say plainly that the API reference cannot:

Declaring and filling are separate everywhere, and filling never revisits a row
it has already done. That makes refreshes idempotent and appends cheap, and it
means a filled value does not track its inputs — surprising if you expect a
spreadsheet, so it is stated rather than left to be discovered.

The availability split runs the opposite way to expectation. Materialized views
are local only and raise on a remote connection; functions are Cloud and
Enterprise only and raise on a local one. Someone reading about both in one
sitting will otherwise assume the usual direction and be wrong in one of them.

Examples are the tested ones from the docstrings, so they cannot drift from the
behaviour they describe.

The training and tutorial pages that pointed into the geneva documentation now
point here.
This commit is contained in:
Jack Ye
2026-09-03 23:39:51 -07:00
parent e687c64f94
commit c0fe2eacda
7 changed files with 418 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/docs.json
+10 -1
View File
@@ -6,7 +6,7 @@
"theme": "mint",
"name": "LanceDB",
"banner": {
"content": "[Why Multimodal Data Needs a Better Lakehouse? — Download the Research Study](https://lancedb.com/download/)",
"content": "[Why Multimodal Data Needs a Better Lakehouse? \u2014 Download the Research Study](https://lancedb.com/download/)",
"dismissible": true
},
"colors": {
@@ -140,6 +140,15 @@
}
]
},
{
"group": "Feature engineering",
"pages": [
"feature-engineering/index",
"feature-engineering/computed-columns",
"feature-engineering/materialized-views",
"feature-engineering/functions"
]
},
{
"group": "Support",
"pages": [
docs/web/feature-engineering/computed-columns.mdx
+113
View File
@@ -0,0 +1,113 @@
---
title: "Computed columns"
sidebarTitle: "Computed columns"
description: "Declare a column from a SQL expression and fill it when you choose."
icon: table-columns
keywords: ["computed columns", "add_columns", "refresh_column", "backfill"]
---
A computed column is declared with an expression and starts with no values. A
separate refresh fills the rows that lack them. Splitting the two means
declaring a column on a billion-row table costs the same as declaring one on an
empty table: nothing is evaluated until you ask for it.
## Declare and fill {#declare-and-fill}
```python
import lancedb
db = lancedb.connect("./.lancedb")
table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}])
table.add_columns(computed={"doubled": "x * 2"})
# AddColumnsResult(version=2)
table.refresh_column("doubled")
# RefreshColumnResult(rows_filled=2, version=3)
table.to_arrow().sort_by("x").to_pandas()
# x doubled
# 0 1 2
# 1 2 4
```
`add_columns(computed=...)` commits a new table version with the column present
and empty. `refresh_column` fills it and commits another. Both are ordinary
table versions, so the usual [versioning](/tables/versioning) applies.
## `computed` and `transforms` are different {#computed-vs-transforms}
`add_columns` takes either, and the difference matters:
- **`transforms`** evaluates the expression *now*, for every row, as part of the
call. Good for a small table or a column you want populated immediately.
- **`computed`** stores the expression and evaluates nothing. The column is
committed empty and filled later by `refresh_column`.
They cannot be combined in one call. Reach for `computed` when the table is
large enough that you would rather control when the work happens.
## Refreshes do not revisit filled rows {#idempotent}
A refresh gives values to rows that have none. Rows already filled are left
alone. Two consequences follow:
**Appending is cheap.** Rows added since the last refresh are filled by the
next one; everything before it is skipped. Backfilling a column and then
ingesting more data does not mean recomputing the backfill.
**A filled value does not track its input.** If `doubled` derives from `x` and
a row's `x` is later updated, `doubled` keeps the value computed when it was
filled. Recomputing means dropping the column and declaring it again.
That is what makes the call idempotent and safe to retry, but it is the opposite
of a spreadsheet formula, so it is worth being deliberate about which columns
you derive this way.
## Constraints on the inputs {#input-constraints}
While a computed column's declaration references another column, that column
cannot be renamed, retyped, or dropped. The declaration is part of the schema,
and LanceDB will not let you invalidate it out from under itself.
Declaration order is also dependency order: a computed column may reference one
declared earlier in the same mapping.
## Filling as a job {#async}
<Badge color="red">Cloud and Enterprise</Badge>
Against a remote database the refresh runs on the server. `refresh_column`
raises there; use `refresh_column_async`, which returns a job you can wait on or
poll instead of holding a connection open:
```python
job = table.refresh_column_async("doubled")
result = job.wait()
result.rows_assigned
# 2
job.status()
# 'finished'
```
The job handle carries `id`, `status()`, `wait(timeout=...)` and `cancel()`, so
a long backfill can be started, left, and checked on later.
## Filling with Python instead of SQL {#functions}
<Badge color="red">Cloud and Enterprise</Badge>
When the value cannot be expressed in SQL — an embedding, a model call, an image
decode — the column is filled by a registered function instead of an expression.
See [Functions](/feature-engineering/functions).
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Materialized views" icon="table" href="/feature-engineering/materialized-views">
Derive a whole table rather than a column.
</Card>
<Card title="Functions" icon="code" href="/feature-engineering/functions">
Fill a column with Python.
</Card>
</CardGroup>
docs/web/feature-engineering/functions.mdx
+111
View File
@@ -0,0 +1,111 @@
---
title: "Functions"
sidebarTitle: "Functions"
description: "Register a Python function once and apply it as a column, for work SQL cannot express."
icon: code
keywords: ["udf", "functions", "create_function", "embedding", "feature engineering"]
---
<Badge color="red">Cloud and Enterprise</Badge>
Some columns cannot be written as a SQL expression: an embedding, a model call,
an image decode. A function is a Python callable registered with the database,
versioned, and then applied as a column — the code runs where the data is
rather than pulling rows to your process.
Registration is a remote operation. On a local connection `create_function`
raises `NotImplementedError`; use a
[computed column](/feature-engineering/computed-columns) instead.
## Define a function {#define}
`@udf` packages a scalar Python callable. Input and output types are inferred
from annotations:
```python
from lancedb import udf
@udf(pip=["numpy==2.2.0"])
def score(value: float) -> float:
return value * 2
```
The decorated object stays callable, so `score(1.5)` runs the original function
locally and unit tests stay ordinary tests.
Where annotations cannot express the Arrow type precisely, pass `input_schema`
and `output_schema` together — one without the other is rejected. Outputs must
be non-nullable, because a physical NULL is what marks a computed-column row as
unfilled.
## Declare the environment {#environment}
The function runs remotely, so its dependencies are part of its definition:
| | |
| --- | --- |
| `pip` | requirements for the remote environment |
| `conda`, `conda_channels` | conda packages instead of pip; channels in priority order |
| `env` | environment variables |
| `python_version` | remote Python major/minor, defaulting to yours |
| `gpu` | whether every execution requires a GPU |
These are part of the **immutable function version**. Changing a dependency
produces a new version rather than altering the one already registered, so a
column filled by version 1 is unaffected by version 2.
## What gets packaged {#packaging}
<Warning>
The artifact is a **snapshot**: the function source, plus exactly the
module-level names it references — modules as imports, importable classes and
functions as imports, literals inlined.
Code that reaches the module namespace another way is rejected where it can be
seen and unsupported where it cannot: `globals()`, `eval`, `sys.modules`,
`builtins`. Closures and a non-standard `__builtins__` are rejected outright.
</Warning>
In practice this means a function should be self-contained: take its inputs as
arguments, import what it needs at module level, and avoid capturing state from
the surrounding scope.
## Register it {#register}
Registration returns a job; the immutable version exists once the job succeeds.
```python
job = db.create_function_async(score)
version = job.wait()
```
`create_function` is the blocking equivalent. A registered version is fetched
with `db.get_function(name, version=...)`.
## Apply it as a column {#apply}
A registered function is applied through `add_columns`, in the `transforms`
mapping:
```python
table.add_columns(transforms={"doubled": score(table["value"])})
```
A mapping with one function application keeps its scalar or named-struct result
in the named column. A bare named-struct application expands its result fields
as one atomic binding, with aliases from `rename(columns=...)`.
Filling then works exactly as it does for any computed column — see
[Computed columns](/feature-engineering/computed-columns) for refresh
semantics, including the rule that filled rows are never revisited.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/computed-columns">
Refresh semantics, and the SQL alternative.
</Card>
<Card title="Embedding functions" icon="vector-square" href="/embedding">
Built-in embedding providers, which need no registration.
</Card>
</CardGroup>
docs/web/feature-engineering/index.mdx
+71
View File
@@ -0,0 +1,71 @@
---
title: "Feature engineering"
sidebarTitle: "Overview"
description: "Derive columns and tables from data already in LanceDB, without moving it somewhere else to compute."
icon: layer-group
keywords: ["feature engineering", "computed columns", "materialized views", "functions", "udf"]
---
Feature engineering in LanceDB happens where the data already is. Rather than
exporting a table, computing somewhere else and writing the result back, you
declare what a column or a table should contain and let LanceDB fill it.
Three mechanisms cover different shapes of that problem. They compose — a
materialized view can select computed columns, and a computed column can be
filled by a registered function — but each is worth understanding on its own.
## Which one to reach for {#which-one}
| | What it produces | Filled by | Available |
| --- | --- | --- | --- |
| [Computed columns](/feature-engineering/computed-columns) | a new column on an existing table | `refresh_column()` | everywhere |
| [Materialized views](/feature-engineering/materialized-views) | a new table derived by a query | `refresh()` | local databases |
| [Functions](/feature-engineering/functions) | a computed column filled by Python | a registered function | Cloud and Enterprise |
**Computed columns** are the common case. A column is declared with a SQL
expression and starts empty; a refresh gives values to the rows that lack them.
Declaring one costs the same on a billion-row table as on an empty one, because
nothing is evaluated until you ask.
**Materialized views** derive a whole table — a filtered, projected, or
aggregated view of a source — and keep it up to date incrementally as the source
changes. The result is an ordinary table: queryable, indexable, searchable.
**Functions** run Python where the data is, for work SQL cannot express:
embedding a column, calling a model, decoding an image. The function is
registered once, versioned, and then applied as a column.
## The shape they share {#the-shape}
All three separate **declaring** from **filling**, and that separation is the
point:
- Declaring is cheap and immediate. It records intent in the schema and commits
a new table version, with no data written.
- Filling is a separate call you control, can retry, and — on Cloud and
Enterprise — can hand to the server as a job rather than waiting on.
Filling never revisits work it has already done. A refresh gives values to rows
that lack them and leaves the rest alone, so calling it twice is not twice the
work, and appending rows means the next refresh handles only the new ones.
The corollary is worth stating plainly: **a filled value is not recomputed when
its inputs change.** If a column derives from `price` and a row's `price` is
later updated, the derived value stays as it was computed. Recomputing means
dropping the column and declaring it again. That is a deliberate trade — it is
what makes refreshes idempotent and cheap — but it will surprise you if you
expect a spreadsheet.
## Where to go next {#next}
<CardGroup cols={3}>
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/computed-columns">
Declare a column from a SQL expression and fill it.
</Card>
<Card title="Materialized views" icon="table" href="/feature-engineering/materialized-views">
Derive a table from a query and refresh it incrementally.
</Card>
<Card title="Functions" icon="code" href="/feature-engineering/functions">
Run Python over your data to produce a column.
</Card>
</CardGroup>
docs/web/feature-engineering/materialized-views.mdx
+107
View File
@@ -0,0 +1,107 @@
---
title: "Materialized views"
sidebarTitle: "Materialized views"
description: "Derive a table from a query over another table, and keep it current incrementally."
icon: table
keywords: ["materialized view", "create_materialized_view", "refresh", "incremental"]
---
A materialized view is a table whose contents are defined by a query over
another table. The query is recorded in the view's schema; a refresh computes
the rows.
The result is an ordinary table. It can be queried, indexed, and searched, and
it appears in `table_names()` — nothing about reading it is special.
## Create and refresh {#create-and-refresh}
```python
import lancedb
db = lancedb.connect(
"./.lancedb",
storage_options={"new_table_enable_stable_row_ids": "true"},
)
table = db.create_table("people", [{"name": "ada", "age": 36},
{"name": "kid", "age": 7}])
view = db.create_materialized_view(
"adults",
"people",
select=["name", ("shout", "upper(name)")],
where="age >= 18",
)
result = view.refresh()
result.rows_written
# 1
```
`select` takes column names, `(alias, expression)` pairs, or a dict of the same.
Omitting it selects every source column, expanded against the schema as it was
at creation time. `where` filters source rows; `limit` caps the view.
## The source needs stable row ids {#stable-row-ids}
<Warning>
The source table must have been created with stable row ids:
```python
db = lancedb.connect(uri, storage_options={"new_table_enable_stable_row_ids": "true"})
```
They are what keeps the view's provenance valid when the source is compacted,
and **they cannot be enabled after a table exists**. A table created without
them cannot later become the source of a view — it has to be recreated.
</Warning>
This is the one decision here that is hard to undo, so it is worth setting the
option on any table you might later want to derive from.
## Refreshes are incremental {#incremental}
`refresh()` reconciles the source's changes into the view — rows added, changed,
or removed since the last refresh — rather than recomputing from scratch. It
falls back to a full rebuild when the changes cannot be reconciled.
Two arguments override that:
- `full=True` forces a rebuild.
- `source_version=N` refreshes against that source version instead of the
latest, which is how you make a view reproduce a known point in time.
Concurrent refreshes of one view do not duplicate rows. If two plan the same
source rows, they conflict on commit and the loser raises rather than writing
them twice — so a retry loop around `refresh()` is safe.
## Local databases only {#availability}
<Warning>
Materialized views are supported on **local databases**. Calling
`create_materialized_view` on a remote connection raises `NotImplementedError`.
This is the reverse of [functions](/feature-engineering/functions), which run
only against Cloud and Enterprise. If you are deriving tables on a remote
database, a [computed column](/feature-engineering/computed-columns) filled by a
server job is usually the closer fit.
</Warning>
## Working with the view {#working-with-the-view}
`view.table()` returns the underlying table for querying, indexing, or search.
`view.definition()` returns the recorded query — the select list, predicate, and
limit — which is how you check what a view actually contains without inferring
it from the rows.
An existing view is reopened with `db.open_materialized_view(name)`.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/computed-columns">
Derive a column instead of a whole table.
</Card>
<Card title="Versioning" icon="clock-rotate-left" href="/tables/versioning">
Every refresh commits a table version.
</Card>
</CardGroup>
docs/web/training/object-detection.mdx
+3 -3
View File
@@ -42,7 +42,7 @@ A perception model fine-tuned on a generic dataset typically misses the long-tai
| **Nighttime pedestrians** | `timeofday = 'night' AND has_person = true` |
| **Distant pedestrians** | `has_person = true AND person_bbox_area_pct < 30.0` |
Each curated slice becomes a [materialized view](/geneva/jobs/materialized-views) — a named, refreshable SQL filter over the source table — and the training script loads it by name. New footage flows in through `add()` → `backfill()` → `refresh()`; no manifests, no exports, no reshuffling on disk.
Each curated slice becomes a [materialized view](/feature-engineering/materialized-views) — a named, refreshable SQL filter over the source table — and the training script loads it by name. New footage flows in through `add()` → `backfill()` → `refresh()`; no manifests, no exports, no reshuffling on disk.
## 1\. Schema {#1-schema}
@@ -74,7 +74,7 @@ Ingestion streams `pa.RecordBatch`es of raw frames + annotations directly into a
## 2\. Backfill curation features with Geneva {#2-backfill-curation-features-with-geneva}
Curation signals are added as columns on the same table using [Geneva UDFs](/geneva/). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows.
Curation signals are added as columns on the same table using [computed columns](/feature-engineering/computed-columns). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows.
```py Python icon=Python
import pyarrow as pa
@@ -117,7 +117,7 @@ with gconn.local_ray_context():
```
<Note>
Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See the [Geneva end-to-end example](/geneva/end-to-end) for more on the backfill pattern.
Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See [computed columns](/feature-engineering/computed-columns) for how a backfill fills only the rows that lack values.
</Note>
## 3\. Define training splits as materialized views {#3-define-training-splits-as-materialized-views}
docs/web/tutorials/feature-engineering/index.mdx
+3 -3
View File
@@ -17,6 +17,6 @@ The relevant section of the documentation are listed below.
| Feature | Description |
|:--------|:------------|
| [Feature Engineering](/geneva/) | Learn the fundamentals of feature engineering. |
| [Materialized Views](/geneva/jobs/materialized-views) | Learn more about creating materialized views. |
| [Contexts](/geneva/jobs/contexts) | Learn how to run your job on a Ray cluster for production use. |
| [Feature engineering](/feature-engineering) | The three ways to derive columns and tables. |
| [Materialized views](/feature-engineering/materialized-views) | Derive a table from a query and refresh it incrementally. |
| [Computed columns](/feature-engineering/computed-columns) | Declare a column from an expression and fill it. |