diff --git a/docs/web/docs.json b/docs/web/docs.json
index 93c607886..2490ff5b1 100644
--- a/docs/web/docs.json
+++ b/docs/web/docs.json
@@ -6,7 +6,7 @@
"theme": "mint",
"name": "LanceDB",
"banner": {
- "content": "[Why Multimodal Data Needs a Better Lakehouse? — Download the Research Study](https://lancedb.com/download/)",
+ "content": "[Why Multimodal Data Needs a Better Lakehouse? \u2014 Download the Research Study](https://lancedb.com/download/)",
"dismissible": true
},
"colors": {
@@ -140,6 +140,15 @@
}
]
},
+ {
+ "group": "Feature engineering",
+ "pages": [
+ "feature-engineering/index",
+ "feature-engineering/computed-columns",
+ "feature-engineering/materialized-views",
+ "feature-engineering/functions"
+ ]
+ },
{
"group": "Support",
"pages": [
diff --git a/docs/web/feature-engineering/computed-columns.mdx b/docs/web/feature-engineering/computed-columns.mdx
new file mode 100644
index 000000000..6405698e6
--- /dev/null
+++ b/docs/web/feature-engineering/computed-columns.mdx
@@ -0,0 +1,113 @@
+---
+title: "Computed columns"
+sidebarTitle: "Computed columns"
+description: "Declare a column from a SQL expression and fill it when you choose."
+icon: table-columns
+keywords: ["computed columns", "add_columns", "refresh_column", "backfill"]
+---
+
+A computed column is declared with an expression and starts with no values. A
+separate refresh fills the rows that lack them. Splitting the two means
+declaring a column on a billion-row table costs the same as declaring one on an
+empty table: nothing is evaluated until you ask for it.
+
+## Declare and fill {#declare-and-fill}
+
+```python
+import lancedb
+
+db = lancedb.connect("./.lancedb")
+table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}])
+
+table.add_columns(computed={"doubled": "x * 2"})
+# AddColumnsResult(version=2)
+
+table.refresh_column("doubled")
+# RefreshColumnResult(rows_filled=2, version=3)
+
+table.to_arrow().sort_by("x").to_pandas()
+# x doubled
+# 0 1 2
+# 1 2 4
+```
+
+`add_columns(computed=...)` commits a new table version with the column present
+and empty. `refresh_column` fills it and commits another. Both are ordinary
+table versions, so the usual [versioning](/tables/versioning) applies.
+
+## `computed` and `transforms` are different {#computed-vs-transforms}
+
+`add_columns` takes either, and the difference matters:
+
+- **`transforms`** evaluates the expression *now*, for every row, as part of the
+ call. Good for a small table or a column you want populated immediately.
+- **`computed`** stores the expression and evaluates nothing. The column is
+ committed empty and filled later by `refresh_column`.
+
+They cannot be combined in one call. Reach for `computed` when the table is
+large enough that you would rather control when the work happens.
+
+## Refreshes do not revisit filled rows {#idempotent}
+
+A refresh gives values to rows that have none. Rows already filled are left
+alone. Two consequences follow:
+
+**Appending is cheap.** Rows added since the last refresh are filled by the
+next one; everything before it is skipped. Backfilling a column and then
+ingesting more data does not mean recomputing the backfill.
+
+**A filled value does not track its input.** If `doubled` derives from `x` and
+a row's `x` is later updated, `doubled` keeps the value computed when it was
+filled. Recomputing means dropping the column and declaring it again.
+
+That is what makes the call idempotent and safe to retry, but it is the opposite
+of a spreadsheet formula, so it is worth being deliberate about which columns
+you derive this way.
+
+## Constraints on the inputs {#input-constraints}
+
+While a computed column's declaration references another column, that column
+cannot be renamed, retyped, or dropped. The declaration is part of the schema,
+and LanceDB will not let you invalidate it out from under itself.
+
+Declaration order is also dependency order: a computed column may reference one
+declared earlier in the same mapping.
+
+## Filling as a job {#async}
+
+Cloud and Enterprise
+
+Against a remote database the refresh runs on the server. `refresh_column`
+raises there; use `refresh_column_async`, which returns a job you can wait on or
+poll instead of holding a connection open:
+
+```python
+job = table.refresh_column_async("doubled")
+result = job.wait()
+result.rows_assigned
+# 2
+job.status()
+# 'finished'
+```
+
+The job handle carries `id`, `status()`, `wait(timeout=...)` and `cancel()`, so
+a long backfill can be started, left, and checked on later.
+
+## Filling with Python instead of SQL {#functions}
+
+Cloud and Enterprise
+
+When the value cannot be expressed in SQL — an embedding, a model call, an image
+decode — the column is filled by a registered function instead of an expression.
+See [Functions](/feature-engineering/functions).
+
+## What to read next {#next}
+
+
+
+ Derive a whole table rather than a column.
+
+
+ Fill a column with Python.
+
+
diff --git a/docs/web/feature-engineering/functions.mdx b/docs/web/feature-engineering/functions.mdx
new file mode 100644
index 000000000..660ef5517
--- /dev/null
+++ b/docs/web/feature-engineering/functions.mdx
@@ -0,0 +1,111 @@
+---
+title: "Functions"
+sidebarTitle: "Functions"
+description: "Register a Python function once and apply it as a column, for work SQL cannot express."
+icon: code
+keywords: ["udf", "functions", "create_function", "embedding", "feature engineering"]
+---
+
+Cloud and Enterprise
+
+Some columns cannot be written as a SQL expression: an embedding, a model call,
+an image decode. A function is a Python callable registered with the database,
+versioned, and then applied as a column — the code runs where the data is
+rather than pulling rows to your process.
+
+Registration is a remote operation. On a local connection `create_function`
+raises `NotImplementedError`; use a
+[computed column](/feature-engineering/computed-columns) instead.
+
+## Define a function {#define}
+
+`@udf` packages a scalar Python callable. Input and output types are inferred
+from annotations:
+
+```python
+from lancedb import udf
+
+@udf(pip=["numpy==2.2.0"])
+def score(value: float) -> float:
+ return value * 2
+```
+
+The decorated object stays callable, so `score(1.5)` runs the original function
+locally and unit tests stay ordinary tests.
+
+Where annotations cannot express the Arrow type precisely, pass `input_schema`
+and `output_schema` together — one without the other is rejected. Outputs must
+be non-nullable, because a physical NULL is what marks a computed-column row as
+unfilled.
+
+## Declare the environment {#environment}
+
+The function runs remotely, so its dependencies are part of its definition:
+
+| | |
+| --- | --- |
+| `pip` | requirements for the remote environment |
+| `conda`, `conda_channels` | conda packages instead of pip; channels in priority order |
+| `env` | environment variables |
+| `python_version` | remote Python major/minor, defaulting to yours |
+| `gpu` | whether every execution requires a GPU |
+
+These are part of the **immutable function version**. Changing a dependency
+produces a new version rather than altering the one already registered, so a
+column filled by version 1 is unaffected by version 2.
+
+## What gets packaged {#packaging}
+
+
+The artifact is a **snapshot**: the function source, plus exactly the
+module-level names it references — modules as imports, importable classes and
+functions as imports, literals inlined.
+
+Code that reaches the module namespace another way is rejected where it can be
+seen and unsupported where it cannot: `globals()`, `eval`, `sys.modules`,
+`builtins`. Closures and a non-standard `__builtins__` are rejected outright.
+
+
+In practice this means a function should be self-contained: take its inputs as
+arguments, import what it needs at module level, and avoid capturing state from
+the surrounding scope.
+
+## Register it {#register}
+
+Registration returns a job; the immutable version exists once the job succeeds.
+
+```python
+job = db.create_function_async(score)
+version = job.wait()
+```
+
+`create_function` is the blocking equivalent. A registered version is fetched
+with `db.get_function(name, version=...)`.
+
+## Apply it as a column {#apply}
+
+A registered function is applied through `add_columns`, in the `transforms`
+mapping:
+
+```python
+table.add_columns(transforms={"doubled": score(table["value"])})
+```
+
+A mapping with one function application keeps its scalar or named-struct result
+in the named column. A bare named-struct application expands its result fields
+as one atomic binding, with aliases from `rename(columns=...)`.
+
+Filling then works exactly as it does for any computed column — see
+[Computed columns](/feature-engineering/computed-columns) for refresh
+semantics, including the rule that filled rows are never revisited.
+
+## What to read next {#next}
+
+
+
+ Refresh semantics, and the SQL alternative.
+
+
+ Built-in embedding providers, which need no registration.
+
+
diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx
new file mode 100644
index 000000000..af1697bdc
--- /dev/null
+++ b/docs/web/feature-engineering/index.mdx
@@ -0,0 +1,71 @@
+---
+title: "Feature engineering"
+sidebarTitle: "Overview"
+description: "Derive columns and tables from data already in LanceDB, without moving it somewhere else to compute."
+icon: layer-group
+keywords: ["feature engineering", "computed columns", "materialized views", "functions", "udf"]
+---
+
+Feature engineering in LanceDB happens where the data already is. Rather than
+exporting a table, computing somewhere else and writing the result back, you
+declare what a column or a table should contain and let LanceDB fill it.
+
+Three mechanisms cover different shapes of that problem. They compose — a
+materialized view can select computed columns, and a computed column can be
+filled by a registered function — but each is worth understanding on its own.
+
+## Which one to reach for {#which-one}
+
+| | What it produces | Filled by | Available |
+| --- | --- | --- | --- |
+| [Computed columns](/feature-engineering/computed-columns) | a new column on an existing table | `refresh_column()` | everywhere |
+| [Materialized views](/feature-engineering/materialized-views) | a new table derived by a query | `refresh()` | local databases |
+| [Functions](/feature-engineering/functions) | a computed column filled by Python | a registered function | Cloud and Enterprise |
+
+**Computed columns** are the common case. A column is declared with a SQL
+expression and starts empty; a refresh gives values to the rows that lack them.
+Declaring one costs the same on a billion-row table as on an empty one, because
+nothing is evaluated until you ask.
+
+**Materialized views** derive a whole table — a filtered, projected, or
+aggregated view of a source — and keep it up to date incrementally as the source
+changes. The result is an ordinary table: queryable, indexable, searchable.
+
+**Functions** run Python where the data is, for work SQL cannot express:
+embedding a column, calling a model, decoding an image. The function is
+registered once, versioned, and then applied as a column.
+
+## The shape they share {#the-shape}
+
+All three separate **declaring** from **filling**, and that separation is the
+point:
+
+- Declaring is cheap and immediate. It records intent in the schema and commits
+ a new table version, with no data written.
+- Filling is a separate call you control, can retry, and — on Cloud and
+ Enterprise — can hand to the server as a job rather than waiting on.
+
+Filling never revisits work it has already done. A refresh gives values to rows
+that lack them and leaves the rest alone, so calling it twice is not twice the
+work, and appending rows means the next refresh handles only the new ones.
+
+The corollary is worth stating plainly: **a filled value is not recomputed when
+its inputs change.** If a column derives from `price` and a row's `price` is
+later updated, the derived value stays as it was computed. Recomputing means
+dropping the column and declaring it again. That is a deliberate trade — it is
+what makes refreshes idempotent and cheap — but it will surprise you if you
+expect a spreadsheet.
+
+## Where to go next {#next}
+
+
+
+ Declare a column from a SQL expression and fill it.
+
+
+ Derive a table from a query and refresh it incrementally.
+
+
+ Run Python over your data to produce a column.
+
+
diff --git a/docs/web/feature-engineering/materialized-views.mdx b/docs/web/feature-engineering/materialized-views.mdx
new file mode 100644
index 000000000..b6d267ae4
--- /dev/null
+++ b/docs/web/feature-engineering/materialized-views.mdx
@@ -0,0 +1,107 @@
+---
+title: "Materialized views"
+sidebarTitle: "Materialized views"
+description: "Derive a table from a query over another table, and keep it current incrementally."
+icon: table
+keywords: ["materialized view", "create_materialized_view", "refresh", "incremental"]
+---
+
+A materialized view is a table whose contents are defined by a query over
+another table. The query is recorded in the view's schema; a refresh computes
+the rows.
+
+The result is an ordinary table. It can be queried, indexed, and searched, and
+it appears in `table_names()` — nothing about reading it is special.
+
+## Create and refresh {#create-and-refresh}
+
+```python
+import lancedb
+
+db = lancedb.connect(
+ "./.lancedb",
+ storage_options={"new_table_enable_stable_row_ids": "true"},
+)
+table = db.create_table("people", [{"name": "ada", "age": 36},
+ {"name": "kid", "age": 7}])
+
+view = db.create_materialized_view(
+ "adults",
+ "people",
+ select=["name", ("shout", "upper(name)")],
+ where="age >= 18",
+)
+
+result = view.refresh()
+result.rows_written
+# 1
+```
+
+`select` takes column names, `(alias, expression)` pairs, or a dict of the same.
+Omitting it selects every source column, expanded against the schema as it was
+at creation time. `where` filters source rows; `limit` caps the view.
+
+## The source needs stable row ids {#stable-row-ids}
+
+
+The source table must have been created with stable row ids:
+
+```python
+db = lancedb.connect(uri, storage_options={"new_table_enable_stable_row_ids": "true"})
+```
+
+They are what keeps the view's provenance valid when the source is compacted,
+and **they cannot be enabled after a table exists**. A table created without
+them cannot later become the source of a view — it has to be recreated.
+
+
+This is the one decision here that is hard to undo, so it is worth setting the
+option on any table you might later want to derive from.
+
+## Refreshes are incremental {#incremental}
+
+`refresh()` reconciles the source's changes into the view — rows added, changed,
+or removed since the last refresh — rather than recomputing from scratch. It
+falls back to a full rebuild when the changes cannot be reconciled.
+
+Two arguments override that:
+
+- `full=True` forces a rebuild.
+- `source_version=N` refreshes against that source version instead of the
+ latest, which is how you make a view reproduce a known point in time.
+
+Concurrent refreshes of one view do not duplicate rows. If two plan the same
+source rows, they conflict on commit and the loser raises rather than writing
+them twice — so a retry loop around `refresh()` is safe.
+
+## Local databases only {#availability}
+
+
+Materialized views are supported on **local databases**. Calling
+`create_materialized_view` on a remote connection raises `NotImplementedError`.
+
+This is the reverse of [functions](/feature-engineering/functions), which run
+only against Cloud and Enterprise. If you are deriving tables on a remote
+database, a [computed column](/feature-engineering/computed-columns) filled by a
+server job is usually the closer fit.
+
+
+## Working with the view {#working-with-the-view}
+
+`view.table()` returns the underlying table for querying, indexing, or search.
+`view.definition()` returns the recorded query — the select list, predicate, and
+limit — which is how you check what a view actually contains without inferring
+it from the rows.
+
+An existing view is reopened with `db.open_materialized_view(name)`.
+
+## What to read next {#next}
+
+
+
+ Derive a column instead of a whole table.
+
+
+ Every refresh commits a table version.
+
+
diff --git a/docs/web/training/object-detection.mdx b/docs/web/training/object-detection.mdx
index 2b516d64b..136a6e7a9 100644
--- a/docs/web/training/object-detection.mdx
+++ b/docs/web/training/object-detection.mdx
@@ -42,7 +42,7 @@ A perception model fine-tuned on a generic dataset typically misses the long-tai
| **Nighttime pedestrians** | `timeofday = 'night' AND has_person = true` |
| **Distant pedestrians** | `has_person = true AND person_bbox_area_pct < 30.0` |
-Each curated slice becomes a [materialized view](/geneva/jobs/materialized-views) — a named, refreshable SQL filter over the source table — and the training script loads it by name. New footage flows in through `add()` → `backfill()` → `refresh()`; no manifests, no exports, no reshuffling on disk.
+Each curated slice becomes a [materialized view](/feature-engineering/materialized-views) — a named, refreshable SQL filter over the source table — and the training script loads it by name. New footage flows in through `add()` → `backfill()` → `refresh()`; no manifests, no exports, no reshuffling on disk.
## 1\. Schema {#1-schema}
@@ -74,7 +74,7 @@ Ingestion streams `pa.RecordBatch`es of raw frames + annotations directly into a
## 2\. Backfill curation features with Geneva {#2-backfill-curation-features-with-geneva}
-Curation signals are added as columns on the same table using [Geneva UDFs](/geneva/). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows.
+Curation signals are added as columns on the same table using [computed columns](/feature-engineering/computed-columns). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows.
```py Python icon=Python
import pyarrow as pa
@@ -117,7 +117,7 @@ with gconn.local_ray_context():
```
-Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See the [Geneva end-to-end example](/geneva/end-to-end) for more on the backfill pattern.
+Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See [computed columns](/feature-engineering/computed-columns) for how a backfill fills only the rows that lack values.
## 3\. Define training splits as materialized views {#3-define-training-splits-as-materialized-views}
diff --git a/docs/web/tutorials/feature-engineering/index.mdx b/docs/web/tutorials/feature-engineering/index.mdx
index ad18e99a1..2d3a4478b 100644
--- a/docs/web/tutorials/feature-engineering/index.mdx
+++ b/docs/web/tutorials/feature-engineering/index.mdx
@@ -17,6 +17,6 @@ The relevant section of the documentation are listed below.
| Feature | Description |
|:--------|:------------|
-| [Feature Engineering](/geneva/) | Learn the fundamentals of feature engineering. |
-| [Materialized Views](/geneva/jobs/materialized-views) | Learn more about creating materialized views. |
-| [Contexts](/geneva/jobs/contexts) | Learn how to run your job on a Ray cluster for production use. |
+| [Feature engineering](/feature-engineering) | The three ways to derive columns and tables. |
+| [Materialized views](/feature-engineering/materialized-views) | Derive a table from a query and refresh it incrementally. |
+| [Computed columns](/feature-engineering/computed-columns) | Declare a column from an expression and fill it. |