From c0fe2eacda395b26b0709dd318cbe8e3531fae45 Mon Sep 17 00:00:00 2001 From: Jack Ye Date: Thu, 3 Sep 2026 23:39:51 -0700 Subject: [PATCH] docs: document feature engineering against the LanceDB API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The feature engineering documentation describes the geneva package, a separate SDK. The capability it covers now lives in this client — materialized views, computed columns, and registered functions are all shipped API with no pages describing them. Four pages cover the three mechanisms and, more usefully, when each applies. The overview leads with that choice rather than with concepts, because the mechanisms overlap and picking the wrong one is the expensive mistake. Two things they say plainly that the API reference cannot: Declaring and filling are separate everywhere, and filling never revisits a row it has already done. That makes refreshes idempotent and appends cheap, and it means a filled value does not track its inputs — surprising if you expect a spreadsheet, so it is stated rather than left to be discovered. The availability split runs the opposite way to expectation. Materialized views are local only and raise on a remote connection; functions are Cloud and Enterprise only and raise on a local one. Someone reading about both in one sitting will otherwise assume the usual direction and be wrong in one of them. Examples are the tested ones from the docstrings, so they cannot drift from the behaviour they describe. The training and tutorial pages that pointed into the geneva documentation now point here. --- docs/web/docs.json | 11 +- .../feature-engineering/computed-columns.mdx | 113 ++++++++++++++++++ docs/web/feature-engineering/functions.mdx | 111 +++++++++++++++++ docs/web/feature-engineering/index.mdx | 71 +++++++++++ .../materialized-views.mdx | 107 +++++++++++++++++ docs/web/training/object-detection.mdx | 6 +- .../tutorials/feature-engineering/index.mdx | 6 +- 7 files changed, 418 insertions(+), 7 deletions(-) create mode 100644 docs/web/feature-engineering/computed-columns.mdx create mode 100644 docs/web/feature-engineering/functions.mdx create mode 100644 docs/web/feature-engineering/index.mdx create mode 100644 docs/web/feature-engineering/materialized-views.mdx diff --git a/docs/web/docs.json b/docs/web/docs.json index 93c607886..2490ff5b1 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -6,7 +6,7 @@ "theme": "mint", "name": "LanceDB", "banner": { - "content": "[Why Multimodal Data Needs a Better Lakehouse? — Download the Research Study](https://lancedb.com/download/)", + "content": "[Why Multimodal Data Needs a Better Lakehouse? \u2014 Download the Research Study](https://lancedb.com/download/)", "dismissible": true }, "colors": { @@ -140,6 +140,15 @@ } ] }, + { + "group": "Feature engineering", + "pages": [ + "feature-engineering/index", + "feature-engineering/computed-columns", + "feature-engineering/materialized-views", + "feature-engineering/functions" + ] + }, { "group": "Support", "pages": [ diff --git a/docs/web/feature-engineering/computed-columns.mdx b/docs/web/feature-engineering/computed-columns.mdx new file mode 100644 index 000000000..6405698e6 --- /dev/null +++ b/docs/web/feature-engineering/computed-columns.mdx @@ -0,0 +1,113 @@ +--- +title: "Computed columns" +sidebarTitle: "Computed columns" +description: "Declare a column from a SQL expression and fill it when you choose." +icon: table-columns +keywords: ["computed columns", "add_columns", "refresh_column", "backfill"] +--- + +A computed column is declared with an expression and starts with no values. A +separate refresh fills the rows that lack them. Splitting the two means +declaring a column on a billion-row table costs the same as declaring one on an +empty table: nothing is evaluated until you ask for it. + +## Declare and fill {#declare-and-fill} + +```python +import lancedb + +db = lancedb.connect("./.lancedb") +table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}]) + +table.add_columns(computed={"doubled": "x * 2"}) +# AddColumnsResult(version=2) + +table.refresh_column("doubled") +# RefreshColumnResult(rows_filled=2, version=3) + +table.to_arrow().sort_by("x").to_pandas() +# x doubled +# 0 1 2 +# 1 2 4 +``` + +`add_columns(computed=...)` commits a new table version with the column present +and empty. `refresh_column` fills it and commits another. Both are ordinary +table versions, so the usual [versioning](/tables/versioning) applies. + +## `computed` and `transforms` are different {#computed-vs-transforms} + +`add_columns` takes either, and the difference matters: + +- **`transforms`** evaluates the expression *now*, for every row, as part of the + call. Good for a small table or a column you want populated immediately. +- **`computed`** stores the expression and evaluates nothing. The column is + committed empty and filled later by `refresh_column`. + +They cannot be combined in one call. Reach for `computed` when the table is +large enough that you would rather control when the work happens. + +## Refreshes do not revisit filled rows {#idempotent} + +A refresh gives values to rows that have none. Rows already filled are left +alone. Two consequences follow: + +**Appending is cheap.** Rows added since the last refresh are filled by the +next one; everything before it is skipped. Backfilling a column and then +ingesting more data does not mean recomputing the backfill. + +**A filled value does not track its input.** If `doubled` derives from `x` and +a row's `x` is later updated, `doubled` keeps the value computed when it was +filled. Recomputing means dropping the column and declaring it again. + +That is what makes the call idempotent and safe to retry, but it is the opposite +of a spreadsheet formula, so it is worth being deliberate about which columns +you derive this way. + +## Constraints on the inputs {#input-constraints} + +While a computed column's declaration references another column, that column +cannot be renamed, retyped, or dropped. The declaration is part of the schema, +and LanceDB will not let you invalidate it out from under itself. + +Declaration order is also dependency order: a computed column may reference one +declared earlier in the same mapping. + +## Filling as a job {#async} + +Cloud and Enterprise + +Against a remote database the refresh runs on the server. `refresh_column` +raises there; use `refresh_column_async`, which returns a job you can wait on or +poll instead of holding a connection open: + +```python +job = table.refresh_column_async("doubled") +result = job.wait() +result.rows_assigned +# 2 +job.status() +# 'finished' +``` + +The job handle carries `id`, `status()`, `wait(timeout=...)` and `cancel()`, so +a long backfill can be started, left, and checked on later. + +## Filling with Python instead of SQL {#functions} + +Cloud and Enterprise + +When the value cannot be expressed in SQL — an embedding, a model call, an image +decode — the column is filled by a registered function instead of an expression. +See [Functions](/feature-engineering/functions). + +## What to read next {#next} + + + + Derive a whole table rather than a column. + + + Fill a column with Python. + + diff --git a/docs/web/feature-engineering/functions.mdx b/docs/web/feature-engineering/functions.mdx new file mode 100644 index 000000000..660ef5517 --- /dev/null +++ b/docs/web/feature-engineering/functions.mdx @@ -0,0 +1,111 @@ +--- +title: "Functions" +sidebarTitle: "Functions" +description: "Register a Python function once and apply it as a column, for work SQL cannot express." +icon: code +keywords: ["udf", "functions", "create_function", "embedding", "feature engineering"] +--- + +Cloud and Enterprise + +Some columns cannot be written as a SQL expression: an embedding, a model call, +an image decode. A function is a Python callable registered with the database, +versioned, and then applied as a column — the code runs where the data is +rather than pulling rows to your process. + +Registration is a remote operation. On a local connection `create_function` +raises `NotImplementedError`; use a +[computed column](/feature-engineering/computed-columns) instead. + +## Define a function {#define} + +`@udf` packages a scalar Python callable. Input and output types are inferred +from annotations: + +```python +from lancedb import udf + +@udf(pip=["numpy==2.2.0"]) +def score(value: float) -> float: + return value * 2 +``` + +The decorated object stays callable, so `score(1.5)` runs the original function +locally and unit tests stay ordinary tests. + +Where annotations cannot express the Arrow type precisely, pass `input_schema` +and `output_schema` together — one without the other is rejected. Outputs must +be non-nullable, because a physical NULL is what marks a computed-column row as +unfilled. + +## Declare the environment {#environment} + +The function runs remotely, so its dependencies are part of its definition: + +| | | +| --- | --- | +| `pip` | requirements for the remote environment | +| `conda`, `conda_channels` | conda packages instead of pip; channels in priority order | +| `env` | environment variables | +| `python_version` | remote Python major/minor, defaulting to yours | +| `gpu` | whether every execution requires a GPU | + +These are part of the **immutable function version**. Changing a dependency +produces a new version rather than altering the one already registered, so a +column filled by version 1 is unaffected by version 2. + +## What gets packaged {#packaging} + + +The artifact is a **snapshot**: the function source, plus exactly the +module-level names it references — modules as imports, importable classes and +functions as imports, literals inlined. + +Code that reaches the module namespace another way is rejected where it can be +seen and unsupported where it cannot: `globals()`, `eval`, `sys.modules`, +`builtins`. Closures and a non-standard `__builtins__` are rejected outright. + + +In practice this means a function should be self-contained: take its inputs as +arguments, import what it needs at module level, and avoid capturing state from +the surrounding scope. + +## Register it {#register} + +Registration returns a job; the immutable version exists once the job succeeds. + +```python +job = db.create_function_async(score) +version = job.wait() +``` + +`create_function` is the blocking equivalent. A registered version is fetched +with `db.get_function(name, version=...)`. + +## Apply it as a column {#apply} + +A registered function is applied through `add_columns`, in the `transforms` +mapping: + +```python +table.add_columns(transforms={"doubled": score(table["value"])}) +``` + +A mapping with one function application keeps its scalar or named-struct result +in the named column. A bare named-struct application expands its result fields +as one atomic binding, with aliases from `rename(columns=...)`. + +Filling then works exactly as it does for any computed column — see +[Computed columns](/feature-engineering/computed-columns) for refresh +semantics, including the rule that filled rows are never revisited. + +## What to read next {#next} + + + + Refresh semantics, and the SQL alternative. + + + Built-in embedding providers, which need no registration. + + diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx new file mode 100644 index 000000000..af1697bdc --- /dev/null +++ b/docs/web/feature-engineering/index.mdx @@ -0,0 +1,71 @@ +--- +title: "Feature engineering" +sidebarTitle: "Overview" +description: "Derive columns and tables from data already in LanceDB, without moving it somewhere else to compute." +icon: layer-group +keywords: ["feature engineering", "computed columns", "materialized views", "functions", "udf"] +--- + +Feature engineering in LanceDB happens where the data already is. Rather than +exporting a table, computing somewhere else and writing the result back, you +declare what a column or a table should contain and let LanceDB fill it. + +Three mechanisms cover different shapes of that problem. They compose — a +materialized view can select computed columns, and a computed column can be +filled by a registered function — but each is worth understanding on its own. + +## Which one to reach for {#which-one} + +| | What it produces | Filled by | Available | +| --- | --- | --- | --- | +| [Computed columns](/feature-engineering/computed-columns) | a new column on an existing table | `refresh_column()` | everywhere | +| [Materialized views](/feature-engineering/materialized-views) | a new table derived by a query | `refresh()` | local databases | +| [Functions](/feature-engineering/functions) | a computed column filled by Python | a registered function | Cloud and Enterprise | + +**Computed columns** are the common case. A column is declared with a SQL +expression and starts empty; a refresh gives values to the rows that lack them. +Declaring one costs the same on a billion-row table as on an empty one, because +nothing is evaluated until you ask. + +**Materialized views** derive a whole table — a filtered, projected, or +aggregated view of a source — and keep it up to date incrementally as the source +changes. The result is an ordinary table: queryable, indexable, searchable. + +**Functions** run Python where the data is, for work SQL cannot express: +embedding a column, calling a model, decoding an image. The function is +registered once, versioned, and then applied as a column. + +## The shape they share {#the-shape} + +All three separate **declaring** from **filling**, and that separation is the +point: + +- Declaring is cheap and immediate. It records intent in the schema and commits + a new table version, with no data written. +- Filling is a separate call you control, can retry, and — on Cloud and + Enterprise — can hand to the server as a job rather than waiting on. + +Filling never revisits work it has already done. A refresh gives values to rows +that lack them and leaves the rest alone, so calling it twice is not twice the +work, and appending rows means the next refresh handles only the new ones. + +The corollary is worth stating plainly: **a filled value is not recomputed when +its inputs change.** If a column derives from `price` and a row's `price` is +later updated, the derived value stays as it was computed. Recomputing means +dropping the column and declaring it again. That is a deliberate trade — it is +what makes refreshes idempotent and cheap — but it will surprise you if you +expect a spreadsheet. + +## Where to go next {#next} + + + + Declare a column from a SQL expression and fill it. + + + Derive a table from a query and refresh it incrementally. + + + Run Python over your data to produce a column. + + diff --git a/docs/web/feature-engineering/materialized-views.mdx b/docs/web/feature-engineering/materialized-views.mdx new file mode 100644 index 000000000..b6d267ae4 --- /dev/null +++ b/docs/web/feature-engineering/materialized-views.mdx @@ -0,0 +1,107 @@ +--- +title: "Materialized views" +sidebarTitle: "Materialized views" +description: "Derive a table from a query over another table, and keep it current incrementally." +icon: table +keywords: ["materialized view", "create_materialized_view", "refresh", "incremental"] +--- + +A materialized view is a table whose contents are defined by a query over +another table. The query is recorded in the view's schema; a refresh computes +the rows. + +The result is an ordinary table. It can be queried, indexed, and searched, and +it appears in `table_names()` — nothing about reading it is special. + +## Create and refresh {#create-and-refresh} + +```python +import lancedb + +db = lancedb.connect( + "./.lancedb", + storage_options={"new_table_enable_stable_row_ids": "true"}, +) +table = db.create_table("people", [{"name": "ada", "age": 36}, + {"name": "kid", "age": 7}]) + +view = db.create_materialized_view( + "adults", + "people", + select=["name", ("shout", "upper(name)")], + where="age >= 18", +) + +result = view.refresh() +result.rows_written +# 1 +``` + +`select` takes column names, `(alias, expression)` pairs, or a dict of the same. +Omitting it selects every source column, expanded against the schema as it was +at creation time. `where` filters source rows; `limit` caps the view. + +## The source needs stable row ids {#stable-row-ids} + + +The source table must have been created with stable row ids: + +```python +db = lancedb.connect(uri, storage_options={"new_table_enable_stable_row_ids": "true"}) +``` + +They are what keeps the view's provenance valid when the source is compacted, +and **they cannot be enabled after a table exists**. A table created without +them cannot later become the source of a view — it has to be recreated. + + +This is the one decision here that is hard to undo, so it is worth setting the +option on any table you might later want to derive from. + +## Refreshes are incremental {#incremental} + +`refresh()` reconciles the source's changes into the view — rows added, changed, +or removed since the last refresh — rather than recomputing from scratch. It +falls back to a full rebuild when the changes cannot be reconciled. + +Two arguments override that: + +- `full=True` forces a rebuild. +- `source_version=N` refreshes against that source version instead of the + latest, which is how you make a view reproduce a known point in time. + +Concurrent refreshes of one view do not duplicate rows. If two plan the same +source rows, they conflict on commit and the loser raises rather than writing +them twice — so a retry loop around `refresh()` is safe. + +## Local databases only {#availability} + + +Materialized views are supported on **local databases**. Calling +`create_materialized_view` on a remote connection raises `NotImplementedError`. + +This is the reverse of [functions](/feature-engineering/functions), which run +only against Cloud and Enterprise. If you are deriving tables on a remote +database, a [computed column](/feature-engineering/computed-columns) filled by a +server job is usually the closer fit. + + +## Working with the view {#working-with-the-view} + +`view.table()` returns the underlying table for querying, indexing, or search. +`view.definition()` returns the recorded query — the select list, predicate, and +limit — which is how you check what a view actually contains without inferring +it from the rows. + +An existing view is reopened with `db.open_materialized_view(name)`. + +## What to read next {#next} + + + + Derive a column instead of a whole table. + + + Every refresh commits a table version. + + diff --git a/docs/web/training/object-detection.mdx b/docs/web/training/object-detection.mdx index 2b516d64b..136a6e7a9 100644 --- a/docs/web/training/object-detection.mdx +++ b/docs/web/training/object-detection.mdx @@ -42,7 +42,7 @@ A perception model fine-tuned on a generic dataset typically misses the long-tai | **Nighttime pedestrians** | `timeofday = 'night' AND has_person = true` | | **Distant pedestrians** | `has_person = true AND person_bbox_area_pct < 30.0` | -Each curated slice becomes a [materialized view](/geneva/jobs/materialized-views) — a named, refreshable SQL filter over the source table — and the training script loads it by name. New footage flows in through `add()` → `backfill()` → `refresh()`; no manifests, no exports, no reshuffling on disk. +Each curated slice becomes a [materialized view](/feature-engineering/materialized-views) — a named, refreshable SQL filter over the source table — and the training script loads it by name. New footage flows in through `add()` → `backfill()` → `refresh()`; no manifests, no exports, no reshuffling on disk. ## 1\. Schema {#1-schema} @@ -74,7 +74,7 @@ Ingestion streams `pa.RecordBatch`es of raw frames + annotations directly into a ## 2\. Backfill curation features with Geneva {#2-backfill-curation-features-with-geneva} -Curation signals are added as columns on the same table using [Geneva UDFs](/geneva/). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows. +Curation signals are added as columns on the same table using [computed columns](/feature-engineering/computed-columns). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows. ```py Python icon=Python import pyarrow as pa @@ -117,7 +117,7 @@ with gconn.local_ray_context(): ``` -Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See the [Geneva end-to-end example](/geneva/end-to-end) for more on the backfill pattern. +Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See [computed columns](/feature-engineering/computed-columns) for how a backfill fills only the rows that lack values. ## 3\. Define training splits as materialized views {#3-define-training-splits-as-materialized-views} diff --git a/docs/web/tutorials/feature-engineering/index.mdx b/docs/web/tutorials/feature-engineering/index.mdx index ad18e99a1..2d3a4478b 100644 --- a/docs/web/tutorials/feature-engineering/index.mdx +++ b/docs/web/tutorials/feature-engineering/index.mdx @@ -17,6 +17,6 @@ The relevant section of the documentation are listed below. | Feature | Description | |:--------|:------------| -| [Feature Engineering](/geneva/) | Learn the fundamentals of feature engineering. | -| [Materialized Views](/geneva/jobs/materialized-views) | Learn more about creating materialized views. | -| [Contexts](/geneva/jobs/contexts) | Learn how to run your job on a Ray cluster for production use. | +| [Feature engineering](/feature-engineering) | The three ways to derive columns and tables. | +| [Materialized views](/feature-engineering/materialized-views) | Derive a table from a query and refresh it incrementally. | +| [Computed columns](/feature-engineering/computed-columns) | Declare a column from an expression and fill it. |