mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-26 13:36:13 +00:00
docs: publish the Function pages only once Functions launch
The two Function pages declare `launch: functions`, and every other page marks what it says about Functions, so the assembled site leaves them out until the launch. Before it, the pages that offered Functions for Python work point Enterprise readers at Geneva instead. The SQL Function example now uses Arrow type names, which the server requires, and states the 3.12 default for `python_version`.
This commit is contained in:
@@ -38,9 +38,12 @@ including that you can travel back to before the column existed.
|
||||
|
||||
- **SQL expressions in `transforms`** are evaluated *now*, for every row, as
|
||||
part of the call. Good for a small table, or a column you want populated
|
||||
immediately. A `FunctionApplication` passed through `transforms` instead
|
||||
immediately.
|
||||
{/* if functions launched */}
|
||||
A `FunctionApplication` passed through `transforms` instead
|
||||
declares empty Function-bound columns, which a refresh fills; that needs a
|
||||
deployment with Functions enabled, and the embedded library refuses it.
|
||||
{/* end if */}
|
||||
- **`computed`** stores the expression and evaluates nothing. The column is
|
||||
committed empty and filled later by
|
||||
[`refresh_column`](/feature-engineering/refresh-column).
|
||||
@@ -81,11 +84,18 @@ reshape later. Dropping the computed column releases the constraint.
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
{/* if functions launched */}
|
||||
When the value cannot be expressed in SQL -- an embedding, a model call, an
|
||||
image decode -- the column is filled by a registered
|
||||
[function](/management/function) instead of an expression. Declaring it looks
|
||||
the same; what differs is what fills it. It needs a deployment with Functions
|
||||
enabled.
|
||||
{/* end if */}
|
||||
{/* if functions not launched */}
|
||||
When the value cannot be expressed in SQL -- an embedding, a model call, an
|
||||
image decode -- [Geneva](/geneva), the feature engineering package for LanceDB
|
||||
Enterprise, fills the column with a Python function instead.
|
||||
{/* end if */}
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
@@ -93,7 +103,14 @@ enabled.
|
||||
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
|
||||
Fill the rows that lack values.
|
||||
</Card>
|
||||
{/* if functions launched */}
|
||||
<Card title="Functions" icon="code" href="/management/function">
|
||||
Fill a column with Python.
|
||||
</Card>
|
||||
{/* end if */}
|
||||
{/* if functions not launched */}
|
||||
<Card title="Geneva" icon="layer-group" href="/geneva">
|
||||
Fill a column with Python on LanceDB Enterprise.
|
||||
</Card>
|
||||
{/* end if */}
|
||||
</CardGroup>
|
||||
|
||||
@@ -108,10 +108,17 @@ exist when that does not fit:
|
||||
- A [computed column](/feature-engineering/add-computed-column) declares the
|
||||
expression and leaves the column empty, so you choose when the work happens.
|
||||
That is the right default on a large table.
|
||||
{/* if functions launched */}
|
||||
- A [function](/management/function) fills a column with Python, for values SQL
|
||||
cannot express -- an embedding, a model call, an image decode.
|
||||
{/* end if */}
|
||||
{/* if functions not launched */}
|
||||
- On LanceDB Enterprise, [Geneva](/geneva) fills a column with Python, for
|
||||
values SQL cannot express -- an embedding, a model call, an image decode.
|
||||
{/* end if */}
|
||||
|
||||
On a deployment, both have to be enabled there.
|
||||
A deployment has to enable computed columns before you use them.
|
||||
Functions need enabling there too. {/* if functions launched */}
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
|
||||
@@ -3,15 +3,20 @@ title: "Feature engineering"
|
||||
sidebarTitle: "Overview"
|
||||
description: "Derive columns and tables from data already in the lakehouse, without moving it somewhere else to compute."
|
||||
icon: layer-group
|
||||
keywords: ["feature engineering", "computed columns", "materialized views", "functions", "lakehouse", "jobs"]
|
||||
keywords: ["feature engineering", "computed columns", "materialized views", "lakehouse", "jobs"]
|
||||
---
|
||||
|
||||
Feature engineering happens where the data already is. Rather than exporting a
|
||||
table, computing somewhere else and writing the result back, you declare what a
|
||||
column or a table should contain and let LanceDB fill it.
|
||||
|
||||
{/* if functions launched */}
|
||||
[Functions](/management/function) and
|
||||
[materialized views](/feature-engineering/materialized-views) are ordinary
|
||||
{/* end if */}
|
||||
{/* if functions not launched */}
|
||||
[Materialized views](/feature-engineering/materialized-views) are ordinary
|
||||
{/* end if */}
|
||||
[catalog objects](/basics/catalog), alongside tables and indexes -- created,
|
||||
versioned, and refreshed like anything else, not a separate system bolted
|
||||
alongside.
|
||||
@@ -48,7 +53,7 @@ idempotent and cheap -- but it will surprise you if you expect a spreadsheet.
|
||||
| --- | --- | --- | --- |
|
||||
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | embedded, and deployments that enable it |
|
||||
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | embedded, and deployments that enable it |
|
||||
| [Function](/management/function) | a computed column filled by Python | registered code | <Badge color="red">Enterprise</Badge> deployments that enable it |
|
||||
| [Function](/management/function) {/* if functions launched */} | a computed column filled by Python | registered code | <Badge color="red">Enterprise</Badge> deployments that enable it |
|
||||
|
||||
**Computed columns** are the common case: a column declared with a SQL
|
||||
expression, filled by [a refresh](/feature-engineering/refresh-column).
|
||||
@@ -57,12 +62,14 @@ expression, filled by [a refresh](/feature-engineering/refresh-column).
|
||||
aggregated view of a source -- and bring it up to date on each refresh. The
|
||||
result is an ordinary table: queryable, indexable, searchable.
|
||||
|
||||
{/* if functions launched */}
|
||||
**Functions** run Python where the data is, for work SQL cannot express:
|
||||
embedding a column, calling a model, decoding an image. Registered once,
|
||||
versioned, then applied as a column.
|
||||
|
||||
They compose. A materialized view can select computed columns, and a computed
|
||||
column can be filled by a registered function.
|
||||
{/* end if */}
|
||||
They compose. A materialized view can select computed columns.
|
||||
A computed column can be filled by a registered function. {/* if functions launched */}
|
||||
|
||||
## Where the work runs {#where}
|
||||
|
||||
@@ -83,15 +90,18 @@ from a different process -- the connection does not have to stay open.
|
||||
|
||||
That relocation is the main thing to understand about Enterprise feature
|
||||
engineering. The API is the same; what changes is that the code runs somewhere
|
||||
else, which is also why a function must declare the environment it needs.
|
||||
else.
|
||||
That is also why a function must declare the environment it needs. {/* if functions launched */}
|
||||
|
||||
## Recovering from a failure {#failures}
|
||||
|
||||
{/* if functions launched */}
|
||||
A function that raises fails the job. There is no per-row retry or skip policy
|
||||
in the function definition, so a function calling a flaky external service
|
||||
should handle that itself, inside the function body.
|
||||
|
||||
The alternative is to let the job fail, fix the cause, and run the refresh
|
||||
{/* end if */}
|
||||
When a refresh fails, fix the cause and run the refresh
|
||||
again. The new refresh fills what is still empty or stale. How much of the
|
||||
failed run was kept depends on where it ran (see
|
||||
[Refresh a column](/feature-engineering/refresh-column#failures)), and a retry
|
||||
@@ -110,7 +120,9 @@ only the rows that did not complete.
|
||||
<Card title="Refresh a materialized view" icon="table" href="/feature-engineering/materialized-views">
|
||||
Bring a derived table up to date with its source.
|
||||
</Card>
|
||||
{/* if functions launched */}
|
||||
<Card title="Functions" icon="code" href="/management/function">
|
||||
Register Python to fill a column with.
|
||||
</Card>
|
||||
{/* end if */}
|
||||
</CardGroup>
|
||||
|
||||
@@ -69,9 +69,8 @@ REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
|
||||
```
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `computed_refresh` for a SQL-expression column,
|
||||
`function_refresh` for one filled by a registered
|
||||
[function](/management/function).
|
||||
**Enterprise job type:** `computed_refresh` for a SQL-expression column.
|
||||
`function_refresh` for one filled by a registered [function](/management/function). {/* if functions launched */}
|
||||
</Note>
|
||||
|
||||
## As a job {#async}
|
||||
@@ -128,10 +127,12 @@ that were already written -- for example when the record of the inputs they
|
||||
were computed from was not saved -- so do not count on a retry costing only the
|
||||
rows that did not complete.
|
||||
|
||||
{/* if functions launched */}
|
||||
For a column filled by a [function](/management/function), the usual cause is
|
||||
the worker environment rather than the data. See
|
||||
[environment mismatches](/management/function#mismatch-symptoms) for the
|
||||
symptoms, which rarely name their cause.
|
||||
{/* end if */}
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user