rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-12 23:02:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: split feature engineering, move functions to Management

Browse Source 
Feature engineering is now the overview plus one page per operation: add a
computed column, refresh a column, refresh a materialized view. Computed
columns covered declaring and filling on one page, which are separate decisions
with separate failure modes.

Functions move to Management, where the other catalog objects are, and absorb
the environment-mismatch guidance from the Enterprise page -- a symptom table
worth having beside the parameters that cause it.

The Enterprise feature-engineering page is merged into the overview. Two of its
claims were stale: create_materialized_view is implemented for db:// connections
in remote/db.py, so views are not local-only, and the tier is Enterprise rather
than 'Cloud and Enterprise'.

The REST reference leaves the sidebar. Mintlify only generates per-endpoint
pages from a navigation group -- hidden, dropdowns, global anchors and a
top-level openapi key were all tried -- so removing the group removes those
pages. The overview survives as an unlisted page, linked from SDKs & SQL, and
the spec still ships for anything that wants to read it directly.
This commit is contained in:
Jack Ye
2026-09-09 23:19:06 -07:00
parent c3f428acae
commit 8f37234a86
16 changed files with 372 additions and 199 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/web/basics/apis.mdx
+1 -1
View File
@@ -60,7 +60,7 @@ its result. Full-text search *is* available in SQL, as the
## TypeScript has no SQL method {#ts-no-sql}
The TypeScript SDK does not expose SQL, and has no methods for
[functions](/feature-engineering/functions). Reach the endpoint with a Flight
[functions](/management/function). Reach the endpoint with a Flight
SQL client instead -- see [connecting a client](/search/sql). This is a gap in
the SDK rather than in the endpoint: the same statements work.
docs/web/basics/catalog.mdx
+2 -2
View File
@@ -23,7 +23,7 @@ feature store. The catalog part is what this page is about.
| [Table](/tables) | Rows and columns, versioned. What most operations act on. |
| [Materialized view](/feature-engineering/materialized-views) | A table whose contents come from a query, computed and stored. |
| [View](/management/view) | A saved query. Computed when read, not stored. |
| [Function](/feature-engineering/functions) | Registered code, applied to produce column values. |
| [Function](/management/function) | Registered code, applied to produce column values. |
```
database
@@ -47,7 +47,7 @@ these docs is about one of them:
| | |
| --- | --- |
| **Columns** | including vectors, blobs, and columns [computed](/feature-engineering/computed-columns) from other columns |
| **Columns** | including vectors, blobs, and columns [computed](/feature-engineering/add-computed-column) from other columns |
| **Versions** | every write commits one, so reads can [travel back](/tables/versioning) |
| **Indexes** | vector, full-text, and scalar, built over the columns |
| **Branches** | isolated lines of change you can [merge or discard](/tables/branching) |
docs/web/basics/lance.mdx
+1 -1
View File
@@ -103,7 +103,7 @@ than erased, so the data is recoverable for as long as the version is.
Adding a derived column writes only the new column. Existing data -- the
expensive part, the images and videos -- is untouched. That is what makes
[computed columns](/feature-engineering/computed-columns) a reasonable thing to
[computed columns](/feature-engineering/add-computed-column) a reasonable thing to
do repeatedly on a large table rather than a full rewrite each time.
<Card
docs/web/docs.json
+13 -11
View File
@@ -62,6 +62,14 @@
{
"source": "/api-reference",
"destination": "/basics/apis"
},
{
"source": "/feature-engineering/computed-columns",
"destination": "/feature-engineering/add-computed-column"
},
{
"source": "/feature-engineering/functions",
"destination": "/management/function"
}
],
"navigation": {
@@ -83,13 +91,6 @@
"basics/catalog",
"basics/jobs",
"basics/apis",
{
"group": "REST API",
"expanded": true,
"pages": [
"api-reference/rest/index"
]
},
"basics/agentic-skills"
]
},
@@ -113,7 +114,8 @@
"tables/schema",
"tables/multimodal"
]
}
},
"management/function"
]
},
{
@@ -167,9 +169,9 @@
"group": "Feature engineering",
"pages": [
"feature-engineering/index",
"feature-engineering/computed-columns",
"feature-engineering/materialized-views",
"feature-engineering/functions"
"feature-engineering/add-computed-column",
"feature-engineering/refresh-column",
"feature-engineering/materialized-views"
]
},
{
docs/web/feature-engineering/add-computed-column.mdx
+89
View File
@@ -0,0 +1,89 @@
---
title: "Add a computed column"
sidebarTitle: "Add Computed Column"
description: "Declare a column from a SQL expression. Nothing is evaluated until you refresh it."
icon: table-columns
keywords: ["computed columns", "add_columns", "transforms", "schema"]
---
A computed column is declared with an expression and starts with no values.
Declaring it is cheap on any table size, because nothing is evaluated until
[a refresh](/feature-engineering/refresh-column) asks for it.
## Declaring one {#declare}
```python
import lancedb
db = lancedb.connect("./.lancedb")
table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}])
table.add_columns(computed={"doubled": "x * 2"})
# AddColumnsResult(version=2)
```
That commits a new table version with the column present and empty. It is an
ordinary table version, so the usual [versioning](/tables/versioning) applies --
including that you can travel back to before the column existed.
## `computed` and `transforms` are different {#computed-vs-transforms}
`add_columns` takes either, and the difference matters:
- **`transforms`** evaluates the expression *now*, for every row, as part of the
call. Good for a small table, or a column you want populated immediately.
- **`computed`** stores the expression and evaluates nothing. The column is
committed empty and filled later by
[`refresh_column`](/feature-engineering/refresh-column).
They cannot be combined in one call. Reach for `computed` when the table is
large enough that you would rather control when the work happens.
## The expression {#expressions}
The expression is SQL, evaluated over the row. It can reference any column in
the table, including another computed column:
```python
table.add_columns(computed={
"power_score": "(stats.magic + stats.strength) / 2.0",
"is_strong": "power_score > 3.0",
})
```
Declaration order is dependency order -- a computed column may reference one
declared earlier in the same mapping, but not one declared after it.
The functions available are the ones the
[SQL dialect](/sql/functions) documents.
## Its inputs become load-bearing {#input-constraints}
<Warning>
While a computed column's declaration references another column, that column
cannot be renamed, retyped, or dropped. The declaration is part of the schema,
and LanceDB will not let you invalidate it out from under itself.
</Warning>
This is worth knowing before deriving a column from something you expect to
reshape later. Dropping the computed column releases the constraint.
## Filling with Python instead of SQL {#functions}
<Badge color="red">Enterprise</Badge>
When the value cannot be expressed in SQL -- an embedding, a model call, an
image decode -- the column is filled by a registered
[function](/management/function) instead of an expression. Declaring it looks
the same; what differs is what fills it.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
Fill the rows that lack values.
</Card>
<Card title="Functions" icon="code" href="/management/function">
Fill a column with Python.
</Card>
</CardGroup>
docs/web/feature-engineering/computed-columns.mdx
-113
View File
@@ -1,113 +0,0 @@
---
title: "Computed columns"
sidebarTitle: "Computed columns"
description: "Declare a column from a SQL expression and fill it when you choose."
icon: table-columns
keywords: ["computed columns", "add_columns", "refresh_column", "backfill"]
---
A computed column is declared with an expression and starts with no values. A
separate refresh fills the rows that lack them. Splitting the two means
declaring a column on a billion-row table costs the same as declaring one on an
empty table: nothing is evaluated until you ask for it.
## Declare and fill {#declare-and-fill}
```python
import lancedb
db = lancedb.connect("./.lancedb")
table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}])
table.add_columns(computed={"doubled": "x * 2"})
# AddColumnsResult(version=2)
table.refresh_column("doubled")
# RefreshColumnResult(rows_filled=2, version=3)
table.to_arrow().sort_by("x").to_pandas()
# x doubled
# 0 1 2
# 1 2 4
```
`add_columns(computed=...)` commits a new table version with the column present
and empty. `refresh_column` fills it and commits another. Both are ordinary
table versions, so the usual [versioning](/tables/versioning) applies.
## `computed` and `transforms` are different {#computed-vs-transforms}
`add_columns` takes either, and the difference matters:
- **`transforms`** evaluates the expression *now*, for every row, as part of the
call. Good for a small table or a column you want populated immediately.
- **`computed`** stores the expression and evaluates nothing. The column is
committed empty and filled later by `refresh_column`.
They cannot be combined in one call. Reach for `computed` when the table is
large enough that you would rather control when the work happens.
## Refreshes do not revisit filled rows {#idempotent}
A refresh gives values to rows that have none. Rows already filled are left
alone. Two consequences follow:
**Appending is cheap.** Rows added since the last refresh are filled by the
next one; everything before it is skipped. Backfilling a column and then
ingesting more data does not mean recomputing the backfill.
**A filled value does not track its input.** If `doubled` derives from `x` and
a row's `x` is later updated, `doubled` keeps the value computed when it was
filled. Recomputing means dropping the column and declaring it again.
That is what makes the call idempotent and safe to retry, but it is the opposite
of a spreadsheet formula, so it is worth being deliberate about which columns
you derive this way.
## Constraints on the inputs {#input-constraints}
While a computed column's declaration references another column, that column
cannot be renamed, retyped, or dropped. The declaration is part of the schema,
and LanceDB will not let you invalidate it out from under itself.
Declaration order is also dependency order: a computed column may reference one
declared earlier in the same mapping.
## Filling as a job {#async}
<Badge color="red">Enterprise</Badge>
Against a remote database the refresh runs on the server. `refresh_column`
raises there; use `refresh_column_async`, which returns a job you can wait on or
poll instead of holding a connection open:
```python
job = table.refresh_column_async("doubled")
result = job.wait()
result.rows_assigned
# 2
job.status()
# 'finished'
```
The job handle carries `id`, `status()`, `wait(timeout=...)` and `cancel()`, so
a long backfill can be started, left, and checked on later.
## Filling with Python instead of SQL {#functions}
<Badge color="red">Enterprise</Badge>
When the value cannot be expressed in SQL — an embedding, a model call, an image
decode — the column is filled by a registered function instead of an expression.
See [Functions](/feature-engineering/functions).
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Materialized views" icon="table" href="/feature-engineering/materialized-views">
Derive a whole table rather than a column.
</Card>
<Card title="Functions" icon="code" href="/feature-engineering/functions">
Fill a column with Python.
</Card>
</CardGroup>
docs/web/feature-engineering/index.mdx
+78 -44
View File
@@ -1,76 +1,110 @@
---
title: "LanceDB Feature Engineering"
title: "Feature engineering"
sidebarTitle: "Overview"
description: "Derive columns and tables from data already in the lakehouse, without moving it somewhere else to compute."
icon: layer-group
keywords: ["feature engineering", "computed columns", "materialized views", "functions", "lakehouse"]
keywords: ["feature engineering", "computed columns", "materialized views", "functions", "lakehouse", "jobs"]
---
Feature engineering happens where the data already is. Rather than exporting a
table, computing somewhere else and writing the result back, you declare what a
column or a table should contain and let LanceDB fill it.
**Functions and materialized views are objects in the Lance Multimodal
Lakehouse**, alongside tables and indexes. They are created, versioned, and
refreshed like anything else in it — not a separate system bolted alongside, and
not a pipeline you assemble out of external orchestration.
[Functions](/management/function) and
[materialized views](/feature-engineering/materialized-views) are ordinary
[catalog objects](/basics/catalog), alongside tables and indexes -- created,
versioned, and refreshed like anything else, not a separate system bolted
alongside.
Three mechanisms cover different shapes of the problem. They compose — a
materialized view can select computed columns, and a computed column can be
filled by a registered function — but each is worth understanding on its own.
## The shape everything shares {#the-shape}
Every mechanism here separates **declaring** from **filling**, and that
separation is the point:
- Declaring is cheap and immediate. It records intent in the schema and commits
a new table version, with no data written. Declaring a column on a
billion-row table costs what declaring one on an empty table costs.
- Filling is a separate call you control, can retry, and -- against a deployment
-- can hand to the server as a [job](/basics/jobs) rather than waiting on.
**Filling never revisits work it has already done.** A refresh gives values to
rows that lack them and leaves the rest alone, so calling it twice is not twice
the work, and appending rows means the next refresh handles only the new ones.
The corollary is worth stating plainly: **a filled value is not recomputed when
its inputs change.** If a column derives from `price` and a row's `price` is
later updated, the derived value stays as it was computed. Recomputing means
dropping the column and declaring it again. That is a deliberate trade -- it is
what makes refreshes idempotent and cheap -- but it will surprise you if you
expect a spreadsheet.
## Which one to reach for {#which-one}
| | What it produces | Filled by | Available |
| --- | --- | --- | --- |
| [Computed columns](/feature-engineering/computed-columns) | a new column on an existing table | `refresh_column()` | everywhere |
| [Materialized views](/feature-engineering/materialized-views) | a new table derived by a query | `refresh()` | local databases |
| [Functions](/feature-engineering/functions) | a computed column filled by Python | a registered function | Enterprise |
| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | everywhere |
| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | everywhere |
| [Function](/management/function) | a computed column filled by Python | registered code | <Badge color="red">Enterprise</Badge> |
**Computed columns** are the common case. A column is declared with a SQL
expression and starts empty; a refresh gives values to the rows that lack them.
Declaring one costs the same on a billion-row table as on an empty one, because
nothing is evaluated until you ask.
**Computed columns** are the common case: a column declared with a SQL
expression, filled by [a refresh](/feature-engineering/refresh-column).
**Materialized views** derive a whole table — a filtered, projected, or
aggregated view of a source — and keep it up to date incrementally as the source
changes. The result is an ordinary table: queryable, indexable, searchable.
**Materialized views** derive a whole table -- a filtered, projected, or
aggregated view of a source -- and keep it current incrementally. The result is
an ordinary table: queryable, indexable, searchable.
**Functions** run Python where the data is, for work SQL cannot express:
embedding a column, calling a model, decoding an image. The function is
registered once, versioned, and then applied as a column.
embedding a column, calling a model, decoding an image. Registered once,
versioned, then applied as a column.
## The shape they share {#the-shape}
They compose. A materialized view can select computed columns, and a computed
column can be filled by a registered function.
All three separate **declaring** from **filling**, and that separation is the
point:
## Where the work runs {#where}
- Declaring is cheap and immediate. It records intent in the schema and commits
a new table version, with no data written.
- Filling is a separate call you control, can retry, and — on Cloud and
Enterprise — can hand to the server as a job rather than waiting on.
<Badge color="red">Enterprise</Badge>
Filling never revisits work it has already done. A refresh gives values to rows
that lack them and leaves the rest alone, so calling it twice is not twice the
work, and appending rows means the next refresh handles only the new ones.
Locally, a refresh runs in your process. Against a deployment it runs across the
cluster, and the API reflects that: `refresh_column` raises, and
`refresh_column_async` returns a [job](/basics/jobs) handle instead.
The corollary is worth stating plainly: **a filled value is not recomputed when
its inputs change.** If a column derives from `price` and a row's `price` is
later updated, the derived value stays as it was computed. Recomputing means
dropping the column and declaring it again. That is a deliberate trade — it is
what makes refreshes idempotent and cheap — but it will surprise you if you
expect a spreadsheet.
```python
job = table.refresh_column_async("embedding")
result = job.wait()
result.rows_assigned
```
A backfill over a large table can therefore be started, left, and checked on
from a different process -- the connection does not have to stay open.
That relocation is the main thing to understand about Enterprise feature
engineering. The API is the same; what changes is that the code runs somewhere
else, which is also why a function must declare the environment it needs.
## Failure is resumable {#failures}
A function that raises fails the job. There is no per-row retry or skip policy
in the function definition, so a function calling a flaky external service
should handle that itself, inside the function body.
The alternative is to let the job fail and run the refresh again. Since a
refresh never revisits a filled row, re-running after a partial failure costs
only the rows that did not complete -- **a failed backfill is resumed, not
restarted.**
## Where to go next {#next}
<CardGroup cols={3}>
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/computed-columns">
Declare a column from a SQL expression and fill it.
<CardGroup cols={2}>
<Card title="Add a computed column" icon="table-columns" href="/feature-engineering/add-computed-column">
Declare a column from a SQL expression.
</Card>
<Card title="Materialized views" icon="table" href="/feature-engineering/materialized-views">
Derive a table from a query and refresh it incrementally.
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
Fill the rows that lack values, locally or as a job.
</Card>
<Card title="Functions" icon="code" href="/feature-engineering/functions">
Run Python over your data to produce a column.
<Card title="Refresh a materialized view" icon="table" href="/feature-engineering/materialized-views">
Bring a derived table up to date with its source.
</Card>
<Card title="Functions" icon="code" href="/management/function">
Register Python to fill a column with.
</Card>
</CardGroup>
docs/web/feature-engineering/materialized-views.mdx
+21 -16
View File
@@ -1,6 +1,6 @@
---
title: "Materialized views"
sidebarTitle: "Materialized views"
title: "Refresh a materialized view"
sidebarTitle: "Refresh Materialized View"
description: "A lakehouse object: a table defined by a query over another table, kept current incrementally."
icon: table
keywords: ["materialized view", "create_materialized_view", "refresh", "incremental"]
@@ -74,22 +74,27 @@ Concurrent refreshes of one view do not duplicate rows. If two plan the same
source rows, they conflict on commit and the loser raises rather than writing
them twice — so a retry loop around `refresh()` is safe.
## Creating one against a deployment {#availability}
## Against a deployment {#availability}
<Warning>
`create_materialized_view` on this client builds a view over a **local**
database. Called on a remote `db://` connection it raises
`NotImplementedError`.
<Badge color="red">Enterprise</Badge>
Views exist on Enterprise deployments too, where a refresh runs across the
cluster rather than in your process — but they are created through the
Enterprise feature engineering interface rather than this method. See the
Enterprise documentation if that is where your data lives.
</Warning>
`create_materialized_view` works on a `db://` connection as well as a local one.
What changes is where the refresh runs -- across the cluster rather than in your
process -- and that SQL can drive it:
The direction is the reverse of [functions](/feature-engineering/functions),
which register only against Enterprise and raise locally. Worth
knowing before concluding either is unavailable.
```sql
REFRESH MATERIALIZED VIEW chunks
```
`WITH (full = true)` recomputes every row instead of only the rows the source
has added since the last refresh. `SHOW MATERIALIZED VIEWS` lists them, which is
cheaper than `list_materialized_views` -- that method finds views by reading
every table's schema, so it costs one open per table.
<Note>
`CASCADE` is accepted by the parser but not yet implemented. A view built on
another view is refreshed by refreshing each in dependency order.
</Note>
## Working with the view {#working-with-the-view}
@@ -103,7 +108,7 @@ An existing view is reopened with `db.open_materialized_view(name)`.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/computed-columns">
<Card title="Refresh a column" icon="arrows-rotate" href="/feature-engineering/refresh-column">
Derive a column instead of a whole table.
</Card>
<Card title="Versioning" icon="clock-rotate-left" href="/tables/versioning">
docs/web/feature-engineering/refresh-column.mdx
+120
View File
@@ -0,0 +1,120 @@
---
title: "Refresh a column"
sidebarTitle: "Refresh Column"
description: "Fill the rows of a computed column that lack values — in process, or as a server-side job."
icon: arrows-rotate
keywords: ["refresh_column", "backfill", "job", "idempotent", "REFRESH COLUMN"]
---
A [declared column](/feature-engineering/add-computed-column) starts empty. A
refresh gives values to the rows that lack them.
<CodeGroup>
```python Python icon="python"
table.refresh_column("doubled")
# RefreshColumnResult(rows_filled=2, version=3)
table.to_arrow().sort_by("x").to_pandas()
# x doubled
# 0 1 2
# 1 2 4
```
```sql SQL
REFRESH COLUMN doubled ON my_table
```
</CodeGroup>
The refresh commits an ordinary table version, so
[versioning](/tables/versioning) applies to it like any other write.
## It never revisits filled rows {#idempotent}
A refresh fills rows that have no value and leaves the rest alone. Two
consequences follow, and both are load-bearing.
**Appending is cheap.** Rows added since the last refresh are filled by the
next one; everything before it is skipped. Backfilling a column and then
ingesting more data does not mean recomputing the backfill.
**A filled value does not track its input.** If `doubled` derives from `x` and a
row's `x` is later updated, `doubled` keeps the value computed when it was
filled. Recomputing means dropping the column and declaring it again.
That is what makes the call idempotent and safe to retry -- but it is the
opposite of a spreadsheet formula, so be deliberate about which columns you
derive this way.
## Refreshing part of a table {#partial}
<Badge color="red">Enterprise</Badge> In SQL a refresh can be scoped to a
predicate, which is useful for filling a slice first and checking it before
committing to the whole table:
```sql
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
```
## As a job {#async}
<Badge color="red">Enterprise</Badge>
Against a deployment the refresh runs on the cluster. `refresh_column` raises
there; use `refresh_column_async`, which returns a [job](/basics/jobs) you can
wait on or poll instead of holding a connection open:
```python
job = table.refresh_column_async("doubled")
result = job.wait()
result.rows_assigned
# 2
job.status()
# 'finished'
```
Poll with `status()`, resolve with `wait()`. `status()` is a point snapshot and
does not raise, which is what makes it safe in a loop; `wait()` raises
`JobFailedError`, `JobCancelledError` or `TimeoutError`.
A long backfill can therefore be started, left, and checked on from a different
process.
### Resourcing the job {#resources}
SQL exposes the worker budget directly, which is the lever when a backfill is
too slow or too expensive:
```sql
REFRESH COLUMN embedding ON documents
WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2)
```
`num_workers` sets the count instead of a per-worker budget; the two are
alternatives rather than a pair. `batch_size` controls how many rows a worker
takes at a time, and `error_policy = 'skip'` continues past rows that fail
rather than failing the job.
## When a refresh fails {#failures}
A failure is not a rollback. Rows filled before the failure keep their values,
because each is committed as it completes.
That makes recovery simple: run the refresh again. It skips everything already
filled and costs only the rows that did not complete -- **a failed backfill is
resumed, not restarted.**
For a column filled by a [function](/management/function), the usual cause is
the worker environment rather than the data. See
[environment mismatches](/management/function#mismatch-symptoms) for the
symptoms, which rarely name their cause.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Jobs" icon="hourglass-half" href="/basics/jobs">
Waiting on, inspecting, and cancelling server-side work.
</Card>
<Card title="Refresh a materialized view" icon="table" href="/feature-engineering/materialized-views">
The same idea, for a whole derived table.
</Card>
</CardGroup>
docs/web/maintenance/compaction.mdx
+1 -1
View File
@@ -95,7 +95,7 @@ continuous small writes needs it on a schedule.
<Note>
Compaction rewrites files, which conflicts with a concurrent
[computed-column refresh](/feature-engineering/computed-columns) or
[computed-column refresh](/feature-engineering/add-computed-column) or
[view refresh](/feature-engineering/materialized-views) planned against the
version it replaces. Sequence them rather than running both at once.
</Note>
docs/web/management/database.mdx
+1 -1
View File
@@ -42,7 +42,7 @@ storage and does the work itself. The third talks to a **deployment**.
| Table handle | `LanceTable` | `RemoteTable` |
| [SQL](/basics/apis) | -- | yes |
| Server-side [jobs](/basics/jobs) | -- | yes |
| [Functions](/feature-engineering/functions) | -- | yes |
| [Functions](/management/function) | -- | yes |
| [Authorization](/governance/authorization) | storage permissions | granted per object |
![](/static/assets/images/overview/understanding-tables.png)
docs/web/feature-engineering/functions.mdx → docs/web/management/function.mdx
+39 -3
View File
@@ -15,7 +15,7 @@ rather than pulling rows to your process.
Registration is a remote operation. On a local connection `create_function`
raises `NotImplementedError`; use a
[computed column](/feature-engineering/computed-columns) instead.
[computed column](/feature-engineering/refresh-column) instead.
## Define a function {#define}
@@ -54,6 +54,42 @@ These are part of the **immutable function version**. Changing a dependency
produces a new version rather than altering the one already registered, so a
column filled by version 1 is unaffected by version 2.
Declaring them is not optional bookkeeping. If the worker environment differs
from the one the function was written against, the failures are among the
hardest to diagnose in the system, because the error rarely names the cause.
### Symptoms of an environment mismatch {#mismatch-symptoms}
| Symptom | Likely cause |
| --- | --- |
| `TypeError: Enum.__new__() missing 1 required positional argument` | `attrs` version mismatch |
| `TypeError: Can't instantiate abstract class` | package structure differs between versions |
| `ArrowInvalid: cannot cast`, or serialization errors | NumPy 1.x against 2.x |
| `ModuleNotFoundError` on the worker | package installed locally but not declared |
| Model loading failures | PyTorch version mismatch |
| Permission denied | an API key present locally but absent from `env` |
The packages worth pinning explicitly, because a mismatch in them produces the
errors above rather than a clear one: **numpy, torch, pyarrow, attrs,
pydantic**.
<Note>
The client version defaults `python_version`, so a function registered from a
3.11 environment runs on 3.11 unless you say otherwise. That default is usually
right and occasionally surprising -- pin it when the function is sensitive to it.
</Note>
## Failure handling {#failures}
A function that raises fails the job. There is no per-row retry or skip policy
in the function definition: `@udf` takes no error-handling parameter, so a
function that calls a flaky external service should handle that itself, inside
the function body.
Letting the job fail is the other option. Since
[a refresh never revisits a filled row](/feature-engineering/refresh-column#idempotent),
re-running after a partial failure costs only the rows that did not complete.
## What gets packaged {#packaging}
<Warning>
@@ -96,13 +132,13 @@ in the named column. A bare named-struct application expands its result fields
as one atomic binding, with aliases from `rename(columns=...)`.
Filling then works exactly as it does for any computed column — see
[Computed columns](/feature-engineering/computed-columns) for refresh
[Computed columns](/feature-engineering/refresh-column) for refresh
semantics, including the rule that filled rows are never revisited.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/computed-columns">
<Card title="Computed columns" icon="table-columns" href="/feature-engineering/refresh-column">
Refresh semantics, and the SQL alternative.
</Card>
<Card title="Embedding functions" icon="vector-square" href="/embedding">
docs/web/management/index.mdx
+1 -1
View File
@@ -87,7 +87,7 @@ method for them, because the thing a view stores *is* a SQL statement. See
<Badge color="red">Enterprise</Badge> A function runs on the server, so there is
no local form. The TypeScript SDK has no function methods; use Python or SQL.
See [Functions](/feature-engineering/functions).
See [Functions](/management/function).
### Jobs {#jobs}
docs/web/quickstart.mdx
+2 -2
View File
@@ -410,8 +410,8 @@ job = characters.refresh_column_async("power_score")
job.wait()
```
See [Computed columns](/feature-engineering/computed-columns) for expressions,
and [Functions](/feature-engineering/functions) for filling a column with Python
See [Computed columns](/feature-engineering/add-computed-column) for expressions,
and [Functions](/management/function) for filling a column with Python
-- an embedding model, a captioner, anything you can register.
## 7\. Store multimodal data {#7-store-multimodal-data}
docs/web/training/object-detection.mdx
+2 -2
View File
@@ -74,7 +74,7 @@ Ingestion streams `pa.RecordBatch`es of raw frames + annotations directly into a
## 2\. Backfill curation features with Geneva {#2-backfill-curation-features-with-geneva}
Curation signals are added as columns on the same table using [computed columns](/feature-engineering/computed-columns). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows.
Curation signals are added as columns on the same table using [computed columns](/feature-engineering/refresh-column). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows.
```py Python icon=Python
import pyarrow as pa
@@ -117,7 +117,7 @@ with gconn.local_ray_context():
```
<Note>
Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See [computed columns](/feature-engineering/computed-columns) for how a backfill fills only the rows that lack values.
Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See [computed columns](/feature-engineering/refresh-column) for how a backfill fills only the rows that lack values.
</Note>
## 3\. Define training splits as materialized views {#3-define-training-splits-as-materialized-views}
docs/web/tutorials/feature-engineering/index.mdx
+1 -1
View File
@@ -19,4 +19,4 @@ The relevant section of the documentation are listed below.
|:--------|:------------|
| [Feature engineering](/feature-engineering) | The three ways to derive columns and tables. |
| [Materialized views](/feature-engineering/materialized-views) | Derive a table from a query and refresh it incrementally. |
| [Computed columns](/feature-engineering/computed-columns) | Declare a column from an expression and fill it. |
| [Computed columns](/feature-engineering/add-computed-column) | Declare a column from an expression and fill it. |