From 8f37234a8642ee284fffab3f76d4086853712b29 Mon Sep 17 00:00:00 2001 From: Jack Ye Date: Wed, 9 Sep 2026 23:19:06 -0700 Subject: [PATCH] docs: split feature engineering, move functions to Management Feature engineering is now the overview plus one page per operation: add a computed column, refresh a column, refresh a materialized view. Computed columns covered declaring and filling on one page, which are separate decisions with separate failure modes. Functions move to Management, where the other catalog objects are, and absorb the environment-mismatch guidance from the Enterprise page -- a symptom table worth having beside the parameters that cause it. The Enterprise feature-engineering page is merged into the overview. Two of its claims were stale: create_materialized_view is implemented for db:// connections in remote/db.py, so views are not local-only, and the tier is Enterprise rather than 'Cloud and Enterprise'. The REST reference leaves the sidebar. Mintlify only generates per-endpoint pages from a navigation group -- hidden, dropdowns, global anchors and a top-level openapi key were all tried -- so removing the group removes those pages. The overview survives as an unlisted page, linked from SDKs & SQL, and the spec still ships for anything that wants to read it directly. --- docs/web/basics/apis.mdx | 2 +- docs/web/basics/catalog.mdx | 4 +- docs/web/basics/lance.mdx | 2 +- docs/web/docs.json | 24 ++-- .../add-computed-column.mdx | 89 +++++++++++++ .../feature-engineering/computed-columns.mdx | 113 ---------------- docs/web/feature-engineering/index.mdx | 122 +++++++++++------- .../materialized-views.mdx | 37 +++--- .../feature-engineering/refresh-column.mdx | 120 +++++++++++++++++ docs/web/maintenance/compaction.mdx | 2 +- docs/web/management/database.mdx | 2 +- .../functions.mdx => management/function.mdx} | 42 +++++- docs/web/management/index.mdx | 2 +- docs/web/quickstart.mdx | 4 +- docs/web/training/object-detection.mdx | 4 +- .../tutorials/feature-engineering/index.mdx | 2 +- 16 files changed, 372 insertions(+), 199 deletions(-) create mode 100644 docs/web/feature-engineering/add-computed-column.mdx delete mode 100644 docs/web/feature-engineering/computed-columns.mdx create mode 100644 docs/web/feature-engineering/refresh-column.mdx rename docs/web/{feature-engineering/functions.mdx => management/function.mdx} (66%) diff --git a/docs/web/basics/apis.mdx b/docs/web/basics/apis.mdx index 4d5e6c6eb..b302a9a3e 100644 --- a/docs/web/basics/apis.mdx +++ b/docs/web/basics/apis.mdx @@ -60,7 +60,7 @@ its result. Full-text search *is* available in SQL, as the ## TypeScript has no SQL method {#ts-no-sql} The TypeScript SDK does not expose SQL, and has no methods for -[functions](/feature-engineering/functions). Reach the endpoint with a Flight +[functions](/management/function). Reach the endpoint with a Flight SQL client instead -- see [connecting a client](/search/sql). This is a gap in the SDK rather than in the endpoint: the same statements work. diff --git a/docs/web/basics/catalog.mdx b/docs/web/basics/catalog.mdx index f850daadf..aa7732bec 100644 --- a/docs/web/basics/catalog.mdx +++ b/docs/web/basics/catalog.mdx @@ -23,7 +23,7 @@ feature store. The catalog part is what this page is about. | [Table](/tables) | Rows and columns, versioned. What most operations act on. | | [Materialized view](/feature-engineering/materialized-views) | A table whose contents come from a query, computed and stored. | | [View](/management/view) | A saved query. Computed when read, not stored. | -| [Function](/feature-engineering/functions) | Registered code, applied to produce column values. | +| [Function](/management/function) | Registered code, applied to produce column values. | ``` database @@ -47,7 +47,7 @@ these docs is about one of them: | | | | --- | --- | -| **Columns** | including vectors, blobs, and columns [computed](/feature-engineering/computed-columns) from other columns | +| **Columns** | including vectors, blobs, and columns [computed](/feature-engineering/add-computed-column) from other columns | | **Versions** | every write commits one, so reads can [travel back](/tables/versioning) | | **Indexes** | vector, full-text, and scalar, built over the columns | | **Branches** | isolated lines of change you can [merge or discard](/tables/branching) | diff --git a/docs/web/basics/lance.mdx b/docs/web/basics/lance.mdx index 448aaf2fc..adf753beb 100644 --- a/docs/web/basics/lance.mdx +++ b/docs/web/basics/lance.mdx @@ -103,7 +103,7 @@ than erased, so the data is recoverable for as long as the version is. Adding a derived column writes only the new column. Existing data -- the expensive part, the images and videos -- is untouched. That is what makes -[computed columns](/feature-engineering/computed-columns) a reasonable thing to +[computed columns](/feature-engineering/add-computed-column) a reasonable thing to do repeatedly on a large table rather than a full rewrite each time. 3.0", +}) +``` + +Declaration order is dependency order -- a computed column may reference one +declared earlier in the same mapping, but not one declared after it. + +The functions available are the ones the +[SQL dialect](/sql/functions) documents. + +## Its inputs become load-bearing {#input-constraints} + + +While a computed column's declaration references another column, that column +cannot be renamed, retyped, or dropped. The declaration is part of the schema, +and LanceDB will not let you invalidate it out from under itself. + + +This is worth knowing before deriving a column from something you expect to +reshape later. Dropping the computed column releases the constraint. + +## Filling with Python instead of SQL {#functions} + +Enterprise + +When the value cannot be expressed in SQL -- an embedding, a model call, an +image decode -- the column is filled by a registered +[function](/management/function) instead of an expression. Declaring it looks +the same; what differs is what fills it. + +## What to read next {#next} + + + + Fill the rows that lack values. + + + Fill a column with Python. + + diff --git a/docs/web/feature-engineering/computed-columns.mdx b/docs/web/feature-engineering/computed-columns.mdx deleted file mode 100644 index b9f07c4d2..000000000 --- a/docs/web/feature-engineering/computed-columns.mdx +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: "Computed columns" -sidebarTitle: "Computed columns" -description: "Declare a column from a SQL expression and fill it when you choose." -icon: table-columns -keywords: ["computed columns", "add_columns", "refresh_column", "backfill"] ---- - -A computed column is declared with an expression and starts with no values. A -separate refresh fills the rows that lack them. Splitting the two means -declaring a column on a billion-row table costs the same as declaring one on an -empty table: nothing is evaluated until you ask for it. - -## Declare and fill {#declare-and-fill} - -```python -import lancedb - -db = lancedb.connect("./.lancedb") -table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}]) - -table.add_columns(computed={"doubled": "x * 2"}) -# AddColumnsResult(version=2) - -table.refresh_column("doubled") -# RefreshColumnResult(rows_filled=2, version=3) - -table.to_arrow().sort_by("x").to_pandas() -# x doubled -# 0 1 2 -# 1 2 4 -``` - -`add_columns(computed=...)` commits a new table version with the column present -and empty. `refresh_column` fills it and commits another. Both are ordinary -table versions, so the usual [versioning](/tables/versioning) applies. - -## `computed` and `transforms` are different {#computed-vs-transforms} - -`add_columns` takes either, and the difference matters: - -- **`transforms`** evaluates the expression *now*, for every row, as part of the - call. Good for a small table or a column you want populated immediately. -- **`computed`** stores the expression and evaluates nothing. The column is - committed empty and filled later by `refresh_column`. - -They cannot be combined in one call. Reach for `computed` when the table is -large enough that you would rather control when the work happens. - -## Refreshes do not revisit filled rows {#idempotent} - -A refresh gives values to rows that have none. Rows already filled are left -alone. Two consequences follow: - -**Appending is cheap.** Rows added since the last refresh are filled by the -next one; everything before it is skipped. Backfilling a column and then -ingesting more data does not mean recomputing the backfill. - -**A filled value does not track its input.** If `doubled` derives from `x` and -a row's `x` is later updated, `doubled` keeps the value computed when it was -filled. Recomputing means dropping the column and declaring it again. - -That is what makes the call idempotent and safe to retry, but it is the opposite -of a spreadsheet formula, so it is worth being deliberate about which columns -you derive this way. - -## Constraints on the inputs {#input-constraints} - -While a computed column's declaration references another column, that column -cannot be renamed, retyped, or dropped. The declaration is part of the schema, -and LanceDB will not let you invalidate it out from under itself. - -Declaration order is also dependency order: a computed column may reference one -declared earlier in the same mapping. - -## Filling as a job {#async} - -Enterprise - -Against a remote database the refresh runs on the server. `refresh_column` -raises there; use `refresh_column_async`, which returns a job you can wait on or -poll instead of holding a connection open: - -```python -job = table.refresh_column_async("doubled") -result = job.wait() -result.rows_assigned -# 2 -job.status() -# 'finished' -``` - -The job handle carries `id`, `status()`, `wait(timeout=...)` and `cancel()`, so -a long backfill can be started, left, and checked on later. - -## Filling with Python instead of SQL {#functions} - -Enterprise - -When the value cannot be expressed in SQL — an embedding, a model call, an image -decode — the column is filled by a registered function instead of an expression. -See [Functions](/feature-engineering/functions). - -## What to read next {#next} - - - - Derive a whole table rather than a column. - - - Fill a column with Python. - - diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx index 527a790ef..ef4a5dc3f 100644 --- a/docs/web/feature-engineering/index.mdx +++ b/docs/web/feature-engineering/index.mdx @@ -1,76 +1,110 @@ --- -title: "LanceDB Feature Engineering" +title: "Feature engineering" sidebarTitle: "Overview" description: "Derive columns and tables from data already in the lakehouse, without moving it somewhere else to compute." icon: layer-group -keywords: ["feature engineering", "computed columns", "materialized views", "functions", "lakehouse"] +keywords: ["feature engineering", "computed columns", "materialized views", "functions", "lakehouse", "jobs"] --- Feature engineering happens where the data already is. Rather than exporting a table, computing somewhere else and writing the result back, you declare what a column or a table should contain and let LanceDB fill it. -**Functions and materialized views are objects in the Lance Multimodal -Lakehouse**, alongside tables and indexes. They are created, versioned, and -refreshed like anything else in it — not a separate system bolted alongside, and -not a pipeline you assemble out of external orchestration. +[Functions](/management/function) and +[materialized views](/feature-engineering/materialized-views) are ordinary +[catalog objects](/basics/catalog), alongside tables and indexes -- created, +versioned, and refreshed like anything else, not a separate system bolted +alongside. -Three mechanisms cover different shapes of the problem. They compose — a -materialized view can select computed columns, and a computed column can be -filled by a registered function — but each is worth understanding on its own. +## The shape everything shares {#the-shape} + +Every mechanism here separates **declaring** from **filling**, and that +separation is the point: + +- Declaring is cheap and immediate. It records intent in the schema and commits + a new table version, with no data written. Declaring a column on a + billion-row table costs what declaring one on an empty table costs. +- Filling is a separate call you control, can retry, and -- against a deployment + -- can hand to the server as a [job](/basics/jobs) rather than waiting on. + +**Filling never revisits work it has already done.** A refresh gives values to +rows that lack them and leaves the rest alone, so calling it twice is not twice +the work, and appending rows means the next refresh handles only the new ones. + +The corollary is worth stating plainly: **a filled value is not recomputed when +its inputs change.** If a column derives from `price` and a row's `price` is +later updated, the derived value stays as it was computed. Recomputing means +dropping the column and declaring it again. That is a deliberate trade -- it is +what makes refreshes idempotent and cheap -- but it will surprise you if you +expect a spreadsheet. ## Which one to reach for {#which-one} | | What it produces | Filled by | Available | | --- | --- | --- | --- | -| [Computed columns](/feature-engineering/computed-columns) | a new column on an existing table | `refresh_column()` | everywhere | -| [Materialized views](/feature-engineering/materialized-views) | a new table derived by a query | `refresh()` | local databases | -| [Functions](/feature-engineering/functions) | a computed column filled by Python | a registered function | Enterprise | +| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | everywhere | +| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | everywhere | +| [Function](/management/function) | a computed column filled by Python | registered code | Enterprise | -**Computed columns** are the common case. A column is declared with a SQL -expression and starts empty; a refresh gives values to the rows that lack them. -Declaring one costs the same on a billion-row table as on an empty one, because -nothing is evaluated until you ask. +**Computed columns** are the common case: a column declared with a SQL +expression, filled by [a refresh](/feature-engineering/refresh-column). -**Materialized views** derive a whole table — a filtered, projected, or -aggregated view of a source — and keep it up to date incrementally as the source -changes. The result is an ordinary table: queryable, indexable, searchable. +**Materialized views** derive a whole table -- a filtered, projected, or +aggregated view of a source -- and keep it current incrementally. The result is +an ordinary table: queryable, indexable, searchable. **Functions** run Python where the data is, for work SQL cannot express: -embedding a column, calling a model, decoding an image. The function is -registered once, versioned, and then applied as a column. +embedding a column, calling a model, decoding an image. Registered once, +versioned, then applied as a column. -## The shape they share {#the-shape} +They compose. A materialized view can select computed columns, and a computed +column can be filled by a registered function. -All three separate **declaring** from **filling**, and that separation is the -point: +## Where the work runs {#where} -- Declaring is cheap and immediate. It records intent in the schema and commits - a new table version, with no data written. -- Filling is a separate call you control, can retry, and — on Cloud and - Enterprise — can hand to the server as a job rather than waiting on. +Enterprise -Filling never revisits work it has already done. A refresh gives values to rows -that lack them and leaves the rest alone, so calling it twice is not twice the -work, and appending rows means the next refresh handles only the new ones. +Locally, a refresh runs in your process. Against a deployment it runs across the +cluster, and the API reflects that: `refresh_column` raises, and +`refresh_column_async` returns a [job](/basics/jobs) handle instead. -The corollary is worth stating plainly: **a filled value is not recomputed when -its inputs change.** If a column derives from `price` and a row's `price` is -later updated, the derived value stays as it was computed. Recomputing means -dropping the column and declaring it again. That is a deliberate trade — it is -what makes refreshes idempotent and cheap — but it will surprise you if you -expect a spreadsheet. +```python +job = table.refresh_column_async("embedding") +result = job.wait() +result.rows_assigned +``` + +A backfill over a large table can therefore be started, left, and checked on +from a different process -- the connection does not have to stay open. + +That relocation is the main thing to understand about Enterprise feature +engineering. The API is the same; what changes is that the code runs somewhere +else, which is also why a function must declare the environment it needs. + +## Failure is resumable {#failures} + +A function that raises fails the job. There is no per-row retry or skip policy +in the function definition, so a function calling a flaky external service +should handle that itself, inside the function body. + +The alternative is to let the job fail and run the refresh again. Since a +refresh never revisits a filled row, re-running after a partial failure costs +only the rows that did not complete -- **a failed backfill is resumed, not +restarted.** ## Where to go next {#next} - - - Declare a column from a SQL expression and fill it. + + + Declare a column from a SQL expression. - - Derive a table from a query and refresh it incrementally. + + Fill the rows that lack values, locally or as a job. - - Run Python over your data to produce a column. + + Bring a derived table up to date with its source. + + + Register Python to fill a column with. diff --git a/docs/web/feature-engineering/materialized-views.mdx b/docs/web/feature-engineering/materialized-views.mdx index 534c006fd..dc46becf3 100644 --- a/docs/web/feature-engineering/materialized-views.mdx +++ b/docs/web/feature-engineering/materialized-views.mdx @@ -1,6 +1,6 @@ --- -title: "Materialized views" -sidebarTitle: "Materialized views" +title: "Refresh a materialized view" +sidebarTitle: "Refresh Materialized View" description: "A lakehouse object: a table defined by a query over another table, kept current incrementally." icon: table keywords: ["materialized view", "create_materialized_view", "refresh", "incremental"] @@ -74,22 +74,27 @@ Concurrent refreshes of one view do not duplicate rows. If two plan the same source rows, they conflict on commit and the loser raises rather than writing them twice — so a retry loop around `refresh()` is safe. -## Creating one against a deployment {#availability} +## Against a deployment {#availability} - -`create_materialized_view` on this client builds a view over a **local** -database. Called on a remote `db://` connection it raises -`NotImplementedError`. +Enterprise -Views exist on Enterprise deployments too, where a refresh runs across the -cluster rather than in your process — but they are created through the -Enterprise feature engineering interface rather than this method. See the -Enterprise documentation if that is where your data lives. - +`create_materialized_view` works on a `db://` connection as well as a local one. +What changes is where the refresh runs -- across the cluster rather than in your +process -- and that SQL can drive it: -The direction is the reverse of [functions](/feature-engineering/functions), -which register only against Enterprise and raise locally. Worth -knowing before concluding either is unavailable. +```sql +REFRESH MATERIALIZED VIEW chunks +``` + +`WITH (full = true)` recomputes every row instead of only the rows the source +has added since the last refresh. `SHOW MATERIALIZED VIEWS` lists them, which is +cheaper than `list_materialized_views` -- that method finds views by reading +every table's schema, so it costs one open per table. + + +`CASCADE` is accepted by the parser but not yet implemented. A view built on +another view is refreshed by refreshing each in dependency order. + ## Working with the view {#working-with-the-view} @@ -103,7 +108,7 @@ An existing view is reopened with `db.open_materialized_view(name)`. ## What to read next {#next} - + Derive a column instead of a whole table. diff --git a/docs/web/feature-engineering/refresh-column.mdx b/docs/web/feature-engineering/refresh-column.mdx new file mode 100644 index 000000000..4db0fc6a8 --- /dev/null +++ b/docs/web/feature-engineering/refresh-column.mdx @@ -0,0 +1,120 @@ +--- +title: "Refresh a column" +sidebarTitle: "Refresh Column" +description: "Fill the rows of a computed column that lack values — in process, or as a server-side job." +icon: arrows-rotate +keywords: ["refresh_column", "backfill", "job", "idempotent", "REFRESH COLUMN"] +--- + +A [declared column](/feature-engineering/add-computed-column) starts empty. A +refresh gives values to the rows that lack them. + + +```python Python icon="python" +table.refresh_column("doubled") +# RefreshColumnResult(rows_filled=2, version=3) + +table.to_arrow().sort_by("x").to_pandas() +# x doubled +# 0 1 2 +# 1 2 4 +``` + +```sql SQL +REFRESH COLUMN doubled ON my_table +``` + + +The refresh commits an ordinary table version, so +[versioning](/tables/versioning) applies to it like any other write. + +## It never revisits filled rows {#idempotent} + +A refresh fills rows that have no value and leaves the rest alone. Two +consequences follow, and both are load-bearing. + +**Appending is cheap.** Rows added since the last refresh are filled by the +next one; everything before it is skipped. Backfilling a column and then +ingesting more data does not mean recomputing the backfill. + +**A filled value does not track its input.** If `doubled` derives from `x` and a +row's `x` is later updated, `doubled` keeps the value computed when it was +filled. Recomputing means dropping the column and declaring it again. + +That is what makes the call idempotent and safe to retry -- but it is the +opposite of a spreadsheet formula, so be deliberate about which columns you +derive this way. + +## Refreshing part of a table {#partial} + +Enterprise In SQL a refresh can be scoped to a +predicate, which is useful for filling a slice first and checking it before +committing to the whole table: + +```sql +REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01' +``` + +## As a job {#async} + +Enterprise + +Against a deployment the refresh runs on the cluster. `refresh_column` raises +there; use `refresh_column_async`, which returns a [job](/basics/jobs) you can +wait on or poll instead of holding a connection open: + +```python +job = table.refresh_column_async("doubled") +result = job.wait() +result.rows_assigned +# 2 +job.status() +# 'finished' +``` + +Poll with `status()`, resolve with `wait()`. `status()` is a point snapshot and +does not raise, which is what makes it safe in a loop; `wait()` raises +`JobFailedError`, `JobCancelledError` or `TimeoutError`. + +A long backfill can therefore be started, left, and checked on from a different +process. + +### Resourcing the job {#resources} + +SQL exposes the worker budget directly, which is the lever when a backfill is +too slow or too expensive: + +```sql +REFRESH COLUMN embedding ON documents + WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2) +``` + +`num_workers` sets the count instead of a per-worker budget; the two are +alternatives rather than a pair. `batch_size` controls how many rows a worker +takes at a time, and `error_policy = 'skip'` continues past rows that fail +rather than failing the job. + +## When a refresh fails {#failures} + +A failure is not a rollback. Rows filled before the failure keep their values, +because each is committed as it completes. + +That makes recovery simple: run the refresh again. It skips everything already +filled and costs only the rows that did not complete -- **a failed backfill is +resumed, not restarted.** + +For a column filled by a [function](/management/function), the usual cause is +the worker environment rather than the data. See +[environment mismatches](/management/function#mismatch-symptoms) for the +symptoms, which rarely name their cause. + +## What to read next {#next} + + + + Waiting on, inspecting, and cancelling server-side work. + + + The same idea, for a whole derived table. + + diff --git a/docs/web/maintenance/compaction.mdx b/docs/web/maintenance/compaction.mdx index 13fdda503..af8ea05d2 100644 --- a/docs/web/maintenance/compaction.mdx +++ b/docs/web/maintenance/compaction.mdx @@ -95,7 +95,7 @@ continuous small writes needs it on a schedule. Compaction rewrites files, which conflicts with a concurrent -[computed-column refresh](/feature-engineering/computed-columns) or +[computed-column refresh](/feature-engineering/add-computed-column) or [view refresh](/feature-engineering/materialized-views) planned against the version it replaces. Sequence them rather than running both at once. diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx index c1d3e9096..170adb184 100644 --- a/docs/web/management/database.mdx +++ b/docs/web/management/database.mdx @@ -42,7 +42,7 @@ storage and does the work itself. The third talks to a **deployment**. | Table handle | `LanceTable` | `RemoteTable` | | [SQL](/basics/apis) | -- | yes | | Server-side [jobs](/basics/jobs) | -- | yes | -| [Functions](/feature-engineering/functions) | -- | yes | +| [Functions](/management/function) | -- | yes | | [Authorization](/governance/authorization) | storage permissions | granted per object | ![](/static/assets/images/overview/understanding-tables.png) diff --git a/docs/web/feature-engineering/functions.mdx b/docs/web/management/function.mdx similarity index 66% rename from docs/web/feature-engineering/functions.mdx rename to docs/web/management/function.mdx index 8d5ab604c..705603b64 100644 --- a/docs/web/feature-engineering/functions.mdx +++ b/docs/web/management/function.mdx @@ -15,7 +15,7 @@ rather than pulling rows to your process. Registration is a remote operation. On a local connection `create_function` raises `NotImplementedError`; use a -[computed column](/feature-engineering/computed-columns) instead. +[computed column](/feature-engineering/refresh-column) instead. ## Define a function {#define} @@ -54,6 +54,42 @@ These are part of the **immutable function version**. Changing a dependency produces a new version rather than altering the one already registered, so a column filled by version 1 is unaffected by version 2. +Declaring them is not optional bookkeeping. If the worker environment differs +from the one the function was written against, the failures are among the +hardest to diagnose in the system, because the error rarely names the cause. + +### Symptoms of an environment mismatch {#mismatch-symptoms} + +| Symptom | Likely cause | +| --- | --- | +| `TypeError: Enum.__new__() missing 1 required positional argument` | `attrs` version mismatch | +| `TypeError: Can't instantiate abstract class` | package structure differs between versions | +| `ArrowInvalid: cannot cast`, or serialization errors | NumPy 1.x against 2.x | +| `ModuleNotFoundError` on the worker | package installed locally but not declared | +| Model loading failures | PyTorch version mismatch | +| Permission denied | an API key present locally but absent from `env` | + +The packages worth pinning explicitly, because a mismatch in them produces the +errors above rather than a clear one: **numpy, torch, pyarrow, attrs, +pydantic**. + + +The client version defaults `python_version`, so a function registered from a +3.11 environment runs on 3.11 unless you say otherwise. That default is usually +right and occasionally surprising -- pin it when the function is sensitive to it. + + +## Failure handling {#failures} + +A function that raises fails the job. There is no per-row retry or skip policy +in the function definition: `@udf` takes no error-handling parameter, so a +function that calls a flaky external service should handle that itself, inside +the function body. + +Letting the job fail is the other option. Since +[a refresh never revisits a filled row](/feature-engineering/refresh-column#idempotent), +re-running after a partial failure costs only the rows that did not complete. + ## What gets packaged {#packaging} @@ -96,13 +132,13 @@ in the named column. A bare named-struct application expands its result fields as one atomic binding, with aliases from `rename(columns=...)`. Filling then works exactly as it does for any computed column — see -[Computed columns](/feature-engineering/computed-columns) for refresh +[Computed columns](/feature-engineering/refresh-column) for refresh semantics, including the rule that filled rows are never revisited. ## What to read next {#next} - + Refresh semantics, and the SQL alternative. diff --git a/docs/web/management/index.mdx b/docs/web/management/index.mdx index 3ec4622a7..38ef22367 100644 --- a/docs/web/management/index.mdx +++ b/docs/web/management/index.mdx @@ -87,7 +87,7 @@ method for them, because the thing a view stores *is* a SQL statement. See Enterprise A function runs on the server, so there is no local form. The TypeScript SDK has no function methods; use Python or SQL. -See [Functions](/feature-engineering/functions). +See [Functions](/management/function). ### Jobs {#jobs} diff --git a/docs/web/quickstart.mdx b/docs/web/quickstart.mdx index 7811eafc2..c79748e74 100644 --- a/docs/web/quickstart.mdx +++ b/docs/web/quickstart.mdx @@ -410,8 +410,8 @@ job = characters.refresh_column_async("power_score") job.wait() ``` -See [Computed columns](/feature-engineering/computed-columns) for expressions, -and [Functions](/feature-engineering/functions) for filling a column with Python +See [Computed columns](/feature-engineering/add-computed-column) for expressions, +and [Functions](/management/function) for filling a column with Python -- an embedding model, a captioner, anything you can register. ## 7\. Store multimodal data {#7-store-multimodal-data} diff --git a/docs/web/training/object-detection.mdx b/docs/web/training/object-detection.mdx index 136a6e7a9..7704190ba 100644 --- a/docs/web/training/object-detection.mdx +++ b/docs/web/training/object-detection.mdx @@ -74,7 +74,7 @@ Ingestion streams `pa.RecordBatch`es of raw frames + annotations directly into a ## 2\. Backfill curation features with Geneva {#2-backfill-curation-features-with-geneva} -Curation signals are added as columns on the same table using [computed columns](/feature-engineering/computed-columns). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows. +Curation signals are added as columns on the same table using [computed columns](/feature-engineering/refresh-column). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows. ```py Python icon=Python import pyarrow as pa @@ -117,7 +117,7 @@ with gconn.local_ray_context(): ``` -Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See [computed columns](/feature-engineering/computed-columns) for how a backfill fills only the rows that lack values. +Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See [computed columns](/feature-engineering/refresh-column) for how a backfill fills only the rows that lack values. ## 3\. Define training splits as materialized views {#3-define-training-splits-as-materialized-views} diff --git a/docs/web/tutorials/feature-engineering/index.mdx b/docs/web/tutorials/feature-engineering/index.mdx index 2d3a4478b..2576dd026 100644 --- a/docs/web/tutorials/feature-engineering/index.mdx +++ b/docs/web/tutorials/feature-engineering/index.mdx @@ -19,4 +19,4 @@ The relevant section of the documentation are listed below. |:--------|:------------| | [Feature engineering](/feature-engineering) | The three ways to derive columns and tables. | | [Materialized views](/feature-engineering/materialized-views) | Derive a table from a query and refresh it incrementally. | -| [Computed columns](/feature-engineering/computed-columns) | Declare a column from an expression and fill it. | +| [Computed columns](/feature-engineering/add-computed-column) | Declare a column from an expression and fill it. |