diff --git a/docs/web/basics/apis.mdx b/docs/web/basics/apis.mdx index 4d5e6c6eb..b302a9a3e 100644 --- a/docs/web/basics/apis.mdx +++ b/docs/web/basics/apis.mdx @@ -60,7 +60,7 @@ its result. Full-text search *is* available in SQL, as the ## TypeScript has no SQL method {#ts-no-sql} The TypeScript SDK does not expose SQL, and has no methods for -[functions](/feature-engineering/functions). Reach the endpoint with a Flight +[functions](/management/function). Reach the endpoint with a Flight SQL client instead -- see [connecting a client](/search/sql). This is a gap in the SDK rather than in the endpoint: the same statements work. diff --git a/docs/web/basics/catalog.mdx b/docs/web/basics/catalog.mdx index f850daadf..aa7732bec 100644 --- a/docs/web/basics/catalog.mdx +++ b/docs/web/basics/catalog.mdx @@ -23,7 +23,7 @@ feature store. The catalog part is what this page is about. | [Table](/tables) | Rows and columns, versioned. What most operations act on. | | [Materialized view](/feature-engineering/materialized-views) | A table whose contents come from a query, computed and stored. | | [View](/management/view) | A saved query. Computed when read, not stored. | -| [Function](/feature-engineering/functions) | Registered code, applied to produce column values. | +| [Function](/management/function) | Registered code, applied to produce column values. | ``` database @@ -47,7 +47,7 @@ these docs is about one of them: | | | | --- | --- | -| **Columns** | including vectors, blobs, and columns [computed](/feature-engineering/computed-columns) from other columns | +| **Columns** | including vectors, blobs, and columns [computed](/feature-engineering/add-computed-column) from other columns | | **Versions** | every write commits one, so reads can [travel back](/tables/versioning) | | **Indexes** | vector, full-text, and scalar, built over the columns | | **Branches** | isolated lines of change you can [merge or discard](/tables/branching) | diff --git a/docs/web/basics/lance.mdx b/docs/web/basics/lance.mdx index 448aaf2fc..adf753beb 100644 --- a/docs/web/basics/lance.mdx +++ b/docs/web/basics/lance.mdx @@ -103,7 +103,7 @@ than erased, so the data is recoverable for as long as the version is. Adding a derived column writes only the new column. Existing data -- the expensive part, the images and videos -- is untouched. That is what makes -[computed columns](/feature-engineering/computed-columns) a reasonable thing to +[computed columns](/feature-engineering/add-computed-column) a reasonable thing to do repeatedly on a large table rather than a full rewrite each time. 3.0", +}) +``` + +Declaration order is dependency order -- a computed column may reference one +declared earlier in the same mapping, but not one declared after it. + +The functions available are the ones the +[SQL dialect](/sql/functions) documents. + +## Its inputs become load-bearing {#input-constraints} + + +While a computed column's declaration references another column, that column +cannot be renamed, retyped, or dropped. The declaration is part of the schema, +and LanceDB will not let you invalidate it out from under itself. + + +This is worth knowing before deriving a column from something you expect to +reshape later. Dropping the computed column releases the constraint. + +## Filling with Python instead of SQL {#functions} + +Enterprise + +When the value cannot be expressed in SQL -- an embedding, a model call, an +image decode -- the column is filled by a registered +[function](/management/function) instead of an expression. Declaring it looks +the same; what differs is what fills it. + +## What to read next {#next} + + + + Fill the rows that lack values. + + + Fill a column with Python. + + diff --git a/docs/web/feature-engineering/computed-columns.mdx b/docs/web/feature-engineering/computed-columns.mdx deleted file mode 100644 index b9f07c4d2..000000000 --- a/docs/web/feature-engineering/computed-columns.mdx +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: "Computed columns" -sidebarTitle: "Computed columns" -description: "Declare a column from a SQL expression and fill it when you choose." -icon: table-columns -keywords: ["computed columns", "add_columns", "refresh_column", "backfill"] ---- - -A computed column is declared with an expression and starts with no values. A -separate refresh fills the rows that lack them. Splitting the two means -declaring a column on a billion-row table costs the same as declaring one on an -empty table: nothing is evaluated until you ask for it. - -## Declare and fill {#declare-and-fill} - -```python -import lancedb - -db = lancedb.connect("./.lancedb") -table = db.create_table("computed_demo", [{"x": 1}, {"x": 2}]) - -table.add_columns(computed={"doubled": "x * 2"}) -# AddColumnsResult(version=2) - -table.refresh_column("doubled") -# RefreshColumnResult(rows_filled=2, version=3) - -table.to_arrow().sort_by("x").to_pandas() -# x doubled -# 0 1 2 -# 1 2 4 -``` - -`add_columns(computed=...)` commits a new table version with the column present -and empty. `refresh_column` fills it and commits another. Both are ordinary -table versions, so the usual [versioning](/tables/versioning) applies. - -## `computed` and `transforms` are different {#computed-vs-transforms} - -`add_columns` takes either, and the difference matters: - -- **`transforms`** evaluates the expression *now*, for every row, as part of the - call. Good for a small table or a column you want populated immediately. -- **`computed`** stores the expression and evaluates nothing. The column is - committed empty and filled later by `refresh_column`. - -They cannot be combined in one call. Reach for `computed` when the table is -large enough that you would rather control when the work happens. - -## Refreshes do not revisit filled rows {#idempotent} - -A refresh gives values to rows that have none. Rows already filled are left -alone. Two consequences follow: - -**Appending is cheap.** Rows added since the last refresh are filled by the -next one; everything before it is skipped. Backfilling a column and then -ingesting more data does not mean recomputing the backfill. - -**A filled value does not track its input.** If `doubled` derives from `x` and -a row's `x` is later updated, `doubled` keeps the value computed when it was -filled. Recomputing means dropping the column and declaring it again. - -That is what makes the call idempotent and safe to retry, but it is the opposite -of a spreadsheet formula, so it is worth being deliberate about which columns -you derive this way. - -## Constraints on the inputs {#input-constraints} - -While a computed column's declaration references another column, that column -cannot be renamed, retyped, or dropped. The declaration is part of the schema, -and LanceDB will not let you invalidate it out from under itself. - -Declaration order is also dependency order: a computed column may reference one -declared earlier in the same mapping. - -## Filling as a job {#async} - -Enterprise - -Against a remote database the refresh runs on the server. `refresh_column` -raises there; use `refresh_column_async`, which returns a job you can wait on or -poll instead of holding a connection open: - -```python -job = table.refresh_column_async("doubled") -result = job.wait() -result.rows_assigned -# 2 -job.status() -# 'finished' -``` - -The job handle carries `id`, `status()`, `wait(timeout=...)` and `cancel()`, so -a long backfill can be started, left, and checked on later. - -## Filling with Python instead of SQL {#functions} - -Enterprise - -When the value cannot be expressed in SQL — an embedding, a model call, an image -decode — the column is filled by a registered function instead of an expression. -See [Functions](/feature-engineering/functions). - -## What to read next {#next} - - - - Derive a whole table rather than a column. - - - Fill a column with Python. - - diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx index 527a790ef..ef4a5dc3f 100644 --- a/docs/web/feature-engineering/index.mdx +++ b/docs/web/feature-engineering/index.mdx @@ -1,76 +1,110 @@ --- -title: "LanceDB Feature Engineering" +title: "Feature engineering" sidebarTitle: "Overview" description: "Derive columns and tables from data already in the lakehouse, without moving it somewhere else to compute." icon: layer-group -keywords: ["feature engineering", "computed columns", "materialized views", "functions", "lakehouse"] +keywords: ["feature engineering", "computed columns", "materialized views", "functions", "lakehouse", "jobs"] --- Feature engineering happens where the data already is. Rather than exporting a table, computing somewhere else and writing the result back, you declare what a column or a table should contain and let LanceDB fill it. -**Functions and materialized views are objects in the Lance Multimodal -Lakehouse**, alongside tables and indexes. They are created, versioned, and -refreshed like anything else in it — not a separate system bolted alongside, and -not a pipeline you assemble out of external orchestration. +[Functions](/management/function) and +[materialized views](/feature-engineering/materialized-views) are ordinary +[catalog objects](/basics/catalog), alongside tables and indexes -- created, +versioned, and refreshed like anything else, not a separate system bolted +alongside. -Three mechanisms cover different shapes of the problem. They compose — a -materialized view can select computed columns, and a computed column can be -filled by a registered function — but each is worth understanding on its own. +## The shape everything shares {#the-shape} + +Every mechanism here separates **declaring** from **filling**, and that +separation is the point: + +- Declaring is cheap and immediate. It records intent in the schema and commits + a new table version, with no data written. Declaring a column on a + billion-row table costs what declaring one on an empty table costs. +- Filling is a separate call you control, can retry, and -- against a deployment + -- can hand to the server as a [job](/basics/jobs) rather than waiting on. + +**Filling never revisits work it has already done.** A refresh gives values to +rows that lack them and leaves the rest alone, so calling it twice is not twice +the work, and appending rows means the next refresh handles only the new ones. + +The corollary is worth stating plainly: **a filled value is not recomputed when +its inputs change.** If a column derives from `price` and a row's `price` is +later updated, the derived value stays as it was computed. Recomputing means +dropping the column and declaring it again. That is a deliberate trade -- it is +what makes refreshes idempotent and cheap -- but it will surprise you if you +expect a spreadsheet. ## Which one to reach for {#which-one} | | What it produces | Filled by | Available | | --- | --- | --- | --- | -| [Computed columns](/feature-engineering/computed-columns) | a new column on an existing table | `refresh_column()` | everywhere | -| [Materialized views](/feature-engineering/materialized-views) | a new table derived by a query | `refresh()` | local databases | -| [Functions](/feature-engineering/functions) | a computed column filled by Python | a registered function | Enterprise | +| [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | everywhere | +| [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | everywhere | +| [Function](/management/function) | a computed column filled by Python | registered code | Enterprise | -**Computed columns** are the common case. A column is declared with a SQL -expression and starts empty; a refresh gives values to the rows that lack them. -Declaring one costs the same on a billion-row table as on an empty one, because -nothing is evaluated until you ask. +**Computed columns** are the common case: a column declared with a SQL +expression, filled by [a refresh](/feature-engineering/refresh-column). -**Materialized views** derive a whole table — a filtered, projected, or -aggregated view of a source — and keep it up to date incrementally as the source -changes. The result is an ordinary table: queryable, indexable, searchable. +**Materialized views** derive a whole table -- a filtered, projected, or +aggregated view of a source -- and keep it current incrementally. The result is +an ordinary table: queryable, indexable, searchable. **Functions** run Python where the data is, for work SQL cannot express: -embedding a column, calling a model, decoding an image. The function is -registered once, versioned, and then applied as a column. +embedding a column, calling a model, decoding an image. Registered once, +versioned, then applied as a column. -## The shape they share {#the-shape} +They compose. A materialized view can select computed columns, and a computed +column can be filled by a registered function. -All three separate **declaring** from **filling**, and that separation is the -point: +## Where the work runs {#where} -- Declaring is cheap and immediate. It records intent in the schema and commits - a new table version, with no data written. -- Filling is a separate call you control, can retry, and — on Cloud and - Enterprise — can hand to the server as a job rather than waiting on. +Enterprise -Filling never revisits work it has already done. A refresh gives values to rows -that lack them and leaves the rest alone, so calling it twice is not twice the -work, and appending rows means the next refresh handles only the new ones. +Locally, a refresh runs in your process. Against a deployment it runs across the +cluster, and the API reflects that: `refresh_column` raises, and +`refresh_column_async` returns a [job](/basics/jobs) handle instead. -The corollary is worth stating plainly: **a filled value is not recomputed when -its inputs change.** If a column derives from `price` and a row's `price` is -later updated, the derived value stays as it was computed. Recomputing means -dropping the column and declaring it again. That is a deliberate trade — it is -what makes refreshes idempotent and cheap — but it will surprise you if you -expect a spreadsheet. +```python +job = table.refresh_column_async("embedding") +result = job.wait() +result.rows_assigned +``` + +A backfill over a large table can therefore be started, left, and checked on +from a different process -- the connection does not have to stay open. + +That relocation is the main thing to understand about Enterprise feature +engineering. The API is the same; what changes is that the code runs somewhere +else, which is also why a function must declare the environment it needs. + +## Failure is resumable {#failures} + +A function that raises fails the job. There is no per-row retry or skip policy +in the function definition, so a function calling a flaky external service +should handle that itself, inside the function body. + +The alternative is to let the job fail and run the refresh again. Since a +refresh never revisits a filled row, re-running after a partial failure costs +only the rows that did not complete -- **a failed backfill is resumed, not +restarted.** ## Where to go next {#next} - - - Declare a column from a SQL expression and fill it. + + + Declare a column from a SQL expression. - - Derive a table from a query and refresh it incrementally. + + Fill the rows that lack values, locally or as a job. - - Run Python over your data to produce a column. + + Bring a derived table up to date with its source. + + + Register Python to fill a column with. diff --git a/docs/web/feature-engineering/materialized-views.mdx b/docs/web/feature-engineering/materialized-views.mdx index 534c006fd..dc46becf3 100644 --- a/docs/web/feature-engineering/materialized-views.mdx +++ b/docs/web/feature-engineering/materialized-views.mdx @@ -1,6 +1,6 @@ --- -title: "Materialized views" -sidebarTitle: "Materialized views" +title: "Refresh a materialized view" +sidebarTitle: "Refresh Materialized View" description: "A lakehouse object: a table defined by a query over another table, kept current incrementally." icon: table keywords: ["materialized view", "create_materialized_view", "refresh", "incremental"] @@ -74,22 +74,27 @@ Concurrent refreshes of one view do not duplicate rows. If two plan the same source rows, they conflict on commit and the loser raises rather than writing them twice — so a retry loop around `refresh()` is safe. -## Creating one against a deployment {#availability} +## Against a deployment {#availability} - -`create_materialized_view` on this client builds a view over a **local** -database. Called on a remote `db://` connection it raises -`NotImplementedError`. +Enterprise -Views exist on Enterprise deployments too, where a refresh runs across the -cluster rather than in your process — but they are created through the -Enterprise feature engineering interface rather than this method. See the -Enterprise documentation if that is where your data lives. - +`create_materialized_view` works on a `db://` connection as well as a local one. +What changes is where the refresh runs -- across the cluster rather than in your +process -- and that SQL can drive it: -The direction is the reverse of [functions](/feature-engineering/functions), -which register only against Enterprise and raise locally. Worth -knowing before concluding either is unavailable. +```sql +REFRESH MATERIALIZED VIEW chunks +``` + +`WITH (full = true)` recomputes every row instead of only the rows the source +has added since the last refresh. `SHOW MATERIALIZED VIEWS` lists them, which is +cheaper than `list_materialized_views` -- that method finds views by reading +every table's schema, so it costs one open per table. + + +`CASCADE` is accepted by the parser but not yet implemented. A view built on +another view is refreshed by refreshing each in dependency order. + ## Working with the view {#working-with-the-view} @@ -103,7 +108,7 @@ An existing view is reopened with `db.open_materialized_view(name)`. ## What to read next {#next} - + Derive a column instead of a whole table. diff --git a/docs/web/feature-engineering/refresh-column.mdx b/docs/web/feature-engineering/refresh-column.mdx new file mode 100644 index 000000000..4db0fc6a8 --- /dev/null +++ b/docs/web/feature-engineering/refresh-column.mdx @@ -0,0 +1,120 @@ +--- +title: "Refresh a column" +sidebarTitle: "Refresh Column" +description: "Fill the rows of a computed column that lack values — in process, or as a server-side job." +icon: arrows-rotate +keywords: ["refresh_column", "backfill", "job", "idempotent", "REFRESH COLUMN"] +--- + +A [declared column](/feature-engineering/add-computed-column) starts empty. A +refresh gives values to the rows that lack them. + + +```python Python icon="python" +table.refresh_column("doubled") +# RefreshColumnResult(rows_filled=2, version=3) + +table.to_arrow().sort_by("x").to_pandas() +# x doubled +# 0 1 2 +# 1 2 4 +``` + +```sql SQL +REFRESH COLUMN doubled ON my_table +``` + + +The refresh commits an ordinary table version, so +[versioning](/tables/versioning) applies to it like any other write. + +## It never revisits filled rows {#idempotent} + +A refresh fills rows that have no value and leaves the rest alone. Two +consequences follow, and both are load-bearing. + +**Appending is cheap.** Rows added since the last refresh are filled by the +next one; everything before it is skipped. Backfilling a column and then +ingesting more data does not mean recomputing the backfill. + +**A filled value does not track its input.** If `doubled` derives from `x` and a +row's `x` is later updated, `doubled` keeps the value computed when it was +filled. Recomputing means dropping the column and declaring it again. + +That is what makes the call idempotent and safe to retry -- but it is the +opposite of a spreadsheet formula, so be deliberate about which columns you +derive this way. + +## Refreshing part of a table {#partial} + +Enterprise In SQL a refresh can be scoped to a +predicate, which is useful for filling a slice first and checking it before +committing to the whole table: + +```sql +REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01' +``` + +## As a job {#async} + +Enterprise + +Against a deployment the refresh runs on the cluster. `refresh_column` raises +there; use `refresh_column_async`, which returns a [job](/basics/jobs) you can +wait on or poll instead of holding a connection open: + +```python +job = table.refresh_column_async("doubled") +result = job.wait() +result.rows_assigned +# 2 +job.status() +# 'finished' +``` + +Poll with `status()`, resolve with `wait()`. `status()` is a point snapshot and +does not raise, which is what makes it safe in a loop; `wait()` raises +`JobFailedError`, `JobCancelledError` or `TimeoutError`. + +A long backfill can therefore be started, left, and checked on from a different +process. + +### Resourcing the job {#resources} + +SQL exposes the worker budget directly, which is the lever when a backfill is +too slow or too expensive: + +```sql +REFRESH COLUMN embedding ON documents + WITH (cpu_millis = 8000, memory_bytes = 17179869184, gpu_count = 2) +``` + +`num_workers` sets the count instead of a per-worker budget; the two are +alternatives rather than a pair. `batch_size` controls how many rows a worker +takes at a time, and `error_policy = 'skip'` continues past rows that fail +rather than failing the job. + +## When a refresh fails {#failures} + +A failure is not a rollback. Rows filled before the failure keep their values, +because each is committed as it completes. + +That makes recovery simple: run the refresh again. It skips everything already +filled and costs only the rows that did not complete -- **a failed backfill is +resumed, not restarted.** + +For a column filled by a [function](/management/function), the usual cause is +the worker environment rather than the data. See +[environment mismatches](/management/function#mismatch-symptoms) for the +symptoms, which rarely name their cause. + +## What to read next {#next} + + + + Waiting on, inspecting, and cancelling server-side work. + + + The same idea, for a whole derived table. + + diff --git a/docs/web/maintenance/compaction.mdx b/docs/web/maintenance/compaction.mdx index 13fdda503..af8ea05d2 100644 --- a/docs/web/maintenance/compaction.mdx +++ b/docs/web/maintenance/compaction.mdx @@ -95,7 +95,7 @@ continuous small writes needs it on a schedule. Compaction rewrites files, which conflicts with a concurrent -[computed-column refresh](/feature-engineering/computed-columns) or +[computed-column refresh](/feature-engineering/add-computed-column) or [view refresh](/feature-engineering/materialized-views) planned against the version it replaces. Sequence them rather than running both at once. diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx index c1d3e9096..170adb184 100644 --- a/docs/web/management/database.mdx +++ b/docs/web/management/database.mdx @@ -42,7 +42,7 @@ storage and does the work itself. The third talks to a **deployment**. | Table handle | `LanceTable` | `RemoteTable` | | [SQL](/basics/apis) | -- | yes | | Server-side [jobs](/basics/jobs) | -- | yes | -| [Functions](/feature-engineering/functions) | -- | yes | +| [Functions](/management/function) | -- | yes | | [Authorization](/governance/authorization) | storage permissions | granted per object | ![](/static/assets/images/overview/understanding-tables.png) diff --git a/docs/web/feature-engineering/functions.mdx b/docs/web/management/function.mdx similarity index 66% rename from docs/web/feature-engineering/functions.mdx rename to docs/web/management/function.mdx index 8d5ab604c..705603b64 100644 --- a/docs/web/feature-engineering/functions.mdx +++ b/docs/web/management/function.mdx @@ -15,7 +15,7 @@ rather than pulling rows to your process. Registration is a remote operation. On a local connection `create_function` raises `NotImplementedError`; use a -[computed column](/feature-engineering/computed-columns) instead. +[computed column](/feature-engineering/refresh-column) instead. ## Define a function {#define} @@ -54,6 +54,42 @@ These are part of the **immutable function version**. Changing a dependency produces a new version rather than altering the one already registered, so a column filled by version 1 is unaffected by version 2. +Declaring them is not optional bookkeeping. If the worker environment differs +from the one the function was written against, the failures are among the +hardest to diagnose in the system, because the error rarely names the cause. + +### Symptoms of an environment mismatch {#mismatch-symptoms} + +| Symptom | Likely cause | +| --- | --- | +| `TypeError: Enum.__new__() missing 1 required positional argument` | `attrs` version mismatch | +| `TypeError: Can't instantiate abstract class` | package structure differs between versions | +| `ArrowInvalid: cannot cast`, or serialization errors | NumPy 1.x against 2.x | +| `ModuleNotFoundError` on the worker | package installed locally but not declared | +| Model loading failures | PyTorch version mismatch | +| Permission denied | an API key present locally but absent from `env` | + +The packages worth pinning explicitly, because a mismatch in them produces the +errors above rather than a clear one: **numpy, torch, pyarrow, attrs, +pydantic**. + + +The client version defaults `python_version`, so a function registered from a +3.11 environment runs on 3.11 unless you say otherwise. That default is usually +right and occasionally surprising -- pin it when the function is sensitive to it. + + +## Failure handling {#failures} + +A function that raises fails the job. There is no per-row retry or skip policy +in the function definition: `@udf` takes no error-handling parameter, so a +function that calls a flaky external service should handle that itself, inside +the function body. + +Letting the job fail is the other option. Since +[a refresh never revisits a filled row](/feature-engineering/refresh-column#idempotent), +re-running after a partial failure costs only the rows that did not complete. + ## What gets packaged {#packaging} @@ -96,13 +132,13 @@ in the named column. A bare named-struct application expands its result fields as one atomic binding, with aliases from `rename(columns=...)`. Filling then works exactly as it does for any computed column — see -[Computed columns](/feature-engineering/computed-columns) for refresh +[Computed columns](/feature-engineering/refresh-column) for refresh semantics, including the rule that filled rows are never revisited. ## What to read next {#next} - + Refresh semantics, and the SQL alternative. diff --git a/docs/web/management/index.mdx b/docs/web/management/index.mdx index 3ec4622a7..38ef22367 100644 --- a/docs/web/management/index.mdx +++ b/docs/web/management/index.mdx @@ -87,7 +87,7 @@ method for them, because the thing a view stores *is* a SQL statement. See Enterprise A function runs on the server, so there is no local form. The TypeScript SDK has no function methods; use Python or SQL. -See [Functions](/feature-engineering/functions). +See [Functions](/management/function). ### Jobs {#jobs} diff --git a/docs/web/quickstart.mdx b/docs/web/quickstart.mdx index 7811eafc2..c79748e74 100644 --- a/docs/web/quickstart.mdx +++ b/docs/web/quickstart.mdx @@ -410,8 +410,8 @@ job = characters.refresh_column_async("power_score") job.wait() ``` -See [Computed columns](/feature-engineering/computed-columns) for expressions, -and [Functions](/feature-engineering/functions) for filling a column with Python +See [Computed columns](/feature-engineering/add-computed-column) for expressions, +and [Functions](/management/function) for filling a column with Python -- an embedding model, a captioner, anything you can register. ## 7\. Store multimodal data {#7-store-multimodal-data} diff --git a/docs/web/training/object-detection.mdx b/docs/web/training/object-detection.mdx index 136a6e7a9..7704190ba 100644 --- a/docs/web/training/object-detection.mdx +++ b/docs/web/training/object-detection.mdx @@ -74,7 +74,7 @@ Ingestion streams `pa.RecordBatch`es of raw frames + annotations directly into a ## 2\. Backfill curation features with Geneva {#2-backfill-curation-features-with-geneva} -Curation signals are added as columns on the same table using [computed columns](/feature-engineering/computed-columns). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows. +Curation signals are added as columns on the same table using [computed columns](/feature-engineering/refresh-column). Backfills are incremental and checkpointed: re-running the command after new footage arrives only computes the new rows. ```py Python icon=Python import pyarrow as pa @@ -117,7 +117,7 @@ with gconn.local_ray_context(): ``` -Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See [computed columns](/feature-engineering/computed-columns) for how a backfill fills only the rows that lack values. +Because the curation features are flat scalar columns on the same table, all four retrieval modes — SQL, full-text search, vector search, and SQL-filtered vector search — work directly without joins or exports. See [computed columns](/feature-engineering/refresh-column) for how a backfill fills only the rows that lack values. ## 3\. Define training splits as materialized views {#3-define-training-splits-as-materialized-views} diff --git a/docs/web/tutorials/feature-engineering/index.mdx b/docs/web/tutorials/feature-engineering/index.mdx index 2d3a4478b..2576dd026 100644 --- a/docs/web/tutorials/feature-engineering/index.mdx +++ b/docs/web/tutorials/feature-engineering/index.mdx @@ -19,4 +19,4 @@ The relevant section of the documentation are listed below. |:--------|:------------| | [Feature engineering](/feature-engineering) | The three ways to derive columns and tables. | | [Materialized views](/feature-engineering/materialized-views) | Derive a table from a query and refresh it incrementally. | -| [Computed columns](/feature-engineering/computed-columns) | Declare a column from an expression and fill it. | +| [Computed columns](/feature-engineering/add-computed-column) | Declare a column from an expression and fill it. |