From e20664740fadf76541bcaabd7cdc997e3b8ddf7d Mon Sep 17 00:00:00 2001 From: Brendan Clement Date: Fri, 9 Oct 2026 13:07:33 -0700 Subject: [PATCH] docs: publish the Function pages only once Functions launch The two Function pages declare `launch: functions`, and every other page marks what it says about Functions, so the assembled site leaves them out until the launch. Before it, the pages that offered Functions for Python work point Enterprise readers at Geneva instead. The SQL Function example now uses Arrow type names, which the server requires, and states the 3.12 default for `python_version`. --- CONTRIBUTING.md | 20 +++++++++++++++- docs/web/architecture.mdx | 5 ++-- docs/web/basics/catalog.mdx | 18 +++++++++++--- docs/web/basics/jobs.mdx | 10 ++++---- docs/web/basics/lakehouse.mdx | 4 ++-- docs/web/basics/limits.mdx | 2 +- docs/web/basics/query.mdx | 4 ++-- docs/web/basics/sdk.mdx | 6 ++--- docs/web/enterprise/index.mdx | 2 +- .../add-computed-column.mdx | 19 ++++++++++++++- .../feature-engineering/data-evolution.mdx | 9 ++++++- docs/web/feature-engineering/index.mdx | 24 ++++++++++++++----- .../feature-engineering/refresh-column.mdx | 7 +++--- docs/web/governance/authorization.mdx | 16 +++++++++++-- docs/web/maintenance/background-jobs.mdx | 8 ++++--- docs/web/management/database.mdx | 2 +- docs/web/management/function.mdx | 1 + docs/web/management/index.mdx | 2 ++ docs/web/management/table.mdx | 4 ++-- docs/web/quickstart.mdx | 5 ++-- docs/web/sql/ddl/function.mdx | 10 ++++++-- docs/web/sql/ddl/namespace.mdx | 2 +- docs/web/sql/ddl/table.mdx | 6 +++-- docs/web/sql/dml/refresh.mdx | 2 ++ docs/web/sql/index.mdx | 2 +- 25 files changed, 142 insertions(+), 48 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1e111fdf2..a47da71cf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -94,7 +94,7 @@ what the capability is, and what the embedded form does instead. A separate private repository supplies the fuller version of each, which replaces the page at the same path when the published site is assembled. -Three things to know before editing: +Four things to know before editing: **Code examples are not written into the pages.** They live in real tests under `docs/web-tests/{py,ts,rs}`, are extracted into `docs/web/snippets/`, and are @@ -137,6 +137,24 @@ also open the page with a badge of its own, and keep a qualification that only one section needs in that section's text. A `mint dev` preview of `docs/web` alone renders neither the labels nor the comparison. +**A feature can be documented before it launches.** Functions are documented +ahead of their launch. A page that documents only them says so in its +frontmatter: + +```yaml +launch: functions +``` + +Another page marks what it says about them. A section between +`{/* if functions launched */}` and `{/* end if */}`, each on a line of its own, +is published only once Functions launch; `{/* if functions not launched */}` +marks one published only until then. Put the marker at the end of a table row or +list item to gate that line alone. Until the launch, the assembled site leaves +out those pages, their examples, and every marked section, navigation entry and +redirect, and stops the build if anything still published links to them. A +`mint dev` preview of `docs/web` alone shows both sides of every marker. The +documentation repository's README shows how to assemble either site. + ## Release process For information on the release process, see: [release_process.md](release_process.md) diff --git a/docs/web/architecture.mdx b/docs/web/architecture.mdx index 4ee1bc0e7..0ef2210c3 100644 --- a/docs/web/architecture.mdx +++ b/docs/web/architecture.mdx @@ -90,7 +90,8 @@ path. scales for interactive traffic without also scaling indexing capacity, and heavy background work does not steal resources from user queries. Because a server exists, so do the things that need one: [SQL](/sql), -[functions](/management/function), server-side jobs, and +[functions](/management/function), {/* if functions launched */} +server-side jobs, and [authorization](/governance/authorization) per object rather than per bucket. **What it is good at.** Many concurrent queries, continuous ingestion and @@ -108,7 +109,7 @@ terabytes. | Indexing and compaction | when you call it | dedicated indexers | | [Jobs](/basics/jobs) | run in-process, no server record | real jobs: id, status, events, cancel | | [SQL](/sql) | unavailable | Flight SQL endpoint | -| [Functions](/management/function) | unavailable | run on workers | +| [Functions](/management/function) {/* if functions launched */} | unavailable | run on workers | | Access control | object-store permissions | [per-object grants](/governance/authorization) | | Scale bound | your process | the cluster | | What you operate | nothing | a deployment | diff --git a/docs/web/basics/catalog.mdx b/docs/web/basics/catalog.mdx index 497f51ebc..e76d7c19d 100644 --- a/docs/web/basics/catalog.mdx +++ b/docs/web/basics/catalog.mdx @@ -3,7 +3,7 @@ title: "Lance Catalog" sidebarTitle: "Lance Catalog" description: "The naming layer: what objects exist, how they nest, and who may reach them." icon: sitemap -keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view", "function"] +keywords: ["catalog", "namespace", "hierarchy", "governance", "database", "table", "view"] --- A catalog answers one question: **what exists, and what is it called.** It holds @@ -25,12 +25,23 @@ resolves the same names to the same tables. | [Table](/management/table) | Rows and columns, versioned. What most operations act on. | | [Materialized view](/management/materialized-view) | A table whose contents come from a query, computed and stored. | | [View](/management/view) | A saved query. Computed when read, not stored. | -| [Function](/management/function) | Registered code, applied to produce column values. | +| [Function](/management/function) {/* if functions launched */} | Registered code, applied to produce column values. | -Views and functions exist only on an Enterprise deployment that enables them. +Views exist only on an Enterprise deployment that enables them. +So do functions. {/* if functions launched */} Materialized views also exist in the embedded library, and a deployment has to enable them too. +{/* if functions not launched */} +``` +database +└── namespace + ├── table + ├── materialized view (a table, defined by a query) + └── view (a query, not stored) +``` +{/* end if */} +{/* if functions launched */} ``` database └── namespace @@ -39,6 +50,7 @@ database ├── view (a query, not stored) └── function ``` +{/* end if */} A namespace can contain further namespaces, so the hierarchy goes as deep as your organisation needs. A table is reached by its path through it: diff --git a/docs/web/basics/jobs.mdx b/docs/web/basics/jobs.mdx index 21ad7ed2f..10cf2b3b9 100644 --- a/docs/web/basics/jobs.mdx +++ b/docs/web/basics/jobs.mdx @@ -14,8 +14,8 @@ availability: --- Some operations take longer than a request should. Building an index over a -large table, refreshing a computed column, registering a function -- these run -on the server after the call that started them has returned. +large table or refreshing a computed column, for example, runs on the server +after the call that started it has returned. Each is submitted the same way and returns the same handle. @@ -36,7 +36,7 @@ a **job** immediately. | --- | --- | --- | | `create_index` | `create_index_async` | `CREATE INDEX` | | `refresh_column` | `refresh_column_async` | `REFRESH COLUMN` | -| `create_function` | `create_function_async` | `CREATE FUNCTION` | +| `create_function` {/* if functions launched */} | `create_function_async` | `CREATE FUNCTION` | | `drop_table` | `drop_table_async` | `DROP TABLE` | @@ -76,8 +76,8 @@ returns the failed state rather than raising, which is what you want in a polling loop. In Python `wait` takes an optional timeout and returns the operation's typed -result -- a `RefreshColumnJobResult`, a `FunctionVersion`, or `None` for an -operation that has no result: +result -- a `RefreshColumnJobResult`, for example, or `None` for an operation +that has no result: ```python from datetime import timedelta diff --git a/docs/web/basics/lakehouse.mdx b/docs/web/basics/lakehouse.mdx index 7dd631ac3..aa4e154e7 100644 --- a/docs/web/basics/lakehouse.mdx +++ b/docs/web/basics/lakehouse.mdx @@ -54,10 +54,10 @@ operation, not a separate service holding a second copy of your embeddings. See ### Compute {#compute} Deriving new columns and tables from what is already stored: -[computed columns](/feature-engineering/add-computed-column), -[functions](/management/function) that run Python where the data is, and +[computed columns](/feature-engineering/add-computed-column) and [materialized views](/management/materialized-view) that persist a query's result. +[Functions](/management/function) run Python where the data is. {/* if functions launched */} The reason this belongs in the lakehouse rather than in a pipeline outside it is the format. Adding a derived column writes only that column -- the images and diff --git a/docs/web/basics/limits.mdx b/docs/web/basics/limits.mdx index fd9d3878e..c0ad03d80 100644 --- a/docs/web/basics/limits.mdx +++ b/docs/web/basics/limits.mdx @@ -97,7 +97,7 @@ not another. | SQL on a local database | unavailable; `execute_query` raises | | SQL on a deployment | needs its Flight SQL endpoint enabled and reachable | | SQL methods in the TypeScript SDK | none; use a Flight SQL client | -| Function methods in the TypeScript SDK | none; use Python or SQL | +| Function methods in the TypeScript SDK {/* if functions launched */} | none; use Python or SQL | | Vector search in SQL | [depends on the deployment](/basics/sql#no-vector-sql); the query builder always has it | | Geospatial `st_*` functions | behind a build feature LanceDB does not ship | | `tableExists` / `namespaceExists` in TypeScript | not present | diff --git a/docs/web/basics/query.mdx b/docs/web/basics/query.mdx index 1447c4489..11d5d179a 100644 --- a/docs/web/basics/query.mdx +++ b/docs/web/basics/query.mdx @@ -90,7 +90,7 @@ hybrid search has no SQL form. See [Semantic Search](/search). ## When the work outlives the call {#jobs} -Some operations take longer than a query should: building an index, refreshing a -computed column, registering a function. Those return a **job** rather than +Some operations take longer than a query should: building an index, or +refreshing a computed column. Those return a **job** rather than rows, and a job is a different handle with a different shape. See [Jobs](/basics/jobs). diff --git a/docs/web/basics/sdk.mdx b/docs/web/basics/sdk.mdx index 28d094d09..8568107ac 100644 --- a/docs/web/basics/sdk.mdx +++ b/docs/web/basics/sdk.mdx @@ -42,9 +42,9 @@ Not every SDK covers the same surface, though: | Feature engineering, jobs | yes | partly | | Runs against a local database | yes | yes | -Neither SDK aggregates or joins, and TypeScript has no methods for -[functions](/management/function). Reach those with SQL on a deployment -instead -- see [SQL](/basics/sql). +Neither SDK aggregates or joins. +TypeScript has no methods for [functions](/management/function). {/* if functions launched */} +Reach those with SQL on a deployment instead -- see [SQL](/basics/sql). ## REST API {#rest} diff --git a/docs/web/enterprise/index.mdx b/docs/web/enterprise/index.mdx index ed4d232c1..e257f7b84 100644 --- a/docs/web/enterprise/index.mdx +++ b/docs/web/enterprise/index.mdx @@ -21,7 +21,7 @@ What it adds over the embedded form: | | | | --- | --- | | [SQL](/sql) | an Arrow Flight SQL endpoint | -| [Functions](/management/function) | Python running on workers | +| [Functions](/management/function) {/* if functions launched */} | Python running on workers | | [Jobs](/basics/jobs) | server-side work with ids, status and events | | [Authorization](/governance/authorization) | per-object grants | | Scale | a cluster rather than your process | diff --git a/docs/web/feature-engineering/add-computed-column.mdx b/docs/web/feature-engineering/add-computed-column.mdx index afb53eba3..1b57d7839 100644 --- a/docs/web/feature-engineering/add-computed-column.mdx +++ b/docs/web/feature-engineering/add-computed-column.mdx @@ -38,9 +38,12 @@ including that you can travel back to before the column existed. - **SQL expressions in `transforms`** are evaluated *now*, for every row, as part of the call. Good for a small table, or a column you want populated - immediately. A `FunctionApplication` passed through `transforms` instead + immediately. + {/* if functions launched */} + A `FunctionApplication` passed through `transforms` instead declares empty Function-bound columns, which a refresh fills; that needs a deployment with Functions enabled, and the embedded library refuses it. + {/* end if */} - **`computed`** stores the expression and evaluates nothing. The column is committed empty and filled later by [`refresh_column`](/feature-engineering/refresh-column). @@ -81,11 +84,18 @@ reshape later. Dropping the computed column releases the constraint. Enterprise +{/* if functions launched */} When the value cannot be expressed in SQL -- an embedding, a model call, an image decode -- the column is filled by a registered [function](/management/function) instead of an expression. Declaring it looks the same; what differs is what fills it. It needs a deployment with Functions enabled. +{/* end if */} +{/* if functions not launched */} +When the value cannot be expressed in SQL -- an embedding, a model call, an +image decode -- [Geneva](/geneva), the feature engineering package for LanceDB +Enterprise, fills the column with a Python function instead. +{/* end if */} ## What to read next {#next} @@ -93,7 +103,14 @@ enabled. Fill the rows that lack values. + {/* if functions launched */} Fill a column with Python. + {/* end if */} + {/* if functions not launched */} + + Fill a column with Python on LanceDB Enterprise. + + {/* end if */} diff --git a/docs/web/feature-engineering/data-evolution.mdx b/docs/web/feature-engineering/data-evolution.mdx index b568c8c16..9a82a0af9 100644 --- a/docs/web/feature-engineering/data-evolution.mdx +++ b/docs/web/feature-engineering/data-evolution.mdx @@ -108,10 +108,17 @@ exist when that does not fit: - A [computed column](/feature-engineering/add-computed-column) declares the expression and leaves the column empty, so you choose when the work happens. That is the right default on a large table. +{/* if functions launched */} - A [function](/management/function) fills a column with Python, for values SQL cannot express -- an embedding, a model call, an image decode. +{/* end if */} +{/* if functions not launched */} +- On LanceDB Enterprise, [Geneva](/geneva) fills a column with Python, for + values SQL cannot express -- an embedding, a model call, an image decode. +{/* end if */} -On a deployment, both have to be enabled there. +A deployment has to enable computed columns before you use them. +Functions need enabling there too. {/* if functions launched */} ## What to read next {#next} diff --git a/docs/web/feature-engineering/index.mdx b/docs/web/feature-engineering/index.mdx index cc454990d..a2a5ca552 100644 --- a/docs/web/feature-engineering/index.mdx +++ b/docs/web/feature-engineering/index.mdx @@ -3,15 +3,20 @@ title: "Feature engineering" sidebarTitle: "Overview" description: "Derive columns and tables from data already in the lakehouse, without moving it somewhere else to compute." icon: layer-group -keywords: ["feature engineering", "computed columns", "materialized views", "functions", "lakehouse", "jobs"] +keywords: ["feature engineering", "computed columns", "materialized views", "lakehouse", "jobs"] --- Feature engineering happens where the data already is. Rather than exporting a table, computing somewhere else and writing the result back, you declare what a column or a table should contain and let LanceDB fill it. +{/* if functions launched */} [Functions](/management/function) and [materialized views](/feature-engineering/materialized-views) are ordinary +{/* end if */} +{/* if functions not launched */} +[Materialized views](/feature-engineering/materialized-views) are ordinary +{/* end if */} [catalog objects](/basics/catalog), alongside tables and indexes -- created, versioned, and refreshed like anything else, not a separate system bolted alongside. @@ -48,7 +53,7 @@ idempotent and cheap -- but it will surprise you if you expect a spreadsheet. | --- | --- | --- | --- | | [Computed column](/feature-engineering/add-computed-column) | a new column on an existing table | a SQL expression | embedded, and deployments that enable it | | [Materialized view](/feature-engineering/materialized-views) | a new table derived by a query | a refresh | embedded, and deployments that enable it | -| [Function](/management/function) | a computed column filled by Python | registered code | Enterprise deployments that enable it | +| [Function](/management/function) {/* if functions launched */} | a computed column filled by Python | registered code | Enterprise deployments that enable it | **Computed columns** are the common case: a column declared with a SQL expression, filled by [a refresh](/feature-engineering/refresh-column). @@ -57,12 +62,14 @@ expression, filled by [a refresh](/feature-engineering/refresh-column). aggregated view of a source -- and bring it up to date on each refresh. The result is an ordinary table: queryable, indexable, searchable. +{/* if functions launched */} **Functions** run Python where the data is, for work SQL cannot express: embedding a column, calling a model, decoding an image. Registered once, versioned, then applied as a column. -They compose. A materialized view can select computed columns, and a computed -column can be filled by a registered function. +{/* end if */} +They compose. A materialized view can select computed columns. +A computed column can be filled by a registered function. {/* if functions launched */} ## Where the work runs {#where} @@ -83,15 +90,18 @@ from a different process -- the connection does not have to stay open. That relocation is the main thing to understand about Enterprise feature engineering. The API is the same; what changes is that the code runs somewhere -else, which is also why a function must declare the environment it needs. +else. +That is also why a function must declare the environment it needs. {/* if functions launched */} ## Recovering from a failure {#failures} +{/* if functions launched */} A function that raises fails the job. There is no per-row retry or skip policy in the function definition, so a function calling a flaky external service should handle that itself, inside the function body. -The alternative is to let the job fail, fix the cause, and run the refresh +{/* end if */} +When a refresh fails, fix the cause and run the refresh again. The new refresh fills what is still empty or stale. How much of the failed run was kept depends on where it ran (see [Refresh a column](/feature-engineering/refresh-column#failures)), and a retry @@ -110,7 +120,9 @@ only the rows that did not complete. Bring a derived table up to date with its source. + {/* if functions launched */} Register Python to fill a column with. + {/* end if */} diff --git a/docs/web/feature-engineering/refresh-column.mdx b/docs/web/feature-engineering/refresh-column.mdx index 69515e7e9..1c1775849 100644 --- a/docs/web/feature-engineering/refresh-column.mdx +++ b/docs/web/feature-engineering/refresh-column.mdx @@ -69,9 +69,8 @@ REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01' ``` -**Enterprise job type:** `computed_refresh` for a SQL-expression column, -`function_refresh` for one filled by a registered -[function](/management/function). +**Enterprise job type:** `computed_refresh` for a SQL-expression column. +`function_refresh` for one filled by a registered [function](/management/function). {/* if functions launched */} ## As a job {#async} @@ -128,10 +127,12 @@ that were already written -- for example when the record of the inputs they were computed from was not saved -- so do not count on a retry costing only the rows that did not complete. +{/* if functions launched */} For a column filled by a [function](/management/function), the usual cause is the worker environment rather than the data. See [environment mismatches](/management/function#mismatch-symptoms) for the symptoms, which rarely name their cause. +{/* end if */} ## What to read next {#next} diff --git a/docs/web/governance/authorization.mdx b/docs/web/governance/authorization.mdx index 6ad7e23a9..7b4cc9a7f 100644 --- a/docs/web/governance/authorization.mdx +++ b/docs/web/governance/authorization.mdx @@ -44,6 +44,17 @@ not a narrower one. Objects are addressed by kind and path: +{/* if functions not launched */} +``` +system +database:db +namespace:db:ns +table:db:ns$table +view:db:ns$view +secret:db:ns$secret +``` +{/* end if */} +{/* if functions launched */} ``` system database:db @@ -53,6 +64,7 @@ view:db:ns$view function:db:ns$function secret:db:ns$secret ``` +{/* end if */} A nested namespace joins its path with `$`, as in `namespace:db:parent$child` or `table:db:parent$child$table`. @@ -69,7 +81,7 @@ table alone is not enough to traverse to it. | `USAGE` | system | listing jobs and their events, and looking up other identities | | `USAGE` | database | accessing the namespaces inside | | `USAGE` | namespace | accessing the tables inside | -| `USAGE` | function | binding it to a column, and refreshing a column it computes | +| `USAGE` {/* if functions launched */} | function | binding it to a column, and refreshing a column it computes | | `USAGE` | secret | binding and resolving it | | `SELECT` | table, view | reading rows | | `INSERT` | table | appending rows | @@ -81,7 +93,7 @@ table alone is not enough to traverse to it. | `CREATE_TABLE` | namespace | creating a table in it | | `CREATE_MATERIALIZED_VIEW` | namespace | creating a materialized view in it | | `CREATE_VIEW` | namespace | creating a non-materialized view in it | -| `CREATE_FUNCTION` | namespace | registering a function in it | +| `CREATE_FUNCTION` {/* if functions launched */} | namespace | registering a function in it | | `CREATE_SECRET` | namespace | creating a secret in it | | `OPERATE` | system | describing and controlling jobs, managing caches, and reading server configuration | diff --git a/docs/web/maintenance/background-jobs.mdx b/docs/web/maintenance/background-jobs.mdx index 4c236ba64..b064c2d53 100644 --- a/docs/web/maintenance/background-jobs.mdx +++ b/docs/web/maintenance/background-jobs.mdx @@ -64,8 +64,10 @@ export, its events also survive in the exported table. | Job type | What it does | | --- | --- | -| `drop_object` | Deletes a dropped catalog object's content, then clears it from the database's cleanup backlog. Covers namespaces, views, and functions -- everything [`drop_table`](/management/table#drop) does not, since that one fences a directory with its own marker instead. | +| `drop_object` | Deletes a dropped catalog object's content, then clears it from the database's cleanup backlog. Covers namespaces, views and other catalog objects -- everything [`drop_table`](/management/table#drop) does not, since that one fences a directory with its own marker instead. | You trigger this indirectly, by dropping a [namespace](/management/namespace), -[view](/sql/ddl/view#drop), or [function](/management/function#drop) -- the teardown -itself runs as this job rather than inline with the call. +a [view](/sql/ddl/view#drop), +or a [function](/management/function#drop), {/* if functions launched */} +for example -- the teardown itself runs as this job rather than inline with the +call. diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx index 76aeb571a..c29b473cd 100644 --- a/docs/web/management/database.mdx +++ b/docs/web/management/database.mdx @@ -49,7 +49,7 @@ storage and does the work itself. The third talks to a **deployment**. | Table handle | `LanceTable` | `RemoteTable` | | [SQL](/basics/sql) | -- | where enabled | | Server-side [jobs](/basics/jobs) | -- | yes | -| [Functions](/management/function) | -- | where enabled | +| [Functions](/management/function) {/* if functions launched */} | -- | where enabled | | [Authorization](/governance/authorization) | storage permissions | granted per object, where configured | ![](/static/assets/images/overview/understanding-tables.png) diff --git a/docs/web/management/function.mdx b/docs/web/management/function.mdx index 4a24e0ec5..61b46e469 100644 --- a/docs/web/management/function.mdx +++ b/docs/web/management/function.mdx @@ -4,6 +4,7 @@ sidebarTitle: "Function" description: "Register a Python function once and apply it as a column, for work SQL cannot express." icon: code keywords: ["udf", "functions", "create_function", "embedding", "feature engineering"] +launch: functions availability: oss: unavailable enterprise: varies diff --git a/docs/web/management/index.mdx b/docs/web/management/index.mdx index a1a2cde13..8192d8687 100644 --- a/docs/web/management/index.mdx +++ b/docs/web/management/index.mdx @@ -79,6 +79,7 @@ it current. enables them; the SDK methods send the view's SQL query to the deployment. See [Views](/management/view). +{/* if functions launched */} ### Functions {#functions} | | Python | TypeScript | SQL | @@ -92,6 +93,7 @@ enables them; the SDK methods send the view's SQL query to the deployment. See a deployment that enables functions, so there is no local form. The TypeScript SDK has no function methods; use Python or SQL. See [Functions](/management/function). +{/* end if */} ### Jobs {#jobs} diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index c2943e11b..3a4af3c5f 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -225,8 +225,8 @@ also makes shallow clones only. **Enterprise job type:** `drop_table`. On a deployment that drops tables in the background, fences the table's directory with a marker and deletes its content as a job -- see -[Background jobs](/maintenance/background-jobs#teardown) for the namespace, -view, and function equivalent. +[Background jobs](/maintenance/background-jobs#teardown) for the namespace +and view equivalent. ## Dropping {#drop} diff --git a/docs/web/quickstart.mdx b/docs/web/quickstart.mdx index 36001c97e..a507bba96 100644 --- a/docs/web/quickstart.mdx +++ b/docs/web/quickstart.mdx @@ -410,9 +410,8 @@ job = table.refresh_column_async("combat_score") job.wait() ``` -See [Computed columns](/feature-engineering/add-computed-column) for expressions, -and [Functions](/management/function) for filling a column with Python --- an embedding model, a captioner, anything you can register. +See [Computed columns](/feature-engineering/add-computed-column) for expressions. +See [Functions](/management/function) for filling a column with Python. {/* if functions launched */} ## 7\. Store multimodal data {#7-store-multimodal-data} diff --git a/docs/web/sql/ddl/function.mdx b/docs/web/sql/ddl/function.mdx index 39a661fc3..51cff143b 100644 --- a/docs/web/sql/ddl/function.mdx +++ b/docs/web/sql/ddl/function.mdx @@ -4,6 +4,7 @@ sidebarTitle: "Function" description: "CREATE FUNCTION, SHOW FUNCTIONS, and DROP FUNCTION." icon: function keywords: ["create function", "drop function", "show functions", "sql", "udf"] +launch: functions --- Enterprise [SQL runs on a deployment.](/sql) @@ -11,12 +12,17 @@ keywords: ["create function", "drop function", "show functions", "sql", "udf"] ## CREATE FUNCTION {#create} ```sql -CREATE FUNCTION f(x BIGINT) RETURNS BIGINT +CREATE FUNCTION f(x int64) RETURNS int64 LANGUAGE python AS 'def f(x): return x * 2' WITH (pip = 'numpy==1.26.4', python_version = '3.11') ``` +Parameter and result types are Arrow type names, not SQL ones: `int64`, +`float64`, `utf8`, `bool`, `date32`, and lists such as `list` or +`fixed_size_list`. `BIGINT` is refused. A parameter accepts nulls +unless it is declared `NOT NULL`. + The body is Python source. Its dependencies are declared with it and are part of the **immutable function version** -- changing one produces a new version rather than altering the registered one, so a column filled by an earlier @@ -27,7 +33,7 @@ version is unaffected. | `pip` | pip requirements for the worker environment | | `conda` | conda packages instead of pip | | `conda_channels` | channels in priority order; requires `conda` | -| `python_version` | worker Python version, defaulting to yours | +| `python_version` | worker Python version; `3.12` when omitted, whatever Python you send the statement from | | `gpu` | `'true'` when every execution needs a GPU | **Errors and limits** diff --git a/docs/web/sql/ddl/namespace.mdx b/docs/web/sql/ddl/namespace.mdx index daedf9720..3ec08469b 100644 --- a/docs/web/sql/ddl/namespace.mdx +++ b/docs/web/sql/ddl/namespace.mdx @@ -36,7 +36,7 @@ DROP SCHEMA `a$b` RESTRICT `CASCADE` is rejected: **"DROP SCHEMA CASCADE is not supported; drop the -objects inside first."** A namespace's tables, views and functions must be +objects inside first."** A namespace's tables, views and other objects must be dropped explicitly before the namespace itself can go. `RESTRICT` -- the default -- states that requirement rather than changing it. diff --git a/docs/web/sql/ddl/table.mdx b/docs/web/sql/ddl/table.mdx index bd64458be..759bb64b9 100644 --- a/docs/web/sql/ddl/table.mdx +++ b/docs/web/sql/ddl/table.mdx @@ -30,8 +30,8 @@ The table's columns and types, without a plan -- the SQL counterpart of ## ALTER TABLE {#alter} -`ALTER TABLE ... ADD COLUMN` covers three different things, distinguished by -what follows the column name. +What `ALTER TABLE ... ADD COLUMN` does depends on what follows the column +name. ### A plain column {#plain-column} @@ -52,6 +52,7 @@ The column is committed empty and the expression is stored, not evaluated -- declaring one on a billion-row table costs what declaring one on an empty table costs. [`REFRESH COLUMN`](/sql/dml/refresh) fills it. +{/* if functions launched */} ### A function column {#function-column} ```sql @@ -86,6 +87,7 @@ ALTER TABLE t ADD COLUMN c AS f(x) WITH (num_workers = '2') `WITH` on a plain computed column accepts only `version`. Resource options such as `num_workers` apply to function columns, where there are workers to size. +{/* end if */} ### Table properties {#properties} diff --git a/docs/web/sql/dml/refresh.mdx b/docs/web/sql/dml/refresh.mdx index 90ba61c8f..fa1c87eff 100644 --- a/docs/web/sql/dml/refresh.mdx +++ b/docs/web/sql/dml/refresh.mdx @@ -52,11 +52,13 @@ REFRESH COLUMN embedding ON documents `num_workers` sets a count; the budget options size each worker. They are alternatives rather than a pair. +{/* if functions launched */} `error_policy` and `num_workers` are refused for a column filled by a registered [function](/management/function) with its own resource budget -- use the CPU, memory and GPU options there. +{/* end if */} ## REFRESH MATERIALIZED VIEW {#refresh-materialized-view} diff --git a/docs/web/sql/index.mdx b/docs/web/sql/index.mdx index 6b147d0a7..5cf2f7f5c 100644 --- a/docs/web/sql/index.mdx +++ b/docs/web/sql/index.mdx @@ -101,7 +101,7 @@ operations that have no SQL form, such as a merge insert -- see | [View](/sql/ddl/view) | `CREATE VIEW`, `DROP VIEW` | | [Materialized view](/sql/ddl/materialized-view) | `CREATE MATERIALIZED VIEW`, `SHOW MATERIALIZED VIEWS` | | [Index](/sql/ddl/index) | `CREATE INDEX`, `DROP INDEX` | -| [Function](/sql/ddl/function) | `CREATE FUNCTION`, `SHOW FUNCTIONS`, `DROP FUNCTION` | +| [Function](/sql/ddl/function) {/* if functions launched */} | `CREATE FUNCTION`, `SHOW FUNCTIONS`, `DROP FUNCTION` | | [Job](/sql/ddl/job) | `SHOW JOBS`, `SHOW ERRORS`, `PAUSE`/`RESUME`/`CANCEL JOB` | ### DML {#dml}