## Summary
Adds an optional `max_concurrency` setting to Function registration and
explicit refresh:
- A registered Function version can define the default maximum number of
user-code calls in flight for one logical execution.
- A refresh can override that default for that execution only.
- The value must be a positive `u32` and is independent from
`resource_budget`; it is not an account-wide quota.
- Omitting the option preserves the previous request payload. Local
embedded refreshes reject this Enterprise-only option.
Rust, Python sync/async, PyO3, and the existing Node refresh surface
forward the option. Node has no Function registration surface today, so
no registration API is added there. The generated TypeDoc pages are
included for the new public option.
## Validation
The native-tested source covers Function version decoding, registration
validation and serialization, the Flight SQL registration ticket and
returned version, remote refresh forwarding, local rejection, and Node's
numeric boundary before conversion. Exact stage results are Rust `1+3`
plus four remote cases at `1` each; Python `14/1/49/24`; Node `57`
remote and `1` local rejection. Ruff, standard repository Pyright, and
Biome passed.
The tested Python extension was
`21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the
tested Node addon and generated native declaration were
`f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and
`4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The
final 24-file diff preserves the 22 tested
product/test/fixture/source-doc blobs and adds only two generated
TypeDoc pages.
The client registers Functions, but the decorator that packages one was
called `udf` and produced a `UdfDefinition`. Rename them to `function`
and `FunctionDefinition`, and say "function" instead of "UDF" in the
docstrings, error messages and docs around them.
`udf` and `UdfDefinition` stay as deprecated aliases that emit a
`DeprecationWarning`. The source packager strips either spelling of the
decorator, so `@udf` and `@function` produce the same artifact and
digest, and the wire contract is unchanged. DataFusion and Lance UDF
types, and Geneva's `@udf` in the integration docs, are unrelated and
untouched.
Labels early access features and hides features we aren't documenting
yet.
**Early access label:** Functions (4 pages, plus the GPU refresh,
checkpointing and adaptive execution pages), Secrets (3 pages), Tiered
Index Cache, Blob V2, and merge insert, which is labelled for Enterprise
only. Two sidebar titles are shorter so the tag fits.
**Hidden:** branching, JSON columns and geospatial.
- The JSON functions page and the agent branch-experiments tutorial go
in `.mintignore`.
- `tables/branching` stays reachable but out of the nav, as #4859 left
it, because SDK error messages link to it.
- Other pages' mentions of these features become draft comments, so
they're easy to restore.
**Secrets page:** drops references to manual KMS setup, which the
control plane handles. It keeps the encryption-at-rest facts, states the
Azure value-size limit inline, and points a 501 error to support.
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:
**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.
**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.
**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.
**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.
**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.
Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Python UDF registration could not set `batch_size` or
`row_timeout_seconds`, and the Rust registration model discarded these
fields before submitting SQL. Expose both optional controls and preserve
them through registration, with invalid values rejected and unchanged
canonical bytes when the controls are omitted.
The timeout applies to each Python call: one row for scalar Functions
and one batch for vectorized Functions. Initialization keeps its
existing service deadline.
Validation at `f58e55c535a7df310db14d691b329402ca4baaf5`: 6 Rust
registration-contract tests, 2 outbound-SQL tests, and all 103 Python
slice2 tests passed after rebuilding the native extension. Rust
formatting and the repository Python CI checks (Ruff 0.9.9, python/
scope) passed. A real REST + Flight SQL server integration also passed
using the rebuilt SDK: `batch_size=2` limits observed calls in a direct
SELECT over 8 rows, initial materialized-view creation, and
append/refresh to 12 rows. This CPU host test verifies SDK-to-runtime
passthrough; it does not exercise GPU initialization.
Refs ENT-3246. The original GPU initialization and video-decoding
scenario still needs verification on a current deployment; this SDK
change does not close that investigation.
The additional repository-root style checks report existing
docs/web-tests findings (nine format files and 48 lint diagnostics); all
affected files are unchanged from main. These are outside the Python CI
job scope.
Rust compatibility: callers constructing `FunctionRegistrationRequest`
must initialize the two new optional fields (`None` preserves defaults);
the request no longer implements `Eq`.
The final head `798f463c9f68ab73e28c9a1ff06b5e4ee3e5d01f` adds
creation-guide documentation and integrates main’s namespace
repr/optional OCI embedding changes. The Function registration
implementation is unchanged. All 103 Python slice2 tests passed again on
EC2 using the unchanged native extension; Rust and server-bridge
evidence remains applicable.
A Function that produces several blobs per row, such as frames from a video, could not declare `pa.list_(lancedb.blob("item"))` as its output: signatures rejected Blob v2 under any collection.
Outputs now accept Blob v2 inside list and large-list fields; the server stores each element as a Blob v2 value. Inputs, fixed-size lists and maps still reject it (lance stores Blob v2 under neither), and the `udf()` docstring now says where Blob v2 may appear.
## Summary
- Add sync/async `DBConnection.list_gpu_classes()` returning typed
`GpuClass` records for `GET /v1/gpu/classes`. Records include the class
name, Kubernetes resource, GPU model and memory, per-worker GPU maximum,
and worker CPU/memory/ephemeral-storage limits.
- Add `udf(gpu_class=...)` for immutable Function-level GPU placement.
It requires `gpu=True`; Function versions expose their registered
`gpu_class`.
- Add `gpu_class` and `gpu_count` to `refresh_column` /
`refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as
top-level `backfill_column` fields and validated before any request.
`gpu_count` is the aggregate GPU budget across Function workers.
- Validate GPU class names with the server lowercase 63-character
DNS-label rule.
- Document the discovery-to-registration flow, configured hardware
versus free capacity, System Usage permission, and per-refresh GPU
placement in the API and web guides.
Older servers without `/v1/gpu/classes` produce a clear
`NotImplementedError` instead of a bare 404.
Part of ENT-3470. The client follows the in-flight Sophon contract on
`xuanwo/ent-3470-fix` for GPU class discovery and backfill placement
options.
## Breaking change
`PythonRuntimeSpec::PythonV2` adds `gpu_class`, and
`RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers
constructing either value with all fields explicitly must supply the new
fields.
## Validation
At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python
extension build, pyright, ruff, and Python Function/runtime, discovery,
refresh placement, and refresh regressions (147 passed, 6 skipped).
At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright,
Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed).
The final commit only updates documentation and Rust doc comments.
## Summary
- Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause,
and
`dedupe()`/`duplicate_pairs()` (verified against current sophon source;
these were recently renamed from
`vector_dedup`/`vector_duplicate_pairs`).
Fix the blob-column default-projection claim and add the previously
undocumented `blob_descriptor()`.
- Enrich the gated Functions draft (still `.mintignore`'d, pre-release):
chained Functions over blob outputs, GPU-produced blobs, Secrets,
multi-input/output binding, `code=[module]` packaging.
- Upgrade the materialized-view and computed-column auto-refresh drafts
with
precise, verified catch-up semantics, and add the missing `ALTER
MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the
automatic-refresh scheduler is still dev/staging-only).
- Thread one running example (video frames → GPU extract/caption → dedup
→
auto-refreshing materialized view) through Functions, materialized
views,
and the LeRobotDataset integration guide instead of a separate tutorial.
- Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout
(52 files); activate the existing availability-comparison mechanism on
`basics/offerings.mdx` for a full feature-by-feature comparison table.
Xuanwo
Wyatt Alt
Brendan Clement
Jack YeandClaude Sonnet 5