mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
3bfa02337596bab28fc2aa6e2d2a19ef76cb0de1
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Brendan ClementandClaude
|
e2501b0dab |
docs: update example names (#4887)
Use neutral placeholder names and values in a few code examples in the docs and Python docstrings. Text-only; no behavior change. --- _Generated by [Claude Code](https://claude.ai/code/session_019iqZdtD3ZMnvTMeoNsULgX)_ --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
 Jack YeandClaude Sonnet 5
|
5e38120bdc |
docs: add storage integration SDK examples, abstract RTE write, simplify cache pages (#4865)
## Summary - **Define Storage Integration**: un-gates the Python (and Rust, for S3) `create_storage_integration` examples, drafted against #4618 and held back until it shipped -- it's now in v0.42.0-beta.0+, and the signature matches the drafts exactly. - **Naming rules**: drops the `Branch` row and its "Learn more" link, consistent with de-emphasizing branches elsewhere in this nav. - **Distributed RTE Write**: removes the REST path, the `__lance_mem_wal` index name, and "WAL node" terminology in favor of an abstract framing -- a query node routes writes to RTE nodes, each of which can host shards from many different tables. Both diagram variants are updated to match. - **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop Helm-specific and exact-percentage detail (disk-budget math, `startup_prewarm_block_query_ready`/`healthPort`/failure-policy table) in favor of a short feature → architecture → why-it-matters shape, each closing with a contact-sales pointer. Data Page Cache's on-demand prewarm section is removed outright. ## Note I wasn't able to get a clean visual render of the updated Distributed RTE Write diagram in local preview -- Mermaid isn't rendering in headless Chrome in this environment for *any* page right now, confirmed against an unmodified reference page showing the identical symptom. The diagram source follows the exact pattern already shipped and working in the 21 diagrams from the earlier brand-theming pass, and `mint validate`/`mint broken-links` both pass clean. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
 Yang Cen
|
69ac2158a1 |
fix: preserve take_row_ids request order (#4826)
`take_row_ids` used an `IN` filter, so the backend could reorder rows and collapse duplicate IDs. That can silently misalign training rows with labels or sample weights. Restore the requested order and duplicate occurrences in the shared Rust take path for native and remote tables, and retain that information through Python query serialization. Missing or deleted IDs are omitted. Explicit Rust `order_by` overrides request order, and pagination applies after restoring occurrences. Empty projections and `_rowid` projection aliases retain their output schema. RTE tables reject unsupported takes with the existing base-table guidance before checking the internal row-ID projection used for ordering. An explicit `use_rte(False)` reads the base table with the same requested order and occurrences. A pre-existing fixture defect also blocked CI: `test_table_add_in_threadpool` responded without reading uploads, allowing a connection reset during response decoding. This PR includes a fix confined to that test's handler: consume each request body before responding and verify one create row plus ten inserted rows. This was not a regression in the take implementation. Production write retries, concurrency and HTTP client code are unchanged. The selective sampling guide scopes the row-ID guarantees to SDK releases containing this fix, preserves the distinct offset-take semantics, and explains that ordered takes collect and combine matching rows before output batching. It recommends smaller ID requests and narrow projections because `max_batch_length` does not cap working memory. This uses the existing REST query protocol and requires a new SDK release; no deployed service was changed. Validation: - Docs follow-up: the changed page compiles with the Mint 4.2.888 MDX compiler; Typos 1.26.0 passes and Vale 3.24.0 has no findings on changed lines. Full-site Mint validation reports four existing snippet-import warnings on the reranking/vector-search pages, identical with the original page restored in a baseline comparison. - The applied threadpool test passed locally after the fixture change. The identical candidate previously passed 10 trials with the ordinary reader and 10 with an unbuffered reader; each trial verifies all uploaded rows. Those checks used macOS arm64/Python 3.14/default features, not the full failing CI environment. The normal new-head macOS Python 3.13/experimental-functions job remains the required platform check. - Python package Ruff format/check passed. Repository-root Ruff still reports 59 existing errors and six formatting differences in unchanged `docs/web-tests` files; none are in changed files. - At `a569d3f0`, the RTE diagnostic regression was reproduced and fixed; 5 targeted cases and strict workspace Clippy passed. The fixture-only follow-up changes no production Rust/Python code, so that evidence is reused. - Earlier validation at `6bcc0878`: 439 Rust query/remote-table tests and 24 Python take/getitems tests passed; the focused 49-test suite also covered offset dataloaders. A local 9,216-row, four-fragment reproduction returned 200 seeded random batches of 64 rows with zero ordering or label-alignment mismatches. The ordering and projection implementation is unchanged by the follow-ups. - No live Enterprise run was performed. Fixes [ENT-3469](https://linear.app/lancedb/issue/ENT-3469). |
||
 Wyatt Alt
|
b5c5ede39f |
feat(python)!: rename the @udf decorator to @function (#4837)
The client registers Functions, but the decorator that packages one was called `udf` and produced a `UdfDefinition`. Rename them to `function` and `FunctionDefinition`, and say "function" instead of "UDF" in the docstrings, error messages and docs around them. `udf` and `UdfDefinition` stay as deprecated aliases that emit a `DeprecationWarning`. The source packager strips either spelling of the decorator, so `@udf` and `@function` produce the same artifact and digest, and the wire contract is unchanged. DataFusion and Lance UDF types, and Geneva's `@udf` in the integration docs, are unrelated and untouched. |
||
|
Brendan Clement
|
1db0d3ca5f |
docs: label early access features and hide ones not ready to document (#4855)
Labels early access features and hides features we aren't documenting yet. **Early access label:** Functions (4 pages, plus the GPU refresh, checkpointing and adaptive execution pages), Secrets (3 pages), Tiered Index Cache, Blob V2, and merge insert, which is labelled for Enterprise only. Two sidebar titles are shorter so the tag fits. **Hidden:** branching, JSON columns and geospatial. - The JSON functions page and the agent branch-experiments tutorial go in `.mintignore`. - `tables/branching` stays reachable but out of the nav, as #4859 left it, because SDK error messages link to it. - Other pages' mentions of these features become draft comments, so they're easy to restore. **Secrets page:** drops references to manual KMS setup, which the control plane handles. It keeps the encryption-at-rest facts, states the Azure value-size limit inline, and points a 501 error to support. |
||
|
Jack YeandClaude Sonnet 5
|
d0b0efcd47 |
style: increase Mermaid diagram font size and tone down dark-mode accent borders (#4828)
## Summary Follow-up to the Mermaid brand-theming PR, addressing review feedback: - Diagram text was hard to read at the previous 15px font size; bumped to 18px, light and dark alike. - The dark-mode orange accent border was brighter and more saturated than its fill or the default border, reading as too loud against the dark background. It now reuses the same muted orange already used for the light-mode accent border, which was fine as is. ## Companion change lancedb/sophon#9719 makes the same two changes across its own diagrams. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Jack YeandClaude Sonnet 5
|
dfbf682cb5 |
style: re-theme Mermaid diagrams to the brand palette, light and dark (#4822)
## Summary Every Mermaid diagram now renders in the brand palette, with separate light and dark variants that switch with the site's theme toggle. Default node styling stays neutral; one accent color marks each diagram's actual subject, chosen per diagram. A couple of diagrams changed orientation where their old layout was too wide to read comfortably. Companion change in lancedb/sophon uses the same colors and pattern. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Jack YeandClaude Sonnet 5
|
de95ed3f6d |
docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
 Xuanwo
|
ad70597435 |
feat(python): expose Function batch size and row timeout controls (#4807)
Python UDF registration could not set `batch_size` or `row_timeout_seconds`, and the Rust registration model discarded these fields before submitting SQL. Expose both optional controls and preserve them through registration, with invalid values rejected and unchanged canonical bytes when the controls are omitted. The timeout applies to each Python call: one row for scalar Functions and one batch for vectorized Functions. Initialization keeps its existing service deadline. Validation at `f58e55c535a7df310db14d691b329402ca4baaf5`: 6 Rust registration-contract tests, 2 outbound-SQL tests, and all 103 Python slice2 tests passed after rebuilding the native extension. Rust formatting and the repository Python CI checks (Ruff 0.9.9, python/ scope) passed. A real REST + Flight SQL server integration also passed using the rebuilt SDK: `batch_size=2` limits observed calls in a direct SELECT over 8 rows, initial materialized-view creation, and append/refresh to 12 rows. This CPU host test verifies SDK-to-runtime passthrough; it does not exercise GPU initialization. Refs ENT-3246. The original GPU initialization and video-decoding scenario still needs verification on a current deployment; this SDK change does not close that investigation. The additional repository-root style checks report existing docs/web-tests findings (nine format files and 48 lint diagnostics); all affected files are unchanged from main. These are outside the Python CI job scope. Rust compatibility: callers constructing `FunctionRegistrationRequest` must initialize the two new optional fields (`None` preserves defaults); the request no longer implements `Eq`. The final head `798f463c9f68ab73e28c9a1ff06b5e4ee3e5d01f` adds creation-guide documentation and integrates main’s namespace repr/optional OCI embedding changes. The Function registration implementation is unchanged. All 103 Python slice2 tests passed again on EC2 using the unchanged native extension; Rust and server-bridge evidence remains applicable. |
||
|
Wyatt Alt
|
8ee7698545 |
feat(python): allow Blob v2 under a list in Function outputs
A Function that produces several blobs per row, such as frames from a video, could not declare `pa.list_(lancedb.blob("item"))` as its output: signatures rejected Blob v2 under any collection.
Outputs now accept Blob v2 inside list and large-list fields; the server stores each element as a Blob v2 value. Inputs, fixed-size lists and maps still reject it (lance stores Blob v2 under neither), and the `udf()` docstring now says where Blob v2 may appear.
|
||
|
Xuanwo
|
63aa06ef7d |
feat(python): expose Function GPU classes and placement (#4791)
## Summary - Add sync/async `DBConnection.list_gpu_classes()` returning typed `GpuClass` records for `GET /v1/gpu/classes`. Records include the class name, Kubernetes resource, GPU model and memory, per-worker GPU maximum, and worker CPU/memory/ephemeral-storage limits. - Add `udf(gpu_class=...)` for immutable Function-level GPU placement. It requires `gpu=True`; Function versions expose their registered `gpu_class`. - Add `gpu_class` and `gpu_count` to `refresh_column` / `refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as top-level `backfill_column` fields and validated before any request. `gpu_count` is the aggregate GPU budget across Function workers. - Validate GPU class names with the server lowercase 63-character DNS-label rule. - Document the discovery-to-registration flow, configured hardware versus free capacity, System Usage permission, and per-refresh GPU placement in the API and web guides. Older servers without `/v1/gpu/classes` produce a clear `NotImplementedError` instead of a bare 404. Part of ENT-3470. The client follows the in-flight Sophon contract on `xuanwo/ent-3470-fix` for GPU class discovery and backfill placement options. ## Breaking change `PythonRuntimeSpec::PythonV2` adds `gpu_class`, and `RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers constructing either value with all fields explicitly must supply the new fields. ## Validation At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python extension build, pyright, ruff, and Python Function/runtime, discovery, refresh placement, and refresh regressions (147 passed, 6 skipped). At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright, Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed). The final commit only updates documentation and Rust doc comments. |
||
|
Brendan ClementandClaude Opus 5.5
|
ca343a5c06 |
docs: fix table, accuracy and structure problems found in the pre-launch review (#4745)
## Summary Fixes the problems a page-by-page review of the docs found before launch, after checking each one against the current source and, where noted, the code. The matching fixes for the dataset and Geneva pages, which live in `lancedb/docs`, are in lancedb/docs#402. **Tables** - `integrations/embedding/cohere`: each input-type description was split across two table rows and wrapped in literal quotes (`"`search_document`"`). Now one row per type. - `integrations/embedding/gemini`: removes the same stray quotes from the task-type table. - `integrations/ai/langchain`: the MMR search table's `kwargs` row was missing a cell, so its description sat under Type. - Six reranker pages (answerdotai, cohere, colbert, jina, openai, voyageai): closes the backtick before `_relevance_score`. - `indexing/vector-index`: finishes a cell cut off at "with scalar". - `sql/dql/dedupe`: labels the empty header cell in the resolution table. **Corrections** - `curation/sampling`: sampling works in LanceDB Embedded through the Python SDK, so the page is available in both, with the SQL sections marked Enterprise. - `feature-engineering/index`, `add-computed-column`: automatic refresh is on by default for each computed column and takes effect once a deployment enables its scheduler. `refresh-column` already said so; these two implied it was off. - `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates `array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also merges scalar-index's two "nested fields" sections into one. - `integrations/embedding/ibm`: lists the four current models from `CURRENT_MODELS` (it listed a deprecated one) and explains that the default stays a legacy ID so existing tables reload with their model. - `integrations/reranking/openai`: the class is `OpenaiReranker`, not `OpenAI()`. - `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit` throughout, each step runs one command, each step passes the file it writes to the next, and `save-as` uses `-f`/`-o`. - `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the pipeline reads, and the run command runs the file the page creates. - `search/vector-search`: the multivector walkthrough connected to Enterprise in step 1 and a local path in step 2. It now runs locally throughout, with a note on connecting to Enterprise, and is renamed so it doesn't read as a duplicate of the short section above. **Structure and wording** - `integrations/embedding/nvidia`: adds headings, and drops test-suite details aimed at contributors. - `integrations/ai/genkit`: makes the retrieval test a real heading and describes the query step. - `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a `<Note>`. - `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading, not bold text. - `curation/windowing`: sidebar title matches the curation overview ("Rank and Deduplicate"). - `deployment/storage`, `basics/agentic-skills`: fixes a garbled sentence and a stale reference to the quickstart. **Checked:** no new Vale alerts on the changed pages. Assembled together with lancedb/docs#402, the site passes `mint validate` and `mint broken-links --check-anchors --check-redirects`. Every changed page was reviewed in a local preview. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Jack Ye
|
ddb318567a |
docs: complete SQL reference, enrich Functions drafts, rename OSS to Embedded (#4732)
## Summary - Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause, and `dedupe()`/`duplicate_pairs()` (verified against current sophon source; these were recently renamed from `vector_dedup`/`vector_duplicate_pairs`). Fix the blob-column default-projection claim and add the previously undocumented `blob_descriptor()`. - Enrich the gated Functions draft (still `.mintignore`'d, pre-release): chained Functions over blob outputs, GPU-produced blobs, Secrets, multi-input/output binding, `code=[module]` packaging. - Upgrade the materialized-view and computed-column auto-refresh drafts with precise, verified catch-up semantics, and add the missing `ALTER MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the automatic-refresh scheduler is still dev/staging-only). - Thread one running example (video frames → GPU extract/caption → dedup → auto-refreshing materialized view) through Functions, materialized views, and the LeRobotDataset integration guide instead of a separate tutorial. - Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout (52 files); activate the existing availability-comparison mechanism on `basics/offerings.mdx` for a full feature-by-feature comparison table. |
||
 Will JonesandClaude Opus 5.5
|
4d7a64712f |
docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Brendan Clement
|
06aa526ae1 |
docs: correct data workflows, SQL and operations (#4534)
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The curation, SQL and operations pages contain outdated claims and examples that do not reliably work as written. This corrects those pages against the reviewed SDK and local server evidence, including prepared statements, background jobs, cleanup and monitoring. Integration guides are grouped with their examples in #4533. Function and Secret-object documentation stays unpublished until its release. The two Function pages are listed in `.mintignore` and left out of the navigation, and the sections about them on other pages are MDX comments. Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after #4533 is squash-merged. Local assembly, Mint validation and link checks pass, and the assembled site shows no Function or Secret-object content. A local release rehearsal that uncomments the drafts and restores the navigation also passes. Hold publication until this PR and lancedb/sophon#8688 are on `main`. |
||
|
Jack YeandBrendan Clement
|
47b7babc08 |
docs: bring the open-source documentation into this repository (#4122)
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (
|