mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
e6b04c5da7822c48bddb7f1a7982ffdaf00a53df
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Jack YeandClaude Sonnet 5
|
d0b0efcd47 |
style: increase Mermaid diagram font size and tone down dark-mode accent borders (#4828)
## Summary Follow-up to the Mermaid brand-theming PR, addressing review feedback: - Diagram text was hard to read at the previous 15px font size; bumped to 18px, light and dark alike. - The dark-mode orange accent border was brighter and more saturated than its fill or the default border, reading as too loud against the dark background. It now reuses the same muted orange already used for the light-mode accent border, which was fine as is. ## Companion change lancedb/sophon#9719 makes the same two changes across its own diagrams. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Jack YeandClaude Sonnet 5
|
dfbf682cb5 |
style: re-theme Mermaid diagrams to the brand palette, light and dark (#4822)
## Summary Every Mermaid diagram now renders in the brand palette, with separate light and dark variants that switch with the site's theme toggle. Default node styling stays neutral; one accent color marks each diagram's actual subject, chosen per diagram. A couple of diagrams changed orientation where their old layout was too wide to read comfortably. Companion change in lancedb/sophon uses the same colors and pattern. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Jack YeandClaude Sonnet 5
|
de95ed3f6d |
docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
 Xuanwo
|
ad70597435 |
feat(python): expose Function batch size and row timeout controls (#4807)
Python UDF registration could not set `batch_size` or `row_timeout_seconds`, and the Rust registration model discarded these fields before submitting SQL. Expose both optional controls and preserve them through registration, with invalid values rejected and unchanged canonical bytes when the controls are omitted. The timeout applies to each Python call: one row for scalar Functions and one batch for vectorized Functions. Initialization keeps its existing service deadline. Validation at `f58e55c535a7df310db14d691b329402ca4baaf5`: 6 Rust registration-contract tests, 2 outbound-SQL tests, and all 103 Python slice2 tests passed after rebuilding the native extension. Rust formatting and the repository Python CI checks (Ruff 0.9.9, python/ scope) passed. A real REST + Flight SQL server integration also passed using the rebuilt SDK: `batch_size=2` limits observed calls in a direct SELECT over 8 rows, initial materialized-view creation, and append/refresh to 12 rows. This CPU host test verifies SDK-to-runtime passthrough; it does not exercise GPU initialization. Refs ENT-3246. The original GPU initialization and video-decoding scenario still needs verification on a current deployment; this SDK change does not close that investigation. The additional repository-root style checks report existing docs/web-tests findings (nine format files and 48 lint diagnostics); all affected files are unchanged from main. These are outside the Python CI job scope. Rust compatibility: callers constructing `FunctionRegistrationRequest` must initialize the two new optional fields (`None` preserves defaults); the request no longer implements `Eq`. The final head `798f463c9f68ab73e28c9a1ff06b5e4ee3e5d01f` adds creation-guide documentation and integrates main’s namespace repr/optional OCI embedding changes. The Function registration implementation is unchanged. All 103 Python slice2 tests passed again on EC2 using the unchanged native extension; Rust and server-bridge evidence remains applicable. |
||
 Wyatt Alt
|
8ee7698545 |
feat(python): allow Blob v2 under a list in Function outputs
A Function that produces several blobs per row, such as frames from a video, could not declare `pa.list_(lancedb.blob("item"))` as its output: signatures rejected Blob v2 under any collection.
Outputs now accept Blob v2 inside list and large-list fields; the server stores each element as a Blob v2 value. Inputs, fixed-size lists and maps still reject it (lance stores Blob v2 under neither), and the `udf()` docstring now says where Blob v2 may appear.
|
||
|
Xuanwo
|
63aa06ef7d |
feat(python): expose Function GPU classes and placement (#4791)
## Summary - Add sync/async `DBConnection.list_gpu_classes()` returning typed `GpuClass` records for `GET /v1/gpu/classes`. Records include the class name, Kubernetes resource, GPU model and memory, per-worker GPU maximum, and worker CPU/memory/ephemeral-storage limits. - Add `udf(gpu_class=...)` for immutable Function-level GPU placement. It requires `gpu=True`; Function versions expose their registered `gpu_class`. - Add `gpu_class` and `gpu_count` to `refresh_column` / `refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as top-level `backfill_column` fields and validated before any request. `gpu_count` is the aggregate GPU budget across Function workers. - Validate GPU class names with the server lowercase 63-character DNS-label rule. - Document the discovery-to-registration flow, configured hardware versus free capacity, System Usage permission, and per-refresh GPU placement in the API and web guides. Older servers without `/v1/gpu/classes` produce a clear `NotImplementedError` instead of a bare 404. Part of ENT-3470. The client follows the in-flight Sophon contract on `xuanwo/ent-3470-fix` for GPU class discovery and backfill placement options. ## Breaking change `PythonRuntimeSpec::PythonV2` adds `gpu_class`, and `RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers constructing either value with all fields explicitly must supply the new fields. ## Validation At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python extension build, pyright, ruff, and Python Function/runtime, discovery, refresh placement, and refresh regressions (147 passed, 6 skipped). At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright, Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed). The final commit only updates documentation and Rust doc comments. |
||
 Brendan ClementandClaude Opus 5.5
|
ca343a5c06 |
docs: fix table, accuracy and structure problems found in the pre-launch review (#4745)
## Summary Fixes the problems a page-by-page review of the docs found before launch, after checking each one against the current source and, where noted, the code. The matching fixes for the dataset and Geneva pages, which live in `lancedb/docs`, are in lancedb/docs#402. **Tables** - `integrations/embedding/cohere`: each input-type description was split across two table rows and wrapped in literal quotes (`"`search_document`"`). Now one row per type. - `integrations/embedding/gemini`: removes the same stray quotes from the task-type table. - `integrations/ai/langchain`: the MMR search table's `kwargs` row was missing a cell, so its description sat under Type. - Six reranker pages (answerdotai, cohere, colbert, jina, openai, voyageai): closes the backtick before `_relevance_score`. - `indexing/vector-index`: finishes a cell cut off at "with scalar". - `sql/dql/dedupe`: labels the empty header cell in the resolution table. **Corrections** - `curation/sampling`: sampling works in LanceDB Embedded through the Python SDK, so the page is available in both, with the SQL sections marked Enterprise. - `feature-engineering/index`, `add-computed-column`: automatic refresh is on by default for each computed column and takes effect once a deployment enables its scheduler. `refresh-column` already said so; these two implied it was off. - `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates `array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also merges scalar-index's two "nested fields" sections into one. - `integrations/embedding/ibm`: lists the four current models from `CURRENT_MODELS` (it listed a deprecated one) and explains that the default stays a legacy ID so existing tables reload with their model. - `integrations/reranking/openai`: the class is `OpenaiReranker`, not `OpenAI()`. - `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit` throughout, each step runs one command, each step passes the file it writes to the next, and `save-as` uses `-f`/`-o`. - `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the pipeline reads, and the run command runs the file the page creates. - `search/vector-search`: the multivector walkthrough connected to Enterprise in step 1 and a local path in step 2. It now runs locally throughout, with a note on connecting to Enterprise, and is renamed so it doesn't read as a duplicate of the short section above. **Structure and wording** - `integrations/embedding/nvidia`: adds headings, and drops test-suite details aimed at contributors. - `integrations/ai/genkit`: makes the retrieval test a real heading and describes the query step. - `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a `<Note>`. - `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading, not bold text. - `curation/windowing`: sidebar title matches the curation overview ("Rank and Deduplicate"). - `deployment/storage`, `basics/agentic-skills`: fixes a garbled sentence and a stale reference to the quickstart. **Checked:** no new Vale alerts on the changed pages. Assembled together with lancedb/docs#402, the site passes `mint validate` and `mint broken-links --check-anchors --check-redirects`. Every changed page was reviewed in a local preview. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Jack Ye
|
ddb318567a |
docs: complete SQL reference, enrich Functions drafts, rename OSS to Embedded (#4732)
## Summary - Complete the SQL reference: `ATTACH DATABASE`, the `CHANGES` clause, and `dedupe()`/`duplicate_pairs()` (verified against current sophon source; these were recently renamed from `vector_dedup`/`vector_duplicate_pairs`). Fix the blob-column default-projection claim and add the previously undocumented `blob_descriptor()`. - Enrich the gated Functions draft (still `.mintignore`'d, pre-release): chained Functions over blob outputs, GPU-produced blobs, Secrets, multi-input/output binding, `code=[module]` packaging. - Upgrade the materialized-view and computed-column auto-refresh drafts with precise, verified catch-up semantics, and add the missing `ALTER MATERIALIZED VIEW` / `CREATE ... WITH(...)` sections (also gated — the automatic-refresh scheduler is still dev/staging-only). - Thread one running example (video frames → GPU extract/caption → dedup → auto-refreshing materialized view) through Functions, materialized views, and the LeRobotDataset integration guide instead of a separate tutorial. - Rename "LanceDB OSS"/"OSS" to "LanceDB Embedded"/"Embedded" throughout (52 files); activate the existing availability-comparison mechanism on `basics/offerings.mdx` for a full feature-by-feature comparison table. |
||
 Will JonesandClaude Opus 5.5
|
4d7a64712f |
docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Brendan Clement
|
06aa526ae1 |
docs: correct data workflows, SQL and operations (#4534)
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The curation, SQL and operations pages contain outdated claims and examples that do not reliably work as written. This corrects those pages against the reviewed SDK and local server evidence, including prepared statements, background jobs, cleanup and monitoring. Integration guides are grouped with their examples in #4533. Function and Secret-object documentation stays unpublished until its release. The two Function pages are listed in `.mintignore` and left out of the navigation, and the sections about them on other pages are MDX comments. Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after #4533 is squash-merged. Local assembly, Mint validation and link checks pass, and the assembled site shows no Function or Secret-object content. A local release rehearsal that uncomments the drafts and restores the navigation also passes. Hold publication until this PR and lancedb/sophon#8688 are on `main`. |
||
|
Jack YeandBrendan Clement
|
47b7babc08 |
docs: bring the open-source documentation into this repository (#4122)
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (
|