mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-29 14:56:51 +00:00
c167b99fa075b10b97ebce976d509861b0974807
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  lancedb-gatefixer[bot]andXuanwo
|
e3dce2cbad |
fix(rust): prevent oversized SQ remap panics (#3861)
## Summary - preflight IVF-SQ and IVF-HNSW-SQ storage segments touched by compaction - preserve unknown legacy fragment coverage as conservatively affected - cover both fixed-size vector and multivector SQ storage dimensions - reject only affected partitions whose fixed-size-list child exceeds Arrow's `u32` take-index limit - return an actionable `InvalidInput` error before Arrow can panic ## Root cause Arrow 58 builds fixed-size-list child indices with `u32`. For a large SQ partition, `partition_rows * vector_dimension` can exceed `u32::MAX`; the child range then wraps inside `take_fixed_size_list`, producing the reported `ArrayData::slice` assertion during index remapping. The preflight now reads raw segment metadata, mirrors Lance's fragment-bitmap selection (including `None` as affected/unknown), opens only affected vector segments, and checks their actual storage quantizer and partition sizes. It derives the stored dimension for both `FixedSizeList` vectors and flattened `List<FixedSizeList>` multivectors. Stable-row-ID datasets, deferred remapping, custom remappers, unaffected segments, non-SQ indexes, safe partitions, and no-op compactions are unchanged. Because the upstream Arrow kernel still has the limit, an affected index must be recreated with default IVF partitioning (or enough partitions to stay below the reported maximum) before compaction. This changes the failure from a process panic to a targeted recovery error; it does not silently alter index structure. ## Validation - `cargo fmt --all -- --check` - `cargo test --quiet --features aws,remote -p lancedb table::optimize::tests --lib` - `cargo test --quiet --features remote -p lancedb table::create_index::tests::test_create_index_ivf_hnsw_sq` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples -- -A clippy::redundant-pub-crate` (the allowance is for pre-existing Rust 1.97 warnings in remote-table code) Fixes #2866 <!-- lance-gatekeeper-fix:v1 agent=ba2f07b1ee6f9e68a0de1961c164f70e generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
3352bf49df |
fix(python): expose optimize compaction options (#3870)
## Summary - expose typed compaction options on synchronous and asynchronous table optimization - translate validated Python options into native Lance compaction settings - document the row-based steady-state limit and first-pass/source-budget caveats - carry the Lance beta.15 fragment-exclusion option through the typed Python API ## Root cause The native Python optimize binding always constructed default compaction settings, and the public AsyncTable API offered no way to lower the fragment row target. Wide-row fragments therefore remained compaction candidates on every later optimize call, repeatedly rewriting the accumulated table. ## Limitations This is an explicit row-based mitigation, not byte-native task sizing or a hard memory cap. Callers must derive target_rows_per_fragment from expected row size. Existing oversized fragments are rewritten whole on the first pass; max_bytes_per_file limits output file size rather than compaction memory; and source budgets admit whole planned tasks, so a budget below the first task performs no compaction. Splitting tasks by a byte budget or returning a planner error requires a Lance-side change. ## Validation - cargo check --release -p lancedb-python - cargo clippy --quiet -p lancedb-python - cargo fmt --all --check - ruff format --check python tests - ruff check python tests - uv run --no-sync pytest python/tests/test_table.py -k optimize_compaction -q (16 passed) Fixes #2325 <!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
  
|
951ae51ef6 |
docs: add realtime engine (RTE) getting-started guide (#4548)
Adds a user-facing realtime engine (RTE) page at `docs/web/tables/realtime-engine.mdx`, listed under **Management** after Branches. It is adapted from the internal WAL field guide (lancedb/sophon#6213), with the deployment and operations material left out. The page covers: - what RTE is and how it works (memtable → SSTables → compaction; reads merge the base table with the fresh tier) - getting started: `set_unenforced_primary_key`, indexes, `enable_rte` / `ALTER TABLE ... SET RTE`, `merge_insert` upserts and queries, partial updates, deletes, and `checkpoint_rte` / `disable_rte` - performance: write throughput and shard scaling, and read latency with and without RTE - a note on operations not yet supported (time travel, branching, `count_rows`, row IDs, `TABLESAMPLE`) Also aligns related docs with current RTE naming and behavior: - `sql/ddl/table.mdx`: `SET MEMWAL` / `UNSET MEMWAL` → `SET RTE` / `UNSET RTE` - `maintenance/background-jobs.mdx`: job names are `memwal_compaction` and `memwal_catchup_<type>` - `feature-engineering/materialized-views.mdx`: MemWAL → realtime engine - Python `AsyncTable.enable_rte` / `RTEWriteSpec.with_maintained_indexes` docstrings: by default, indexes created after enabling RTE are maintained too The examples are written directly in the page rather than generated from `docs/web-tests`, because RTE needs an Enterprise server to run. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: XY Zhan <zhaner08@hotmail.com> |
||
|
Xuanwo
|
9f616e6218 |
docs: explain managed blob ingestion, functions, and cleanup (#4604)
Document explicit Managed Blob v2 opt-in, descriptor queries and payload fetching, Function inputs and outputs, and whole-object cleanup. Explain null versus empty values, snapshot ownership, packed-object retention, and the distinction from the legacy large-binary Blob API. The local ingestion and Enterprise Function examples have been executed end to end. The Function lifecycle description depends on lance-format/lance#10176 and the corresponding Enterprise migration. The existing Functions page remains outside the published navigation until launch. Validation: the 347-file assembled site has no broken links. Mintlify 4.2.888 build validation reports four existing `RsExampleBody`/`RsExampleTail` import warnings in the reranking and vector-search pages; the same diagnostics reproduce with the unchanged base versions of all three pages edited here. |
||
 Brendan Clement
|
198812fd53 |
docs: clean up prose flagged by the shared Vale rules (#4596)
The existing docs have 254 findings from our shared Vale rules. Contributors would otherwise inherit these when editing the same lines. Clean up the prose across 80 pages: correct product names and a repeated word, simplify a few phrases, and replace dashes with punctuation that fits each sentence. The pinned Vale scan now reports zero findings, with no suppressions. |
||
|
Dan RammerandClaude Opus 5
|
fe34eb3549 |
refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What Renames the MemWAL surface in two passes, across Rust, Python, TypeScript, and Java, including the wire format. ### 1. MemWAL APIs drop "LSM" The write path these APIs drive is Lance's MemWAL; "LSM" described its shape rather than naming it. Turning it on and off now reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` / `optimize`, which compact the base table. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm` modules) are unchanged. ### 2. Stats use SSTable and table shard Aligns the stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an **SSTable**. **Generation → SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding *transform*, so it was already wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `stats.buckets` | `stats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed generations" → SSTables, "bucket" (meaning a shard) → table shard. `MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard status value are **unchanged** — those name the sharding transform or a server value, not the shard. ## Compatibility Breaking, for an experimental API. The wire format changes with the SDKs, so this must roll out together with the matching server change: - Routes: `/set_lsm_write_spec/` → `/enable_memwal/`, `/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` → `/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` → `/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/` - Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` → `memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables` - Request fields: `include_generation_rows` → `include_sstable_rows`; `use_lsm` → `use_memwal` on query and merge-insert bodies ## Validation - Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467 lib tests and all integration test targets pass; `cargo clippy` clean - Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`, `test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table` — 655 passed; `ruff format` / `ruff check` clean - TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593 passed; `biome check` clean; `docs/src/js` regenerated with `npm run docs` - Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned google-java-format does not run on JDK 21) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Brendan Clement
|
06aa526ae1 |
docs: correct data workflows, SQL and operations (#4534)
This PR is part of the documentation revamp. Once the full set of PRs is merged, we’ll publish the combined site to staging for review before updating docs.lancedb.com. The curation, SQL and operations pages contain outdated claims and examples that do not reliably work as written. This corrects those pages against the reviewed SDK and local server evidence, including prepared statements, background jobs, cleanup and monitoring. Integration guides are grouped with their examples in #4533. Function and Secret-object documentation stays unpublished until its release. The two Function pages are listed in `.mintignore` and left out of the navigation, and the sections about them on other pages are MDX comments. Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after #4533 is squash-merged. Local assembly, Mint validation and link checks pass, and the assembled site shows no Function or Secret-object content. A local release rehearsal that uncomments the drafts and restores the navigation also passes. Hold publication until this PR and lancedb/sophon#8688 are on `main`. |
||
 Jack YeandBrendan Clement
|
47b7babc08 |
docs: bring the open-source documentation into this repository (#4122)
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (
|