`mkdocs build --strict` only catches references it cannot resolve. A bare
anchor such as `[limit][]` or `[vector search][search]` is matched by
autorefs against any heading on the site, so six of them silently linked
into the JavaScript reference instead. The relative links in
`permutation.py` and `remote/errors.py` pointed at in-page anchors and
paths that do not exist.
Targets that still exist here or in an imported inventory now use
mkdocstrings references; the guide pages deleted in #2770 use their
lancedb.com URLs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`mkdocs build` emitted 61 warnings on main, and rendering the previously
undocumented classes in this PR pushed that to 158. That backlog is what
blocks turning on strict mode (#3707), so clear it here rather than leave
it worse than we found it.
Most of it was one systematic false positive: griffe cannot see the
generated `__init__` of a pydantic dataclass, so every documented
parameter looked unknown. `warn_unknown_params` turns that check off.
The rest were real docstring bugs, in 15 docstrings:
* Prose trailing a `Parameters` section is read as parameter names, which
invented parameters called `The`, `you` and `To`. Moved into `Notes` or
the summary.
* numpydoc only reads a type when the colon has spaces around it. Where
the documented name is a pydantic attribute rather than a signature
parameter, griffe has no signature to fall back on and the type was
dropped. Affects nine embedding classes.
* `num_partitions, default sqrt(num_rows)` and friends parse as a list of
names, rendering a bogus `default` parameter.
* One parameter indented five spaces instead of four.
`nodejs/CONTRIBUTING.md` links to the repo-root CONTRIBUTING.md, which
does not resolve once typedoc copies the file into `docs/src/js/_media/`;
an absolute URL works from both places.
`mkdocs build --strict` now exits 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four packages are now rendered by a single mkdocstrings directive each,
driven by the module's `__all__`, instead of a hand-maintained list of
symbols. These were where most of the drift was: 7 of 12 rerankers and
14 of 17 embedding functions had never been listed.
`lancedb.embeddings` had no `__all__`; without one mkdocstrings renders
no members at all for a re-export package, so one is added.
AGENTS.md gains a section describing how the reference page is wired up
and how to check a docs build locally, plus a step in the "adding a new
method on Table" checklist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Python API reference page had drifted from the public API. Branch
management (`Branches` / `AsyncBranches`, which own `diff` and `merge`),
structured full-text query classes, take queries, blob helpers,
namespace connections, most rerankers and embedding functions, the
PyTorch dataloader, and several other public symbols were never listed,
so they did not appear in the rendered docs.
Also fixes docstring cross-references that pointed at guide pages which
have since moved off this site, and at unresolvable relative targets
(`[Table](Table)`, `[PyArrow Table](pyarrow.Table)`).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`test_read_consistency_interval` asserted that a table opened with a
100ms `read_consistency_interval` still read stale data immediately
after a concurrent write. The cache timestamp is set when the table is
opened and reads within the interval do not refresh it, so that
assertion only held if the intervening open/count/commit/count sequence
finished within 100ms of real wall-clock time. On a loaded CI runner it
did not: the TTL expired, `count_rows` refreshed synchronously, and the
test failed with `left: 1, right: 0`. This broke the Rust workflow on
`main` at 0bc08160 (a Python-only commit).
This pins the `background_cache` mock clock once `table2` has seeded its
cache, and advances it explicitly in place of `tokio::time::sleep`, so
the test controls when the interval elapses. Same approach as #3547.
With the clock pinned there is no real sleep left to be imprecise, so
the `cfg(not(target_os = "windows"))` guard is dropped and the test now
runs on Windows too.
Verified by inserting a stall before the write: 120ms reproduces the
original failure deterministically, and with this change the test still
passes with a 500ms stall.
Fixes#3712
## What
Expose custom FTS stop-word lists in the Python and TypeScript public
APIs, including their standalone tokenize helpers and remote index
creation.
This PR supports concrete string lists only. It does not add file or
LanceDB-table stop-word sources.
## Why
Rust already exposes Lance's custom stop-word list option. The Python
and TypeScript APIs did not pass it through, and local index details did
not retain the full tokenizer parameters needed by index-backed
tokenization after reopening a table.
## How
- Add `custom_stop_words` / `customStopWords` to the Python and
TypeScript FTS and tokenize options.
- Preserve `None` / `undefined`, empty lists, and list contents without
normalization.
- Load the persisted FTS segment parameters when returning local index
details.
- Serialize the concrete list in remote create-index requests.
- Keep Python and TypeScript tests thin; behavior, persistence, query
tokenization, and remote JSON coverage live primarily in Rust.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
- Python extension rebuild with `uv` and `maturin`
- Targeted Python tests: 4 passed
- Python `ruff format --check` and `ruff check`
- TypeScript build, typecheck, Biome lint, generated docs, and targeted
tests
---------
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
## Summary
- add a keyword-only `transform_parallelism` option to
`StreamingDataset`
- preserve CPU auto-detection by default and fall back to one worker
when unavailable
- apply the configured limit to both the transform executor and
concurrency semaphore
- document and test explicit, default, fallback, and invalid values
## Testing
- `uv run --extra tests --with torch pytest
python/tests/test_elastic_dataloader.py -q` (`136 passed`)
- `uvx ruff check python/lancedb/streaming.py
python/tests/test_elastic_dataloader.py`
- `uvx ruff format --check python/lancedb/streaming.py
python/tests/test_elastic_dataloader.py`
- `git diff --check origin/main...HEAD`
Closes#3695
Co-authored-by: buduoqiu <yaodong-shen@users.noreply.github.com>
Standard GitHub-hosted runners are free on public repos, so all Actions
spend here is on the `*-8x-*` / `4x` larger runners. Measured over 30
days at current (post-Jan-2026) larger-runner rates, that is ~$1,400/mo,
and `npm-publish` is ~70% of it.
## Changes
**Fat LTO was forcing builds onto large runners.** `[profile.release]`
in `.cargo/config.toml` sets `lto = "fat"` with `codegen-units = 1`,
which is single-threaded and the peak-memory step. The macOS
`npm-publish` build was 111 of its 113 minutes in one `napi build` step,
making it the critical path of the whole publish pipeline. The ThinLTO
override already applied to Windows now covers macOS too, and both
Windows builds move from `windows-2025-8x-x64` to the free standard
`windows-2025`.
**The npm-publish cargo cache never existed.** There are zero caches
with its key prefix. The key was static, so `actions/cache` (which only
writes on a miss) could never refresh it, and a multi-GB release
`target/` per target could never fit the repo's 10 GB budget anyway. Now
caches only the crate registry, keyed on `Cargo.lock`. The docker builds
also mounted `.cargo/registry/*` while the cache saved `.cargo-cache`,
so containers re-downloaded the registry every run.
**Cache eviction thrash.** Repo cache usage is 10.4 GB against GitHub's
10 GB cap, so every PR run evicted main's warm entries. `rust.yml` and
`nodejs.yml` now restore everywhere but only save from `main`.
**npm-publish moves to nightly + tags** instead of every push to main
(~90/month). The cross-compiled targets do need watching, so
`report-failure` now fires on scheduled runs, and dedupes onto an
existing open issue rather than filing one per night.
**rust.yml aarch64-pc-windows-msvc** cross-compiled its tests and then
skipped them, paying full codegen and link cost for a compile check.
`windows-11-arm` is now GA and free on public repos, so it builds and
tests natively. Its test step also passes `--target` — without it cargo
used `target/ci/` rather than `target/<triple>/ci/` and rebuilt the
entire dependency graph a second time.
**pypi-publish.yml had no concurrency group**, so force-pushes left a
~74 minute Windows job running.
## What is cost vs. wall-clock
| Change | Cost | Wall-clock |
|---|---|---|
| Windows npm-publish → free runners | **−$570/mo** | slower per job
(8→4 cores) |
| npm-publish nightly | **−$125/mo** | — |
| pypi-publish concurrency | small | — |
| macOS ThinLTO | $0 (already free) | **−~50 min** per release |
| rust aarch64 Windows native | $0 (already free) | **−~25 min** |
| rust `--target` on test step | $0 | large, avoids a second full build
|
| rust-cache `save-if` | small | faster via real cache hits |
## Risks
- The two Windows builds now have 4 cores instead of 8 and ~14 GB of
free disk. If they fail, it is most likely disk rather than memory;
fallback is `windows-2025-4x-x64`, which still halves that line.
- `windows-11-arm` has a thinner toolset (choco/vcpkg/protoc under
emulation) and this enables a test step that has never run, so it may
surface real aarch64 failures. That is the point, but it is the change
most likely to need iteration.
- ThinLTO applies to published macOS and Windows binaries, typically
within a few percent of fat LTO. Linux release builds are untouched.
## Follow-ups
- `python.yml` `pydantic1x` (37 min) and `Doctest` (33 min) each rebuild
the extension from source via `pip install -e .` with no Rust cache;
they should consume the wheel the `linux` job already builds. Worth
~$235/mo and ~70 min of compute per run. Separate PR.
- The three `ubuntu-2404-8x-x64` npm-publish builds (~$420/mo at the old
cadence) are the remaining large-runner spend;
`aarch64-unknown-linux-gnu` could run natively on free
`ubuntu-24.04-arm`. Worth doing after this lands so the ThinLTO change
can be validated first.
- The wheel composite actions declare `python-minor-version` as required
but never use it, and every caller omits it (actionlint warns).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
`list_versions()` against a remote table on a server that uses
lance-namespace was failing. The server was returning
`timestamp_millis`, while db-catalog deployments were returning
`timestamp`, and the client was only accepting `timestamp`. So, updated
the client to accept both. (assuming we're migrating over time;
eventually we can turn off the `timestamp` code path I suppose.)
## Summary
Lance can now plan multiple byte ranges for the same blob in one
`read_blob_ranges` operation, but LanceDB users currently cannot expose
a complete set of logical ranges to that planner.
This complements `BlobFile`: file-like consumers such as PyAV can
continue to discover ranges dynamically, while callers that already know
the ranges for a batch can submit them together.
## Motivating example
A training table may store a large video blob together with a small
application-level clip index:
```text
video: blob
clips: [{offset, length}, ...]
```
The caller can select the videos and clips for a batch, obtain their row
IDs from the query, and read all of the selected windows together:
```python
rows = (
table.search()
.select(["clips"])
.with_row_id(True)
.limit(64)
.to_arrow()
.to_pylist()
)
requests = []
for row in rows:
clip = sample_clip(row["clips"])
requests.append(
(row["_rowid"], clip["offset"], clip["length"])
)
chunks = table.fetch_blob_ranges("video", requests)
```
Here, `_rowid` comes from the LanceDB query, while `offset` and `length`
come from the application's clip index and are relative to that row's
video blob. The caller describes only the logical reads; Lance still
handles validation, source grouping, coalescing, scheduling, and byte
backpressure.
Lance v10.0.0-beta.5 returns one logical result per blob selector or
range request and explicitly distinguishes null blobs from valid empty
values. LanceDB consumes that aligned result contract directly and only
adds a cardinality check for unresolved row IDs.
This PR exposes batched blob-range reads on local Rust and Python
tables. Results preserve request identity, duplicates, null slots, and
valid empty ranges while allowing Lance to execute the physical reads
out of order. Scheduler buffer sizing remains an internal Lance concern,
so the LanceDB API does not expose `io_buffer_size`.
Cloud tables continue to report this operation as unsupported until
there is a corresponding remote API.
SELECT COUNT(*) FROM t WHERE <predicate> — and any query that plans an
empty-projection scan — panics the executing query task:
InvalidArgumentError("must either specify a row count or at least one
column")
Root cause
MetadataEraserExec wraps every LanceDB table scan to strip schema-level
metadata, rebuilding each batch in execute():
RecordBatch::try_new(schema.clone(), batch.columns().to_vec()).unwrap()
RecordBatch::try_new infers the row count from the columns. COUNT(*)
with a filter is planned with an empty projection, so the scan emits
zero-column batches — there are no columns to infer a length from,
try_new returns Err, and the .unwrap() panics.
(This is specific to the empty-projection case: COUNT(*) with no filter
is answered from statistics and never scans, and COUNT(<col>) projects a
column — both already work.)
Updates the Rust workspace Lance dependencies and Java lance-core
dependency to v10.0.0-beta.5. No compatibility fixes were required;
full-workspace Clippy passes with warnings denied. Lance tag:
https://github.com/lance-format/lance/releases/tag/v10.0.0-beta.5
## What
MemWAL LSM **read** support. When a table has an LSM write spec
(`set_lsm_write_spec`), `merge_insert` upserts live in the MemWAL
active/frozen memtables and flushed SSTables until an external
compaction merges them into the base table, so a normal scan returns
**stale** data. This routes reads through Lance's `LsmScanner` so
queries also surface that in-flight data, deduplicated by primary key
(newest generation wins).
## How
- Adds a **`use_lsm: Option<bool>`** query flag, symmetric with the
`merge_insert` flag:
- **unset** — auto-route through the LSM scanner when the table carries
a write spec
- **`use_lsm(true)`** — force the LSM path; error if there is no spec
- **`use_lsm(false)`** — read the base table only (the escape hatch)
- Plain scan, single-column full-text search, and single-vector ANN all
run through one `LsmScanner` (assembled from on-disk shard manifests
plus the cached writer's in-memory memtables), so a `where` predicate is
honored as a **prefilter** uniformly — including for vector search.
- **Compaction-aware snapshots:** an SSTable generation is dropped only
once it is both compacted into the base table and covered by the arm's
base-index catch-up (`index_catchup`); plain scans use the compaction
watermark alone.
- Query shapes the scanner cannot honor hard-error with guidance to set
`use_lsm(false)`: hybrid, multi/binary vectors, `with_row_id`,
reranking, `order_by`, dynamic/Substrait projection or filters,
`distance_range`, `use_index(false)`, postfilter, take-by-row-id/offset,
reads from a time-traveled version, and an unmaintained or ambiguous
(multiple) FTS/vector index. Namespace-pushdown queries fall back to
local execution when a spec is present; WAL-only writers are handled.
- Exposed across the Rust core and the Python (`use_lsm`) and TypeScript
(`useLsm`) bindings, including `TakeQuery`.
Rebased from Lance `7.2.0-beta.3` to `10.0.0-beta.3`.
Python was versioned and tagged separately from the Rust, Java, and
Node.js SDKs, and had drifted three minor versions ahead (0.36 vs 0.33).
Users had no way to tell which Python version corresponded to which Rust
or Node release, and the gap had no meaning behind it.
This unifies the two tracks so there is one version and one tag for all
four SDKs.
## Version
The shared version is set to `0.37.0-beta.0`. Python continues its own
sequence (highest published: 0.36 → 0.37) while Rust, Java, and Node.js
jump 0.33 → 0.37 to meet it. Picking Python's next minor means Python
users see no discontinuity at all, and only the other SDKs skip forward.
Note that `main` trails the `release/v0.32` branch on both lines (main
is at 0.32.0-beta.3 / 0.35.0-beta.3; the release branch carries
0.33.0-beta.0 / 0.36.0-beta.0), so 0.37 is chosen to clear the highest
tag on either branch. Every index stays monotonic:
| index | publishes | last published | next |
|---|---|---|---|
| PyPI | stable only | 0.34.0 | 0.37.0 |
| Fury | previews | 0.36.0b0 | 0.37.0-beta.1 |
| npm | both | 0.33.0-beta.0 | 0.37.0-beta.1 |
| crates.io | stable only | 0.31.0 | 0.37.0 |
| Maven | both | 0.33.0-beta.0 | 0.37.0-beta.1 |
A one-time jump for three SDKs, versus explaining the offset
indefinitely.
## Mechanism
* `python/.bumpversion.toml` is removed. `python/Cargo.toml` — the
source of the Python package version, since `pyproject.toml` declares
`dynamic = ["version"]` — becomes a tracked file of the root config. Its
`cargo update -p lancedb-python` pre-commit hook is dropped as
redundant: `ci/update_lockfiles.sh` already refreshes every workspace
member version in `Cargo.lock`.
* `pypi-publish.yml` triggers on `v*` instead of `python-v*`, so one tag
releases all four packages. `ci/bump_version.sh` and
`make-release-commit.yml` lose their now-dead tag-prefix and
per-language plumbing, including the `python` / `other` dispatch inputs.
* The two byte-identical GH release jobs in `npm-publish.yml` and
`pypi-publish.yml` are replaced by a single `gh-release.yml`. One
release per tag, named `LanceDB vX.Y.Z`, instead of separate "Python
LanceDB" and "Node/Rust LanceDB" releases for the same commit.
The trade-off: there is no longer a way to ship a Python-only patch
without also releasing crates.io, Maven, and npm. That is the cost of
making drift structurally impossible.
## Beta releases marked "Latest" (#3666)
Both GH release jobs used:
```yaml
prerelease: ${{ contains('beta', github.ref) }}
```
The arguments are reversed. `contains(search, item)` asks whether
*`search`* contains *`item`*, so this evaluated "does the literal string
`'beta'` contain `refs/tags/python-v0.35.0-beta.2`?" — always `false`.
Every beta was published as a full release, and GitHub awards "Latest"
to the newest non-prerelease.
The new workflow derives the flag from the parsed version rather than
the raw ref, and sets `make_latest` explicitly:
```yaml
prerelease: ${{ steps.extract_version.outputs.prerelease }}
make_latest: ${{ steps.extract_version.outputs.prerelease == 'false' }}
```
npm was never affected (`--tag preview` uses correct bash), and PyPI
already excludes pre-releases from resolution.
This only fixes releases published from here on. Already-published betas
need a one-time backfill:
```shell
gh api --paginate /repos/lancedb/lancedb/releases \
--jq '.[] | select(.prerelease == false) | select(.tag_name | test("beta")) | .id' \
| xargs -I{} gh api -X PATCH /repos/lancedb/lancedb/releases/{} -F prerelease=true
```
## Verification
Ran `ci/bump_version.sh` end-to-end against this branch with the release
tooling installed:
* `preview` → tags `v0.37.0-beta.1` (previous tag `v0.33.0-beta.0`
detected, `pre_n` bump)
* `stable` → tags `v0.37.0`
* Both paths update `.bumpversion.toml`, `rust/lancedb/Cargo.toml`,
`nodejs/Cargo.toml`, `python/Cargo.toml`, `nodejs/package.json`, the 7
`nodejs/npm/*/package.json` files, both Java poms, and
`docs/src/java/java.md` together
* `check_breaking_changes.py` resolves the last stable as `v0.31.0`, so
the minor-version gate passes
All five touched workflows parse as valid YAML and the pre-commit hooks
pass.
## Notes for review
* This targets `main` only, so it takes effect at the next
release-branch cut. The in-flight `release/v0.32` branch still carries
`v0.33.0-beta.0` / `python-v0.36.0-beta.0`; if we want the imminent
stable to be 0.37.0, this needs to be applied there too.
* Historical `python-v*` tags are left alone. The changelog builder
scans `^v`, which does not match them, so the first unified release's
notes will compute `fromTag` from the Rust/Node line only — a one-time
gap in the Python-side changelog.
* Pre-existing and not addressed here: `ci/update_lockfiles.sh --amend`
amends the commit that `bump-my-version` has already tagged, so the
lockfile update lands outside the tag on stable releases.
Fixes#3666
## What changed
- add `block_size` to Python FTS configuration and the deprecated
local/remote helpers
- add `blockSize` to the TypeScript FTS options and propagate it through
the NAPI binding
- serialize the value as `block_size` for remote index creation
- document the existing Rust builder API and generate the TypeScript API
reference
- add local, remote, metadata, search, and invalid-value regression
coverage
## Why
Lance supports configuring the number of documents per compressed FTS
posting block, but LanceDB's Python and TypeScript APIs did not expose
the setting. This made the experimental FTS V3 layout unavailable
through those clients and allowed the value to be dropped before index
creation.
## How it works
The default remains `128`. Supported values are `128` and `256`;
selecting `256` uses the experimental FTS V3 format. Invalid values are
rejected by the Lance builder and surfaced as Python or JavaScript
errors.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo +1.94.0 clippy --quiet --features remote --tests --examples --
-D warnings`
- targeted Rust local and remote index tests
- Rust doctests: 34 passed
- Python Ruff checks, doctest, and targeted local/remote tests: 5 passed
- TypeScript build, Biome lint, generated docs, and targeted Jest tests:
9 passed
- `git diff --check`
## Limitations
The Java client remains unchanged because its external remote REST model
does not currently expose `block_size`.
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
## What
`AnswerdotaiRerankers(return_score="all").rerank_hybrid(...)` (and
`ColbertReranker`, which subclasses it without overriding
`rerank_hybrid`) raises:
```
pyarrow.lib.ArrowInvalid: Invalid sort key column: No match for FieldRef.Name(_relevance_score) in _rowid: int64 ...
```
## Why
```python
combined_results = self.merge_results(vector_results, fts_results)
combined_results = self._rerank(combined_results, query)
if self.score == "relevance":
combined_results = self._keep_relevance_score(combined_results)
elif self.score == "all":
combined_results = self._merge_and_keep_scores(vector_results, fts_results)
```
When `score == "all"`, `combined_results` is unconditionally overwritten
by `_merge_and_keep_scores(vector_results, fts_results)` **after**
`_rerank()` already computed and appended `_relevance_score` —
discarding it. The following `sort_by("_relevance_score", ...)` then has
nothing to sort on.
Every sibling reranker that supports `return_score="all"`
(`cross_encoder`, `openai`, `cohere`, `jinaai`, `voyageai`, `watsonx`)
instead calls `_merge_and_keep_scores()` **before** `_rerank()`. This
file is the one place the ordering got inverted when `"all"` support was
added (#2509) — a copy/paste inconsistency across the six files that PR
touched. Fix mirrors the pattern already used (and tested) by the other
five rerankers.
Also drops the now-stale `"Only 'relevance' is supported for now"`
docstring line on both classes, left over from before `"all"` support
existed.
## Testing
Added `test_answerdotai_reranker_return_all`, mirroring the existing
`test_cross_encoder_reranker_return_all`. Verified locally with the real
built Rust extension: red (reproduces the exact `ArrowInvalid` above) →
green, using the actual `rerank_hybrid`/`_rerank`/`base.py` code path
with the model call mocked out — my local environment's
`rerankers==0.10.0` fails to load the real ColBERT model against the
available `transformers` version (`AttributeError: 'ColBERTModel' object
has no attribute 'all_tied_weights_keys'`), which I confirmed also
breaks the **pre-existing**, unmodified
`test_colbert_reranker`/`test_answerdotai_reranker` baseline tests
identically — an unrelated local dependency-version issue, not a
regression from this change. `ruff check`/`ruff format` clean; full
`test_rerankers.py` run: 9 passed / 8 skipped / 3 failed (the 3 failures
are exactly those two pre-existing tests plus my new one, all failing at
model-loading time for the same unrelated reason before reaching the
changed code).
---
Disclosure: this PR was drafted with AI assistance (Claude); I reviewed,
tested, and take responsibility for the change.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
This updates the Java API reference to close the documentation gaps that
can be fixed from the current Java source and generated namespace API.
The patch adds an empty table example, shows how to wrap returned Arrow
IPC query bytes in a reusable `ArrowFileReader` helper, and documents
the Java index operations that are currently exposed by the namespace
client: vector indexes, scalar indexes, full text search indexes, and
listing indexes.
## Issue Links
Fixes https://github.com/lancedb/docs/issues/157
Fixes https://github.com/lancedb/docs/issues/160
Partially addresses https://github.com/lancedb/docs/issues/159 by
documenting the index parameters currently exposed by Java. The
requested `num_partitions` example is still blocked because
`CreateTableIndexRequest` does not expose IVF training parameters yet.
Not included: https://github.com/lancedb/docs/issues/158. The current
Java docs and source remain remote namespace oriented, so local DB
connection documentation should wait until the Java local DB API is
available and can be verified.
## Validation
- Built the Java core module with OpenJDK 17:
`./mvnw -pl lancedb-core -am -DskipTests compile`
- Checked the Markdown diff:
`git diff --check -- docs/src/java/java.md`
The Java build succeeds. It still reports pre-existing checkstyle
warnings in the namespace client builder, but the Maven build is green.
## Summary
- reconstruct foreign Arrow Map schemas from their single sanitized
entries field
- reject malformed Map types with anything other than one child
- preserve the complete Map schema and `keysSorted` value through
empty-table creation and IPC round trips across Arrow 15–18
## Testing
- `./node_modules/.bin/jest --runInBand __test__/arrow.test.ts
__test__/sanitize.test.ts`
- `pnpm lint`
- `pnpm build`
- `pnpm run docs`
Fixes#2337
## What
- Replace legacy model names in `WatsonxEmbeddings` with the current
supported set:
- `ibm/granite-embedding-278m-multilingual` (new default, 768-dim)
- `ibm/slate-125m-english-rtrvr-v2` (768-dim)
- `ibm/slate-30m-english-rtrvr-v2` (384-dim)
- `intfloat/multilingual-e5-large` (1024-dim)
- `sentence-transformers/all-minilm-l6-v2` (384-dim)
- Add `space_id` field — mutually exclusive with `project_id`, mirrors
the
existing pattern in `WatsonxReranker`
- `project_id` / `space_id` resolution now falls back to
`WATSONX_PROJECT_ID` /
`WATSONX_SPACE_ID` env vars; exactly one must be supplied
## Why
The previously hardcoded models (`ibm/slate-125m-english-rtrvr`,
`sentence-transformers/all-minilm-l12-v2`) are legacy and no longer
listed as
supported by the watsonx.ai platform. `space_id` scoping was already
supported
by `WatsonxReranker` but was missing from the embeddings counterpart.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Updates the Rust workspace Lance dependencies and Java lance-core from
v9.1.0-beta.5 to v9.1.0-beta.7, including the generated Cargo lockfile.
No LanceDB compatibility changes were required for this release. See the
[Lance v9.1.0-beta.7
release](https://github.com/lance-format/lance/releases/tag/v9.1.0-beta.7).
This PR adds some support for `diff` / `merge` in the remote client as
for local tables we stay `NotSupported` until
https://github.com/lance-format/lance/issues/7263.
This wires the two review-and-land calls against the remote REST API:
- `POST /v1/table/{id}/branches/diff`
- `POST /v1/table/{id}/branches/merge`
Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python
returns the wire JSON, same shape as the REST response.
Merge here means promoting a branch's added columns onto `main`.
### Behavior
- Remote only. Local raises `NotSupported`.
- A rejected merge is not an exception. HTTP 409 still returns `Ok` / a
dict with `status="rejected"` and blockers in `diff.mergeBlockers`.
- Unknown blocker / status codes parse as `Unknown` so a newer server
does not break older clients.
- `MergePreview` tolerates missing fields for the same reason.
- Merge requests are not retried. 409 is final and carries the body you
need.
### Example
```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")
exp.add_columns({"tag": "cast('draft' as string)"})
diff = table.branches.diff("exp")
preview = table.branches.merge("exp", dry_run=True)
result = table.branches.merge("exp", dry_run=False)
if result["status"] == "merged":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "rejected":
print(result["diff"]["mergeBlockers"])
```
### Testing
cargo test -p lancedb --features remote diff_branch
cargo test -p lancedb --features remote merge_branch
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Problem
On the remote (LanceDB Cloud) write path, each write partition is
uploaded as a **single** `/insert?upload_id=...` request that stays open
until the whole partition has been streamed and the server has written
it to object storage. For large bulk ingests a partition can be many GB,
so a single request can run longer than the client read timeout (default
300s), surfacing as:
```
lancedb.remote.errors.HttpError: operation timed out
```
The server already supports staging **multiple** parts under one
`upload_id` (each `/insert` writes a separate transaction that
`complete` merges atomically), but the client never used that — it sent
one part per partition.
## Change
Split each partition into multiple parts of at most
`max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded
as its own `/insert?upload_id=...&upload_part_id=...` request. This
bounds how long any single request stays open, independent of total data
size or write parallelism.
Key properties:
- **Still streamed, not buffered.** Each part's body is driven through a
bounded channel while the request is in flight (`futures::join!` of a
producer + the send), so peak memory stays at a couple of batches per
partition regardless of the part size. Backpressure from a
slow/throttled server still propagates upstream.
- **Correct part accounting.** An empty partition still sends exactly
one (schema-only) part so `complete` has a transaction to commit; a size
cut landing exactly on the end of input does not emit a trailing empty
part.
- **Multipart only.** The single-request (non-multipart) path is
unchanged.
## Config
New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable
via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable.
**Default 1 GiB** (`Some(0)` disables splitting → one request per
partition). Python users pick up the default/env automatically through
the remote client.
## Tests
- `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each
batch in its own part → N requests, each carrying the shared `upload_id`
and a distinct `upload_part_id`.
- `test_multipart_single_part_when_under_budget`: a large budget keeps
the partition in a single request.
- Verified end-to-end against a live remote table: a forced-chunked
multipart add (many parts) assembles to the correct row count.
Related to ENT-1883.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Fixes#1653.
`infer_vector_column_name` in `util.py` could silently return `None`
when `query is None` and `query_type` is not `"fts"` or `"hybrid"`. This
`None` then propagated into downstream code, causing a cryptic
`TypeError: expected bytes, NoneType found` rather than a clear error
message.
## Changes
- **Removes the no-op `try/except Exception as e: raise e`** around
`inf_vector_column_query` (it was catching and immediately re-raising
without adding any value)
- - **Adds a `None` guard** after the inference block: if
`vector_column_name` is still `None` at this point, raise a clear
`ValueError` pointing the user to pass `vector_column_name` explicitly
## Before / After
**Before:** cryptic `TypeError: expected bytes, NoneType found` deep in
schema lookup code
**After:**
```
ValueError: No vector column found in the schema. Please specify the vector column name explicitly via the `vector_column_name` parameter.
```
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Some additions to our lancedb skill to enable agents to use the jobs
methods that we recently added. Eval tests (below, with and without
these additions to the skill) suggest that they're helping, mostly to
find the right method calls. These are a little unusual because they
require REST server connection, they're not yet implemented in the SDKs.
```
┌─────────────────────┬───────────┬────────────┬─────────────┬──────────┬───────────┬──────────┬───────────┐
│ eval │ grade w/o │ grade with │ improvement │ time w/o │ time with │ cost w/o │ cost with │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ 8-list-running-jobs │ 2.5/3 │ 3/3 │ +0.5 │ 123s │ 29s │ $0.58 │ $0.18 │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ 9-describe-job │ 1/5 │ 5/5 │ +4.0 │ 159s │ 52s │ $0.62 │ $0.25 │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ 10-cancel-job │ 3/3 │ 3/3 │ +0.0 │ 99s │ 35s │ $0.55 │ $0.21 │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ TOTAL │ 6.5/11 │ 11/11 │ +4.5 │ 381s │ 116s │ $1.75 │ $0.65 │
└─────────────────────┴───────────┴────────────┴─────────────┴──────────┴───────────┴──────────┴───────────┘
```
Failure reasons are because the agent didn't know the right method to
call, spent all its turns guessing REST calls, tried to inspect lancedb
code, but didn't find the answer in here.
## Problem
`table.add(dataset)` with a `pyarrow.dataset.Dataset` OOMs the client
during bulk ingestion of wide rows (e.g. embedding columns), even
against a remote table where the upload itself is streaming.
The cause is in `to_scannable`: a `Dataset` is scanned with pyarrow's
default scanner settings (`batch_size=131072` rows,
`batch_readahead=16`, `fragment_readahead=4`). pyarrow's internal
threads prefetch that read-ahead window independently of LanceDB's
backpressure, so for wide rows a large fraction of the dataset is held
in memory. On the remote path this is then multiplied across the
multipart write partitions (one in-flight batch per partition, up to
CPU-core count).
Reproduced on a 10 GB / 1.55M-row dataset with two 768-dim float32
embeddings: peak client RSS ~11.7 GB for the scan alone (6.8 GB after
consuming a *single* batch), ~15.4 GB for the full remote `add()`.
## Fix
`to_scannable` now sizes the scanner from an estimate of bytes-per-row
derived from the schema:
- **Narrow datasets keep pyarrow's defaults** (empty scanner kwargs) —
no throughput regression. The bound only engages above ~410 bytes/row.
- **Wide rows** get a smaller `batch_size` (~16 MiB/batch) and reduced
read-ahead (`batch_readahead=2`, `fragment_readahead=1`) so peak
in-flight memory stays near a ~1 GiB budget. Read-ahead (not just batch
size) has to drop, because pyarrow pins whole row-group buffers.
On the 10 GB dataset this drops peak client RSS to ~1.4 GB, and it stays
flat as the dataset grows. The `Dataset`/`LanceDataset` scannables
remain rescannable (retry-safe).
## Also: expose `write_parallelism` on `add()`
`AddDataBuilder::write_parallelism` already existed in Rust but was not
exposed in Python. This PR forwards it through the async, sync, and
remote `add()` methods, so users can cap the number of parallel write
partitions (each buffers data in flight) to trade throughput for memory
on large uploads.
## Tests
- `test_scannable.py`: bytes-per-row estimation; narrow → defaults; wide
→ bounded; `Dataset` reader streams bounded batches and stays
rescannable.
- `test_table.py`: `write_parallelism` on sync and async `add()`, and
that `write_parallelism=0` is rejected.
Fixes ENT-1883
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Moves the skill from .agents/skills/lancedb to
plugins/lancedb/skills/lancedb, as recommended by codex and claude.
Install path now is:
### Codex/ChatGPT app
Codex: Plugins -> "Create" arrow -> Add plugin marketplace
search for lancedb plugin and install
### Codex CLI
```
codex plugin marketplace add lancedb/lancedb
codex plugin install lancedb@lancedb # name@marketplace
```
### Claude app
Settings -> Plugins -> Add -> Add marketplace
search for lancedb plugin and install
### Claude CLI
```
claude plugin marketplace add lancedb/lancedb
claude plugin install lancedb@lancedb
```
Here's how it looks on ChatGPT/Codex app:
(the main icon has light and dark modes; the smaller one on the skill
doesn't so I made it gray 🤷 )
<img width="764" height="560" alt="Screenshot 2026-07-16 at 2 49 24 PM"
src="https://github.com/user-attachments/assets/b82cda16-3392-4740-ac47-b2f187cb2655"
/>
Hi, and thank you for LanceDB.
Small CI supply-chain hardening. In `make-release-commit.yml`, the
release job checks out with `LANCEDB_RELEASE_TOKEN` (a push-capable PAT)
and its final step pushes the version tag using a third-party action
pinned to a **mutable branch**:
```yaml
- name: Push new version tag
uses: ad-m/github-push-action@master
with:
github_token: ${{ secrets.LANCEDB_RELEASE_TOKEN }}
```
`@master` can move after review; whatever it points at then runs with
that release token in scope. This PR pins it to the commit behind the
current release (`v1.3.0` → `881a6320…`), keeping the version visible as
a comment. Behavior today is unchanged.
For transparency: I used AI assistance to spot and draft this; I
verified the workflow and resolved the SHA myself.
Tracks #3324. On x86_64 CPUs without AVX2 (Sandy Bridge / Ivy Bridge /
Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD), `import
lancedb` SIGILLs because the wheel bakes AVX2 + FMA into every compiled
function. Per [westonpace's
review](https://github.com/lancedb/lancedb/issues/3324#issuecomment-4328944354),
the default `lancedb` wheel stays fast; pre-Haswell users get a
separately-published `lancedb-compat` wheel.
## Summary
- Adds a `lancedb-compat` matrix entry to `pypi-publish.yml` that builds
with `RUSTFLAGS="-C target-cpu=x86-64-v2"` (Nehalem-class baseline).
Same Python API (`import lancedb` works) — files install to the same
namespace, so the two wheels conflict at install time and users pick
one. Same pattern as `psycopg2` / `psycopg2-binary` and `tensorflow` /
`tensorflow-cpu`.
- Generalizes `build_linux_wheel` and `upload_wheel` composites with
optional `package-name` and `rustflags` inputs (defaults preserve the
existing 4 `lancedb` matrix entries verbatim).
- Documents the choice in `python/README.md`: `pip install
lancedb-compat` for pre-Haswell hosts.
The default `.cargo/config.toml` baseline is unchanged.
## Sequencing
1. ~~lance-format/lance#6630 merges → runtime SIMD dispatch lands in
lance.~~ **Done — merged.**
2. lancedb's lance dep is bumped to a release that includes it (separate
PR / normal cadence).
3. This PR's `lancedb-compat` wheel build path starts producing a wheel
that runs on pre-Haswell hardware. **Maintainer setup**: register
`lancedb-compat` on PyPI and configure trusted publishing.
## Verified end-to-end on Sandy Bridge Xeon E5-2609
Verification was done locally against a fork-pinned lance dep that
includes the runtime dispatch implementation, using the same
`RUSTFLAGS="-C target-cpu=x86-64-v2"` flags this PR uses in CI:
```
$ RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
$ pip install ./target/wheels/lancedb-*.whl
$ python verify.py
PASS: import + simd dispatch + table create + vector search all work.
```
Pre-fix on the same CPU (default `pip install lancedb`): `Illegal
instruction (core dumped)`. Full reproducer (deps + clone + build +
verification):
https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df.
Fork-internal verification PR with the dep bump and full logs:
[`tobocop2/lancedb#2`](https://github.com/tobocop2/lancedb/pull/2).
## Benchmarks — no regressions on modern CPUs from the lance-side change
These are the numbers I ran for the lance PR, confirming the runtime
dispatch doesn't slow down the default (`target-cpu=haswell`) wheel that
existing users install. Criterion, one machine, one session, base → PR,
no `RUSTFLAGS` override. Full methodology, null experiments, and logs:
[lance-format/lance#6630 benchmark
comment](https://github.com/lance-format/lance/pull/6630#issuecomment-4933063394)
and the [logs
gist](https://gist.github.com/tobocop2/3c6d0f449cbd736aa2501f89a7fe56a2).
| benchmark | EPYC 7B13 (`avx2`, `fma`, no `avx512f`) | Xeon Cascade
Lake (`avx512f`) |
|---|---|---|
| `Cosine(f32, scalar)` *(control)* | +0.04% | +0.09% |
| `Cosine(f64, scalar)` | −0.34% | −1.94% |
| `Cosine(u8, SIMD)` | +2.30% | +3.63% |
| `Dot(f16, SIMD)` | −0.58% | +0.61% |
| `Dot(f32, SIMD)` | +0.34% | **−6.08%** |
| `Dot(f32, arrow_arity)` | +0.02% | −0.00% |
| `L2(f32, scalar)` | −0.10% | −0.02% |
| `L2(f32, simd)` (dim 1024) | +2.63% | −0.53% |
| **`L2(simd,f32x8)` (dim 8)** | **−45.9%** | **−25.1%** |
| `L2(u8, SIMD)` | +0.42% | −3.11% |
| `NormL2(f32, SIMD)` | −1.02% | −4.17% |
| `NormL2(f64, SIMD)` | +3.51% | −0.58% |
Nothing regresses beyond the noise floor. Dim 8 — the PQ sub-vector
width — improves 25–46%.
---
To be transparent: this isn't my domain of expertise and the lance-side
implementation is AI-generated. I verified it works end-to-end on the
failing hardware. Happy to roll in feedback.
Routes local sync child-namespace operations through the Rust-backed
connection instead of the Python namespace-client fallback.
Also keeps lazy namespace-client construction for table-to-Lance
conversion and preserves public namespace error mappings.
Validated locally with ruff format/check and targeted namespace pytest.
2026-07-16 10:53:49 -07:00
157 changed files with 8628 additions and 1170 deletions
"description":"Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables.",
description:"Override [project] name in python/pyproject.toml (e.g. 'lancedb-compat'). Default keeps 'lancedb'."
required:false
default:"lancedb"
rustflags:
description:"RUSTFLAGS for the build container, as a single whitespace-free token (e.g. '-Ctarget-cpu=x86-64-v2'). Empty leaves RUSTFLAGS unset, keeping the defaults from .cargo/config.toml."
required:false
default:""
runs:
using:"composite"
steps:
@@ -27,6 +35,18 @@ runs:
ARM_BUILD:${{ inputs.arm-build }}
run:|
echo "ARM BUILD: $ARM_BUILD"
- name:Patch package name for variant build
if:${{ inputs.package-name != 'lancedb' }}
shell:bash
env:
PACKAGE_NAME:${{ inputs.package-name }}
run:|
# Swap the [project] name so this build produces e.g. lancedb-compat
# wheels. The package still installs files under the lancedb/
# namespace -- import lancedb still works after pip install.
sed -i.bak 's/^name = "lancedb"$/name = "'"$PACKAGE_NAME"'"/' python/pyproject.toml
"description":"Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables, with idiomatic query/search patterns and performance defaults for ingestion, indexing, filtering, and diagnostics.",
"description":"Codex plugin for building LanceDB pipelines in Python and TypeScript.",
"author":{
"name":"LanceDB"
},
"keywords":[
"lancedb",
"vector-search",
"full-text-search",
"hybrid-search",
"python",
"typescript",
"pipelines"
],
"skills":"./skills/",
"interface":{
"displayName":"LanceDB",
"shortDescription":"Build LanceDB pipelines in Python and TypeScript.",
"longDescription":"Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables, with idiomatic query/search patterns and performance defaults for ingestion, indexing, filtering, and diagnostics.",
"developerName":"LanceDB",
"websiteURL":"https://www.lancedb.com",
"category":"Developer Tools",
"capabilities":[
"Developer Tools"
],
"defaultPrompt":"Create a LanceDB table, embed sample text, and run a vector search.",
description: Use when writing, reviewing, debugging, or documenting LanceDB pipelines in Python or TypeScript, especially code that should work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables. Helps avoid non-portable full-table materialization, choose idiomatic query/search patterns, and apply LanceDB performance defaults for ingestion, indexing, filtering, and diagnostics.
description: Use when writing, reviewing, debugging, or documenting LanceDB pipelines in Python or TypeScript, especially code that should work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables. Helps avoid non-portable full-table materialization, choose idiomatic query/search patterns, apply LanceDB performance defaults for ingestion, indexing, filtering, and diagnostics, and resolve connections to the remote server for Enterprise-only operations such as jobs.
---
# Building LanceDB Pipelines
@@ -19,7 +19,7 @@ Do NOT assume local-only table helpers exist on remote tables. If the user asks
## Workflow
1. Identify the SDK: Python, TypeScript, or both.
2. Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path.
2. Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path. If the task involves jobs in any way (listing, inspecting, creating, or canceling jobs), it is always the remote path and requires a remote server connection — see "Connecting to the LanceDB remote server" below before doing anything else.
3. Read the matching language branch before writing or changing code:
4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main.
- Remote server connection resolution (jobs, raw REST): `references/remote_connect.md`
- Job operations REST API (list/describe/cancel/query_events): `references/remote_jobs.md`
4. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets. Read `column_metadata.md` when the task is documenting, tagging, classifying, or grouping table columns (field descriptions, `lancedb:tag:*` tags, logical column families). Read `branch_ops.md` when the task involves branch lifecycle (list/create/delete), writing to a non-main branch, or verifying a change stayed off main. Read `remote_connect.md` when the task involves jobs or direct REST access to an Enterprise deployment, and `remote_jobs.md` for the job REST methods themselves (list, describe, cancel, query_events).
5. For Python schemas, favor Pydantic models and validate records before writing. Use PyArrow schemas when Arrow-native, streaming, or highly dynamic data makes them materially better suited.
6. Prefer `search()` or `query()` builders with explicit `select()` and `limit()` for reads.
7. Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote read pitfall.
@@ -70,6 +72,10 @@ Rules for portable Enterprise ingestion:
This is Enterprise/Cloud-specific. Local/OSS tables have no separate data plane, so `mode="overwrite"` and immediate same-name reuse are fine there.
## Connecting to the LanceDB remote server
LanceDB Enterprise/Cloud deployments are served by a server implementing the lance-namespace OpenAPI spec (<https://github.com/lance-format/lance-namespace/blob/main/docs/src/spec.yaml>). Every remote (`db://...`) connection talks to such a server, and some operations exist only there. In particular, **all operations around jobs (listing, inspecting, creating, or canceling jobs) run server-side** — there is no local/OSS equivalent. Before any job work, or any direct REST call to an Enterprise deployment, read `references/remote_connect.md` to resolve the base URL, credentials, and database header and to validate the connection. Then use the four job REST methods documented in `references/remote_jobs.md` (list, describe, cancel, query_events).
## Script
Run the scanner when reviewing or modifying an existing codebase:
Manage branches on a LanceDB table: list what exists, create new ones, delete stale ones, and direct read/write operations at a specific branch without touching main. Use for branch lifecycle tasks, experimental/isolated table versions, targeting an operation at a non-main branch, or confirming a mutation did not affect main.
Works on local/OSS and remote Enterprise/Cloud tables.
Works on local/OSS and remote Enterprise/Cloud tables, except merging a branch into main, which is Enterprise-only.
## The branch model (important)
@@ -101,6 +101,69 @@ assert b"lancedb:description" not in (table.schema.field("category").metadata or
Two handles on the same branch see each other's writes (e.g. `table.branches.create("exp")` and `db.open_table(name, branch="exp")`); main stays isolated.
## Merging a branch into main (Enterprise only)
Merge is available through the SDKs (`table.branches.merge(...)`) on **Enterprise tables only** — it is not supported on Cloud or local/OSS tables, which raise `NotSupported`.
`merge` takes the branch to merge **from** and a `dry_run` flag. Both the SDK method and the underlying REST endpoint **actually merge by default** (`dry_run=False`); pass `dry_run=True` to only preview. A rejected merge is **not an exception** — it returns a result with `status="rejected"` rather than raising, so inspect the return value. Use `branches.diff(from_branch)` to inspect a branch's pending diff without attempting a merge.
```python
exp = "experiment-reindex"
# preview only — returns status="ready" if it would merge cleanly
preview = table.branches.merge(exp, dry_run=True)
# actually merge (default)
result = table.branches.merge(exp)
if result["status"] == "merged":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "rejected":
print(result["diff"]["mergeBlockers"]) # why it was refused
The result is the wire JSON, containing `status` (`ready` on a passing dry run, `merged` on success, `rejected` when refused — also `notImplemented`/`unknown`), the branch `diff` (including `mergeBlockers` explaining any rejection), a `preview` of the columns that would be promoted, and — after a real merge — `mainVersionAfter`.
### Merge preconditions
Merge only **promotes newly added columns** onto main; it does not replay arbitrary commits. Practically, a branch is mergeable only if it has **exactly one commit since it was created, and that commit added a column**. The merge is rejected (`status: "rejected"`, with `mergeBlockers` set) if:
- the branch was forked from another branch rather than directly from main
- main has advanced since the branch was forked
- the branch's rows changed since the fork (row counts must match main exactly)
- the branch removed columns or changed a column's type/nullability
- the branch added no columns (index-only changes are not merged)
### Adding a column in a single commit
Because the branch must contain just one column-adding commit, add the column with its values in one operation rather than add-then-backfill:
1. **SQL transformation** — `add_columns` with a SQL expression computed from existing columns, so the column lands populated in one commit.
2. **Precompute the values** — compute the column's values externally, then add the fully-populated column in a single operation (e.g. via `merge_insert`/`add_columns` with the data ready).
3. **Lance-format-level data evolution (pylance)** — use Lance's data evolution with backfill, documented at <https://lance.org/guide/data_evolution/#with-data-backfill>.
## Quick reference
| Goal | Python | TypeScript |
@@ -113,5 +176,7 @@ Two handles on the same branch see each other's writes (e.g. `table.branches.cre
@@ -4,12 +4,19 @@ Quick method reference for Python LanceDB code. Cross-check source for non-trivi
## Connect
If you're connecting to a remote database, use this:
```python
import lancedb
db = lancedb.connect("./camelot-db") # local/OSS
db = lancedb.connect("db://my-db", api_key=api_key, region=region) # remote
db = lancedb.connect("db://my-db", api_key=api_key, host_override=host_override) # remote
```
(values may be found in LANCEDB_API_KEY and LANCEDB_HOST_OVERRIDE, either in env vars or a .env file)
If you're connecting to a local table using OSS LanceDB, use this:
```python
db = lancedb.connect("./camelot-db") # local/OSS
```
If you're not sure which, or if you can't find the api_key or host_override params, ask the user.
**Place the local database directory next to the script/entrypoint that opens it** (i.e. resolve the path relative to the script, `Path(__file__).parent / "camelot-db"`), not buried under a shared `data/` folder. The Lance dataset is the database, not a data file — keeping it beside its code makes ownership obvious and paths stable regardless of the working directory the script is launched from.
Every remote (`db://...`) connection talks to such a server, and some operations
exist only there. In particular, all operations around jobs (listing, inspecting,
creating, or canceling jobs) run server-side — there is no local/OSS equivalent, so
resolve a server connection before attempting any job work. The job REST methods
themselves are documented in `references/remote_jobs.md`.
Every request needs two things:
1. **Base URL** — the server endpoint
2. **Credentials** — an API key (`x-api-key` header over REST), and usually a database name (`x-lancedb-database` header)
## Resolution steps
1. If the user already gave a URL and API key (or said which environment they're working against), use that.
2. Otherwise, look for credentials already available in the environment:
- Env vars like `LANCEDB_URI` / `LANCEDB_HOST` / `LANCEDB_API_KEY`
- A server endpoint already running or port-forwarded locally (the REST default port is 2333, i.e. `http://localhost:2333`)
3. If you didn't find both pieces, ask the user directly: **"What's your LanceDB endpoint's URL, and what's your API key?"** Also ask which database to use if it isn't obvious. Don't guess or probe further — the user knows their deployment.
## Validating the connection
Make a cheap authenticated request and check the status before starting real work:
@@ -8,6 +8,27 @@ A Python library for [LanceDB](https://github.com/lancedb/lancedb).
pip install lancedb
```
### Pre-Haswell x86_64 hosts: `lancedb-compat`
The default `lancedb` wheel targets `x86-64-haswell` (AVX2 + FMA + F16C) for full performance on modern hardware. Pre-Haswell hosts — Intel Sandy Bridge / Ivy Bridge / Westmere; AMD Bulldozer / Piledriver / Steamroller — don't have AVX2 and crash with `Illegal instruction` at `import lancedb`.
For those hosts, install the `lancedb-compat` package instead:
```bash
pip install lancedb-compat
```
Same Python API (`import lancedb` works as usual). The compat wheel is compiled at the `x86-64-v2` baseline (Nehalem-class) and uses runtime SIMD dispatch in the embedded lance crate to pick the right kernel tier (scalar / AVX / AVX+FMA / AVX2+FMA / AVX-512) at load time, so it still goes fast on modern hardware while running cleanly on the pre-Haswell silicon. Use `lance.simd_info()` from Python to verify which tier was selected.
`lancedb` and `lancedb-compat` install to the same `lancedb/` namespace and conflict at install time. Pick one. To switch, `pip uninstall lancedb` first, then `pip install lancedb-compat` (or vice-versa).
If you need a custom baseline (or `lancedb-compat` isn't yet published for your platform), build from source with the override:
Stable releases are created about every 2 weeks. For the latest features and bug fixes, you can install the preview release. These releases receive the same level of testing as stable releases, but are not guaranteed to be available for more than 6 months after they are released. Once your application is stable, we recommend switching to stable releases.
source_instruction:str,default"represent the document for retrieval"
source_instruction:str,default"represent the document for retrieval"
Theinstructionforthesourcecolumn
query_instruction:str,default"represent the document for retrieving the most
query_instruction:str,default"represent the document for retrieving the most
similardocuments"
Theinstructionforthequery
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Will Jones
Yang Cen
Lance Release
buduoqiu
kid
dependabot[bot]
Dan Tasse
Xuanwo
Vivek
LanceDB Robot
Heng Ge
Andrew Chen
Eran Dagan
Prashanth Rao
Expyron
Mateusz Szewczyk
Drew Gallardo
Vitaliy
Kobi Hikri
Tobias
Jack Ye
