Compare commits

..
Author SHA1 Message Date
dependabot[bot] 13f9dd630b build(deps): bump prost from 0.14.3 to 0.14.4 in the rust-minor-patch group (#4135)
Bumps the rust-minor-patch group with 1 update:
[prost](https://github.com/tokio-rs/prost).

Updates `prost` from 0.14.3 to 0.14.4
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/tokio-rs/prost/blob/master/CHANGELOG.md">prost's
changelog</a>.</em></p>
<blockquote>
<h1>Prost version 0.14.4</h1>
<p><em>PROST!</em> is a <a href="https://protobuf.dev/">Protocol
Buffers</a> implementation for the <a
href="https://www.rust-lang.org/">Rust Language</a>. <code>prost</code>
generates simple, idiomatic Rust code from <code>proto2</code> and
<code>proto3</code> files.</p>
<h3>🚀 Features</h3>
<ul>
<li><em>(prost-derive)</em> Make is_valid a constant function (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1401">#1401</a>)</li>
<li>Increase MSRV to 1.85 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1428">#1428</a>)</li>
</ul>
<h3>🐛 Bug Fixes</h3>
<ul>
<li>Use Display instead of Debug for generated enumeration attributes
(<a
href="https://redirect.github.com/tokio-rs/prost/issues/1419">#1419</a>)</li>
<li><em>(prost-derive)</em> Return error for invalid enumeration default
identifiers (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1426">#1426</a>)</li>
<li><em>(build)</em> Grab binary path from cargo (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1429">#1429</a>)</li>
<li><em>(build)</em> Fix C++ build on GCC 15 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1395">#1395</a>)</li>
</ul>
<h3>📚 Documentation</h3>
<ul>
<li>Add example for <code>decode_length_delimiter</code> (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1311">#1311</a>)</li>
<li>Update protobuf-src example to avoid unsafe set_var</li>
</ul>
<h3>🧪 Testing</h3>
<ul>
<li>Test derive Eq behavior (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1422">#1422</a>)</li>
<li><em>(groups)</em> Actually construct <code>NestedGroup</code> (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1363">#1363</a>)</li>
</ul>
<h3>💼 Dependencies</h3>
<ul>
<li><em>(deps)</em> Update criterion requirement from 0.7 to 0.8 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1374">#1374</a>)</li>
<li><em>(deps)</em> Remove <code>getrandom@0.4.1</code> from
build-dependencies (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1400">#1400</a>)</li>
<li><em>(deps)</em> Update rand requirement from 0.9 to 0.10 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1397">#1397</a>)</li>
<li><em>(deps)</em> Bump actions/upload-artifact from 6 to 7 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1409">#1409</a>)</li>
<li><em>(deps)</em> Update <code>cargo clippy</code> to 1.89 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1433">#1433</a>)</li>
<li><em>(deps)</em> Update <code>cargo clippy</code> to 1.91 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1435">#1435</a>)</li>
<li><em>(deps)</em> Update and improve nix devshell (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1393">#1393</a>)</li>
</ul>
<h3>🎨 Styling</h3>
<ul>
<li>Prevent needless borrow (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1404">#1404</a>)</li>
<li>Use <code>std::hint::black_box()</code> (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1403">#1403</a>)</li>
<li>Use variables directly in <code>format!()</code> (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1432">#1432</a>)</li>
<li>Remove explicit <code>.into_iter()</code> (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1434">#1434</a>)</li>
<li>Run clippy on benches (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1405">#1405</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/tokio-rs/prost/commit/13646cde7eab75c81b3047767aa0a86e7dbecf12"><code>13646cd</code></a>
chore: Release version 0.14.4 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1437">#1437</a>)</li>
<li><a
href="https://github.com/tokio-rs/prost/commit/dad79d5c8e3549d93ebe6f6c723bb42928d805d8"><code>dad79d5</code></a>
fix(prost-derive): return error for invalid enumeration default
identifiers (...</li>
<li><a
href="https://github.com/tokio-rs/prost/commit/b0b6c93e3aac89df28690a4967a8bbe93ec95391"><code>b0b6c93</code></a>
ci: Update <code>cargo clippy</code> to 1.91 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1435">#1435</a>)</li>
<li><a
href="https://github.com/tokio-rs/prost/commit/32cfffbc494f2faf461cab85e04a42412484c0e4"><code>32cfffb</code></a>
style: remove explicit <code>.into_iter()</code> (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1434">#1434</a>)</li>
<li><a
href="https://github.com/tokio-rs/prost/commit/2710efdb9978d9c75fb19b0b092a369a2d385b55"><code>2710efd</code></a>
ci: Update <code>cargo clippy</code> to 1.89 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1433">#1433</a>)</li>
<li><a
href="https://github.com/tokio-rs/prost/commit/18ea4e42bbc307d33d65e05ad47b3c45623c0500"><code>18ea4e4</code></a>
style: use variables directly in <code>format!()</code> (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1432">#1432</a>)</li>
<li><a
href="https://github.com/tokio-rs/prost/commit/2821bd1d8c20137c83ead4db39f8e1da00b4e854"><code>2821bd1</code></a>
build(deps): bump actions/upload-artifact from 6 to 7 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1409">#1409</a>)</li>
<li><a
href="https://github.com/tokio-rs/prost/commit/3ce3b39f9206b5e3bbe34c6e1aa69fe3c53f0924"><code>3ce3b39</code></a>
test(groups): Actually construct <code>NestedGroup</code> (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1363">#1363</a>)</li>
<li><a
href="https://github.com/tokio-rs/prost/commit/8776405574b3ba0a0fe96ada8799ac8bc61ceb3e"><code>8776405</code></a>
docs: Update changelog for version 0.14.3 (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1431">#1431</a>)</li>
<li><a
href="https://github.com/tokio-rs/prost/commit/33d3ef18c008da13e862d7e7674d751ab2776360"><code>33d3ef1</code></a>
build: Grab binary path from cargo (<a
href="https://redirect.github.com/tokio-rs/prost/issues/1429">#1429</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/tokio-rs/prost/compare/v0.14.3...v0.14.4">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=prost&package-manager=cargo&previous-version=0.14.3&new-version=0.14.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-10 10:15:08 -07:00
LanceDB Robot bc4497b21a chore: update lance dependency to v12.0.0-beta.16 (#4156)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core from v12.0.0-beta.15 to
[v12.0.0-beta.16](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.16).
No compatibility fixes were required; `cargo clippy --quiet --workspace
--tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and
`git diff --check` passed.
2026-09-10 00:32:31 -07:00
Will JonesandClaude Sonnet 5 1da5876870 ci: add spell checking (#4148)
Adds [typos](https://github.com/crate-ci/typos) as a CI check and
pre-commit hook, the same way Lance does it, so misspellings like the
ones fixed in #4146 get caught automatically going forward.

This also fixes the misspellings `typos` found across the repo (Rust,
Python, TypeScript source, comments, and generated docs), and adds a
small `.typos.toml` with `extend-words` entries for terms that are
correct but look like typos: `AKS` (Azure Kubernetes Service), `RabitQ`
(a real quantization algorithm name), `mmaped` (the actual name of a
`candle-core` API we call), and `Writeable` (from Python's
`_typeshed.WriteableBuffer`). Third-party license files are excluded.

Fixes #4147

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-09 15:33:04 +08:00
Madan Kumar 577fb48376 fix(python): apply offset when combining async hybrid results (#4028)
Fixes #4027

## Summary

`AsyncHybridQuery`
(`table.query().nearest_to(...).nearest_to_text(...)`) paginates
incorrectly when `.offset()` is used: the second page repeats rows from
the first page and silently drops others.

`offset()` on a hybrid query pushes the offset down into *both*
sub-queries (`HybridQuery::offset` in `python/src/query.rs` forwards to
`inner_vec` and `inner_fts`), so each sub-query independently skips its
own first `offset` rows before the results are fused.
`AsyncHybridQuery.to_batches` then called `_combine_hybrid_results(...,
limit=self._inner.get_limit())` without an `offset`, so the reranked
table was sliced starting at position 0 and the sub-query limits were
never raised to cover the skipped prefix.

On the 4-row fixture in `test_hybrid_query.py`, with `_rowid` ordering
`[3, 0, 2, 1]`:

| query | before | after |
| --- | --- | --- |
| `.limit(2)` | `[0, 3]` | `[0, 3]` |
| `.offset(2).limit(2)` | `[3, 1]` | `[2, 1]` |

Row `3` was returned on both pages and row `2` was never returned at
all.

This is the async counterpart of #3769 (`Fixes #3765`), which fixed the
same bug in the synchronous `LanceHybridQueryBuilder`. #3765 explicitly
deferred the async path; this PR closes that gap and reuses the `offset`
parameter that #3769 already added to `_combine_hybrid_results`. The
synchronous path is unaffected — it was fixed in #3769.

## Changes

`python/python/lancedb/query.py`, `AsyncHybridQuery.to_batches`:

- Each sub-query now fetches `limit + offset` rows and its own offset is
reset to 0, so the fused result contains the full prefix the window is
sliced out of.
- The combined, reranked table is sliced with `offset=` instead of
always starting at 0.

Both halves are needed: raising the sub-query limits without the final
slice still returns page 1, and slicing without raising the limits still
misses rows.

`nodejs` has no equivalent hybrid combine path, so there is no SDK
parity gap here.

## Test plan

- [x] New regression test `test_async_hybrid_query_offset` in
`python/python/tests/test_hybrid_query.py`, mirroring the sync
`test_hybrid_query_offset`. It asserts the offset window is a suffix of
the un-offset result *and* that page 1 + page 2 together cover every row
exactly once (a row-count-only assertion would pass even with
duplicates).
- [x] `pytest python/tests/test_hybrid_query.py` — 16 passed
- [x] `pytest python/tests/test_rerankers.py` — 9 passed, 11 skipped
- [x] `pytest python/tests/test_query.py` — 86 passed
- [x] `pytest --doctest-modules python/lancedb/query.py` — 13 passed
- [x] `ruff format --check` / `ruff check` — clean
---

## Scope, after review

@lancedb-gatekeeper raised three points. Two were mine and are fixed in
`04d07c2`; the third is deliberately left alone and I'd like a
maintainer's call on it.

**Fixed — effective limit was read from the FTS child only.**
`HybridQuery::get_limit()` (`python/src/query.rs:1159`) returns
`self.inner_fts.inner.current_request().limit`, so an FTS-first hybrid
with no explicit `.limit()` yielded `None`, skipped the widening branch
and passed `limit=None` to the combiner — returning the union of both
candidate lists instead of the documented default of 10. The limit is
now derived from both children with a `DEFAULT_HYBRID_LIMIT = 10`
fallback, so construction order no longer matters.

**Fixed — `explain_plan()` / `analyze_plan()` described a different
query than the one that ran.** Both built their children straight from
`self._inner`, bypassing the limit/offset rewrite in `to_batches`, and
reported `skip=2, fetch=2` while execution used `skip=0, fetch=4`. Child
preparation now lives in one `_create_child_queries()` helper used by
all three.

> **Visible change to `explain_plan()` output:** because the plan is now
built from the real execution children, which carry `with_row_id()`, the
two `ProjectionExec` lines gain a `_rowid` column. The doctest is
updated to match. This is the diagnostic becoming truthful rather than
the assertion being weakened — it is still an exact-match comparison.

**Not fixed here — RRF candidate-pool invariance.** Widening each
sub-query to `limit + offset` does change the candidate pool between
page requests, so the fused ranking can shift and pagination can still
repeat rows. That's a real problem, but it is exactly what the merged
sync path does today:

```python
# LanceHybridQueryBuilder (sync), merged in #3769
sub_query_limit = self._limit + (self._offset or 0)
```

Making the pool invariant means choosing a contract — a fixed candidate
pool, or an explicit cursor — and that ought to apply to sync and async
together rather than letting the two paths diverge. I've asked in the
review thread which way you'd prefer, and I'm happy to do it here or in
a follow-up covering both paths.

So, to be precise about what this PR delivers: it makes `.offset()` take
effect on the async hybrid path and makes the diagnostics honest. It
does not make hybrid pagination stable across pages under reranking —
that needs the contract decision above.
2026-09-08 15:42:06 -07:00
陈志谦 c7b051aff7 docs: fix spelling typos across python package docstrings (#4146)
Six files carried spelling typos in user-visible docstrings:

- `table.py` (×3) + `remote/table.py`: "The **targetted** vector to
search for" → "targeted"
- `query.py`: "pa.Array **wouln't** be allowed" → "wouldn't"
- `embeddings/gte.py`: "mlx package **insalled**" → "installed"
- `rerankers/base.py`: "This is **inteded**" → "intended"
- `index.py`: "dimension **divded** by 8" → "divided"

Docstrings only.
2026-09-08 13:53:38 -07:00
Lance Release 2e205ac9bb Bump version: 0.39.0-beta.5 → 0.39.0-beta.6 2026-09-08 12:14:45 +00:00
LanceDB RobotandJack Ye 3e3878b223 chore: update lance dependency to v12.0.0-beta.15 (#4143)
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core from v12.0.0-beta.14 to
[v12.0.0-beta.15](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.15).
No compatibility fixes were required; `cargo clippy --quiet --workspace
--tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and
`git diff --check` passed.

---------

Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
2026-09-08 05:12:45 -07:00
Lance Release 19fb665c76 Bump version: 0.39.0-beta.4 → 0.39.0-beta.5 2026-09-08 12:03:46 +00:00
Wyatt AltandClaude Fable 5.1 1f95398c34 feat: function columns on materialized views (#4119)
A view could not carry a column it does not compute: the definition
planned every output as a SQL expression, and the refresh engine treated
any commit it did not make as drift and rebuilt. Servers fill such
columns on tables with a separate job, as computed columns bound to a
registered function, and want the same column on a view.

This lets a declaration add computed columns, placed at their positions
in the select list and validated by the existing computed-column
contract, with the view created in one commit. Refresh writes those
columns NULL on every path and never reads them, so a rewritten row
comes back unfilled, and a commit that rewrites only computed columns is
recognised as a fill rather than drift, so the next refresh carries on
incrementally. A source column a computed column reads without the view
projecting it is held as an internal projection, so the select list
stays the view's column list. Nothing in the stored definition changes;
an older reader fails closed on the schema check.

Two smaller changes ride along because the feature needs them: an
identity projection keeps its source column's nullability, with the
schema check accepting a nullable physical field for a non-null planned
one so existing views keep refreshing; and `prepare_declaration` takes
`Option` projections, so an empty list declares no projection rather
than `SELECT *`.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 04:38:22 -07:00
Lance Release 0111a72dc3 Bump version: 0.39.0-beta.3 → 0.39.0-beta.4 2026-09-06 20:46:55 +00:00
LanceDB RobotandJack Ye a487d4033e chore: update lance dependency to v12.0.0-beta.14 (#4141)
Update the Rust workspace Lance dependencies and Java lance-core from
v12.0.0-beta.11 to
[v12.0.0-beta.14](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.14),
refreshing Cargo.lock. Resolve two Clippy diagnostics by making an
internal Node.js helper private and using a byte string literal in a
remote-table test fixture.

Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, `git diff --check`, and `pnpm
build` in nodejs.

---------

Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
2026-09-06 13:43:35 -07:00
dependabot[bot] 02ea0dda9f build(deps-dev): bump the nodejs-deps group across 1 directory with 2 updates (#4134)
Bumps the nodejs-deps group with 2 updates in the /nodejs directory:
[@opentelemetry/sdk-metrics](https://github.com/open-telemetry/opentelemetry-js)
and [ts-jest](https://github.com/kulshekhar/ts-jest).

Updates `@opentelemetry/sdk-metrics` from 2.10.0 to 2.11.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/open-telemetry/opentelemetry-js/releases">@​opentelemetry/sdk-metrics's
releases</a>.</em></p>
<blockquote>
<h2>v2.11.0</h2>
<h2>2.11.0</h2>
<h3>🚀 Features</h3>
<ul>
<li>feat(context-async-hooks): implement <code>attach()</code> on
<code>AsyncLocalStorageContextManager</code> <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6845">#6845</a>
<a href="https://github.com/pichlermarc"><code>@​pichlermarc</code></a>
<ul>
<li>On Node.js 25.9+, delegates to
<code>AsyncLocalStorage.withScope()</code> returning a native
<code>RunScope</code>. On older Node.js, falls back to
<code>enterWith()</code> with a manual disposable wrapper.</li>
</ul>
</li>
<li>feat(sdk-trace): allow configuring the force flush timeout per call
<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6929">#6929</a>
<a
href="https://github.com/LarryHu0217"><code>@​LarryHu0217</code></a></li>
</ul>
<h3>🐛 Bug Fixes</h3>
<ul>
<li>fix(sdk-metrics): ignore <code>Infinity</code> in exponential
histograms <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7015">#7015</a>
<a href="https://github.com/mwear"><code>@​mwear</code></a></li>
</ul>
<h3>🏠 Internal</h3>
<ul>
<li>perf(sdk-metrics): reuse a single DataView for exponential histogram
bit reads <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6998">#6998</a>
<a href="https://github.com/mwear"><code>@​mwear</code></a></li>
<li>chore(ci): run documentation tests on a weekly schedule <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6920">#6920</a>
<a
href="https://github.com/LarryHu0217"><code>@​LarryHu0217</code></a></li>
<li>feat(ci): support pre-releases and major version bumps in the
release workflow <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6768">#6768</a>
<a
href="https://github.com/pichlermarc"><code>@​pichlermarc</code></a></li>
<li>chore(resources): Ensure that multiple uses of
serviceInstanceIdDetector.detect() return the <em>same</em> value for
<code>service.instance.id</code></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/open-telemetry/opentelemetry-js/blob/main/CHANGELOG.md">@​opentelemetry/sdk-metrics's
changelog</a>.</em></p>
<blockquote>
<h2>2.11.0</h2>
<h3>🚀 Features</h3>
<ul>
<li>feat(context-async-hooks): implement <code>attach()</code> on
<code>AsyncLocalStorageContextManager</code> <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6845">#6845</a>
<a href="https://github.com/pichlermarc"><code>@​pichlermarc</code></a>
<ul>
<li>On Node.js 25.9+, delegates to
<code>AsyncLocalStorage.withScope()</code> returning a native
<code>RunScope</code>. On older Node.js, falls back to
<code>enterWith()</code> with a manual disposable wrapper.</li>
</ul>
</li>
<li>feat(sdk-trace): allow configuring the force flush timeout per call
<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6929">#6929</a>
<a
href="https://github.com/LarryHu0217"><code>@​LarryHu0217</code></a></li>
</ul>
<h3>🐛 Bug Fixes</h3>
<ul>
<li>fix(sdk-trace-base): avoid a Webpack self-reference error in
CommonJS output <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6981">#6981</a>
<a href="https://github.com/sansynx"><code>@​sansynx</code></a></li>
<li>fix(sdk-metrics): ignore <code>Infinity</code> in exponential
histograms <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7015">#7015</a>
<a href="https://github.com/mwear"><code>@​mwear</code></a></li>
</ul>
<h3>🏠 Internal</h3>
<ul>
<li>perf(sdk-metrics): reuse a single DataView for exponential histogram
bit reads <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6998">#6998</a>
<a href="https://github.com/mwear"><code>@​mwear</code></a></li>
<li>chore(ci): run documentation tests on a weekly schedule <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6920">#6920</a>
<a
href="https://github.com/LarryHu0217"><code>@​LarryHu0217</code></a></li>
<li>feat(ci): support pre-releases and major version bumps in the
release workflow <a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6768">#6768</a>
<a
href="https://github.com/pichlermarc"><code>@​pichlermarc</code></a></li>
<li>chore(resources): Ensure that multiple uses of
serviceInstanceIdDetector.detect() return the <em>same</em> value for
<code>service.instance.id</code></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/open-telemetry/opentelemetry-js/commit/0b72a81636fa476e8f1f1afd2ae0c90a1362194c"><code>0b72a81</code></a>
chore: prepare next release (<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7044">#7044</a>)</li>
<li><a
href="https://github.com/open-telemetry/opentelemetry-js/commit/a9c5338a9f485f1433df9308f24bd7397a4a0321"><code>a9c5338</code></a>
ci: roll prerelease changelog into one final release changelog (<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7045">#7045</a>)</li>
<li><a
href="https://github.com/open-telemetry/opentelemetry-js/commit/f41805e769ba10fb6dae72a4b7a5a3dc67cca82e"><code>f41805e</code></a>
chore: prepare next release (<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7042">#7042</a>)</li>
<li><a
href="https://github.com/open-telemetry/opentelemetry-js/commit/b85eb28343ff6234e2bb4d46b7b4a3d360e5ea2f"><code>b85eb28</code></a>
chore(instrumentation-http): fix lint errors (<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7039">#7039</a>)</li>
<li><a
href="https://github.com/open-telemetry/opentelemetry-js/commit/3f9253009be2ae48419432e79a597ead7be8be6a"><code>3f92530</code></a>
ci: support pre-releases and major version bumps in release workflow (<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7035">#7035</a>)</li>
<li><a
href="https://github.com/open-telemetry/opentelemetry-js/commit/82a58316a63b6fde62afd268e145b82222d328cf"><code>82a5831</code></a>
docs(otlp-exporter-base): document HTTP exporter options (<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6735">#6735</a>)</li>
<li><a
href="https://github.com/open-telemetry/opentelemetry-js/commit/e086dec7f9304107ef6d50b5877be88895c06aa7"><code>e086dec</code></a>
Merge commit from fork</li>
<li><a
href="https://github.com/open-telemetry/opentelemetry-js/commit/59dac70d00d46fa56b2b921cf721fd922730f23d"><code>59dac70</code></a>
chore(deps): update jamesives/github-pages-deploy-action action to
v4.9.0 (<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7">#7</a>...</li>
<li><a
href="https://github.com/open-telemetry/opentelemetry-js/commit/d0ce7532b058631ec9eec111c04fefe7fd873e1f"><code>d0ce753</code></a>
chore: add <a
href="https://github.com/maryliag"><code>@​maryliag</code></a> to
maintainers (<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7024">#7024</a>)</li>
<li><a
href="https://github.com/open-telemetry/opentelemetry-js/commit/03469a129f97265c8eec93d566e9e00d4f741db3"><code>03469a1</code></a>
chore(deps): update open-telemetry/shared-workflows action to v0.10.0
(<a
href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7032">#7032</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/open-telemetry/opentelemetry-js/compare/v2.10.0...v2.11.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `ts-jest` from 29.4.9 to 29.4.12
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/kulshekhar/ts-jest/releases">ts-jest's
releases</a>.</em></p>
<blockquote>
<h2>v29.4.12</h2>
<p>Please refer to <a
href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">CHANGELOG.md</a>
for details.</p>
<h2>v29.4.11</h2>
<p>Please refer to <a
href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">CHANGELOG.md</a>
for details.</p>
<h2>v29.4.10</h2>
<p>Please refer to <a
href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">CHANGELOG.md</a>
for details.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">ts-jest's
changelog</a>.</em></p>
<blockquote>
<h2><a
href="https://github.com/kulshekhar/ts-jest/compare/v29.4.11...v29.4.12">29.4.12</a>
(2026-07-22)</h2>
<h3>Features</h3>
<ul>
<li><strong>compiler:</strong> support TypeScript 7 projects through
compatibility aliases (<a
href="https://redirect.github.com/kulshekhar/ts-jest/pull/5386">#5386</a>)</li>
</ul>
<h2><a
href="https://github.com/kulshekhar/ts-jest/compare/v29.4.10...v29.4.11">29.4.11</a>
(2026-05-21)</h2>
<h3>Bug Fixes</h3>
<ul>
<li>preserve Bundler on the CJS path under TypeScript &gt;= 6 (<a
href="https://github.com/kulshekhar/ts-jest/commit/39418187515f11b6584d35a4e3ddf50231f74936">3941818</a>),
closes <a
href="https://redirect.github.com/kulshekhar/ts-jest/issues/4198">#4198</a></li>
</ul>
<h2><a
href="https://github.com/kulshekhar/ts-jest/compare/v29.4.9...v29.4.10">29.4.10</a>
(2026-05-18)</h2>
<h3>Bug Fixes</h3>
<ul>
<li>pass <code>resolutionMode</code> to
<code>ts.resolveModuleName</code> for hybrid module support (<a
href="https://github.com/kulshekhar/ts-jest/commit/b557a85f85c3fd34523ec3a15293afbdc9dea83c">b557a85</a>)</li>
<li>rebuild <code>Program</code> when consecutive compiles need
different module kinds (<a
href="https://github.com/kulshekhar/ts-jest/commit/a82a2b32c4987a5249fd5284283117dd2fa3be47">a82a2b3</a>),
closes <a
href="https://redirect.github.com/kulshekhar/ts-jest/issues/4774">#4774</a></li>
<li>respect tsconfig <code>moduleResolution</code> instead of forcing
<code>Node10</code> (<a
href="https://github.com/kulshekhar/ts-jest/commit/1bffffc667557c173ae0c1f93dd436920775dac4">1bffffc</a>)</li>
<li><strong>transformer:</strong> transpile <code>mjs</code> files from
<code>node_modules</code> for CJS mode (<a
href="https://github.com/kulshekhar/ts-jest/commit/96d025dd912ea2bceb18b67d2d509ada7a756d9d">96d025d</a>)</li>
<li><strong>transformer:</strong> use a consistent comparator in
hoist-jest sortStatements (<a
href="https://github.com/kulshekhar/ts-jest/commit/8a8fd2fb8446655bba18367db9306a1089490e62">8a8fd2f</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/kulshekhar/ts-jest/commit/3f05625da10da954fdf0a10394385008275ddbb3"><code>3f05625</code></a>
chore(release): 29.4.12</li>
<li><a
href="https://github.com/kulshekhar/ts-jest/commit/df28b27f2e60edf275866763a3cdf745360d3eae"><code>df28b27</code></a>
docs: clarify TypeScript version prerequisites</li>
<li><a
href="https://github.com/kulshekhar/ts-jest/commit/c8a614af419b1dcc90f3d1a7a48238ac1b637e6b"><code>c8a614a</code></a>
docs: mention TypeScript 7 setup in README</li>
<li><a
href="https://github.com/kulshekhar/ts-jest/commit/06c79d4cebfa785749b2f96ef2dbeffc12798c47"><code>06c79d4</code></a>
fix: address TypeScript 7 review feedback</li>
<li><a
href="https://github.com/kulshekhar/ts-jest/commit/f10746008e512203ad7d8c581e2c58cc7dcd43c8"><code>f107460</code></a>
docs: explain TypeScript 7 compatibility setup</li>
<li><a
href="https://github.com/kulshekhar/ts-jest/commit/33882274c1e4fc3a06cece3b21f4873182c8fee7"><code>3388227</code></a>
test(e2e): add TypeScript compatibility matrix</li>
<li><a
href="https://github.com/kulshekhar/ts-jest/commit/891dc731ff9f31785ad9698e4d7cfa6078991fe6"><code>891dc73</code></a>
fix(compiler): support TypeScript 7 compatibility aliases</li>
<li><a
href="https://github.com/kulshekhar/ts-jest/commit/eb135ebe819991b1e10c998915cc6db2057c4de1"><code>eb135eb</code></a>
build(deps-dev): bump shell-quote from 1.8.4 to 1.10.0 in /examples</li>
<li><a
href="https://github.com/kulshekhar/ts-jest/commit/d5d80a34bc575be130db2b9f5cd0958173982cda"><code>d5d80a3</code></a>
ci: pin google osv scan action at v2.3.5</li>
<li><a
href="https://github.com/kulshekhar/ts-jest/commit/6bf293f0a4ddf468735d81fcdf04f923278e030c"><code>6bf293f</code></a>
build(deps): bump shell-quote from 1.8.4 to 1.10.0 in /website</li>
<li>Additional commits viewable in <a
href="https://github.com/kulshekhar/ts-jest/compare/v29.4.9...v29.4.12">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-06 09:41:24 -07:00
68 changed files with 2211 additions and 299 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
[tool.bumpversion] [tool.bumpversion]
current_version = "0.39.0-beta.3" current_version = "0.39.0-beta.6"
parse = """(?x) parse = """(?x)
(?P<major>0|[1-9]\\d*)\\. (?P<major>0|[1-9]\\d*)\\.
(?P<minor>0|[1-9]\\d*)\\. (?P<minor>0|[1-9]\\d*)\\.
+20
View File
@@ -0,0 +1,20 @@
name: Typo checker
on:
push:
branches:
- main
pull_request:
permissions:
contents: read
jobs:
run:
name: Spell Check with Typos
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v6
- name: Check spelling of the entire repository
uses: crate-ci/typos@6802cc60d4e7f78b9d5454f6cf3935c042d5e1e3 # v1.26.0
+4
View File
@@ -10,6 +10,10 @@ repos:
rev: v0.9.9 rev: v0.9.9
hooks: hooks:
- id: ruff - id: ruff
- repo: https://github.com/crate-ci/typos
rev: v1.26.0
hooks:
- id: typos
# - repo: https://github.com/RobertCraigie/pyright-python # - repo: https://github.com/RobertCraigie/pyright-python
# rev: v1.1.395 # rev: v1.1.395
# hooks: # hooks:
+19
View File
@@ -0,0 +1,19 @@
[default]
extend-ignore-re = ["(?Rm)^.*(#|//)\\s*spellchecker:disable-line$"]
[default.extend-words]
# Azure Kubernetes Service, mentioned in rust/lancedb/src/remote/oauth.rs.
AKS = "AKS"
# RabitQ is the name of a vector quantization algorithm, not a typo of "Rabbit".
Rabit = "Rabit"
# `VarBuilder::from_mmaped_safetensors` is the real (if oddly-spelled) name of
# the candle-core API we call in rust/lancedb/src/embeddings/sentence_transformers.rs.
mmaped = "mmaped"
# `WriteableBuffer` is the real name of a type from Python's `_typeshed` stubs,
# used in python/python/lancedb/_blob.py.
Writeable = "Writeable"
[files]
extend-exclude = [
"*_THIRD_PARTY_LICENSES.*",
]
Generated
+53 -51
View File
@@ -3526,8 +3526,8 @@ checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]] [[package]]
name = "fsst" name = "fsst"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"rand 0.9.5", "rand 0.9.5",
@@ -4886,8 +4886,8 @@ checksum = "e037a2e1d8d5fdbd49b16a4ea09d5d6401c1f29eca5ff29d03d3824dba16256a"
[[package]] [[package]]
name = "lance" name = "lance"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arc-swap", "arc-swap",
"arrow", "arrow",
@@ -4959,8 +4959,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-arrow" name = "lance-arrow"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"arrow-buffer", "arrow-buffer",
@@ -4982,7 +4982,7 @@ dependencies = [
[[package]] [[package]]
name = "lance-arrow-scalar" name = "lance-arrow-scalar"
version = "58.0.0" version = "58.0.0"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"arrow-buffer", "arrow-buffer",
@@ -4996,7 +4996,7 @@ dependencies = [
[[package]] [[package]]
name = "lance-arrow-stats" name = "lance-arrow-stats"
version = "58.0.0" version = "58.0.0"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"arrow-schema", "arrow-schema",
@@ -5005,8 +5005,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-bitpacking" name = "lance-bitpacking"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrayref", "arrayref",
"crunchy", "crunchy",
@@ -5016,8 +5016,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-core" name = "lance-core"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"arrow-buffer", "arrow-buffer",
@@ -5054,8 +5054,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-datafusion" name = "lance-datafusion"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow", "arrow",
"arrow-array", "arrow-array",
@@ -5085,8 +5085,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-datagen" name = "lance-datagen"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow", "arrow",
"arrow-array", "arrow-array",
@@ -5103,8 +5103,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-derive" name = "lance-derive"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"proc-macro2", "proc-macro2",
"quote", "quote",
@@ -5113,8 +5113,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-encoding" name = "lance-encoding"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-arith", "arrow-arith",
"arrow-array", "arrow-array",
@@ -5147,8 +5147,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-file" name = "lance-file"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-arith", "arrow-arith",
"arrow-array", "arrow-array",
@@ -5179,8 +5179,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-index" name = "lance-index"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arc-swap", "arc-swap",
"arrow", "arrow",
@@ -5244,8 +5244,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-index-core" name = "lance-index-core"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"arrow-schema", "arrow-schema",
@@ -5267,8 +5267,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-io" name = "lance-io"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow", "arrow",
"arrow-array", "arrow-array",
@@ -5308,8 +5308,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-linalg" name = "lance-linalg"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"arrow-schema", "arrow-schema",
@@ -5323,21 +5323,23 @@ dependencies = [
[[package]] [[package]]
name = "lance-namespace" name = "lance-namespace"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow", "arrow",
"async-trait", "async-trait",
"bytes", "bytes",
"lance-core", "lance-core",
"lance-namespace-reqwest-client", "lance-namespace-reqwest-client",
"serde",
"serde_json",
"snafu 0.9.0", "snafu 0.9.0",
] ]
[[package]] [[package]]
name = "lance-namespace-impls" name = "lance-namespace-impls"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow", "arrow",
"arrow-ipc", "arrow-ipc",
@@ -5376,9 +5378,9 @@ dependencies = [
[[package]] [[package]]
name = "lance-namespace-reqwest-client" name = "lance-namespace-reqwest-client"
version = "0.11.1" version = "0.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d06b1fbb5d41f93bc652b61e2872af92e8a6c5f6b4ce8839a8ecfa05365d359" checksum = "d8d23e54b1634d5bbb434f8dd33dc3c05f6e58d876a9a27b3b4aef58ddbe11af"
dependencies = [ dependencies = [
"reqwest 0.12.28", "reqwest 0.12.28",
"serde", "serde",
@@ -5390,8 +5392,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-select" name = "lance-select"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"arrow-buffer", "arrow-buffer",
@@ -5405,8 +5407,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-table" name = "lance-table"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow", "arrow",
"arrow-array", "arrow-array",
@@ -5446,8 +5448,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-testing" name = "lance-testing"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"arrow-schema", "arrow-schema",
@@ -5460,8 +5462,8 @@ dependencies = [
[[package]] [[package]]
name = "lance-tokenizer" name = "lance-tokenizer"
version = "12.0.0-beta.11" version = "12.0.0-beta.16"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.11#4a0e26895729feb86d0cb9c09d551bfd619c6472" source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.16#f7df098f5860cf9b3ac5339e7a641430cfbc6351"
dependencies = [ dependencies = [
"frostem", "frostem",
"icu_segmenter", "icu_segmenter",
@@ -5474,7 +5476,7 @@ dependencies = [
[[package]] [[package]]
name = "lancedb" name = "lancedb"
version = "0.39.0-beta.3" version = "0.39.0-beta.6"
dependencies = [ dependencies = [
"ahash", "ahash",
"anyhow", "anyhow",
@@ -5565,7 +5567,7 @@ dependencies = [
[[package]] [[package]]
name = "lancedb-nodejs" name = "lancedb-nodejs"
version = "0.39.0-beta.3" version = "0.39.0-beta.6"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"arrow-buffer", "arrow-buffer",
@@ -5590,7 +5592,7 @@ dependencies = [
[[package]] [[package]]
name = "lancedb-python" name = "lancedb-python"
version = "0.39.0-beta.3" version = "0.39.0-beta.6"
dependencies = [ dependencies = [
"arrow", "arrow",
"async-trait", "async-trait",
@@ -7701,9 +7703,9 @@ dependencies = [
[[package]] [[package]]
name = "prost" name = "prost"
version = "0.14.3" version = "0.14.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d2ea70524a2f82d518bce41317d0fae74151505651af45faf1ffbd6fd33f0568" checksum = "528ac67416ff8646872a3c02cad9cc4ee5dc9f9540c9b10771855c95cb2e5ae1"
dependencies = [ dependencies = [
"bytes", "bytes",
"prost-derive", "prost-derive",
@@ -7730,9 +7732,9 @@ dependencies = [
[[package]] [[package]]
name = "prost-derive" name = "prost-derive"
version = "0.14.3" version = "0.14.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "27c6023962132f4b30eb4c172c91ce92d933da334c59c23cddee82358ddafb0b" checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"itertools 0.14.0", "itertools 0.14.0",
+14 -14
View File
@@ -13,20 +13,20 @@ categories = ["database-implementations"]
rust-version = "1.91.0" rust-version = "1.91.0"
[workspace.dependencies] [workspace.dependencies]
lance = { "version" = "=12.0.0-beta.11", default-features = false, "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance = { "version" = "=12.0.0-beta.16", default-features = false, "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-core = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-core = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-datagen = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-datagen = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-file = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-file = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-io = { "version" = "=12.0.0-beta.11", default-features = false, "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-io = { "version" = "=12.0.0-beta.16", default-features = false, "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-index = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-index = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-linalg = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-linalg = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-namespace = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace-impls = { "version" = "=12.0.0-beta.11", default-features = false, "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-namespace-impls = { "version" = "=12.0.0-beta.16", default-features = false, "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-table = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-table = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-testing = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-testing = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-datafusion = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-datafusion = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-encoding = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-encoding = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lance-arrow = { "version" = "=12.0.0-beta.11", "tag" = "v12.0.0-beta.11", "git" = "https://github.com/lance-format/lance.git" } lance-arrow = { "version" = "=12.0.0-beta.16", "tag" = "v12.0.0-beta.16", "git" = "https://github.com/lance-format/lance.git" }
lancedb = { path = "rust/lancedb", default-features = false } lancedb = { path = "rust/lancedb", default-features = false }
ahash = "0.8" ahash = "0.8"
# Note that this one does not include pyarrow # Note that this one does not include pyarrow
+1 -1
View File
@@ -155,7 +155,7 @@ paths:
vector: vector:
type: FixedSizeList type: FixedSizeList
description: | description: |
The targetted vector to search for. Required. The targeted vector to search for. Required.
vector_column: vector_column:
type: string type: string
description: | description: |
+1 -1
View File
@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
<dependency> <dependency>
<groupId>com.lancedb</groupId> <groupId>com.lancedb</groupId>
<artifactId>lancedb-core</artifactId> <artifactId>lancedb-core</artifactId>
<version>0.39.0-beta.3</version> <version>0.39.0-beta.6</version>
</dependency> </dependency>
``` ```
+1 -1
View File
@@ -141,7 +141,7 @@ Currently this causes multiple copies of the row to be created
but that behavior is subject to change. but that behavior is subject to change.
An optional condition may be specified. If it is, then only An optional condition may be specified. If it is, then only
matched rows that satisfy the condtion will be updated. Any matched rows that satisfy the condition will be updated. Any
rows that do not satisfy the condition will be left as they rows that do not satisfy the condition will be left as they
are. Failing to satisfy the condition does not cause a are. Failing to satisfy the condition does not cause a
"matched row" to become a "not matched" row. "matched row" to become a "not matched" row.
+1 -1
View File
@@ -1266,7 +1266,7 @@ value is 0")
Note: if your condition is something like "some_id_column == 7" and Note: if your condition is something like "some_id_column == 7" and
you are updating many rows (with different ids) then you will get you are updating many rows (with different ids) then you will get
better performance with a single [`merge_insert`] call instead of better performance with a single [`merge_insert`] call instead of
repeatedly calilng this method. repeatedly calling this method.
##### Parameters ##### Parameters
+1 -1
View File
@@ -118,7 +118,7 @@ Number of sub-vectors of PQ.
This value controls how much the vector is compressed during the quantization step. This value controls how much the vector is compressed during the quantization step.
The more sub vectors there are the less the vector is compressed. The default is The more sub vectors there are the less the vector is compressed. The default is
the dimension of the vector divided by 16. If the dimension is not evenly divisible the dimension of the vector divided by 16. If the dimension is not evenly divisible
by 16 we use the dimension divded by 8. by 16 we use the dimension divided by 8.
The above two cases are highly preferred. Having 8 or 16 values per subvector allows The above two cases are highly preferred. Having 8 or 16 values per subvector allows
us to use efficient SIMD instructions. us to use efficient SIMD instructions.
+1 -1
View File
@@ -16,7 +16,7 @@ optional config: Index;
Advanced index configuration Advanced index configuration
This option allows you to specify a specfic index to create and also This option allows you to specify a specific index to create and also
allows you to pass in configuration for training the index. allows you to pass in configuration for training the index.
See the static methods on Index for details on the various index types. See the static methods on Index for details on the various index types.
+1 -1
View File
@@ -112,7 +112,7 @@ Number of sub-vectors of PQ.
This value controls how much the vector is compressed during the quantization step. This value controls how much the vector is compressed during the quantization step.
The more sub vectors there are the less the vector is compressed. The default is The more sub vectors there are the less the vector is compressed. The default is
the dimension of the vector divided by 16. If the dimension is not evenly divisible the dimension of the vector divided by 16. If the dimension is not evenly divisible
by 16 we use the dimension divded by 8. by 16 we use the dimension divided by 8.
The above two cases are highly preferred. Having 8 or 16 values per subvector allows The above two cases are highly preferred. Having 8 or 16 values per subvector allows
us to use efficient SIMD instructions. us to use efficient SIMD instructions.
+1 -1
View File
@@ -8,7 +8,7 @@
<parent> <parent>
<groupId>com.lancedb</groupId> <groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId> <artifactId>lancedb-parent</artifactId>
<version>0.39.0-beta.3</version> <version>0.39.0-beta.6</version>
<relativePath>../pom.xml</relativePath> <relativePath>../pom.xml</relativePath>
</parent> </parent>
+2 -2
View File
@@ -6,7 +6,7 @@
<groupId>com.lancedb</groupId> <groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId> <artifactId>lancedb-parent</artifactId>
<version>0.39.0-beta.3</version> <version>0.39.0-beta.6</version>
<packaging>pom</packaging> <packaging>pom</packaging>
<name>${project.artifactId}</name> <name>${project.artifactId}</name>
<description>LanceDB Java SDK Parent POM</description> <description>LanceDB Java SDK Parent POM</description>
@@ -28,7 +28,7 @@
<properties> <properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<arrow.version>15.0.0</arrow.version> <arrow.version>15.0.0</arrow.version>
<lance-core.version>12.0.0-beta.11</lance-core.version> <lance-core.version>12.0.0-beta.16</lance-core.version>
<spotless.skip>false</spotless.skip> <spotless.skip>false</spotless.skip>
<spotless.version>2.30.0</spotless.version> <spotless.version>2.30.0</spotless.version>
<spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version> <spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version>
+1 -1
View File
@@ -1,7 +1,7 @@
[package] [package]
name = "lancedb-nodejs" name = "lancedb-nodejs"
edition.workspace = true edition.workspace = true
version = "0.39.0-beta.3" version = "0.39.0-beta.6"
publish = false publish = false
license.workspace = true license.workspace = true
description.workspace = true description.workspace = true
+4 -4
View File
@@ -281,7 +281,7 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
numIndices: 0, numIndices: 0,
numRows: 3, numRows: 3,
// Full on-disk size of the two data files, footers and metadata included. // Full on-disk size of the two data files, footers and metadata included.
totalBytes: 684, totalBytes: 550,
}); });
// Index files count toward totalBytes too (only deletion files and // Index files count toward totalBytes too (only deletion files and
@@ -289,7 +289,7 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
await table.createIndex("id", { config: Index.btree() }); await table.createIndex("id", { config: Index.btree() });
const statsWithIndex = await table.stats(); const statsWithIndex = await table.stats();
expect(statsWithIndex.numIndices).toBe(1); expect(statsWithIndex.numIndices).toBe(1);
expect(statsWithIndex.totalBytes).toBeGreaterThan(684); expect(statsWithIndex.totalBytes).toBeGreaterThan(550);
}); });
it("should overwrite data if asked", async () => { it("should overwrite data if asked", async () => {
@@ -3252,7 +3252,7 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
const db = await connect(tmpDir.name); const db = await connect(tmpDir.name);
const data = [ const data = [
{ text: "fa", vector: [0.1, 0.2, 0.3] }, { text: "fa", vector: [0.1, 0.2, 0.3] },
{ text: "fo", vector: [0.4, 0.5, 0.6] }, { text: "fo", vector: [0.4, 0.5, 0.6] }, // spellchecker:disable-line
{ text: "fob", vector: [0.4, 0.5, 0.6] }, { text: "fob", vector: [0.4, 0.5, 0.6] },
{ text: "focus", vector: [0.4, 0.5, 0.6] }, { text: "focus", vector: [0.4, 0.5, 0.6] },
{ text: "foo", vector: [0.4, 0.5, 0.6] }, { text: "foo", vector: [0.4, 0.5, 0.6] },
@@ -3277,7 +3277,7 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
const resultSet = new Set(fuzzyResults.map((r) => r.text)); const resultSet = new Set(fuzzyResults.map((r) => r.text));
expect(resultSet.has("foo")).toBe(true); expect(resultSet.has("foo")).toBe(true);
expect(resultSet.has("fob")).toBe(true); expect(resultSet.has("fob")).toBe(true);
expect(resultSet.has("fo")).toBe(true); expect(resultSet.has("fo")).toBe(true); // spellchecker:disable-line
expect(resultSet.has("food")).toBe(true); expect(resultSet.has("food")).toBe(true);
const prefixResults = await table const prefixResults = await table
+2 -2
View File
@@ -600,7 +600,7 @@ function makeVector(
} }
if (values.length === 0) { if (values.length === 0) {
throw Error( throw Error(
"makeVector requires at least one value or the type must be specfied", "makeVector requires at least one value or the type must be specified",
); );
} }
const sampleValue = values.find((val) => val !== null && val !== undefined); const sampleValue = values.find((val) => val !== null && val !== undefined);
@@ -858,7 +858,7 @@ async function applyEmbeddings<T>(
* customized by the `embeddingDataType` property of the embedding function. * customized by the `embeddingDataType` property of the embedding function.
* *
* If a schema is provided in `makeTableOptions` then it should include the * If a schema is provided in `makeTableOptions` then it should include the
* embedding columns. If no schema is provded then embedding columns will * embedding columns. If no schema is provided then embedding columns will
* be placed at the end of the table, after all of the input columns. * be placed at the end of the table, after all of the input columns.
*/ */
export async function convertToTable( export async function convertToTable(
+3 -3
View File
@@ -26,7 +26,7 @@ export interface IvfPqOptions {
* This value controls how much the vector is compressed during the quantization step. * This value controls how much the vector is compressed during the quantization step.
* The more sub vectors there are the less the vector is compressed. The default is * The more sub vectors there are the less the vector is compressed. The default is
* the dimension of the vector divided by 16. If the dimension is not evenly divisible * the dimension of the vector divided by 16. If the dimension is not evenly divisible
* by 16 we use the dimension divded by 8. * by 16 we use the dimension divided by 8.
* *
* The above two cases are highly preferred. Having 8 or 16 values per subvector allows * The above two cases are highly preferred. Having 8 or 16 values per subvector allows
* us to use efficient SIMD instructions. * us to use efficient SIMD instructions.
@@ -228,7 +228,7 @@ export interface HnswPqOptions {
* This value controls how much the vector is compressed during the quantization step. * This value controls how much the vector is compressed during the quantization step.
* The more sub vectors there are the less the vector is compressed. The default is * The more sub vectors there are the less the vector is compressed. The default is
* the dimension of the vector divided by 16. If the dimension is not evenly divisible * the dimension of the vector divided by 16. If the dimension is not evenly divisible
* by 16 we use the dimension divded by 8. * by 16 we use the dimension divided by 8.
* *
* The above two cases are highly preferred. Having 8 or 16 values per subvector allows * The above two cases are highly preferred. Having 8 or 16 values per subvector allows
* us to use efficient SIMD instructions. * us to use efficient SIMD instructions.
@@ -825,7 +825,7 @@ export interface IndexOptions {
/** /**
* Advanced index configuration * Advanced index configuration
* *
* This option allows you to specify a specfic index to create and also * This option allows you to specify a specific index to create and also
* allows you to pass in configuration for training the index. * allows you to pass in configuration for training the index.
* *
* See the static methods on Index for details on the various index types. * See the static methods on Index for details on the various index types.
+1 -1
View File
@@ -27,7 +27,7 @@ export class MergeInsertBuilder {
* but that behavior is subject to change. * but that behavior is subject to change.
* *
* An optional condition may be specified. If it is, then only * An optional condition may be specified. If it is, then only
* matched rows that satisfy the condtion will be updated. Any * matched rows that satisfy the condition will be updated. Any
* rows that do not satisfy the condition will be left as they * rows that do not satisfy the condition will be left as they
* are. Failing to satisfy the condition does not cause a * are. Failing to satisfy the condition does not cause a
* "matched row" to become a "not matched" row. * "matched row" to become a "not matched" row.
+1 -1
View File
@@ -3,7 +3,7 @@
// The utilities in this file help sanitize data from the user's arrow // The utilities in this file help sanitize data from the user's arrow
// library into the types expected by vectordb's arrow library. Node // library into the types expected by vectordb's arrow library. Node
// generally allows for mulitple versions of the same library (and sometimes // generally allows for multiple versions of the same library (and sometimes
// even multiple copies of the same version) to be installed at the same // even multiple copies of the same version) to be installed at the same
// time. However, arrow-js uses instanceof which expected that the input // time. However, arrow-js uses instanceof which expected that the input
// comes from the exact same library instance. This is not always the case // comes from the exact same library instance. This is not always the case
+1 -1
View File
@@ -313,7 +313,7 @@ export abstract class Table {
* Note: if your condition is something like "some_id_column == 7" and * Note: if your condition is something like "some_id_column == 7" and
* you are updating many rows (with different ids) then you will get * you are updating many rows (with different ids) then you will get
* better performance with a single [`merge_insert`] call instead of * better performance with a single [`merge_insert`] call instead of
* repeatedly calilng this method. * repeatedly calling this method.
* @param {Map<string, string> | Record<string, string>} updates - the * @param {Map<string, string> | Record<string, string>} updates - the
* columns to update * columns to update
* @returns {Promise<UpdateResult>} A promise that resolves to an object * @returns {Promise<UpdateResult>} A promise that resolves to an object
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-darwin-arm64", "name": "@lancedb/lancedb-darwin-arm64",
"version": "0.39.0-beta.3", "version": "0.39.0-beta.6",
"os": ["darwin"], "os": ["darwin"],
"cpu": ["arm64"], "cpu": ["arm64"],
"main": "lancedb.darwin-arm64.node", "main": "lancedb.darwin-arm64.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-arm64-gnu", "name": "@lancedb/lancedb-linux-arm64-gnu",
"version": "0.39.0-beta.3", "version": "0.39.0-beta.6",
"os": ["linux"], "os": ["linux"],
"cpu": ["arm64"], "cpu": ["arm64"],
"main": "lancedb.linux-arm64-gnu.node", "main": "lancedb.linux-arm64-gnu.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-arm64-musl", "name": "@lancedb/lancedb-linux-arm64-musl",
"version": "0.39.0-beta.3", "version": "0.39.0-beta.6",
"os": ["linux"], "os": ["linux"],
"cpu": ["arm64"], "cpu": ["arm64"],
"main": "lancedb.linux-arm64-musl.node", "main": "lancedb.linux-arm64-musl.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-x64-gnu", "name": "@lancedb/lancedb-linux-x64-gnu",
"version": "0.39.0-beta.3", "version": "0.39.0-beta.6",
"os": ["linux"], "os": ["linux"],
"cpu": ["x64"], "cpu": ["x64"],
"main": "lancedb.linux-x64-gnu.node", "main": "lancedb.linux-x64-gnu.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-x64-musl", "name": "@lancedb/lancedb-linux-x64-musl",
"version": "0.39.0-beta.3", "version": "0.39.0-beta.6",
"os": ["linux"], "os": ["linux"],
"cpu": ["x64"], "cpu": ["x64"],
"main": "lancedb.linux-x64-musl.node", "main": "lancedb.linux-x64-musl.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-win32-arm64-msvc", "name": "@lancedb/lancedb-win32-arm64-msvc",
"version": "0.39.0-beta.3", "version": "0.39.0-beta.6",
"os": [ "os": [
"win32" "win32"
], ],
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-win32-x64-msvc", "name": "@lancedb/lancedb-win32-x64-msvc",
"version": "0.39.0-beta.3", "version": "0.39.0-beta.6",
"os": ["win32"], "os": ["win32"],
"cpu": ["x64"], "cpu": ["x64"],
"main": "lancedb.win32-x64-msvc.node", "main": "lancedb.win32-x64-msvc.node",
+1 -1
View File
@@ -11,7 +11,7 @@
"ann" "ann"
], ],
"private": false, "private": false,
"version": "0.39.0-beta.3", "version": "0.39.0-beta.6",
"main": "dist/index.js", "main": "dist/index.js",
"exports": { "exports": {
".": "./dist/index.js", ".": "./dist/index.js",
+24 -24
View File
@@ -41,7 +41,7 @@ importers:
version: 3.7.0(@emnapi/core@1.10.0)(@emnapi/runtime@1.11.3)(@types/node@22.7.4) version: 3.7.0(@emnapi/core@1.10.0)(@emnapi/runtime@1.11.3)(@types/node@22.7.4)
'@opentelemetry/sdk-metrics': '@opentelemetry/sdk-metrics':
specifier: ^2.10.0 specifier: ^2.10.0
version: 2.10.0(@opentelemetry/api@1.9.1) version: 2.11.0(@opentelemetry/api@1.9.1)
'@types/axios': '@types/axios':
specifier: ^0.14.0 specifier: ^0.14.0
version: 0.14.4 version: 0.14.4
@@ -80,7 +80,7 @@ importers:
version: 0.2.7 version: 0.2.7
ts-jest: ts-jest:
specifier: ^29.1.2 specifier: ^29.1.2
version: 29.4.9(@babel/core@7.29.7)(@jest/transform@29.7.0)(@jest/types@29.6.3)(babel-jest@29.7.0(@babel/core@7.29.7))(jest-util@29.7.0)(jest@29.7.0(@types/node@22.7.4))(typescript@5.5.4) version: 29.4.12(@babel/core@7.29.7)(@jest/transform@29.7.0)(@jest/types@29.6.3)(babel-jest@29.7.0(@babel/core@7.29.7))(jest-util@29.7.0)(jest@29.7.0(@types/node@22.7.4))(typescript@5.5.4)
typedoc: typedoc:
specifier: 0.26.4 specifier: 0.26.4
version: 0.26.4(typescript@5.5.4) version: 0.26.4(typescript@5.5.4)
@@ -1394,20 +1394,20 @@ packages:
resolution: {integrity: sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==} resolution: {integrity: sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==}
engines: {node: '>=8.0.0'} engines: {node: '>=8.0.0'}
'@opentelemetry/core@2.10.0': '@opentelemetry/core@2.11.0':
resolution: {integrity: sha512-/wNZ8twnEQQA4HoHu22+vcsdru6pWPWxW+7w+FlxT6Id7PE/WIbZmVKkte+PF72e0F2dnImFeHD2syyE1Mw6MQ==} resolution: {integrity: sha512-7YP44XH0tV6+Mb54x2YGf84i7yi+31MBZlE8JwvozkxyTvXbSp10X7cI7YE49ChJ3shMJoBmCJF3+1QFBJctGA==}
engines: {node: ^18.19.0 || >=20.6.0} engines: {node: ^18.19.0 || >=20.6.0}
peerDependencies: peerDependencies:
'@opentelemetry/api': '>=1.0.0 <1.10.0' '@opentelemetry/api': '>=1.0.0 <1.10.0'
'@opentelemetry/resources@2.10.0': '@opentelemetry/resources@2.11.0':
resolution: {integrity: sha512-q6MMm2zhggzsHVNbabYwut+a6nbuQQe3URUoxaojM/8K1IBfwwPzvxIjNi2/lI1TFe+fMHMW9MWhrtDLEXEnkA==} resolution: {integrity: sha512-Ie7+8q8MDF4FAEQCKVMTx3ReUvxiIAgIiiW3c9JdmP8+HMcDy20puT+AHjexnExgnbvBxjQ9fjkFDWrikJ2jQA==}
engines: {node: ^18.19.0 || >=20.6.0} engines: {node: ^18.19.0 || >=20.6.0}
peerDependencies: peerDependencies:
'@opentelemetry/api': '>=1.3.0 <1.10.0' '@opentelemetry/api': '>=1.3.0 <1.10.0'
'@opentelemetry/sdk-metrics@2.10.0': '@opentelemetry/sdk-metrics@2.11.0':
resolution: {integrity: sha512-t6r1VSvXNtSDnPXU1FbZeetJb7yyovHmgu0wRSoftxtE0g2rSNhQZQUy69sRUCL+iioJpX8SN/S6wq6ZtvLySQ==} resolution: {integrity: sha512-7GXXcObyHyDUUSG+L+kJoquty01bzm7ivE7+SSgXXJcHuPzGviptxwARmI2c+bnnxjexGQbJnyNlN8HxBP/Y7A==}
engines: {node: ^18.19.0 || >=20.6.0} engines: {node: ^18.19.0 || >=20.6.0}
peerDependencies: peerDependencies:
'@opentelemetry/api': '>=1.9.0 <1.10.0' '@opentelemetry/api': '>=1.9.0 <1.10.0'
@@ -1480,6 +1480,7 @@ packages:
'@smithy/core@3.24.1': '@smithy/core@3.24.1':
resolution: {integrity: sha512-3mT7o4qQyUWttYnVK3A0Z/u3Xha3E81tXn32Tz6vjZiUXhBrkEivpw1hBYfh84iFF9CSzkBU9Y1DJ3Q6RQ231g==} resolution: {integrity: sha512-3mT7o4qQyUWttYnVK3A0Z/u3Xha3E81tXn32Tz6vjZiUXhBrkEivpw1hBYfh84iFF9CSzkBU9Y1DJ3Q6RQ231g==}
engines: {node: '>=18.0.0'} engines: {node: '>=18.0.0'}
deprecated: Deprecated due to bug in browser bundling instructions https://github.com/smithy-lang/smithy-typescript/issues/2025
'@smithy/credential-provider-imds@4.3.1': '@smithy/credential-provider-imds@4.3.1':
resolution: {integrity: sha512-0S/acwHnqX4WrjXzhdiDRxsG2s9SC0cpPIK9nZ1R6UOHd+j7uL28+4bHu22urbLk2TVw3fkp6na/+fkUt/pLNQ==} resolution: {integrity: sha512-0S/acwHnqX4WrjXzhdiDRxsG2s9SC0cpPIK9nZ1R6UOHd+j7uL28+4bHu22urbLk2TVw3fkp6na/+fkUt/pLNQ==}
@@ -3238,8 +3239,8 @@ packages:
peerDependencies: peerDependencies:
typescript: '>=4.2.0' typescript: '>=4.2.0'
ts-jest@29.4.9: ts-jest@29.4.12:
resolution: {integrity: sha512-LTb9496gYPMCqjeDLdPrKuXtncudeV1yRZnF4Wo5l3SFi0RYEnYRNgMrFIdg+FHvfzjCyQk1cLncWVqiSX+EvQ==} resolution: {integrity: sha512-Ov6ClY53Fflh6BGAnY2DlTq1hYDrTycz2PVTXBWFW2CU+9zrEqAp9fWdGXl42EXO5RLSFAcAZ2JFKbP+zBTFfw==}
engines: {node: ^14.15.0 || ^16.10.0 || ^18.0.0 || >=20.0.0} engines: {node: ^14.15.0 || ^16.10.0 || ^18.0.0 || >=20.0.0}
hasBin: true hasBin: true
peerDependencies: peerDependencies:
@@ -5110,22 +5111,22 @@ snapshots:
'@opentelemetry/api@1.9.1': {} '@opentelemetry/api@1.9.1': {}
'@opentelemetry/core@2.10.0(@opentelemetry/api@1.9.1)': '@opentelemetry/core@2.11.0(@opentelemetry/api@1.9.1)':
dependencies: dependencies:
'@opentelemetry/api': 1.9.1 '@opentelemetry/api': 1.9.1
'@opentelemetry/semantic-conventions': 1.43.0 '@opentelemetry/semantic-conventions': 1.43.0
'@opentelemetry/resources@2.10.0(@opentelemetry/api@1.9.1)': '@opentelemetry/resources@2.11.0(@opentelemetry/api@1.9.1)':
dependencies: dependencies:
'@opentelemetry/api': 1.9.1 '@opentelemetry/api': 1.9.1
'@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) '@opentelemetry/core': 2.11.0(@opentelemetry/api@1.9.1)
'@opentelemetry/semantic-conventions': 1.43.0 '@opentelemetry/semantic-conventions': 1.43.0
'@opentelemetry/sdk-metrics@2.10.0(@opentelemetry/api@1.9.1)': '@opentelemetry/sdk-metrics@2.11.0(@opentelemetry/api@1.9.1)':
dependencies: dependencies:
'@opentelemetry/api': 1.9.1 '@opentelemetry/api': 1.9.1
'@opentelemetry/core': 2.10.0(@opentelemetry/api@1.9.1) '@opentelemetry/core': 2.11.0(@opentelemetry/api@1.9.1)
'@opentelemetry/resources': 2.10.0(@opentelemetry/api@1.9.1) '@opentelemetry/resources': 2.11.0(@opentelemetry/api@1.9.1)
'@opentelemetry/semantic-conventions@1.43.0': {} '@opentelemetry/semantic-conventions@1.43.0': {}
@@ -5574,7 +5575,7 @@ snapshots:
globby: 11.1.0 globby: 11.1.0
is-glob: 4.0.3 is-glob: 4.0.3
minimatch: 9.0.9 minimatch: 9.0.9
semver: 7.8.0 semver: 7.8.5
ts-api-utils: 1.4.3(typescript@5.5.4) ts-api-utils: 1.4.3(typescript@5.5.4)
optionalDependencies: optionalDependencies:
typescript: 5.5.4 typescript: 5.5.4
@@ -6426,7 +6427,7 @@ snapshots:
'@babel/parser': 7.29.3 '@babel/parser': 7.29.3
'@istanbuljs/schema': 0.1.6 '@istanbuljs/schema': 0.1.6
istanbul-lib-coverage: 3.2.2 istanbul-lib-coverage: 3.2.2
semver: 7.8.0 semver: 7.8.5
transitivePeerDependencies: transitivePeerDependencies:
- supports-color - supports-color
@@ -6705,7 +6706,7 @@ snapshots:
jest-util: 29.7.0 jest-util: 29.7.0
natural-compare: 1.4.0 natural-compare: 1.4.0
pretty-format: 29.7.0 pretty-format: 29.7.0
semver: 7.8.0 semver: 7.8.5
transitivePeerDependencies: transitivePeerDependencies:
- supports-color - supports-color
@@ -6828,7 +6829,7 @@ snapshots:
make-dir@4.0.0: make-dir@4.0.0:
dependencies: dependencies:
semver: 7.8.0 semver: 7.8.5
make-error@1.3.6: {} make-error@1.3.6: {}
@@ -7162,8 +7163,7 @@ snapshots:
semver@7.8.0: {} semver@7.8.0: {}
semver@7.8.5: semver@7.8.5: {}
optional: true
sharp@0.35.4(@types/node@22.7.4): sharp@0.35.4(@types/node@22.7.4):
dependencies: dependencies:
@@ -7327,7 +7327,7 @@ snapshots:
dependencies: dependencies:
typescript: 5.5.4 typescript: 5.5.4
ts-jest@29.4.9(@babel/core@7.29.7)(@jest/transform@29.7.0)(@jest/types@29.6.3)(babel-jest@29.7.0(@babel/core@7.29.7))(jest-util@29.7.0)(jest@29.7.0(@types/node@22.7.4))(typescript@5.5.4): ts-jest@29.4.12(@babel/core@7.29.7)(@jest/transform@29.7.0)(@jest/types@29.6.3)(babel-jest@29.7.0(@babel/core@7.29.7))(jest-util@29.7.0)(jest@29.7.0(@types/node@22.7.4))(typescript@5.5.4):
dependencies: dependencies:
bs-logger: 0.2.6 bs-logger: 0.2.6
fast-json-stable-stringify: 2.1.0 fast-json-stable-stringify: 2.1.0
@@ -7336,7 +7336,7 @@ snapshots:
json5: 2.2.3 json5: 2.2.3
lodash.memoize: 4.1.2 lodash.memoize: 4.1.2
make-error: 1.3.6 make-error: 1.3.6
semver: 7.8.0 semver: 7.8.5
type-fest: 4.41.0 type-fest: 4.41.0
typescript: 5.5.4 typescript: 5.5.4
yargs-parser: 21.1.1 yargs-parser: 21.1.1
+1 -1
View File
@@ -127,7 +127,7 @@ impl Job {
} }
/// Serialise Arrow batches as a single IPC stream for the TypeScript layer. /// Serialise Arrow batches as a single IPC stream for the TypeScript layer.
pub(crate) fn batches_to_ipc_buffer(batches: &[RecordBatch]) -> napi::Result<Buffer> { fn batches_to_ipc_buffer(batches: &[RecordBatch]) -> napi::Result<Buffer> {
let Some(first) = batches.first() else { let Some(first) = batches.first() else {
return Ok(Buffer::from(Vec::<u8>::new())); return Ok(Buffer::from(Vec::<u8>::new()));
}; };
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "lancedb-python" name = "lancedb-python"
version = "0.39.0-beta.3" version = "0.39.0-beta.6"
publish = false publish = false
edition.workspace = true edition.workspace = true
description = "Python bindings for LanceDB" description = "Python bindings for LanceDB"
+1 -1
View File
@@ -21,7 +21,7 @@ class GteEmbeddings(TextEmbeddingFunction):
An embedding function that uses GTE-LARGE MLX format(for Apple silicon devices only) An embedding function that uses GTE-LARGE MLX format(for Apple silicon devices only)
as well as the standard cpu/gpu version from: https://huggingface.co/thenlper/gte-large. as well as the standard cpu/gpu version from: https://huggingface.co/thenlper/gte-large.
For Apple users, you will need the mlx package insalled, which can be done with: For Apple users, you will need the mlx package installed, which can be done with:
pip install mlx pip install mlx
Parameters Parameters
@@ -60,7 +60,7 @@ class InstructorEmbeddingFunction(TextEmbeddingFunction):
import lancedb import lancedb
from lancedb.pydantic import LanceModel, Vector from lancedb.pydantic import LanceModel, Vector
from lancedb.embeddings import get_registry, InstuctorEmbeddingFunction from lancedb.embeddings import get_registry, InstructorEmbeddingFunction
instructor = get_registry().get("instructor").create( instructor = get_registry().get("instructor").create(
source_instruction="represent the document for retrieval", source_instruction="represent the document for retrieval",
+1 -1
View File
@@ -751,7 +751,7 @@ class IvfPq:
This value controls how much the vector is compressed during the This value controls how much the vector is compressed during the
quantization step. The more sub vectors there are the less the vector is quantization step. The more sub vectors there are the less the vector is
compressed. The default is the dimension of the vector divided by 16. If compressed. The default is the dimension of the vector divided by 16. If
the dimension is not evenly divisible by 16 we use the dimension divded by the dimension is not evenly divisible by 16 we use the dimension divided by
8. 8.
The above two cases are highly preferred. Having 8 or 16 values per The above two cases are highly preferred. Having 8 or 16 values per
+58 -17
View File
@@ -78,6 +78,10 @@ if TYPE_CHECKING:
T = TypeVar("T", bound="LanceModel") T = TypeVar("T", bound="LanceModel")
AnalyzePlanDistributedMetrics = Literal["aggregate", "per_worker", "full"] AnalyzePlanDistributedMetrics = Literal["aggregate", "per_worker", "full"]
# Number of rows a hybrid query returns when no limit was set on it. This
# mirrors the default the Rust query builder applies to its sub-queries.
DEFAULT_HYBRID_LIMIT = 10
@runtime_checkable @runtime_checkable
class _LanceScanner(Protocol): class _LanceScanner(Protocol):
@@ -859,7 +863,7 @@ class Query(pydantic.BaseModel):
return query return query
# This tells pydantic to allow custom types (needed for the `vector` query since # This tells pydantic to allow custom types (needed for the `vector` query since
# pa.Array wouln't be allowed otherwise) # pa.Array wouldn't be allowed otherwise)
model_config = pydantic.ConfigDict(arbitrary_types_allowed=True) model_config = pydantic.ConfigDict(arbitrary_types_allowed=True)
@@ -3893,14 +3897,54 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
return self return self
def _create_child_queries(
self,
) -> Tuple["AsyncFTSQuery", "AsyncVectorQuery", int, int]:
"""Build the sub-queries that make up this hybrid query.
Execution, `explain_plan` and `analyze_plan` all go through here so that
the plans that are reported are the plans that actually run.
Returns the two sub-queries along with the effective limit and offset of
the hybrid query itself.
"""
fts_query = AsyncFTSQuery(self._inner.to_fts_query(), self._table)
vec_query = AsyncVectorQuery(self._inner.to_vector_query(), self._table)
fts_req = fts_query._inner.to_query_request()
vec_req = vec_query._inner.to_query_request()
# Only one of the two sub-queries carries the limit when it was never
# set explicitly: nearest_to()/nearest_to_text() build the sibling query
# from scratch, and that is where the default gets filled in. Which one
# that is depends on the order the hybrid query was built in, so look at
# both rather than at a single side.
limit = fts_req.limit if fts_req.limit is not None else vec_req.limit
if limit is None:
limit = DEFAULT_HYBRID_LIMIT
offset = fts_req.offset or vec_req.offset or 0
fts_query.with_row_id()
vec_query.with_row_id()
# offset() pushes the offset down into both sub-queries, which would make
# each of them skip its own first `offset` rows. The window has to be
# taken out of the combined, reranked results instead, so fetch the
# skipped prefix here too and slice it off afterwards.
fts_query.limit(limit + offset)
vec_query.limit(limit + offset)
fts_query.offset(0)
vec_query.offset(0)
return fts_query, vec_query, limit, offset
async def to_batches( async def to_batches(
self, self,
*, *,
max_batch_length: Optional[int] = None, max_batch_length: Optional[int] = None,
timeout: Optional[timedelta] = None, timeout: Optional[timedelta] = None,
) -> AsyncRecordBatchReader: ) -> AsyncRecordBatchReader:
fts_query = AsyncFTSQuery(self._inner.to_fts_query(), self._table) fts_query, vec_query, limit, offset = self._create_child_queries()
vec_query = AsyncVectorQuery(self._inner.to_vector_query(), self._table)
req = fts_query._inner.to_query_request() req = fts_query._inner.to_query_request()
blob_auto_row_id = False blob_auto_row_id = False
@@ -3920,9 +3964,6 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
self._blob_auto_row_id = blob_auto_row_id self._blob_auto_row_id = blob_auto_row_id
self._blob_paths = blob_paths self._blob_paths = blob_paths
fts_query.with_row_id()
vec_query.with_row_id()
fts_results, vector_results = await asyncio.gather( fts_results, vector_results = await asyncio.gather(
fts_query.to_arrow(timeout=timeout), fts_query.to_arrow(timeout=timeout),
vec_query.to_arrow(timeout=timeout), vec_query.to_arrow(timeout=timeout),
@@ -3934,8 +3975,9 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
norm=self._norm, norm=self._norm,
fts_query=fts_query.get_query(), fts_query=fts_query.get_query(),
reranker=self._reranker, reranker=self._reranker,
limit=self._inner.get_limit(), limit=limit,
with_row_ids=True, with_row_ids=True,
offset=offset,
) )
if ( if (
not self._user_requested_row_id() not self._user_requested_row_id()
@@ -3964,14 +4006,14 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
... print(plan) ... print(plan)
>>> asyncio.run(doctest_example()) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE >>> asyncio.run(doctest_example()) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
RRFReranker(K=60) RRFReranker(K=60)
ProjectionExec: expr=[vector@0 as vector, text@3 as text, _distance@2 as _distance] ProjectionExec: expr=[vector@0 as vector, text@3 as text, _distance@2 as _distance, _rowid@1 as _rowid]
LanceRead: uri=..., projection=[text], source=stream(_rowid) LanceRead: uri=..., projection=[text], source=stream(_rowid)
GlobalLimitExec: skip=0, fetch=10 GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false] SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2 KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], ... LanceRead: uri=..., projection=[vector], ...
ProjectionExec: expr=[vector@2 as vector, text@3 as text, _score@1 as _score] ProjectionExec: expr=[vector@2 as vector, text@3 as text, _score@1 as _score, _rowid@0 as _rowid]
LanceRead: uri=..., projection=[vector, text], source=stream(_rowid) LanceRead: uri=..., projection=[vector, text], source=stream(_rowid)
GlobalLimitExec: skip=0, fetch=10 GlobalLimitExec: skip=0, fetch=10
MatchQuery: column=text, query=[hello] MatchQuery: column=text, query=[hello]
@@ -3986,8 +4028,9 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
plan : str plan : str
""" # noqa: E501 """ # noqa: E501
vector_plan = await self._inner.to_vector_query().explain_plan(verbose) fts_query, vec_query, _, _ = self._create_child_queries()
fts_plan = await self._inner.to_fts_query().explain_plan(verbose) vector_plan = await vec_query.explain_plan(verbose)
fts_plan = await fts_query.explain_plan(verbose)
# Indent sub-plans under the reranker # Indent sub-plans under the reranker
indented_vector = "\n".join(" " + line for line in vector_plan.splitlines()) indented_vector = "\n".join(" " + line for line in vector_plan.splitlines())
indented_fts = "\n".join(" " + line for line in fts_plan.splitlines()) indented_fts = "\n".join(" " + line for line in fts_plan.splitlines())
@@ -4014,14 +4057,12 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
------- -------
plan : str plan : str
""" """
fts_query, vec_query, _, _ = self._create_child_queries()
results = ["Vector Search Query:"] results = ["Vector Search Query:"]
results.append( results.append(await vec_query.analyze_plan(distributed_metrics))
await self._inner.to_vector_query().analyze_plan(distributed_metrics)
)
results.append("FTS Search Query:") results.append("FTS Search Query:")
results.append( results.append(await fts_query.analyze_plan(distributed_metrics))
await self._inner.to_fts_query().analyze_plan(distributed_metrics)
)
return "\n".join(results) return "\n".join(results)
+1 -1
View File
@@ -720,7 +720,7 @@ class RemoteTable(Table):
Parameters Parameters
---------- ----------
query: list/np.ndarray/str/PIL.Image.Image, default None query: list/np.ndarray/str/PIL.Image.Image, default None
The targetted vector to search for. The targeted vector to search for.
- *default None*. - *default None*.
Acceptable types are: list, np.ndarray, PIL.Image.Image Acceptable types are: list, np.ndarray, PIL.Image.Image
+1 -1
View File
@@ -175,7 +175,7 @@ class Reranker(ABC):
if the results haven't been executed yet or the results in arrow format. if the results haven't been executed yet or the results in arrow format.
query : str or None, query : str or None,
The input query. Some rerankers might not need the query to rerank. The input query. Some rerankers might not need the query to rerank.
In that case, it can be set to None explicitly. This is inteded to In that case, it can be set to None explicitly. This is intended to
be handled by the reranker implementations. be handled by the reranker implementations.
deduplicate : bool, optional deduplicate : bool, optional
Whether to deduplicate the results based on the `_rowid` column, Whether to deduplicate the results based on the `_rowid` column,
+4 -4
View File
@@ -1619,7 +1619,7 @@ class Table(ABC):
Parameters Parameters
---------- ----------
query: list/np.ndarray/str/PIL.Image.Image, default None query: list/np.ndarray/str/PIL.Image.Image, default None
The targetted vector to search for. The targeted vector to search for.
- *default None*. - *default None*.
Acceptable types are: list, np.ndarray, PIL.Image.Image Acceptable types are: list, np.ndarray, PIL.Image.Image
@@ -3841,7 +3841,7 @@ class LanceTable(Table):
Parameters Parameters
---------- ----------
query: list/np.ndarray/str/PIL.Image.Image, default None query: list/np.ndarray/str/PIL.Image.Image, default None
The targetted vector to search for. The targeted vector to search for.
- *default None*. - *default None*.
Acceptable types are: list, np.ndarray, PIL.Image.Image Acceptable types are: list, np.ndarray, PIL.Image.Image
@@ -5638,7 +5638,7 @@ class AsyncTable:
if fill_value is None: if fill_value is None:
fill_value = 0.0 fill_value = 0.0
# _santitize_data is an old code path, but we will use it until the # _sanitize_data is an old code path, but we will use it until the
# new code path is ready. # new code path is ready.
if mode == "overwrite": if mode == "overwrite":
# For overwrite, apply the same preprocessing as create_table # For overwrite, apply the same preprocessing as create_table
@@ -5814,7 +5814,7 @@ class AsyncTable:
Parameters Parameters
---------- ----------
query: list/np.ndarray/str/PIL.Image.Image, default None query: list/np.ndarray/str/PIL.Image.Image, default None
The targetted vector to search for. The targeted vector to search for.
- *default None*. - *default None*.
Acceptable types are: list, np.ndarray, PIL.Image.Image Acceptable types are: list, np.ndarray, PIL.Image.Image
+4 -1
View File
@@ -297,7 +297,10 @@ def test_blob_v2_projection_sources_use_typed_column_name():
def _legacy_v1_table(name): def _legacy_v1_table(name):
db = lancedb.connect("memory:///") # Legacy v1 blob columns are only writable at file version <= 2.1.
db = lancedb.connect(
"memory:///", storage_options={"new_table_data_storage_version": "2.1"}
)
schema = pa.schema( schema = pa.schema(
[ [
pa.field("id", pa.int64()), pa.field("id", pa.int64()),
+5 -5
View File
@@ -327,8 +327,8 @@ def test_embedding_function_with_pandas(tmp_path):
) -> List[np.array]: ) -> List[np.array]:
return [np.random.randn(self.ndims()).tolist() for _ in range(len(texts))] return [np.random.randn(self.ndims()).tolist() for _ in range(len(texts))]
registery = get_registry() registry = get_registry()
func = registery.get("mock-embedding").create() func = registry.get("mock-embedding").create()
class TestSchema(LanceModel): class TestSchema(LanceModel):
text: str = func.SourceField() text: str = func.SourceField()
@@ -394,9 +394,9 @@ def test_multiple_embeddings_for_pandas(tmp_path):
) -> List[np.array]: ) -> List[np.array]:
return [np.random.randn(self.ndims()).tolist() for _ in range(len(texts))] return [np.random.randn(self.ndims()).tolist() for _ in range(len(texts))]
registery = get_registry() registry = get_registry()
func1 = registery.get("mock-embedding").create() func1 = registry.get("mock-embedding").create()
func2 = registery.get("mock-embedding2").create() func2 = registry.get("mock-embedding2").create()
class TestSchema(LanceModel): class TestSchema(LanceModel):
text: str = func1.SourceField() text: str = func1.SourceField()
+14 -4
View File
@@ -1011,8 +1011,13 @@ def test_fts_ngram(mem_db: DBConnection):
assert set(r["text"] for r in results) == {"lance database", "lance is cool"} assert set(r["text"] for r in results) == {"lance database", "lance is cool"}
results = ( results = (
table.search("nce", query_type="fts").limit(10).to_list() table.search(
) # spellchecker:disable-line "nce", # spellchecker:disable-line
query_type="fts",
)
.limit(10)
.to_list()
)
assert len(results) == 2 assert len(results) == 2
assert set(r["text"] for r in results) == {"lance database", "lance is cool"} assert set(r["text"] for r in results) == {"lance database", "lance is cool"}
@@ -1034,8 +1039,13 @@ def test_fts_ngram(mem_db: DBConnection):
assert set(r["text"] for r in results) == {"lance database", "lance is cool"} assert set(r["text"] for r in results) == {"lance database", "lance is cool"}
results = ( results = (
table.search("nce", query_type="fts").limit(10).to_list() table.search(
) # spellchecker:disable-line "nce", # spellchecker:disable-line
query_type="fts",
)
.limit(10)
.to_list()
)
assert len(results) == 0 assert len(results) == 0
results = table.search("la", query_type="fts").limit(10).to_list() results = table.search("la", query_type="fts").limit(10).to_list()
+87
View File
@@ -203,6 +203,93 @@ async def test_async_hybrid_query_default_limit(table: AsyncTable):
assert texts.count("a") == 1 assert texts.count("a") == 1
@pytest.mark.asyncio
async def test_async_hybrid_query_offset(table: AsyncTable):
# The offset window of a hybrid query must be a suffix of the same query
# run without an offset. Skipping the first rows of each sub-query instead
# of the first rows of the fused result silently changes which rows land in
# the window.
full = await (
table.query()
.nearest_to([0.0, 0.4])
.nearest_to_text("dog")
.limit(4)
.with_row_id()
.to_arrow()
)
assert len(full) == 4
second_page = await (
table.query()
.nearest_to([0.0, 0.4])
.nearest_to_text("dog")
.offset(2)
.limit(2)
.with_row_id()
.to_arrow()
)
assert second_page["_rowid"].to_pylist() == full["_rowid"].to_pylist()[2:]
first_page = await (
table.query()
.nearest_to([0.0, 0.4])
.nearest_to_text("dog")
.limit(2)
.with_row_id()
.to_arrow()
)
# Paging through the result must visit every row exactly once: no row
# repeated from the previous page and none dropped between the two.
paged = first_page["_rowid"].to_pylist() + second_page["_rowid"].to_pylist()
assert sorted(paged) == sorted(full["_rowid"].to_pylist())
@pytest.mark.asyncio
async def test_async_hybrid_query_fts_first_default_limit(table: AsyncTable):
# nearest_to() and nearest_to_text() build their new sibling sub-query from
# scratch, and that is the sub-query the default limit ends up on. So the
# side that carries the limit depends on the order the hybrid query was
# built in, and looking at only one side loses the limit for half the ways
# a hybrid query can be written. Without a limit the combined results are
# not truncated at all and the whole union of both candidate lists is
# returned.
await table.add([{"text": "dog", "vector": [50.0 + i, 50.0]} for i in range(10)])
result = await (
table.query().nearest_to_text("dog").nearest_to([0.1, 0.1]).to_arrow()
)
assert len(result) == 10
offset_result = await (
table.query().nearest_to_text("dog").nearest_to([0.1, 0.1]).offset(2).to_arrow()
)
assert len(offset_result) == 10
@pytest.mark.asyncio
async def test_async_hybrid_query_explain_plan_matches_execution(table: AsyncTable):
# Paging rewrites the sub-queries: each one fetches limit + offset rows with
# no offset of its own, and the window is sliced out after fusion. The plans
# have to be built from those rewritten sub-queries, otherwise explain_plan
# and analyze_plan describe a query that is never run.
query = (
table.query().nearest_to([0.0, 0.4]).nearest_to_text("dog").offset(2).limit(2)
)
await query.to_arrow()
plan = await query.explain_plan()
assert [
line.strip() for line in plan.splitlines() if "GlobalLimitExec" in line
] == [
"GlobalLimitExec: skip=0, fetch=4",
"GlobalLimitExec: skip=0, fetch=4",
]
analyzed = await query.analyze_plan()
assert analyzed.count("skip=0, fetch=4") == 2
assert "skip=2" not in analyzed
def test_hybrid_query_offset(sync_table: Table): def test_hybrid_query_offset(sync_table: Table):
# The offset window of a hybrid query must be a suffix of the same query # The offset window of a hybrid query must be a suffix of the same query
# run without an offset -- it must not be silently ignored. # run without an offset -- it must not be silently ignored.
+7 -1
View File
@@ -193,7 +193,13 @@ class TestNamespaceConnection:
), ),
) )
table = db.create_table("blob_table", data, namespace_path=["test_ns"]) # Legacy v1 blob columns are only writable at file version <= 2.1.
table = db.create_table(
"blob_table",
data,
namespace_path=["test_ns"],
storage_options={"new_table_data_storage_version": "2.1"},
)
df = table.to_pandas(blob_mode="lazy").sort_values("id") df = table.to_pandas(blob_mode="lazy").sort_values("id")
blob = df["blob"].iloc[0] blob = df["blob"].iloc[0]
+38 -10
View File
@@ -40,6 +40,10 @@ from utils import exception_output
from importlib.util import find_spec from importlib.util import find_spec
# Legacy v1 blob columns are only writable at file version <= 2.1.
LEGACY_BLOB_STORAGE_OPTIONS = {"new_table_data_storage_version": "2.1"}
def _blob_query_data(): def _blob_query_data():
return pa.table( return pa.table(
{ {
@@ -119,13 +123,17 @@ def _assert_blob_bytes_projection(df):
def _blob_query_table(db, name, blob_schema): def _blob_query_table(db, name, blob_schema):
if blob_schema == "v1": if blob_schema == "v1":
return db.create_table(name, _blob_query_data()) return db.create_table(
name, _blob_query_data(), storage_options=LEGACY_BLOB_STORAGE_OPTIONS
)
return _create_blob_v2_query_table(db, name) return _create_blob_v2_query_table(db, name)
async def _blob_query_table_async(db, name, blob_schema): async def _blob_query_table_async(db, name, blob_schema):
if blob_schema == "v1": if blob_schema == "v1":
return await db.create_table(name, _blob_query_data()) return await db.create_table(
name, _blob_query_data(), storage_options=LEGACY_BLOB_STORAGE_OPTIONS
)
return await _create_blob_v2_query_table_async(db, name) return await _create_blob_v2_query_table_async(db, name)
@@ -275,7 +283,9 @@ async def test_query_to_pandas_kwargs(table, table_async):
def test_plain_scan_query_to_pandas_blob_modes(tmp_db, blob_mode): def test_plain_scan_query_to_pandas_blob_modes(tmp_db, blob_mode):
pytest.importorskip("lance") pytest.importorskip("lance")
table = tmp_db.create_table( table = tmp_db.create_table(
f"test_query_to_pandas_blob_{blob_mode}", _blob_query_data() f"test_query_to_pandas_blob_{blob_mode}",
_blob_query_data(),
storage_options=LEGACY_BLOB_STORAGE_OPTIONS,
) )
df = ( df = (
@@ -322,7 +332,9 @@ def test_plain_scan_query_to_pandas_blob_mode_does_not_collect_arrow(
): ):
pytest.importorskip("lance") pytest.importorskip("lance")
table = tmp_db.create_table( table = tmp_db.create_table(
"test_query_to_pandas_blob_no_arrow_collect", _blob_query_data() "test_query_to_pandas_blob_no_arrow_collect",
_blob_query_data(),
storage_options=LEGACY_BLOB_STORAGE_OPTIONS,
) )
query = table.search().where("id = 1").select(["id", "blob"]) query = table.search().where("id = 1").select(["id", "blob"])
@@ -347,7 +359,9 @@ def test_plain_scan_query_to_pandas_blob_descriptions_flatten_uses_scanner(
): ):
pytest.importorskip("lance") pytest.importorskip("lance")
table = tmp_db.create_table( table = tmp_db.create_table(
"test_query_to_pandas_blob_desc_flatten", _blob_query_data() "test_query_to_pandas_blob_desc_flatten",
_blob_query_data(),
storage_options=LEGACY_BLOB_STORAGE_OPTIONS,
) )
query = table.search().where("id = 1").select(["id", "blob"]) query = table.search().where("id = 1").select(["id", "blob"])
@@ -365,7 +379,11 @@ def test_plain_scan_query_to_pandas_blob_descriptions_flatten_uses_scanner(
def test_plain_scan_query_to_pandas_scanner_state(tmp_db): def test_plain_scan_query_to_pandas_scanner_state(tmp_db):
pytest.importorskip("lance") pytest.importorskip("lance")
data = _blob_query_data() data = _blob_query_data()
table = tmp_db.create_table("test_query_to_pandas_scanner_state", data.slice(0, 2)) table = tmp_db.create_table(
"test_query_to_pandas_scanner_state",
data.slice(0, 2),
storage_options=LEGACY_BLOB_STORAGE_OPTIONS,
)
table.add(data.slice(2, 2)) table.add(data.slice(2, 2))
fragments = table.to_lance().get_fragments() fragments = table.to_lance().get_fragments()
@@ -400,7 +418,9 @@ def test_plain_scan_query_to_pandas_scanner_state(tmp_db):
async def test_async_plain_scan_query_to_pandas_blob_projection(tmp_db_async): async def test_async_plain_scan_query_to_pandas_blob_projection(tmp_db_async):
pytest.importorskip("lance") pytest.importorskip("lance")
table = await tmp_db_async.create_table( table = await tmp_db_async.create_table(
"test_async_query_to_pandas_blob_projection", _blob_query_data() "test_async_query_to_pandas_blob_projection",
_blob_query_data(),
storage_options=LEGACY_BLOB_STORAGE_OPTIONS,
) )
lazy_df = await ( lazy_df = await (
@@ -452,7 +472,9 @@ async def test_async_plain_scan_query_to_pandas_blob_mode_does_not_collect_arrow
): ):
pytest.importorskip("lance") pytest.importorskip("lance")
table = await tmp_db_async.create_table( table = await tmp_db_async.create_table(
"test_async_query_to_pandas_blob_no_arrow_collect", _blob_query_data() "test_async_query_to_pandas_blob_no_arrow_collect",
_blob_query_data(),
storage_options=LEGACY_BLOB_STORAGE_OPTIONS,
) )
query = table.query().where("id = 1").select(["id", "blob"]) query = table.query().where("id = 1").select(["id", "blob"])
@@ -474,7 +496,11 @@ async def test_async_plain_scan_query_to_pandas_blob_mode_does_not_collect_arrow
def test_vector_query_to_pandas_blob_mode_requires_native_path(tmp_db): def test_vector_query_to_pandas_blob_mode_requires_native_path(tmp_db):
pytest.importorskip("lance") pytest.importorskip("lance")
table = tmp_db.create_table("test_vector_query_blob_mode", _blob_query_data()) table = tmp_db.create_table(
"test_vector_query_blob_mode",
_blob_query_data(),
storage_options=LEGACY_BLOB_STORAGE_OPTIONS,
)
with pytest.raises(RuntimeError, match="Lance native pandas conversion"): with pytest.raises(RuntimeError, match="Lance native pandas conversion"):
table.search([1.0, 0.0]).select(["blob", "vector"]).limit(1).to_pandas( table.search([1.0, 0.0]).select(["blob", "vector"]).limit(1).to_pandas(
@@ -485,7 +511,9 @@ def test_vector_query_to_pandas_blob_mode_requires_native_path(tmp_db):
def test_vector_query_to_pandas_blob_descriptions_requires_plain_scan(tmp_db): def test_vector_query_to_pandas_blob_descriptions_requires_plain_scan(tmp_db):
pytest.importorskip("lance") pytest.importorskip("lance")
table = tmp_db.create_table( table = tmp_db.create_table(
"test_vector_query_blob_descriptions", _blob_query_data() "test_vector_query_blob_descriptions",
_blob_query_data(),
storage_options=LEGACY_BLOB_STORAGE_OPTIONS,
) )
with pytest.raises(RuntimeError, match="plain scan query"): with pytest.raises(RuntimeError, match="plain scan query"):
+1 -1
View File
@@ -81,7 +81,7 @@ def get_test_table(tmp_path):
"but his son was mortal", "but his son was mortal",
"there hasn't been a good battlefield game since 2142", "there hasn't been a good battlefield game since 2142",
"I wish they would make another one", "I wish they would make another one",
"campains are not as good as they used to be", "campaigns are not as good as they used to be",
"Multiplayer and open world games have destroyed the single player experience", "Multiplayer and open world games have destroyed the single player experience",
"Maybe the future is console games", "Maybe the future is console games",
"I don't know", "I don't know",
+17 -5
View File
@@ -64,15 +64,23 @@ async def _blob_v2_table_async(db: AsyncConnection, name: str):
return table return table
# Legacy v1 blob columns are only writable at file version <= 2.1.
LEGACY_BLOB_STORAGE_OPTIONS = {"new_table_data_storage_version": "2.1"}
def _blob_table(db: DBConnection, name: str, blob_schema: str): def _blob_table(db: DBConnection, name: str, blob_schema: str):
if blob_schema == "v1": if blob_schema == "v1":
return db.create_table(name, data=_blob_test_data()) return db.create_table(
name, data=_blob_test_data(), storage_options=LEGACY_BLOB_STORAGE_OPTIONS
)
return _blob_v2_table(db, name) return _blob_v2_table(db, name)
async def _blob_table_async(db: AsyncConnection, name: str, blob_schema: str): async def _blob_table_async(db: AsyncConnection, name: str, blob_schema: str):
if blob_schema == "v1": if blob_schema == "v1":
return await db.create_table(name, data=_blob_test_data()) return await db.create_table(
name, data=_blob_test_data(), storage_options=LEGACY_BLOB_STORAGE_OPTIONS
)
return await _blob_v2_table_async(db, name) return await _blob_v2_table_async(db, name)
@@ -147,7 +155,11 @@ def test_table_to_pandas_invalid_blob_mode_non_blob_table(tmp_db: DBConnection):
@pytest.mark.parametrize("blob_mode", ["lazy", "bytes", "descriptions"]) @pytest.mark.parametrize("blob_mode", ["lazy", "bytes", "descriptions"])
def test_table_to_pandas_blob_modes(tmp_db: DBConnection, blob_mode): def test_table_to_pandas_blob_modes(tmp_db: DBConnection, blob_mode):
pytest.importorskip("lance") pytest.importorskip("lance")
table = tmp_db.create_table(f"test_to_pandas_blob_{blob_mode}", _blob_test_data()) table = tmp_db.create_table(
f"test_to_pandas_blob_{blob_mode}",
_blob_test_data(),
storage_options=LEGACY_BLOB_STORAGE_OPTIONS,
)
df = table.to_pandas(blob_mode=blob_mode) df = table.to_pandas(blob_mode=blob_mode)
@@ -3342,7 +3354,7 @@ def test_empty_query(mem_db: DBConnection):
# None is the same as default # None is the same as default
df = table.search().select(["id"]).limit(None).to_arrow() df = table.search().select(["id"]).limit(None).to_arrow()
assert df.num_rows == 100 assert df.num_rows == 100
# invalid limist is the same as None, wihch is the same as default # invalid limist is the same as None, which is the same as default
df = table.search().select(["id"]).limit(-1).to_arrow() df = table.search().select(["id"]).limit(-1).to_arrow()
assert df.num_rows == 100 assert df.num_rows == 100
# valid limit should work # valid limit should work
@@ -3959,7 +3971,7 @@ def test_stats(mem_db: DBConnection):
print(f"{stats=}") print(f"{stats=}")
assert stats == { assert stats == {
# Full on-disk size of the data file, footer and metadata included. # Full on-disk size of the data file, footer and metadata included.
"total_bytes": 633, "total_bytes": 637,
"num_rows": 2, "num_rows": 2,
"num_indices": 0, "num_indices": 0,
"fragment_stats": { "fragment_stats": {
+1 -1
View File
@@ -334,7 +334,7 @@ pub struct PyQueryRequest {
pub column: Option<String>, pub column: Option<String>,
pub query_vector: Option<PyQueryVectors>, pub query_vector: Option<PyQueryVectors>,
pub minimum_nprobes: Option<usize>, pub minimum_nprobes: Option<usize>,
// None means user did not set it and default shoud be used (currenty 20) // None means user did not set it and default should be used (currently 20)
// Some(0) means user set it to None and there is no limit // Some(0) means user set it to None and there is no limit
pub maximum_nprobes: Option<usize>, pub maximum_nprobes: Option<usize>,
pub lower_bound: Option<f32>, pub lower_bound: Option<f32>,
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "lancedb" name = "lancedb"
version = "0.39.0-beta.3" version = "0.39.0-beta.6"
edition.workspace = true edition.workspace = true
description = "LanceDB: A serverless, low-latency vector database for AI applications" description = "LanceDB: A serverless, low-latency vector database for AI applications"
license.workspace = true license.workspace = true
+1 -1
View File
@@ -163,7 +163,7 @@ pub struct PolarsDataFrameRecordBatchReader {
impl PolarsDataFrameRecordBatchReader { impl PolarsDataFrameRecordBatchReader {
/// Creates a new `PolarsDataFrameRecordBatchReader` from a given Polars DataFrame. /// Creates a new `PolarsDataFrameRecordBatchReader` from a given Polars DataFrame.
/// If the input dataframe does not have aligned chunks, this function undergoes /// If the input dataframe does not have aligned chunks, this function undergoes
/// the costly operation of reallocating each series as a single contigous chunk. /// the costly operation of reallocating each series as a single contiguous chunk.
pub fn new(mut df: DataFrame) -> Result<Self> { pub fn new(mut df: DataFrame) -> Result<Self> {
df.align_chunks(); df.align_chunks();
let arrow_schema = let arrow_schema =
+10 -8
View File
@@ -532,10 +532,11 @@ mod tests {
fn storage_version_bumps_to_v2_2() { fn storage_version_bumps_to_v2_2() {
let mut params = WriteParams::default(); let mut params = WriteParams::default();
ensure_blob_storage_version(&blob_schema(), &mut params); ensure_blob_storage_version(&blob_schema(), &mut params);
assert_eq!( let resolved = params
params.data_storage_version.unwrap().resolve(), .data_storage_version
ConcreteFileVersion::V2_2 .unwrap_or(LanceFileVersion::Stable)
); .resolve();
assert_eq!(resolved, ConcreteFileVersion::V2_2);
assert!(!params.enable_stable_row_ids); assert!(!params.enable_stable_row_ids);
} }
@@ -547,10 +548,11 @@ mod tests {
}; };
ensure_blob_storage_version(&blob_schema(), &mut params); ensure_blob_storage_version(&blob_schema(), &mut params);
assert!(params.enable_stable_row_ids); assert!(params.enable_stable_row_ids);
assert_eq!( let resolved = params
params.data_storage_version.unwrap().resolve(), .data_storage_version
ConcreteFileVersion::V2_2 .unwrap_or(LanceFileVersion::Stable)
); .resolve();
assert_eq!(resolved, ConcreteFileVersion::V2_2);
} }
#[test] #[test]
+1 -1
View File
@@ -827,7 +827,7 @@ impl Connection {
pub struct ConnectRequest { pub struct ConnectRequest {
/// Database URI /// Database URI
/// ///
/// ### Accpeted URI formats /// ### Accepted URI formats
/// ///
/// - `/path/to/database` - local database on file system. /// - `/path/to/database` - local database on file system.
/// - `s3://bucket/path/to/database` or `gs://bucket/path/to/database` - database on cloud object store /// - `s3://bucket/path/to/database` or `gs://bucket/path/to/database` - database on cloud object store
+9 -9
View File
@@ -512,7 +512,7 @@ impl ListingDatabase {
// iter thru the query params and extract the commit store param // iter thru the query params and extract the commit store param
let mut engine = None; let mut engine = None;
let mut mirrored_store = None; let mut mirrored_store = None;
let mut filtered_querys = vec![]; let mut filtered_queries = vec![];
// WARNING: specifying engine is NOT a publicly supported feature in lancedb yet // WARNING: specifying engine is NOT a publicly supported feature in lancedb yet
// THE API WILL CHANGE // THE API WILL CHANGE
@@ -528,13 +528,13 @@ impl ListingDatabase {
mirrored_store = Some(value.to_string()); mirrored_store = Some(value.to_string());
} else { } else {
// to owned so we can modify the url // to owned so we can modify the url
filtered_querys.push((key.to_string(), value.to_string())); filtered_queries.push((key.to_string(), value.to_string()));
} }
} }
// Filter out the commit store query param -- it's a lancedb param // Filter out the commit store query param -- it's a lancedb param
url.query_pairs_mut().clear(); url.query_pairs_mut().clear();
url.query_pairs_mut().extend_pairs(filtered_querys); url.query_pairs_mut().extend_pairs(filtered_queries);
// Take a copy of the query string so we can propagate it to lance. // Take a copy of the query string so we can propagate it to lance.
// `query_pairs_mut()` leaves the URL with `Some("")` even when no // `query_pairs_mut()` leaves the URL with `Some("")` even when no
// pairs survive (or none existed in the first place), so an empty // pairs survive (or none existed in the first place), so an empty
@@ -896,11 +896,11 @@ impl Database for ListingDatabase {
} }
async fn read_consistency(&self) -> Result<ReadConsistency> { async fn read_consistency(&self) -> Result<ReadConsistency> {
if let Some(read_consistency_inverval) = self.read_consistency_interval { if let Some(interval) = self.read_consistency_interval {
if read_consistency_inverval.is_zero() { if interval.is_zero() {
Ok(ReadConsistency::Strong) Ok(ReadConsistency::Strong)
} else { } else {
Ok(ReadConsistency::Eventual(read_consistency_inverval)) Ok(ReadConsistency::Eventual(interval))
} }
} else { } else {
Ok(ReadConsistency::Manual) Ok(ReadConsistency::Manual)
@@ -3043,15 +3043,15 @@ mod tests {
/// across platforms — see the `file://` test below). /// across platforms — see the `file://` test below).
fn capture_query_like_connect(input_uri: &str) -> Option<String> { fn capture_query_like_connect(input_uri: &str) -> Option<String> {
let mut url = url::Url::parse(input_uri).unwrap(); let mut url = url::Url::parse(input_uri).unwrap();
let mut filtered_querys = Vec::new(); let mut filtered_queries = Vec::new();
for (key, value) in url.query_pairs() { for (key, value) in url.query_pairs() {
if key == ENGINE || key == MIRRORED_STORE { if key == ENGINE || key == MIRRORED_STORE {
continue; continue;
} }
filtered_querys.push((key.to_string(), value.to_string())); filtered_queries.push((key.to_string(), value.to_string()));
} }
url.query_pairs_mut().clear(); url.query_pairs_mut().clear();
url.query_pairs_mut().extend_pairs(filtered_querys); url.query_pairs_mut().extend_pairs(filtered_queries);
url.query().filter(|q| !q.is_empty()).map(|s| s.to_string()) url.query().filter(|q| !q.is_empty()).map(|s| s.to_string())
} }
+8 -3
View File
@@ -3,6 +3,7 @@
//! Namespace-based database implementation that delegates table management to lance-namespace //! Namespace-based database implementation that delegates table management to lance-namespace
use lance_datafusion::utils::StreamingWriteSource;
use std::collections::{HashMap, HashSet}; use std::collections::{HashMap, HashSet};
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
@@ -250,11 +251,11 @@ impl Database for LanceNamespaceDatabase {
} }
async fn read_consistency(&self) -> Result<ReadConsistency> { async fn read_consistency(&self) -> Result<ReadConsistency> {
if let Some(read_consistency_inverval) = self.read_consistency_interval { if let Some(interval) = self.read_consistency_interval {
if read_consistency_inverval.is_zero() { if interval.is_zero() {
Ok(ReadConsistency::Strong) Ok(ReadConsistency::Strong)
} else { } else {
Ok(ReadConsistency::Eventual(read_consistency_inverval)) Ok(ReadConsistency::Eventual(interval))
} }
} else { } else {
Ok(ReadConsistency::Manual) Ok(ReadConsistency::Manual)
@@ -304,6 +305,10 @@ impl Database for LanceNamespaceDatabase {
} }
async fn create_table(&self, request: DbCreateTableRequest) -> Result<Arc<dyn BaseTable>> { async fn create_table(&self, request: DbCreateTableRequest) -> Result<Arc<dyn BaseTable>> {
// Refuse a bad declaration before the namespace records a table.
crate::table::computed_columns::ensure_declarations_are_planned(
&request.data.arrow_schema(),
)?;
let mut table_id = request.namespace_path.clone(); let mut table_id = request.namespace_path.clone();
table_id.push(request.name.clone()); table_id.push(request.name.clone());
let mut existing_table = None; let mut existing_table = None;
+1 -1
View File
@@ -125,7 +125,7 @@ macro_rules! impl_pq_params_setter {
/// This value controls how much the vector is compressed during the quantization step. /// This value controls how much the vector is compressed during the quantization step.
/// The more sub vectors there are the less the vector is compressed. The default is /// The more sub vectors there are the less the vector is compressed. The default is
/// the dimension of the vector divided by 16. If the dimension is not evenly divisible /// the dimension of the vector divided by 16. If the dimension is not evenly divisible
/// by 16 we use the dimension divded by 8. /// by 16 we use the dimension divided by 8.
/// ///
/// The above two cases are highly preferred. Having 8 or 16 values per subvector allows /// The above two cases are highly preferred. Having 8 or 16 values per subvector allows
/// us to use efficient SIMD instructions. /// us to use efficient SIMD instructions.
File diff suppressed because it is too large Load Diff
+536 -17
View File
@@ -24,8 +24,8 @@ use std::time::{SystemTime, UNIX_EPOCH};
use arrow_array::cast::AsArray; use arrow_array::cast::AsArray;
use arrow_array::types::UInt64Type; use arrow_array::types::UInt64Type;
use arrow_array::{RecordBatch, UInt64Array}; use arrow_array::{RecordBatch, UInt64Array, new_null_array};
use arrow_schema::{Schema as ArrowSchema, SchemaRef}; use arrow_schema::{FieldRef, Schema as ArrowSchema, SchemaRef};
use datafusion::common::ScalarValue; use datafusion::common::ScalarValue;
use datafusion::error::DataFusionError; use datafusion::error::DataFusionError;
use datafusion::physical_plan::SendableRecordBatchStream; use datafusion::physical_plan::SendableRecordBatchStream;
@@ -34,7 +34,7 @@ use datafusion::prelude::{col, lit};
use futures::{StreamExt, TryStreamExt}; use futures::{StreamExt, TryStreamExt};
use lance::Dataset; use lance::Dataset;
use lance::dataset::mem_wal::DatasetMemWalExt; use lance::dataset::mem_wal::DatasetMemWalExt;
use lance::dataset::transaction::{Operation, Transaction}; use lance::dataset::transaction::{Operation, Transaction, UpdateMode};
use lance::dataset::write::delete::DeleteBuilder; use lance::dataset::write::delete::DeleteBuilder;
use lance::dataset::write::merge_insert::inserted_rows::{ use lance::dataset::write::merge_insert::inserted_rows::{
KeyExistenceFilter, KeyExistenceFilterBuilder, KeyValue, KeyExistenceFilter, KeyExistenceFilterBuilder, KeyValue,
@@ -51,6 +51,9 @@ use super::{
definition_to_metadata, definition_to_metadata,
}; };
use crate::database::OpenTableRequest; use crate::database::OpenTableRequest;
use crate::table::computed_columns::{
computed_column_from_field, computed_columns, ensure_declarations_are_planned,
};
use crate::table::{NativeTable, NativeTableExt, Table}; use crate::table::{NativeTable, NativeTableExt, Table};
use crate::{Error, Result}; use crate::{Error, Result};
@@ -167,30 +170,52 @@ pub(crate) async fn execute_refresh(
.map(|p| (p.output.clone(), p.expression.clone())) .map(|p| (p.output.clone(), p.expression.clone()))
.collect(); .collect();
validate_inputs(&source_ds, definition)?; validate_inputs(&source_ds, definition)?;
let (replanned, mut planned_fields, _renames) = super::plan( let (replanned, planned_fields, _renames) = super::plan(
source_schema, source_schema,
&definition.source_table, &definition.source_table,
&definition.source_namespace, &definition.source_namespace,
&projections, Some(&projections),
definition.filter.as_deref(), definition.filter.as_deref(),
definition.limit, definition.limit,
)?; )?;
let mut planned_fields = planned_fields;
planned_fields.push(arrow_schema::Field::new( planned_fields.push(arrow_schema::Field::new(
SOURCE_ROW_ID_COLUMN, SOURCE_ROW_ID_COLUMN,
arrow_schema::DataType::UInt64, arrow_schema::DataType::UInt64,
false, false,
)); ));
// A computed column is not planned from the source: refresh writes it
// NULL and its declaration's owner fills it. Its declaration must still
// be complete, and it must be able to hold NULL.
let physical = ArrowSchema::from(view_ds.schema()); let physical = ArrowSchema::from(view_ds.schema());
let planned_shape: Vec<_> = planned_fields let mut computed = computed_columns(&physical).into_iter().map(|c| c.name);
.iter() if let Some(name) = computed.by_ref().find(|name| {
.map(|f| (f.name().clone(), f.data_type().clone(), f.is_nullable())) physical
.collect(); .field_with_name(name)
let physical_shape: Vec<_> = physical .is_ok_and(|f| !f.is_nullable())
}) {
return Err(Error::Schema {
message: format!(
"computed column '{name}' of view '{}' cannot hold NULL; recreate the view",
view.name()
),
});
}
ensure_declarations_are_planned(&physical)?;
let physical_planned: Vec<&FieldRef> = physical
.fields() .fields()
.iter() .iter()
.map(|f| (f.name().clone(), f.data_type().clone(), f.is_nullable())) .filter(|f| computed_column_from_field(f).is_none())
.collect(); .collect();
if planned_shape != physical_shape { // A projected column that became nullable at the source still fits the
// view's nullable field; the reverse would not.
let matches = planned_fields.len() == physical_planned.len()
&& planned_fields.iter().zip(&physical_planned).all(|(e, p)| {
e.name() == p.name()
&& e.data_type() == p.data_type()
&& (p.is_nullable() || !e.is_nullable())
});
if !matches {
return Err(Error::Schema { return Err(Error::Schema {
message: format!( message: format!(
"the stored definition of view '{}' does not produce this \ "the stored definition of view '{}' does not produce this \
@@ -229,11 +254,18 @@ pub(crate) async fn execute_refresh(
.get(SOURCE_VERSION_TS_META_KEY) .get(SOURCE_VERSION_TS_META_KEY)
.and_then(|raw| raw.parse().ok()); .and_then(|raw| raw.parse().ok());
// The watermark speaks only for the view state its refresh left behind; // The watermark speaks only for the view state its refresh left behind;
// any other commit on the view since then is drift. // any other commit on the view since then is drift, except a fill of its
let view_intact = metadata // computed columns, which rewrites nothing refresh certifies.
let recorded_view_version = metadata
.get(VIEW_VERSION_META_KEY) .get(VIEW_VERSION_META_KEY)
.and_then(|raw| raw.parse::<u64>().ok()) .and_then(|raw| raw.parse::<u64>().ok());
== Some(view_ds.version().version); let view_intact = match recorded_view_version {
Some(recorded) if recorded == view_ds.version().version => true,
Some(recorded) if recorded < view_ds.version().version => {
only_computed_rewrites_since(&view_ds, recorded).await?
}
_ => false,
};
if !full && watermark == Some(source_version) && view_intact && recorded_ts == Some(source_ts) { if !full && watermark == Some(source_version) && view_intact && recorded_ts == Some(source_ts) {
return Ok(RefreshMaterializedViewResult { return Ok(RefreshMaterializedViewResult {
@@ -1090,6 +1122,69 @@ struct RowScope {
limit: Option<u64>, limit: Option<u64>,
} }
/// Whether every commit on the view after `recorded` is a fill of its
/// computed columns: a column rewrite or data replacement touching only
/// those fields and neither adding nor removing rows. A version whose
/// transaction cannot be read is not proven, so it counts as drift.
async fn only_computed_rewrites_since(view_ds: &Dataset, recorded: u64) -> Result<bool> {
// A fill may write any field under a computed column, so the whole
// subtree counts, not only the root.
let physical = ArrowSchema::from(view_ds.schema());
fn subtree(field: &lance_core::datatypes::Field, ids: &mut Vec<u32>) {
ids.push(field.id as u32);
for child in &field.children {
subtree(child, ids);
}
}
let mut computed_fields = Vec::new();
for column in computed_columns(&physical) {
if let Some(field) = view_ds.schema().field(&column.name) {
subtree(field, &mut computed_fields);
}
}
if computed_fields.is_empty() {
return Ok(false);
}
for version in recorded + 1..=view_ds.version().version {
let Some(transaction) = view_ds.read_transaction_by_version(version).await? else {
return Ok(false);
};
let fill = match &transaction.operation {
Operation::Update {
removed_fragment_ids,
new_fragments,
fields_modified,
update_mode: Some(UpdateMode::RewriteColumns),
..
} => {
removed_fragment_ids.is_empty()
&& new_fragments.is_empty()
&& !fields_modified.is_empty()
&& fields_modified
.iter()
.all(|field| computed_fields.contains(field))
}
// What `refresh_column` commits for a SQL declaration.
Operation::DataReplacement { replacements } => {
!replacements.is_empty()
&& replacements.iter().all(|group| {
!group.1.fields.is_empty()
&& group
.1
.fields
.iter()
.all(|field| computed_fields.contains(&(*field as u32)))
})
}
_ => false,
};
if !fill {
return Ok(false);
}
}
Ok(true)
}
async fn compute_stream( async fn compute_stream(
source: &Dataset, source: &Dataset,
definition: &MaterializedViewDefinition, definition: &MaterializedViewDefinition,
@@ -1158,6 +1253,10 @@ async fn compute_stream(
let batch = batch.map_err(|e| DataFusionError::External(Box::new(e)))?; let batch = batch.map_err(|e| DataFusionError::External(Box::new(e)))?;
let mut columns = Vec::with_capacity(out_schema.fields().len()); let mut columns = Vec::with_capacity(out_schema.fields().len());
for field in out_schema.fields() { for field in out_schema.fields() {
if computed_column_from_field(field).is_some() {
columns.push(new_null_array(field.data_type(), batch.num_rows()));
continue;
}
let name = if field.name() == SOURCE_ROW_ID_COLUMN { let name = if field.name() == SOURCE_ROW_ID_COLUMN {
ROW_ID ROW_ID
} else { } else {
@@ -2768,7 +2867,7 @@ mod tests {
let (conn, source) = db_with_source(vec![1]).await; let (conn, source) = db_with_source(vec![1]).await;
let prepared = crate::materialized_view::prepare_declaration( let prepared = crate::materialized_view::prepare_declaration(
&source, &source,
&[("x".into(), "x".into()), ("twice".into(), "x * 2".into())], Some(&[("x".into(), "x".into()), ("twice".into(), "x * 2".into())]),
None, None,
None, None,
) )
@@ -3132,4 +3231,424 @@ mod tests {
let err = view.refresh().execute().await.unwrap_err(); let err = view.refresh().execute().await.unwrap_err();
assert!(err.to_string().contains("source table 'src'"), "{err}"); assert!(err.to_string().contains("source table 'src'"), "{err}");
} }
/// A view with a computed column, declared over `people` and refreshed.
async fn refreshed_computed_view(conn: &Connection) -> MaterializedView {
use crate::materialized_view::tests::{computed_field, people, test_binding};
let source = people(conn).await;
let view = crate::materialized_view::prepare_declaration(
&source,
Some(&[
("id".to_string(), "id".to_string()),
("name".to_string(), "name".to_string()),
]),
None,
None,
)
.await
.unwrap()
.with_computed_columns(
vec![(2, computed_field("emb", "fb_1", "name"))],
&[test_binding("fb_1", "name", "emb")],
)
.unwrap()
.create("v")
.await
.unwrap();
let result = view.refresh().execute().await.unwrap();
assert_eq!(result.mode, RefreshMode::Rebuild);
view
}
async fn unfilled(view: &MaterializedView) -> usize {
view.table()
.count_rows(Some("emb IS NULL".to_string()))
.await
.unwrap()
}
async fn append_people(conn: &Connection, ids: Vec<i32>, names: Vec<&str>) {
let batch = record_batch!(("id", Int32, ids), ("name", Utf8, names)).unwrap();
conn.open_table("people")
.execute()
.await
.unwrap()
.add(batch)
.execute()
.await
.unwrap();
}
/// Commit the fill job's shape on the view: a column rewrite of
/// `fields`, touching no rows. The data is left as it is; what matters
/// here is how the next refresh classifies the commit.
async fn commit_column_rewrite(view: &MaterializedView, fields: &[&str]) {
let native = view.table().as_native().unwrap();
native.dataset.reload().await.unwrap();
let dataset = native.dataset.get().await.unwrap().as_ref().clone();
let fields_modified = fields
.iter()
.map(|name| dataset.schema().field(name).unwrap().id as u32)
.collect();
let updated_fragments = dataset
.get_fragments()
.iter()
.map(|fragment| fragment.metadata().clone())
.collect();
let operation = Operation::Update {
removed_fragment_ids: Vec::new(),
updated_fragments,
new_fragments: Vec::new(),
fields_modified,
compacted_sstables: Vec::new(),
fields_for_preserving_frag_bitmap: Vec::new(),
update_mode: Some(UpdateMode::RewriteColumns),
inserted_rows_filter: None,
updated_fragment_offsets: None,
};
let read_version = dataset.version().version;
CommitBuilder::new(WriteDestination::Dataset(Arc::new(dataset)))
.execute(Transaction::new(read_version, operation, None))
.await
.unwrap();
}
/// Refresh never computes a computed column: every row it writes, on a
/// rebuild, an append and a rewrite, carries NULL there, and the
/// declaration survives all three.
#[tokio::test]
async fn test_computed_columns_are_written_null_and_kept() {
let conn = connect("memory://").execute().await.unwrap();
let view = refreshed_computed_view(&conn).await;
assert_eq!(unfilled(&view).await, 3);
append_people(&conn, vec![4, 5], vec!["d", "e"]).await;
let result = view.refresh().execute().await.unwrap();
assert_eq!(result.mode, RefreshMode::Incremental);
assert_eq!(unfilled(&view).await, 5);
conn.open_table("people")
.execute()
.await
.unwrap()
.update()
.column("name", "'z'")
.only_if("id = 1")
.execute()
.await
.unwrap();
view.refresh().execute().await.unwrap();
assert_eq!(unfilled(&view).await, 5);
assert_eq!(read(view.table(), "id").await, vec![1, 2, 3, 4, 5]);
let schema = view.table().schema().await.unwrap();
assert!(
crate::table::computed_columns::function_bindings(&schema)
.unwrap()
.iter()
.any(|b| b.binding_id() == "fb_1"),
"the binding envelope was lost"
);
assert!(
computed_column_from_field(schema.field_with_name("emb").unwrap()).is_some(),
"the declaration was lost"
);
assert_eq!(
view.refresh().execute().await.unwrap().mode,
RefreshMode::NoOp
);
}
/// The fill job's commit rewrites only computed columns. It is the one
/// commit on a view that is not drift: the next refresh carries on from
/// its watermark instead of rebuilding, which would null what the fill
/// just wrote.
#[tokio::test]
async fn test_a_computed_column_fill_is_not_drift() {
let conn = connect("memory://").execute().await.unwrap();
let view = refreshed_computed_view(&conn).await;
commit_column_rewrite(&view, &["emb"]).await;
assert_eq!(
view.refresh().execute().await.unwrap().mode,
RefreshMode::NoOp
);
commit_column_rewrite(&view, &["emb"]).await;
append_people(&conn, vec![4], vec!["d"]).await;
let result = view.refresh().execute().await.unwrap();
assert_eq!(result.mode, RefreshMode::Incremental);
assert_eq!(result.rows_written, 1);
assert_eq!(read(view.table(), "id").await, vec![1, 2, 3, 4]);
}
/// A column rewrite that reaches a projected column is drift like any
/// other write: refresh certifies those columns and must recompute them.
#[tokio::test]
async fn test_a_rewrite_of_a_projected_column_is_drift() {
let conn = connect("memory://").execute().await.unwrap();
let view = refreshed_computed_view(&conn).await;
commit_column_rewrite(&view, &["emb", "name"]).await;
assert_eq!(
view.refresh().execute().await.unwrap().mode,
RefreshMode::Rebuild
);
}
/// The declaration contract is checked before any refresh mutation: a
/// missing binding envelope and a column that lost its declaration both
/// fail closed.
#[tokio::test]
async fn test_a_broken_declaration_is_refused_before_refresh() {
let conn = connect("memory://").execute().await.unwrap();
let view = refreshed_computed_view(&conn).await;
let native = view.table().as_native().unwrap();
let mut dataset = native.dataset.get().await.unwrap().as_ref().clone();
dataset
.update_schema_metadata(vec![(
crate::table::computed_columns::FUNCTION_BINDINGS_META_KEY.to_string(),
None,
)])
.await
.unwrap();
let err = view.refresh().execute().await.unwrap_err().to_string();
assert!(err.contains("references missing binding 'fb_1'"), "{err}");
let conn = connect("memory://").execute().await.unwrap();
let view = refreshed_computed_view(&conn).await;
let native = view.table().as_native().unwrap();
let mut dataset = native.dataset.get().await.unwrap().as_ref().clone();
dataset
.replace_field_metadata(vec![(
dataset.schema().field("emb").unwrap().id as u32,
HashMap::new(),
)])
.await
.unwrap();
let err = view.refresh().execute().await.unwrap_err().to_string();
assert!(err.contains("does not match binding 'fb_1'"), "{err}");
}
/// An input the view does not project is materialized on every refresh
/// path, before the provenance column, with the source's values.
#[tokio::test]
async fn test_internal_inputs_are_materialized_and_refreshed() {
use crate::materialized_view::tests::{computed_field, strict_people, test_binding};
let conn = connect("memory://").execute().await.unwrap();
let source = strict_people(&conn).await;
let mut prepared = crate::materialized_view::prepare_declaration(
&source,
Some(&[("id".to_string(), "id".to_string())]),
None,
None,
)
.await
.unwrap();
let input = prepared.input_column("name").unwrap();
let view = prepared
.with_computed_columns(
vec![(1, computed_field("emb", "fb_1", &input))],
&[test_binding("fb_1", &input, "emb")],
)
.unwrap()
.create("v")
.await
.unwrap();
let names: Vec<String> = view
.table()
.schema()
.await
.unwrap()
.fields()
.iter()
.map(|f| f.name().clone())
.collect();
assert_eq!(names, ["id", "emb", "__input_name", SOURCE_ROW_ID_COLUMN]);
let unfilled_inputs = || async {
view.table()
.count_rows(Some("__input_name IS NULL".to_string()))
.await
.unwrap()
};
assert_eq!(
view.refresh().execute().await.unwrap().mode,
RefreshMode::Rebuild
);
assert_eq!(view.table().count_rows(None).await.unwrap(), 3);
assert_eq!(unfilled_inputs().await, 0);
let more = arrow_array::RecordBatch::try_new(
source.schema().await.unwrap(),
vec![
Arc::new(Int32Array::from(vec![4])),
Arc::new(arrow_array::StringArray::from(vec!["d"])),
],
)
.unwrap();
source.add(more).execute().await.unwrap();
assert_eq!(
view.refresh().execute().await.unwrap().mode,
RefreshMode::Incremental
);
assert_eq!(unfilled_inputs().await, 0);
assert_eq!(
view.table()
.count_rows(Some("__input_name = 'd'".to_string()))
.await
.unwrap(),
1
);
source
.update()
.column("name", "'z'")
.only_if("id = 1")
.execute()
.await
.unwrap();
view.refresh().execute().await.unwrap();
assert_eq!(
view.table()
.count_rows(Some("__input_name = 'z'".to_string()))
.await
.unwrap(),
1
);
assert_eq!(
unfilled(&view).await,
4,
"rewritten and new rows are unfilled"
);
}
/// A SQL declaration is filled by `refresh_column` on the view, which
/// commits a data replacement; the next refresh continues from its
/// watermark and keeps what the fill wrote, and only rows the view added
/// since come back unfilled.
#[tokio::test]
async fn test_a_sql_fill_is_not_drift() {
use crate::materialized_view::tests::{people, sql_field};
let conn = connect("memory://").execute().await.unwrap();
let source = people(&conn).await;
let view = crate::materialized_view::prepare_declaration(
&source,
Some(&[("id".to_string(), "id".to_string())]),
None,
None,
)
.await
.unwrap()
.with_computed_columns(
vec![(
1,
sql_field("next", arrow_schema::DataType::Int32, "id + 1", r#"["id"]"#),
)],
&[],
)
.unwrap()
.create("v")
.await
.unwrap();
let filled = || async {
view.table()
.count_rows(Some("next = id + 1".to_string()))
.await
.unwrap()
};
assert_eq!(
view.refresh().execute().await.unwrap().mode,
RefreshMode::Rebuild
);
assert_eq!(
view.table()
.refresh_column("next")
.await
.unwrap()
.rows_filled,
3
);
assert_eq!(filled().await, 3);
assert_eq!(
view.refresh().execute().await.unwrap().mode,
RefreshMode::NoOp
);
assert_eq!(filled().await, 3);
append_people(&conn, vec![4], vec!["d"]).await;
assert_eq!(
view.refresh().execute().await.unwrap().mode,
RefreshMode::Incremental
);
assert_eq!(filled().await, 3);
assert_eq!(
view.table()
.refresh_column("next")
.await
.unwrap()
.rows_filled,
1
);
assert_eq!(filled().await, 4);
assert_eq!(
view.refresh().execute().await.unwrap().mode,
RefreshMode::NoOp
);
}
/// A fill of a nested computed column writes its child fields; that is
/// still a fill, not drift.
#[tokio::test]
async fn test_a_nested_sql_fill_is_not_drift() {
use crate::materialized_view::tests::{people, sql_field};
let conn = connect("memory://").execute().await.unwrap();
let source = people(&conn).await;
let payload = sql_field(
"payload",
arrow_schema::DataType::Struct(
vec![arrow_schema::Field::new(
"value",
arrow_schema::DataType::Utf8,
true,
)]
.into(),
),
"named_struct('value', name)",
r#"["name"]"#,
);
let view = crate::materialized_view::prepare_declaration(
&source,
Some(&[("name".to_string(), "name".to_string())]),
None,
None,
)
.await
.unwrap()
.with_computed_columns(vec![(1, payload)], &[])
.unwrap()
.create("v")
.await
.unwrap();
view.refresh().execute().await.unwrap();
assert_eq!(
view.table()
.refresh_column("payload")
.await
.unwrap()
.rows_filled,
3
);
assert_eq!(
view.refresh().execute().await.unwrap().mode,
RefreshMode::NoOp
);
assert_eq!(
view.table()
.count_rows(Some("payload.value = name".to_string()))
.await
.unwrap(),
3
);
}
} }
+1 -1
View File
@@ -1299,7 +1299,7 @@ impl VectorQuery {
/// This can be useful when there is a narrow filter to allow these queries to /// This can be useful when there is a narrow filter to allow these queries to
/// spend more time searching and avoid potential false negatives. /// spend more time searching and avoid potential false negatives.
/// ///
/// Set to None to search all partitions, if needed, to satsify the limit /// Set to None to search all partitions, if needed, to satisfy the limit
pub fn maximum_nprobes(mut self, maximum_nprobes: Option<usize>) -> Result<Self> { pub fn maximum_nprobes(mut self, maximum_nprobes: Option<usize>) -> Result<Self> {
if let Some(maximum_nprobes) = maximum_nprobes { if let Some(maximum_nprobes) = maximum_nprobes {
if maximum_nprobes == 0 { if maximum_nprobes == 0 {
+3 -43
View File
@@ -688,7 +688,6 @@ impl<S: HttpSend> Database for RemoteDatabase<S> {
async fn list_jobs(&self) -> Result<Vec<JobInfo>> { async fn list_jobs(&self) -> Result<Vec<JobInfo>> {
let mut out = Vec::new(); let mut out = Vec::new();
let mut page_token: Option<String> = None; let mut page_token: Option<String> = None;
let mut seen_page_tokens = HashSet::new();
for page in 0..MAX_LIST_JOBS_PAGES { for page in 0..MAX_LIST_JOBS_PAGES {
let mut body = serde_json::json!({}); let mut body = serde_json::json!({});
if let Some(token) = &page_token { if let Some(token) = &page_token {
@@ -697,8 +696,7 @@ impl<S: HttpSend> Database for RemoteDatabase<S> {
let req = self.client.post("/v1/jobs/list").json(&body); let req = self.client.post("/v1/jobs/list").json(&body);
let (request_id, rsp) = self.client.send(req).await?; let (request_id, rsp) = self.client.send(req).await?;
let rsp = self.client.check_response(&request_id, rsp).await?; let rsp = self.client.check_response(&request_id, rsp).await?;
let status = rsp.status(); let body: RemoteListJobsResponse = rsp.json().await.err_to_http(request_id)?;
let body: RemoteListJobsResponse = rsp.json().await.err_to_http(request_id.clone())?;
out.extend(body.jobs.into_iter().map(|row| JobInfo { out.extend(body.jobs.into_iter().map(|row| JobInfo {
job_id: row.job_id, job_id: row.job_id,
table: row.table, table: row.table,
@@ -706,17 +704,10 @@ impl<S: HttpSend> Database for RemoteDatabase<S> {
state: job_state_to_client(&row.state), state: job_state_to_client(&row.state),
created_at_millis: row.created_at_millis, created_at_millis: row.created_at_millis,
})); }));
let Some(next_page_token) = body.page_token.filter(|token| !token.is_empty()) else { page_token = body.page_token;
if page_token.is_none() {
break; break;
};
if !seen_page_tokens.insert(next_page_token.clone()) {
return Err(Error::Http {
source: "Job listing response repeated a page_token".into(),
request_id,
status_code: Some(status),
});
} }
page_token = Some(next_page_token);
if page + 1 == MAX_LIST_JOBS_PAGES { if page + 1 == MAX_LIST_JOBS_PAGES {
log::warn!( log::warn!(
"list_jobs truncated after {} pages ({} jobs)", "list_jobs truncated after {} pages ({} jobs)",
@@ -2631,37 +2622,6 @@ mod tests {
assert_eq!(jobs[2].state, "failed"); assert_eq!(jobs[2].state, "failed");
} }
#[tokio::test]
async fn test_list_jobs_rejects_a_page_token_cycle() {
let requests = Arc::new(AtomicUsize::new(0));
let seen = requests.clone();
let conn = Connection::new_with_handler(move |request| {
let body: serde_json::Value =
serde_json::from_slice(request.body().unwrap().as_bytes().unwrap()).unwrap();
match seen.fetch_add(1, Ordering::SeqCst) {
0 => assert!(body.get("page_token").is_none()),
_ => assert_eq!(body["page_token"], "loop"),
}
http::Response::builder()
.status(200)
.body(r#"{"jobs": [], "page_token": "loop"}"#)
.unwrap()
});
let error = conn.list_jobs().await.unwrap_err();
assert!(
matches!(
&error,
Error::Http {
status_code: Some(http::StatusCode::OK),
..
}
),
"got {error:?}"
);
assert_eq!(requests.load(Ordering::SeqCst), 2);
}
#[tokio::test] #[tokio::test]
async fn test_open_job() { async fn test_open_job() {
let conn = Connection::new_with_handler(|request| { let conn = Connection::new_with_handler(|request| {
+1 -1
View File
@@ -8021,7 +8021,7 @@ mod tests {
match request.url().path() { match request.url().path() {
"/v1/table/my_table/backfill_column" => http::Response::builder() "/v1/table/my_table/backfill_column" => http::Response::builder()
.status(202) .status(202)
.body(r#"{"job_id": "j-42"}"#.as_bytes().to_vec()) .body(br#"{"job_id": "j-42"}"#.to_vec())
.unwrap(), .unwrap(),
"/v1/jobs/describe" => http::Response::builder() "/v1/jobs/describe" => http::Response::builder()
.status(200) .status(200)
+5 -4
View File
@@ -240,7 +240,7 @@ enum BadVectorHandling {
/// An error is returned /// An error is returned
#[default] #[default]
Error, Error,
/// The offending row is droppped /// The offending row is dropped
Drop, Drop,
/// The invalid/missing items are replaced by fill_value /// The invalid/missing items are replaced by fill_value
Fill(f32), Fill(f32),
@@ -1326,7 +1326,7 @@ impl Table {
/// Note: if your condition is something like "some_id_column == 7" and /// Note: if your condition is something like "some_id_column == 7" and
/// you are updating many rows (with different ids) then you will get /// you are updating many rows (with different ids) then you will get
/// better performance with a single [`merge_insert`] call instead of /// better performance with a single [`merge_insert`] call instead of
/// repeatedly calilng this method. /// repeatedly calling this method.
pub fn update(&self) -> UpdateBuilder { pub fn update(&self) -> UpdateBuilder {
UpdateBuilder::new(self.inner.clone()) UpdateBuilder::new(self.inner.clone())
} }
@@ -2804,7 +2804,7 @@ impl NativeTable {
namespace_client: Option<Arc<dyn LanceNamespace>>, namespace_client: Option<Arc<dyn LanceNamespace>>,
pushdown_operations: HashSet<NamespaceClientPushdownOperation>, pushdown_operations: HashSet<NamespaceClientPushdownOperation>,
) -> Result<Self> { ) -> Result<Self> {
computed_columns::ensure_no_foreign_declarations(batches.arrow_schema().fields())?; let batches = computed_columns::admit_create_source(batches)?;
// Default params uses format v1. // Default params uses format v1.
let params = params.unwrap_or(WriteParams { let params = params.unwrap_or(WriteParams {
..Default::default() ..Default::default()
@@ -2904,6 +2904,7 @@ impl NativeTable {
pushdown_operations: HashSet<NamespaceClientPushdownOperation>, pushdown_operations: HashSet<NamespaceClientPushdownOperation>,
session: Option<Arc<lance::session::Session>>, session: Option<Arc<lance::session::Session>>,
) -> Result<Self> { ) -> Result<Self> {
let batches = computed_columns::admit_create_source(batches)?;
// Build table_id from namespace + name for the storage options provider // Build table_id from namespace + name for the storage options provider
let mut table_id = namespace.clone(); let mut table_id = namespace.clone();
table_id.push(name.to_string()); table_id.push(name.to_string());
@@ -5677,7 +5678,7 @@ mod tests {
TableStatistics { TableStatistics {
num_rows: 250, num_rows: 250,
num_indices: 0, num_indices: 0,
total_bytes: 8925, total_bytes: 8969,
fragment_stats: FragmentStatistics { fragment_stats: FragmentStatistics {
num_fragments: 11, num_fragments: 11,
num_small_fragments: 11, num_small_fragments: 11,
+152 -1
View File
@@ -21,6 +21,7 @@
//! [`computed_columns`] and [`computed_column_from_field`] read declarations //! [`computed_columns`] and [`computed_column_from_field`] read declarations
//! back off a schema. //! back off a schema.
use futures::StreamExt;
use std::collections::{BTreeSet, HashMap, HashSet}; use std::collections::{BTreeSet, HashMap, HashSet};
use std::sync::Arc; use std::sync::Arc;
@@ -1338,6 +1339,106 @@ pub(crate) fn ensure_batch_writes_no_computed_values(
Ok(()) Ok(())
} }
/// Validate every computed-column declaration `schema` carries against the
/// schema itself: every field with declaration metadata is a complete
/// declaration, a SQL declaration re-plans to the field it declares, a
/// Function declaration satisfies the binding contract, and no declaration
/// reads another computed column. What passes here is what `refresh_column`
/// can execute.
pub(crate) fn ensure_declarations_are_planned(schema: &ArrowSchema) -> Result<()> {
let invalid = |message: String| Error::InvalidInput { message };
// A field with any declaration key is a declaration; a partial one is
// not "no declaration", it is a broken one.
for field in schema.fields() {
if field.metadata().keys().any(|k| is_declaration_key(k))
&& computed_column_from_field(field).is_none()
{
return Err(invalid(format!(
"field '{}' carries an incomplete computed-column declaration",
field.name()
)));
}
}
let declared: HashSet<String> = computed_columns(schema)
.into_iter()
.map(|c| c.name)
.collect();
for column in computed_columns(schema) {
let field = schema.field_with_name(&column.name)?;
if !field.is_nullable() {
return Err(invalid(format!(
"computed column '{}' must be nullable until a refresh fills it",
column.name
)));
}
match &column.kind {
ComputedColumnKind::Sql { expression } => {
let others: Vec<ArrowField> = schema
.fields()
.iter()
.filter(|f| f.name() != &column.name)
.map(|f| f.as_ref().clone())
.collect();
let bound = bind(Arc::new(ArrowSchema::new(others)), &column.name, expression)?;
if let Some(input) = bound.roots.iter().find(|r| declared.contains(*r)) {
return Err(invalid(format!(
"computed column '{}' reads computed column '{input}'",
column.name
)));
}
if &bound.data_type != field.data_type() {
return Err(invalid(format!(
"computed column '{}' is declared as {} but its expression yields {}",
column.name,
field.data_type(),
bound.data_type
)));
}
let mut declared_inputs = column.inputs.clone();
declared_inputs.sort();
if declared_inputs != bound.inputs {
return Err(invalid(format!(
"computed column '{}' declares inputs {:?} but its expression reads {:?}",
column.name, declared_inputs, bound.inputs
)));
}
}
ComputedColumnKind::Function { binding_id, .. } => {
// The binding validator resolves each input's leaf; the
// no-computed-input rule is about the root it hangs from.
let bindings = function_bindings(schema)?;
let Some(binding) = bindings.iter().find(|b| b.binding_id() == binding_id) else {
continue; // reported by the binding validator below
};
// Roots come from the canonical path parser: a quoted
// top-level name may itself contain a dot.
if let Some(input) = binding
.inputs()
.iter()
.filter_map(|input| resolve_field_path(schema, &input.field_path).ok())
.map(|resolved| resolved.root.name().as_str())
.find(|r| declared.contains(*r))
{
return Err(invalid(format!(
"computed column '{}' reads computed column '{input}'",
column.name
)));
}
}
ComputedColumnKind::Unrecognized { kind } => {
return Err(Error::NotSupported {
message: format!(
"computed column '{}' is defined by '{kind}', which this version \
of lancedb cannot fill",
column.name
),
});
}
}
}
ensure_supported_function_metadata(schema)
}
/// Reject fields carrying declaration metadata that did not come through /// Reject fields carrying declaration metadata that did not come through
/// [`plan`]. One authority for creation, overwrite and raw transforms. /// [`plan`]. One authority for creation, overwrite and raw transforms.
pub(crate) fn ensure_no_foreign_declarations<'a>( pub(crate) fn ensure_no_foreign_declarations<'a>(
@@ -1796,6 +1897,54 @@ pub(super) async fn add_foreign_kind(table: &crate::Table, name: &str, kind: &st
.unwrap(); .unwrap();
} }
/// Admit a table's initial data: every declaration it carries is validated,
/// and the stream refuses any batch with values in a computed column, whose
/// values come from refresh alone. One boundary for every way a table is
/// created.
pub(crate) fn admit_create_source<S: lance_datafusion::utils::StreamingWriteSource>(
batches: S,
) -> Result<UnfilledDeclarations<S>> {
let schema = batches.arrow_schema();
ensure_declarations_are_planned(&schema)?;
let declared = computed_columns(&schema)
.into_iter()
.map(|c| c.name)
.collect();
Ok(UnfilledDeclarations {
inner: batches,
declared,
})
}
/// A write source whose computed columns must arrive unfilled.
pub(crate) struct UnfilledDeclarations<S> {
inner: S,
declared: Vec<String>,
}
impl<S: lance_datafusion::utils::StreamingWriteSource> lance_datafusion::utils::StreamingWriteSource
for UnfilledDeclarations<S>
{
fn arrow_schema(&self) -> SchemaRef {
self.inner.arrow_schema()
}
fn into_stream(self) -> datafusion_physical_plan::SendableRecordBatchStream {
if self.declared.is_empty() {
return self.inner.into_stream();
}
let schema = self.inner.arrow_schema();
let declared = self.declared;
let stream = self.inner.into_stream().map(move |batch| {
let batch = batch?;
ensure_batch_writes_no_computed_values(&declared, &batch)
.map_err(|e| datafusion_common::DataFusionError::External(Box::new(e)))?;
Ok(batch)
});
Box::pin(datafusion_physical_plan::stream::RecordBatchStreamAdapter::new(schema, stream))
}
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
/// The gate's reproducer: the validator applies the same schema-level /// The gate's reproducer: the validator applies the same schema-level
@@ -2646,6 +2795,8 @@ mod tests {
); );
} }
/// A create carries a declaration only if it re-plans completely; this
/// one lacks its inputs and is refused before its forged value matters.
#[tokio::test] #[tokio::test]
async fn test_create_table_cannot_inject_a_declaration() { async fn test_create_table_cannot_inject_a_declaration() {
let conn = connect("memory://").execute().await.unwrap(); let conn = connect("memory://").execute().await.unwrap();
@@ -2673,7 +2824,7 @@ mod tests {
.await .await
.unwrap_err(); .unwrap_err();
assert!( assert!(
matches!(&err, Error::InvalidInput { message } if message.contains("computed()")), matches!(&err, Error::InvalidInput { message } if message.contains("computed column 'doubled'")),
"{err:?}" "{err:?}"
); );
} }
@@ -5,12 +5,17 @@
//! //!
//! [`super::cast::cast_to_table_schema`] calls [`coerce_blob_expr`]. //! [`super::cast::cast_to_table_schema`] calls [`coerce_blob_expr`].
use std::fmt;
use std::hash::{Hash, Hasher};
use std::sync::Arc; use std::sync::Arc;
use arrow_schema::{DataType, Field, FieldRef, Fields}; use arrow_array::{Array, BooleanArray, RecordBatch};
use arrow_schema::{DataType, Field, FieldRef, Fields, Schema};
use arrow_select::nullif::nullif;
use datafusion::functions::core::{get_field, named_struct}; use datafusion::functions::core::{get_field, named_struct};
use datafusion_common::ScalarValue; use datafusion_common::ScalarValue;
use datafusion_common::config::ConfigOptions; use datafusion_common::config::ConfigOptions;
use datafusion_expr::ColumnarValue;
use datafusion_physical_expr::ScalarFunctionExpr; use datafusion_physical_expr::ScalarFunctionExpr;
use datafusion_physical_expr::expressions::{CastExpr, Literal}; use datafusion_physical_expr::expressions::{CastExpr, Literal};
use datafusion_physical_plan::PhysicalExpr; use datafusion_physical_plan::PhysicalExpr;
@@ -133,16 +138,102 @@ pub(super) fn coerce_blob_expr(
ns_args.push(value); ns_args.push(value);
} }
let expr: Arc<dyn PhysicalExpr> = Arc::new(ScalarFunctionExpr::new( let built: Arc<dyn PhysicalExpr> = Arc::new(ScalarFunctionExpr::new(
&format!("named_struct({})", table_field.name()), &format!("named_struct({})", table_field.name()),
named_struct(), named_struct(),
ns_args, ns_args,
table_field.clone(), table_field.clone(),
config.clone(), config.clone(),
)); ));
// `named_struct` always yields a valid struct, so a null input would land
// as a row that set neither `data` nor `uri` -- not an absent blob but a
// malformed one, which Lance rejects on write.
let expr: Arc<dyn PhysicalExpr> = Arc::new(AbsentBlobIsNull {
source: input_expr,
built,
field: table_field.clone(),
});
Ok((expr, table_field.clone())) Ok((expr, table_field.clone()))
} }
/// Carries the source column's nullity onto the struct built for it.
///
/// This is its own expression rather than a `CASE` because the projection
/// takes its output field from `return_field`, and the generic implementation
/// rebuilds a bare field -- which would drop the `lance.blob.v2` extension
/// metadata and stop the column being recognised as a blob at all.
#[derive(Debug, Clone)]
struct AbsentBlobIsNull {
source: Arc<dyn PhysicalExpr>,
built: Arc<dyn PhysicalExpr>,
field: FieldRef,
}
impl fmt::Display for AbsentBlobIsNull {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "absent_blob_is_null({}, {})", self.source, self.built)
}
}
impl PartialEq for AbsentBlobIsNull {
fn eq(&self, other: &Self) -> bool {
self.source.eq(&other.source) && self.built.eq(&other.built) && self.field == other.field
}
}
impl Eq for AbsentBlobIsNull {}
impl Hash for AbsentBlobIsNull {
fn hash<H: Hasher>(&self, state: &mut H) {
self.source.hash(state);
self.built.hash(state);
self.field.hash(state);
}
}
impl PhysicalExpr for AbsentBlobIsNull {
fn return_field(&self, _input_schema: &Schema) -> datafusion_common::Result<FieldRef> {
Ok(self.field.clone())
}
fn nullable(&self, _input_schema: &Schema) -> datafusion_common::Result<bool> {
Ok(true)
}
fn evaluate(&self, batch: &RecordBatch) -> datafusion_common::Result<ColumnarValue> {
let rows = batch.num_rows();
let built = self.built.evaluate(batch)?.into_array(rows)?;
let source = self.source.evaluate(batch)?.into_array(rows)?;
let Some(nulls) = source.logical_nulls() else {
return Ok(ColumnarValue::Array(built));
};
// `nullif` nulls the rows the mask marks true, which is where the
// source had no value.
let absent = BooleanArray::new(!nulls.inner(), None);
Ok(ColumnarValue::Array(nullif(built.as_ref(), &absent)?))
}
fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
vec![&self.source, &self.built]
}
fn with_new_children(
self: Arc<Self>,
children: Vec<Arc<dyn PhysicalExpr>>,
) -> datafusion_common::Result<Arc<dyn PhysicalExpr>> {
Ok(Arc::new(Self {
source: children[0].clone(),
built: children[1].clone(),
field: self.field.clone(),
}))
}
fn fmt_sql(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{self}")
}
}
enum BlobInputShape<'a> { enum BlobInputShape<'a> {
Bytes, Bytes,
String, String,
@@ -313,6 +404,11 @@ mod tests {
let data = image.column_by_name("data").unwrap(); let data = image.column_by_name("data").unwrap();
assert!(!data.is_null(0)); assert!(!data.is_null(0));
assert!(data.is_null(1)); assert!(data.is_null(1));
// The row itself has to be null, not merely a struct whose children
// are. A present-but-empty struct set neither `data` nor `uri`, which
// Lance rejects as malformed rather than reading as an absent blob.
assert!(!image.is_null(0));
assert!(image.is_null(1));
} }
#[tokio::test] #[tokio::test]
+1 -1
View File
@@ -52,7 +52,7 @@ enum ConsistencyMode {
/// refresh_window = min(3s, TTL/4) /// refresh_window = min(3s, TTL/4)
/// ///
/// | t < TTL - refresh_window | t < TTL | t >= TTL | /// | t < TTL - refresh_window | t < TTL | t >= TTL |
/// | Return value | Background refresh & return value | syncronous refresh | /// | Return value | Background refresh & return value | synchronous refresh |
Eventual(BackgroundCache<Arc<Dataset>, Error>), Eventual(BackgroundCache<Arc<Dataset>, Error>),
} }
+1 -1
View File
@@ -103,7 +103,7 @@ impl MergeInsertBuilder {
/// but that behavior is subject to change. /// but that behavior is subject to change.
/// ///
/// An optional condition may be specified. If it is, then only /// An optional condition may be specified. If it is, then only
/// matched rows that satisfy the condtion will be updated. Any /// matched rows that satisfy the condition will be updated. Any
/// rows that do not satisfy the condition will be left as they /// rows that do not satisfy the condition will be left as they
/// are. Failing to satisfy the condition does not cause a /// are. Failing to satisfy the condition does not cause a
/// "matched row" to become a "not matched" row. /// "matched row" to become a "not matched" row.
+1 -1
View File
@@ -904,7 +904,7 @@ fn unsharded_shard_id() -> Uuid {
/// Build a [`ShardWriterConfig`] from the persisted `writer_config_defaults`. /// Build a [`ShardWriterConfig`] from the persisted `writer_config_defaults`.
/// ///
/// Unknown or unparseable keys are ignored; absent keys keep the /// Unknown or unparsable keys are ignored; absent keys keep the
/// [`ShardWriterConfig`] default. The shard id is set by `mem_wal_writer`. /// [`ShardWriterConfig`] default. The shard id is set by `mem_wal_writer`.
fn shard_writer_config_from_defaults(defaults: &HashMap<String, String>) -> ShardWriterConfig { fn shard_writer_config_from_defaults(defaults: &HashMap<String, String>) -> ShardWriterConfig {
let mut config = ShardWriterConfig::default().with_shard_spec_id(SHARDING_SPEC_ID); let mut config = ShardWriterConfig::default().with_shard_spec_id(SHARDING_SPEC_ID);
+10 -2
View File
@@ -19,6 +19,7 @@ use lancedb::{
connect, connect_namespace, connect, connect_namespace,
database::listing::{ database::listing::{
ListingDatabaseOptions, NewTableConfig, OPT_NEW_TABLE_ENABLE_STABLE_ROW_IDS, ListingDatabaseOptions, NewTableConfig, OPT_NEW_TABLE_ENABLE_STABLE_ROW_IDS,
OPT_NEW_TABLE_STORAGE_VERSION,
}, },
query::{ExecutableQuery, QueryBase}, query::{ExecutableQuery, QueryBase},
table::{AddDataMode, CompactionOptions, OptimizeAction, OptimizeStats, WriteOptions}, table::{AddDataMode, CompactionOptions, OptimizeAction, OptimizeStats, WriteOptions},
@@ -146,7 +147,10 @@ async fn non_blob_table_keeps_default_format_and_row_id_setting() -> Result<()>
let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)])); let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)]));
let table = db.create_empty_table("t", schema).execute().await?; let table = db.create_empty_table("t", schema).execute().await?;
assert!(!supports_blob_v2(storage_format_version(&table).await)); assert_eq!(
storage_format_version(&table).await,
LanceFileVersion::Stable.resolve()
);
assert!(!uses_stable_row_ids(&table).await); assert!(!uses_stable_row_ids(&table).await);
Ok(()) Ok(())
} }
@@ -809,7 +813,11 @@ async fn fetch_blobs_rejects_unknown_column() -> Result<()> {
#[tokio::test] #[tokio::test]
async fn fetch_blobs_rejects_legacy_v1_blob_column() -> Result<()> { async fn fetch_blobs_rejects_legacy_v1_blob_column() -> Result<()> {
let tmp = tempdir().unwrap(); let tmp = tempdir().unwrap();
let db = connect(tmp.path().to_str().unwrap()).execute().await?; // Legacy v1 blob columns are only writable at file version <= 2.1.
let db = connect(tmp.path().to_str().unwrap())
.storage_options([(OPT_NEW_TABLE_STORAGE_VERSION, "2.1")])
.execute()
.await?;
let legacy = Field::new("image", DataType::LargeBinary, true).with_metadata( let legacy = Field::new("image", DataType::LargeBinary, true).with_metadata(
std::collections::HashMap::from([("lance-encoding:blob".to_string(), "true".to_string())]), std::collections::HashMap::from([("lance-encoding:blob".to_string(), "true".to_string())]),
); );