Compare commits

...

19 Commits

Author SHA1 Message Date
Xuanwo d383c37967 feat: support nested blob function signatures 2026-09-01 16:54:15 +08:00
Xuanwo 5cbd979455 fix: preserve namespace drop errors (#4099)
## Summary

- preserve typed namespace errors returned by `drop_table`
- return `TableNotFound` when dropping an absent namespace table
- cover repeated drop behavior in the namespace database test

## Validation

- `cargo test --quiet --features remote -p lancedb
database::namespace::tests::test_namespace_drop_table --lib`
- `cargo clippy --quiet --features remote -p lancedb --lib --tests -- -D
warnings`

## Context

Sophon SQL implements `DROP TABLE IF EXISTS` by matching
`lancedb::Error::TableNotFound`. The namespace database previously
wrapped this error as `Runtime`, causing cleanup to fail and mask an
earlier statement error.
2026-08-31 14:02:40 -07:00
dependabot[bot] e773d1e093 build(deps): bump the rust-minor-patch group across 1 directory with 9 updates (#4084)
Bumps the rust-minor-patch group with 9 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [async-trait](https://github.com/dtolnay/async-trait) | `0.1.91` |
`0.1.92` |
| [log](https://github.com/rust-lang/log) | `0.4.33` | `0.4.34` |
| [moka](https://github.com/moka-rs/moka) | `0.12.15` | `0.12.16` |
| [uuid](https://github.com/uuid-rs/uuid) | `1.24.0` | `1.26.0` |
| [serde_with](https://github.com/jonasbb/serde_with) | `3.21.0` |
`3.22.0` |
| [roaring](https://github.com/RoaringBitmap/roaring-rs) | `0.11.4` |
`0.11.5` |
| [napi](https://github.com/napi-rs/napi-rs) | `3.11.0` | `3.12.0` |
| [napi-derive](https://github.com/napi-rs/napi-rs) | `3.6.1` | `3.6.3`
|
| [napi-build](https://github.com/napi-rs/napi-rs) | `2.4.0` | `2.4.1` |


Updates `async-trait` from 0.1.91 to 0.1.92
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/async-trait/releases">async-trait's
releases</a>.</em></p>
<blockquote>
<h2>0.1.92</h2>
<ul>
<li>Resolve double_must_use clippy lint in generated code (<a
href="https://redirect.github.com/dtolnay/async-trait/issues/303">#303</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/async-trait/commit/82e7e9edd60f622294373a23c0ce9c0077ad0263"><code>82e7e9e</code></a>
Release 0.1.92</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/9a35cb87f9366cd992bbc00d430e1b5fe1aa0cdd"><code>9a35cb8</code></a>
Merge pull request <a
href="https://redirect.github.com/dtolnay/async-trait/issues/303">#303</a>
from dtolnay/mustuse</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/875ceecb100bab2cf369178633b4791336d92b75"><code>875ceec</code></a>
Resolve double_must_use clippy lint</li>
<li><a
href="https://github.com/dtolnay/async-trait/commit/62993a57bc6a8d5bd3de23fbae48cede333cb925"><code>62993a5</code></a>
Raise minimum tested compiler to rust 1.88</li>
<li>See full diff in <a
href="https://github.com/dtolnay/async-trait/compare/0.1.91...0.1.92">compare
view</a></li>
</ul>
</details>
<br />

Updates `log` from 0.4.33 to 0.4.34
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/log/releases">log's
releases</a>.</em></p>
<blockquote>
<h2>0.4.34</h2>
<h2>What's Changed</h2>
<ul>
<li>doc: Add context-logger utility to README by <a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li>Add alloc support for boxed loggers by <a
href="https://github.com/malezjaa"><code>@​malezjaa</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li><a href="https://github.com/malezjaa"><code>@​malezjaa</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rust-lang/log/compare/0.4.33...0.4.34">https://github.com/rust-lang/log/compare/0.4.33...0.4.34</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/log/blob/master/CHANGELOG.md">log's
changelog</a>.</em></p>
<blockquote>
<h2>[0.4.34] - 2026-08-22</h2>
<h2>What's Changed</h2>
<ul>
<li>doc: Add context-logger utility to README by <a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li>Add alloc support for boxed loggers by <a
href="https://github.com/malezjaa"><code>@​malezjaa</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/alekseysidorov"><code>@​alekseysidorov</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/735">rust-lang/log#735</a></li>
<li><a href="https://github.com/malezjaa"><code>@​malezjaa</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/737">rust-lang/log#737</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rust-lang/log/compare/0.4.33...0.4.34">https://github.com/rust-lang/log/compare/0.4.33...0.4.34</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rust-lang/log/commit/8034743dd9d7f7583bd9a670271483d176130911"><code>8034743</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/log/issues/738">#738</a>
from rust-lang/cargo/0.4.34</li>
<li><a
href="https://github.com/rust-lang/log/commit/7d1e24e3506d4ffa1badf6c9ea357779877adaf0"><code>7d1e24e</code></a>
prepare for 0.4.34 release</li>
<li><a
href="https://github.com/rust-lang/log/commit/3b939b6714616dc32193c12019861c7c518c5edb"><code>3b939b6</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/log/issues/737">#737</a>
from malezjaa/master</li>
<li><a
href="https://github.com/rust-lang/log/commit/b88266cfed8b287f8c35b2015808b09b056f61af"><code>b88266c</code></a>
Add alloc support for boxed loggers</li>
<li><a
href="https://github.com/rust-lang/log/commit/037d7a58f6ad184abb3afc4db81d37c43a5696ec"><code>037d7a5</code></a>
doc: Add context-logger utility to README</li>
<li>See full diff in <a
href="https://github.com/rust-lang/log/compare/0.4.33...0.4.34">compare
view</a></li>
</ul>
</details>
<br />

Updates `moka` from 0.12.15 to 0.12.16
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/moka-rs/moka/releases">moka's
releases</a>.</em></p>
<blockquote>
<h2>v0.12.16</h2>
<h2>Version 0.12.16</h2>
<h3>Fixed</h3>
<ul>
<li>Fixed a bug where cache eviction could stall permanently when the
cache was configured with the <strong>non-default</strong> LRU eviction
policy (<code>EvictionPolicy::lru()</code>) by a race between insert and
remove operations on the same key (<a
href="https://redirect.github.com/moka-rs/moka/issues/592">#592</a><a
href="https://redirect.github.com/moka-rs/moka/pull/592/">gh-pull-0592</a>
by <a
href="https://github.com/kim-jhyeon"><code>@​kim-jhyeon</code></a>,
reported in <a
href="https://redirect.github.com/moka-rs/moka/issues/590">#590</a><a
href="https://redirect.github.com/moka-rs/moka/issues/590/">gh-issue-0590</a>):
<ul>
<li>This bug was introduced in v0.12.0 and affected
<code>sync::Cache</code>, <code>sync::SegmentedCache</code> and
<code>future::Cache</code>.</li>
<li>A race between applying a write recording for an entry and
concurrently removing that entry from the internal concurrent hash table
could leave an orphaned node at the front of the LRU queue. Once
present, no entry was ever evicted again and the cache grew unboundedly
past <code>max_capacity</code>.</li>
<li>The same race also affected the default TinyLFU eviction policy, but
with a milder symptom: each occurrence permanently leaked one phantom
entry slot, causing <code>entry_count</code> and
<code>weighted_size</code> to over-report and the usable capacity to
shrink by one entry per occurrence. Fixed by the same change.</li>
</ul>
</li>
</ul>
<h3>Changed</h3>
<ul>
<li>Worked around a ThreadSanitizer false positive (<a
href="https://redirect.github.com/moka-rs/moka/issues/602">#602</a><a
href="https://redirect.github.com/moka-rs/moka/pull/602/">gh-pull-0602</a>):
<ul>
<li>Replaced the standalone <code>fence(Acquire)</code> in the internal
<code>MiniArc</code>'s drop path with an <code>Acquire</code> load of
the reference count, so that downstream projects can now run
ThreadSanitizer on code using Moka without hitting this false
positive.</li>
<li><code>std::sync::Arc</code> has a similar workaround.</li>
</ul>
</li>
<li>Raised the minimum version of the <code>crossbeam-epoch</code> crate
from <code>v0.9.18</code> to <code>v0.9.20</code> to avoid the following
advisory (<a
href="https://redirect.github.com/moka-rs/moka/issues/603">#603</a><a
href="https://redirect.github.com/moka-rs/moka/pull/603/">gh-pull-0603</a>):
<ul>
<li>[RUSTSEC-2026-0204] crossbeam-epoch: invalid pointer dereference in
<code>fmt::Pointer</code> for <code>Atomic</code> and
<code>Shared</code></li>
<li>Moka is <em>not</em> affected by this advisory because it never
formats these pointer types. However, raising the minimum version
prevents downstream lockfiles from resolving to an affected
<code>crossbeam-epoch</code> version via Moka.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/moka-rs/moka/blob/main/CHANGELOG.md">moka's
changelog</a>.</em></p>
<blockquote>
<h2>Version 0.12.16</h2>
<h3>Fixed</h3>
<ul>
<li>Fixed a bug where cache eviction could stall permanently when the
cache was
configured with the <strong>non-default</strong> LRU eviction policy
(<code>EvictionPolicy::lru()</code>)
by a race between insert and remove operations on the same key
(<a
href="https://redirect.github.com/moka-rs/moka/issues/592">#592</a>[gh-pull-0592]
by [<a
href="https://github.com/kim-jhyeon"><code>@​kim-jhyeon</code></a>][gh-kim-jhyeon],
reported in
<a
href="https://redirect.github.com/moka-rs/moka/issues/590">#590</a>[gh-issue-0590]):
<ul>
<li>This bug was introduced in v0.12.0 and affected
<code>sync::Cache</code>,
<code>sync::SegmentedCache</code> and <code>future::Cache</code>.</li>
<li>A race between applying a write recording for an entry and
concurrently
removing that entry from the internal concurrent hash table could leave
an
orphaned node at the front of the LRU queue. Once present, no entry was
ever
evicted again and the cache grew unboundedly past
<code>max_capacity</code>.</li>
<li>The same race also affected the default TinyLFU eviction policy, but
with
a milder symptom: each occurrence permanently leaked one phantom entry
slot, causing <code>entry_count</code> and <code>weighted_size</code> to
over-report and the
usable capacity to shrink by one entry per occurrence. Fixed by the same
change.</li>
</ul>
</li>
</ul>
<h3>Changed</h3>
<ul>
<li>Worked around a ThreadSanitizer false positive (<a
href="https://redirect.github.com/moka-rs/moka/issues/602">#602</a>[gh-pull-0602]):
<ul>
<li>Replaced the standalone <code>fence(Acquire)</code> in the internal
<code>MiniArc</code>'s drop
path with an <code>Acquire</code> load of the reference count, so that
downstream
projects can now run ThreadSanitizer on code using Moka without hitting
this false positive.</li>
<li><code>std::sync::Arc</code> has a similar workaround.</li>
</ul>
</li>
<li>Raised the minimum version of the <code>crossbeam-epoch</code> crate
from <code>v0.9.18</code> to
<code>v0.9.20</code> to avoid the following advisory (<a
href="https://redirect.github.com/moka-rs/moka/issues/603">#603</a>[gh-pull-0603]):
<ul>
<li>[RUSTSEC-2026-0204] crossbeam-epoch: invalid pointer dereference in
<code>fmt::Pointer</code> for <code>Atomic</code> and
<code>Shared</code></li>
<li>Moka is <em>not</em> affected by this advisory because it never
formats these
pointer types. However, raising the minimum version prevents downstream
lockfiles from resolving to an affected <code>crossbeam-epoch</code>
version via
Moka.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/moka-rs/moka/commit/a616ec19e8d4ed938caf8b2c88090331d778d5da"><code>a616ec1</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/604">#604</a> from
moka-rs/chore/bump-v0.12.16</li>
<li><a
href="https://github.com/moka-rs/moka/commit/3b140a627e9faa4ec6a8e682224c7f81efc2b6e4"><code>3b140a6</code></a>
Bump the version to v0.12.16</li>
<li><a
href="https://github.com/moka-rs/moka/commit/51b802dc5cfc9e0da21de04177d79028bfbe5d47"><code>51b802d</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/603">#603</a> from
moka-rs/bump-crossbeam-epoch-floor</li>
<li><a
href="https://github.com/moka-rs/moka/commit/4f9071684161d59212c32a0e89762c3a5d6385a4"><code>4f90716</code></a>
Raise the minimum crossbeam-epoch version to 0.9.20</li>
<li><a
href="https://github.com/moka-rs/moka/commit/08d0e0458bd95af7f9435ff3ffbba6d1e91647c1"><code>08d0e04</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/602">#602</a> from
moka-rs/gh600-tsan-workaround</li>
<li><a
href="https://github.com/moka-rs/moka/commit/14447a7cbe441639e2aa3570e411fe493c71c9ac"><code>14447a7</code></a>
Restructure the v0.12.16 TSan workaround CHANGELOG entry</li>
<li><a
href="https://github.com/moka-rs/moka/commit/7b14c37b009a25c9a2ec27e2669dc5f8db7ce254"><code>7b14c37</code></a>
Avoid a TSan false positive by replacing the fence in MiniArc::drop</li>
<li><a
href="https://github.com/moka-rs/moka/commit/05b37c63098473034e7e961c1010284163ad8634"><code>05b37c6</code></a>
Merge pull request <a
href="https://redirect.github.com/moka-rs/moka/issues/599">#599</a> from
moka-rs/gh590-deterministic-tests</li>
<li><a
href="https://github.com/moka-rs/moka/commit/fc318584d25c0ea01109872da37d395754647e04"><code>fc31858</code></a>
Replace private doc references in gh590 test comments</li>
<li><a
href="https://github.com/moka-rs/moka/commit/57435922036ff4ab9f1b5bd0c3bffe6c8acd9921"><code>5743592</code></a>
Improve the v0.12.16 CHANGELOG entry</li>
<li>Additional commits viewable in <a
href="https://github.com/moka-rs/moka/compare/v0.12.15...v0.12.16">compare
view</a></li>
</ul>
</details>
<br />

Updates `uuid` from 1.24.0 to 1.26.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/uuid-rs/uuid/releases">uuid's
releases</a>.</em></p>
<blockquote>
<h2>v1.26.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Add ContextV7::with_additional_precision_bits by <a
href="https://github.com/ChrisJr404"><code>@​ChrisJr404</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/904">uuid-rs/uuid#904</a></li>
<li>Prepare for 1.26.0 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/905">uuid-rs/uuid#905</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/1.25.0...v1.26.0">https://github.com/uuid-rs/uuid/compare/1.25.0...v1.26.0</a></p>
<h2>1.25.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Add a serde::bytes module that encodes a Uuid as a byte string by <a
href="https://github.com/ChrisJr404"><code>@​ChrisJr404</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/902">uuid-rs/uuid#902</a></li>
<li>Prepare for 1.25.0 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/903">uuid-rs/uuid#903</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/ChrisJr404"><code>@​ChrisJr404</code></a> made
their first contribution in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/902">uuid-rs/uuid#902</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/v1.24.1...1.25.0">https://github.com/uuid-rs/uuid/compare/v1.24.1...1.25.0</a></p>
<h2>v1.24.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Fix non-ASCII character handling in parse diagnostics by <a
href="https://github.com/questfever"><code>@​questfever</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/899">uuid-rs/uuid#899</a></li>
<li>Prepare for 1.24.1 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/900">uuid-rs/uuid#900</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/questfever"><code>@​questfever</code></a> made
their first contribution in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/899">uuid-rs/uuid#899</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/v1.24.0...v1.24.1">https://github.com/uuid-rs/uuid/compare/v1.24.0...v1.24.1</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/uuid-rs/uuid/commit/cdc96a87bddc38d0eb8f894c764e151d2299b4b3"><code>cdc96a8</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/905">#905</a> from
uuid-rs/cargo/v1.26.0</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/34e4f49c0d50c12f1b3021baf98b8fb91f6407bb"><code>34e4f49</code></a>
don't test macros under miri</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/d9e7242b37755d844d19fa74559a88e1c46c5206"><code>d9e7242</code></a>
update nightly used for miri</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/ec16819865b89aa3c52456c8afd0ce9a90f0fcdb"><code>ec16819</code></a>
prepare for 1.26.0 release</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/162cd208a4521138f1d8ce05b63342ba7ba5c4e6"><code>162cd20</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/904">#904</a> from
ChrisJr404/v7-additional-precision-bits</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/97eceffa708f87969792af604291d3e4984dfc90"><code>97eceff</code></a>
Add ContextV7::with_additional_precision_bits for microsecond
clocks</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/302e0bf6dc5abf949c06973a37f1f3a093cc2699"><code>302e0bf</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/903">#903</a> from
uuid-rs/cargo/1.25.0</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/b7ccde885d770d013f413a2685ebe7f38932e1d0"><code>b7ccde8</code></a>
prepare for 1.25.0 release</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/c62dffbc038034ff045f3009f2536362e313bf34"><code>c62dffb</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/902">#902</a> from
ChrisJr404/serde-bytes-module</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/8c198b24b1aa55948c0fa4b3433c1954be19c8c8"><code>8c198b2</code></a>
Add a serde::bytes module that encodes as a byte string</li>
<li>Additional commits viewable in <a
href="https://github.com/uuid-rs/uuid/compare/v1.24.0...v1.26.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `serde_with` from 3.21.0 to 3.22.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/jonasbb/serde_with/releases">serde_with's
releases</a>.</em></p>
<blockquote>
<h2>serde_with v3.22.0</h2>
<h3>Added</h3>
<ul>
<li>Add support for <code>jiff</code> v0.2 behind the new
<code>jiff_0_2</code> feature flag (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/936">#936</a>)
<code>jiff::SignedDuration</code> works with
<code>DurationSeconds</code> and its variants.
<code>jiff::Timestamp</code>, <code>jiff::Zoned</code>, and
<code>jiff::civil::DateTime</code> work with
<code>TimestampSeconds</code> and its variants.
Deserializing a <code>jiff::Zoned</code> uses the system time zone, like
<code>chrono::DateTime&lt;Local&gt;</code>.</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>Extend the <a
href="https://github.com/jonasbb/serde_with/security/advisories/GHSA-7gcf-g7xr-8hxj">GHSA-7gcf-g7xr-8hxj</a>
fix to the duplicate-key-prevention collections.
The <code>rust::sets_duplicate_value_is_error</code>,
<code>rust::maps_duplicate_key_is_error</code>,
<code>rust::sets_last_value_wins</code>, and
<code>rust::maps_first_key_wins</code> adapters created their backing
sets/maps with <code>with_capacity_and_hasher</code> using the raw
deserializer <code>size_hint</code>, bypassing the
<code>size_hint_cautious</code> cap added in <a
href="https://redirect.github.com/jonasbb/serde_with/issues/966">#966</a>
(the <code>clippy.toml</code> <code>disallowed_methods</code> lint only
covers <code>Vec::with_capacity</code>, not
<code>with_capacity_and_hasher</code>, so these sites were not flagged).
Attacker-controlled input claiming a huge length could panic with
<code>Hash table capacity overflow</code> before a single element was
read. All such constructions now route through
<code>size_hint_cautious</code>.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/jonasbb/serde_with/commit/88f576a17c5cd45cea6a30252ef10653dde69fa8"><code>88f576a</code></a>
Bump version to 3.22.0 (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/991">#991</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/931e664445c2139446b84e76b339924f136a1565"><code>931e664</code></a>
Bump version to 3.22.0</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/e26930e0b7a6c1e6463086a2447e7cc8fc6f0a24"><code>e26930e</code></a>
Bump github/codeql-action from 4.37.3 to 4.37.4 in the github-actions
group (...</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/92cd5a0bd5c7a80fc7eae90bb99b873c40429aa3"><code>92cd5a0</code></a>
Bump github/codeql-action in the github-actions group</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/32be66fecc5c1fe4c90ac0230c0af04d1977df53"><code>32be66f</code></a>
Guard with_capacity_and_hasher against untrusted size_hint (DoS) (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/971">#971</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/33871cd4dd1ecef2c3af0ead9528c8b407c16f04"><code>33871cd</code></a>
Merge branch 'master' into
fix/duplicate-key-impls-capacity-overflow</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/bb1e06484261595c8cec7fd7f4ed33ecdfb951c0"><code>bb1e064</code></a>
Change function position within impl (<a
href="https://redirect.github.com/jonasbb/serde_with/issues/968">#968</a>)</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/202d3dd617d7b5a9db5f490fa752d6ccb48454e8"><code>202d3dd</code></a>
Improve the time unit macros to remove unnecessary repetition and make
the co...</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/b347efb536caf83c850d4808f90503835fd78755"><code>b347efb</code></a>
Move the <code>use_duration_signed_ser</code>/<code>*_de</code> macros
utils</li>
<li><a
href="https://github.com/jonasbb/serde_with/commit/65905455527c0abf51f2f906bc08724426b2b922"><code>6590545</code></a>
chrono_0_4: Implement the same time unit macro cleanup as jiff_0_2</li>
<li>Additional commits viewable in <a
href="https://github.com/jonasbb/serde_with/compare/v3.21.0...v3.22.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `roaring` from 0.11.4 to 0.11.5
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/RoaringBitmap/roaring-rs/releases">roaring's
releases</a>.</em></p>
<blockquote>
<h2>v0.11.5</h2>
<h2>What's Changed</h2>
<ul>
<li>Implement std Error for IntegerTooSmall by <a
href="https://github.com/Kerollmops"><code>@​Kerollmops</code></a> in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/362">RoaringBitmap/roaring-rs#362</a></li>
<li>fix: invalid treemap iter advance by <a
href="https://github.com/silver-ymz"><code>@​silver-ymz</code></a> in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/360">RoaringBitmap/roaring-rs#360</a></li>
<li>Fix off-by-one that corrupts a bitmap in
remove_smallest/remove_biggest (<a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/359">#359</a>)
by <a href="https://github.com/youdie006"><code>@​youdie006</code></a>
in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/363">RoaringBitmap/roaring-rs#363</a></li>
<li>Upgrade dependencies bump version by <a
href="https://github.com/Kerollmops"><code>@​Kerollmops</code></a> in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/364">RoaringBitmap/roaring-rs#364</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/silver-ymz"><code>@​silver-ymz</code></a> made
their first contribution in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/360">RoaringBitmap/roaring-rs#360</a></li>
<li><a href="https://github.com/youdie006"><code>@​youdie006</code></a>
made their first contribution in <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/pull/363">RoaringBitmap/roaring-rs#363</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/RoaringBitmap/roaring-rs/compare/v0.11.4...v0.11.5">https://github.com/RoaringBitmap/roaring-rs/compare/v0.11.4...v0.11.5</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/0ce3fc8b55b193ce220253bfbc0c3e09bd171375"><code>0ce3fc8</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/364">#364</a>
from RoaringBitmap/upgrade-dependencies-bump-version</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/a961a042db8d325515e6b5273a2e9369fe5c931d"><code>a961a04</code></a>
Remove the once_cell dependency</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/5e8445b2d6914e8e56d85de340f9156825f4e91b"><code>5e8445b</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/363">#363</a>
from youdie006/fix/359-interval-remove-boundary</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/bf2961d99fb4da2c540a55eb699228a7bb00a732"><code>bf2961d</code></a>
Bump version to v0.11.5</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/048a8b05fcae08636354607f0c00d6e95f262107"><code>048a8b0</code></a>
Fix off-by-one that corrupts a bitmap in
remove_smallest/remove_biggest</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/27d84f567dd85243194d2b87683262ef43a5dd97"><code>27d84f5</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/360">#360</a>
from silver-ymz/fix/treemap-iter-advance-across-bitmaps</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/aac2de82a7f9e44fd73d8364de840169447d580b"><code>aac2de8</code></a>
Make clippy happy</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/a3d1d54be985fe22c01e882f85c3ee7055ad9c8b"><code>a3d1d54</code></a>
Merge pull request <a
href="https://redirect.github.com/RoaringBitmap/roaring-rs/issues/362">#362</a>
from RoaringBitmap/std-error-for-integer-too-small</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/9a3c33e42c0b14bdd3f296313ee367526092aa81"><code>9a3c33e</code></a>
Implement std Error for IntegerTooSmall</li>
<li><a
href="https://github.com/RoaringBitmap/roaring-rs/commit/f46c0ffe90b6d5d52a93106253bb6fa51a08c137"><code>f46c0ff</code></a>
fix: invalid treemap iter advance</li>
<li>See full diff in <a
href="https://github.com/RoaringBitmap/roaring-rs/compare/v0.11.4...v0.11.5">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi` from 3.11.0 to 3.12.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi's
releases</a>.</em></p>
<blockquote>
<h2>napi-v3.12.0</h2>
<h3>Added</h3>
<ul>
<li><em>(cli)</em> support non-threaded WASI targets (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3353">#3353</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/58bd87fa524a837a7c962ab4103e5588557ccd81"><code>58bd87f</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3414">#3414</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/9da87236dbc4fef99f066b7a130f4d0377308d44"><code>9da8723</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/8d22196aa98a1e6e70584561f5446d117d9c802c"><code>8d22196</code></a>
chore(deps): update dependency oxc-parser to ^0.142.0 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3422">#3422</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/abc30fbafc2e3967d499cef970c68b3edfefd850"><code>abc30fb</code></a>
build(deps): bump postcss from 8.5.17 to 8.5.23 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3421">#3421</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/55421392cbaa24d4df69419e4c6d4958fbcb6a12"><code>5542139</code></a>
build(deps): bump fast-xml-parser from 5.9.3 to 5.10.1 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3418">#3418</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/dc4ee8c89cc27ce30e239482199b3b3d786bf8b6"><code>dc4ee8c</code></a>
build(deps): bump fast-uri from 3.1.3 to 3.1.4 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3419">#3419</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/050d985196174b4be830cdb813d09e2705258455"><code>050d985</code></a>
feat(async-runtime): drain-linger surface + lock-free scheduler
internals (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3">#3</a>...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/e0b87086eefe0e7efeea6d269e9403c4be4ba9aa"><code>e0b8708</code></a>
chore(deps): update dependency oxc-parser to ^0.141.0 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3417">#3417</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/fc8494010697d078a93a528c3180271f6f187504"><code>fc84940</code></a>
chore(deps): update actions/setup-node action to v7 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3413">#3413</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ee598db45985ef11e18c7340801c28bb2452b688"><code>ee598db</code></a>
build(deps): bump protobufjs from 7.6.4 to 7.6.5 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3410">#3410</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-v3.11.0...napi-v3.12.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi-derive` from 3.6.1 to 3.6.3
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi-derive's
releases</a>.</em></p>
<blockquote>
<h2>napi-derive-v3.6.3</h2>
<h3>Other</h3>
<ul>
<li>updated the following local packages: napi-derive-backend</li>
</ul>
<h2>napi-derive-v3.6.2</h2>
<h3>Other</h3>
<ul>
<li>updated the following local packages: napi-derive-backend</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/956e4525fea6a676ea3680b711382f167b899af9"><code>956e452</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3448">#3448</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/73048f5a7fdbd42cdc2f46f2d5ac60ef27417bfa"><code>73048f5</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61fae8a1440ad8b7249f3cd7838fc2bafe00a906"><code>61fae8a</code></a>
fix(napi): stop unloading addons with live native code, preserve
non-Error re...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/93e86ce167095e84f2be2ae1c66a6c0bb96fec49"><code>93e86ce</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/2c905991899b9f12a0df4072c4bff6d62ef70d26"><code>2c90599</code></a>
fix(cli): support npm 12 pack output (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3449">#3449</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/360b1ec99ab0d001147d29e11c416c8338d3d1c9"><code>360b1ec</code></a>
fix(wasi): avoid randomness during module registration (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3447">#3447</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/b648c4090e7518ced18c9eca6059d27af3ab511b"><code>b648c40</code></a>
build(deps): bump nanoid from 3.3.16 to 3.3.18 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3446">#3446</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ffda4efff4bc4ebb6bef1f629dd0a6f09dc8f210"><code>ffda4ef</code></a>
chore(deps): update dependency js-yaml to v4.3.1 [security] (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3445">#3445</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/387b0dc7986018e44a4a0b466b030dc414170411"><code>387b0dc</code></a>
feat(cli): size WASI browser worker pools from
navigator.hardwareConcurrency ...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61e4346ce3d9a9c13e5c5dd6fb3b7d5e1b1d6e0d"><code>61e4346</code></a>
build(deps): bump fast-uri from 3.1.4 to 3.1.5 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3440">#3440</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-derive-v3.6.1...napi-derive-v3.6.3">compare
view</a></li>
</ul>
</details>
<br />

Updates `napi-build` from 2.4.0 to 2.4.1
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/napi-rs/napi-rs/releases">napi-build's
releases</a>.</em></p>
<blockquote>
<h2>napi-build-v2.4.1</h2>
<h3>Fixed</h3>
<ul>
<li><em>(napi)</em> stop unloading addons with live native code,
preserve non-Error rejections, and add the wasm teardown barrier (<a
href="https://redirect.github.com/napi-rs/napi-rs/pull/3423">#3423</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/956e4525fea6a676ea3680b711382f167b899af9"><code>956e452</code></a>
chore: release (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3448">#3448</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/73048f5a7fdbd42cdc2f46f2d5ac60ef27417bfa"><code>73048f5</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61fae8a1440ad8b7249f3cd7838fc2bafe00a906"><code>61fae8a</code></a>
fix(napi): stop unloading addons with live native code, preserve
non-Error re...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/93e86ce167095e84f2be2ae1c66a6c0bb96fec49"><code>93e86ce</code></a>
chore(release): publish</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/2c905991899b9f12a0df4072c4bff6d62ef70d26"><code>2c90599</code></a>
fix(cli): support npm 12 pack output (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3449">#3449</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/360b1ec99ab0d001147d29e11c416c8338d3d1c9"><code>360b1ec</code></a>
fix(wasi): avoid randomness during module registration (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3447">#3447</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/b648c4090e7518ced18c9eca6059d27af3ab511b"><code>b648c40</code></a>
build(deps): bump nanoid from 3.3.16 to 3.3.18 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3446">#3446</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/ffda4efff4bc4ebb6bef1f629dd0a6f09dc8f210"><code>ffda4ef</code></a>
chore(deps): update dependency js-yaml to v4.3.1 [security] (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3445">#3445</a>)</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/387b0dc7986018e44a4a0b466b030dc414170411"><code>387b0dc</code></a>
feat(cli): size WASI browser worker pools from
navigator.hardwareConcurrency ...</li>
<li><a
href="https://github.com/napi-rs/napi-rs/commit/61e4346ce3d9a9c13e5c5dd6fb3b7d5e1b1d6e0d"><code>61e4346</code></a>
build(deps): bump fast-uri from 3.1.4 to 3.1.5 (<a
href="https://redirect.github.com/napi-rs/napi-rs/issues/3440">#3440</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/napi-rs/napi-rs/compare/napi-build-v2.4.0...napi-build-v2.4.1">compare
view</a></li>
</ul>
</details>
<br />

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-31 10:40:11 -07:00
Yang Cen c8fd3e97d1 test(python): cover stable main udf registration identity (#4094)
## Other changes

### What changed?

- Add a subprocess regression harness for an ordinary `@udf` function
defined in `__main__`.
- Verify the full registration request, artifact digest, and Function
signature stay identical across independent Python processes and
renamed/moved script paths.
- Verify body, referenced-global, and annotation changes still produce
distinct artifact identities, with annotation changes also producing a
distinct Function signature.

### Why is the change needed?


[ENT-2441](https://linear.app/lancedb/issue/ENT-2441/make-sure-function-defined-in-main-gets-stable-signature)
tracks the stability guarantee. Investigation on the exact `840e1d73`
main base found that LanceDB already packages canonical source instead
of cloudpickle bytes, so the unchanged `__main__` function is stable and
no production-code fix is needed. This change closes the missing
regression-test coverage.


[GEN-950](https://linear.app/lancedb/issue/GEN-950/class-based-udfs-defined-in-main-get-a-new-auto-version-on-every-run)
remains a separate Geneva checkpoint-version issue for class-based
callables. LanceDB's Function API continues to accept synchronous Python
functions only.

## Validation

- `cd python && uv run --extra tests pytest
python/tests/test_first_class_function_slice2.py -q` (`40 passed`)
- `uv run --project python --extra dev ruff format .`
- `uv run --project python --extra dev ruff check .` (`All checks
passed!`)
2026-08-31 22:34:34 +08:00
Xuanwo c196d033e9 feat: add drop_function client APIs (#4097) 2026-08-31 22:17:11 +08:00
Yang Cen 16753b805a revert: restore the lance v12.0.0-beta.5 pin on main (#4095)
Reverts #4093 (`c4ee8ae`), restoring main's lance dependency to
v12.0.0-beta.5 and the v12 integration surface it carried — the
`read_dir_page` paginated-listing pushdown from #3979, the v12
object-store wrapper APIs, and the shard-manifest call sites.

Pinning lance v11.0.0 stable belonged on a dedicated release branch for
cutting v0.38.0, not on main: main was already on the v12 beta train, so
#4093 was a downgrade of the development line. The released **v0.38.0
stands as published** — this only moves main forward again.

Verified on this branch: `cargo check --features remote --tests
--examples` clean, all 48 `database::listing` tests pass (the restored
store-pushdown pagination versions), `cargo fmt --check` and `cargo
clippy --features remote --tests --examples` clean. The root
`Cargo.lock` is restored by the revert and resolves as-is.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01BX7w9fUMQbPAXuxe9YgQKc

---
_Generated by [Claude
Code](https://claude.ai/code/session_01BX7w9fUMQbPAXuxe9YgQKc)_

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-31 20:40:46 +08:00
Lance Release 840e1d7313 Bump version: 0.38.0-beta.16 → 0.38.0 2026-08-31 07:38:51 +00:00
Lance Release 1a9414c47c Bump version: 0.38.0-beta.15 → 0.38.0-beta.16 2026-08-31 07:38:25 +00:00
LanceDB Robot c4ee8ae670 feat: update lance dependency to v11.0.0 (#4093)
Updates the Rust workspace and Java lance-core dependency to Lance v11.0.0.

Includes compatibility adjustments for the Lance 11 object-store, table-listing, and shard-manifest APIs.
2026-08-31 15:35:09 +08:00
Lance Release 57b8d3bf05 Bump version: 0.38.0-beta.14 → 0.38.0-beta.15 2026-08-31 07:33:34 +00:00
Jack Ye c6dfe830d9 feat(python): support large_utf8 function signatures (#4092)
Teach the Python Function signature emitter to serialize PyArrow large
strings as the canonical Arrow type name `large_utf8`.

Extend the shared Function Arrow type fixture and explicit-schema
coverage for scalar, nested, list, and large-list compositions.
2026-08-31 00:32:04 -07:00
Jack Ye d5dac65a21 feat: support Blob v2 UDF signatures (#4091)
Make Function authoring and declaration planning treat Blob v2 as a
scalar semantic type while preserving exact Blob metadata in binding
schemas.

Covers scalar Blob outputs, expanded named-struct outputs, and
whole-result structs with Blob children.
2026-08-30 23:33:30 -07:00
Lance Release 1b0fc2c465 Bump version: 0.38.0-beta.13 → 0.38.0-beta.14 2026-08-30 15:16:33 +00:00
Xuanwo a417e46bfa feat(functions): support GPU resource requirements (#4085)
Functions can describe their Python environment today, but cannot
declare accelerator requirements. That prevents Sophon from scheduling
computed-column UDF refreshes onto GPU workers from the immutable
Function definition.

Add `num_gpus` to Python `@udf` through a typed
`FunctionResourceRequirements` value and represent resource-aware
definitions with the `python_v2` runtime discriminator. CPU Functions
retain their existing `python` encoding and canonical identity.

The new discriminator is intentional for mixed-version safety:
deployments that do not understand execution resources reject the
runtime instead of accepting a new field and silently running the
Function on CPU. Required resources are part of Function version
identity; priority, concurrency, and retry policy remain Job concerns.
The actual resource scheduling remains owned by Sophon.
2026-08-30 08:10:08 -07:00
Jack Ye fcdc3f949e fix: allow multiple function bindings per table (#4090)
Allow a remote Function declaration when the table already contains
valid, supported Function binding metadata. Existing bindings remain
fully validated, including fail-closed handling for newer or
inconsistent contracts, while other schema mutations retain their
existing no-binding guard. Add planner and remote request-path
regression coverage for a second binding and reject dependent Function
inputs, including nested paths.
2026-08-30 01:16:57 -07:00
Lance Release 0c4e0667bc Bump version: 0.38.0-beta.12 → 0.38.0-beta.13 2026-08-30 06:09:21 +00:00
Jack Ye 101f524e47 feat(python): support nested Function Arrow types (#4088)
Teach Python Function authoring to retain the compact V1 grammar for
existing types and emit canonical exact JSON for nested struct
signatures. Adds coverage for recursive struct/list schemas and exact
field properties.
2026-08-29 23:07:59 -07:00
LanceDB Robot 36c142fa2e chore: update lance dependency to v12.0.0-beta.5 (#4089)
Updates the Rust workspace and Java lance-core dependency to Lance
v12.0.0-beta.5. Includes minimal Rust 1.97 Clippy compatibility fixes
required by validation. Lance tag:
https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.5

---------

Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
2026-08-29 14:51:41 -07:00
Will Jones a87cada90e feat(node)!: require Node >= 22 and drop npm lockfiles (#4074)
The bindings are built, installed and published with pnpm everywhere,
but a parallel npm dependency graph was still being maintained beside
it. This removes it, raises the supported Node floor to the versions we
actually test, and gives Dependabot the npm coverage it was missing.

## Dropping npm

`nodejs/package-lock.json` was regenerated by `ci/update_lockfiles.sh`
on every release commit and read by nothing — no workflow runs `npm ci`
or `npm install` in `nodejs/`, and npm never publishes a lockfile in a
package tarball. It could not even agree with the real install, since
npm does not see pnpm's `overrides`. Because GitHub's dependency graph
parses `package-lock.json`, it was also reporting vulnerabilities for a
tree we neither install nor ship.

`docs/package.json`, `docs/package-lock.json` and `docs/tsconfig.json`
go too. They depend on `file:../node` and
`file:../node/node_modules/apache-arrow` — the `node/` directory was
removed long ago — the tsconfig compiles `src/*.ts` where no TypeScript
files exist, and nothing installs any of it. `docs.yml` only referenced
the lockfile to configure an npm cache for an install it never ran.

Two `workflow_dispatch` workflows for regenerating those lockfiles are
removed as well. Both were already broken: they `uses:` composite
actions at `.github/workflows/update_package_lock{,_nodejs}` that do not
exist, so dispatching either failed immediately.

The remaining `npx` calls become direct `node_modules/.bin/...`
invocations. These were already running locally installed binaries
rather than resolving anything, but naming the binary removes the npm
CLI from the loop and does not depend on which Node version is active.
`dev.yml`'s commitlint check was the last place doing real npm
dependency resolution — an unpinned `npm install
@commitlint/config-conventional` that also bypassed the
`minimumReleaseAge` hold configured for `nodejs/` — and is now a pinned
`pnpm dlx`.

## Node support

Node 18 and 20 both reached end-of-life, in April 2025 and April 2026.
The matrix moves to 22, 24 and 26, and `engines` rises from `>= 18` to
`>= 22` so the declared floor is one the matrix actually covers. Node 22
is LTS until April 2027; 24 is LTS; 26 is Current and becomes LTS in
October 2026.

This also removes the reason the workflows reached for `npx` in the
first place: pnpm 11 requires Node >= 22.13, which every matrix version
now satisfies.

The prebuilt-binary smoke test in `npm-publish.yml` moves from Node 20
to Node 22 — the floor, where a napi ABI problem would surface first —
rather than fanning out across all three, to keep the publish matrix
from tripling.

## Dependabot

There were no npm-ecosystem entries at all, which is why the advisories
behind #4073 went unnoticed. Both pnpm lockfiles are now watched —
`nodejs/` and `nodejs/examples/`, which is a separate install — using
the same `lockfile-only` strategy as the existing cargo and pip entries,
so version ranges in `package.json` are left alone.

## Pre-commit biome

The hook ran `npx @biomejs/biome@1.8.3` while `nodejs/package.json`
resolved 1.9.4. The two disagree about formatting, so the hook rejected
code that `pnpm lint` accepts, and failed on unmodified `main` for
anyone touching `nodejs/`. It now uses the pnpm-managed biome, which
fixes the drift with no source changes.

## Testing

`dev.yml`'s commitlint job does not check out the repo, so it runs in an
empty workspace, and I could not verify `pnpm/action-setup` there
locally. It triggers on `pull_request_target`, so this PR exercises it
directly — worth confirming green before merge. I did verify the `pnpm
dlx` invocation itself locally: it accepts a conventional title and
rejects a non-conventional one with exit 1.

Node 26 is new enough that the examples job may surface gaps in prebuilt
native binaries (`onnxruntime-node`, `sharp`) before their maintainers
publish for it.

## Not included

`nodejs/examples/` still pins `sharp: "0.33.5"` and has its own audit
findings. Raising the Node floor unblocks that work — sharp 0.35
requires Node >= 20.9, which the matrix now satisfies — but it is a
dependency bump rather than tooling cleanup, so it is left separate.

## Breaking changes

`@lancedb/lancedb` now requires Node >= 22; previously >= 18. The
`@types/node` peer range moves from `>=18` to `>=22` to match. Users on
Node 18 or 20 must upgrade their runtime; both have been end-of-life for
some time. Existing installs are unaffected, since `engines` is only
checked on install.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 09:47:47 -07:00
54 changed files with 2331 additions and 11595 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
[tool.bumpversion]
current_version = "0.38.0-beta.12"
current_version = "0.38.0"
parse = """(?x)
(?P<major>0|[1-9]\\d*)\\.
(?P<minor>0|[1-9]\\d*)\\.
+24
View File
@@ -44,3 +44,27 @@ updates:
python-deps:
patterns:
- "*"
# The npm ecosystem covers pnpm lockfiles. There are two separate installs:
# the bindings themselves and the examples, which have their own lockfile.
# As with cargo and pip above, only bump the lockfile — the version ranges
# in package.json are our consumers' constraints, not ours.
- package-ecosystem: npm
directory: /nodejs
schedule:
interval: weekly
versioning-strategy: lockfile-only
groups:
nodejs-deps:
patterns:
- "*"
- package-ecosystem: npm
directory: /nodejs/examples
schedule:
interval: weekly
versioning-strategy: lockfile-only
groups:
nodejs-examples-deps:
patterns:
- "*"
+10 -4
View File
@@ -29,12 +29,14 @@ jobs:
steps:
- uses: actions/setup-node@v6
with:
node-version: "18"
node-version: "24"
- uses: pnpm/action-setup@v6
with:
version: 11.1.1
# These rules are disabled because Github will always ensure there
# is a blank line between the title and the body and Github will
# word wrap the description field to ensure a reasonable max line
# length.
- run: npm install @commitlint/config-conventional
- run: >
echo 'module.exports = {
"rules": {
@@ -43,7 +45,11 @@ jobs:
"body-leading-blank": [0, "always"]
}
}' > .commitlintrc.js
- run: npx commitlint --extends @commitlint/config-conventional --verbose <<< $COMMIT_MSG
- run: >
pnpm dlx
--package @commitlint/cli@21.2.2
--package @commitlint/config-conventional@21.2.2
commitlint --extends @commitlint/config-conventional --verbose <<< $COMMIT_MSG
env:
COMMIT_MSG: >
${{ github.event.pull_request.title }}
@@ -54,7 +60,7 @@ jobs:
with:
script: |
const message = `**ACTION NEEDED**
Lance follows the [Conventional Commits specification](https://www.conventionalcommits.org/en/v1.0.0/) for release automation.
The PR title and description are used as the merge commit message.\
+1 -1
View File
@@ -56,7 +56,7 @@ jobs:
uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2.9.0
with:
# Restricted to http(s) on purpose. Much of docs/src is generated
# API reference (the js/ tree comes from `npm run docs` in nodejs)
# API reference (the js/ tree comes from `pnpm run docs` in nodejs)
# and the hand-written pages use mkdocstrings cross-references and
# nav-relative paths that only resolve in the site mkdocs builds,
# not in this checkout, so relative links would be reported as
+1 -3
View File
@@ -55,9 +55,7 @@ jobs:
- name: Set up node
uses: actions/setup-node@v6
with:
node-version: 20
cache: 'npm'
cache-dependency-path: docs/package-lock.json
node-version: 24
- name: Install node dependencies
working-directory: nodejs
run: |
+11 -13
View File
@@ -47,9 +47,8 @@ jobs:
version: 11.1.1
- uses: actions/setup-node@v6
with:
# pnpm 11 requires Node >= 22.13; use 24 since 22 hits EOL
# in October. The library itself still supports Node >= 18
# (see test matrix below).
# Build on a supported LTS; the matrix job below covers every
# Node version the library claims to support.
node-version: 24
cache: 'pnpm'
cache-dependency-path: nodejs/pnpm-lock.yaml
@@ -84,7 +83,7 @@ jobs:
timeout-minutes: 30
strategy:
matrix:
node-version: [ "18", "20" ]
node-version: [ "22", "24", "26" ]
runs-on: "ubuntu-22.04"
defaults:
run:
@@ -101,9 +100,9 @@ jobs:
- uses: actions/setup-node@v6
name: Setup Node.js 24 for build
with:
# pnpm 11 requires Node >= 22.13; use 24 since 22 hits EOL
# in October. Build/install runs on Node 24; tests run on the
# matrix version below using direct jest invocation.
# Build and install once on a fixed version so the generated docs
# are identical across matrix legs; the tests below then run on each
# supported Node version.
node-version: 24
cache: 'pnpm'
cache-dependency-path: nodejs/pnpm-lock.yaml
@@ -152,9 +151,9 @@ jobs:
S3_TEST: "1"
# Newer @smithy/core uses dynamic ESM imports.
NODE_OPTIONS: "--experimental-vm-modules"
# Invoke jest directly because pnpm 11 itself requires Node 22+
# while the matrix tests on older Node versions.
run: npx jest --verbose
# Invoke the installed jest binary directly; the pnpm shim is set up
# against the build-phase Node, not the version selected above.
run: node_modules/.bin/jest --verbose
- name: Test examples
working-directory: ./
env:
@@ -164,7 +163,7 @@ jobs:
run: |
python ci/mock_openai.py &
cd nodejs/examples
npx jest --testEnvironment jest-environment-node-single-context --verbose
node_modules/.bin/jest --testEnvironment jest-environment-node-single-context --verbose
macos:
timeout-minutes: 30
# macos-15 ships a newer linker; the older macos-14 linker fails to insert
@@ -185,8 +184,7 @@ jobs:
version: 11.1.1
- uses: actions/setup-node@v6
with:
# pnpm 11 requires Node >= 22.13; use 24 since 22 hits EOL
# in October.
# pnpm 11 requires Node >= 22.13.
node-version: 24
cache: 'pnpm'
cache-dependency-path: nodejs/pnpm-lock.yaml
+7 -8
View File
@@ -168,8 +168,7 @@ jobs:
- name: Setup node
uses: actions/setup-node@v6
with:
# pnpm 11 requires Node >= 22.13; use 24 since 22 hits EOL
# in October.
# pnpm 11 requires Node >= 22.13.
node-version: 24
cache: pnpm
cache-dependency-path: nodejs/pnpm-lock.yaml
@@ -251,7 +250,7 @@ jobs:
run: |
set -e
${{ matrix.settings.pre_build }}
npx napi build --platform --release \
node_modules/.bin/napi build --platform --release \
--features ${{ matrix.settings.features }} \
--target ${{ matrix.settings.target }} \
--dts ../lancedb/native.d.ts \
@@ -271,7 +270,7 @@ jobs:
- name: Build
run: |
${{ matrix.settings.pre_build }}
npx napi build --platform --release \
node_modules/.bin/napi build --platform --release \
--features ${{ matrix.settings.features }} \
--target ${{ matrix.settings.target }} \
--dts ../lancedb/native.d.ts \
@@ -339,7 +338,7 @@ jobs:
- target: aarch64-unknown-linux-gnu
host: ubuntu-2404-8x-arm64
node:
- '20'
- '22'
runs-on: ${{ matrix.settings.host }}
defaults:
run:
@@ -385,9 +384,9 @@ jobs:
- name: Move built files
run: cp dist/native.d.ts dist/native.js dist/*.node lancedb/
- name: Test bindings
# Invoke jest directly because pnpm 11 itself requires Node 22+
# while the matrix tests on older Node versions.
run: npx jest --verbose
# Invoke the installed jest binary directly; the pnpm shim is set up
# against the install-phase Node, not the version selected above.
run: node_modules/.bin/jest --verbose
publish:
name: Publish
runs-on: ubuntu-latest
@@ -1,22 +0,0 @@
name: Update package-lock.json
on:
workflow_dispatch:
permissions:
contents: read
jobs:
publish:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ref: main
persist-credentials: false
fetch-depth: 0
lfs: true
- uses: ./.github/workflows/update_package_lock
with:
github_token: ${{ secrets.LANCEDB_RELEASE_TOKEN }}
@@ -1,22 +0,0 @@
name: Update NodeJs package-lock.json
on:
workflow_dispatch:
permissions:
contents: read
jobs:
publish:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ref: main
persist-credentials: false
fetch-depth: 0
lfs: true
- uses: ./.github/workflows/update_package_lock_nodejs
with:
github_token: ${{ secrets.LANCEDB_RELEASE_TOKEN }}
+4 -1
View File
@@ -20,7 +20,10 @@ repos:
hooks:
- id: local-biome-check
name: biome check
entry: npx @biomejs/biome@1.8.3 check --config-path nodejs/biome.json nodejs/
# Use the biome from nodejs/package.json rather than a separately
# pinned one: the two drifted apart and disagreed on formatting, so
# this hook rejected code that `pnpm lint` accepted.
entry: nodejs/node_modules/.bin/biome check --config-path nodejs/biome.json nodejs/
language: system
types: [text]
files: "nodejs/.*"
+3 -3
View File
@@ -38,7 +38,7 @@ Before committing changes, run formatting for every language you touched. At min
* Rust changes: run `cargo fmt --all`.
* Python changes: run `ruff format .` and `ruff check .` from the repository root,
and run targeted tests through `cd python && uv run ...`.
* TypeScript changes: run the relevant `npm`/`pnpm` lint, format, build, and docs commands in `nodejs`.
* TypeScript changes: run the relevant `pnpm` lint, format, build, and docs commands in `nodejs`.
Before creating a PR, the exact value passed to `gh pr create --title` must follow
Conventional Commits, such as `fix: support nested field paths in native index creation`
@@ -101,12 +101,12 @@ Python bindings changes:
TypeScript bindings changes:
1. Add napi-rs method binding on `Table` in `nodejs/src/table.rs`.
2. Run `npm run build` to generate TypeScript definitions.
2. Run `pnpm build` to generate TypeScript definitions.
3. Add typescript method on abstract class `Table` in `nodejs/src/table.ts`.
4. Add concrete method on `LocalTable` class in `nodejs/src/native_table.ts`.
* Note: despite the name, this class is also used for remote tables.
5. Add test in `nodejs/__test__/table.test.ts`.
6. Run `npm run docs` to generate TypeScript documentation.
6. Run `pnpm run docs` to generate TypeScript documentation.
## Python API reference
Generated
+71 -69
View File
@@ -535,9 +535,9 @@ dependencies = [
[[package]]
name = "async-trait"
version = "0.1.91"
version = "0.1.92"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae36dc4177970ef04fde5178d3e2429882def40e57a451f919c098f72baa6cec"
checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667"
dependencies = [
"proc-macro2",
"quote",
@@ -1443,9 +1443,9 @@ checksum = "175812e0be2bccb6abe50bb8d566126198344f707e304f45c648fd8f2cc0365e"
[[package]]
name = "bytemuck"
version = "1.25.0"
version = "1.25.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec"
checksum = "95832e849adfb21180ccb6826a99da14e5d266ae5c2e668e1602cf234f153797"
dependencies = [
"bytemuck_derive",
]
@@ -3455,8 +3455,8 @@ checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "fsst"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-array",
"rand 0.9.5",
@@ -4815,8 +4815,8 @@ checksum = "e037a2e1d8d5fdbd49b16a4ea09d5d6401c1f29eca5ff29d03d3824dba16256a"
[[package]]
name = "lance"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arc-swap",
"arrow",
@@ -4888,8 +4888,8 @@ dependencies = [
[[package]]
name = "lance-arrow"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -4911,7 +4911,7 @@ dependencies = [
[[package]]
name = "lance-arrow-scalar"
version = "58.0.0"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -4925,7 +4925,7 @@ dependencies = [
[[package]]
name = "lance-arrow-stats"
version = "58.0.0"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-array",
"arrow-schema",
@@ -4934,8 +4934,8 @@ dependencies = [
[[package]]
name = "lance-bitpacking"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrayref",
"crunchy",
@@ -4945,8 +4945,8 @@ dependencies = [
[[package]]
name = "lance-core"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -4983,8 +4983,8 @@ dependencies = [
[[package]]
name = "lance-datafusion"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow",
"arrow-array",
@@ -5013,8 +5013,8 @@ dependencies = [
[[package]]
name = "lance-datagen"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow",
"arrow-array",
@@ -5031,8 +5031,8 @@ dependencies = [
[[package]]
name = "lance-derive"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"proc-macro2",
"quote",
@@ -5041,8 +5041,8 @@ dependencies = [
[[package]]
name = "lance-encoding"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-arith",
"arrow-array",
@@ -5075,8 +5075,8 @@ dependencies = [
[[package]]
name = "lance-file"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-arith",
"arrow-array",
@@ -5107,8 +5107,8 @@ dependencies = [
[[package]]
name = "lance-index"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arc-swap",
"arrow",
@@ -5172,8 +5172,8 @@ dependencies = [
[[package]]
name = "lance-index-core"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-array",
"arrow-schema",
@@ -5195,8 +5195,8 @@ dependencies = [
[[package]]
name = "lance-io"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow",
"arrow-array",
@@ -5236,8 +5236,8 @@ dependencies = [
[[package]]
name = "lance-linalg"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-array",
"arrow-schema",
@@ -5251,8 +5251,8 @@ dependencies = [
[[package]]
name = "lance-namespace"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow",
"async-trait",
@@ -5264,8 +5264,8 @@ dependencies = [
[[package]]
name = "lance-namespace-impls"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow",
"arrow-ipc",
@@ -5318,8 +5318,8 @@ dependencies = [
[[package]]
name = "lance-select"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -5333,8 +5333,8 @@ dependencies = [
[[package]]
name = "lance-table"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow",
"arrow-array",
@@ -5374,8 +5374,8 @@ dependencies = [
[[package]]
name = "lance-testing"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"arrow-array",
"arrow-schema",
@@ -5388,8 +5388,8 @@ dependencies = [
[[package]]
name = "lance-tokenizer"
version = "12.0.0-beta.2"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.2#dafa4642658d996b3e31dde91e02f72db7860d7e"
version = "12.0.0-beta.5"
source = "git+https://github.com/lance-format/lance.git?tag=v12.0.0-beta.5#556637791d0048c2b4f1342dd84b67c8bbd65259"
dependencies = [
"frostem",
"icu_segmenter",
@@ -5402,7 +5402,7 @@ dependencies = [
[[package]]
name = "lancedb"
version = "0.38.0-beta.12"
version = "0.38.0"
dependencies = [
"ahash",
"anyhow",
@@ -5490,7 +5490,7 @@ dependencies = [
[[package]]
name = "lancedb-nodejs"
version = "0.38.0-beta.12"
version = "0.38.0"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -5515,7 +5515,7 @@ dependencies = [
[[package]]
name = "lancedb-python"
version = "0.38.0-beta.12"
version = "0.38.0"
dependencies = [
"arrow",
"async-trait",
@@ -5748,9 +5748,9 @@ dependencies = [
[[package]]
name = "log"
version = "0.4.33"
version = "0.4.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
[[package]]
name = "loom"
@@ -6001,9 +6001,9 @@ dependencies = [
[[package]]
name = "moka"
version = "0.12.15"
version = "0.12.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "957228ad12042ee839f93c8f257b62b4c0ab5eaae1d4fa60de53b27c9d7c5046"
checksum = "4293f18e7567a1caf3c584855554377025c65e0aa445344d04171f5ad63d19b9"
dependencies = [
"async-lock",
"crossbeam-channel",
@@ -6097,14 +6097,15 @@ dependencies = [
[[package]]
name = "napi"
version = "3.11.0"
version = "3.12.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "de33522036981030a75c231829566bc63414e08101a6f5ff4ac6cef19c8e0941"
checksum = "58c5f4d5375213fdb7be2655e152386e82f026f9a5ba36a75556e11359aafe09"
dependencies = [
"bitflags 2.11.1",
"chrono",
"ctor 1.0.12",
"futures",
"libc",
"napi-build",
"napi-sys",
"nohash-hasher",
@@ -6116,15 +6117,15 @@ dependencies = [
[[package]]
name = "napi-build"
version = "2.4.0"
version = "2.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5282704fbe8d49b0cf8b08e3f33233416a528658f205c7e5ace63b582de0b11c"
checksum = "60fdf9b392c50e7c4170fa633bd909490ed7835cea4c046776d1a4dd8d2ae0ab"
[[package]]
name = "napi-derive"
version = "3.6.1"
version = "3.6.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4d5c9c02556ea6dc99dffd36c1ce60141411657438501a125b675776d011ce92"
checksum = "0fa55ea69990c90b888e9e77044410e304ce7f35de599dc6d0b5c1923d2e59af"
dependencies = [
"convert_case",
"ctor 1.0.12",
@@ -6136,9 +6137,9 @@ dependencies = [
[[package]]
name = "napi-derive-backend"
version = "6.1.1"
version = "6.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d60b5d773ad46c698c8cc2cd9fde0b283d39cbb7f71c04bee633c7bdba4423bd"
checksum = "df4056ac7c18e4438ccf0edaed4340ca0d269278c8ec19284f7b23cb039fd0ae"
dependencies = [
"convert_case",
"proc-macro2",
@@ -8601,9 +8602,9 @@ dependencies = [
[[package]]
name = "roaring"
version = "0.11.4"
version = "0.11.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1dedc5658c6ecb3bdb5ef5f3295bb9253f42dcf3fd1402c03f6b1f7659c3c4a9"
checksum = "18bd8a37d17a58532776dcdf6041ce64929adca78e8489d5cacbafe99229d3e1"
dependencies = [
"bytemuck",
"byteorder",
@@ -9063,9 +9064,9 @@ dependencies = [
[[package]]
name = "serde_with"
version = "3.21.0"
version = "3.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76a5c54c7310e7b8b9577c286d7e399ddd876c3e12b3ed917a8aabc4b96e9e8c"
checksum = "ee78f1fbe43ac4a0e47aadb3dbd357b69eb0d3793e948624cd03dd2750ab1c0a"
dependencies = [
"base64 0.22.1",
"bs58",
@@ -9073,6 +9074,7 @@ dependencies = [
"hex",
"indexmap 1.9.3",
"indexmap 2.14.0",
"jiff",
"schemars 0.9.0",
"schemars 1.2.1",
"serde_core",
@@ -9083,9 +9085,9 @@ dependencies = [
[[package]]
name = "serde_with_macros"
version = "3.21.0"
version = "3.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "84d57bc0c8b9a17920c178daa6bb924850d54a9c97ab45194bb8c17ad66bb660"
checksum = "8705578779c2b6bd90d84d66eb2e206b708b1a4d7b9f17641b293545bf1c7e46"
dependencies = [
"darling 0.23.0",
"proc-macro2",
@@ -10452,9 +10454,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.24.0"
version = "1.26.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239"
checksum = "b5772d71c9be8a8a6ac2117d949c5b224c1b72241bb611d9a3012edcf8af7812"
dependencies = [
"getrandom 0.4.2",
"js-sys",
+14 -14
View File
@@ -13,20 +13,20 @@ categories = ["database-implementations"]
rust-version = "1.91.0"
[workspace.dependencies]
lance = { "version" = "=12.0.0-beta.2", default-features = false, "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-core = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-datagen = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-file = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-io = { "version" = "=12.0.0-beta.2", default-features = false, "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-index = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-linalg = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace-impls = { "version" = "=12.0.0-beta.2", default-features = false, "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-table = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-testing = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-datafusion = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-encoding = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance-arrow = { "version" = "=12.0.0-beta.2", "tag" = "v12.0.0-beta.2", "git" = "https://github.com/lance-format/lance.git" }
lance = { "version" = "=12.0.0-beta.5", default-features = false, "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-core = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-datagen = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-file = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-io = { "version" = "=12.0.0-beta.5", default-features = false, "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-index = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-linalg = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace-impls = { "version" = "=12.0.0-beta.5", default-features = false, "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-table = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-testing = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-datafusion = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-encoding = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lance-arrow = { "version" = "=12.0.0-beta.5", "tag" = "v12.0.0-beta.5", "git" = "https://github.com/lance-format/lance.git" }
lancedb = { path = "rust/lancedb", default-features = false }
ahash = "0.8"
# Note that this one does not include pyarrow
+1 -1
View File
@@ -5,5 +5,5 @@ licenses:
cd python && cargo about generate ../about.hbs -o RUST_THIRD_PARTY_LICENSES.html -c ../about.toml
cd python && uv sync --all-extras && uv tool run pip-licenses --python .venv/bin/python --format=markdown --with-urls --output-file=PYTHON_THIRD_PARTY_LICENSES.md
cd nodejs && cargo about generate ../about.hbs -o RUST_THIRD_PARTY_LICENSES.html -c ../about.toml
cd nodejs && npx license-checker --markdown --out NODEJS_THIRD_PARTY_LICENSES.md
cd nodejs && pnpm dlx license-checker@25 --markdown --out NODEJS_THIRD_PARTY_LICENSES.md
cd java && ./mvnw license:aggregate-add-third-party -q
+2 -6
View File
@@ -12,16 +12,12 @@ done
# This updates the lockfile without building
cargo metadata --quiet > /dev/null
pushd nodejs || exit 1
npm install --package-lock-only --silent
popd
if git diff --quiet --exit-code; then
echo "No lockfile changes to commit; skipping amend."
elif $AMEND; then
git add Cargo.lock nodejs/package-lock.json
git add Cargo.lock
git commit --amend --no-edit
else
git add Cargo.lock nodejs/package-lock.json
git add Cargo.lock
git commit -m "Update lockfiles"
fi
+11 -8
View File
@@ -47,22 +47,24 @@ pytest -vv python/tests/docs
### Checking typescript examples
The `@lancedb/lancedb` package must be built before running the tests:
The examples depend on `@lancedb/lancedb` at `file:../dist`, so the package must be
built before running the tests. This uses pnpm; see the
[Typescript contributing guide](../nodejs/CONTRIBUTING.md) for the toolchain setup.
```shell
pushd nodejs
npm ci
npm run build
pnpm install
pnpm build
popd
```
Then you can run the examples by going to the `nodejs/examples` directory and
running the tests like a normal npm package:
Then you can run the examples by going to the `nodejs/examples` directory, which is a
separate pnpm package with its own lockfile:
```shell
pushd nodejs/examples
npm ci
npm test
pnpm install
pnpm test
popd
```
@@ -84,6 +86,7 @@ The new files should be checked into the repository.
```shell
pushd nodejs
npm run docs
# `pnpm docs` would invoke pnpm's built-in `docs` command, not the script.
pnpm run docs
popd
```
-135
View File
@@ -1,135 +0,0 @@
{
"name": "lancedb-docs-test",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "lancedb-docs-test",
"version": "1.0.0",
"license": "Apache 2",
"dependencies": {
"apache-arrow": "file:../node/node_modules/apache-arrow",
"vectordb": "file:../node"
},
"devDependencies": {
"@types/node": "^20.11.8",
"typescript": "^5.3.3"
}
},
"../node": {
"name": "vectordb",
"version": "0.21.2-beta.0",
"cpu": [
"x64",
"arm64"
],
"license": "Apache-2.0",
"os": [
"darwin",
"linux",
"win32"
],
"dependencies": {
"@neon-rs/load": "^0.0.74",
"axios": "^1.4.0"
},
"devDependencies": {
"@neon-rs/cli": "^0.0.160",
"@types/chai": "^4.3.4",
"@types/chai-as-promised": "^7.1.5",
"@types/mocha": "^10.0.1",
"@types/node": "^18.16.2",
"@types/sinon": "^10.0.15",
"@types/temp": "^0.9.1",
"@types/uuid": "^9.0.3",
"@typescript-eslint/eslint-plugin": "^5.59.1",
"apache-arrow-old": "npm:apache-arrow@13.0.0",
"cargo-cp-artifact": "^0.1",
"chai": "^4.3.7",
"chai-as-promised": "^7.1.1",
"eslint": "^8.39.0",
"eslint-config-standard-with-typescript": "^34.0.1",
"eslint-plugin-import": "^2.26.0",
"eslint-plugin-n": "^15.7.0",
"eslint-plugin-promise": "^6.1.1",
"mocha": "^10.2.0",
"openai": "^4.24.1",
"sinon": "^15.1.0",
"temp": "^0.9.4",
"ts-node": "^10.9.1",
"ts-node-dev": "^2.0.0",
"typedoc": "^0.24.7",
"typedoc-plugin-markdown": "^3.15.3",
"typescript": "^5.1.0",
"uuid": "^9.0.0"
},
"optionalDependencies": {
"@lancedb/vectordb-darwin-arm64": "0.21.2-beta.0",
"@lancedb/vectordb-darwin-x64": "0.21.2-beta.0",
"@lancedb/vectordb-linux-arm64-gnu": "0.21.2-beta.0",
"@lancedb/vectordb-linux-x64-gnu": "0.21.2-beta.0",
"@lancedb/vectordb-win32-x64-msvc": "0.21.2-beta.0"
},
"peerDependencies": {
"@apache-arrow/ts": "^14.0.2",
"apache-arrow": "^14.0.2"
}
},
"../node/node_modules/apache-arrow": {
"version": "14.0.2",
"license": "Apache-2.0",
"dependencies": {
"@types/command-line-args": "5.2.0",
"@types/command-line-usage": "5.0.2",
"@types/node": "20.3.0",
"@types/pad-left": "2.1.1",
"command-line-args": "5.2.1",
"command-line-usage": "7.0.1",
"flatbuffers": "23.5.26",
"json-bignum": "^0.0.3",
"pad-left": "^2.1.0",
"tslib": "^2.5.3"
},
"bin": {
"arrow2csv": "bin/arrow2csv.js"
}
},
"node_modules/@types/node": {
"version": "20.11.8",
"resolved": "https://registry.npmjs.org/@types/node/-/node-20.11.8.tgz",
"integrity": "sha512-i7omyekpPTNdv4Jb/Rgqg0RU8YqLcNsI12quKSDkRXNfx7Wxdm6HhK1awT3xTgEkgxPn3bvnSpiEAc7a7Lpyow==",
"dev": true,
"dependencies": {
"undici-types": "~5.26.4"
}
},
"node_modules/apache-arrow": {
"resolved": "../node/node_modules/apache-arrow",
"link": true
},
"node_modules/typescript": {
"version": "5.3.3",
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.3.3.tgz",
"integrity": "sha512-pXWcraxM0uxAS+tN0AG/BF2TyqmHO014Z070UsJ+pFvYuRSq8KH8DmWpnbXe0pEPDHXZV3FcAbJkijJ5oNEnWw==",
"dev": true,
"bin": {
"tsc": "bin/tsc",
"tsserver": "bin/tsserver"
},
"engines": {
"node": ">=14.17"
}
},
"node_modules/undici-types": {
"version": "5.26.5",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-5.26.5.tgz",
"integrity": "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==",
"dev": true
},
"node_modules/vectordb": {
"resolved": "../node",
"link": true
}
}
}
-20
View File
@@ -1,20 +0,0 @@
{
"name": "lancedb-docs-test",
"version": "1.0.0",
"description": "auto-generated tests from doc",
"author": "dev@lancedb.com",
"license": "Apache 2",
"dependencies": {
"apache-arrow": "file:../node/node_modules/apache-arrow",
"vectordb": "file:../node"
},
"scripts": {
"build": "tsc -b && cd ../node && npm run build-release",
"example": "npm run build && node",
"test": "npm run build && ls dist/*.js | xargs -n 1 node"
},
"devDependencies": {
"@types/node": "^20.11.8",
"typescript": "^5.3.3"
}
}
+1 -1
View File
@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
<dependency>
<groupId>com.lancedb</groupId>
<artifactId>lancedb-core</artifactId>
<version>0.38.0-beta.12</version>
<version>0.38.0</version>
</dependency>
```
-17
View File
@@ -1,17 +0,0 @@
{
"include": [
"src/*.ts",
],
"compilerOptions": {
"target": "es2022",
"module": "nodenext",
"declaration": true,
"outDir": "./dist",
"strict": true,
"allowJs": true,
"resolveJsonModule": true,
},
"exclude": [
"./dist/*",
]
}
+1 -1
View File
@@ -8,7 +8,7 @@
<parent>
<groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId>
<version>0.38.0-beta.12</version>
<version>0.38.0-final.0</version>
<relativePath>../pom.xml</relativePath>
</parent>
+2 -2
View File
@@ -6,7 +6,7 @@
<groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId>
<version>0.38.0-beta.12</version>
<version>0.38.0-final.0</version>
<packaging>pom</packaging>
<name>${project.artifactId}</name>
<description>LanceDB Java SDK Parent POM</description>
@@ -28,7 +28,7 @@
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<arrow.version>15.0.0</arrow.version>
<lance-core.version>12.0.0-beta.2</lance-core.version>
<lance-core.version>12.0.0-beta.5</lance-core.version>
<spotless.skip>false</spotless.skip>
<spotless.version>2.30.0</spotless.version>
<spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version>
+1 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "lancedb-nodejs"
edition.workspace = true
version = "0.38.0-beta.12"
version = "0.38.0"
publish = false
license.workspace = true
description.workspace = true
+2 -2
View File
@@ -5,8 +5,8 @@ import packageJson = require("../package.json");
describe("package metadata", () => {
it("requires Node.js type declarations compatible with the runtime", () => {
expect(packageJson.engines.node).toBe(">= 18");
expect(packageJson.peerDependencies["@types/node"]).toBe(">=18");
expect(packageJson.engines.node).toBe(">= 22");
expect(packageJson.peerDependencies["@types/node"]).toBe(">=22");
expect(packageJson.peerDependenciesMeta["@types/node"]).toEqual({
optional: true,
});
+9 -2
View File
@@ -3,6 +3,7 @@
import * as http from "http";
import { RequestListener } from "http";
import packageJson = require("../package.json");
import {
ClientConfig,
Connection,
@@ -70,7 +71,13 @@ async function withMockDatabase(
try {
await callback(db);
} finally {
server.close();
// `close()` alone leaves the port bound until keep-alive sockets drain, so
// a single failing test would cascade into EADDRINUSE for every test after
// it. Destroy the connections and wait for the port to actually be free.
await new Promise<void>((resolve) => {
server.closeAllConnections();
server.close(() => resolve());
});
}
}
@@ -131,7 +138,7 @@ describe("remote connection", () => {
(req, res) => {
expect(req.headers["x-api-key"]).toEqual("fake");
expect(req.headers["user-agent"]).toEqual(
`LanceDB-Node-Client/${process.env.npm_package_version}`,
`LanceDB-Node-Client/${packageJson.version}`,
);
const body = JSON.stringify({ tables: [] });
+2 -1
View File
@@ -8,7 +8,8 @@
"//1": "--experimental-vm-modules is needed to run jest with sentence-transformers",
"//2": "--testEnvironment is needed to run jest with sentence-transformers",
"//3": "See: https://github.com/huggingface/transformers.js/issues/57",
"test": "node --experimental-vm-modules node_modules/.bin/jest --testEnvironment jest-environment-node-single-context --verbose",
"//4": "jest is invoked by its JS entry, not node_modules/.bin/jest: under pnpm that path is a shell shim, which `node` cannot execute",
"test": "node --experimental-vm-modules node_modules/jest/bin/jest.js --testEnvironment jest-environment-node-single-context --verbose",
"lint": "biome check *.ts && biome format *.ts",
"lint-ci": "biome ci .",
"lint-fix": "biome check --write *.ts && pnpm format",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-darwin-arm64",
"version": "0.38.0-beta.12",
"version": "0.38.0",
"os": ["darwin"],
"cpu": ["arm64"],
"main": "lancedb.darwin-arm64.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-arm64-gnu",
"version": "0.38.0-beta.12",
"version": "0.38.0",
"os": ["linux"],
"cpu": ["arm64"],
"main": "lancedb.linux-arm64-gnu.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-arm64-musl",
"version": "0.38.0-beta.12",
"version": "0.38.0",
"os": ["linux"],
"cpu": ["arm64"],
"main": "lancedb.linux-arm64-musl.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-x64-gnu",
"version": "0.38.0-beta.12",
"version": "0.38.0",
"os": ["linux"],
"cpu": ["x64"],
"main": "lancedb.linux-x64-gnu.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-x64-musl",
"version": "0.38.0-beta.12",
"version": "0.38.0",
"os": ["linux"],
"cpu": ["x64"],
"main": "lancedb.linux-x64-musl.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-win32-arm64-msvc",
"version": "0.38.0-beta.12",
"version": "0.38.0",
"os": [
"win32"
],
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-win32-x64-msvc",
"version": "0.38.0-beta.12",
"version": "0.38.0",
"os": ["win32"],
"cpu": ["x64"],
"main": "lancedb.win32-x64-msvc.node",
-11106
View File
File diff suppressed because it is too large Load Diff
+3 -3
View File
@@ -11,7 +11,7 @@
"ann"
],
"private": false,
"version": "0.38.0-beta.12",
"version": "0.38.0",
"main": "dist/index.js",
"exports": {
".": "./dist/index.js",
@@ -67,7 +67,7 @@
"timeout": "3m"
},
"engines": {
"node": ">= 18"
"node": ">= 22"
},
"packageManager": "pnpm@11.1.1",
"cpu": ["x64", "arm64"],
@@ -101,7 +101,7 @@
"openai": "4.29.2"
},
"peerDependencies": {
"@types/node": ">=18",
"@types/node": ">=22",
"apache-arrow": ">=15.0.0 <=18.1.0"
},
"peerDependenciesMeta": {
+5 -1
View File
@@ -664,7 +664,11 @@ impl JsFullTextQuery {
}
fn parse_fts_query(query: Object) -> napi::Result<FullTextSearchQuery> {
if let Ok(Some(query)) = query.get::<&JsFullTextQuery>("query") {
// `&JsFullTextQuery` recovers a native class reference through napi's borrow-tracked
// path, which is only usable from generated `#[napi]` argument conversion. This is a
// manual lookup on a nested `Object` property instead, so use `ClassInstance`, which
// unwraps the class without requiring a borrow scope.
if let Ok(Some(query)) = query.get::<ClassInstance<JsFullTextQuery>>("query") {
Ok(FullTextSearchQuery::new_query(query.inner.clone()))
} else if let Ok(Some(query_text)) = query.get::<String>("query") {
let mut query_text = query_text;
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "lancedb-python"
version = "0.38.0-beta.12"
version = "0.38.0"
publish = false
edition.workspace = true
description = "Python bindings for LanceDB"
+1
View File
@@ -150,6 +150,7 @@ class Connection(object):
def job(self, job_id: str) -> Job: ...
async def create_function_async(self, request_json: str) -> Job: ...
async def get_function(self, name: str, version: str) -> str: ...
async def drop_function(self, name: str, version: str) -> bool: ...
async def list_jobs(self) -> List[JobInfo]: ...
async def get_job(self, job_id: str) -> Optional[JobDescription]: ...
async def cancel_job(self, job_id: str) -> bool: ...
+18
View File
@@ -712,6 +712,16 @@ class DBConnection(EnforceOverrides):
"Function catalog operations are not supported for this connection type"
)
def drop_function(self, name: str, *, version: str) -> bool:
"""Drop one exact immutable Function version from the remote catalog.
Returns True when the version changed to Dropped and False for an
idempotent replay. Local connections raise NotImplementedError.
"""
raise NotImplementedError(
"Function catalog operations are not supported for this connection type"
)
def job(self, job_id: str) -> Job:
"""A [Job][lancedb.job.Job] handle for a server-side job by id.
@@ -1413,6 +1423,10 @@ class LanceDBConnection(DBConnection):
def get_function(self, name: str, *, version: str) -> FunctionVersion:
return LOOP.run(self._conn.get_function(name, version=version))
@override
def drop_function(self, name: str, *, version: str) -> bool:
return LOOP.run(self._conn.drop_function(name, version=version))
@override
def list_jobs(self) -> List[JobInfo]:
"""List server-side jobs across the database's tables."""
@@ -2243,6 +2257,10 @@ class AsyncConnection(object):
"""Open one exact immutable Function version from the remote catalog."""
return FunctionVersion.from_json(await self._inner.get_function(name, version))
async def drop_function(self, name: str, *, version: str) -> bool:
"""Drop one exact immutable Function version from the remote catalog."""
return await self._inner.drop_function(name, version)
async def list_jobs(self) -> List[JobInfo]:
"""List server-side jobs across the database's tables."""
return await self._inner.list_jobs()
+327 -20
View File
@@ -49,11 +49,25 @@ from pydantic import (
model_validator,
)
from .schema import is_blob_v2_field as _is_blob_v2_field
_Int32 = conint(strict=True, ge=-(2**31), le=2**31 - 1)
_UInt32 = conint(strict=True, ge=0, le=2**32 - 1)
_UInt64 = conint(strict=True, ge=0, le=2**64 - 1)
def _validate_gpu_wire_marker(value: Any) -> bool:
if value is not True:
raise ValueError("runtime.gpu must be true")
return True
def _normalize_gpu_marker(value: bool) -> Optional[bool]:
if not isinstance(value, bool):
raise ValueError("gpu must be a boolean")
return True if value else None
class _FrozenDict(dict):
def _immutable(self, *args, **kwargs):
raise TypeError("remote canonical values are immutable")
@@ -239,6 +253,23 @@ class PythonRuntimeSpec(_RemoteValue):
python_version: Optional[str] = None
environment: Optional[PythonEnvironmentSpec] = None
env: Optional[Mapping[str, str]] = None
gpu: Optional[bool] = None
@model_validator(mode="before")
@classmethod
def _discard_unknown_runtime_payload(cls, value):
if isinstance(value, Mapping):
kind = value.get("kind")
if isinstance(kind, str) and kind not in {"python", "python_v2"}:
return {"kind": kind}
return value
@field_validator("gpu", mode="before")
@classmethod
def _validate_gpu_marker(cls, value):
if value is None:
return None
return _validate_gpu_wire_marker(value)
@model_validator(mode="after")
def _validate_runtime_kind(self):
@@ -247,18 +278,28 @@ class PythonRuntimeSpec(_RemoteValue):
raise ValueError("python runtime requires python_version")
if self.environment is None:
raise ValueError("python runtime requires environment")
if self.gpu is not None:
raise ValueError("python runtime with gpu requires kind='python_v2'")
elif self.kind == "python_v2":
if self.python_version is None:
raise ValueError("python_v2 runtime requires python_version")
if self.environment is None:
raise ValueError("python_v2 runtime requires environment")
if self.gpu is None:
raise ValueError("python_v2 runtime requires gpu")
else:
object.__setattr__(self, "python_version", None)
object.__setattr__(self, "environment", None)
object.__setattr__(self, "env", None)
object.__setattr__(self, "gpu", None)
return self
class FunctionVersion(_RemoteValue):
"""An exact immutable Function version returned by Enterprise.
Scheduling resources, priority, concurrency, and retry policy belong to
the submitting Job and are not part of this identity.
The GPU execution requirement is part of this identity. CPU and memory sizing,
priority, concurrency, and retry policy belong to the execution platform.
"""
name: str
@@ -479,6 +520,13 @@ class RefreshColumnResult(_RemoteValue):
_FUNCTION_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_.-]*$")
_FUNCTION_BLOB_V2_TYPE = "blob_v2"
_ARROW_EXTENSION_NAME_KEY = "ARROW:extension:name"
_BLOB_V2_EXTENSION_NAME = "lance.blob.v2"
_NESTED_BLOB_COLLECTION_ERROR = (
"unsupported Arrow type for Function signature: Blob v2 fields nested under "
"collection types are not supported"
)
_GRAMMAR_PRIMITIVES = (
@@ -495,6 +543,7 @@ _GRAMMAR_PRIMITIVES = (
(pa.float32(), "float32"),
(pa.float64(), "float64"),
(pa.string(), "utf8"),
(pa.large_string(), "large_utf8"),
(pa.binary(), "binary"),
(pa.date32(), "date32"),
(pa.date64(), "date64"),
@@ -502,31 +551,258 @@ _GRAMMAR_PRIMITIVES = (
def _canonical_arrow_type(data_type: pa.DataType) -> str:
"""The server's V1 Function type grammar. Anything outside it is rejected
here rather than at registration."""
"""The compact Function grammar, or canonical exact JSON for nested types."""
grammar = _grammar_arrow_type(data_type)
if grammar is not None:
return grammar
exact = _exact_arrow_type(data_type)
return json.dumps(exact, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
def _grammar_arrow_type(data_type: pa.DataType) -> Optional[str]:
for candidate, name in _GRAMMAR_PRIMITIVES:
if data_type == candidate:
return name
if pa.types.is_list(data_type) or pa.types.is_large_list(data_type):
item = _grammar_list_item(data_type)
if item is None:
return None
prefix = "list" if pa.types.is_list(data_type) else "large_list"
return f"{prefix}<{_canonical_list_item(data_type)}>"
return f"{prefix}<{item}>"
if pa.types.is_fixed_size_list(data_type) and data_type.list_size > 0:
return (
f"fixed_size_list<{_canonical_list_item(data_type)}, {data_type.list_size}>"
)
raise TypeError(f"unsupported Arrow type for Function signature: {data_type}")
item = _grammar_list_item(data_type)
if item is not None:
return f"fixed_size_list<{item}, {data_type.list_size}>"
return None
def _canonical_list_item(data_type: pa.DataType) -> str:
def _grammar_list_item(data_type: pa.DataType) -> Optional[str]:
"""The grammar names only the item type; it always means a non-nullable
child called `item`, so any other child metadata cannot be represented."""
child called `item`, so other child properties require exact JSON."""
child = data_type.value_field
if child.name != "item" or child.nullable or child.metadata:
return None
return _grammar_arrow_type(child.type)
def _validate_exact_arrow_field(field: pa.Field) -> None:
if not field.name:
raise TypeError(
"unsupported Arrow type for Function signature: list items must be a "
f"non-nullable field named 'item', got {child}"
"unsupported Arrow type for Function signature: field names "
"must not be empty"
)
return _canonical_arrow_type(child.type)
if _is_blob_v2_field(field):
if not _has_supported_blob_v2_layout(field):
raise TypeError(
"unsupported Arrow type for Function signature: lance.blob.v2 "
f"requires a supported Blob storage layout, got {field}"
)
metadata = {
(key.decode() if isinstance(key, bytes) else key): (
value.decode() if isinstance(value, bytes) else value
)
for key, value in (field.metadata or {}).items()
}
if metadata and metadata != {
_ARROW_EXTENSION_NAME_KEY: _BLOB_V2_EXTENSION_NAME
}:
raise TypeError(
"unsupported Arrow type for Function signature: lance.blob.v2 "
"field metadata must contain only its canonical extension marker"
)
elif field.metadata:
raise TypeError(
"unsupported Arrow type for Function signature: field metadata "
f"is not supported, got {field}"
)
def _has_supported_blob_v2_layout(field: pa.Field) -> bool:
data_type = field.type
if isinstance(data_type, pa.ExtensionType):
data_type = data_type.storage_type
if not pa.types.is_struct(data_type):
return False
fields = tuple(data_type)
def matches(spec, compare_nullable) -> bool:
return len(fields) == len(spec) and all(
actual.name == name
and actual.type == expected_type
and (not check_nullable or actual.nullable == nullable)
for actual, (name, expected_type, nullable), check_nullable in zip(
fields, spec, compare_nullable
)
)
logical_minimal = (
("data", pa.large_binary(), True),
("uri", pa.utf8(), True),
)
logical_full = logical_minimal + (
("position", pa.uint64(), True),
("size", pa.uint64(), True),
)
prepared = (
("kind", pa.uint8(), True),
("data", pa.large_binary(), True),
("uri", pa.utf8(), True),
("blob_id", pa.uint32(), True),
("blob_size", pa.uint64(), True),
("position", pa.uint64(), True),
)
descriptor = (
("kind", pa.uint8(), False),
("position", pa.uint64(), False),
("size", pa.uint64(), False),
("blob_id", pa.uint32(), False),
("blob_uri", pa.utf8(), False),
)
return (
matches(logical_minimal, (True, True))
or matches(logical_full, (True, True, False, False))
or matches(prepared, (True,) * len(prepared))
or matches(descriptor, (False,) * len(descriptor))
)
def _canonical_arrow_field(field: pa.Field) -> str:
_validate_exact_arrow_field(field)
if _is_blob_v2_field(field):
return _FUNCTION_BLOB_V2_TYPE
return _canonical_arrow_type(field.type)
def _blob_storage_type(field: pa.Field) -> pa.DataType:
data_type = field.type
if isinstance(data_type, pa.ExtensionType):
return data_type.storage_type
return data_type
def _exact_blob_storage_type(field: pa.Field) -> dict[str, Any]:
storage = _blob_storage_type(field)
if not pa.types.is_struct(storage):
raise TypeError(
"unsupported Arrow type for Function signature: lance.blob.v2 "
"requires struct storage"
)
return {
"type": "struct",
"fields": [
{
"name": child.name,
"nullable": child.nullable,
"type": (
{"type": "large_binary"}
if pa.types.is_large_binary(child.type)
else _exact_arrow_type(child.type)
),
}
for child in storage
],
}
def _data_type_has_blob_v2(data_type: pa.DataType) -> bool:
if pa.types.is_struct(data_type):
return any(
_is_blob_v2_field(field) or _data_type_has_blob_v2(field.type)
for field in data_type
)
if (
pa.types.is_list(data_type)
or pa.types.is_large_list(data_type)
or pa.types.is_fixed_size_list(data_type)
):
field = data_type.value_field
return _is_blob_v2_field(field) or _data_type_has_blob_v2(field.type)
if pa.types.is_map(data_type):
return any(
_is_blob_v2_field(field) or _data_type_has_blob_v2(field.type)
for field in (data_type.key_field, data_type.item_field)
)
return False
def _exact_arrow_field(
field: pa.Field, *, inside_collection: bool = False
) -> dict[str, Any]:
_validate_exact_arrow_field(field)
if _is_blob_v2_field(field):
if inside_collection:
raise TypeError(_NESTED_BLOB_COLLECTION_ERROR)
return {
"name": field.name,
"nullable": field.nullable,
"type": _exact_blob_storage_type(field),
"metadata": {
_ARROW_EXTENSION_NAME_KEY: _BLOB_V2_EXTENSION_NAME,
},
}
value = {
"name": field.name,
"nullable": field.nullable,
"type": _exact_arrow_type(field.type, inside_collection=inside_collection),
}
return value
def _exact_arrow_type(
data_type: pa.DataType, *, inside_collection: bool = False
) -> dict[str, Any]:
for candidate, name in _GRAMMAR_PRIMITIVES:
if data_type == candidate:
return {"type": name}
if pa.types.is_struct(data_type):
fields = list(data_type)
names = [field.name for field in fields]
if not fields or len(set(names)) != len(names):
raise TypeError(
"unsupported Arrow type for Function signature: structs must have "
"non-empty, uniquely named fields"
)
return {
"type": "struct",
"fields": [
_exact_arrow_field(field, inside_collection=inside_collection)
for field in fields
],
}
if (
pa.types.is_list(data_type)
or pa.types.is_large_list(data_type)
or pa.types.is_fixed_size_list(data_type)
):
if pa.types.is_fixed_size_list(data_type):
if data_type.value_field.name != "item":
raise TypeError(
"unsupported Arrow type for Function signature: fixed-size list "
"items must be named 'item'"
)
if data_type.list_size <= 0:
raise TypeError(
f"unsupported Arrow type for Function signature: {data_type}"
)
value: dict[str, Any] = {
"type": (
"list"
if pa.types.is_list(data_type)
else "large_list"
if pa.types.is_large_list(data_type)
else "fixed_size_list"
),
"fields": [
_exact_arrow_field(data_type.value_field, inside_collection=True)
],
}
if pa.types.is_fixed_size_list(data_type):
value["length"] = data_type.list_size
return value
if pa.types.is_map(data_type) and _data_type_has_blob_v2(data_type):
raise TypeError(_NESTED_BLOB_COLLECTION_ERROR)
raise TypeError(f"unsupported Arrow type for Function signature: {data_type}")
def _list_of(item: pa.DataType) -> pa.DataType:
@@ -600,8 +876,15 @@ def _callable_parameters(function: Callable[..., Any]) -> tuple[inspect.Paramete
def _function_output(output: pa.DataType | pa.Field | pa.Schema) -> FunctionOutput:
if isinstance(output, pa.Schema):
if output.metadata:
raise TypeError("Function output schema metadata is not supported")
fields = tuple(output)
elif isinstance(output, pa.Field) and pa.types.is_struct(output.type):
elif (
isinstance(output, pa.Field)
and not _is_blob_v2_field(output)
and pa.types.is_struct(output.type)
):
_validate_exact_arrow_field(output)
if output.nullable:
raise ValueError("Function output must be non-nullable")
fields = tuple(output.type)
@@ -617,11 +900,12 @@ def _function_output(output: pa.DataType | pa.Field | pa.Schema) -> FunctionOutp
raise TypeError(
"output_schema must be a PyArrow DataType, Field, or Schema"
)
_validate_exact_arrow_field(field)
if field.nullable:
raise ValueError("Function output must be non-nullable")
return FunctionOutput(
kind="scalar",
arrow_type=_canonical_arrow_type(field.type),
arrow_type=_canonical_arrow_field(field),
nullable=False,
)
@@ -629,6 +913,8 @@ def _function_output(output: pa.DataType | pa.Field | pa.Schema) -> FunctionOutp
raise ValueError("named-struct Function output must contain at least one field")
if any(field.nullable for field in fields):
raise ValueError("Function output fields must be non-nullable")
for field in fields:
_validate_exact_arrow_field(field)
names = [field.name for field in fields]
if len(set(names)) != len(names):
raise ValueError("Function output field names must be unique")
@@ -637,7 +923,7 @@ def _function_output(output: pa.DataType | pa.Field | pa.Schema) -> FunctionOutp
fields=tuple(
FunctionResultField(
name=field.name,
arrow_type=_canonical_arrow_type(field.type),
arrow_type=_canonical_arrow_field(field),
nullable=False,
)
for field in fields
@@ -657,6 +943,10 @@ def _infer_signature(
if input_schema is not None:
if not isinstance(input_schema, pa.Schema):
raise TypeError("input_schema must be a PyArrow Schema")
if input_schema.metadata:
raise TypeError("Function input schema metadata is not supported")
for field in input_schema:
_validate_exact_arrow_field(field)
expected = tuple(parameter.name for parameter in parameters)
actual = tuple(input_schema.names)
if actual != expected:
@@ -667,7 +957,7 @@ def _infer_signature(
inputs = tuple(
FunctionParameter(
name=field.name,
arrow_type=_canonical_arrow_type(field.type),
arrow_type=_canonical_arrow_field(field),
nullable=field.nullable,
)
for field in input_schema
@@ -690,7 +980,9 @@ def _infer_signature(
inputs.append(
FunctionParameter(
name=parameter.name,
arrow_type=_canonical_arrow_type(data_type),
arrow_type=_canonical_arrow_field(
pa.field(parameter.name, data_type, nullable=nullable)
),
nullable=nullable,
)
)
@@ -910,6 +1202,7 @@ class UdfDefinition:
pip: tuple[str, ...],
env: Mapping[str, str],
python_version: Optional[str],
gpu: bool = False,
conda: tuple[str, ...] = (),
conda_channels: tuple[str, ...] = (),
):
@@ -938,12 +1231,14 @@ class UdfDefinition:
signature = _infer_signature(function, input_schema, output_schema)
source = _package_source(function)
digest = f"sha256:{hashlib.sha256(source).hexdigest()}"
gpu_marker = _normalize_gpu_marker(gpu)
runtime = PythonRuntimeSpec(
kind="python",
kind="python_v2" if gpu_marker is not None else "python",
python_version=python_version
or f"{sys.version_info.major}.{sys.version_info.minor}",
environment=environment_spec,
env=environment,
gpu=gpu_marker,
)
self._function = function
self._request = FunctionRegistrationRequest(
@@ -989,6 +1284,7 @@ def udf(
pip: tuple[str, ...] | list[str] = (),
env: Optional[Mapping[str, str]] = None,
python_version: Optional[str] = None,
gpu: bool = False,
conda: tuple[str, ...] | list[str] = (),
conda_channels: tuple[str, ...] | list[str] = (),
) -> Callable[[Callable[..., Any]], UdfDefinition]: ...
@@ -1003,6 +1299,7 @@ def udf(
pip: tuple[str, ...] | list[str] = (),
env: Optional[Mapping[str, str]] = None,
python_version: Optional[str] = None,
gpu: bool = False,
conda: tuple[str, ...] | list[str] = (),
conda_channels: tuple[str, ...] | list[str] = (),
):
@@ -1035,6 +1332,10 @@ def udf(
Environment variables included in the Function definition.
python_version : str, optional
Remote Python major/minor version. Defaults to the client version.
gpu : bool, default False
Whether every remote execution requires a GPU. The execution platform
selects one compatible GPU for each worker. The requirement is part of
the immutable Function version.
The packaged artifact is a snapshot: the function source plus exactly
the module-level names it references (modules as imports, importable
@@ -1059,6 +1360,11 @@ def udf(
... return value * 2
>>> score(1.5)
3.0
>>> @udf(pip=["cupy-cuda12x"], gpu=True)
... def gpu_score(value: int) -> int:
... return value * 2
>>> gpu_score.registration_request.runtime.gpu
True
"""
def decorate(target: Callable[..., Any]) -> UdfDefinition:
@@ -1070,6 +1376,7 @@ def udf(
pip=tuple(pip),
env={} if env is None else env,
python_version=python_version,
gpu=gpu,
conda=tuple(conda),
conda_channels=tuple(conda_channels),
)
+4
View File
@@ -749,6 +749,10 @@ class RemoteDBConnection(DBConnection):
def get_function(self, name: str, *, version: str) -> FunctionVersion:
return LOOP.run(self._conn.get_function(name, version=version))
@override
def drop_function(self, name: str, *, version: str) -> bool:
return LOOP.run(self._conn.drop_function(name, version=version))
@override
def list_jobs(self) -> List["JobInfo"]:
"""List server-side jobs across the database's tables."""
@@ -12,6 +12,8 @@ from datetime import date
import http.server
import json
from pathlib import Path
import subprocess
import sys
import threading
from typing import Optional
@@ -19,7 +21,13 @@ import pyarrow as pa
import pytest
import lancedb
from lancedb.functions import UdfDefinition, udf
from lancedb.functions import (
PythonRuntimeSpec,
UdfDefinition,
_canonical_arrow_type,
_GRAMMAR_PRIMITIVES,
udf,
)
THRESHOLD = 20
_CACHE = None
@@ -61,6 +69,80 @@ def test_scalar_udf_matches_shared_registration_golden_and_remains_callable():
}
def _main_udf_source(
*, threshold: int = 20, input_annotation: str = "int", comparison: str = ">="
) -> str:
return (
"from __future__ import annotations\n"
"from lancedb.functions import udf\n"
f"THRESHOLD = {threshold}\n"
"\n"
"@udf\n"
f"def label(value: {input_annotation}) -> str:\n"
f" return 'big' if value {comparison} THRESHOLD else 'small'\n"
"\n"
"assert label.__module__ == '__main__'\n"
"print(label.registration_request.to_canonical_json())\n"
)
def _run_main_udf(path: Path, source: str) -> dict:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(source)
result = subprocess.run(
[sys.executable, str(path)],
check=True,
capture_output=True,
text=True,
)
return json.loads(result.stdout)
def test_main_udf_registration_identity_is_stable_across_processes_and_paths(
tmp_path,
):
source = _main_udf_source()
original_path = tmp_path / "original" / "job.py"
moved_path = tmp_path / "moved" / "renamed_job.py"
original_runs = [_run_main_udf(original_path, source) for _ in range(2)]
moved_run = _run_main_udf(moved_path, source)
assert len({run["artifact"]["digest"] for run in [*original_runs, moved_run]}) == 1
assert all(
run["signature"] == original_runs[0]["signature"]
for run in [original_runs[1], moved_run]
)
assert original_runs[0] == original_runs[1] == moved_run
body_change = _run_main_udf(
tmp_path / "changes" / "body.py", _main_udf_source(comparison=">")
)
global_change = _run_main_udf(
tmp_path / "changes" / "global.py", _main_udf_source(threshold=21)
)
annotation_change = _run_main_udf(
tmp_path / "changes" / "annotation.py",
_main_udf_source(input_annotation="float"),
)
baseline = original_runs[0]
assert baseline["signature"] == body_change["signature"]
assert baseline["signature"] == global_change["signature"]
assert baseline["signature"] != annotation_change["signature"]
assert (
len(
{
baseline["artifact"]["digest"],
body_change["artifact"]["digest"],
global_change["artifact"]["digest"],
annotation_change["artifact"]["digest"],
}
)
== 4
)
def _run_packaged(definition, *args):
"""Execute the shipped artifact in a fresh namespace, as a worker would."""
source = base64.b64decode(definition.registration_request.artifact.content.data)
@@ -89,6 +171,58 @@ def test_udf_conda_environment():
udf(name="channels", conda_channels=["conda-forge"])(lambda value: value)
def test_udf_gpu_marker_uses_gpu_runtime():
@udf(pip=["cupy-cuda12x"], gpu=True)
def double_on_gpu(value: int) -> int:
return value * 2
request = json.loads(double_on_gpu.registration_request.to_canonical_json())
assert request["runtime"]["kind"] == "python_v2"
assert request["runtime"]["gpu"] is True
@udf(pip=["pyarrow"])
def cpu_function(value: int) -> int:
return value
cpu_runtime = json.loads(cpu_function.registration_request.to_canonical_json())[
"runtime"
]
assert cpu_runtime["kind"] == "python"
assert "gpu" not in cpu_runtime
def identity(value: int) -> int:
return value
for invalid in [None, 0, 1, -1, 1.5, "", "true", "1", "H100"]:
with pytest.raises(ValueError, match="gpu must be a boolean"):
udf(name="invalid_gpu", gpu=invalid)(identity)
base_runtime = {
"kind": "python_v2",
"python_version": "3.12",
"environment": {"kind": "pip"},
}
runtime = PythonRuntimeSpec.model_validate({**base_runtime, "gpu": True})
assert runtime.gpu is True
for invalid in [False, 1, 0, "", "true", "1", "H100"]:
with pytest.raises(ValueError, match="runtime.gpu must be true"):
PythonRuntimeSpec.model_validate({**base_runtime, "gpu": invalid})
def test_unknown_runtime_discards_payload_before_known_field_validation():
for payload in [
{"kind": "python_v3", "gpu": {"model": "H100"}},
{"kind": "python_v3", "resources": []},
{
"kind": "python_v3",
"environment": {"kind": []},
"python_version": 3.15,
},
]:
runtime = PythonRuntimeSpec.model_validate(payload)
assert runtime.to_canonical_json() == '{"kind":"python_v3"}'
def test_udf_packages_attribute_access_and_body_imports():
@udf
def word_norm(body: str) -> float:
@@ -168,9 +302,7 @@ def test_udf_resolves_module_globals_before_builtins(tmp_path):
udf(module.uses_callable_shadow)
def test_canonical_arrow_type_is_exactly_the_grammar():
from lancedb.functions import _GRAMMAR_PRIMITIVES, _canonical_arrow_type
def test_canonical_arrow_type_prefers_the_compact_grammar():
golden = json.loads(
(
Path(__file__).parents[3]
@@ -181,14 +313,19 @@ def test_canonical_arrow_type_is_exactly_the_grammar():
case["arrow_type"] for case in golden["valid"] if "<" not in case["arrow_type"]
]
assert [name for _, name in _GRAMMAR_PRIMITIVES] == primitives
assert _canonical_arrow_type(pa.list_(pa.field("item", pa.float32(), False))) == (
"list<float32>"
)
assert (
_canonical_arrow_type(pa.large_list(pa.field("item", pa.float32(), False)))
== "large_list<float32>"
)
for outside in [
pa.timestamp("us"),
pa.decimal128(10, 2),
pa.large_string(),
pa.large_binary(),
pa.binary(4),
pa.duration("s"),
pa.struct([pa.field("a", pa.int32())]),
pa.list_(pa.float32(), 0),
pa.list_(pa.timestamp("us")),
]:
@@ -378,14 +515,27 @@ def test_udf_recursion_versus_a_rebound_module_name(tmp_path):
udf(raw_fact)
def test_canonical_arrow_type_rejects_unrepresentable_list_children():
from lancedb.functions import _canonical_arrow_type
def test_canonical_arrow_type_uses_exact_json_for_list_child_properties():
nullable = pa.list_(pa.float32())
assert json.loads(_canonical_arrow_type(nullable)) == {
"type": "list",
"fields": [
{
"name": "item",
"nullable": True,
"type": {"type": "float32"},
}
],
}
named = pa.list_(pa.field("custom", pa.float32(), nullable=False))
assert json.loads(_canonical_arrow_type(named))["fields"][0]["name"] == "custom"
for outside in [
pa.list_(pa.float32()), # pyarrow default: nullable child
pa.list_(pa.field("custom", pa.float32(), nullable=False)),
pa.list_(pa.field("item", pa.float32(), nullable=False, metadata={"k": "v"})),
pa.list_(pa.field("item", pa.float32(), nullable=False), 0),
pa.list_(
pa.field("item", pa.float32(), nullable=False, metadata={"k": "v"}), 3
),
pa.list_(pa.field("custom", pa.float32(), nullable=False), 3),
]:
with pytest.raises(TypeError, match="unsupported Arrow type"):
_canonical_arrow_type(outside)
@@ -395,6 +545,29 @@ def test_canonical_arrow_type_rejects_unrepresentable_list_children():
)
== "fixed_size_list<float32, 3>"
)
fixed = json.loads(_canonical_arrow_type(pa.list_(pa.float32(), 3)))
assert fixed == {
"type": "fixed_size_list",
"fields": [
{
"name": "item",
"nullable": True,
"type": {"type": "float32"},
}
],
"length": 3,
}
large = json.loads(_canonical_arrow_type(pa.large_list(pa.float32())))
assert large["type"] == "large_list"
assert large["fields"][0]["nullable"] is True
for invalid_struct in [
pa.struct([]),
pa.struct([pa.field("a", pa.int32()), pa.field("a", pa.int64())]),
pa.struct([pa.field("", pa.int32())]),
]:
with pytest.raises(TypeError, match="unsupported Arrow type"):
_canonical_arrow_type(invalid_struct)
def _calls_missing(value: int) -> int:
@@ -432,6 +605,7 @@ def _arrow_type_from_golden(spec: dict) -> pa.DataType:
"null": pa.null(),
"bool": pa.bool_(),
"utf8": pa.string(),
"large_utf8": pa.large_string(),
"binary": pa.binary(),
"float16": pa.float16(),
"float32": pa.float32(),
@@ -448,8 +622,6 @@ def test_arrow_type_grammar_matches_the_shared_golden():
/ "rust/lancedb/tests/fixtures/first_class_functions/v1/arrow_types.json"
).read_text()
)
from lancedb.functions import _canonical_arrow_type
emitted = {
case["arrow_type"]: _canonical_arrow_type(_arrow_type_from_golden(case["json"]))
for case in golden["valid"]
@@ -482,6 +654,395 @@ def test_explicit_arrow_schema_is_deterministic():
assert signature.output.nullable is False
def test_blob_fields_use_the_scalar_function_semantic_type():
@udf(
input_schema=pa.schema([lancedb.blob("image", nullable=False)]),
output_schema=lancedb.blob("result", nullable=False),
)
def copy_blob(image):
return image
signature = copy_blob.registration_request.signature
assert signature.inputs[0].arrow_type == "blob_v2"
assert signature.output.kind == "scalar"
assert signature.output.arrow_type == "blob_v2"
def test_whole_named_struct_function_can_include_a_blob_result_field():
@udf(
input_schema=pa.schema([lancedb.blob("image", nullable=False)]),
output_schema=pa.field(
"payload",
pa.struct(
[
pa.field("mime_type", pa.string(), nullable=False),
lancedb.blob("image", nullable=False),
]
),
nullable=False,
),
)
def inspect_blob(image):
return {"mime_type": "image/png", "image": image}
output = inspect_blob.registration_request.signature.output
assert output.kind == "named_struct"
assert [(field.name, field.arrow_type) for field in output.fields] == [
("mime_type", "utf8"),
("image", "blob_v2"),
]
def test_struct_blob_signature_fields_preserve_exact_metadata_and_nullability():
nested_input = pa.field(
"payload",
pa.struct(
[
pa.field("mime_type", pa.string(), nullable=False),
pa.field(
"nested",
pa.struct([lancedb.blob("image", nullable=True)]),
nullable=True,
),
]
),
nullable=True,
)
nested_output = pa.field(
"result",
pa.struct(
[
pa.field("mime_type", pa.string(), nullable=False),
pa.field(
"nested",
pa.struct([lancedb.blob("image", nullable=True)]),
nullable=False,
),
]
),
nullable=False,
)
@udf(input_schema=pa.schema([nested_input]), output_schema=nested_output)
def copy_payload(payload):
return payload
signature = copy_payload.registration_request.signature
input_type = json.loads(signature.inputs[0].arrow_type)
assert input_type["fields"][1]["nullable"] is True
input_blob = input_type["fields"][1]["type"]["fields"][0]
assert input_blob["nullable"] is True
assert input_blob["metadata"] == {"ARROW:extension:name": "lance.blob.v2"}
assert signature.output.kind == "named_struct"
nested_result = next(
field for field in signature.output.fields if field.name == "nested"
)
output_type = json.loads(nested_result.arrow_type)
output_blob = output_type["fields"][0]
assert output_blob["nullable"] is True
assert output_blob["metadata"] == {"ARROW:extension:name": "lance.blob.v2"}
def test_struct_blob_signature_supports_multiple_struct_levels():
recursive = pa.field(
"value",
pa.struct(
[
pa.field(
"level_1",
pa.struct(
[
pa.field(
"level_2",
pa.struct([lancedb.blob("image", nullable=False)]),
nullable=False,
)
]
),
nullable=False,
)
]
),
nullable=False,
)
@udf(
input_schema=pa.schema([recursive]),
output_schema=pa.field("size", pa.int64(), nullable=False),
)
def blob_size(value):
return len(value["level_1"]["level_2"]["image"])
encoded = json.loads(blob_size.registration_request.signature.inputs[0].arrow_type)
blob = encoded["fields"][0]["type"]["fields"][0]["type"]["fields"][0]
assert blob["metadata"]["ARROW:extension:name"] == "lance.blob.v2"
@pytest.mark.parametrize(
"data_type",
[
pa.list_(lancedb.blob("item", nullable=False)),
pa.large_list(lancedb.blob("item", nullable=False)),
pa.list_(lancedb.blob("item", nullable=False), 2),
pa.map_(pa.string(), lancedb.blob("value", nullable=False).type),
],
)
def test_blob_signature_rejects_collection_ancestors(data_type):
with pytest.raises(
TypeError,
match="Blob v2 fields nested under collection types are not supported",
):
@udf(
input_schema=pa.schema([pa.field("value", data_type, nullable=False)]),
output_schema=pa.field("size", pa.int64(), nullable=False),
)
def blob_size(value):
return len(value)
def test_blob_signature_rejects_collection_below_a_struct():
nested = pa.field(
"value",
pa.struct(
[
pa.field(
"images",
pa.list_(lancedb.blob("item", nullable=False)),
nullable=False,
)
]
),
nullable=False,
)
with pytest.raises(
TypeError,
match="Blob v2 fields nested under collection types are not supported",
):
@udf(
input_schema=pa.schema([nested]),
output_schema=pa.field("size", pa.int64(), nullable=False),
)
def blob_size(value):
return len(value["images"])
def test_named_struct_function_can_include_a_blob_result_field():
@udf(
input_schema=pa.schema([lancedb.blob("image", nullable=False)]),
output_schema=pa.schema(
[
lancedb.blob("thumbnail", nullable=False),
pa.field("width", pa.int32(), nullable=False),
]
),
)
def inspect_blob(image):
return {"thumbnail": image, "width": 1}
output = inspect_blob.registration_request.signature.output
assert output.kind == "named_struct"
assert [(field.name, field.arrow_type) for field in output.fields] == [
("thumbnail", "blob_v2"),
("width", "int32"),
]
def test_metadata_marked_blob_field_uses_the_semantic_type():
extension = lancedb.blob("image", nullable=False).type
storage = (
extension.storage_type if isinstance(extension, pa.ExtensionType) else extension
)
metadata_blob = pa.field(
"image",
storage,
nullable=False,
metadata={"ARROW:extension:name": "lance.blob.v2"},
)
@udf(
input_schema=pa.schema([metadata_blob]),
output_schema=pa.field("size", pa.int64(), nullable=False),
)
def blob_size(image):
return len(image)
assert blob_size.registration_request.signature.inputs[0].arrow_type == "blob_v2"
def test_blob_marker_rejects_invalid_storage_layout():
malformed = pa.field(
"image",
pa.int64(),
nullable=False,
metadata={"ARROW:extension:name": "lance.blob.v2"},
)
with pytest.raises(TypeError, match="requires a supported Blob storage layout"):
@udf(
input_schema=pa.schema([malformed]),
output_schema=pa.field("size", pa.int64(), nullable=False),
)
def blob_size(image):
return len(image)
def test_nested_non_blob_extension_is_not_silently_unwrapped():
class TestExtension(pa.ExtensionType):
def __init__(self):
super().__init__(pa.int64(), "test.function.extension")
def __arrow_ext_serialize__(self):
return b""
@classmethod
def __arrow_ext_deserialize__(cls, storage_type, serialized):
return cls()
nested = pa.field(
"value",
pa.struct([pa.field("extended", TestExtension(), nullable=False)]),
nullable=False,
)
with pytest.raises(TypeError, match="unsupported Arrow type"):
@udf(
input_schema=pa.schema([nested]),
output_schema=pa.field("result", pa.int64(), nullable=False),
)
def extension_value(value):
return value["extended"]
def test_explicit_large_utf8_schemas_use_the_canonical_function_name():
input_schema = pa.schema([pa.field("text", pa.large_string(), nullable=True)])
output_schema = pa.field("result", pa.large_string(), nullable=False)
@udf(input_schema=input_schema, output_schema=output_schema)
def preserve(text):
return text
signature = preserve.registration_request.signature
assert signature.inputs[0].arrow_type == "large_utf8"
assert signature.inputs[0].nullable is True
assert signature.output.arrow_type == "large_utf8"
assert signature.output.nullable is False
nested = pa.struct([pa.field("text", pa.large_string(), nullable=True)])
assert json.loads(_canonical_arrow_type(nested)) == {
"type": "struct",
"fields": [
{
"name": "text",
"nullable": True,
"type": {"type": "large_utf8"},
}
],
}
def test_nested_struct_output_uses_canonical_exact_json():
token = pa.struct(
[
pa.field("position", pa.int32(), nullable=False),
pa.field("value", pa.string(), nullable=False),
pa.field("length", pa.int32(), nullable=False),
]
)
analysis = pa.struct(
[
pa.field("normalized_text", pa.string(), nullable=False),
pa.field("has_content", pa.bool_(), nullable=False),
pa.field(
"metrics",
pa.struct(
[
pa.field("character_count", pa.int64(), nullable=False),
pa.field("word_count", pa.int32(), nullable=False),
pa.field("average_word_length", pa.float64(), nullable=False),
]
),
nullable=False,
),
pa.field(
"diagnostics",
pa.struct(
[
pa.field("status", pa.string(), nullable=False),
pa.field(
"normalization",
pa.struct(
[
pa.field("changed", pa.bool_(), nullable=False),
pa.field(
"original_length", pa.int64(), nullable=False
),
]
),
nullable=False,
),
]
),
nullable=False,
),
pa.field(
"token_preview",
pa.list_(pa.field("item", token, nullable=False)),
nullable=False,
),
]
)
@udf(
input_schema=pa.schema([pa.field("text", pa.string(), nullable=False)]),
output_schema=pa.field("analysis", analysis, nullable=False),
)
def analyze(text):
return {"normalized_text": text}
output = analyze.registration_request.signature.output
assert output.kind == "named_struct"
assert [field.name for field in output.fields] == [
"normalized_text",
"has_content",
"metrics",
"diagnostics",
"token_preview",
]
metrics = json.loads(output.fields[2].arrow_type)
assert metrics == {
"type": "struct",
"fields": [
{
"name": "character_count",
"nullable": False,
"type": {"type": "int64"},
},
{
"name": "word_count",
"nullable": False,
"type": {"type": "int32"},
},
{
"name": "average_word_length",
"nullable": False,
"type": {"type": "float64"},
},
],
}
preview = json.loads(output.fields[4].arrow_type)
assert preview["type"] == "list"
assert preview["fields"][0]["type"]["type"] == "struct"
assert [field["name"] for field in preview["fields"][0]["type"]["fields"]] == [
"position",
"value",
"length",
]
def test_annotation_and_explicit_schema_validation_fail_closed():
with pytest.raises(TypeError, match="missing Function annotations"):
@@ -525,6 +1086,72 @@ def test_annotation_and_explicit_schema_validation_fail_closed():
def nullable_explicit(value):
return value
for invalid_field in [
pa.field("", pa.int32(), nullable=False),
pa.field("result", pa.int32(), nullable=False, metadata={"k": "v"}),
]:
with pytest.raises(TypeError, match="unsupported Arrow type"):
@udf(
input_schema=pa.schema([pa.field("value", pa.int64())]),
output_schema=pa.schema([invalid_field]),
)
def invalid_explicit_field(value):
return value
with pytest.raises(TypeError, match="unsupported Arrow type"):
@udf(
input_schema=pa.schema(
[pa.field("value", pa.int64(), metadata={"k": "v"})]
),
output_schema=pa.int64(),
)
def input_field_metadata(value):
return value
with pytest.raises(TypeError, match="unsupported Arrow type"):
@udf(
input_schema=pa.schema([pa.field("value", pa.int64())]),
output_schema=pa.field(
"result", pa.int64(), nullable=False, metadata={"k": "v"}
),
)
def scalar_output_field_metadata(value):
return value
struct_type = pa.struct([pa.field("value", pa.int64(), nullable=False)])
with pytest.raises(TypeError, match="unsupported Arrow type"):
@udf(
input_schema=pa.schema([pa.field("value", pa.int64())]),
output_schema=pa.field(
"result", struct_type, nullable=False, metadata={"k": "v"}
),
)
def struct_output_field_metadata(value):
return {"value": value}
for input_schema, output_schema in [
(
pa.schema([pa.field("value", pa.int64())], metadata={"k": "v"}),
pa.int64(),
),
(
pa.schema([pa.field("value", pa.int64())]),
pa.schema(
[pa.field("result", pa.int64(), nullable=False)],
metadata={"k": "v"},
),
),
]:
with pytest.raises(TypeError, match="schema metadata"):
@udf(input_schema=input_schema, output_schema=output_schema)
def schema_metadata(value):
return value
def test_local_function_catalog_operations_are_not_supported(tmp_path):
db = lancedb.connect(tmp_path)
@@ -535,6 +1162,8 @@ def test_local_function_catalog_operations_are_not_supported(tmp_path):
db.create_function_async(normalize_score)
with pytest.raises(NotImplementedError, match=message):
db.get_function("normalize_score", version="fv_exact")
with pytest.raises(NotImplementedError, match=message):
db.drop_function("normalize_score", version="fv_exact")
@contextlib.contextmanager
@@ -580,6 +1209,12 @@ def _mock_remote_function_catalog():
"version": "fv_exact",
}
response = state["version"]
elif self.path == "/v1/functions/drop":
assert body == {
"name": "normalize_score",
"version": "fv_exact",
}
response = {"dropped": True}
else:
status = 404
response = {"error": "not found"}
@@ -638,3 +1273,40 @@ def test_blocking_remote_registration_returns_function_version():
"/v1/functions/create",
"/v1/jobs/describe",
]
def test_remote_drop_function_sends_exact_version():
with _mock_remote_function_catalog() as (host, state):
db = lancedb.connect(
"db://dev",
api_key="fake",
host_override=host,
client_config={"retry_config": {"retries": 0}},
)
assert db.drop_function("normalize_score", version="fv_exact") is True
assert state["requests"] == [
(
"/v1/functions/drop",
{"name": "normalize_score", "version": "fv_exact"},
)
]
@pytest.mark.asyncio
async def test_async_remote_drop_function_sends_exact_version():
with _mock_remote_function_catalog() as (host, state):
db = await lancedb.connect_async(
"db://dev",
api_key="fake",
host_override=host,
client_config={"retry_config": {"retries": 0}},
)
assert await db.drop_function("normalize_score", version="fv_exact") is True
assert state["requests"] == [
(
"/v1/functions/drop",
{"name": "normalize_score", "version": "fv_exact"},
)
]
+11
View File
@@ -629,6 +629,17 @@ impl Connection {
})
}
pub fn drop_function(
self_: PyRef<'_, Self>,
name: String,
version: String,
) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.get_inner()?.clone();
future_into_py(self_.py(), async move {
inner.drop_function(name, version).await.infer_error()
})
}
pub fn list_jobs(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.get_inner()?.clone();
future_into_py(self_.py(), async move {
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "lancedb"
version = "0.38.0-beta.12"
version = "0.38.0"
edition.workspace = true
description = "LanceDB: A serverless, low-latency vector database for AI applications"
license.workspace = true
+15
View File
@@ -523,6 +523,21 @@ impl Connection {
.await
}
/// Drop one exact immutable Function version from the remote catalog.
///
/// Returns `true` when the server appended a Dropped transition and
/// `false` for an idempotent replay. Local databases return
/// [`Error::NotSupported`].
pub async fn drop_function(
&self,
name: impl AsRef<str>,
version: impl AsRef<str>,
) -> Result<bool> {
self.internal
.drop_function(name.as_ref(), version.as_ref())
.await
}
/// Rename a table in the database.
///
/// This is only supported in LanceDB Cloud.
+4
View File
@@ -307,6 +307,10 @@ pub trait Database:
) -> Result<crate::function::FunctionVersion> {
function_catalog_not_supported()
}
/// Drop one exact immutable Function version from the remote catalog.
async fn drop_function(&self, _name: &str, _version: &str) -> Result<bool> {
function_catalog_not_supported()
}
/// A [`crate::job::Job`] handle for a server-side job by id, suitable for
/// waiting on or cancelling the job. The handle is constructed without a
/// server round trip; an unknown id surfaces when the handle is used.
+10 -3
View File
@@ -539,9 +539,7 @@ impl Database for LanceNamespaceDatabase {
self.namespace
.drop_table(drop_request)
.await
.map_err(|e| Error::Runtime {
message: format!("Failed to drop table: {}", e),
})?;
.map_err(|e| map_namespace_lance_error(e, name))?;
Ok(())
}
@@ -1495,6 +1493,15 @@ mod tests {
.expect("Failed to list tables");
assert!(!table_names_after.contains(&"drop_test".to_string()));
let error = conn
.drop_table("drop_test", &["test_ns".into()])
.await
.expect_err("dropping a missing table should fail");
assert!(
matches!(error, Error::TableNotFound { ref name, .. } if name == "drop_test"),
"expected TableNotFound, got: {error:?}"
);
// Verify: Cannot open dropped table
let open_result = conn.open_table("drop_test").execute().await;
assert!(open_result.is_err());
+153 -27
View File
@@ -15,6 +15,9 @@ use serde_json::Value;
use crate::{Error, Result};
/// Semantic Function type for a Blob v2 value.
pub const FUNCTION_BLOB_V2_TYPE: &str = "blob_v2";
fn invalid_json(error: impl std::fmt::Display) -> Error {
Error::InvalidInput {
message: format!("invalid remote Function JSON: {error}"),
@@ -207,6 +210,33 @@ pub enum PythonRuntimeSpec {
environment: PythonEnvironmentSpec,
env: BTreeMap<String, String>,
},
/// The GPU-enabled Sophon-managed Python runtime.
///
/// # Examples
///
/// ```
/// use std::collections::BTreeMap;
/// use lancedb::function::{PythonEnvironmentSpec, PythonRuntimeSpec};
///
/// let runtime = PythonRuntimeSpec::PythonV2 {
/// python_version: "3.12".to_string(),
/// environment: PythonEnvironmentSpec {
/// kind: "pip".to_string(),
/// packages: vec!["cupy-cuda12x".to_string()],
/// channels: Vec::new(),
/// path: None,
/// modules: Vec::new(),
/// image: None,
/// },
/// env: BTreeMap::new(),
/// };
/// assert!(runtime.requires_gpu());
/// ```
PythonV2 {
python_version: String,
environment: PythonEnvironmentSpec,
env: BTreeMap<String, String>,
},
/// A runtime kind introduced by a newer server.
///
/// Unknown payload fields are intentionally not retained because the
@@ -219,22 +249,27 @@ impl PythonRuntimeSpec {
pub fn kind(&self) -> &str {
match self {
Self::Python { .. } => "python",
Self::PythonV2 { .. } => "python_v2",
Self::Unrecognized { kind } => kind,
}
}
/// The Python version for the V1 runtime, or `None` for an unknown kind.
/// The Python version for a known Python runtime, or `None` for an unknown kind.
pub fn python_version(&self) -> Option<&str> {
match self {
Self::Python { python_version, .. } => Some(python_version),
Self::Python { python_version, .. } | Self::PythonV2 { python_version, .. } => {
Some(python_version)
}
Self::Unrecognized { .. } => None,
}
}
/// The Python environment for the V1 runtime, or `None` for an unknown kind.
/// The Python environment for a known Python runtime, or `None` for an unknown kind.
pub fn environment(&self) -> Option<&PythonEnvironmentSpec> {
match self {
Self::Python { environment, .. } => Some(environment),
Self::Python { environment, .. } | Self::PythonV2 { environment, .. } => {
Some(environment)
}
Self::Unrecognized { .. } => None,
}
}
@@ -242,38 +277,73 @@ impl PythonRuntimeSpec {
/// Environment variables, or `None` for an unknown kind.
pub fn env(&self) -> Option<&BTreeMap<String, String>> {
match self {
Self::Python { env, .. } => Some(env),
Self::Python { env, .. } | Self::PythonV2 { env, .. } => Some(env),
Self::Unrecognized { .. } => None,
}
}
/// Whether the runtime requires a GPU selected by the execution platform.
pub fn requires_gpu(&self) -> bool {
matches!(self, Self::PythonV2 { .. })
}
}
#[derive(Deserialize)]
struct PythonRuntimeWire {
kind: String,
#[serde(default)]
python_version: Option<String>,
#[serde(default)]
environment: Option<PythonEnvironmentSpec>,
struct PythonRuntimeV1Wire {
python_version: String,
environment: PythonEnvironmentSpec,
#[serde(default)]
env: BTreeMap<String, String>,
#[serde(default)]
gpu: Option<Value>,
}
#[derive(Deserialize)]
struct PythonRuntimeV2Wire {
python_version: String,
environment: PythonEnvironmentSpec,
#[serde(default)]
env: BTreeMap<String, String>,
gpu: bool,
}
impl<'de> Deserialize<'de> for PythonRuntimeSpec {
fn deserialize<D: Deserializer<'de>>(deserializer: D) -> std::result::Result<Self, D::Error> {
let wire = PythonRuntimeWire::deserialize(deserializer)?;
if wire.kind == "python" {
Ok(Self::Python {
python_version: wire
.python_version
.ok_or_else(|| de::Error::missing_field("python_version"))?,
environment: wire
.environment
.ok_or_else(|| de::Error::missing_field("environment"))?,
env: wire.env,
})
} else {
Ok(Self::Unrecognized { kind: wire.kind })
let value = Value::deserialize(deserializer)?;
let kind = value
.get("kind")
.ok_or_else(|| de::Error::missing_field("kind"))?
.as_str()
.ok_or_else(|| de::Error::custom("runtime.kind must be a string"))?
.to_string();
match kind.as_str() {
"python" => {
let wire: PythonRuntimeV1Wire =
serde_json::from_value(value).map_err(de::Error::custom)?;
if wire.gpu.is_some() {
return Err(de::Error::custom(
"python runtime with gpu requires kind='python_v2'",
));
}
Ok(Self::Python {
python_version: wire.python_version,
environment: wire.environment,
env: wire.env,
})
}
"python_v2" => {
let wire: PythonRuntimeV2Wire =
serde_json::from_value(value).map_err(de::Error::custom)?;
if !wire.gpu {
return Err(de::Error::custom("runtime.gpu must be true"));
}
Ok(Self::PythonV2 {
python_version: wire.python_version,
environment: wire.environment,
env: wire.env,
})
}
_ => Ok(Self::Unrecognized { kind }),
}
}
}
@@ -287,6 +357,8 @@ impl Serialize for PythonRuntimeSpec {
environment: &'a PythonEnvironmentSpec,
#[serde(skip_serializing_if = "BTreeMap::is_empty")]
env: &'a BTreeMap<String, String>,
#[serde(skip_serializing_if = "Option::is_none")]
gpu: Option<bool>,
}
#[derive(Serialize)]
@@ -304,6 +376,19 @@ impl Serialize for PythonRuntimeSpec {
python_version,
environment,
env,
gpu: None,
}
.serialize(serializer),
Self::PythonV2 {
python_version,
environment,
env,
} => PythonRuntimeRef {
kind: "python_v2",
python_version,
environment,
env,
gpu: Some(true),
}
.serialize(serializer),
Self::Unrecognized { kind } => UnrecognizedRuntimeRef { kind }.serialize(serializer),
@@ -313,8 +398,8 @@ impl Serialize for PythonRuntimeSpec {
/// Immutable Function version returned by the Enterprise catalog.
///
/// Scheduling resources, priority, concurrency, and retry policy belong to
/// the submitting Job and are not part of this identity.
/// The GPU execution requirement is part of this identity. CPU and memory sizing,
/// priority, concurrency, and retry policy belong to the execution platform.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct FunctionVersion {
name: String,
@@ -589,7 +674,7 @@ impl_json!(RefreshColumnResult);
#[cfg(test)]
mod conda_environment_tests {
use super::PythonEnvironmentSpec;
use super::{PythonEnvironmentSpec, PythonRuntimeSpec};
#[test]
fn conda_channels_round_trip_and_pip_stays_bare() {
@@ -608,4 +693,45 @@ mod conda_environment_tests {
serde_json::from_str(r#"{"kind":"pip","packages":["numpy"]}"#).unwrap();
assert!(!serde_json::to_string(&pip).unwrap().contains("channels"));
}
#[test]
fn gpu_python_runtime_marker_round_trips_and_validates() {
let runtime: PythonRuntimeSpec = serde_json::from_str(
r#"{"kind":"python_v2","python_version":"3.12","environment":{"kind":"pip"},"gpu":true}"#,
)
.unwrap();
assert_eq!(runtime.kind(), "python_v2");
assert!(runtime.requires_gpu());
assert_eq!(
super::canonical_json(&runtime).unwrap(),
r#"{"environment":{"kind":"pip"},"gpu":true,"kind":"python_v2","python_version":"3.12"}"#
);
for invalid in [
r#"{"kind":"python","python_version":"3.12","environment":{"kind":"pip"},"gpu":true}"#,
r#"{"kind":"python_v2","python_version":"3.12","environment":{"kind":"pip"}}"#,
r#"{"kind":"python_v2","python_version":"3.12","environment":{"kind":"pip"},"gpu":1}"#,
r#"{"kind":"python_v2","python_version":"3.12","environment":{"kind":"pip"},"gpu":false}"#,
r#"{"kind":"python_v2","python_version":"3.12","environment":{"kind":"pip"},"gpu":"true"}"#,
r#"{"kind":"python_v2","python_version":"3.12","environment":{"kind":"pip"},"gpu":"H100"}"#,
] {
assert!(serde_json::from_str::<PythonRuntimeSpec>(invalid).is_err());
}
}
#[test]
fn unknown_runtime_discards_payload_before_known_field_validation() {
for encoded in [
r#"{"kind":"python_v3","gpu":{"model":"H100"}}"#,
r#"{"kind":"python_v3","resources":[]}"#,
r#"{"kind":"python_v3","python_version":3.15,"environment":{"kind":[]}}"#,
] {
let runtime: PythonRuntimeSpec = serde_json::from_str(encoded).unwrap();
assert_eq!(runtime.kind(), "python_v3");
assert_eq!(
super::canonical_json(&runtime).unwrap(),
r#"{"kind":"python_v3"}"#
);
}
}
}
+38
View File
@@ -533,6 +533,11 @@ struct RemoteListJobsResponse {
page_token: Option<String>,
}
#[derive(serde::Deserialize)]
struct RemoteDropFunctionResponse {
dropped: bool,
}
/// Bound on `list_jobs` page walking; a warning is logged when the listing
/// is truncated at this many pages.
const MAX_LIST_JOBS_PAGES: usize = 100;
@@ -583,6 +588,20 @@ impl<S: HttpSend> Database for RemoteDatabase<S> {
response.json().await.err_to_http(request_id)
}
async fn drop_function(&self, name: &str, version: &str) -> Result<bool> {
let req = self
.client
.post("/v1/functions/drop")
.json(&serde_json::json!({
"name": name,
"version": version,
}));
let (request_id, response) = self.client.send(req).await?;
let response = self.client.check_response(&request_id, response).await?;
let response: RemoteDropFunctionResponse = response.json().await.err_to_http(request_id)?;
Ok(response.dropped)
}
fn job(&self, job_id: &str) -> Result<crate::job::Job> {
Ok(crate::job::Job::new(Box::new(super::job::RemoteJob::new(
self.client.clone(),
@@ -2689,6 +2708,25 @@ mod tests {
assert_eq!(version.version(), "fv_01K3EXACT");
}
#[tokio::test]
async fn test_drop_function_sends_exact_version_and_decodes_replay() {
let conn = Connection::new_with_handler(|request| {
assert_eq!(request.method(), &reqwest::Method::POST);
assert_eq!(request.url().path(), "/v1/functions/drop");
let body: serde_json::Value =
serde_json::from_slice(request.body().unwrap().as_bytes().unwrap()).unwrap();
assert_eq!(
body,
serde_json::json!({"name": "embed", "version": "fv_01K3EXACT"})
);
http::Response::builder()
.status(200)
.body(r#"{"dropped":false}"#)
.unwrap()
});
assert!(!conn.drop_function("embed", "fv_01K3EXACT").await.unwrap());
}
#[tokio::test]
async fn test_conn_job_waits_to_done() {
let polls = Arc::new(AtomicUsize::new(0));
+89 -3
View File
@@ -5734,9 +5734,8 @@ mod tests {
))
.execute()
.await;
let err = match result {
Ok(_) => panic!("legacy remote query unexpectedly succeeded"),
Err(err) => err,
let Err(err) = result else {
panic!("legacy remote query unexpectedly succeeded")
};
assert!(
@@ -7465,6 +7464,93 @@ mod tests {
assert_eq!(result.version, 8);
}
#[tokio::test]
async fn test_add_function_column_allows_an_existing_binding() {
let binding = crate::function::FunctionBinding::from_json(include_str!(
"../../tests/fixtures/first_class_functions/v1/remote_function_binding.json"
))
.unwrap();
let binding_metadata = crate::table::computed_columns::function_bindings_metadata(
std::slice::from_ref(&binding),
)
.unwrap();
let mut fields = vec![
Field::new("title", DataType::Utf8, true),
Field::new("body", DataType::Utf8, true),
];
fields.extend(binding.outputs().iter().map(|output| {
let data_type = match output.arrow_type.as_str() {
"utf8" => DataType::Utf8,
"int64" => DataType::Int64,
other => panic!("unexpected fixture output type {other}"),
};
Field::new(&output.output_name, data_type, true).with_metadata(
crate::table::computed_columns::function_computed_column_metadata(
binding.binding_id(),
output.output_ordinal,
&["title".into(), "body".into()],
),
)
}));
let schema = Schema::new_with_metadata(
fields,
HashMap::from([(
crate::table::computed_columns::FUNCTION_BINDINGS_META_KEY.to_string(),
binding_metadata,
)]),
);
let table =
Table::new_with_handler("my_table", move |request| match request.url().path() {
"/v1/table/my_table/describe/" => http::Response::builder()
.status(200)
.body(describe_response(&schema))
.unwrap(),
"/v1/table/my_table/add_columns/" => {
let actual: serde_json::Value =
serde_json::from_slice(request.body().unwrap().as_bytes().unwrap())
.unwrap();
assert_eq!(
actual["new_columns"],
serde_json::json!([
{"name":"secondary_text","all_null":true},
{"name":"secondary_token_count","all_null":true}
])
);
http::Response::builder()
.status(200)
.body(r#"{"version":10}"#.to_string())
.unwrap()
}
path => panic!("Unexpected path: {path}"),
});
let application = crate::function::FunctionApplication::from_json(
r#"{
"function":{"name":"text_features","version":"fv_01K3TEXT"},
"inputs":[
{"parameter":"title","kind":"column","value":{"path":"title"}},
{"parameter":"body","kind":"column","value":{"path":"body"}}
],
"output":{"kind":"named_struct","fields":[
{"name":"normalized_text","arrow_type":"utf8","nullable":false},
{"name":"token_count","arrow_type":"int64","nullable":false}
]},
"columns":{
"normalized_text":"secondary_text",
"token_count":"secondary_token_count"
}
}"#,
)
.unwrap();
let result = table
.add_columns()
.function(application)
.execute()
.await
.unwrap();
assert_eq!(result.version, 10);
}
#[tokio::test]
async fn test_add_fixed_size_list_function_column_declares_the_vector_type() {
let table = Table::new_with_handler("my_table", |request| {
+7
View File
@@ -5763,6 +5763,13 @@ mod tests {
assert!(index_bytes > 0);
assert_eq!(with_index, data_only + index_bytes);
// Release builds reject unstable overlay datasets unless explicitly opted in.
if !lance_table::feature_flags::can_read_dataset(
lance_table::feature_flags::FLAG_UNSTABLE_DATA_OVERLAY_FILES,
) {
return;
}
// Commit an overlay file supplying new `foo` values for the first three
// rows of fragment 0. There is no high-level API that writes overlays
// yet, so write the overlay's data file and commit the `DataOverlay`
+731 -50
View File
@@ -29,14 +29,16 @@ use datafusion_common::{ScalarValue, tree_node::TreeNode};
use datafusion_expr::Expr;
use datafusion_physical_plan::PhysicalExpr;
use lance::dataset::NewColumnTransform;
use lance_arrow::FieldExt;
use lance_core::datatypes::{BLOB_V2_DESC_FIELD, format_field_path_minimal, parse_field_path};
use lance_arrow::{ARROW_EXT_NAME_KEY, BLOB_V2_EXT_NAME, FieldExt};
use lance_core::datatypes::{
BLOB_V2_DESC_FIELD, BlobV2Layout, format_field_path_minimal, parse_field_path,
};
use lance_datafusion::planner::Planner;
use lance_namespace::models::{JsonArrowDataType, JsonArrowField, JsonArrowSchema};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use crate::function::{FunctionApplication, FunctionBinding};
use crate::function::{FUNCTION_BLOB_V2_TYPE, FunctionApplication, FunctionBinding};
use crate::utils::resolve_arrow_field_path;
use crate::{Error, Result};
@@ -547,7 +549,12 @@ fn ensure_known_binding_shape(value: &Value) -> Result<()> {
Ok(())
}
fn resolve_field_path<'a>(schema: &'a ArrowSchema, path: &str) -> Result<&'a ArrowField> {
struct ResolvedFieldPath<'a> {
root: &'a ArrowField,
leaf: &'a ArrowField,
}
fn resolve_field_path<'a>(schema: &'a ArrowSchema, path: &str) -> Result<ResolvedFieldPath<'a>> {
let parts = lance_core::datatypes::parse_field_path(path).map_err(|e| {
invalid_function(format!("invalid Function input field path '{path}': {e}"))
})?;
@@ -556,25 +563,40 @@ fn resolve_field_path<'a>(schema: &'a ArrowSchema, path: &str) -> Result<&'a Arr
"Function input field path cannot be empty",
));
};
let mut field = schema
let root = schema
.field_with_name(root)
.map_err(|_| invalid_function(format!("unknown Function input column '{path}'")))?;
let mut leaf = root;
for child in children {
let DataType::Struct(fields) = field.data_type() else {
let DataType::Struct(fields) = leaf.data_type() else {
return Err(invalid_function(format!(
"Function input field path '{path}' traverses a non-struct field"
)));
};
field = fields
leaf = fields
.iter()
.find(|field| field.name() == child)
.map(AsRef::as_ref)
.ok_or_else(|| invalid_function(format!("unknown Function input column '{path}'")))?;
}
Ok(field)
Ok(ResolvedFieldPath { root, leaf })
}
fn canonical_input_arrow_type(field: &JsonArrowField) -> Result<String> {
let is_blob_v2 = field
.metadata
.as_ref()
.and_then(|metadata| metadata.get(ARROW_EXT_NAME_KEY))
.map(String::as_str)
== Some(BLOB_V2_EXT_NAME);
if is_blob_v2 || field.r#type.fields.is_some() {
let arrow_field = lance_namespace::schema::convert_json_arrow_field(field)
.map_err(|e| invalid_function(format!("invalid Function input field: {e}")))?;
validate_function_blob_nesting(&arrow_field, false)?;
if is_blob_v2 {
return Ok(FUNCTION_BLOB_V2_TYPE.to_string());
}
}
if field.r#type.fields.is_none() && field.r#type.length.is_none() {
Ok(field.r#type.r#type.clone())
} else {
@@ -584,6 +606,42 @@ fn canonical_input_arrow_type(field: &JsonArrowField) -> Result<String> {
}
}
fn has_supported_blob_v2_layout(field: &ArrowField) -> bool {
field.is_blob_v2()
&& matches!(
field.data_type(),
DataType::Struct(fields) if BlobV2Layout::classify(fields).is_some()
)
}
fn validate_function_blob_nesting(field: &ArrowField, inside_collection: bool) -> Result<()> {
if field.is_blob_v2() {
if inside_collection {
return Err(invalid_function(format!(
"Function field '{}' nests Blob v2 under a collection, which Function signatures do not support",
field.name()
)));
}
if !has_supported_blob_v2_layout(field) {
return Err(invalid_function(format!(
"Function field '{}' has an invalid Blob v2 storage layout",
field.name()
)));
}
return Ok(());
}
match field.data_type() {
DataType::Struct(fields) => fields
.iter()
.try_for_each(|field| validate_function_blob_nesting(field, inside_collection)),
DataType::List(field)
| DataType::LargeList(field)
| DataType::FixedSizeList(field, _)
| DataType::Map(field, _) => validate_function_blob_nesting(field, true),
_ => Ok(()),
}
}
/// `fixed_size_list<item, size>` -> (`item`, `size`); the comma must sit outside
/// any nested `<...>`.
fn split_fixed_size_list(raw: &str) -> Option<(&str, i32)> {
@@ -663,10 +721,82 @@ fn parse_output_arrow_type(raw: &str) -> Result<JsonArrowDataType> {
Ok(data_type)
}
fn function_output_field(name: &str, nullable: bool, raw: &str) -> Result<JsonArrowField> {
let field = if raw == FUNCTION_BLOB_V2_TYPE {
lance_namespace::schema::arrow_schema_to_json(&ArrowSchema::new(vec![crate::blob(
name, nullable,
)]))
.map_err(|e| invalid_function(format!("could not encode Blob v2 output field: {e}")))?
.fields
.into_iter()
.next()
.ok_or_else(|| invalid_function("Blob v2 output field is missing"))?
} else {
JsonArrowField::new(name.to_string(), nullable, parse_output_arrow_type(raw)?)
};
let arrow_field = lance_namespace::schema::convert_json_arrow_field(&field)
.map_err(|e| invalid_function(format!("invalid Function output field: {e}")))?;
validate_function_blob_nesting(&arrow_field, false)?;
Ok(field)
}
fn function_output_field_matches(expected: &ArrowField, actual: &ArrowField) -> bool {
expected.name() == actual.name()
&& expected.is_nullable() == actual.is_nullable()
&& if expected.is_blob_v2() {
has_supported_blob_v2_layout(expected) && has_supported_blob_v2_layout(actual)
} else {
function_output_type_matches(expected.data_type(), actual.data_type())
}
}
fn function_output_type_matches(expected: &DataType, actual: &DataType) -> bool {
if expected == actual {
return true;
}
match (expected, actual) {
(DataType::Struct(expected), DataType::Struct(actual)) => {
expected.len() == actual.len()
&& expected
.iter()
.zip(actual)
.all(|(expected, actual)| function_output_field_matches(expected, actual))
}
(DataType::List(expected), DataType::List(actual))
| (DataType::LargeList(expected), DataType::LargeList(actual)) => {
function_output_field_matches(expected, actual)
}
(
DataType::FixedSizeList(expected, expected_size),
DataType::FixedSizeList(actual, actual_size),
) => expected_size == actual_size && function_output_field_matches(expected, actual),
(DataType::Map(expected, expected_sorted), DataType::Map(actual, actual_sorted)) => {
expected_sorted == actual_sorted && function_output_field_matches(expected, actual)
}
_ => false,
}
}
fn function_output_type_has_blob(data_type: &DataType) -> bool {
match data_type {
DataType::Struct(fields) => fields
.iter()
.any(|field| field.is_blob_v2() || function_output_type_has_blob(field.data_type())),
DataType::List(field)
| DataType::LargeList(field)
| DataType::FixedSizeList(field, _)
| DataType::Map(field, _) => {
field.is_blob_v2() || function_output_type_has_blob(field.data_type())
}
_ => false,
}
}
fn ensure_binding_matches_schema(schema: &ArrowSchema, binding: &FunctionBinding) -> Result<()> {
let mut input_fields = Vec::with_capacity(binding.inputs().len());
for input in binding.inputs() {
let field = resolve_field_path(schema, &input.field_path)?;
let resolved = resolve_field_path(schema, &input.field_path)?;
let field = resolved.leaf;
if field
.metadata()
.get(COMPUTED_COLUMN_META_KEY)
@@ -721,6 +851,11 @@ fn ensure_binding_matches_schema(schema: &ArrowSchema, binding: &FunctionBinding
)));
}
let expected_inputs = binding
.inputs()
.iter()
.map(|input| input.field_path.clone())
.collect::<Vec<_>>();
let mut output_fields = Vec::with_capacity(binding.outputs().len());
for output in binding.outputs() {
let field = schema.field_with_name(&output.output_name).map_err(|_| {
@@ -737,25 +872,61 @@ fn ensure_binding_matches_schema(schema: &ArrowSchema, binding: &FunctionBinding
binding.binding_id()
)));
}
let expected_type = parse_output_arrow_type(&output.arrow_type)?;
let expected_type = lance_namespace::schema::convert_json_arrow_type(&expected_type)
.map_err(|e| invalid_function(format!("invalid Function output type: {e}")))?;
if field.data_type() != &expected_type {
let (type_matches, has_semantic_blob) = if output.arrow_type == FUNCTION_BLOB_V2_TYPE {
(has_supported_blob_v2_layout(field), true)
} else {
let expected_type = parse_output_arrow_type(&output.arrow_type)?;
let expected_type = lance_namespace::schema::convert_json_arrow_type(&expected_type)
.map_err(|e| invalid_function(format!("invalid Function output type: {e}")))?;
(
function_output_type_matches(&expected_type, field.data_type()),
function_output_type_has_blob(&expected_type),
)
};
if !type_matches {
return Err(invalid_function(format!(
"Function output '{}' type no longer matches binding '{}'",
output.output_name,
binding.binding_id()
)));
}
output_fields.push(ArrowField::new(
field.name().clone(),
field.data_type().clone(),
true,
));
}
let output_schema =
lance_namespace::schema::arrow_schema_to_json(&ArrowSchema::new(output_fields))
let metadata = field.metadata();
let declared_inputs = metadata
.get(INPUTS_META_KEY)
.and_then(|raw| serde_json::from_str::<Vec<String>>(raw).ok());
if metadata.get(COMPUTED_COLUMN_META_KEY).map(String::as_str) != Some("true")
|| metadata.get(KIND_META_KEY).map(String::as_str) != Some(FUNCTION_KIND)
|| metadata
.get(FUNCTION_BINDING_ID_META_KEY)
.map(String::as_str)
!= Some(binding.binding_id())
|| metadata
.get(FUNCTION_OUTPUT_ORDINAL_META_KEY)
.and_then(|value| value.parse::<u32>().ok())
!= Some(output.output_ordinal)
|| declared_inputs.as_deref() != Some(expected_inputs.as_slice())
{
return Err(invalid_function(format!(
"Function output '{}' declaration metadata does not match binding '{}'",
output.output_name,
binding.binding_id()
)));
}
if has_semantic_blob {
output_fields.push(function_output_field(
field.name(),
true,
&output.arrow_type,
)?);
} else {
let json = lance_namespace::schema::arrow_schema_to_json(&ArrowSchema::new(vec![
ArrowField::new(field.name().clone(), field.data_type().clone(), true),
]))
.map_err(|e| invalid_function(format!("invalid Function output schema: {e}")))?;
output_fields.push(json.fields.into_iter().next().unwrap());
}
}
let output_schema = JsonArrowSchema::new(output_fields);
let output_schema = serde_json::to_value(output_schema).map_err(|e| {
invalid_function(format!(
"could not encode exact Function output schema: {e}"
@@ -778,7 +949,7 @@ pub(crate) fn plan_function_application(
application: &FunctionApplication,
output_name: Option<&str>,
) -> Result<FunctionDeclarationPlan> {
ensure_no_function_bindings_for_mutation(schema, "Function binding declaration")?;
ensure_supported_function_metadata(schema)?;
if application.has_unknown_fields() {
return Err(Error::NotSupported {
message: "Function application contains fields from a newer contract".into(),
@@ -828,8 +999,9 @@ pub(crate) fn plan_function_application(
input.parameter
))
})?;
let field = resolve_field_path(schema, path)?;
if field
let resolved = resolve_field_path(schema, path)?;
if resolved
.root
.metadata()
.get(COMPUTED_COLUMN_META_KEY)
.map(String::as_str)
@@ -839,6 +1011,7 @@ pub(crate) fn plan_function_application(
"Function input '{path}' is computed; computed-on-computed bindings are not supported"
)));
}
let field = resolved.leaf;
let parameter_field = ArrowField::new(
input.parameter.clone(),
field.data_type().clone(),
@@ -882,16 +1055,15 @@ pub(crate) fn plan_function_application(
"Function logical outputs must be non-nullable during NULL assignment",
));
}
let data_type =
parse_output_arrow_type(output.arrow_type.as_deref().ok_or_else(|| {
invalid_function("scalar Function output is missing its Arrow type")
})?)?;
let arrow_type = output.arrow_type.as_deref().ok_or_else(|| {
invalid_function("scalar Function output is missing its Arrow type")
})?;
outputs.push(FunctionOutputTarget {
result_field: WHOLE_RESULT_FIELD.to_string(),
output_name: name.to_string(),
output_ordinal: 0,
});
output_fields.push(JsonArrowField::new(name.to_string(), true, data_type));
output_fields.push(function_output_field(name, true, arrow_type)?);
}
"named_struct" => {
if output.fields.is_empty() {
@@ -935,13 +1107,7 @@ pub(crate) fn plan_function_application(
let fields = output
.fields
.iter()
.map(|field| {
Ok(JsonArrowField::new(
field.name.clone(),
false,
parse_output_arrow_type(&field.arrow_type)?,
))
})
.map(|field| function_output_field(&field.name, false, &field.arrow_type))
.collect::<Result<Vec<_>>>()?;
let mut data_type = JsonArrowDataType::new("struct".to_string());
data_type.fields = Some(fields);
@@ -968,11 +1134,7 @@ pub(crate) fn plan_function_application(
output_name: name.clone(),
output_ordinal: ordinal as u32,
});
output_fields.push(JsonArrowField::new(
name.clone(),
true,
parse_output_arrow_type(&field.arrow_type)?,
));
output_fields.push(function_output_field(name, true, &field.arrow_type)?);
}
}
}
@@ -1462,9 +1624,7 @@ fn plan_declarations(schema: SchemaRef, columns: &[(String, String)]) -> Result<
for (name, expression) in columns {
if schema.field_with_name(name).is_ok() {
return Err(Error::ColumnAlreadyExists {
name: name.to_string(),
});
return Err(Error::ColumnAlreadyExists { name: name.clone() });
}
let bound = bind(schema.clone(), name, expression)?;
@@ -1611,7 +1771,7 @@ mod tests {
}
use arrow_array::record_batch;
use arrow_schema::DataType;
use arrow_schema::{DataType, TimeUnit};
use futures::TryStreamExt;
use lance::dataset::ColumnAlteration;
@@ -2572,6 +2732,95 @@ mod tests {
.unwrap()
}
fn blob_application(output: &str) -> FunctionApplication {
FunctionApplication::from_json(&format!(
r#"{{
"function":{{"name":"blob_features","version":"fv_blob"}},
"inputs":[
{{"parameter":"image","kind":"column","value":{{"path":"image"}}}}
],
"output":{output}
}}"#
))
.unwrap()
}
fn exact_arrow_type(field: ArrowField) -> String {
let json =
lance_namespace::schema::arrow_schema_to_json(&ArrowSchema::new(vec![field])).unwrap();
serde_json::to_string(json.fields[0].r#type.as_ref()).unwrap()
}
fn single_input_application(path: &str) -> FunctionApplication {
FunctionApplication::from_json(
&serde_json::json!({
"function": {"name": "inspect", "version": "fv_nested_blob"},
"inputs": [{
"parameter": "value",
"kind": "column",
"value": {"path": path}
}],
"output": {"kind": "scalar", "arrow_type": "int64", "nullable": false}
})
.to_string(),
)
.unwrap()
}
fn binding_from_plan(plan: &FunctionDeclarationPlan) -> FunctionBinding {
let inputs = plan
.input_bindings
.iter()
.enumerate()
.map(|(index, input)| {
serde_json::json!({
"parameter": input.parameter,
"field_id": index,
"field_path": input.field_path,
"arrow_type": input.arrow_type,
"nullable": input.nullable,
})
})
.collect::<Vec<_>>();
let outputs = plan
.outputs
.iter()
.zip(&plan.output_schema.fields)
.enumerate()
.map(|(index, (output, field))| {
serde_json::json!({
"result_field": output.result_field,
"output_name": output.output_name,
"output_field_id": 100 + index,
"output_ordinal": output.output_ordinal,
"arrow_type": canonical_input_arrow_type(field).unwrap(),
"nullable": false,
})
})
.collect::<Vec<_>>();
FunctionBinding::from_json(
&serde_json::json!({
"binding_id": "fb_blob",
"function": plan.application.function(),
"inputs": inputs,
"outputs": outputs,
"input_schema": plan.input_schema,
"output_schema": plan.output_schema,
})
.to_string(),
)
.unwrap()
}
fn full_blob_field(name: &str, nullable: bool) -> ArrowField {
ArrowField::new(
name,
DataType::Struct(lance_core::datatypes::BLOB_V2_LOGICAL_FIELDS.clone()),
nullable,
)
.with_metadata(crate::blob(name, nullable).metadata().clone())
}
fn function_binding_schema(title_nullable: bool, body_nullable: bool) -> ArrowSchema {
ArrowSchema::new(vec![
ArrowField::new("title", DataType::Utf8, title_nullable),
@@ -2581,6 +2830,37 @@ mod tests {
])
}
fn valid_function_binding_schema(
title_nullable: bool,
body_nullable: bool,
binding: &FunctionBinding,
) -> ArrowSchema {
let mut fields = function_binding_schema(title_nullable, body_nullable)
.fields()
.iter()
.map(|field| field.as_ref().clone())
.collect::<Vec<_>>();
let inputs = binding
.inputs()
.iter()
.map(|input| input.field_path.clone())
.collect::<Vec<_>>();
for output in binding.outputs() {
let index = fields
.iter()
.position(|field| field.name() == &output.output_name)
.unwrap();
fields[index] = fields[index]
.clone()
.with_metadata(function_computed_column_metadata(
binding.binding_id(),
output.output_ordinal,
&inputs,
));
}
ArrowSchema::new(fields)
}
#[test]
fn test_non_nullable_function_inputs_can_bind_to_nullable_parameters() {
let binding = FunctionBinding::from_json(include_str!(
@@ -2588,7 +2868,11 @@ mod tests {
))
.unwrap();
ensure_binding_matches_schema(&function_binding_schema(false, false), &binding).unwrap();
ensure_binding_matches_schema(
&valid_function_binding_schema(false, false, &binding),
&binding,
)
.unwrap();
}
#[test]
@@ -2601,8 +2885,11 @@ mod tests {
raw_binding["input_schema"]["fields"][0]["nullable"] = Value::Bool(false);
let binding: FunctionBinding = serde_json::from_value(raw_binding).unwrap();
let err = ensure_binding_matches_schema(&function_binding_schema(true, false), &binding)
.unwrap_err();
let err = ensure_binding_matches_schema(
&valid_function_binding_schema(true, false, &binding),
&binding,
)
.unwrap_err();
assert!(
matches!(&err, Error::InvalidInput { message }
if message.contains("input column 'title' is nullable")
@@ -2613,6 +2900,73 @@ mod tests {
);
}
#[test]
fn test_second_binding_rejects_outputs_without_reciprocal_metadata() {
let binding = FunctionBinding::from_json(include_str!(
"../../tests/fixtures/first_class_functions/v1/remote_function_binding.json"
))
.unwrap();
let schema = ArrowSchema::new_with_metadata(
function_binding_schema(true, true).fields().to_vec(),
HashMap::from([(
FUNCTION_BINDINGS_META_KEY.to_string(),
function_bindings_metadata(std::slice::from_ref(&binding)).unwrap(),
)]),
);
let err = plan_function_application(
&schema,
&named_struct_application(
r#"{"normalized_text":"secondary_text","token_count":"secondary_token_count"}"#,
),
None,
)
.unwrap_err();
assert!(
matches!(&err, Error::InvalidInput { message }
if message.contains("declaration metadata")
&& message.contains("fb_01K3TEXT")),
"{err:?}"
);
}
#[test]
fn test_persisted_nested_input_keeps_leaf_level_validation() {
let mut raw_binding: Value = serde_json::from_str(include_str!(
"../../tests/fixtures/first_class_functions/v1/remote_function_binding.json"
))
.unwrap();
raw_binding["inputs"][0]["field_path"] = Value::String("title.value".to_string());
let binding: FunctionBinding = serde_json::from_value(raw_binding).unwrap();
let title = ArrowField::new(
"title",
DataType::Struct(vec![ArrowField::new("value", DataType::Utf8, true)].into()),
true,
)
.with_metadata(HashMap::from([
(COMPUTED_COLUMN_META_KEY.to_string(), "true".to_string()),
(KIND_META_KEY.to_string(), SQL_KIND.to_string()),
(EXPRESSION_META_KEY.to_string(), "title".to_string()),
]));
let mut fields = vec![title, ArrowField::new("body", DataType::Utf8, true)];
fields.extend(binding.outputs().iter().map(|output| {
let data_type = match output.arrow_type.as_str() {
"utf8" => DataType::Utf8,
"int64" => DataType::Int64,
other => panic!("unexpected fixture output type {other}"),
};
ArrowField::new(&output.output_name, data_type, true).with_metadata(
function_computed_column_metadata(
binding.binding_id(),
output.output_ordinal,
&["title.value".into(), "body".into()],
),
)
}));
ensure_binding_matches_schema(&ArrowSchema::new(fields), &binding).unwrap();
}
#[test]
fn test_function_binding_metadata_survives_schema_round_trip() {
let binding = FunctionBinding::from_json(include_str!(
@@ -2661,9 +3015,36 @@ mod tests {
output_ordinal: 1,
} if binding_id == "fb_01K3TEXT"
));
let err = plan_function_application(&reopened, &named_struct_application("{}"), None)
let dependent_application = FunctionApplication::from_json(
r#"{
"function":{"name":"dependent","version":"fv_dependent"},
"inputs":[
{"parameter":"text","kind":"column","value":{"path":"search_text"}}
],
"output":{"kind":"scalar","arrow_type":"int64","nullable":false}
}"#,
)
.unwrap();
let err = plan_function_application(&reopened, &dependent_application, Some("dependent"))
.unwrap_err();
assert!(matches!(err, Error::NotSupported { .. }));
assert!(
matches!(&err, Error::InvalidInput { message } if message.contains("computed-on-computed"))
);
let plan = plan_function_application(
&reopened,
&named_struct_application(
r#"{"normalized_text":"secondary_text","token_count":"secondary_token_count"}"#,
),
None,
)
.unwrap();
assert_eq!(
plan.outputs
.iter()
.map(|output| output.output_name.as_str())
.collect::<Vec<_>>(),
["secondary_text", "secondary_token_count"]
);
}
#[test]
@@ -2713,6 +3094,273 @@ mod tests {
);
}
#[test]
fn test_blob_function_plans_semantic_input_and_scalar_output() {
let schema = ArrowSchema::new(vec![crate::blob("image", false)]);
let application =
blob_application(r#"{"kind":"scalar","arrow_type":"blob_v2","nullable":false}"#);
let plan = plan_function_application(&schema, &application, Some("thumbnail")).unwrap();
assert_eq!(plan.input_bindings[0].arrow_type, FUNCTION_BLOB_V2_TYPE);
let input_schema =
lance_namespace::schema::convert_json_arrow_schema(&plan.input_schema).unwrap();
assert!(input_schema.field(0).is_blob_v2());
let output_schema =
lance_namespace::schema::convert_json_arrow_schema(&plan.output_schema).unwrap();
assert!(output_schema.field(0).is_blob_v2());
}
#[test]
fn test_blob_scalar_binding_accepts_full_logical_layout() {
let input = crate::blob("image", false);
let application =
blob_application(r#"{"kind":"scalar","arrow_type":"blob_v2","nullable":false}"#);
let plan = plan_function_application(
&ArrowSchema::new(vec![input.clone()]),
&application,
Some("thumbnail"),
)
.unwrap();
let binding = binding_from_plan(&plan);
let mut metadata = full_blob_field("thumbnail", true).metadata().clone();
metadata.extend(function_computed_column_metadata(
binding.binding_id(),
0,
&["image".into()],
));
let output = full_blob_field("thumbnail", true).with_metadata(metadata);
ensure_binding_matches_schema(&ArrowSchema::new(vec![input, output]), &binding).unwrap();
}
#[test]
fn test_blob_binding_rejects_marker_on_invalid_storage_layout() {
let input = crate::blob("image", false);
let application =
blob_application(r#"{"kind":"scalar","arrow_type":"blob_v2","nullable":false}"#);
let plan = plan_function_application(
&ArrowSchema::new(vec![input.clone()]),
&application,
Some("thumbnail"),
)
.unwrap();
let binding = binding_from_plan(&plan);
let malformed = ArrowField::new("thumbnail", DataType::Int64, true)
.with_metadata(crate::blob("thumbnail", true).metadata().clone());
ensure_binding_matches_schema(&ArrowSchema::new(vec![input, malformed]), &binding)
.unwrap_err();
}
#[test]
fn test_blob_input_rejects_marker_on_invalid_storage_layout() {
let malformed = ArrowField::new("image", DataType::Int64, false)
.with_metadata(crate::blob("image", false).metadata().clone());
let application =
blob_application(r#"{"kind":"scalar","arrow_type":"blob_v2","nullable":false}"#);
plan_function_application(
&ArrowSchema::new(vec![malformed]),
&application,
Some("thumbnail"),
)
.unwrap_err();
}
#[test]
fn test_non_blob_input_does_not_require_json_round_trip() {
let json = lance_namespace::schema::arrow_schema_to_json(&ArrowSchema::new(vec![
ArrowField::new("event_time", DataType::Time64(TimeUnit::Microsecond), false),
]))
.unwrap();
assert_eq!(
canonical_input_arrow_type(&json.fields[0]).unwrap(),
"time64"
);
}
#[test]
fn test_blob_named_struct_plans_expanded_and_whole_outputs() {
let schema = ArrowSchema::new(vec![crate::blob("image", false)]);
let application = blob_application(
r#"{"kind":"named_struct","fields":[
{"name":"thumbnail","arrow_type":"blob_v2","nullable":false},
{"name":"width","arrow_type":"int32","nullable":false}
]}"#,
);
let expanded = plan_function_application(&schema, &application, None).unwrap();
let expanded_schema =
lance_namespace::schema::convert_json_arrow_schema(&expanded.output_schema).unwrap();
assert!(expanded_schema.field(0).is_blob_v2());
assert_eq!(expanded_schema.field(1).data_type(), &DataType::Int32);
let whole = plan_function_application(&schema, &application, Some("analysis")).unwrap();
let whole_schema =
lance_namespace::schema::convert_json_arrow_schema(&whole.output_schema).unwrap();
let DataType::Struct(fields) = whole_schema.field(0).data_type() else {
panic!("whole Function output should be a struct");
};
assert!(fields[0].is_blob_v2());
assert_eq!(fields[1].data_type(), &DataType::Int32);
}
#[test]
fn test_struct_blob_input_preserves_exact_schema_and_nullability() {
let payload = ArrowField::new(
"payload",
DataType::Struct(Fields::from(vec![
ArrowField::new("mime_type", DataType::Utf8, false),
ArrowField::new(
"nested",
DataType::Struct(Fields::from(vec![crate::blob("image", true)])),
true,
),
])),
true,
);
let plan = plan_function_application(
&ArrowSchema::new(vec![payload]),
&single_input_application("payload"),
Some("size"),
)
.unwrap();
let declared: JsonArrowDataType =
serde_json::from_str(&plan.input_bindings[0].arrow_type).unwrap();
let DataType::Struct(fields) =
lance_namespace::schema::convert_json_arrow_type(&declared).unwrap()
else {
panic!("expected a struct Function input")
};
assert!(fields[1].is_nullable());
let DataType::Struct(nested) = fields[1].data_type() else {
panic!("expected a recursive struct Function input")
};
assert!(nested[0].is_blob_v2());
assert!(nested[0].is_nullable());
let exact = lance_namespace::schema::convert_json_arrow_schema(&plan.input_schema).unwrap();
let DataType::Struct(fields) = exact.field(0).data_type() else {
panic!("expected exact input schema to retain the struct")
};
let DataType::Struct(nested) = fields[1].data_type() else {
panic!("expected exact input schema to retain the nested struct")
};
assert!(nested[0].is_blob_v2());
}
#[test]
fn test_recursive_blob_result_plans_one_whole_named_struct_column() {
let details_type = exact_arrow_type(ArrowField::new(
"details",
DataType::Struct(Fields::from(vec![crate::blob("image", true)])),
false,
));
let application = FunctionApplication::from_json(
&serde_json::json!({
"function": {"name": "inspect", "version": "fv_nested_blob"},
"inputs": [],
"output": {
"kind": "named_struct",
"fields": [
{"name": "mime_type", "arrow_type": "utf8", "nullable": false},
{"name": "details", "arrow_type": details_type, "nullable": false}
]
}
})
.to_string(),
)
.unwrap();
let plan = plan_function_application(&ArrowSchema::empty(), &application, Some("payload"))
.unwrap();
assert_eq!(plan.outputs.len(), 1);
assert_eq!(plan.outputs[0].result_field, WHOLE_RESULT_FIELD);
let schema =
lance_namespace::schema::convert_json_arrow_schema(&plan.output_schema).unwrap();
assert_eq!(schema.field(0).name(), "payload");
let DataType::Struct(fields) = schema.field(0).data_type() else {
panic!("whole named result must be one struct column")
};
assert_eq!(
fields.iter().map(|field| field.name()).collect::<Vec<_>>(),
["mime_type", "details"]
);
let DataType::Struct(details) = fields[1].data_type() else {
panic!("expected recursive result struct")
};
assert!(details[0].is_blob_v2());
assert!(!fields.iter().any(|field| field.name() == "payload"));
}
#[test]
fn test_blob_children_under_collections_are_rejected() {
let collections = vec![
DataType::List(Arc::new(crate::blob("item", false))),
DataType::LargeList(Arc::new(crate::blob("item", false))),
DataType::FixedSizeList(Arc::new(crate::blob("item", false)), 2),
DataType::Map(
Arc::new(ArrowField::new(
"entries",
DataType::Struct(Fields::from(vec![
ArrowField::new("key", DataType::Utf8, false),
crate::blob("value", false),
])),
false,
)),
false,
),
];
for data_type in collections {
let schema = ArrowSchema::new(vec![ArrowField::new("value", data_type, false)]);
let error = plan_function_application(
&schema,
&single_input_application("value"),
Some("size"),
)
.unwrap_err();
assert!(
error.to_string().contains("under a collection"),
"got: {error}"
);
}
}
#[test]
fn test_blob_whole_struct_binding_accepts_full_logical_layout() {
let input = crate::blob("image", false);
let application = blob_application(
r#"{"kind":"named_struct","fields":[
{"name":"thumbnail","arrow_type":"blob_v2","nullable":false},
{"name":"width","arrow_type":"int32","nullable":false}
]}"#,
);
let plan = plan_function_application(
&ArrowSchema::new(vec![input.clone()]),
&application,
Some("analysis"),
)
.unwrap();
let binding = binding_from_plan(&plan);
let output = ArrowField::new(
"analysis",
DataType::Struct(Fields::from(vec![
full_blob_field("thumbnail", false),
ArrowField::new("width", DataType::Int32, false),
])),
true,
)
.with_metadata(function_computed_column_metadata(
binding.binding_id(),
0,
&["image".into()],
));
ensure_binding_matches_schema(&ArrowSchema::new(vec![input, output]), &binding).unwrap();
}
#[test]
fn test_function_mapping_and_sibling_collisions_fail_before_request() {
let unknown = named_struct_application(r#"{"missing":"renamed"}"#);
@@ -2819,5 +3467,38 @@ mod tests {
assert!(
matches!(&err, Error::InvalidInput { message } if message.contains("computed-on-computed"))
);
let nested_title = ArrowField::new(
"title",
DataType::Struct(vec![ArrowField::new("value", DataType::Utf8, true)].into()),
true,
)
.with_metadata(HashMap::from([
(COMPUTED_COLUMN_META_KEY.to_string(), "true".to_string()),
(KIND_META_KEY.to_string(), SQL_KIND.to_string()),
(
EXPRESSION_META_KEY.to_string(),
"struct('value')".to_string(),
),
]));
let nested_schema = ArrowSchema::new(vec![nested_title, schema.field(1).as_ref().clone()]);
let nested_application = FunctionApplication::from_json(
r#"{
"function":{"name":"text_features","version":"fv_exact"},
"inputs":[
{"parameter":"title","kind":"column","value":{"path":"title.value"}},
{"parameter":"body","kind":"column","value":{"path":"body"}}
],
"output":{"kind":"named_struct","fields":[
{"name":"normalized_text","arrow_type":"utf8","nullable":false},
{"name":"token_count","arrow_type":"int64","nullable":false}
]}
}"#,
)
.unwrap();
let err = plan_function_application(&nested_schema, &nested_application, None).unwrap_err();
assert!(
matches!(&err, Error::InvalidInput { message } if message.contains("computed-on-computed"))
);
}
}
@@ -45,7 +45,11 @@ async fn local_function_catalog_operations_return_stable_not_supported() {
.get_function("normalize_score", "fv_exact")
.await
.unwrap_err();
for error in [create_error, lookup_error] {
let drop_error = connection
.drop_function("normalize_score", "fv_exact")
.await
.unwrap_err();
for error in [create_error, lookup_error, drop_error] {
assert!(matches!(
error,
Error::NotSupported { message }
@@ -78,6 +78,12 @@
"type": "utf8"
}
},
{
"arrow_type": "large_utf8",
"json": {
"type": "large_utf8"
}
},
{
"arrow_type": "binary",
"json": {
@@ -171,6 +177,21 @@
]
}
},
{
"arrow_type": "list<large_utf8>",
"json": {
"type": "list",
"fields": [
{
"name": "item",
"nullable": false,
"type": {
"type": "large_utf8"
}
}
]
}
},
{
"arrow_type": "large_list<utf8>",
"json": {
@@ -186,6 +207,21 @@
]
}
},
{
"arrow_type": "large_list<large_utf8>",
"json": {
"type": "large_list",
"fields": [
{
"name": "item",
"nullable": false,
"type": {
"type": "large_utf8"
}
}
]
}
},
{
"arrow_type": "fixed_size_list<float32, 384>",
"json": {
@@ -330,4 +366,4 @@
"timestamp[us]",
"struct<a: int32>"
]
}
}