mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-29 14:56:51 +00:00
ece54d8917ecce44b64539c44b68fc5841797f5d
3330
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Lance Release
|
ece54d8917 | Bump version: 0.41.0-beta.16 → 0.41.0-beta.17 | ||
 Brendan Clement
|
882be89778 |
docs: clarify how to contribute website documentation
The contributor guide mixes up shared documentation, Enterprise placeholders and the SDK reference. It also describes section merging that the assembler doesn't do. This replaces the overlapping instructions with guidance on where to write, how whole-page replacement works, how to preview changes and how to keep unfinished docs hidden. It explains that source changes enter a saved release edition before they appear on staging, and points SDK contributors to the existing reference guide. |
||
  
|
659ea11ac4 |
fix(embeddings): default Cohere to embed-multilingual-v3.0, the v2.0 models are retired (#4698)
Cohere retired `embed-english-v2.0`, `embed-english-light-v2.0` and `embed-multilingual-v2.0` on April 4, 2026 ([deprecations](https://docs.cohere.com/docs/deprecations), which lists `embed-english-v3.0`, `embed-multilingual-v3.0` and `embed-v4.0` as the embedding alternatives). `CohereEmbeddingFunction` still defaulted to `embed-multilingual-v2.0`, so every call without an explicit `name` fails. The default is now `embed-multilingual-v3.0`, the multilingual v3 model, and the retired models are removed from the docstring and the Cohere integration page (the page also listed `embed-english-v2.0` as the default, which did not match the code). The docs example, its generated snippet, the docs test and the slow Cohere test use the new name. This changes the default vector size: `ndims()` returns 1024 for `embed-multilingual-v3.0`, where the old default gave 768. Tables created from a schema that uses `cohere.ndims()` get 1024-dimension vectors from now on. A table that already holds 768-dimension vectors from the retired model needs a re-embed with a current model, since the old model can no longer be called. The `ndims()` entries for the v2 names are left as they are. Test: `test_cohere_default_model_is_not_a_retired_v2_model` in `python/python/tests/test_embeddings.py` checks the default name and that `ndims()` is 1024. The repository's pytest session could not be run for this change (its `conftest.py` needs a newer native module than the released wheel); the body of that test was run against the released `lancedb` 0.40.0 with this version of `cohere.py` in place, where it passes, and it returns `embed-multilingual-v2.0` / 768 without the change. ruff check and format are clean on the touched Python files. No Cohere request was made. This change was made with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Brendan Clement
|
4359688f32 |
ci: let the Vale action install its pinned CLI (#4708)
Let the pinned Vale action install Vale 3.24.0 instead of maintaining a separate download and extraction step |
||
|
Brendan Clement
|
149cf1052a |
fix(ci): bundle Vale rules to avoid private release downloads (#4707)
The prose check fails because it downloads its Vale rules from the now-private docs repository. Keep the five unchanged rules from `vale-lancedb-v0.1.0` in this repository and disable `vale sync` in CI. This also works for contributors without access to the docs repository. The update instructions record the release and archive checksum. |
||
|
Lance Release
|
45f987a29b | Bump version: 0.41.0-beta.15 → 0.41.0-beta.16 | ||
 Wyatt Alt
|
bb6aa1a147 |
feat: add a cascade option to drop_database (#4697)
The remote catalog could only drop an empty database: it always asked
the server for a restricted drop.
`DropDatabaseRequest::cascade`, `drop_database(..., cascade=True)` in
Python and `dropDatabase(name, { cascade: true })` in TypeScript ask the
server to drop everything in the database first. The name stays taken
until the server finishes cleaning up the dropped objects' storage. A
restricted drop remains the default.
|
||
 Jack YeandXuanwo
|
45dbf5eff5 |
fix: use SQL for all remote materialized view APIs (#4673)
Remote materialized-view APIs now use SQL for the full lifecycle. Create and refresh already submit SQL; this change moves describe, paginated listing, and drop onto `DESCRIBE MATERIALIZED VIEW`, `SHOW MATERIALIZED VIEWS`, and `DROP MATERIALIZED VIEW`. Async create, refresh, and drop return the existing Job handle with `job_type = "query"`. The handle has one `id`: the connection-scoped SQL query ID, also accepted by `describe_query`. Submission starts the query, the job drains its result once, and repeated waits reuse the completed result. Synchronous drop waits for catalog removal; physical cleanup proceeds in the background. Listing uses namespace-scoped keyset pagination with opaque resume tokens. Streamed and deferred SQL responses share the same path. Adds transport coverage and updates Python/TypeScript documentation for query-backed drop jobs. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Jay PokaleandXuanwo
|
f6a348ac5b |
fix(python): keep FTS scores in RRF and MRR rerankers with return_score="all" (#4677)
`RRFReranker.rerank_hybrid` and `MRRReranker.rerank_hybrid` merge the
vector and FTS results with the base `merge_results`, which keeps only
the first copy of a duplicated row. With `return_score="all"`, a row
that both searches found therefore comes back with `_score = None`, even
though the FTS search scored it. `LinearCombinationReranker` already
copies the FTS score onto such rows.
Reproduction against `lancedb` 0.40.0:
```python
import pyarrow as pa
from lancedb.rerankers import RRFReranker
vector = pa.table({"_rowid": pa.array([1, 2], pa.uint64()), "text": ["a", "b"],
"_distance": pa.array([0.1, 0.2], pa.float32())})
fts = pa.table({"_rowid": pa.array([2, 3], pa.uint64()), "text": ["b", "c"],
"_score": pa.array([5.0, 4.0], pa.float32())})
RRFReranker(return_score="all").rerank_hybrid("q", vector, fts).to_pylist()
# row 2: _distance=0.2, _score=None (expected _score=5.0)
```
`table.search(query_type="hybrid")` is not affected, because
`_combine_hybrid_results` restores the original `_distance` and `_score`
by `_rowid` after reranking. The wrong value shows up when
`rerank_hybrid` is called directly, for example from a custom reranker
that wraps RRF or MRR.
The fix uses `_merge_and_keep_scores` when `return_score="all"`, the
same pattern the model-based rerankers (`CohereReranker`,
`CrossEncoderReranker`, `JinaReranker`, `OpenaiReranker`,
`VoyageAIReranker`, ...) already use. The relevance-only path is
unchanged.
Tests: added `test_rank_fusion_return_score_all_keeps_both_scores` for
both rerankers. It fails without the fix and passes with it. I ran it
against the released 0.40.0 package with this branch's
`rerankers/{base,rrf,mrr}.py` dropped in; `rrf.py` and `mrr.py` on
`main` are identical to 0.40.0. The other reranker tests give the same
results with and without the change. `ruff format` and `ruff check` are
clean.
I used an AI assistant (Claude) while working on this; I reviewed and
tested the change.
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
 lancedb-gatefixer[bot]andXuanwo
|
ac99ab173b |
fix(python): support native blob conversion in pandas and batches (#4647)
Plain local blob v2 scans tried to open a pylance dataset before reaching LanceDB's byte-fetch path, so `to_pandas(blob_mode="bytes")` failed when the optional dependency was absent. Batch readers also had no `blob_mode` option. Route supported blob v2 pandas conversions through the native Arrow path. Add `blob_mode` to sync and async batch readers, convert payloads during iteration, and expose the converted schema without fetching the first batch. Apply conversion after reranking and retain the default descriptor behavior. Clean up scanner error punctuation and remove the misleading plain-scan suggestion. Bytes mode retains the existing `to_arrow` restrictions for legacy/list blobs, remote tables, and namespace query pushdown. Lazy pandas handles and legacy pandas scans still require pylance; sync FTS/hybrid batching remains unsupported. Validation: - UV-managed editable Python development build completed successfully. - The absent-pylance regression tests fail against the original query implementation and pass with this change. - Blob and query suites: 403 passed, 1 skipped, 1 deselected. - Remote query suite: 190 passed. - Python-tree Ruff lint/format checks and the async reader Pyright check pass. Repository-wide Ruff reports existing errors outside the Python tree. Fixes #4645 <!-- lance-gatekeeper-fix:v1 agent=fd85998f14bb0b3c92b2d050df93a615 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Dan Tasse
|
6e4f523e14 |
docs: remove logical-column from docs (#4693)
We've removed this feature from LanceDB Enterprise and no longer privilege the "lancedb:logical-column" key. So, removing that from these docs. |
||
|
Xuanwo
|
b44a0d2362 |
chore: update Lance to v15.0.0-beta.3 (#4692)
Upgrade Lance to 15.0.0-beta.3 so `Dataset::concat_data_file_parts` returns a `Send` future and downstream workers can run it on Tokio. The upstream fix is https://github.com/lance-format/lance/pull/10370. Keep the vendored Arrow validity fix; upstream Arrow sources are unchanged between beta.2 and beta.3. |
||
 dependabot[bot]
|
d44609b6f2 |
chore(deps): bump the nodejs-deps group across 1 directory with 3 updates (#4661)
Bumps the nodejs-deps group with 3 updates in the /nodejs directory: [@opentelemetry/api](https://github.com/open-telemetry/opentelemetry-js), [@opentelemetry/sdk-metrics](https://github.com/open-telemetry/opentelemetry-js) and [ts-jest](https://github.com/kulshekhar/ts-jest). Updates `@opentelemetry/api` from 1.9.1 to 1.10.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/open-telemetry/opentelemetry-js/releases">@opentelemetry/api's releases</a>.</em></p> <blockquote> <h2>api/v1.10.0</h2> <h2>1.10.0</h2> <h3>🚀 Features</h3> <ul> <li>feat(api): integrate the Logs API into <code>@opentelemetry/api</code> <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/4862">#4862</a> <a href="https://github.com/hectorhdzg"><code>@hectorhdzg</code></a></li> <li>feat(api): add imperative <code>context.attach()</code> for setting context across callback boundaries that <code>with()</code> cannot wrap. <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6845">#6845</a> <a href="https://github.com/pichlermarc"><code>@pichlermarc</code></a> <ul> <li><code>attach()</code> returns a <code>Token</code> whose <code>dispose()</code> method restores the previous context. Use <code>token.dispose()</code> to detach.</li> <li><code>attach</code> is an optional method on the <code>ContextManager</code> interface; when the active context manager does not implement it, <code>context.attach()</code> logs a warning and returns a no-op token.</li> </ul> </li> <li>feat(api): add <code>AnyValue</code> type (of type <code>unknown</code>) and extend <code>Attributes</code> type values to use <code>AnyValue</code> <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6780">#6780</a> <a href="https://github.com/trentm"><code>@trentm</code></a> <ul> <li>This is part of supporting <a href="https://github.com/open-telemetry/opentelemetry-specification/blob/main/oteps/4485-extending-attributes-to-support-complex-values.md#how">OTEP 4485</a>.</li> <li>This is a <strong>breaking change for SDK implementations</strong>, in that they may need to support more attribute types for any APIs that accept <code>Attributes</code> or <code>AnyValue</code>. Implementors should refer to the OTEP 4485 link above for "MUST / SHOULD" requirements for complex attributes handling. See also the <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/opentelemetry-core/src/common/attributes.ts">attributes-handling utilities provided by <code>@opentelemetry/core</code> to help</a>. For example, <code>cleanSimpleAttributes()</code> can be used to limit user-provided attributes to the set of value types supported in earlier <code>@opentelemetry/api</code> versions.</li> <li>This change is <strong>typically not a breaking change for users</strong> of <code>@opentelemetry/api</code>. There is one possible breaking case: If a TypeScript user was using the <code>Attributes</code> type in their own code and using <a href="https://www.typescriptlang.org/docs/handbook/2/narrowing.html">TypeScript Narrowing</a> to reduce an attribute value to an expected type (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6579#issuecomment-4345316200">example here</a>). It is expected that this usage should be rare and, likely, limited to SDK implementations, if at all. (If this impacted your code, <a href="https://github.com/open-telemetry/opentelemetry-js/issues/new">please let us know</a>.)</li> <li><strong>Note for users of attributes:</strong> Simple attributes SHOULD be used whenever possible. Instrumentations SHOULD assume that backends do not index individual properties of complex attributes, that querying or aggregating on such properties is inefficient and complicated, and that reporting complex attributes carries higher performance overhead.</li> </ul> </li> </ul> <h3>🐛 Bug Fixes</h3> <ul> <li>fix(api): cap tracestate list-members and total length in the deprecated <code>TraceState.set()</code> so it stays in step with <code>TraceState</code> from <code>@opentelemetry/core</code> <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6964">#6964</a> <a href="https://github.com/Zuhef"><code>@Zuhef</code></a></li> <li>fix: resolve the ESM build through the <code>default</code> export condition instead of <code>import</code>, so a resolver that activates neither <code>import</code> nor <code>require</code> still finds an entry point <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7132">#7132</a> <a href="https://github.com/overbalance"><code>@overbalance</code></a></li> </ul> <h3>📚 Documentation</h3> <ul> <li>docs(api): complete <code>@since</code> annotations for the public API <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7023">#7023</a> <a href="https://github.com/nabeelamjadsheikh"><code>@nabeelamjadsheikh</code></a> <ul> <li>Adds the three missing exported-type annotations and the member-level annotations for methods and properties introduced after their enclosing type.</li> <li>Corrects <code>Attributes</code>, <code>AttributeValue</code> and <code>TracerOptions</code> from <code>1.3.0</code> to <code>1.1.0</code>; all three were already importable in 1.1.0 via wildcard re-export.</li> </ul> </li> </ul> <h3>🏠 Internal</h3> <ul> <li>build: migrate the package build from <code>tsc</code> to <code>tsdown</code>, emitting dual CJS/ESM output from a single <code>dist/</code> directory <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6293">#6293</a> <a href="https://github.com/overbalance"><code>@overbalance</code></a> <ul> <li>The public entry points are unchanged: <code>@opentelemetry/api</code> and <code>@opentelemetry/api/experimental</code>. This package already declared an <code>exports</code> map, so deep imports such as <code>@opentelemetry/api/build/src/...</code> were rejected before this change and continue to be.</li> <li>The emitted files moved out of <code>build/src</code> (CJS), <code>build/esm</code> and <code>build/esnext</code> (ESM) into <code>dist/</code>, using <code>.cjs</code>/<code>.mjs</code> extensions with matching <code>.d.cts</code>/<code>.d.mts</code> declarations. The non-standard <code>esnext</code> condition has been dropped in favour of the standard <code>import</code>/<code>require</code> conditions.</li> </ul> </li> <li>perf(api): add getGlobal fast-path <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6956">#6956</a> <a href="https://github.com/legendecas"><code>@legendecas</code></a></li> <li>chore: remove the redundant <code>module</code> field from <code>package.json</code>; bundlers resolve the ESM build through the <code>exports</code> map <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7130">#7130</a> <a href="https://github.com/overbalance"><code>@overbalance</code></a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/CHANGELOG.md">@opentelemetry/api's changelog</a>.</em></p> <blockquote> <h2>1.10.0</h2> <h3>🚀 (Enhancement)</h3> <ul> <li>feat(resource): create sync resource with some attributes that resolve asynchronously <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3460">#3460</a> <a href="https://github.com/samimusallam"><code>@samimusallam</code></a></li> <li>feat: collect host id for non-cloud environments <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3575">#3575</a> <a href="https://github.com/mwear"><code>@mwear</code></a></li> <li>feat (api-logs): separate Events API into its own package <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3550">3550</a> <a href="https://github.com/martinkuba"><code>@martinkuba</code></a></li> <li>feat(sdk-metrics): apply binary search in histogram recording <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3539">#3539</a> <a href="https://github.com/legendecas"><code>@legendecas</code></a></li> <li>perf(propagator-jaeger): improve deserializeSpanContext performance <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3541">#3541</a> <a href="https://github.com/doochik"><code>@doochik</code></a></li> <li>feat: support TraceState in SamplingResult <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3530">#3530</a> <a href="https://github.com/raphael-theriault-swi"><code>@raphael-theriault-swi</code></a></li> <li>feat(sdk-trace-base): add diagnostic logging when spans are dropped <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3610">#3610</a> <a href="https://github.com/neoeinstein"><code>@neoeinstein</code></a></li> <li>feat: add unit to view instrument selection criteria <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3647">#3647</a> <a href="https://github.com/jlabatut"><code>@jlabatut</code></a></li> <li>feat(tracing): expose dropped counts for attributes, events and links on span <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3576">#3576</a> <a href="https://github.com/mohitk05"><code>@mohitk05</code></a></li> </ul> <h3>🐛 (Bug Fix)</h3> <ul> <li>fix(core): added falsy check to make otel core work with browser where webpack config had process as false or null <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3613">#3613</a> <a href="https://github.com/ravindra-dyte"><code>@ravindra-dyte</code></a></li> <li>fix(instrumentation-http): include query params in http.target <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3646">#3646</a> <a href="https://github.com/kobi-co"><code>@kobi-co</code></a></li> </ul> <h3>📚 (Refine Doc)</h3> <ul> <li>chore: update http example <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3651">#3651</a> <a href="https://github.com/JamieDanielson"><code>@JamieDanielson</code></a></li> </ul> <h3>🏠 (Internal)</h3> <ul> <li>chore(exporter-jaeger): deprecate jaeger exporter <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3585">#3585</a> <a href="https://github.com/pichlermarc"><code>@pichlermarc</code></a></li> <li>fix(sdk-metrics): fix flaky LastValueAggregator test by using fake timer <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3587">#3587</a> <a href="https://github.com/pichlermarc"><code>@pichlermarc</code></a></li> <li>fix(test): fix failing tests by preventing source-map generation <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/3642">#3642</a> <a href="https://github.com/pichlermarc"><code>@pichlermarc</code></a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/56e6b1bb890f844b8963a146780d0b9cfa8abd0d"><code>56e6b1b</code></a> chore: Release API 1.4.1, SDK 1.10.0, Experimental 0.36.0 (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3654">#3654</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/ecb5ebe86eebc5a1880041b2523adf5f6d022282"><code>ecb5ebe</code></a> fix: Minor typing issues for web sandbox (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3660">#3660</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/48ecf229eb0b468cf433a152daf414313b381edc"><code>48ecf22</code></a> feat(sdk-trace-base): add dropped attributes and events count on span (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3576">#3576</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/95bf6fc7cc105469def055f1f93446c9cd09882d"><code>95bf6fc</code></a> fix(build): add <code>example/http</code> to ignore list in <code>update-ts-configs</code> (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3657">#3657</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/3a2ca58075cc7e0eea8233c51735db5d583c11d0"><code>3a2ca58</code></a> feat: add unit to view instrument selection criteria (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3647">#3647</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/ebc8575a0c215a7b7c84866116a261ffd8aff015"><code>ebc8575</code></a> chore(jeager-exporter): deprecate jaeger exporter (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3585">#3585</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/2f715bd78dd7fb66e2aa2b76fb6e7c623deddd60"><code>2f715bd</code></a> feat: allow to add attributes from grpc metadata in the patched server (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3589">#3589</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/cbbdfd379854fcb41bd70841ae4890200a2fda09"><code>cbbdfd3</code></a> fix(sdk-node): update instrumentations once MeterProvider is initialized (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3624">#3624</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/4a4484a910201466121ed45fa12566737f6ac19d"><code>4a4484a</code></a> chore: update http example (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3651">#3651</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/494efffcf4648e3b1207c3c925c1a52bcc7f7350"><code>494efff</code></a> fix(otlp-exporter-base): fix changelog for merged retry pr (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/3650">#3650</a>)</li> <li>Additional commits viewable in <a href="https://github.com/open-telemetry/opentelemetry-js/compare/v1.9.1...v1.10.0">compare view</a></li> </ul> </details> <br /> Updates `@opentelemetry/sdk-metrics` from 2.11.0 to 2.12.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/open-telemetry/opentelemetry-js/releases">@opentelemetry/sdk-metrics's releases</a>.</em></p> <blockquote> <h2>v2.12.0</h2> <h2>2.12.0</h2> <h3>🚀 Features</h3> <ul> <li>chore(sdk-trace): add <code>StackContextManager</code> in <code>@opentelemetry/sdk-trace</code> package <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7086">#7086</a></li> </ul> <h3>🏠 Internal</h3> <ul> <li>feat(ci): support releasing from maintenance branches <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/6767">#6767</a> <a href="https://github.com/pichlermarc"><code>@pichlermarc</code></a> <ul> <li>The API documentation site is only redeployed for releases whose commit is reachable from <code>main</code>, so a maintenance release no longer overwrites it.</li> </ul> </li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/CHANGELOG.md">@opentelemetry/sdk-metrics's changelog</a>.</em></p> <blockquote> <!-- raw HTML omitted --> <h1>CHANGELOG</h1> <p>All notable changes to this project will be documented in this file.</p> <p>For API changes, see the <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/api/CHANGELOG.md">API CHANGELOG</a>. For experimental package changes, see the <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/experimental/CHANGELOG.md">experimental CHANGELOG</a>. For semantic convention package changes, see the <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/semantic-conventions/CHANGELOG.md">semconv CHANGELOG</a>. For notes on migrating to 2.x / 0.200.x see <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/doc/upgrade-to-2.x.md">the upgrade guide</a>. For notes on migrating to 3.x see <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/doc/3.x/migration-guide.md">the 3.x migration guide</a>.</p> <h2>Unreleased</h2> <h3>💥 Breaking Changes</h3> <h3>🚀 Features</h3> <h3>🐛 Bug Fixes</h3> <h3>📚 Documentation</h3> <h3>🏠 Internal</h3> <h2>3.0.0</h2> <h3>💥 Breaking Changes</h3> <ul> <li>chore(sdk-trace-node)!: remove the <code>@opentelemetry/sdk-trace-node</code> package <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7054">#7054</a> <ul> <li>The sdk-trace-node package has been replaced by the <code>@opentelemetry/sdk-trace</code> package. See the <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/doc/3.x/migration-guide.md">3.x migration guide</a> for full migration instructions.</li> </ul> </li> <li>chore(shim-opentracing)!: remove the <code>@opentelemetry/shim-opentracing</code> package <ul> <li>In the <a href="https://github.com/open-telemetry/opentelemetry-specification/releases/tag/v1.56.0">OpenTelemetry Specification v1.56.0</a> the <a href="https://redirect.github.com/open-telemetry/opentelemetry-specification/pull/4938">OpenTracing compatibility requirements were deprecated</a>. The JavaScript OpenTracing shim package will not receive any more releases after the current <a href="https://www.npmjs.com/package/@opentelemetry/shim-opentracing/v/2.12.0">2.12.0 release</a> (<a href="https://github.com/open-telemetry/opentelemetry-js/tree/v2.12.0/packages/opentelemetry-shim-opentracing/">source code for last release</a>).</li> </ul> </li> <li>chore(context-async-hooks)!: remove the unused class <code>AsyncHooksContextManager</code> <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7078">#7078</a></li> <li>feat!: migrate package builds from <code>tsc</code> to <code>tsdown</code>, emitting dual CJS/ESM output from a single <code>dist/</code> directory and declaring an <code>exports</code> map on every package <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/6293">#6293</a> <a href="https://github.com/overbalance"><code>@overbalance</code></a> <ul> <li>Importing a package by its name is unaffected in both CommonJS and ESM, as is every subpath listed in its <code>exports</code> map.</li> <li><strong>Deep imports into the build output no longer resolve.</strong> An <code>exports</code> map is an allowlist that Node.js and bundlers enforce, so specifiers such as <code>@opentelemetry/core/build/src/...</code> or <code>@opentelemetry/core/build/esm/...</code> now fail with <code>ERR_PACKAGE_PATH_NOT_EXPORTED</code>. Rewriting them to the new file layout does not help — unlisted subpaths are rejected whether or not the file exists.</li> <li>The emitted files moved out of <code>build/src</code> (CJS), <code>build/esm</code> and <code>build/esnext</code> (ESM) into <code>dist/</code>, using <code>.cjs</code>/<code>.mjs</code> extensions with matching <code>.d.cts</code>/<code>.d.mts</code> declarations.</li> <li><code><package>/package.json</code> is no longer importable for the same reason. Read a package's version from your own dependency metadata, or use <code>SDK_INFO</code> from <code>@opentelemetry/core</code> for the SDK version.</li> <li>The non-standard <code>esnext</code> entry point has been removed; tools that preferred it fall back to <code>module</code> (ESM) or <code>main</code> (CJS).</li> <li>If you depend on something that is only reachable through a deep import, please open an issue so it can be considered for the public API.</li> </ul> </li> <li>feat(core)!: remove deprecated <code>getTimeOrigin</code>, <code>otperformance</code>, <code>_globalThis</code>, and <code>unrefTimer</code> from <code>@opentelemetry/core</code> <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7053">#7053</a> <ul> <li><code>getTimeOrigin()</code> — use <code>performance.timeOrigin</code> directly.</li> <li><code>otperformance</code> — use the global <code>performance</code> object directly.</li> <li><code>_globalThis</code> — use <code>globalThis</code> directly.</li> <li><code>unrefTimer(timer)</code> — call <code>timer.unref()</code> directly in your own code.</li> </ul> </li> <li>feat(sdk-trace)!: remove deprecated <code>TracerProviderOptions.forceFlushTimeoutMillis</code> <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7057">#7057</a> <ul> <li>Pass <code>timeoutMillis</code> to <code>provider.forceFlush({ timeoutMillis })</code> instead. The default timeout is 30000ms.</li> </ul> </li> <li>feat(propagator-jaeger)!: remove <code>@opentelemetry/propagator-jaeger</code> package <a href="https://redirect.github.com/open-telemetry/opentelemetry-js/pull/7077">#7077</a> <ul> <li>The Jaeger propagator is deprecated by the OpenTelemetry specification in favour of <code>W3CTraceContextPropagator</code>. Use <code>W3CTraceContextPropagator</code> from <code>@opentelemetry/core</code> instead.</li> <li>See the <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/doc/3.x/migration-guide.md">3.x migration guide</a> for full instructions.</li> </ul> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/92b844e8aa7d9655e4b3560154e912c4c1ced64d"><code>92b844e</code></a> chore: prepare next v2.x release (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7151">#7151</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/a859aff9825ec8138cf287d08ac86401f00aad7c"><code>a859aff</code></a> fix(web-common): export getElementXPath utility (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7124">#7124</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/2bd5a73e7af7ab363f3b239f8cb1f5f56eaa3ad5"><code>2bd5a73</code></a> chore(web-common)!: deprecate <code>addSpanNetworkEvents()</code> et al instrumentation-...</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/d60f05edeac6aec8e64736c10672077fd4a766b5"><code>d60f05e</code></a> doc(sdk-trace): add description for StackContextManager (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7089">#7089</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/5bd476e6f7a22e6cf2e94a645833247284aa295d"><code>5bd476e</code></a> chore: move context manager to sdk-trace (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7086">#7086</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/554c0c6467c77e9ef1f2676f2bd20b505e228417"><code>554c0c6</code></a> chore: move sdk-trace-web utils to web-common (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7058">#7058</a>)</li> <li><a href="https://github.com/open-telemetry/opentelemetry-js/commit/788922c5780a17d5aaace76a0da458858b2ad88a"><code>788922c</code></a> ci: support maintenance branch releases (<a href="https://redirect.github.com/open-telemetry/opentelemetry-js/issues/7036">#7036</a>)</li> <li>See full diff in <a href="https://github.com/open-telemetry/opentelemetry-js/compare/v2.11.0...v2.12.0">compare view</a></li> </ul> </details> <br /> Updates `ts-jest` from 29.4.12 to 29.4.15 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/kulshekhar/ts-jest/releases">ts-jest's releases</a>.</em></p> <blockquote> <h2>v29.4.15</h2> <p>Please refer to <a href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>v29.4.14</h2> <p>Please refer to <a href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>v29.4.13</h2> <p>Please refer to <a href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">CHANGELOG.md</a> for details.</p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md">ts-jest's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/kulshekhar/ts-jest/compare/v29.4.14...v29.4.15">29.4.15</a> (2026-10-15)</h2> <h3>Bug Fixes</h3> <ul> <li><strong>compiler:</strong> resolve modules with tsconfig options regardless of cache state (<a href="https://github.com/kulshekhar/ts-jest/commit/268746c0fcaf72476cebb348c5543e5c2aac3622">268746c</a>), closes <a href="https://redirect.github.com/kulshekhar/ts-jest/issues/5216">#5216</a></li> <li>support <code>.cjs</code> config files in <code>config:migrate</code> (<a href="https://github.com/kulshekhar/ts-jest/commit/4001b4e6d28aa461e7468f09157413e113f0bd92">4001b4e</a>), closes <a href="https://redirect.github.com/kulshekhar/ts-jest/issues/4485">#4485</a></li> </ul> <h2><a href="https://github.com/kulshekhar/ts-jest/compare/v29.4.13...v29.4.14">29.4.14</a> (2026-09-25)</h2> <h3>Reverts</h3> <ul> <li>bring back prior changes of compiler util (<a href="https://redirect.github.com/kulshekhar/ts-jest/issues/5471">#5471</a>) (<a href="https://github.com/kulshekhar/ts-jest/commit/01e5a2e17ac9c940880865d6a81751832dcc4184">01e5a2e</a>), closes <a href="https://redirect.github.com/kulshekhar/ts-jest/issues/5469">#5469</a></li> </ul> <h2><a href="https://github.com/kulshekhar/ts-jest/compare/v29.4.12...v29.4.13">29.4.13</a> (2026-09-22)</h2> <h3>Features</h3> <ul> <li>support Babel 8 (<a href="https://redirect.github.com/kulshekhar/ts-jest/issues/5466">#5466</a>) (<a href="https://github.com/kulshekhar/ts-jest/commit/c3a0b57c00d8b6acfb72b020e2411752d39073e2">c3a0b57</a>), closes <a href="https://redirect.github.com/kulshekhar/ts-jest/issues/5412">#5412</a></li> </ul> <h3>Performance Improvements</h3> <ul> <li><strong>compiler:</strong> cache module resolution modes (<a href="https://redirect.github.com/kulshekhar/ts-jest/issues/5419">#5419</a>) (<a href="https://github.com/kulshekhar/ts-jest/commit/697d337c1b79a0804972a3d30c0875b02e73bf62">697d337</a>), closes <a href="https://redirect.github.com/kulshekhar/ts-jest/issues/5402">#5402</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/kulshekhar/ts-jest/commit/197d4de96002589658d8a6c2d3b9b9554d17af6b"><code>197d4de</code></a> chore(release): 29.4.15</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/ebad800d2c080cb72d7f0ea3225221b0cee066da"><code>ebad800</code></a> build(deps): update dependency node to v24.22.0</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/e5f6e986fe7c9e8115a7afe49ec9d5e8a576e93b"><code>e5f6e98</code></a> test(compiler): clean up temp dirs from module resolution tests</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/77dbb184d5895f8e315d574971fc83ab6a8a43b2"><code>77dbb18</code></a> test(compiler): make module resolution tests path-independent on Windows</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/268746c0fcaf72476cebb348c5543e5c2aac3622"><code>268746c</code></a> fix(compiler): resolve modules with tsconfig options regardless of cache state</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/15362016c490ae230c3d69f7a9399513d3600ed7"><code>1536201</code></a> build(deps): update dependency vite to ^8.3.4</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/141e39c092759ca6173095fe2dd4ebcdd3117e62"><code>141e39c</code></a> build(deps): update dependency <code>@babel/core</code> to ^8.0.7</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/6a6686e5374525db09ac67f37cf0f2d53c372f28"><code>6a6686e</code></a> build(deps-dev): bump shell-quote from 1.10.0 to 1.12.0 in /examples</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/98e6e50fe934e8e86c903719048891a3a071a2b7"><code>98e6e50</code></a> build(deps): bump proxy-addr from 2.0.7 to 2.0.8 in /website</li> <li><a href="https://github.com/kulshekhar/ts-jest/commit/d2439ff5ed9f6425bed60d0202f37771022f0526"><code>d2439ff</code></a> build(deps): bump compression from 1.8.1 to 1.8.2 in /website</li> <li>Additional commits viewable in <a href="https://github.com/kulshekhar/ts-jest/compare/v29.4.12...v29.4.15">compare view</a></li> </ul> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
0cccc9c991 |
chore(deps): bump rust-minor-patch group and napi 3.15 (#4669)
Bumps the rust-minor-patch group: serde_json 1.0.154, tokio 1.54.1, uuid 1.28.0, reqwest 0.13.5, webbrowser 1.2.5, napi-derive 3.6.13, napi-build 2.6.0, arc-swap 1.9.3, libc 0.2.191. On top of the Dependabot update: * napi is bumped to 3.15.0 instead of 3.13.0. napi-derive 3.6.13 generates code that references `bindgen_prelude::NativeBorrowScopeRelease`, which napi 3.13.0 doesn't have, so the nodejs crate failed to compile. The napi and napi-derive floors in `nodejs/Cargo.toml` are raised so they can't drift apart again. * The RUSTSEC-2026-0410 ignore from #4683 is removed from `deny.toml`. #4690 already moved the lockfile to zstd-sys 2.1.2, which fixes it. The other three zstd ignores stay until we can move to Arrow 59+. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
lancedb-gatefixer[bot]
|
fe079eb421 | fix!: restore blob SQL inserts with explicit adapter schemas (#4687) | ||
 Ozawa SongandXuanwo
|
4b373b2070 |
fix(python)!: support compat-only imports (#3973)
Read `__version__` from the native `_lancedb` module so `import lancedb` works when only `lancedb-compat` is installed. Add the type declaration and a subprocess regression test. Ruff and Linux-targeted Pyright pass. Import tests: 1 passed and 1 Linux-only test skipped on Windows. Runtime tests used the official same-version Windows native module; a fresh source build still needs CI. BREAKING CHANGE: `lancedb.__version__` now uses the native version string. Prereleases may report `0.41.0-beta.13` instead of `0.41.0b13`. Fixes #3950 --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Jay PokaleandXuanwo
|
dcac834233 |
fix(python)!: reject tuple fields with mixed element types in pydantic models (#4679)
`pydantic_to_schema` maps a `Tuple[...]` field to an Arrow list of its
**first** element type. A tuple with different element types gets the
wrong schema, and the error only surfaces when a row is written:
```python
class Pair(LanceModel):
pair: Tuple[int, str]
pydantic_to_schema(Pair) # pair: list<item: int64>
db.create_table("t", schema=Pair).add([Pair(pair=(1, "a"))])
# pyarrow.lib.ArrowInvalid: Could not convert 'a' with type str: tried to convert to int64
```
An Arrow list has a single element type, so such a tuple cannot be
stored as one. This raises a `TypeError` when the schema is built,
saying the tuple elements must share one type and pointing to a nested
`LanceModel` for a fixed set of differently typed fields. Homogeneous
tuples are unchanged: `Tuple[int, int]` and `Tuple[float, ...]` still
map to lists.
**Breaking change:** a model with a mixed-type tuple field, which used
to get a (wrong) `list<...>` schema from `pydantic_to_schema`, now
raises `TypeError` when the schema is built. Such a schema could not
hold the model's values: writing a row failed with `ArrowInvalid`. The
requirement and the nested-`LanceModel` alternative are documented in
the `pydantic_to_schema` docstring.
Tests (in `test_pydantic.py`): `TypeError` for `Tuple[int, str]` (fails
on `main`), and the list mapping for `Tuple[int, int]` and `Tuple[float,
...]`. The whole file passes; `ruff format` and `ruff check` are clean.
I used an AI assistant (Claude) while working on this; I reviewed and
tested the change.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
 陈志谦andXuanwo
|
3575f972ac |
fix(python): accept tuples and pyarrow arrays as hybrid search vectors (#4662)
The synchronous hybrid query builder rejected vector containers that the
rest of the Python API accepts:
```python
table.search(query_type="hybrid").vector((1.0, 2.0)).text("hello").to_pandas() # ValueError
table.search(query_type="hybrid").vector(pa.array([1.0, 2.0], pa.float32()))...
# ValueError: Vector query must be either a string or a vector
```
`lancedb.common.VEC` documents `list[float] | tuple[float, ...] |
np.ndarray | pa.Array`, and the same two containers work for
`table.search(vector)` and for the async hybrid API
(`AsyncQuery.nearest_to(...).nearest_to_text(...)`).
`LanceHybridQueryBuilder._validate_query` only listed `str | list |
np.ndarray`, so a tuple or a pyarrow array was refused before reaching
the query planner.
This normalises the vector exactly the way `LanceQueryBuilder.create`
already does — list/tuple to a `np.float32` array, `np.ndarray` through
`astype(float32)`, `pa.Array`/`pa.ChunkedArray` through `np.array(...,
dtype=float32)`, strings passed through — and keeps the existing
`ValueError` for anything else.
`python/python/tests/test_hybrid_query.py::test_hybrid_query_accepts_every_vector_container`
is parametrised over tuple, pyarrow array and pyarrow chunked array, and
asserts the returned `_rowid`s match the `list` form: 3 failed on `main`
(`ValueError` from `query.py:2452`), 3 passed with the change.
Neighbouring suites (`test_hybrid_query.py`, `test_query.py`,
`test_rerankers.py`) are 220 passed / 9 skipped, and `test_table.py`,
`test_remote_db.py`, `test_fts.py`, `test_index.py`,
`test_permutation.py` are 724 passed / 5 skipped. `ruff` at the version
pinned in the repository's pre-commit config (0.9.9) reports no findings
and `ruff format --check` is clean.
One source file (`python/python/lancedb/query.py`), no behaviour change
for containers that already worked.
Prepared with AI assistance; every line was verified by running the
cases above.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
lancedb-gatefixer[bot]andXuanwo
|
2cd635d41c |
fix: correct remote materialized view refresh results and options (#4672)
Remote materialized-view refreshes drained the SQL result and then fabricated `mode=rebuild`, `rows_written=<total view rows>`, and `source_version=0`. This made incremental refreshes report incorrect statistics. Make these three result fields optional across Rust, Python, and TypeScript, and leave them unset for remote SQL refreshes. Remove the post-refresh row count request. Local refreshes continue to populate all metrics, and `version` remains the view version observed after completion. Remote per-operation statistics remain unavailable until the SQL job adapter can retrieve them. Expose the forced-rebuild option through the Rust builder, synchronous/asynchronous Python refresh and job methods, and TypeScript refresh. Remote `full=true` sends `WITH (full = true)`; the default lets the server choose its refresh mode. Local source-version pinning and remote job cancellation are preserved. Local refresh regression tests respect the `experimental-functions` build gate, while remote refresh coverage also runs in default builds. Validation: - Rust: 20 materialized-view tests and 21 remote SQL tests passed with `remote,experimental-functions`; `cargo fmt --all` passed. - Python: rebuilt the extension in the uv-managed environment. The default build passed 23 tests with 27 feature-dependent skips. The opt-in build passed 61 tests with 3 expected default-build-only skips, covering the materialized-view suite and 14 async query-serialization cases from current main. Python Ruff lint and formatting checks passed. - TypeScript: default and opt-in native builds and TypeScript compilation passed. The default build passed 5 tests with 10 expected feature-dependent skips; the opt-in build passed all 14 materialized-view tests with the default-build-only test skipped. Lint/format checks and documentation generation passed. - Repository-wide `ruff check .` reports 48 pre-existing violations in unchanged `docs/web-tests` files; `ruff check python` passes. Fixes #4670 <!-- lance-gatekeeper-fix:v1 agent=ece43ec4b88a48d1789f60005fa3ecfc generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
c167b99fa0 |
fix: read legacy blob columns through blob APIs (#4649)
Legacy blob v1 columns were rejected by LanceDB's blob validation and omitted from `blob_columns()`, even though the pinned Lance reader already supports them. Accept both blob versions in read validation and discovery so `fetch_blobs`, `fetch_blob_files`, and `fetch_blob_ranges` can read legacy payloads through the existing native reader. Keep list-nested blobs excluded from row-based reads and document how to fetch legacy blobs using explicit row IDs. Regression coverage checks sync and async APIs, independently written Lance datasets, ordering and duplicates across fragments, null and empty blobs, lazy seeking, ranges, and deleted rows. Fixes #4644 Validation: - Both new Python regressions failed on the original implementation with the reported legacy-column rejection and empty discovery result, then passed with the fix. - Python blob suite: 202 passed, 1 skipped. - Remote blob tests: 29 passed. - Focused Rust blob unit tests with `remote`: 13 passed. - Rust blob integration suite with `remote`: 59 passed, including legacy storage 2.0 and 2.1. - Rust compile and Clippy checks with `remote`, tests, and examples passed. - Python package Ruff lint and format checks passed; Rust formatting passed. - Repository-wide `ruff check .` reports 49 existing errors in unchanged `docs/web-tests` examples; the Python package is clean. <!-- lance-gatekeeper-fix:v1 agent=02b7edcbfa8ff0e7fad4615159249b3e generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
83b7275137 |
fix(remote): make HTTP timeout errors actionable (#4650)
Remote `create_table` reader uploads can exceed the read deadline while waiting for response headers, but reqwest's main error message only says "error sending request". Make HTTP and retry errors explicitly identify timeouts and name the relevant `client_config.timeout_config` settings and environment variables, preserving typed sources, request metadata, and retry classification. Document that the read timeout covers waiting for headers during upload. This implements the issue's diagnostic option: creation still uses one streaming request, so large or slow readers may require a longer `client_config.timeout_config.read_timeout`. Validation: - 48 focused Rust client, error, and retry tests passed. Regression coverage holds an upload open until its read timeout and also exercises TLS connection timeouts, with and without retries. - 8 Python create-table and HTTP/retry error tests passed in the uv-managed environment after rebuilding the extension, including reader creation with read and overall timeouts. - `cargo check --quiet --features remote --tests --examples` passed. - `cargo clippy --quiet --features remote --tests --examples` passed. - Rust formatting and Python package Ruff lint/format checks passed. - Repository-wide `ruff check .` reports 49 existing findings under `docs/web-tests`; those files are unchanged. Fixes #4646 <!-- lance-gatekeeper-fix:v1 agent=ce1c040fa581f1748166ecb686d9c9c0 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
ad1212fb32 |
chore: update Lance to v15.0.0-beta.2 (#4690)
**Blocked on the corrected Lance release:** https://github.com/lance-format/lance/pull/10370 restores the `Send` contract required by downstream Function workers. Refresh this upgrade with that release before merging. Upgrade the Rust, Java, and Python Lance dependencies to https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.2. This brings in bounded vector-index consolidation without retraining (https://github.com/lance-format/lance/pull/10335) and deletion filtering for index entries whose fragment coverage was pruned (https://github.com/lance-format/lance/pull/10325). Synchronize the vendored `lance-arrow` with the release and retain its offset-safe validity patch. The existing `nullable_struct_merge` regression still panics with the unpatched release, so removing that patch would regress nullable nested queries. Refresh both lockfiles, including the `zstd-sys` version required by the new Lance encoding dependency. Update the custom memtable bitmap test adapter to train on row addresses, matching the new bitmap contract; remove its obsolete row-ID conversion. Preserve the Blob APIs' existing missing-row error messages when newer Lance rejects a missing fragment earlier in the read path. Retain upstream attribution in vendored sources while satisfying the repository license-header check. Validation of the compatibility fixes: all 58 Rust Blob integration tests and 219 Python Blob tests pass (one Python test skipped). Strict workspace Clippy, formatting, and diff checks pass. |
||
|
lancedb-gatefixer[bot]andXuanwo
|
7fca4d666a |
fix(python): preserve async query search options during serialization (#4659)
Async `to_query_object()` dropped enabled `fast_search` on every query type and `postfilter` on plain and full-text queries. Re-executing a serialized fast search could therefore return unindexed rows. Copy enabled `fast_search` into the Python `Query` and serialize enabled plain/FTS postfilter from the Rust request's prefilter setting. Preserve the existing unset defaults: `None` for these options, with the vector query's existing `False` postfilter default. Validation: - Added 12 serialization cases across plain, FTS, and vector queries, covering both enabled and unset options, plus two FTS execution round-trip cases. Before the fix, nine cases failed, including returning an unindexed row after serialization. - Rebuilt the Python extension using the uv-managed environment. - Query tests and existing FTS fast-search tests: 192 passed, 1 slow test deselected. - Rust formatting and Python package lint/format checks pass. Repository-wide `ruff check .` reports 49 pre-existing errors in unchanged `docs/web-tests` files. Fixes #4658 <!-- lance-gatekeeper-fix:v1 agent=f828b24871a2dc928668be2de4f85078 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
1edfa6077b |
fix(python): allow phrase queries without a column (#4653)
Quoted async full-text queries without `columns` carry an unset phrase column, but converting them to Python unwrapped it and raised `PanicException`. Python also required `PhraseQuery.column`, preventing callers from constructing the equivalent query object. Make the Python column optional and preserve `Option<String>` in both Rust conversion directions, leaving column selection to query planning. Regression tests cover quoted strings and structured queries with omitted, `None`, and explicit columns, query reuse, JSON serialization, slop, and document granularity. Validation: - Reproduced both missing-column failures before the fix; explicit-column controls passed. - Rebuilt the extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`. - `cd python && uv run --no-sync --extra tests pytest python/tests/test_fts.py python/tests/test_query.py::test_query_serialization_async -q`: 85 passed. - `cargo fmt --all -- --check` and `ruff check python` pass; touched Python files pass formatting checks. - `cargo clippy --quiet -p lancedb-python --lib --features remote` passes. - Root-wide `ruff check .` reports 49 existing findings in unchanged documentation examples. Fixes #4651 <!-- lance-gatekeeper-fix:v1 agent=62d36fad70c8717236283b8e97dd6b30 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Avanish GargandXuanwo
|
8887c82789 |
fix(python)!: raise on bool(Expr) instead of silently dropping predicates (#4682)
Python's `and` / `or` / `not` and chained comparisons call `bool()` on
an `Expr`. `Expr` had no
`__bool__`, so that was always `True` and the filter silently lost a
predicate: `where(lit(2) <
col("id") < lit(5))` returned ids 0..4 (lower bound dropped),
`(col("id") > 7) and (col("g") == 0)`
returned all even ids, and `(a) or (b)` kept only `a` (see the issue for
the repro and output).
Changes (`python/python/lancedb/expr.py`,
`python/python/lancedb/query.py`):
- `Expr.__bool__` raises `TypeError("The truth value of an Expr is
ambiguous. ...")` with a hint to
use `&`, `|`, `~` with parentheses and `Expr.isin()` for membership.
`TypeError` rather than
`ValueError` because the object does not support truthiness at all
(polars `Expr` and SQLAlchemy
clauses raise `TypeError`; the codebase raises `TypeError` for
unsupported `lit()` values).
pandas, numpy and pyarrow `Expression` use `ValueError`.
- `LanceHybridQueryBuilder` tested `if self._where:`, which would have
started raising for every
`.where(Expr)` on a hybrid query. It is now `isinstance(self._where,
Expr) or self._where`, so
an Expr is always applied and an empty SQL string is still skipped as
before.
- Notes section in the `Expr` docstring.
Not changed: no other code in `python/python/lancedb` tests the
truthiness of a filter or of an
`Expr` (checked `where`/`filter`/`predicate`/`condition` handling in
`query.py`, `table.py`,
`remote/`, `merge.py`, `functions.py`, `pydantic.py`: they use `is None`
/ `isinstance`). Cross-SDK:
`nodejs` has no `Expr` class on `main` yet, and JavaScript cannot
overload `&&` / `||` / `!`
either, so there is nothing equivalent to port; the Rust core is not
affected.
Behaviour changes to be aware of: code that already used `and` / `or` /
`not` / chained
comparisons on an `Expr` (which returned wrong rows) now raises
`TypeError`. `x in [expr, ...]`,
`any([...])` and `all([...])` on Exprs, and `==` between two `Query`
objects whose `filter` are
different Expr objects (pydantic compares the fields), now raise instead
of returning a meaningless
answer; `Expr` stays unhashable and `isinstance` / `is None` checks are
unaffected.
Tests (`python/python/tests/test_expr.py`): `TestExprTruthiness`
(`bool`, `and`, `or`, `not`,
chained comparison, `if`, list membership, `any`/`all` raise with a
helpful message; `&`, `|`, `~`
still combine) and `TestExprWhereBuilders` (plain, vector, FTS, hybrid
and async `.where(Expr)`
return the right rows, hybrid with an SQL string and with an empty
string is unchanged, chained
comparison is rejected before any query runs). With the source change
stashed 11 of the new tests
fail; with only `__bool__` added and the hybrid change reverted the
hybrid test fails with the
new `TypeError`, so it guards the second change. `test_expr.py`: 144
passed (123 before this change).
Verification caveat: `main`'s Python layer is newer than the newest
published wheel, so the tests
were run against the 0.41.0b14 native module
(`python/python/lancedb/expr.py` and `query.py` are
identical to the wheel); the full suite (`pytest python/tests
--ignore=python/tests/docs -m "not slow and not s3_test"`)
has the identical 25 failures before and after (wheel/main skew in
remote options, remote db,
index and table tests; none in expr/query tests): 2604 passed before,
2625 passed after (the 21
new tests). `ruff format --check python` and `ruff check python` are
clean. Not run: Rust build, Node tests, remote (cloud)
tests.
Fixes #4681
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
Xuanwo
|
8a4effa06a |
fix: gate experimental local functions behind an opt-in feature (#4688)
Published SDK builds currently expose local SQL computed columns and materialized-view execution that are not ready for release. Add the opt-in `experimental-functions` Cargo feature to the Rust core and forward it through the Python and Node bindings; published packages leave it disabled. Local declarations and refreshes now return `NotSupported` before mutating tables or spawning jobs, including prepared materialized-view creation. Remote Cloud/Enterprise client operations, Python UDF packaging, ordinary eager `add_columns`, and access to existing data remain available. Stored metadata and write protections stay active, and shared server commit helpers remain usable. CI covers both feature configurations. Existing local execution tests run with the feature enabled; default-build regressions cover early rejection, unchanged table versions, existing metadata protections, and normal eager transforms. API and source-build documentation explain the opt-in. Validation: Rust library tests pass with the feature disabled (1,869 tests) and enabled (1,947 tests), plus three default-build integration regressions. Python and TypeScript bindings were rebuilt and tested in both configurations, including mocked remote requests; Rust formatting, Python package lint, TypeScript lint/build, and generated docs were checked. Repository-wide Ruff still reports 48 pre-existing errors in unchanged `docs/web-tests` files. |
||
|
lancedb-gatefixer[bot]
|
dcbf5ce66a |
fix: preserve null blob descriptor fields in SQL (#4571)
SQL descriptor access returns NULL for every field of a NULL Blob v2 value, so `payload.size = 0` matches only empty blobs and `payload.size IS NULL` matches NULL blobs. DataFusion 54.1.0 extracts struct children without combining their validity with the parent struct. Lance descriptors keep nullness in the parent and can store zero-filled children beneath NULL blobs. The adapter derives the descriptor schema from Lance's default projection, marks descriptor children nullable, and masks them before SQL evaluation. Schema derivation requires no query request, and the mask handles struct and list nesting while preserving existing child nulls and value buffers without fetching blob bytes. Filter classification follows the referenced struct field path. Blob predicates stay above the mask; scalar siblings such as `extra.note` retain exact filter pushdown even when `extra.clip` is a blob. **Merge hold:** as requested in the [review discussion](https://github.com/lancedb/lancedb/pull/4571#issuecomment-6253443077), blob SQL INSERT compatibility remains pending a separate write-mode design. DataFusion uses the provider's single schema for reads and INSERT coercion, so the descriptor schema still rejects blob write structs. This follow-up does not restore those INSERTs; the merge should wait until the write contract is settled. Validation: - Reproduced the original NULL blob returning `kind = 0` before enabling the mask. - Reproduced both the extra remote query and lost nested sibling pushdown before the follow-up fixes. - All 441 DataFusion adapter tests passed after merging main at `e6789a41`, including SQL projections, filters, mixed predicates, LIMIT, COUNT, schema agreement, sliced arrays, nested structs, list variants, and EXPLAIN assertions for scalar sibling pushdown. The blob fixtures use the current four-field write format, and NULL-blob filtering is also covered with row IDs and row addresses enabled. - Remote constructor regression passed: one describe request and no query request, with nullable descriptor fields preserved. - `cargo fmt --all` - `cargo clippy --locked --profile ci -p lancedb --features remote --tests --examples -- -D warnings` Fixes #4569 <!-- lance-gatekeeper-fix:v1 agent=2ca7389efd48d11a8debe9111abeb00e generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
Lance Release
|
37b46f6b4f | Bump version: 0.41.0-beta.14 → 0.41.0-beta.15 | ||
|
Wyatt Alt
|
fe07c25a0a |
fix: refuse Function-bound writes by column instead of by table (#4686)
A table with a Function binding rejected every update and merge_insert, whatever columns the write named: updating an unrelated column, or a merge that only inserts rows, failed with "not supported on a table with registered Function bindings". Expression computed columns on the same table accept those writes. Both paths now use the per-column guard that schema evolution already uses. An update is checked on its SET columns. A merge is checked on the columns a matched row would take from the source, every source column but the join keys; an insert-only merge is an append, and its new rows get their Function outputs from the next refresh. Writes to a binding's inputs or outputs are still refused, now naming the column. |
||
|
lancedb-gatefixer[bot]andXuanwo
|
e3dce2cbad |
fix(rust): prevent oversized SQ remap panics (#3861)
## Summary - preflight IVF-SQ and IVF-HNSW-SQ storage segments touched by compaction - preserve unknown legacy fragment coverage as conservatively affected - cover both fixed-size vector and multivector SQ storage dimensions - reject only affected partitions whose fixed-size-list child exceeds Arrow's `u32` take-index limit - return an actionable `InvalidInput` error before Arrow can panic ## Root cause Arrow 58 builds fixed-size-list child indices with `u32`. For a large SQ partition, `partition_rows * vector_dimension` can exceed `u32::MAX`; the child range then wraps inside `take_fixed_size_list`, producing the reported `ArrayData::slice` assertion during index remapping. The preflight now reads raw segment metadata, mirrors Lance's fragment-bitmap selection (including `None` as affected/unknown), opens only affected vector segments, and checks their actual storage quantizer and partition sizes. It derives the stored dimension for both `FixedSizeList` vectors and flattened `List<FixedSizeList>` multivectors. Stable-row-ID datasets, deferred remapping, custom remappers, unaffected segments, non-SQ indexes, safe partitions, and no-op compactions are unchanged. Because the upstream Arrow kernel still has the limit, an affected index must be recreated with default IVF partitioning (or enough partitions to stay below the reported maximum) before compaction. This changes the failure from a process panic to a targeted recovery error; it does not silently alter index structure. ## Validation - `cargo fmt --all -- --check` - `cargo test --quiet --features aws,remote -p lancedb table::optimize::tests --lib` - `cargo test --quiet --features remote -p lancedb table::create_index::tests::test_create_index_ivf_hnsw_sq` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples -- -A clippy::redundant-pub-crate` (the allowance is for pre-existing Rust 1.97 warnings in remote-table code) Fixes #2866 <!-- lance-gatekeeper-fix:v1 agent=ba2f07b1ee6f9e68a0de1961c164f70e generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
a9fb16ac44 |
fix: return cosine-scaled ANN distances (#3852)
## Summary - convert unrefined cosine ANN scores from normalized squared-L2 scale to the public cosine scale - translate public distance-range bounds to the internal ANN scale before execution - add regression coverage for indexed distances and range filtering while leaving refined, exact, and flat paths unchanged ## Root cause Cosine PQ, SQ, and RQ quantizers normalize vectors and search with squared L2 internally. The unrefined native query plan returned those internal scores directly, making the values twice the public cosine distance for unit-normalized embeddings. ## Validation - `cargo test --quiet --features remote --tests` - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo fmt --all -- --check` - rebuilt the Python extension and reran the issue reproduction through the Python API Fixes #3530 <!-- lance-gatekeeper-fix:v1 agent=fa75cad2d924c92a97215fcab4e21b14 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
3352bf49df |
fix(python): expose optimize compaction options (#3870)
## Summary - expose typed compaction options on synchronous and asynchronous table optimization - translate validated Python options into native Lance compaction settings - document the row-based steady-state limit and first-pass/source-budget caveats - carry the Lance beta.15 fragment-exclusion option through the typed Python API ## Root cause The native Python optimize binding always constructed default compaction settings, and the public AsyncTable API offered no way to lower the fragment row target. Wide-row fragments therefore remained compaction candidates on every later optimize call, repeatedly rewriting the accumulated table. ## Limitations This is an explicit row-based mitigation, not byte-native task sizing or a hard memory cap. Callers must derive target_rows_per_fragment from expected row size. Existing oversized fragments are rewritten whole on the first pass; max_bytes_per_file limits output file size rather than compaction memory; and source budgets admit whole planned tasks, so a budget below the first task performs no compaction. Splitting tasks by a byte budget or returning a planner error requires a Lance-side change. ## Validation - cargo check --release -p lancedb-python - cargo clippy --quiet -p lancedb-python - cargo fmt --all --check - ruff format --check python tests - ruff check python tests - uv run --no-sync pytest python/tests/test_table.py -k optimize_compaction -q (16 passed) Fixes #2325 <!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
311ded06d0 |
fix(ci): acknowledge Arrow-pinned zstd advisories (#4683)
RustSec published four `zstd` advisories after #3819 merged, causing the `deny` job to fail on `main`. `arrow-ipc 58.x` pins `zstd ^0.13`, while the patched releases require `zstd >= 0.14.3`; moving to Arrow 59+ requires a coordinated DataFusion and Lance upgrade. This records temporary exceptions with their reachability rationale. LanceDB uses the `zstd::bulk` API through `arrow-ipc`; the affected `with_context`, seekable, and streaming-decoder APIs are not used, and the `zstd-sys` advisory is wasm32-only while wasm32 is not a shipped target. Validation: `cargo-deny 0.20.2` reports `advisories ok, bans ok, licenses ok, sources ok`. |
||
 Drew
|
436c209580 | docs: document Blob V2 in the multimodal guide (#4623) | ||
|
lancedb-gatefixer[bot]andXuanwo
|
7952ee2a71 |
fix(python): cover FP16 GPU index queries (#3819)
Keep FP16 vectors through IVF-PQ index creation and search in the synchronous and asynchronous quickstarts. Add native and conditional CUDA regression coverage using a float64 query against an explicitly typed FP16 column. This is test and documentation-example coverage for dtype preservation already present in Lance; it does not change production code. Related #2120. Validation on `bbad20940632b21a9ee85bdcb46b4cc19dbd65e3` with Python 3.12.12, Lance 15.0.0-beta.1, PyArrow 24.0.0, and Torch 2.14.1: - The native FP16 parameter, Arrow FP16 ingestion test, and both quickstarts passed: 4 passed, 1 CUDA-only skip. - The two affected test files passed: 386 passed, 1 CUDA-only skip (`-m "not slow and not s3_test"`). - Ruff 0.9.9 lint and formatting checks passed for the Python package. Real CUDA execution has not been performed. Torch is installed, but `torch.cuda.is_available()` is false; `test_create_f16_table[cuda]` skips with `CUDA not available`. The native CPU result does not establish that the original GPU failure is resolved. Issue #2120 remains open for verification on an actual CUDA device. Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 KarthikSandXuanwo
|
180eaaf68e |
fix!: reject unsupported remote connection and table options (#4486)
Fixes #4399. Remote connections currently discard local-only settings, including table write options that determine storage format. The synchronous Python client also binds the fourth positional `create_table` argument to `on_bad_vectors`, so an intended overwrite is sent as a create. This change aligns the remote Python signature with `DBConnection.create_table`, preserves the legacy remote timeout keywords, and raises `ValueError` for other unknown connection keywords. The shared Rust client rejects unsupported connection, create, and open options before sending HTTP requests; open validation also runs before a table-cache hit. Python reports these as `NotImplementedError`, including explicit `enable_v2_manifest_paths=False`. Supported Enterprise connection-level storage options remain available. This implements option 2 from the issue: explicit rejection. It deliberately does not add unverified server write parameters. Calls that previously supplied ignored options must omit them to use server-managed storage and caching. Database URI prefixes are already preserved on current main and are unchanged here. Regression tests cover Python sync/async, TypeScript create/open calls, and the Rust remote client, including empty-table creation, overwrite mode, and cached table handles. Python API documentation describes the behavior and corrected positional order. Reconciled with main at `0ea932da`, preserving upstream blob and error-handling tests and updating the Python test that previously expected open options to be ignored. Validation on the merged source: - Rebuilt the Python native extension; 304 tests passed across remote options, database, remote database, and session suites. - Rust remote database suite: 135 passed. Rebuilt the Node native extension; TypeScript remote suite: 62 passed. TypeScript compilation and documentation generation passed. - Full Python SDK lint and formatting pass (159 files); full TypeScript lint and formatting pass (66 files). Rust formatting and `git diff --check` pass. - Repository-wide Python lint/format reports existing issues in unchanged `docs/web-tests` files (49 lint findings and 11 formatting files). Environment: Ubuntu under WSL2, Linux x86-64, CPython 3.12.13, Rust 1.97.0, and Node.js 22.16.0. Locked dependencies and the repository `ci` build profile. Remote tests use HTTP fixtures; this does not claim validation against the original ARM64 single-node deployment. Documentation review follow-up: added a draft Enterprise migration note in `docs/web/basics/sdk.mdx`. It explains that `db://` connections reject local session/cache and per-table storage/write options, while supported connection-level Enterprise storage options remain available. Following the repository's release conventions, the note stays in an MDX comment below the migration list until the behavior is released. Vale 3.24.0 with the pinned LanceDB style package reports no findings; `git diff --check` passes. This follow-up changes documentation only. BREAKING CHANGE: Remote connections now reject unsupported local session/cache and per-table storage/write options instead of silently ignoring them. Callers must omit these options and use server-managed storage and caching; supported Enterprise connection-level storage options remain available. Integration with current main preserves its synchronous API signatures, including the remote `create_table(mode=None)` default, and the table compatibility defaults. The positional-order repair is already on main via #4608; this PR retains the distinct unsupported-option rejection behavior. CI follow-up: the inherited IVF search-effort test occasionally measured recall 0.375 against its unchanged 0.5 threshold after random index training. A matching local reproducer observed this once in 30 builds. The fixture now places the eight exact neighbors together, preserving the 16-partition setup, two writes, all probe/precedence assertions, and the recall threshold. The adjusted fixture returned recall 1.0 in 100 builds; the full query and remote suites passed (102 tests). This stabilizes test data and does not change search behavior or promise production recall. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
KarthikSandXuanwo
|
982b2a1229 |
fix(python): restore sync remote connection and table API parity (#4473)
Fixes #4400. Sync remote connections fail on `uri`, `len(db)`, and `name in db`, and remote tables lack `namespace`, `id`, and `replace_field_metadata`. Share the paginated connection helpers and table identifier contract through the base classes, and delegate remote metadata replacement to the existing async implementation. Unsupported remote session and read-consistency properties raise explicit `NotImplementedError`s. Reconciled with main at `3dd57978`, including the synchronous method signatures from #4498. The local session fix has already landed in #4433, so this PR preserves its supplied-session-or-None contract and adds no Rust/PyO3 session access. Namespace connections retain their writable properties. Existing custom `Table` subclasses remain instantiable: `namespace` defaults to a fresh empty list, and deprecated `replace_field_metadata` delegates to `update_field_metadata` with `replace=True` while retaining its `None` return. Native and remote overrides are preserved. Regression tests instantiate an old-style subclass through normal ABC class creation and verify identifiers, independent namespace lists, deprecation, and exact metadata delegation. Historical validation at `03c43f7b` before the latest main integration and subclass compatibility repair, Ubuntu x86-64, Python 3.12.13: - Rebuilt the native extension with the repository `ci` profile and locked Python dependencies. - 273 tests passed across `test_sync_api_parity.py`, `test_db.py`, `test_remote_db.py`, and `test_session.py`. - 84 namespace and catalog compatibility tests passed (357 tests total). - Python SDK lint and formatting pass (158 files); changed-file checks and `git diff --check` pass. - Repository-wide lint/format checks report existing issues in unchanged `docs/web-tests` files (49 lint findings and 11 formatting files). These are outside this patch. Remote tests use the existing HTTP fixtures through the rebuilt native client; the original aarch64 server deployment was not recreated. Validation after merging main `3dd57978` and the compatibility repair (macOS ARM64, uv-managed Python 3.12.12, rebuilt editable native extension): - 60 focused parity, signature, and error-contract tests passed. - 381 database, remote, session, namespace, catalog, and sync parity tests passed; 2 platform-dependent tests skipped. - The two new subclass regressions fail against the pre-repair implementation with the expected abstract-member instantiation error, then pass with the repair. - Python SDK Ruff lint/format passes (167 files); repository-wide findings remain confined to unchanged docs/web-tests (49 lint findings, 11 formatting files). --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
KarthikSandXuanwo
|
3dd579785e |
fix(python)!: align synchronous table method signatures (#4498)
Closes #4396. Calls written against `Table` can fail or bind different arguments when used with `RemoteTable`. Align the affected sync signatures, positional order, and defaults; preserve renamed remote keywords as deprecated aliases. Share legacy vector configuration and tokenizer inference, forward index tuning and timeout options, and accept `fast_search` consistently. Preserve current upstream conversion behavior: local `to_polars(batch_size=...)` returns a LazyFrame; remote tables accept the same argument signature but raise the documented unsupported-operation error and direct callers to `table.search().to_polars()`. The abstract `Table` method retains its unsupported-operation error, with the implementation confined to `LanceTable`. Regression coverage checks argument binding, deprecated aliases, option forwarding, native behavior through both wrappers, HTTP index requests, and remote conversion rejection. The merge preserves current tokenizer defaults, remote query defaults, HNSW-PQ support, and unsupported-operation handling. Runtime validation at `c4fb01f2` (Linux x86_64, uv-managed Python 3.12.15): - Focused Ruff lint and formatting passed for all four changed Python files. - Rebuilt and installed the current native extension successfully with the repository's `ci` Cargo profile. - Seven regression suites passed: `test_errors.py`, `test_table_signatures.py`, `test_remote_db.py`, `test_table.py`, `test_index.py`, `test_fts.py`, and `test_query.py`: **874 passed, 1 deselected**, with slow and S3 tests excluded. - CI caught an abstract-base conversion regression in the initial merge. The existing error-contract test reproduced it locally (1 failed, 7 passed); restoring the implementation to `LanceTable` fixes it while preserving signature parity. - Full-root Ruff reports 49 lint errors and 11 files needing formatting, all in unchanged upstream `docs/web-tests` files. The four changed Python files pass both checks. - Validated against upstream `197403e8`, using Lance 15.0.0-beta.1 and LanceDB 0.41.0-beta.14 bindings. Historical validation before this merge: 618 tests passed and one was deselected. That result does not establish correctness of the current merge. The original client-side reproduction produced nine argument-binding errors without contacting a server; the initial regression suite failed 32 cases before the fix and passed 39 after it. Documentation review follow-up: restored the `batch_size` parameter description on the abstract `Table.to_polars` method so it appears in the public API reference. This adds six docstring lines and changes no runtime behavior. After this edit, all 47 focused error-contract and signature tests pass; changed-file Ruff lint/format and `git diff --check` pass. Repository-wide Ruff still reports the same 49 findings and 11 formatting files in unchanged `docs/web-tests`. The full documentation site and seven-suite runtime run were not repeated for this documentation-only follow-up. The reported ARM64 production-server deployment has not been recreated. HTTP integration tests use the repository's local mock server. BREAKING CHANGE: Remote synchronous table methods now use the positional argument order and defaults documented by `Table`. Migrate calls using the former remote-only positional order to named arguments, particularly `create_index` and `search`; pass `index_type` by keyword to `create_scalar_index`. Remote scalar and vector index creation now default to `replace=True`; specify `replace=False` to retain rejection of an existing index. Renamed remote keywords remain supported as deprecated aliases, as documented above. --------- Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
66357275bb |
fix: inherit Lance defaults for IVF RQ quantization (#4585)
`IvfRq()` forced `num_bits=1` through Python's Rust binding, overriding Lance's five-bit default. The shared native index creation path also substituted one bit when the Rust or Node builder left the value unset. Make Python's bit count optional and forward only explicit values. Use `RQBuildParams::default()` for an unset native bit count, and omit unset values when dispatching to pylance for accelerator builds. Update the Python, Rust, Node, and vector-index documentation. New indexes with an unset bit count now inherit Lance's default, currently five bits, across Python, Rust, and Node. Five-bit vector codes use roughly five times the storage of one-bit codes, plus other index overhead. This also changes reader compatibility: indexes created with `num_bits >= 2` use a newer on-disk layout that some older LanceDB versions cannot read. Upgrade readers to versions that support multi-bit IVF_RQ, or explicitly set `num_bits=1` (for example, `IvfRq(num_bits=1)` in Python) when sharing indexes with readers limited to one-bit IVF_RQ. Validation: - Reproduced six regression failures before the fix: one-bit default metadata, rejected `None`, and incorrect remote/accelerator forwarding. - After merging `main` through `b762ec22`, rebuilt both native bindings against Lance 15.0.0-beta.1. Python validation passed with 106 tests and 15 optional Ollama skips. Coverage includes the full index suite, omitted and explicit bit counts, reading Lance-created IVF_RQ indexes, accelerator forwarding, CUDA-unavailability handling, timeout validation, remote requests, and table/namespace listing behavior. - Rust formatting and Python package Ruff lint/format checks pass. Node `pnpm lint`, `pnpm build` (including TypeScript checks), `pnpm run docs`, the full connection suite, and the existing IVF_RQ regression pass. Generated documentation adds no further changes. The merge preserves upstream's explicit RaBitQ heading anchor alongside the `IvfRq()` example and five-bit default/reader-compatibility guidance. Repository-wide `ruff check .` still reports 49 existing errors in unchanged documentation examples under `docs/web-tests`; root formatting also finds 11 unchanged documentation example files. The Python package checks pass. Fixes #4582 <!-- lance-gatekeeper-fix:v1 agent=6ef9406f0142b1ab66d53d9d56098223 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 
|
22fc2a2087 |
fix(python)!: put OpenCLIP models in eval mode before embedding (#4536)
`OpenClipEmbeddings` loads its model with
`open_clip.create_model_and_transforms()`, which hands the model back in
training mode (the open_clip README calls `model.eval()` right after it
for that reason). `OpenClipEmbeddings` never switches it out of training
mode, so layers that behave differently during training stay active
while we embed: stochastic depth in the timm-based image towers
(ViTamin, ConvNeXt, MobileCLIP2-L-14), dropout in the Hugging Face text
towers (`xlm-roberta-*`, `roberta-ViT-B-32`, `nllb-clip-*`), and batch
norm in the ResNet towers loaded with non-OpenAI weights. The
`torch.no_grad()` blocks turn off autograd but don't change how these
layers behave.
The default `ViT-B-32` has none of those layers, which is probably why
nobody has run into this. With dropout or stochastic depth active, the
same input gives a different vector on every call, so the vector stored
at ingestion and the one computed for an identical query don't match.
With batch norm in training mode, an image's vector depends on the other
images in its batch:
```python
import numpy as np
from PIL import Image
from lancedb.embeddings import get_registry
func = get_registry().get("open-clip").create(name="ViTamin-S", pretrained="datacomp1b")
image = Image.new("RGB", (256, 256), "red")
a = func.generate_image_embedding(image)
b = func.generate_image_embedding(image)
print(func._model.training, np.array_equal(a, b))
# before: True False
# after: False True
```
The fix calls `model.eval()` once after loading, same as the SigLIP,
ColPali and ImageBind embedding functions already do.
The new test swaps in a stand-in for the `open_clip` module, so it runs
without torch or a model download, and checks that the model handed back
by `create_model_and_transforms` ends up in eval mode.
BREAKING CHANGE: Embeddings change for OpenCLIP models with
training-sensitive layers, including ViTamin, ConvNeXt, MobileCLIP2,
Hugging Face RoBERTa/XLM-RoBERTa/NLLB text towers, and ResNet models
loaded with non-OpenAI weights. For tables embedded with affected
models, regenerate the stored embeddings in eval mode and rebuild their
vector indexes before querying with the new embeddings. Rebuilding an
index alone does not regenerate embeddings. The default ViT-B-32 model
is unaffected.
Co-authored-by: devjoinedthechat <235875317+devjoinedthechat@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
lancedb-gatefixer[bot]andXuanwo
|
7ed03e1c40 |
fix(python): honor bad-vector policies for pandas NaNs (#4503)
Pandas converts NaN vector components to Arrow nulls. Both Python's NaN mask and Rust's default append validator ignored null children, so `add()` silently stored incomplete vectors despite `on_bad_vectors="error"`. Reject null components in the existing Rust ingestion validator, and include null components in Python's bad-vector detection and filling. `drop` removes affected rows, `fill` repairs missing components while preserving valid ones, and `null` replaces affected vectors. Explicit null components in Arrow and list inputs follow the same policies. Rust allows whole null vectors and retains the explicit `NaNVectorBehavior::Keep` opt-out. The ingestion guide and synchronous/asynchronous `create_table()` option docs describe null components, all four policies, and filling that preserves valid components. Validation: - Reproduced the default pandas append failure with the regression test before changing the Rust validator. - Rebuilt the native Python extension from the merged sources. The Python table and utility suites passed: 481 passed, 1 skipped. Coverage includes create/append/overwrite, pandas/rows/Arrow inputs, every bad-vector policy, float16/32/64, sliced and chunked arrays, all-null components, and hidden children of null lists. - All 43 focused Rust tests passed: 7 validator tests and 36 ingestion tests. - Rust Clippy with `remote`, tests, and examples passed. Ruff lint passes for the Python package, formatting checks pass for modified Python files, and `cargo fmt --all` is clean. - Documentation follow-up: all 56 focused vector-policy tests passed; Python package lint and creation-docstring formatting checks passed. Repository-wide Ruff reports 49 existing errors in untouched documentation examples. Create/overwrite retain their existing `RuntimeError` wrapping for errors raised while consuming lazy Arrow readers. Fixes #4435 <!-- lance-gatekeeper-fix:v1 agent=b5108e27d28ef613a78c1e4354c03c48 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Joaquin Hui GomezandXuanwo
|
256db55a51 |
fix(python)!: align RemoteDBConnection.create_table parameter order (#4608)
`RemoteDBConnection.create_table` orders its parameters `(name, data,
schema, on_bad_vectors, fill_value, mode, exist_ok, ...)`, while
`DBConnection`, `LanceDBConnection` and `LanceNamespaceDBConnection` all
use `(name, data, schema, mode, exist_ok, on_bad_vectors, fill_value,
...)`. Since `RemoteDBConnection` subclasses `DBConnection`, identical
positional calls bind to different parameters: switching a working
script from `lancedb.connect(path)` to `lancedb.connect("db://...")`
silently turns `mode="overwrite"` into `on_bad_vectors="overwrite"` and
`exist_ok=True` into `fill_value=True`, and nothing client-side
validates either. `inspect.signature(...).bind(None, "t", data, None,
"overwrite", True)` shows it deterministically, no cloud needed.
This reorders the remote signature to match the other three; every
parameter keeps its own existing default (including the remote's
`mode=None`, which its body relies on), and the body already forwards by
keyword so nothing else changes. The new test asserts positional parity
with `DBConnection` through `embedding_functions` and fails on `main`.
Blast radius, stated plainly: only remote callers passing four or more
positional arguments are affected — positions 4–7 move; positions 1–3
and 8 are unchanged and keyword callers are unaffected. No in-repo
caller passes more than two positionals. Anyone relying on the current
order was relying on behaviour that contradicts the documented base
contract and diverges from local connections. Not added here: the
`storage_options`/`data_storage_version`/`enable_v2_manifest_paths`
parameters the remote signature still lacks.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
|
||
|
lancedb-gatefixer[bot]
|
e6789a4118 |
fix: support system columns in DataFusion table adapters (#4667)
DataFusion plans SQL against the adapter's declared schema, which omitted `_rowid` and `_rowaddr` even though Lance can project both. SQL therefore rejected these columns before reaching the scan. Add `BaseTableAdapter::with_system_columns` to opt into either column using Lance's nullable `UInt64` fields. Scans without a projection now select the full declared column list so their output matches the provider schema. Default adapters retain data-only `SELECT *` and remain compatible with `INSERT INTO ... SELECT *`; opted-in adapters include the requested system columns in `SELECT *`. Regression tests compare SQL results with native queries and cover projection order, system-only projections, row-ID predicates, scans without a projection, duplicate/unsupported names, and default SQL inserts. Validation: - DataFusion unit suite: 436 tests passed with `cargo test --locked --quiet --profile ci -p lancedb --features remote --lib table::datafusion`. - `cargo clippy --locked --quiet --profile ci -p lancedb --features remote --tests --examples -- -D warnings` passed. - The new builder's documentation example passed its doctest. - `cargo fmt --all -- --check` and `git diff --check` passed. Fixes #4666 <!-- lance-gatekeeper-fix:v1 agent=58d267206580227b3820df65af15b0a7 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
Jack YeandLance Release
|
197403e891 |
fix: honor downstream Lance dependency overrides (#4671)
Consumers that provide a local Lance checkout cannot resolve the dependency graph because the SDK directly selects a vendored `lance-arrow`, producing two path packages with the same name and version. The Rust core now declares its upstream Git dependency separately so the consuming workspace can override that source consistently. Python continues inheriting the vendored workspace dependency, keeping the sliced-struct validity fix in Maturin source distributions. Standalone SDK builds still select that same fix through the existing root patch. Adds a downstream consumer CI regression that resolves a local Lance override and checks that exactly one canonical `lance-arrow` package is selected. The Lance version updater also updates the core declaration in stable, preview and local modes. Public APIs and query behavior are unchanged. The workspace declaration lint excludes path dependencies because the Python binding intentionally uses the bundled source. Duplicate and unused external declarations remain errors; advisory, license, banned-package and source checks continue covering the full dependency graph. --------- Co-authored-by: Lance Release <lance-dev@lancedb.com> |
||
|
Lance Release
|
de0c5edc7d | Bump version: 0.41.0-beta.13 → 0.41.0-beta.14 | ||
|
Jack Ye
|
dcfef7c5ab |
feat(python): open lazy blob files from SQL descriptors (#4668)
SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.
### Python experience
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.
The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.
For an async table:
```python
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
```
### Execution
The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.
Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.
Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.
Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
|
||
|
Jack Ye
|
9af7757053 |
fix: execute remote materialized view creation through SQL (#4665)
Remote materialized-view creation currently expects a REST response containing a registry job ID, while refresh already submits SQL. Route both through the SQL query backend and preserve Python's synchronous and asynchronous return types. ### Query to Job mapping `SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through the same backend as `execute_query_async()`. Once submission succeeds, it retains the returned `Query` and starts one client-side task that consumes its result reader to completion. `Job::from_query_task()` wraps that task, its cached outcome, a single ID and a cancellation callback. Python exposes this as `Job[T]` or `AsyncJob[T]`. The existing `job_type` property maps SQL-backed handles to the new `"query"` kind. The shared `id` contains that query's UUID string, so Python callers can inspect it with `db.describe_query(job.id)` on the submitting connection. No separate `query_id`, backend enum or `type()` API is added. | Handle | `job_type` | `id` | Available lookup | | --- | --- | --- | --- | | SQL-backed | `"query"`, known at submission | Connection-scoped query UUID string | `describe_query(job.id)` | | Registry-backed | Existing server-reported kind, after `status()` or `refresh()` | Registry job ID string | Registry job APIs, descriptions and events | | Local | `None` | `None` | The current in-process handle | | Job concept | SQL-backed behavior | | --- | --- | | Submission | Await initial SQL submission before returning the handle. Submission errors propagate immediately. | | `status()` | Tracks the adapter task: running during result consumption/finalization, finished after both succeed, failed if either fails, or cancelled after cancellation. | | `wait()` | Returns the operation's typed terminal result. One task consumes the query reader once; repeated waits reuse its cached outcome without resubmitting SQL or reopening the reader. | | `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported cancellation leaves the task running; an accepted request stops the task. Cancelling a finished operation is a no-op. | | Registry details | Query-backed handles expose no registry description, event history, pause/resume or reattachment after a client restart. `refresh()` updates cached lifecycle state. | The Job tracks command completion rather than exposing SQL batches. A wait timeout does not cancel the query. Query IDs remain scoped to the submitting connection and cannot be used with registry APIs or as durable resume tokens. Python `describe_query()` accepts both UUID objects and the string `id` exposed by query-backed jobs. Rust, Python and TypeScript expose the same classification through their existing job-type accessors. ### Materialized-view operations - **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace and optional auto-refresh settings. After command completion, resolve the job to `None`. Synchronous creation waits on the job, then opens and returns the materialized view. - **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same adapter. After command completion, read the refreshed table's version and row count to construct the existing typed refresh result. Synchronous refresh waits on the same path. Remote creation requires the SQL service. Execution placement remains a server decision. REST endpoints are unchanged. The job guide documents the ID/job-type contract and query-backed limitations. Regression coverage exercises SQL submission, namespaces, refresh options, job-type/ID preservation through typed result mapping, repeated waits, failures and cancellation rejection. No local builds or tests were run. |
||
|
Jack Ye
|
24c2669a50 |
feat(sql): follow adaptive streaming and polling responses (#4664)
SQL execution now starts with one Flight call. The server can stream
rows immediately or return a polling continuation, which the SDK follows
through the existing query handle and reader APIs. The same behavior
applies with and without parameters.
Both initial transports share direct-stream lifecycle handling. Handoffs
are explicitly negotiated; existing servers can continue returning
ordinary streams. Add coverage for streaming, deferred results,
cancellation, malformed decisions, and legacy responses.
```mermaid
flowchart TD
A[execute_query or execute_query_async] --> B{Parameters supplied?}
B -->|No| C[DoGet: SQL command ticket]
B -->|Yes| D[DoExchange: SQL and Arrow parameter row]
C --> E[Server chooses execution path]
D --> E
E -->|Stream| F[Initial response: streaming decision and Arrow batches]
E -->|Defer| G[Initial response: PollInfo continuation]
G --> H[SDK follows PollFlightInfo]
H -->|Still running| H
H -->|Result endpoints available| I[SDK fetches tickets with DoGet]
F --> J[Query reader yields Arrow batches]
I --> J
```
`execute_query` opens the reader; `execute_query_async` returns its
query handle. The caller does not select the transport or execution
path.
|
||
|
Jack Ye
|
6b6778e684 |
feat: infer lazy blob reader Function inputs (#4663)
Infer Blob V2 input access from the Python Function signature. `video:
BlobReader` requests a lazy, seekable reader; `video: bytes` requests
the complete payload. Users do not need a separate `blob_inputs`
registration option.
## Python experience
For a video table joined to a table of frame timestamps, the Function
can pass its reader directly to a media decoder:
```python
import io
import av
from lancedb import BlobReader, udf
@udf(pip=["av", "Pillow"])
def extract_frame(video: BlobReader, timestamp_s: float) -> bytes:
with av.open(video) as container:
stream = container.streams.video[0]
container.seek(int(timestamp_s / stream.time_base), stream=stream)
for frame in container.decode(stream):
if frame.time is not None and frame.time >= timestamp_s:
output = io.BytesIO()
frame.to_image().save(output, format="JPEG")
return output.getvalue()
raise ValueError("No frame at or after the requested timestamp")
```
After registering `extract_frame`, the query keeps the usual Function
call:
```sql
SELECT f.video_id, f.timestamp_s,
extract_frame(v.video, f.timestamp_s) AS image
FROM videos AS v
JOIN frames AS f ON v.id = f.video_id
```
The returned JPEG can feed a subsequent image-model step. The reader
provides `read`, `readinto`, `seek`, `tell`, `size()`, and
`read_range(offset, length)`; range reads leave the sequential cursor
unchanged. Each row invocation owns its reader, which closes after the
call. Decoder seeks can request the parts of the MP4 needed to locate
and decode the requested frame; the amount read depends on the file
layout and codec.
Functions that need the whole value can keep a bytes signature:
```python
@udf
def video_length(video: bytes) -> int:
return len(video)
```
## Execution contract
- Signature inference maps `BlobReader` to a Blob V2 field and records
`blob_access="reader"`. `bytes` keeps its binary Arrow type and records
`blob_access="bytes"`; it can also bind to a Blob V2 source. Explicit
schemas support the same choice, including a Blob V2 field annotated as
`bytes`.
- Reader execution carries a compact blob reference into the Function
worker. Payload reads happen when Python calls the reader, with requests
split into at most 1 MiB ranges. The full video does not need to enter
the Arrow input batch just to invoke the Function.
- Bytes execution materializes the full source before the Function runs
and remains subject to eager-input memory admission. Calling
`reader.read()` without a size also materializes the remaining contents
in Python; lazy access depends on how the Function consumes it.
- Input access metadata survives Python/Rust Function serialization and
registration. Packaged Functions import the worker-provided reader
implementation. This PR defines the SDK contract; execution requires
corresponding server support.
- This changes blob inputs. The example still returns ordinary image
bytes; it does not add streamed outputs or run an image model.
Adds coverage for signature inference, explicit schemas, nullable
readers, signature round trips, seeking, bounded range reads, buffering,
and invalid responses.
|
||
 Shiduo LiandXuanwo
|
0ea932dafd |
fix(rust): compile jobs without remote feature (#4103)
## What Move typed job decode error construction behind a feature-aware helper so builds without the `remote` feature do not reference `Error::Http`. Local decode failures continue to surface as `Error::Runtime`, while remote builds retain HTTP errors with request IDs. Add a regression test for local typed-result decode failures. Closes #4096. ## Testing - `cargo fmt --all -- --check` - `cargo check -p lancedb --no-default-features` - `cargo test -p lancedb --no-default-features job::tests::local_typed_result_decode_errors_are_runtime_errors` Co-authored-by: Xuanwo <github@xuanwo.io> |