Compare commits

...

73 Commits

Author SHA1 Message Date
Lance Release
590c0c1e77 Bump version: 0.30.2 → 0.31.0-beta.0 2026-04-03 08:45:29 +00:00
LanceDB Robot
382ecd65e3 chore: update lance dependency to v5.0.0-beta.4 (#3234)
## Summary
- Update Rust Lance workspace dependencies to `v5.0.0-beta.4` using
`ci/set_lance_version.py` (including lockfile refresh).
- Update Java `lance-core` dependency property to `5.0.0-beta.4` in
`java/pom.xml`.

## Verification
- `cargo clippy --workspace --tests --all-features -- -D warnings`
- `cargo fmt --all`

## Triggering tag
- https://github.com/lance-format/lance/releases/tag/v5.0.0-beta.4
2026-04-03 01:33:36 -07:00
Jack Ye
e26b22bcca refactor!: consolidate namespace related naming and enterprise integration (#3205)
1. Refactored every client (Rust core, Python, Node/TypeScript) so
“namespace” usage is explicit: code now keeps namespace paths
(namespace_path) separate from namespace clients (namespace_client).
Connections propagate the client, table creation routes through it, and
managed versioning defaults are resolved from namespace metadata. Python
gained LanceNamespaceDBConnection/async counterparts, and the
namespace-focused tests were rewritten to match the clarified API
surface.
2. Synchronized the workspace with Lance 5.0.0-beta.3 (see
https://github.com/lance-format/lance/pull/6186 for the upstream
namespace refactor), updating Cargo/uv lockfiles and ensuring all
bindings align with the new namespace semantics.
3. Added a namespace-backed code path to lancedb.connect() via new
keyword arguments (namespace_client_impl, namespace_client_properties,
plus the existing pushdown-ops flag). When those kwargs are supplied,
connect() delegates to connect_namespace, so users can opt into
namespace clients without changing APIs. (The async helper will gain
parity in a later change)
2026-04-03 00:09:03 -07:00
Lance Release
3ba46135a5 Bump version: 0.27.2-beta.2 → 0.27.2 2026-03-31 21:26:04 +00:00
Lance Release
f903d07887 Bump version: 0.27.2-beta.1 → 0.27.2-beta.2 2026-03-31 21:25:36 +00:00
Lance Release
5d550124bd Bump version: 0.30.2-beta.2 → 0.30.2 2026-03-31 21:25:04 +00:00
Lance Release
c57cb310a2 Bump version: 0.30.2-beta.1 → 0.30.2-beta.2 2026-03-31 21:25:02 +00:00
Dan Tasse
97754f5123 fix: change _client reference to _conn (#3188)
This code previously referenced `self._client`, which does not exist.
This change makes it correctly call `self._conn.close()`
2026-03-31 13:29:17 -07:00
Pratik Dey
7b1c063848 feat(python): add type-safe expression builder API (#3150)
Introduces col(), lit(), func(), and Expr class as alternatives to raw
SQL strings in .where() and .select(). Expressions are backed by
DataFusion's Expr AST and serialized to SQL for remote table compat.

Resolves: 
- https://github.com/lancedb/lancedb/issues/3044 (python api's)
- https://github.com/lancedb/lancedb/issues/3043 (support for filter)
- https://github.com/lancedb/lancedb/issues/3045 (support for
projection)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 11:32:49 -07:00
Will Jones
c7f189f27b chore: upgrade lance to stable 4.0.0 (#3207)
Bumps all lance-* workspace dependencies from `4.0.0-rc.3` (git source)
to the stable `4.0.0` release on crates.io, removing the `git`/`tag`
overrides.

No code changes were required — compiles and passes clippy cleanly.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 17:05:45 -07:00
yaommen
a0a2942ad5 fix: respect max_batch_length for Rust vector and hybrid queries (#3172)
Fixes #1540

I could not reproduce this on current `main` from Python, but I could
still reproduce it from the Rust SDK.

Python no longer reproduces because the current Python vector/hybrid
query paths re-chunk results into a `pyarrow.Table` before returning
batches. Rust still reproduced because `max_batch_length` was passed
into planning/scanning, but vector search could still emit larger
`RecordBatch`es later in execution (for example after KNN / TopK), so it
was not enforced on the final Rust output stream.

This PR enforces `max_batch_length` on the final Rust query output
stream and adds Rust regression coverage.

Before the fix, the Rust repro produced:
`num_batches=2, max_batch=8192, min_batch=1808, all_le_100=false`

After the fix, the same repro produces batches `<= 100`.

## Runnable Rust repro

Before this fix, current `main` could still return batches like `[8192,
1808]` here even with `max_batch_length = 100`:

```rust
use std::sync::Arc;

use arrow_array::{
    types::Float32Type, FixedSizeListArray, RecordBatch, RecordBatchReader, StringArray,
};
use arrow_schema::{DataType, Field, Schema};
use futures::TryStreamExt;
use lancedb::query::{ExecutableQuery, QueryBase, QueryExecutionOptions};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let tmp = tempfile::tempdir()?;
    let uri = tmp.path().to_str().unwrap();

    let rows = 10_000;
    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::Utf8, false),
        Field::new(
            "vector",
            DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 4),
            false,
        ),
    ]));

    let ids = StringArray::from_iter_values((0..rows).map(|i| format!("row-{i}")));
    let vectors = FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
        (0..rows).map(|i| Some(vec![Some(i as f32), Some(1.0), Some(2.0), Some(3.0)])),
        4,
    );
    let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(ids), Arc::new(vectors)])?;
    let reader: Box<dyn RecordBatchReader + Send> = Box::new(
        arrow_array::RecordBatchIterator::new(vec![Ok(batch)].into_iter(), schema),
    );

    let db = lancedb::connect(uri).execute().await?;
    let table = db.create_table("test", reader).execute().await?;

    let mut opts = QueryExecutionOptions::default();
    opts.max_batch_length = 100;

    let mut stream = table
        .query()
        .nearest_to(vec![0.0, 1.0, 2.0, 3.0])?
        .limit(rows)
        .execute_with_options(opts)
        .await?;

    let mut sizes = Vec::new();
    while let Some(batch) = stream.try_next().await? {
        sizes.push(batch.num_rows());
    }

    println!("{sizes:?}");
    Ok(())
}
```

Signed-off-by: yaommen <myanstu@163.com>
2026-03-30 15:43:58 -07:00
Will Jones
e3d53dd185 fix(python): skip test_url_retrieve_downloads_image when PIL not installed (#3208)
The test added in #3190 unconditionally imports `PIL`, which is an
optional dependency. This causes CI failures in environments where
Pillow isn't installed (`ModuleNotFoundError: No module named 'PIL'`).

Use `pytest.importorskip` to skip gracefully when Pillow is unavailable.

Fixes CI failure on main.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 14:48:49 -07:00
Will Jones
66804e99fc fix(python): use correct exception types in namespace tests (#3206)
## Summary
- Namespace tests expected `RuntimeError` for table-not-found and
namespace-not-empty cases, but `lance_namespace` raises
`TableNotFoundError` and `NamespaceNotEmptyError` which inherit from
`Exception`, not `RuntimeError`.
- Updated `pytest.raises` to use the correct exception types.

## Test plan
- [x] CI passes on `test_namespace.py`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 12:55:54 -07:00
lennylxx
9f85d4c639 fix(embeddings): add missing urllib.request import in url_retrieve (#3190)
url_retrieve() calls urllib.request.urlopen() but only urllib.error was
imported, causing AttributeError for any HTTP URL input. This affects
open-clip, siglip, and jinaai embedding functions when processing image
URLs.

The bug has existed since the embeddings API refactor (#580) but was
masked because most users pass local file paths or bytes rather than
HTTP URLs.
2026-03-30 12:03:44 -07:00
Vedant Madane
1ba19d728e feat(node): support Float16, Float64, and Uint8 vector queries (#3193)
Fixes #2716

## Summary

Add support for querying with Float16Array, Float64Array, and Uint8Array
vectors in the Node.js SDK, eliminating precision loss from the previous
\Float32Array.from()\ conversion.

## Implementation

Follows @wjones127's [5-step
plan](https://github.com/lancedb/lancedb/issues/2716#issuecomment-3447750543):

### Rust (\
odejs/src/query.rs\)

1. \ytes_to_arrow_array(data: Uint8Array, dtype: String)\ helper that:
   - Creates an Arrow \Buffer\ from the raw bytes
   - Wraps it in a typed \ScalarBuffer<T>\ based on the dtype enum
   - Constructs a \PrimitiveArray\ and returns \Arc<dyn Array>\
2. \
earest_to_raw(data, dtype)\ and \dd_query_vector_raw(data, dtype)\ NAPI
methods that pass the type-erased array to the core \
earest_to\/\dd_query_vector\ which already accept \impl
IntoQueryVector\ for \Arc<dyn Array>\

### TypeScript (\
odejs/lancedb/query.ts\, \rrow.ts\)

3. Extended \IntoVector\ type to include \Uint8Array\ (and
\Float16Array\ via runtime check for Node 22+)
4. \xtractVectorBuffer()\ helper detects non-Float32 typed arrays and
extracts their underlying byte buffer + dtype string
5. \
earestTo()\ and \ddQueryVector()\ route through the raw NAPI path when
the input is Float16/Float64/Uint8

### Backward compatibility

Existing \Float32Array\ and \
umber[]\ inputs are unchanged -- they still use the original \
earest_to(Float32Array)\ NAPI method. The new raw path is only used when
a non-Float32 typed array is detected.

## Usage

\\\	ypescript
// Float16Array (Node 22+) -- no precision loss
const f16vec = new Float16Array([0.1, 0.2, 0.3]);
const results = await
table.query().nearestTo(f16vec).limit(10).toArray();

// Float64Array -- no precision loss
const f64vec = new Float64Array([0.1, 0.2, 0.3]);
const results = await
table.query().nearestTo(f64vec).limit(10).toArray();

// Uint8Array (binary embeddings)
const u8vec = new Uint8Array([1, 0, 1, 1, 0]);
const results = await
table.query().nearestTo(u8vec).limit(10).toArray();

// Existing usage unchanged
const results = await table.query().nearestTo([0.1, 0.2,
0.3]).limit(10).toArray();
\\\

## Note on dependencies

The Rust side uses \rrow_array\, \rrow_buffer\, and \half\ crates.
These should already be in the dependency tree via \lancedb\ core, but
\Cargo.toml\ may need explicit entries for \half\ and the arrow
sub-crates in the nodejs workspace.

---------

Signed-off-by: Vedant Madane <6527493+VedantMadane@users.noreply.github.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
2026-03-30 11:15:35 -07:00
lif
4c44587af0 fix: table.add(mode='overwrite') infers vector column types (#3184)
Fixes #3183

## Summary

When `table.add(mode='overwrite')` is called, PyArrow infers input data
types (e.g. `list<double>`) which differ from the original table schema
(e.g. `fixed_size_list<float32>`). Previously, overwrite mode bypassed
`cast_to_table_schema()` entirely, so the inferred types replaced the
original schema, breaking vector search.

This fix builds a merged target schema for overwrite: columns present in
the existing table schema keep their original types, while columns
unique to the input pass through as-is. This way
`cast_to_table_schema()` is applied unconditionally, preserving vector
column types without blocking schema evolution.

## Changes

- `rust/lancedb/src/table/add_data.rs`: For overwrite mode, construct a
target schema by matching input columns against the existing table
schema, then cast. Non-overwrite (append) path is unchanged.
- Added `test_add_overwrite_preserves_vector_type` test that creates a
table with `fixed_size_list<float32>`, overwrites with `list<double>`
input, and asserts the original type is preserved.

## Test Plan

- `cargo test --features remote -p lancedb -- test_add_overwrite` — all
4 overwrite tests pass
- Full suite: 454 passed, 2 failed (pre-existing `remote::retry` flakes
unrelated to this change)

---------

Signed-off-by: majiayu000 <1835304752@qq.com>
2026-03-30 10:57:33 -07:00
lennylxx
1d1cafb59c fix(python): don't assign dict.update() return value in _sanitize_data (#3198)
dict.update() mutates in place and returns None. Assigning its result
caused with_metadata(None) to strip all schema metadata when embedding
metadata was merged during create_table with embedding_functions.
2026-03-30 10:15:45 -07:00
aikido-autofix[bot]
4714598155 ci: mitigate template injection attack in build_linux_wheel (#3195)
This patch mitigates template injection vulnerabilities in GitHub
Workflows by replacing direct references with an environment variable.

Aikido used AI to generate this PR.

High confidence: Aikido has a robust set of benchmarks for similar
fixes, and they are proven to be effective.

Co-authored-by: aikido-autofix[bot] <119856028+aikido-autofix[bot]@users.noreply.github.com>
2026-03-30 09:29:24 -07:00
lennylxx
74f457a0f2 fix(rust): handle Mutex lock poisoning gracefully across codebase (#3196)
Replace ~30 production `lock().unwrap()` calls that would cascade-panic
on a poisoned Mutex. Functions returning `Result` now propagate the
poison as an error via `?` (leveraging the existing `From<PoisonError>`
impl). Functions without a `Result` return recover via
`unwrap_or_else(|e| e.into_inner())`, which is safe because the guarded
data (counters, caches, RNG state) remains logically valid after a
panic.
2026-03-30 09:25:18 -07:00
Dan Tasse
cca6a7c989 fix: raise instead of return ValueError (#3189)
These couple of cases used to return ValueError; should raise it
instead.
2026-03-25 18:49:29 -07:00
Lance Release
ad96489114 Bump version: 0.27.2-beta.0 → 0.27.2-beta.1 2026-03-25 16:22:09 +00:00
Lance Release
76429730c0 Bump version: 0.30.2-beta.0 → 0.30.2-beta.1 2026-03-25 16:21:26 +00:00
Weston Pace
874b74dd3c feat: update lance dependency to v4.0.0-rc.3 (#3187)
## Summary
- Update all lance workspace dependencies from v3.0.1 (crates.io) to
v4.0.0-rc.3 (git tag)
- Pin AWS SDK transitive dependencies to versions compatible with Rust
1.91.0 MSRV

## Test plan
- [x] `cargo check --features remote --tests --examples` passes
- [x] `cargo clippy --features remote --tests --examples` passes
- [x] Python bindings compile (`cargo check -p lancedb-python`)
- [ ] CI passes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-25 09:20:29 -07:00
Lance Release
61de47f3a5 Bump version: 0.27.1 → 0.27.2-beta.0 2026-03-25 03:23:28 +00:00
Lance Release
f4d613565e Bump version: 0.30.1 → 0.30.2-beta.0 2026-03-25 03:22:55 +00:00
Wyatt Alt
410ab9b6fe Revert "feat: allow passing azure client/tenant ID through remote SDK" (#3185)
Reverts lancedb/lancedb#3102
2026-03-24 20:17:40 -07:00
Will Jones
1d6e00b902 feat: progress bar for add() (#3067)
## Summary

Adds progress reporting for `table.add()` so users can track large write
operations. The progress callback is available in Rust, Python (sync and
async), and through the PyO3 bindings.

### Usage

Pass `progress=True` to get an automatic tqdm bar:

```python
table.add(data, progress=True)
# 100%|██████████| 1000000/1000000 [00:12<00:00, 82345 rows/s, 45.2 MB/s | 4/4 workers]
```

Or pass a tqdm bar for more control:

```python
from tqdm import tqdm

with tqdm(unit=" rows") as pbar:
    table.add(data, progress=pbar)
```

Or use a callback for custom progress handling:

```python
def on_progress(p):
    print(f"{p['output_rows']}/{p['total_rows']} rows, "
          f"{p['active_tasks']}/{p['total_tasks']} workers, "
          f"done={p['done']}")

table.add(data, progress=on_progress)
```

In Rust:

```rust
table.add(data)
    .progress(|p| println!("{}/{:?} rows", p.output_rows(), p.total_rows()))
    .execute()
    .await?;
```

### Details

- `WriteProgress` struct in Rust with getters for `elapsed`,
`output_rows`, `output_bytes`, `total_rows`, `active_tasks`,
`total_tasks`, and `done`. Fields are private behind getters so new
fields can be added without breaking changes.
- `WriteProgressTracker` tracks progress across parallel write tasks
using a mutex for row/byte counts and atomics for active task counts.
- Active task tracking uses an RAII guard pattern (`ActiveTaskGuard`)
that increments on creation and decrements on drop.
- For remote writes, `output_bytes` reflects IPC wire bytes rather than
in-memory Arrow size. For local writes it uses in-memory Arrow size as a
proxy (see TODO below).
- tqdm postfix displays throughput (MB/s) and worker utilization
(active/total).
- The `done` callback always fires, even on error (via `FinishOnDrop`),
so progress bars are always finalized.

### TODO

- Track actual bytes written to disk for local tables. This requires
Lance to expose a progress callback from its write path. See
lance-format/lance#6247.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-23 16:14:13 -07:00
Esteban Gutierrez
a0228036ae ci: fix unused PreprocessingOutput (#3180)
Simple fix to for CI due unused import of PreprocessingOutput in
table.rs

Co-authored-by: Esteban Gutierrez <esteban@lancedb.com>
2026-03-23 13:45:44 -07:00
Esteban Gutierrez
d8fc071a7d fix(ci): bump AWS SDK MSRV pins to March 2025 release (#3179)
Lance v4.1.0-beta requires the default-https-client feature on
aws-sdk-dynamodb and aws-sdk-s3, which was introduced in the March
2025 AWS SDK release. Update all AWS SDK pins to versions from the
same AWS SDK release to maintain internal dependency compatibility.

Co-authored-by: Esteban Gutierrez <esteban@lancedb.com>
2026-03-23 15:30:33 -05:00
Will Jones
e6fd8d071e feat(rust): parallel inserts for remote tables via multipart write (#3071)
Similar to https://github.com/lancedb/lancedb/pull/3062, we can write in
parallel to remote tables if the input data source is large enough.

We take advantage of new endpoints coming in server version 0.4.0, which
allow writing data in multiple requests, and the committing at the end
in a single request.

To make testing easier, I also introduce a `write_parallelism`
parameter. In the future, we can expose that in Python and NodeJS so
users can manually specify the parallelism they get.

Closes #2861

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-20 13:19:07 -07:00
LanceDB Robot
670dcca551 feat: update lance dependency to v3.0.1 (#3168)
## Summary
- Updated Lance Rust workspace dependencies to `3.0.1` using
`ci/set_lance_version.py`.
- Updated Java `lance-core` dependency property in `java/pom.xml` to
`3.0.1`.
- Refreshed `Cargo.lock` entries for Lance crates at `3.0.1`.

## Verification
- `cargo clippy --workspace --tests --all-features -- -D warnings`
- `cargo fmt --all`

## Trigger
- Tag:
[`refs/tags/v3.0.1`](https://github.com/lancedb/lance/tree/v3.0.1)

Co-authored-by: Esteban Gutierrez <estebangtz@gmail.com>
2026-03-20 09:53:20 -07:00
Prashanth Rao
ed7e01a58b docs: fix rendering issues with missing index types in API docs (#3143)
## Problem

The generated Python API docs for
`lancedb.table.IndexStatistics.index_type` were misleading because
mkdocstrings renders that field’s type annotation directly, and the
existing `Literal[...]` listed only a subset of the actual canonical SDK
index type strings.

Current (missing index types):
<img width="823" height="83" alt="image"
src="https://github.com/user-attachments/assets/f6f29fe3-4c16-4d00-a4e9-28a7cd6e19ec"
/>


## Fix

- Update the `IndexStatistics.index_type` annotation in
`python/python/lancedb/table.py` to include the full supported set of
canonical values, so the generated docs show all valid index_type
strings inline.
- Add a small regression test in `python/python/tests/test_index.py` to
ensure the docs-facing annotation does not drift silently again in case
we add a new index/quantization type in the future.
- Bumps mkdocs and material theme versions to mkdocs 1.6 to allow access
to more features like hooks

After fix (all index types are included and tested for in the
annotations):
<img width="1017" height="93" alt="image"
src="https://github.com/user-attachments/assets/66c74d5c-34b3-4b44-8173-3ee23e3648ac"
/>
2026-03-20 09:34:42 -07:00
Lance Release
3450ccaf7f Bump version: 0.27.1-beta.0 → 0.27.1 2026-03-20 00:35:36 +00:00
Lance Release
9b229f1e7c Bump version: 0.27.0 → 0.27.1-beta.0 2026-03-20 00:35:19 +00:00
Lance Release
f5b21c0aa4 Bump version: 0.30.1-beta.0 → 0.30.1 2026-03-20 00:35:03 +00:00
Lance Release
e927924d26 Bump version: 0.30.0 → 0.30.1-beta.0 2026-03-20 00:35:02 +00:00
Weston Pace
11a4966bfc feat: upgrade lance dependency to v3.0.1 (#3157)
## Summary
- Upgrade all lance-* dependencies from v3.0.0 to v3.0.1 (stable, from
crates.io)

## Test plan
- [x] `cargo check --features remote --tests --examples` passes
- [x] `cargo clippy --features remote --tests --examples` passes
- [x] `cargo fmt --all --check` passes
- [ ] CI tests pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 17:30:46 -07:00
Weston Pace
dd5aaa72dc ci: modify check_lance_release.py to prefer stable releases over betas (#3146)
When Lance 3.0.0 released the check_lance_release.py script did not make
a PR for it because it was a pre-release. This change may not be perfect
but it always ranks stable releases above non-stable releases.
2026-03-17 09:21:30 -07:00
marca116
3a200d77ef fix: pre-filtering on hybrid search (#3096)
When using hybrid search with a where filter, the prefilter argument is
silently inverted. Passing prefilter=True actually performs
post-filtering, and prefilter=False actually performs pre-filtering.
2026-03-16 21:48:42 -07:00
Lance Release
bd09c53938 Bump version: 0.27.0-beta.6 → 0.27.0 2026-03-16 22:47:06 +00:00
Lance Release
0b18e33180 Bump version: 0.27.0-beta.5 → 0.27.0-beta.6 2026-03-16 22:46:48 +00:00
Lance Release
c89240b16c Bump version: 0.30.0-beta.6 → 0.30.0 2026-03-16 22:46:19 +00:00
Lance Release
099ff355a4 Bump version: 0.30.0-beta.5 → 0.30.0-beta.6 2026-03-16 22:46:17 +00:00
Weston Pace
c5995fda67 feat: update lance dependency to 3.0.0 release (#3137)
## Summary
- Update all 14 lance crates from `3.0.0-rc.3` (git source) to `3.0.0`
(crates.io release)
- Remove git/tag source references since 3.0.0 is published on crates.io

## Test plan
- [x] `cargo check --features remote --tests --examples` passes
- [x] `cargo clippy --features remote --tests --examples` passes
- [ ] CI passes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 15:29:18 -07:00
Weston Pace
25eb1fbfa4 fix: restore storage options on copy in localstack tests (#3148) 2026-03-16 14:02:19 -07:00
Weston Pace
4ac41c5c3f fix(ci): upgrade LocalStack to 4.0 for S3 integration tests (#3147)
## Summary
- Upgrade LocalStack from 3.3 to 4.0 in `docker-compose.yml` to fix S3
integration test failures in CI
- Version 3.3 has compatibility issues with newer Python 3.13 and
updated boto3 dependencies
- Matches the LocalStack version used successfully in the lance
repository

## Test plan
- [ ] Verify `docker compose up --detach --wait` completes successfully
in CI
- [ ] All tests in `test_s3.py` pass (5 tests)
- [ ] All `@pytest.mark.s3_test` tests in
`test_namespace_integration.py` pass (7 tests)
- [ ] No regressions in non-integration test jobs (Mac, Windows)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 09:02:11 -07:00
Will Jones
9a5b0398ec chore: fix ci (#3139)
* Move away from buildjet, which is shutting down runners for GHA [^1]
* Add `Cargo.lock` to build jobs, so when we upgrade locked dependencies
we check the builds actually pass. CI started failing because
dependencies were changed in #3116 without running all build jobs.
* Add fixes for aws-lc-rs build in NodeJS.

[^1]: https://buildjet.com/for-github-actions/blog/we-are-shutting-down

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 06:25:40 -07:00
Pratik Dey
d1d720d08a feat(nodejs): support field/data type input in add_columns() method (#3114)
Add support for passing field/data type information into add_columns()
method, bringing parity with Python bindings. The method now accepts:

- AddColumnsSql[] - SQL expressions (existing functionality)
- Field - single Arrow field with explicit data type
- Field[] - array of Arrow fields with explicit data types
- Schema - Arrow schema with explicit data types

New columns added via Field/Schema are initialized with null values. All
field-based columns must be nullable due to null initialization.

Resolves #3107

---------

Signed-off-by: Pratik <pratikrocks.dey11@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-03-13 12:57:14 -07:00
Mesut-Doner
c2e543f1b7 feat(rust): support Expr in projection query (#3069)
Referred and followed [`Select::Dynamic`] implementation. 

Closes #3039
2026-03-13 12:54:26 -07:00
Weston Pace
216c1b5f77 docs: remove experimental label from optimize and warn about delete_unverified (#3128)
## Summary
- Removes the "Experimental API" section from `optimize` method
documentation across Rust, Python, and TypeScript
- Adds a warning to `delete_unverified` documentation in all bindings:
this should only be set to true if you can guarantee no other process is
working on the dataset, otherwise it could be corrupted
- Fixes a typo ("shoudl" → "should")

Closes #3125


🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 14:37:42 +08:00
Xin Sun
fc1867da83 chore: remove the duplicate snafu-derive dependency in the lockfile (#3124) 2026-03-10 21:43:51 -07:00
Esteban Gutierrez
f951da2b00 feat: support prewarm_index and prewarm_data on remote tables (#3110)
## Summary

- Implement `RemoteTable.prewarm_data(columns)` calling `POST
/v1/table/{id}/page_cache/prewarm/`
- Implement `RemoteTable.prewarm_index(name)` calling `POST
/v1/table/{id}/index/{name}/prewarm/` (previously returned
`NotSupported`)
- Add `BaseTable::prewarm_data(columns)` trait method and `Table` public
API in Rust core
- Add PyO3 bindings and Python API (`AsyncTable`, `LanceTable`,
`RemoteTable`) for `prewarm_data`
- Add type stubs for `prewarm_index` and `prewarm_data` in
`_lancedb.pyi`
- Upgrade Lance to 3.0.0-rc.3 with breaking change fixes

Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 15:39:39 -05:00
Esteban Gutierrez
6530d82690 chore: dependency updates and security fixes (#3116)
## Summary

- Update dependencies across Rust, Python, Node.js, Java, Docker, and
docs
- Pin unpinned dependency lower bounds to prevent silent downgrades
- Bump CI actions to current major versions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 20:04:27 -07:00
Lance Release
b3fc9c444f Bump version: 0.27.0-beta.4 → 0.27.0-beta.5 2026-03-09 19:58:12 +00:00
Lance Release
6de8f42dcd Bump version: 0.30.0-beta.4 → 0.30.0-beta.5 2026-03-09 19:56:15 +00:00
Will Jones
5c3bd68e58 feat: upgrade Lance to 3.0.0-rc.3 (#3104)
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
2026-03-09 12:55:20 -07:00
Weston Pace
4be85444f0 feat: infer js native arrays (#3119)
When we create tables without using Arrow by parsing JS records we
always infer to float64. Many times embeddings are not float64 and it
would be nice to be able to use the native type without requiring users
to pull in Arrow. We can utilize JS's builtin Float32Array to do this.
This PR also adds support for UInt8/16/32 and Int8/16/32 arrays as well.

Closes #3115
2026-03-09 10:13:59 -07:00
Xuanwo
68c07f333f chore: unify component README titles (#3066) 2026-03-09 21:47:58 +08:00
Lance Release
814a379e08 Bump version: 0.27.0-beta.3 → 0.27.0-beta.4 2026-03-09 08:47:17 +00:00
Lance Release
f31561c5bb Bump version: 0.30.0-beta.3 → 0.30.0-beta.4 2026-03-09 08:45:25 +00:00
Jack Ye
e0c5ceac03 fix: propagate managed versioning for namespace connection (#3111)
Without this fix, if user directly use the native table to do operations
like `add_columns`, even if it is configured to use namespace db
connection, it is not really propagated through.

The fix is to bring lancedb's python binding up to date and do a similar
implementation as https://github.com/lance-format/lance/pull/5968, and
make sure the namespace is fully propagated through all the related
calls.

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-03-09 01:44:31 -07:00
Prashanth Rao
e93bb3355a docs: add meth/func names to mkdocstrings (#3101)
LanceDB's SDK API docs do not currently show method names under any
given object, and this makes it harder to quickly understand and find
relevant method names for a given class. Geneva docs show the available
methods in the right navigation.

This PR standardizes the appearance of the LanceDB SDK API in the docs
to be more similar to Geneva's.
<img width="1386" height="792" alt="image"
src="https://github.com/user-attachments/assets/30816591-d6d5-495d-886d-e234beeb6059"
/>

<img width="897" height="540" alt="image"
src="https://github.com/user-attachments/assets/d5491b6b-c7bf-4d3b-8b15-1a1a7700e7c9"
/>
2026-03-06 08:54:45 -08:00
Will Jones
b75991eb07 fix: propagate cast errors in add() (#3075)
When we write data with `add()`, we can input data to the table's
schema. However, we were using "safe" mode, which propagates errors as
nulls. For example, if you pass `u64::max` into a field that is a `u32`,
it will just write null instead of giving overflow error. Now it
propagates the overflow. This is the same behavior as other systems like
DuckDB.

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 20:24:50 -08:00
Wyatt Alt
97ca9bb943 feat: allow passing azure client/tenant ID through remote SDK (#3102)
Prior to this commit we supported passing the azure storage account name
to the lancedb remote SDK through headers. This adds support for client
ID and tenant ID as well.
2026-03-04 11:11:36 -08:00
Xuanwo
fa1b04f341 chore: migrate Rust crates to edition 2024 and fix clippy warnings (#3098)
This PR migrates all Rust crates in the workspace to Rust 2024 edition
and addresses the resulting compatibility updates. It also fixes all
clippy warnings surfaced by the workspace checks so the codebase remains
warning-free under the current lint configuration.

Context:
- Scope: workspace edition bump (`2021` -> `2024`) plus follow-up
refactors required by new edition and clippy rules.
- Validation: `cargo fmt --all` and `cargo clippy --quiet --features
remote --tests --examples -- -D warnings` both pass.
2026-03-03 16:23:29 -08:00
mrncstt
367abe99d2 feat(python): support dict to SQL struct conversion in table.update() (#3089)
## Summary

- Add `@value_to_sql.register(dict)` handler that converts Python dicts
to DataFusion's `named_struct()` SQL syntax
- Enables updating struct-typed columns via `table.update(values={"col":
{"field_a": 1, "field_b": "hello"}})`
- Recursively handles nested structs, lists, nulls, and all existing
scalar types

Closes #1363

## Details

The `named_struct` function was introduced in DataFusion 38 and is now
available (LanceDB uses DataFusion 52.1). The implementation follows the
existing `singledispatch` pattern in `util.py`.

**Example conversion:**
```python
value_to_sql({"field_a": 1, "field_b": "hello"})
# => "named_struct('field_a', 1, 'field_b', 'hello')"
```

## Test plan

- [x] Unit tests for flat struct, nested struct, list inside struct,
mixed types, null values, and empty dict
- [ ] CI integration tests with actual table.update() on struct columns

🔗 [DataFusion named_struct
docs](https://datafusion.apache.org/user-guide/sql/scalar_functions.html#named-struct)
2026-03-03 13:36:08 -08:00
Xuanwo
52ce2c995c fix(ci): only run npm publish on release tags (#3093)
This PR fixes the npm publish dry-run failure for prerelease versions
without changing the existing workflow trigger behavior. The publish
step now detects prerelease versions from `nodejs/package.json` and
always appends `--tag preview` when needed.

Context:
- On `main` pushes, the workflow still runs `npm publish --dry-run` by
design.
- Recent failures were caused by prerelease versions (for example
`0.27.0-beta.3`) running without `--tag`, which npm rejects.
- The previous `refs/tags/v...-beta...` check did not apply on branch
pushes, so dry-run could fail even though release tags worked.
2026-03-04 01:35:10 +08:00
Sean Mackrory
e71a00998c ci: add regression test for fastSearch in FTS queries in TypeScript (#3090)
We recently added support for this for the Python bindings, and wanted
to confirm this already worked as expected in the TS bindings.
2026-03-03 07:09:09 -08:00
Sean Mackrory
39a2ac0a1c feat: add parity between fast_search keyword argument between vector and FTS searches (#3091)
We don't necessarily need to do this, but one user was confused having
used `fast_search=True` as a keyword argument for vector searches, but
being unable to do so for FTS, even after the most recent changes. I
think this is the only discrepancy in where that is possible.
2026-03-03 05:21:36 -08:00
Wyatt Alt
bc7b344fa4 feat: add support for remote index params (#3087)
Prior to this commit the remote SDK did not support the full set of
index parameters. This extends the SDK to support them.
2026-03-02 11:14:28 -08:00
Will Jones
f91d2f5fec ci(python): pin maturin to work around bug (#3088)
Work around for https://github.com/PyO3/maturin/issues/3059
2026-03-02 09:38:54 -08:00
Wyatt Alt
cf81b6419f feat: add num_deleted_rows to delete result (#3077) 2026-03-02 08:37:14 -08:00
Lance Release
0498ac1f2f Bump version: 0.27.0-beta.2 → 0.27.0-beta.3 2026-02-28 01:31:51 +00:00
163 changed files with 11376 additions and 7191 deletions

View File

@@ -1,5 +1,5 @@
[tool.bumpversion] [tool.bumpversion]
current_version = "0.27.0-beta.2" current_version = "0.27.2"
parse = """(?x) parse = """(?x)
(?P<major>0|[1-9]\\d*)\\. (?P<major>0|[1-9]\\d*)\\.
(?P<minor>0|[1-9]\\d*)\\. (?P<minor>0|[1-9]\\d*)\\.

View File

@@ -23,12 +23,15 @@ runs:
steps: steps:
- name: CONFIRM ARM BUILD - name: CONFIRM ARM BUILD
shell: bash shell: bash
env:
ARM_BUILD: ${{ inputs.arm-build }}
run: | run: |
echo "ARM BUILD: ${{ inputs.arm-build }}" echo "ARM BUILD: $ARM_BUILD"
- name: Build x86_64 Manylinux wheel - name: Build x86_64 Manylinux wheel
if: ${{ inputs.arm-build == 'false' }} if: ${{ inputs.arm-build == 'false' }}
uses: PyO3/maturin-action@v1 uses: PyO3/maturin-action@v1
with: with:
maturin-version: "1.12.4"
command: build command: build
working-directory: python working-directory: python
docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'" docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'"
@@ -44,6 +47,7 @@ runs:
if: ${{ inputs.arm-build == 'true' }} if: ${{ inputs.arm-build == 'true' }}
uses: PyO3/maturin-action@v1 uses: PyO3/maturin-action@v1
with: with:
maturin-version: "1.12.4"
command: build command: build
working-directory: python working-directory: python
docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'" docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'"

View File

@@ -20,6 +20,7 @@ runs:
uses: PyO3/maturin-action@v1 uses: PyO3/maturin-action@v1
with: with:
command: build command: build
maturin-version: "1.12.4"
# TODO: pass through interpreter # TODO: pass through interpreter
args: ${{ inputs.args }} args: ${{ inputs.args }}
docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'" docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'"

View File

@@ -25,6 +25,7 @@ runs:
uses: PyO3/maturin-action@v1 uses: PyO3/maturin-action@v1
with: with:
command: build command: build
maturin-version: "1.12.4"
args: ${{ inputs.args }} args: ${{ inputs.args }}
docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'" docker-options: "-e PIP_EXTRA_INDEX_URL='https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/'"
working-directory: python working-directory: python

View File

@@ -15,7 +15,7 @@ jobs:
name: Label PR name: Label PR
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: srvaroa/labeler@master - uses: srvaroa/labeler@v1
env: env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
commitlint: commitlint:
@@ -24,7 +24,7 @@ jobs:
name: Verify PR title / description conforms to semantic-release name: Verify PR title / description conforms to semantic-release
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/setup-node@v3 - uses: actions/setup-node@v4
with: with:
node-version: "18" node-version: "18"
# These rules are disabled because Github will always ensure there # These rules are disabled because Github will always ensure there
@@ -47,7 +47,7 @@ jobs:
${{ github.event.pull_request.body }} ${{ github.event.pull_request.body }}
- if: failure() - if: failure()
uses: actions/github-script@v6 uses: actions/github-script@v7
with: with:
script: | script: |
const message = `**ACTION NEEDED** const message = `**ACTION NEEDED**

View File

@@ -53,7 +53,7 @@ jobs:
python -m pip install --extra-index-url https://pypi.fury.io/lance-format/ --extra-index-url https://pypi.fury.io/lancedb/ -e . python -m pip install --extra-index-url https://pypi.fury.io/lance-format/ --extra-index-url https://pypi.fury.io/lancedb/ -e .
python -m pip install --extra-index-url https://pypi.fury.io/lance-format/ --extra-index-url https://pypi.fury.io/lancedb/ -r ../docs/requirements.txt python -m pip install --extra-index-url https://pypi.fury.io/lance-format/ --extra-index-url https://pypi.fury.io/lancedb/ -r ../docs/requirements.txt
- name: Set up node - name: Set up node
uses: actions/setup-node@v3 uses: actions/setup-node@v4
with: with:
node-version: 20 node-version: 20
cache: 'npm' cache: 'npm'
@@ -68,7 +68,7 @@ jobs:
run: | run: |
PYTHONPATH=. mkdocs build PYTHONPATH=. mkdocs build
- name: Setup Pages - name: Setup Pages
uses: actions/configure-pages@v2 uses: actions/configure-pages@v5
- name: Upload artifact - name: Upload artifact
uses: actions/upload-pages-artifact@v3 uses: actions/upload-pages-artifact@v3
with: with:

View File

@@ -7,6 +7,7 @@ on:
pull_request: pull_request:
paths: paths:
- Cargo.toml - Cargo.toml
- Cargo.lock
- nodejs/** - nodejs/**
- rust/** - rust/**
- docs/src/js/** - docs/src/js/**
@@ -37,7 +38,7 @@ jobs:
with: with:
fetch-depth: 0 fetch-depth: 0
lfs: true lfs: true
- uses: actions/setup-node@v3 - uses: actions/setup-node@v4
with: with:
node-version: 20 node-version: 20
cache: 'npm' cache: 'npm'
@@ -77,7 +78,7 @@ jobs:
with: with:
fetch-depth: 0 fetch-depth: 0
lfs: true lfs: true
- uses: actions/setup-node@v3 - uses: actions/setup-node@v4
name: Setup Node.js 20 for build name: Setup Node.js 20 for build
with: with:
# @napi-rs/cli v3 requires Node >= 20.12 (via @inquirer/prompts@8). # @napi-rs/cli v3 requires Node >= 20.12 (via @inquirer/prompts@8).
@@ -94,7 +95,7 @@ jobs:
run: | run: |
npm ci --include=optional npm ci --include=optional
npm run build:debug -- --profile ci npm run build:debug -- --profile ci
- uses: actions/setup-node@v3 - uses: actions/setup-node@v4
name: Setup Node.js ${{ matrix.node-version }} for test name: Setup Node.js ${{ matrix.node-version }} for test
with: with:
node-version: ${{ matrix.node-version }} node-version: ${{ matrix.node-version }}
@@ -143,7 +144,7 @@ jobs:
with: with:
fetch-depth: 0 fetch-depth: 0
lfs: true lfs: true
- uses: actions/setup-node@v3 - uses: actions/setup-node@v4
with: with:
node-version: 20 node-version: 20
cache: 'npm' cache: 'npm'

View File

@@ -19,6 +19,7 @@ on:
paths: paths:
- .github/workflows/npm-publish.yml - .github/workflows/npm-publish.yml
- Cargo.toml # Change in dependency frequently breaks builds - Cargo.toml # Change in dependency frequently breaks builds
- Cargo.lock
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.ref }} group: ${{ github.workflow }}-${{ github.ref }}
@@ -124,7 +125,12 @@ jobs:
pre_build: |- pre_build: |-
set -e && set -e &&
apt-get update && apt-get update &&
apt-get install -y protobuf-compiler pkg-config apt-get install -y protobuf-compiler pkg-config &&
# The base image (manylinux2014-cross) sets TARGET_CC to the old
# GCC 4.8 cross-compiler. aws-lc-sys checks TARGET_CC before CC,
# so it picks up GCC even though the napi-rs image sets CC=clang.
# Override to use the image's clang-18 which supports -fuse-ld=lld.
export TARGET_CC=clang TARGET_CXX=clang++
- target: x86_64-unknown-linux-musl - target: x86_64-unknown-linux-musl
# This one seems to need some extra memory # This one seems to need some extra memory
host: ubuntu-2404-8x-x64 host: ubuntu-2404-8x-x64
@@ -144,9 +150,10 @@ jobs:
set -e && set -e &&
apt-get update && apt-get update &&
apt-get install -y protobuf-compiler pkg-config && apt-get install -y protobuf-compiler pkg-config &&
# https://github.com/aws/aws-lc-rs/issues/737#issuecomment-2725918627 export TARGET_CC=clang TARGET_CXX=clang++ &&
ln -s /usr/aarch64-unknown-linux-gnu/lib/gcc/aarch64-unknown-linux-gnu/4.8.5/crtbeginS.o /usr/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/usr/lib/crtbeginS.o && # The manylinux2014 sysroot has glibc 2.17 headers which lack
ln -s /usr/aarch64-unknown-linux-gnu/lib/gcc /usr/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/usr/lib/gcc && # AT_HWCAP2 (added in Linux 3.17). Define it for aws-lc-sys.
export CFLAGS="$CFLAGS -DAT_HWCAP2=26" &&
rustup target add aarch64-unknown-linux-gnu rustup target add aarch64-unknown-linux-gnu
- target: aarch64-unknown-linux-musl - target: aarch64-unknown-linux-musl
host: ubuntu-2404-8x-x64 host: ubuntu-2404-8x-x64
@@ -266,7 +273,7 @@ jobs:
- target: x86_64-unknown-linux-gnu - target: x86_64-unknown-linux-gnu
host: ubuntu-latest host: ubuntu-latest
- target: aarch64-unknown-linux-gnu - target: aarch64-unknown-linux-gnu
host: buildjet-16vcpu-ubuntu-2204-arm host: ubuntu-2404-8x-arm64
node: node:
- '20' - '20'
runs-on: ${{ matrix.settings.host }} runs-on: ${{ matrix.settings.host }}
@@ -356,7 +363,8 @@ jobs:
if [[ $DRY_RUN == "true" ]]; then if [[ $DRY_RUN == "true" ]]; then
ARGS="$ARGS --dry-run" ARGS="$ARGS --dry-run"
fi fi
if [[ $GITHUB_REF =~ refs/tags/v(.*)-beta.* ]]; then VERSION=$(node -p "require('./package.json').version")
if [[ $VERSION == *-* ]]; then
ARGS="$ARGS --tag preview" ARGS="$ARGS --tag preview"
fi fi
npm publish $ARGS npm publish $ARGS

View File

@@ -9,6 +9,7 @@ on:
paths: paths:
- .github/workflows/pypi-publish.yml - .github/workflows/pypi-publish.yml
- Cargo.toml # Change in dependency frequently breaks builds - Cargo.toml # Change in dependency frequently breaks builds
- Cargo.lock
env: env:
PIP_EXTRA_INDEX_URL: "https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/" PIP_EXTRA_INDEX_URL: "https://pypi.fury.io/lance-format/ https://pypi.fury.io/lancedb/"

View File

@@ -7,9 +7,14 @@ on:
pull_request: pull_request:
paths: paths:
- Cargo.toml - Cargo.toml
- Cargo.lock
- python/** - python/**
- rust/** - rust/**
- .github/workflows/python.yml - .github/workflows/python.yml
- .github/workflows/build_linux_wheel/**
- .github/workflows/build_mac_wheel/**
- .github/workflows/build_windows_wheel/**
- .github/workflows/run_tests/**
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}

View File

@@ -7,6 +7,7 @@ on:
pull_request: pull_request:
paths: paths:
- Cargo.toml - Cargo.toml
- Cargo.lock
- rust/** - rust/**
- .github/workflows/rust.yml - .github/workflows/rust.yml
@@ -206,14 +207,14 @@ jobs:
- name: Downgrade dependencies - name: Downgrade dependencies
# These packages have newer requirements for MSRV # These packages have newer requirements for MSRV
run: | run: |
cargo update -p aws-sdk-bedrockruntime --precise 1.64.0 cargo update -p aws-sdk-bedrockruntime --precise 1.77.0
cargo update -p aws-sdk-dynamodb --precise 1.55.0 cargo update -p aws-sdk-dynamodb --precise 1.68.0
cargo update -p aws-config --precise 1.5.10 cargo update -p aws-config --precise 1.6.0
cargo update -p aws-sdk-kms --precise 1.51.0 cargo update -p aws-sdk-kms --precise 1.63.0
cargo update -p aws-sdk-s3 --precise 1.65.0 cargo update -p aws-sdk-s3 --precise 1.79.0
cargo update -p aws-sdk-sso --precise 1.50.0 cargo update -p aws-sdk-sso --precise 1.62.0
cargo update -p aws-sdk-ssooidc --precise 1.51.0 cargo update -p aws-sdk-ssooidc --precise 1.63.0
cargo update -p aws-sdk-sts --precise 1.51.0 cargo update -p aws-sdk-sts --precise 1.63.0
cargo update -p home --precise 0.5.9 cargo update -p home --precise 0.5.9
- name: cargo +${{ matrix.msrv }} check - name: cargo +${{ matrix.msrv }} check
env: env:

2447
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -5,7 +5,7 @@ exclude = ["python"]
resolver = "2" resolver = "2"
[workspace.package] [workspace.package]
edition = "2021" edition = "2024"
authors = ["LanceDB Devs <dev@lancedb.com>"] authors = ["LanceDB Devs <dev@lancedb.com>"]
license = "Apache-2.0" license = "Apache-2.0"
repository = "https://github.com/lancedb/lancedb" repository = "https://github.com/lancedb/lancedb"
@@ -15,20 +15,20 @@ categories = ["database-implementations"]
rust-version = "1.91.0" rust-version = "1.91.0"
[workspace.dependencies] [workspace.dependencies]
lance = { "version" = "=3.0.0-rc.2", default-features = false, "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance = { "version" = "=5.0.0-beta.4", default-features = false, "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-core = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-core = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-datagen = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-datagen = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-file = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-file = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-io = { "version" = "=3.0.0-rc.2", default-features = false, "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-io = { "version" = "=5.0.0-beta.4", default-features = false, "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-index = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-index = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-linalg = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-linalg = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-namespace = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-namespace-impls = { "version" = "=3.0.0-rc.2", default-features = false, "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-namespace-impls = { "version" = "=5.0.0-beta.4", default-features = false, "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-table = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-table = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-testing = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-testing = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-datafusion = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-datafusion = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-encoding = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-encoding = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
lance-arrow = { "version" = "=3.0.0-rc.2", "tag" = "v3.0.0-rc.2", "git" = "https://github.com/lance-format/lance.git" } lance-arrow = { "version" = "=5.0.0-beta.4", "tag" = "v5.0.0-beta.4", "git" = "https://github.com/lance-format/lance.git" }
ahash = "0.8" ahash = "0.8"
# Note that this one does not include pyarrow # Note that this one does not include pyarrow
arrow = { version = "57.2", optional = false } arrow = { version = "57.2", optional = false }

View File

@@ -3,6 +3,7 @@
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import functools
import json import json
import os import os
import re import re
@@ -26,6 +27,7 @@ SEMVER_RE = re.compile(
) )
@functools.total_ordering
@dataclass(frozen=True) @dataclass(frozen=True)
class SemVer: class SemVer:
major: int major: int
@@ -156,7 +158,9 @@ def read_current_version(repo_root: Path) -> str:
def determine_latest_tag(tags: Iterable[TagInfo]) -> TagInfo: def determine_latest_tag(tags: Iterable[TagInfo]) -> TagInfo:
return max(tags, key=lambda tag: tag.semver) # Stable releases (no prerelease) are always preferred over pre-releases.
# Within each group, standard semver ordering applies.
return max(tags, key=lambda tag: (not tag.semver.prerelease, tag.semver))
def write_outputs(args: argparse.Namespace, payload: dict) -> None: def write_outputs(args: argparse.Namespace, payload: dict) -> None:

View File

@@ -1,7 +1,7 @@
version: "3.9" version: "3.9"
services: services:
localstack: localstack:
image: localstack/localstack:3.3 image: localstack/localstack:4.0
ports: ports:
- 4566:4566 - 4566:4566
environment: environment:

View File

@@ -1,27 +1,27 @@
#Simple base dockerfile that supports basic dependencies required to run lance with FTS and Hybrid Search # Simple base dockerfile that supports basic dependencies required to run lance with FTS and Hybrid Search
#Usage docker build -t lancedb:latest -f Dockerfile . # Usage: docker build -t lancedb:latest -f Dockerfile .
FROM python:3.10-slim-buster FROM python:3.12-slim-bookworm
# Install Rust # Install build dependencies in a single layer
RUN apt-get update && apt-get install -y curl build-essential && \ RUN apt-get update && \
curl https://sh.rustup.rs -sSf | sh -s -- -y apt-get install -y --no-install-recommends \
curl \
# Set the environment variable for Rust build-essential \
ENV PATH="/root/.cargo/bin:${PATH}" protobuf-compiler \
git \
# Install protobuf compiler ca-certificates && \
RUN apt-get install -y protobuf-compiler && \
apt-get clean && \ apt-get clean && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*
RUN apt-get -y update &&\ # Install Rust (pinned installer, non-interactive)
apt-get -y upgrade && \ RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable --profile minimal
apt-get -y install git
# Set the environment variable for Rust
ENV PATH="/root/.cargo/bin:${PATH}"
# Verify installations # Verify installations
RUN python --version && \ RUN python --version && \
rustc --version && \ rustc --version && \
protoc --version protoc --version
RUN pip install tantivy lancedb RUN pip install --no-cache-dir tantivy lancedb

View File

@@ -52,14 +52,21 @@ plugins:
options: options:
docstring_style: numpy docstring_style: numpy
heading_level: 3 heading_level: 3
show_source: true
show_symbol_type_in_heading: true
show_signature_annotations: true show_signature_annotations: true
show_root_heading: true show_root_heading: true
show_docstring_examples: true
show_docstring_attributes: false
show_docstring_other_parameters: true
show_symbol_type_heading: true
show_labels: false
show_if_no_docstring: true
show_source: false
members_order: source members_order: source
docstring_section_style: list docstring_section_style: list
signature_crossrefs: true signature_crossrefs: true
separate_signature: true separate_signature: true
filters:
- "!^_"
import: import:
# for cross references # for cross references
- https://arrow.apache.org/docs/objects.inv - https://arrow.apache.org/docs/objects.inv
@@ -113,7 +120,7 @@ markdown_extensions:
emoji_index: !!python/name:material.extensions.emoji.twemoji emoji_index: !!python/name:material.extensions.emoji.twemoji
emoji_generator: !!python/name:material.extensions.emoji.to_svg emoji_generator: !!python/name:material.extensions.emoji.to_svg
- markdown.extensions.toc: - markdown.extensions.toc:
toc_depth: 3 toc_depth: 4
permalink: true permalink: true
permalink_title: Anchor link to this section permalink_title: Anchor link to this section

View File

@@ -1,9 +1,9 @@
mkdocs==1.5.3 mkdocs==1.6.1
mkdocs-jupyter==0.24.1 mkdocs-jupyter==0.24.1
mkdocs-material==9.5.3 mkdocs-material==9.6.23
mkdocs-autorefs<=1.0 mkdocs-autorefs>=0.5,<=1.0
mkdocstrings[python]==0.25.2 mkdocstrings[python]>=0.24,<1.0
griffe griffe>=0.40,<1.0
mkdocs-render-swagger-plugin mkdocs-render-swagger-plugin>=0.1.0
pydantic pydantic>=2.0,<3.0
mkdocs-redirects mkdocs-redirects>=1.2.0

View File

@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
<dependency> <dependency>
<groupId>com.lancedb</groupId> <groupId>com.lancedb</groupId>
<artifactId>lancedb-core</artifactId> <artifactId>lancedb-core</artifactId>
<version>0.27.0-beta.2</version> <version>0.27.2</version>
</dependency> </dependency>
``` ```
@@ -57,32 +57,32 @@ LanceNamespace namespaceClient = LanceDbNamespaceClientBuilder.newBuilder()
## Metadata Operations ## Metadata Operations
### Creating a Namespace ### Creating a Namespace Path
Namespaces organize tables hierarchically. Create a namespace before creating tables within it: Namespace paths organize tables hierarchically. Create the desired namespace path before creating tables within it:
```java ```java
import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceRequest;
import org.lance.namespace.model.CreateNamespaceResponse; import org.lance.namespace.model.CreateNamespaceResponse;
// Create a child namespace // Create a child namespace path
CreateNamespaceRequest request = new CreateNamespaceRequest(); CreateNamespaceRequest request = new CreateNamespaceRequest();
request.setId(Arrays.asList("my_namespace")); request.setId(Arrays.asList("my_namespace"));
CreateNamespaceResponse response = namespaceClient.createNamespace(request); CreateNamespaceResponse response = namespaceClient.createNamespace(request);
``` ```
You can also create nested namespaces: You can also create nested namespace paths:
```java ```java
// Create a nested namespace: parent/child // Create a nested namespace path: parent/child
CreateNamespaceRequest request = new CreateNamespaceRequest(); CreateNamespaceRequest request = new CreateNamespaceRequest();
request.setId(Arrays.asList("parent_namespace", "child_namespace")); request.setId(Arrays.asList("parent_namespace", "child_namespace"));
CreateNamespaceResponse response = namespaceClient.createNamespace(request); CreateNamespaceResponse response = namespaceClient.createNamespace(request);
``` ```
### Describing a Namespace ### Describing a Namespace Path
```java ```java
import org.lance.namespace.model.DescribeNamespaceRequest; import org.lance.namespace.model.DescribeNamespaceRequest;
@@ -95,22 +95,22 @@ DescribeNamespaceResponse response = namespaceClient.describeNamespace(request);
System.out.println("Namespace properties: " + response.getProperties()); System.out.println("Namespace properties: " + response.getProperties());
``` ```
### Listing Namespaces ### Listing Namespace Paths
```java ```java
import org.lance.namespace.model.ListNamespacesRequest; import org.lance.namespace.model.ListNamespacesRequest;
import org.lance.namespace.model.ListNamespacesResponse; import org.lance.namespace.model.ListNamespacesResponse;
// List all namespaces at root level // List all namespace paths at the root level
ListNamespacesRequest request = new ListNamespacesRequest(); ListNamespacesRequest request = new ListNamespacesRequest();
request.setId(Arrays.asList()); // Empty for root request.setId(Arrays.asList()); // Empty for root
ListNamespacesResponse response = namespaceClient.listNamespaces(request); ListNamespacesResponse response = namespaceClient.listNamespaces(request);
for (String ns : response.getNamespaces()) { for (String ns : response.getNamespaces()) {
System.out.println("Namespace: " + ns); System.out.println("Namespace path: " + ns);
} }
// List child namespaces under a parent // List child namespace paths under a parent path
ListNamespacesRequest childRequest = new ListNamespacesRequest(); ListNamespacesRequest childRequest = new ListNamespacesRequest();
childRequest.setId(Arrays.asList("parent_namespace")); childRequest.setId(Arrays.asList("parent_namespace"));
@@ -123,7 +123,7 @@ ListNamespacesResponse childResponse = namespaceClient.listNamespaces(childReque
import org.lance.namespace.model.ListTablesRequest; import org.lance.namespace.model.ListTablesRequest;
import org.lance.namespace.model.ListTablesResponse; import org.lance.namespace.model.ListTablesResponse;
// List tables in a namespace // List tables in a namespace path
ListTablesRequest request = new ListTablesRequest(); ListTablesRequest request = new ListTablesRequest();
request.setId(Arrays.asList("my_namespace")); request.setId(Arrays.asList("my_namespace"));
@@ -133,7 +133,7 @@ for (String table : response.getTables()) {
} }
``` ```
### Dropping a Namespace ### Dropping a Namespace Path
```java ```java
import org.lance.namespace.model.DropNamespaceRequest; import org.lance.namespace.model.DropNamespaceRequest;
@@ -175,7 +175,7 @@ DropTableResponse response = namespaceClient.dropTable(request);
### Creating a Table ### Creating a Table
Tables are created within a namespace by providing data in Apache Arrow IPC format: Tables are created within a namespace path by providing data in Apache Arrow IPC format:
```java ```java
import org.lance.namespace.LanceNamespace; import org.lance.namespace.LanceNamespace;
@@ -242,7 +242,7 @@ try (BufferAllocator allocator = new RootAllocator();
} }
byte[] tableData = out.toByteArray(); byte[] tableData = out.toByteArray();
// Create table in a namespace // Create a table in a namespace path
CreateTableRequest request = new CreateTableRequest(); CreateTableRequest request = new CreateTableRequest();
request.setId(Arrays.asList("my_namespace", "my_table")); request.setId(Arrays.asList("my_namespace", "my_table"));
CreateTableResponse response = namespaceClient.createTable(request, tableData); CreateTableResponse response = namespaceClient.createTable(request, tableData);

View File

@@ -61,8 +61,8 @@ sharing the same data, deletion, and index files.
* **options.sourceVersion?**: `number` * **options.sourceVersion?**: `number`
The version of the source table to clone. The version of the source table to clone.
* **options.targetNamespace?**: `string`[] * **options.targetNamespacePath?**: `string`[]
The namespace for the target table (defaults to root namespace). The namespace path for the target table (defaults to root namespace).
#### Returns #### Returns
@@ -116,13 +116,13 @@ Creates a new empty Table
`Promise`&lt;[`Table`](Table.md)&gt; `Promise`&lt;[`Table`](Table.md)&gt;
#### createEmptyTable(name, schema, namespace, options) #### createEmptyTable(name, schema, namespacePath, options)
```ts ```ts
abstract createEmptyTable( abstract createEmptyTable(
name, name,
schema, schema,
namespace?, namespacePath?,
options?): Promise<Table> options?): Promise<Table>
``` ```
@@ -136,8 +136,8 @@ Creates a new empty Table
* **schema**: [`SchemaLike`](../type-aliases/SchemaLike.md) * **schema**: [`SchemaLike`](../type-aliases/SchemaLike.md)
The schema of the table The schema of the table
* **namespace?**: `string`[] * **namespacePath?**: `string`[]
The namespace to create the table in (defaults to root namespace) The namespace path to create the table in (defaults to root namespace)
* **options?**: `Partial`&lt;[`CreateTableOptions`](../interfaces/CreateTableOptions.md)&gt; * **options?**: `Partial`&lt;[`CreateTableOptions`](../interfaces/CreateTableOptions.md)&gt;
Additional options Additional options
@@ -150,10 +150,10 @@ Creates a new empty Table
### createTable() ### createTable()
#### createTable(options, namespace) #### createTable(options, namespacePath)
```ts ```ts
abstract createTable(options, namespace?): Promise<Table> abstract createTable(options, namespacePath?): Promise<Table>
``` ```
Creates a new Table and initialize it with new data. Creates a new Table and initialize it with new data.
@@ -163,8 +163,8 @@ Creates a new Table and initialize it with new data.
* **options**: `object` & `Partial`&lt;[`CreateTableOptions`](../interfaces/CreateTableOptions.md)&gt; * **options**: `object` & `Partial`&lt;[`CreateTableOptions`](../interfaces/CreateTableOptions.md)&gt;
The options object. The options object.
* **namespace?**: `string`[] * **namespacePath?**: `string`[]
The namespace to create the table in (defaults to root namespace) The namespace path to create the table in (defaults to root namespace)
##### Returns ##### Returns
@@ -197,13 +197,13 @@ Creates a new Table and initialize it with new data.
`Promise`&lt;[`Table`](Table.md)&gt; `Promise`&lt;[`Table`](Table.md)&gt;
#### createTable(name, data, namespace, options) #### createTable(name, data, namespacePath, options)
```ts ```ts
abstract createTable( abstract createTable(
name, name,
data, data,
namespace?, namespacePath?,
options?): Promise<Table> options?): Promise<Table>
``` ```
@@ -218,8 +218,8 @@ Creates a new Table and initialize it with new data.
Non-empty Array of Records Non-empty Array of Records
to be inserted into the table to be inserted into the table
* **namespace?**: `string`[] * **namespacePath?**: `string`[]
The namespace to create the table in (defaults to root namespace) The namespace path to create the table in (defaults to root namespace)
* **options?**: `Partial`&lt;[`CreateTableOptions`](../interfaces/CreateTableOptions.md)&gt; * **options?**: `Partial`&lt;[`CreateTableOptions`](../interfaces/CreateTableOptions.md)&gt;
Additional options Additional options
@@ -247,15 +247,15 @@ Return a brief description of the connection
### dropAllTables() ### dropAllTables()
```ts ```ts
abstract dropAllTables(namespace?): Promise<void> abstract dropAllTables(namespacePath?): Promise<void>
``` ```
Drop all tables in the database. Drop all tables in the database.
#### Parameters #### Parameters
* **namespace?**: `string`[] * **namespacePath?**: `string`[]
The namespace to drop tables from (defaults to root namespace). The namespace path to drop tables from (defaults to root namespace).
#### Returns #### Returns
@@ -266,7 +266,7 @@ Drop all tables in the database.
### dropTable() ### dropTable()
```ts ```ts
abstract dropTable(name, namespace?): Promise<void> abstract dropTable(name, namespacePath?): Promise<void>
``` ```
Drop an existing table. Drop an existing table.
@@ -276,8 +276,8 @@ Drop an existing table.
* **name**: `string` * **name**: `string`
The name of the table to drop. The name of the table to drop.
* **namespace?**: `string`[] * **namespacePath?**: `string`[]
The namespace of the table (defaults to root namespace). The namespace path of the table (defaults to root namespace).
#### Returns #### Returns
@@ -304,7 +304,7 @@ Return true if the connection has not been closed
```ts ```ts
abstract openTable( abstract openTable(
name, name,
namespace?, namespacePath?,
options?): Promise<Table> options?): Promise<Table>
``` ```
@@ -315,8 +315,8 @@ Open a table in the database.
* **name**: `string` * **name**: `string`
The name of the table The name of the table
* **namespace?**: `string`[] * **namespacePath?**: `string`[]
The namespace of the table (defaults to root namespace) The namespace path of the table (defaults to root namespace)
* **options?**: `Partial`&lt;[`OpenTableOptions`](../interfaces/OpenTableOptions.md)&gt; * **options?**: `Partial`&lt;[`OpenTableOptions`](../interfaces/OpenTableOptions.md)&gt;
Additional options Additional options
@@ -349,10 +349,10 @@ Tables will be returned in lexicographical order.
`Promise`&lt;`string`[]&gt; `Promise`&lt;`string`[]&gt;
#### tableNames(namespace, options) #### tableNames(namespacePath, options)
```ts ```ts
abstract tableNames(namespace?, options?): Promise<string[]> abstract tableNames(namespacePath?, options?): Promise<string[]>
``` ```
List all the table names in this database. List all the table names in this database.
@@ -361,8 +361,8 @@ Tables will be returned in lexicographical order.
##### Parameters ##### Parameters
* **namespace?**: `string`[] * **namespacePath?**: `string`[]
The namespace to list tables from (defaults to root namespace) The namespace path to list tables from (defaults to root namespace)
* **options?**: `Partial`&lt;[`TableNamesOptions`](../interfaces/TableNamesOptions.md)&gt; * **options?**: `Partial`&lt;[`TableNamesOptions`](../interfaces/TableNamesOptions.md)&gt;
options to control the options to control the

View File

@@ -71,11 +71,12 @@ Add new columns with defined values.
#### Parameters #### Parameters
* **newColumnTransforms**: [`AddColumnsSql`](../interfaces/AddColumnsSql.md)[] * **newColumnTransforms**: `Field`&lt;`any`&gt; \| `Field`&lt;`any`&gt;[] \| `Schema`&lt;`any`&gt; \| [`AddColumnsSql`](../interfaces/AddColumnsSql.md)[]
pairs of column names and Either:
the SQL expression to use to calculate the value of the new column. These - An array of objects with column names and SQL expressions to calculate values
expressions will be evaluated for each row in the table, and can - A single Arrow Field defining one column with its data type (column will be initialized with null values)
reference existing columns in the table. - An array of Arrow Fields defining columns with their data types (columns will be initialized with null values)
- An Arrow Schema defining columns with their data types (columns will be initialized with null values)
#### Returns #### Returns
@@ -484,19 +485,7 @@ Modeled after ``VACUUM`` in PostgreSQL.
- Prune: Removes old versions of the dataset - Prune: Removes old versions of the dataset
- Index: Optimizes the indices, adding new data to existing indices - Index: Optimizes the indices, adding new data to existing indices
Experimental API The frequency an application should call optimize is based on the frequency of
----------------
The optimization process is undergoing active development and may change.
Our goal with these changes is to improve the performance of optimization and
reduce the complexity.
That being said, it is essential today to run optimize if you want the best
performance. It should be stable and safe to use in production, but it our
hope that the API may be simplified (or not even need to be called) in the
future.
The frequency an application shoudl call optimize is based on the frequency of
data modifications. If data is frequently added, deleted, or updated then data modifications. If data is frequently added, deleted, or updated then
optimize should be run frequently. A good rule of thumb is to run optimize if optimize should be run frequently. A good rule of thumb is to run optimize if
you have added or modified 100,000 or more records or run more than 20 data you have added or modified 100,000 or more records or run more than 20 data

View File

@@ -8,6 +8,14 @@
## Properties ## Properties
### numDeletedRows
```ts
numDeletedRows: number;
```
***
### version ### version
```ts ```ts

View File

@@ -37,3 +37,12 @@ tbl.optimize({cleanupOlderThan: new Date()});
```ts ```ts
deleteUnverified: boolean; deleteUnverified: boolean;
``` ```
Because they may be part of an in-progress transaction, files newer than
7 days old are not deleted by default. If you are sure that there are no
in-progress transactions, then you can set this to true to delete all
files older than `cleanupOlderThan`.
**WARNING**: This should only be set to true if you can guarantee that
no other process is currently working on this dataset. Otherwise the
dataset could be put into a corrupted state.

View File

@@ -52,7 +52,7 @@ new EmbeddingFunction<T, M>(): EmbeddingFunction<T, M>
### computeQueryEmbeddings() ### computeQueryEmbeddings()
```ts ```ts
computeQueryEmbeddings(data): Promise<number[] | Float32Array | Float64Array> computeQueryEmbeddings(data): Promise<number[] | Uint8Array | Float32Array | Float64Array>
``` ```
Compute the embeddings for a single query Compute the embeddings for a single query
@@ -63,7 +63,7 @@ Compute the embeddings for a single query
#### Returns #### Returns
`Promise`&lt;`number`[] \| `Float32Array` \| `Float64Array`&gt; `Promise`&lt;`number`[] \| `Uint8Array` \| `Float32Array` \| `Float64Array`&gt;
*** ***

View File

@@ -37,7 +37,7 @@ new TextEmbeddingFunction<M>(): TextEmbeddingFunction<M>
### computeQueryEmbeddings() ### computeQueryEmbeddings()
```ts ```ts
computeQueryEmbeddings(data): Promise<number[] | Float32Array | Float64Array> computeQueryEmbeddings(data): Promise<number[] | Uint8Array | Float32Array | Float64Array>
``` ```
Compute the embeddings for a single query Compute the embeddings for a single query
@@ -48,7 +48,7 @@ Compute the embeddings for a single query
#### Returns #### Returns
`Promise`&lt;`number`[] \| `Float32Array` \| `Float64Array`&gt; `Promise`&lt;`number`[] \| `Uint8Array` \| `Float32Array` \| `Float64Array`&gt;
#### Overrides #### Overrides

View File

@@ -7,5 +7,10 @@
# Type Alias: IntoVector # Type Alias: IntoVector
```ts ```ts
type IntoVector: Float32Array | Float64Array | number[] | Promise<Float32Array | Float64Array | number[]>; type IntoVector:
| Float32Array
| Float64Array
| Uint8Array
| number[]
| Promise<Float32Array | Float64Array | Uint8Array | number[]>;
``` ```

View File

@@ -36,6 +36,20 @@ is also an [asynchronous API client](#connections-asynchronous).
::: lancedb.table.Tags ::: lancedb.table.Tags
## Expressions
Type-safe expression builder for filters and projections. Use these instead
of raw SQL strings with [where][lancedb.query.LanceQueryBuilder.where] and
[select][lancedb.query.LanceQueryBuilder.select].
::: lancedb.expr.Expr
::: lancedb.expr.col
::: lancedb.expr.lit
::: lancedb.expr.func
## Querying (Synchronous) ## Querying (Synchronous)
::: lancedb.query.Query ::: lancedb.query.Query

View File

@@ -1,4 +1,4 @@
# LanceDB Java SDK # LanceDB Java Enterprise Client
## Configuration and Initialization ## Configuration and Initialization

View File

@@ -8,7 +8,7 @@
<parent> <parent>
<groupId>com.lancedb</groupId> <groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId> <artifactId>lancedb-parent</artifactId>
<version>0.27.0-beta.2</version> <version>0.27.2-final.0</version>
<relativePath>../pom.xml</relativePath> <relativePath>../pom.xml</relativePath>
</parent> </parent>
@@ -56,21 +56,21 @@
<dependency> <dependency>
<groupId>org.apache.logging.log4j</groupId> <groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId> <artifactId>log4j-slf4j2-impl</artifactId>
<version>2.24.3</version> <version>2.25.3</version>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.apache.logging.log4j</groupId> <groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId> <artifactId>log4j-core</artifactId>
<version>2.24.3</version> <version>2.25.3</version>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.apache.logging.log4j</groupId> <groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId> <artifactId>log4j-api</artifactId>
<version>2.24.3</version> <version>2.25.3</version>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
</dependencies> </dependencies>

View File

@@ -6,7 +6,7 @@
<groupId>com.lancedb</groupId> <groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId> <artifactId>lancedb-parent</artifactId>
<version>0.27.0-beta.2</version> <version>0.27.2-final.0</version>
<packaging>pom</packaging> <packaging>pom</packaging>
<name>${project.artifactId}</name> <name>${project.artifactId}</name>
<description>LanceDB Java SDK Parent POM</description> <description>LanceDB Java SDK Parent POM</description>
@@ -28,7 +28,7 @@
<properties> <properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<arrow.version>15.0.0</arrow.version> <arrow.version>15.0.0</arrow.version>
<lance-core.version>3.1.0-beta.2</lance-core.version> <lance-core.version>5.0.0-beta.4</lance-core.version>
<spotless.skip>false</spotless.skip> <spotless.skip>false</spotless.skip>
<spotless.version>2.30.0</spotless.version> <spotless.version>2.30.0</spotless.version>
<spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version> <spotless.java.googlejavaformat.version>1.7</spotless.java.googlejavaformat.version>
@@ -111,7 +111,7 @@
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId> <artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version> <version>3.3.1</version>
<executions> <executions>
<execution> <execution>
<id>attach-sources</id> <id>attach-sources</id>
@@ -124,7 +124,7 @@
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId> <artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version> <version>3.11.2</version>
<executions> <executions>
<execution> <execution>
<id>attach-javadocs</id> <id>attach-javadocs</id>
@@ -178,15 +178,15 @@
<plugins> <plugins>
<plugin> <plugin>
<artifactId>maven-clean-plugin</artifactId> <artifactId>maven-clean-plugin</artifactId>
<version>3.1.0</version> <version>3.4.1</version>
</plugin> </plugin>
<plugin> <plugin>
<artifactId>maven-resources-plugin</artifactId> <artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version> <version>3.3.1</version>
</plugin> </plugin>
<plugin> <plugin>
<artifactId>maven-compiler-plugin</artifactId> <artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version> <version>3.14.0</version>
<configuration> <configuration>
<compilerArgs> <compilerArgs>
<arg>-h</arg> <arg>-h</arg>
@@ -205,11 +205,11 @@
</plugin> </plugin>
<plugin> <plugin>
<artifactId>maven-jar-plugin</artifactId> <artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version> <version>3.4.2</version>
</plugin> </plugin>
<plugin> <plugin>
<artifactId>maven-install-plugin</artifactId> <artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version> <version>3.1.3</version>
</plugin> </plugin>
<plugin> <plugin>
<groupId>com.diffplug.spotless</groupId> <groupId>com.diffplug.spotless</groupId>
@@ -327,7 +327,7 @@
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId> <artifactId>maven-gpg-plugin</artifactId>
<version>1.5</version> <version>3.2.7</version>
<executions> <executions>
<execution> <execution>
<id>sign-artifacts</id> <id>sign-artifacts</id>

View File

@@ -1,7 +1,7 @@
[package] [package]
name = "lancedb-nodejs" name = "lancedb-nodejs"
edition.workspace = true edition.workspace = true
version = "0.27.0-beta.2" version = "0.27.2"
license.workspace = true license.workspace = true
description.workspace = true description.workspace = true
repository.workspace = true repository.workspace = true
@@ -15,6 +15,8 @@ crate-type = ["cdylib"]
async-trait.workspace = true async-trait.workspace = true
arrow-ipc.workspace = true arrow-ipc.workspace = true
arrow-array.workspace = true arrow-array.workspace = true
arrow-buffer = "57.2"
half.workspace = true
arrow-schema.workspace = true arrow-schema.workspace = true
env_logger.workspace = true env_logger.workspace = true
futures.workspace = true futures.workspace = true
@@ -25,12 +27,12 @@ napi = { version = "3.8.3", default-features = false, features = [
] } ] }
napi-derive = "3.5.2" napi-derive = "3.5.2"
# Prevent dynamic linking of lzma, which comes from datafusion # Prevent dynamic linking of lzma, which comes from datafusion
lzma-sys = { version = "*", features = ["static"] } lzma-sys = { version = "0.1", features = ["static"] }
log.workspace = true log.workspace = true
# Workaround for build failure until we can fix it. # Pin to resolve build failures; update periodically for security patches.
aws-lc-sys = "=0.28.0" aws-lc-sys = "=0.38.0"
aws-lc-rs = "=1.13.0" aws-lc-rs = "=1.16.1"
[build-dependencies] [build-dependencies]
napi-build = "2.3.1" napi-build = "2.3.1"

View File

@@ -63,6 +63,7 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
tableFromIPC, tableFromIPC,
DataType, DataType,
Dictionary, Dictionary,
Uint8: ArrowUint8,
// biome-ignore lint/suspicious/noExplicitAny: <explanation> // biome-ignore lint/suspicious/noExplicitAny: <explanation>
} = <any>arrow; } = <any>arrow;
type Schema = ApacheArrow["Schema"]; type Schema = ApacheArrow["Schema"];
@@ -362,6 +363,38 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
).toEqual(new Float64().toString()); ).toEqual(new Float64().toString());
}); });
it("will infer FixedSizeList<Float32> from Float32Array values", async function () {
const table = makeArrowTable([
{ id: "a", vector: new Float32Array([0.1, 0.2, 0.3]) },
{ id: "b", vector: new Float32Array([0.4, 0.5, 0.6]) },
]);
expect(DataType.isFixedSizeList(table.getChild("vector")?.type)).toBe(
true,
);
const vectorType = table.getChild("vector")?.type;
expect(vectorType.listSize).toBe(3);
expect(vectorType.children[0].type.toString()).toEqual(
new Float32().toString(),
);
});
it("will infer FixedSizeList<Uint8> from Uint8Array values", async function () {
const table = makeArrowTable([
{ id: "a", vector: new Uint8Array([1, 2, 3]) },
{ id: "b", vector: new Uint8Array([4, 5, 6]) },
]);
expect(DataType.isFixedSizeList(table.getChild("vector")?.type)).toBe(
true,
);
const vectorType = table.getChild("vector")?.type;
expect(vectorType.listSize).toBe(3);
expect(vectorType.children[0].type.toString()).toEqual(
new ArrowUint8().toString(),
);
});
it("will use dictionary encoded strings if asked", async function () { it("will use dictionary encoded strings if asked", async function () {
const table = makeArrowTable([{ str: "hello" }]); const table = makeArrowTable([{ str: "hello" }]);
expect(DataType.isUtf8(table.getChild("str")?.type)).toBe(true); expect(DataType.isUtf8(table.getChild("str")?.type)).toBe(true);

View File

@@ -103,7 +103,7 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
}, },
numIndices: 0, numIndices: 0,
numRows: 3, numRows: 3,
totalBytes: 24, totalBytes: 44,
}); });
}); });
@@ -1259,6 +1259,98 @@ describe("schema evolution", function () {
expect(await table.schema()).toEqual(expectedSchema); expect(await table.schema()).toEqual(expectedSchema);
}); });
it("can add columns with schema for explicit data types", async function () {
const con = await connect(tmpDir.name);
const table = await con.createTable("vectors", [
{ id: 1n, vector: [0.1, 0.2] },
]);
// Define schema for new columns with explicit data types
// Note: All columns must be nullable when using addColumns with Schema
// because they are initially populated with null values
const newColumnsSchema = new Schema([
new Field("price", new Float64(), true),
new Field("category", new Utf8(), true),
new Field("rating", new Int32(), true),
]);
const result = await table.addColumns(newColumnsSchema);
expect(result).toHaveProperty("version");
expect(result.version).toBe(2);
const expectedSchema = new Schema([
new Field("id", new Int64(), true),
new Field(
"vector",
new FixedSizeList(2, new Field("item", new Float32(), true)),
true,
),
new Field("price", new Float64(), true),
new Field("category", new Utf8(), true),
new Field("rating", new Int32(), true),
]);
expect(await table.schema()).toEqual(expectedSchema);
// Verify that new columns are populated with null values
const results = await table.query().toArray();
expect(results).toHaveLength(1);
expect(results[0].price).toBeNull();
expect(results[0].category).toBeNull();
expect(results[0].rating).toBeNull();
});
it("can add a single column using Field", async function () {
const con = await connect(tmpDir.name);
const table = await con.createTable("vectors", [
{ id: 1n, vector: [0.1, 0.2] },
]);
// Add a single field
const priceField = new Field("price", new Float64(), true);
const result = await table.addColumns(priceField);
expect(result).toHaveProperty("version");
expect(result.version).toBe(2);
const expectedSchema = new Schema([
new Field("id", new Int64(), true),
new Field(
"vector",
new FixedSizeList(2, new Field("item", new Float32(), true)),
true,
),
new Field("price", new Float64(), true),
]);
expect(await table.schema()).toEqual(expectedSchema);
});
it("can add multiple columns using array of Fields", async function () {
const con = await connect(tmpDir.name);
const table = await con.createTable("vectors", [
{ id: 1n, vector: [0.1, 0.2] },
]);
// Add multiple fields as array
const fields = [
new Field("price", new Float64(), true),
new Field("category", new Utf8(), true),
];
const result = await table.addColumns(fields);
expect(result).toHaveProperty("version");
expect(result.version).toBe(2);
const expectedSchema = new Schema([
new Field("id", new Int64(), true),
new Field(
"vector",
new FixedSizeList(2, new Field("item", new Float32(), true)),
true,
),
new Field("price", new Float64(), true),
new Field("category", new Utf8(), true),
]);
expect(await table.schema()).toEqual(expectedSchema);
});
it("can alter the columns in the schema", async function () { it("can alter the columns in the schema", async function () {
const con = await connect(tmpDir.name); const con = await connect(tmpDir.name);
const schema = new Schema([ const schema = new Schema([
@@ -1697,6 +1789,65 @@ describe.each([arrow15, arrow16, arrow17, arrow18])(
expect(results2[0].text).toBe(data[1].text); expect(results2[0].text).toBe(data[1].text);
}); });
test("full text search fast search", async () => {
const db = await connect(tmpDir.name);
const data = [{ text: "hello world", vector: [0.1, 0.2, 0.3], id: 1 }];
const table = await db.createTable("test", data);
await table.createIndex("text", {
config: Index.fts(),
});
// Insert unindexed data after creating the index.
await table.add([{ text: "xyz", vector: [0.4, 0.5, 0.6], id: 2 }]);
const withFlatSearch = await table
.search("xyz", "fts")
.limit(10)
.toArray();
expect(withFlatSearch.length).toBeGreaterThan(0);
const fastSearchResults = await table
.search("xyz", "fts")
.fastSearch()
.limit(10)
.toArray();
expect(fastSearchResults.length).toBe(0);
const nearestToTextFastSearch = await table
.query()
.nearestToText("xyz")
.fastSearch()
.limit(10)
.toArray();
expect(nearestToTextFastSearch.length).toBe(0);
// fastSearch should be chainable with other methods.
const chainedFastSearch = await table
.search("xyz", "fts")
.fastSearch()
.select(["text"])
.limit(5)
.toArray();
expect(chainedFastSearch.length).toBe(0);
await table.optimize();
const indexedFastSearch = await table
.search("xyz", "fts")
.fastSearch()
.limit(10)
.toArray();
expect(indexedFastSearch.length).toBeGreaterThan(0);
const indexedNearestToTextFastSearch = await table
.query()
.nearestToText("xyz")
.fastSearch()
.limit(10)
.toArray();
expect(indexedNearestToTextFastSearch.length).toBeGreaterThan(0);
});
test("prewarm full text search index", async () => { test("prewarm full text search index", async () => {
const db = await connect(tmpDir.name); const db = await connect(tmpDir.name);
const data = [ const data = [
@@ -2145,3 +2296,36 @@ describe("when creating an empty table", () => {
expect((actualSchema.fields[1].type as Float64).precision).toBe(2); expect((actualSchema.fields[1].type as Float64).precision).toBe(2);
}); });
}); });
// Ensure we can create float32 arrays without using Arrow
// by utilizing native JS TypedArray support
//
// https://github.com/lancedb/lancedb/issues/3115
describe("when creating a table with Float32Array vectors", () => {
let tmpDir: tmp.DirResult;
beforeEach(() => {
tmpDir = tmp.dirSync({ unsafeCleanup: true });
});
afterEach(() => {
tmpDir.removeCallback();
});
it("should persist Float32Array as FixedSizeList<Float32> in the LanceDB schema", async () => {
const db = await connect(tmpDir.name);
const table = await db.createTable("test", [
{ id: "a", vector: new Float32Array([0.1, 0.2, 0.3]) },
{ id: "b", vector: new Float32Array([0.4, 0.5, 0.6]) },
]);
const schema = await table.schema();
const vectorField = schema.fields.find((f) => f.name === "vector");
expect(vectorField).toBeDefined();
expect(vectorField!.type).toBeInstanceOf(FixedSizeList);
const fsl = vectorField!.type as FixedSizeList;
expect(fsl.listSize).toBe(3);
expect(fsl.children[0].type.typeId).toBe(Type.Float);
// precision: HALF=0, SINGLE=1, DOUBLE=2
expect((fsl.children[0].type as Float32).precision).toBe(1);
});
});

View File

@@ -0,0 +1,110 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
import * as tmp from "tmp";
import { type Table, connect } from "../lancedb";
import {
Field,
FixedSizeList,
Float32,
Int64,
Schema,
makeArrowTable,
} from "../lancedb/arrow";
describe("Vector query with different typed arrays", () => {
let tmpDir: tmp.DirResult;
afterEach(() => {
tmpDir?.removeCallback();
});
async function createFloat32Table(): Promise<Table> {
tmpDir = tmp.dirSync({ unsafeCleanup: true });
const db = await connect(tmpDir.name);
const schema = new Schema([
new Field("id", new Int64(), true),
new Field(
"vec",
new FixedSizeList(2, new Field("item", new Float32())),
true,
),
]);
const data = makeArrowTable(
[
{ id: 1n, vec: [1.0, 0.0] },
{ id: 2n, vec: [0.0, 1.0] },
{ id: 3n, vec: [1.0, 1.0] },
],
{ schema },
);
return db.createTable("test_f32", data);
}
it("should search with Float32Array (baseline)", async () => {
const table = await createFloat32Table();
const results = await table
.query()
.nearestTo(new Float32Array([1.0, 0.0]))
.limit(1)
.toArray();
expect(results.length).toBe(1);
expect(Number(results[0].id)).toBe(1);
});
it("should search with number[] (backward compat)", async () => {
const table = await createFloat32Table();
const results = await table
.query()
.nearestTo([1.0, 0.0])
.limit(1)
.toArray();
expect(results.length).toBe(1);
expect(Number(results[0].id)).toBe(1);
});
it("should search with Float64Array via raw path", async () => {
const table = await createFloat32Table();
const results = await table
.query()
.nearestTo(new Float64Array([1.0, 0.0]))
.limit(1)
.toArray();
expect(results.length).toBe(1);
expect(Number(results[0].id)).toBe(1);
});
it("should add multiple query vectors with Float64Array", async () => {
const table = await createFloat32Table();
const results = await table
.query()
.nearestTo(new Float64Array([1.0, 0.0]))
.addQueryVector(new Float64Array([0.0, 1.0]))
.limit(2)
.toArray();
expect(results.length).toBeGreaterThanOrEqual(2);
});
// Float16Array is only available in Node 22+; not in TypeScript's standard lib yet
const float16ArrayCtor = (globalThis as unknown as Record<string, unknown>)
.Float16Array as (new (values: number[]) => unknown) | undefined;
const hasFloat16 = float16ArrayCtor !== undefined;
const f16it = hasFloat16 ? it : it.skip;
f16it("should search with Float16Array via raw path", async () => {
const table = await createFloat32Table();
const results = await table
.query()
.nearestTo(new float16ArrayCtor!([1.0, 0.0]) as Float32Array)
.limit(1)
.toArray();
expect(results.length).toBe(1);
expect(Number(results[0].id)).toBe(1);
});
});

View File

@@ -30,12 +30,15 @@
"x64", "x64",
"arm64" "arm64"
], ],
"dev": true,
"license": "Apache-2.0", "license": "Apache-2.0",
"optional": true,
"os": [ "os": [
"darwin", "darwin",
"linux", "linux",
"win32" "win32"
], ],
"peer": true,
"dependencies": { "dependencies": {
"reflect-metadata": "^0.2.2" "reflect-metadata": "^0.2.2"
}, },
@@ -91,14 +94,15 @@
} }
}, },
"node_modules/@babel/code-frame": { "node_modules/@babel/code-frame": {
"version": "7.26.2", "version": "7.29.0",
"resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.26.2.tgz", "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.0.tgz",
"integrity": "sha512-RJlIHRueQgwWitWgF8OdFYGZX328Ax5BCemNGlqHfplnRT9ESi8JkFlvaVYbS+UubVY6dpv87Fs2u5M29iNFVQ==", "integrity": "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw==",
"dev": true, "dev": true,
"license": "MIT",
"dependencies": { "dependencies": {
"@babel/helper-validator-identifier": "^7.25.9", "@babel/helper-validator-identifier": "^7.28.5",
"js-tokens": "^4.0.0", "js-tokens": "^4.0.0",
"picocolors": "^1.0.0" "picocolors": "^1.1.1"
}, },
"engines": { "engines": {
"node": ">=6.9.0" "node": ">=6.9.0"
@@ -233,19 +237,21 @@
} }
}, },
"node_modules/@babel/helper-string-parser": { "node_modules/@babel/helper-string-parser": {
"version": "7.25.9", "version": "7.27.1",
"resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.25.9.tgz", "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.27.1.tgz",
"integrity": "sha512-4A/SCr/2KLd5jrtOMFzaKjVtAei3+2r/NChoBNoZ3EyP/+GlhoaEGoWOZUmFmoITP7zOJyHIMm+DYRd8o3PvHA==", "integrity": "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA==",
"dev": true, "dev": true,
"license": "MIT",
"engines": { "engines": {
"node": ">=6.9.0" "node": ">=6.9.0"
} }
}, },
"node_modules/@babel/helper-validator-identifier": { "node_modules/@babel/helper-validator-identifier": {
"version": "7.25.9", "version": "7.28.5",
"resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.25.9.tgz", "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.28.5.tgz",
"integrity": "sha512-Ed61U6XJc3CVRfkERJWDz4dJwKe7iLmmJsbOGu9wSloNSFttHV0I8g6UAgb7qnK5ly5bGLPd4oXZlxCdANBOWQ==", "integrity": "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==",
"dev": true, "dev": true,
"license": "MIT",
"engines": { "engines": {
"node": ">=6.9.0" "node": ">=6.9.0"
} }
@@ -260,25 +266,27 @@
} }
}, },
"node_modules/@babel/helpers": { "node_modules/@babel/helpers": {
"version": "7.26.0", "version": "7.28.6",
"resolved": "https://registry.npmjs.org/@babel/helpers/-/helpers-7.26.0.tgz", "resolved": "https://registry.npmjs.org/@babel/helpers/-/helpers-7.28.6.tgz",
"integrity": "sha512-tbhNuIxNcVb21pInl3ZSjksLCvgdZy9KwJ8brv993QtIVKJBBkYXz4q4ZbAv31GdnC+R90np23L5FbEBlthAEw==", "integrity": "sha512-xOBvwq86HHdB7WUDTfKfT/Vuxh7gElQ+Sfti2Cy6yIWNW05P8iUslOVcZ4/sKbE+/jQaukQAdz/gf3724kYdqw==",
"dev": true, "dev": true,
"license": "MIT",
"dependencies": { "dependencies": {
"@babel/template": "^7.25.9", "@babel/template": "^7.28.6",
"@babel/types": "^7.26.0" "@babel/types": "^7.28.6"
}, },
"engines": { "engines": {
"node": ">=6.9.0" "node": ">=6.9.0"
} }
}, },
"node_modules/@babel/parser": { "node_modules/@babel/parser": {
"version": "7.26.2", "version": "7.29.0",
"resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.26.2.tgz", "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.0.tgz",
"integrity": "sha512-DWMCZH9WA4Maitz2q21SRKHo9QXZxkDsbNZoVD62gusNtNBBqDg9i7uOhASfTfIGNzW+O+r7+jAlM8dwphcJKQ==", "integrity": "sha512-IyDgFV5GeDUVX4YdF/3CPULtVGSXXMLh1xVIgdCgxApktqnQV0r7/8Nqthg+8YLGaAtdyIlo2qIdZrbCv4+7ww==",
"dev": true, "dev": true,
"license": "MIT",
"dependencies": { "dependencies": {
"@babel/types": "^7.26.0" "@babel/types": "^7.29.0"
}, },
"bin": { "bin": {
"parser": "bin/babel-parser.js" "parser": "bin/babel-parser.js"
@@ -510,14 +518,15 @@
} }
}, },
"node_modules/@babel/template": { "node_modules/@babel/template": {
"version": "7.25.9", "version": "7.28.6",
"resolved": "https://registry.npmjs.org/@babel/template/-/template-7.25.9.tgz", "resolved": "https://registry.npmjs.org/@babel/template/-/template-7.28.6.tgz",
"integrity": "sha512-9DGttpmPvIxBb/2uwpVo3dqJ+O6RooAFOS+lB+xDqoE2PVCE8nfoHMdZLpfCQRLwvohzXISPZcgxt80xLfsuwg==", "integrity": "sha512-YA6Ma2KsCdGb+WC6UpBVFJGXL58MDA6oyONbjyF/+5sBgxY/dwkhLogbMT2GXXyU84/IhRw/2D1Os1B/giz+BQ==",
"dev": true, "dev": true,
"license": "MIT",
"dependencies": { "dependencies": {
"@babel/code-frame": "^7.25.9", "@babel/code-frame": "^7.28.6",
"@babel/parser": "^7.25.9", "@babel/parser": "^7.28.6",
"@babel/types": "^7.25.9" "@babel/types": "^7.28.6"
}, },
"engines": { "engines": {
"node": ">=6.9.0" "node": ">=6.9.0"
@@ -542,13 +551,14 @@
} }
}, },
"node_modules/@babel/types": { "node_modules/@babel/types": {
"version": "7.26.0", "version": "7.29.0",
"resolved": "https://registry.npmjs.org/@babel/types/-/types-7.26.0.tgz", "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.0.tgz",
"integrity": "sha512-Z/yiTPj+lDVnF7lWeKCIJzaIkI0vYO87dMpZ4bg4TDrFe4XXLFWL1TbXU27gBP3QccxV9mZICCrnjnYlJjXHOA==", "integrity": "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A==",
"dev": true, "dev": true,
"license": "MIT",
"dependencies": { "dependencies": {
"@babel/helper-string-parser": "^7.25.9", "@babel/helper-string-parser": "^7.27.1",
"@babel/helper-validator-identifier": "^7.25.9" "@babel/helper-validator-identifier": "^7.28.5"
}, },
"engines": { "engines": {
"node": ">=6.9.0" "node": ">=6.9.0"
@@ -1151,95 +1161,6 @@
"url": "https://opencollective.com/libvips" "url": "https://opencollective.com/libvips"
} }
}, },
"node_modules/@isaacs/cliui": {
"version": "8.0.2",
"resolved": "https://registry.npmjs.org/@isaacs/cliui/-/cliui-8.0.2.tgz",
"integrity": "sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==",
"dependencies": {
"string-width": "^5.1.2",
"string-width-cjs": "npm:string-width@^4.2.0",
"strip-ansi": "^7.0.1",
"strip-ansi-cjs": "npm:strip-ansi@^6.0.1",
"wrap-ansi": "^8.1.0",
"wrap-ansi-cjs": "npm:wrap-ansi@^7.0.0"
},
"engines": {
"node": ">=12"
}
},
"node_modules/@isaacs/cliui/node_modules/ansi-regex": {
"version": "6.1.0",
"resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-6.1.0.tgz",
"integrity": "sha512-7HSX4QQb4CspciLpVFwyRe79O3xsIZDDLER21kERQ71oaPodF8jL725AgJMFAYbooIqolJoRLuM81SpeUkpkvA==",
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/ansi-regex?sponsor=1"
}
},
"node_modules/@isaacs/cliui/node_modules/ansi-styles": {
"version": "6.2.1",
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-6.2.1.tgz",
"integrity": "sha512-bN798gFfQX+viw3R7yrGWRqnrN2oRkEkUjjl4JNn4E8GxxbjtG3FbrEIIY3l8/hrwUwIeCZvi4QuOTP4MErVug==",
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/ansi-styles?sponsor=1"
}
},
"node_modules/@isaacs/cliui/node_modules/emoji-regex": {
"version": "9.2.2",
"resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-9.2.2.tgz",
"integrity": "sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg=="
},
"node_modules/@isaacs/cliui/node_modules/string-width": {
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/string-width/-/string-width-5.1.2.tgz",
"integrity": "sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA==",
"dependencies": {
"eastasianwidth": "^0.2.0",
"emoji-regex": "^9.2.2",
"strip-ansi": "^7.0.1"
},
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/@isaacs/cliui/node_modules/strip-ansi": {
"version": "7.1.0",
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-7.1.0.tgz",
"integrity": "sha512-iq6eVVI64nQQTRYq2KtEg2d2uU7LElhTJwsH4YzIHZshxlgZms/wIc4VoDQTlG/IvVIrBKG06CrZnp0qv7hkcQ==",
"dependencies": {
"ansi-regex": "^6.0.1"
},
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/strip-ansi?sponsor=1"
}
},
"node_modules/@isaacs/cliui/node_modules/wrap-ansi": {
"version": "8.1.0",
"resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-8.1.0.tgz",
"integrity": "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ==",
"dependencies": {
"ansi-styles": "^6.1.0",
"string-width": "^5.0.1",
"strip-ansi": "^7.0.1"
},
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
}
},
"node_modules/@isaacs/fs-minipass": { "node_modules/@isaacs/fs-minipass": {
"version": "4.0.1", "version": "4.0.1",
"resolved": "https://registry.npmjs.org/@isaacs/fs-minipass/-/fs-minipass-4.0.1.tgz", "resolved": "https://registry.npmjs.org/@isaacs/fs-minipass/-/fs-minipass-4.0.1.tgz",
@@ -1606,15 +1527,6 @@
"resolved": "../dist", "resolved": "../dist",
"link": true "link": true
}, },
"node_modules/@pkgjs/parseargs": {
"version": "0.11.0",
"resolved": "https://registry.npmjs.org/@pkgjs/parseargs/-/parseargs-0.11.0.tgz",
"integrity": "sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==",
"optional": true,
"engines": {
"node": ">=14"
}
},
"node_modules/@protobufjs/aspromise": { "node_modules/@protobufjs/aspromise": {
"version": "1.1.2", "version": "1.1.2",
"resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz",
@@ -1846,6 +1758,7 @@
"version": "5.0.1", "version": "5.0.1",
"resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz",
"integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==",
"dev": true,
"engines": { "engines": {
"node": ">=8" "node": ">=8"
} }
@@ -1854,6 +1767,7 @@
"version": "4.3.0", "version": "4.3.0",
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz", "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz",
"integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==", "integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==",
"dev": true,
"dependencies": { "dependencies": {
"color-convert": "^2.0.1" "color-convert": "^2.0.1"
}, },
@@ -2019,13 +1933,15 @@
"node_modules/balanced-match": { "node_modules/balanced-match": {
"version": "1.0.2", "version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==" "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==",
"dev": true
}, },
"node_modules/brace-expansion": { "node_modules/brace-expansion": {
"version": "1.1.11", "version": "1.1.12",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz", "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.12.tgz",
"integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==", "integrity": "sha512-9T9UjW3r0UW5c1Q7GTwllptXwhvYmEzFhzMfZ9H7FQWt+uZePjZPjBP/W1ZEyZ1twGWom5/56TF4lPcqjnDHcg==",
"dev": true, "dev": true,
"license": "MIT",
"dependencies": { "dependencies": {
"balanced-match": "^1.0.0", "balanced-match": "^1.0.0",
"concat-map": "0.0.1" "concat-map": "0.0.1"
@@ -2102,6 +2018,19 @@
"integrity": "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ==", "integrity": "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ==",
"dev": true "dev": true
}, },
"node_modules/call-bind-apply-helpers": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",
"integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0",
"function-bind": "^1.1.2"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/callsites": { "node_modules/callsites": {
"version": "3.1.0", "version": "3.1.0",
"resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz", "resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz",
@@ -2298,9 +2227,11 @@
} }
}, },
"node_modules/cross-spawn": { "node_modules/cross-spawn": {
"version": "7.0.3", "version": "7.0.6",
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.3.tgz", "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
"integrity": "sha512-iRDPJKUPVEND7dHPO8rkbOnPpyDygcDFtWjpeWNCgy8WP2rXcxXL8TskReQl6OrB2G7+UJrags1q15Fudc7G6w==", "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==",
"dev": true,
"license": "MIT",
"dependencies": { "dependencies": {
"path-key": "^3.1.0", "path-key": "^3.1.0",
"shebang-command": "^2.0.0", "shebang-command": "^2.0.0",
@@ -2384,10 +2315,19 @@
"node": "^14.15.0 || ^16.10.0 || >=18.0.0" "node": "^14.15.0 || ^16.10.0 || >=18.0.0"
} }
}, },
"node_modules/eastasianwidth": { "node_modules/dunder-proto": {
"version": "0.2.0", "version": "1.0.1",
"resolved": "https://registry.npmjs.org/eastasianwidth/-/eastasianwidth-0.2.0.tgz", "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
"integrity": "sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA==" "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==",
"license": "MIT",
"dependencies": {
"call-bind-apply-helpers": "^1.0.1",
"es-errors": "^1.3.0",
"gopd": "^1.2.0"
},
"engines": {
"node": ">= 0.4"
}
}, },
"node_modules/ejs": { "node_modules/ejs": {
"version": "3.1.10", "version": "3.1.10",
@@ -2425,7 +2365,8 @@
"node_modules/emoji-regex": { "node_modules/emoji-regex": {
"version": "8.0.0", "version": "8.0.0",
"resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz",
"integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==" "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==",
"dev": true
}, },
"node_modules/error-ex": { "node_modules/error-ex": {
"version": "1.3.2", "version": "1.3.2",
@@ -2442,6 +2383,51 @@
"integrity": "sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg==", "integrity": "sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg==",
"dev": true "dev": true
}, },
"node_modules/es-define-property": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",
"integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
}
},
"node_modules/es-errors": {
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz",
"integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
}
},
"node_modules/es-object-atoms": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz",
"integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/es-set-tostringtag": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz",
"integrity": "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0",
"get-intrinsic": "^1.2.6",
"has-tostringtag": "^1.0.2",
"hasown": "^2.0.2"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/escalade": { "node_modules/escalade": {
"version": "3.2.0", "version": "3.2.0",
"resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz",
@@ -2554,19 +2540,21 @@
} }
}, },
"node_modules/filelist/node_modules/brace-expansion": { "node_modules/filelist/node_modules/brace-expansion": {
"version": "2.0.1", "version": "2.0.2",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.0.1.tgz", "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.0.2.tgz",
"integrity": "sha512-XnAIvQ8eM+kC6aULx6wuQiwVsnzsi9d3WxzV3FpWTGA19F621kwdbsAcFKXgKUHZWsy+mY6iL1sHTxWEFCytDA==", "integrity": "sha512-Jt0vHyM+jmUBqojB7E1NIYadt0vI0Qxjxd2TErW94wDz+E2LAm5vKMXXwg6ZZBTHPuUlDgQHKXvjGBdfcF1ZDQ==",
"dev": true, "dev": true,
"license": "MIT",
"dependencies": { "dependencies": {
"balanced-match": "^1.0.0" "balanced-match": "^1.0.0"
} }
}, },
"node_modules/filelist/node_modules/minimatch": { "node_modules/filelist/node_modules/minimatch": {
"version": "5.1.6", "version": "5.1.9",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-5.1.6.tgz", "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-5.1.9.tgz",
"integrity": "sha512-lKwV/1brpG6mBUFHtb7NUmtABCb2WZZmm2wNiOA5hAb8VdCS4B3dtMWyvcoViccwAW/COERjXLt0zP1zXUN26g==", "integrity": "sha512-7o1wEA2RyMP7Iu7GNba9vc0RWWGACJOCZBJX2GJWip0ikV+wcOsgVuY9uE8CPiyQhkGFSlhuSkZPavN7u1c2Fw==",
"dev": true, "dev": true,
"license": "ISC",
"dependencies": { "dependencies": {
"brace-expansion": "^2.0.1" "brace-expansion": "^2.0.1"
}, },
@@ -2604,39 +2592,16 @@
"resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-1.12.0.tgz", "resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-1.12.0.tgz",
"integrity": "sha512-c7CZADjRcl6j0PlvFy0ZqXQ67qSEZfrVPynmnL+2zPc+NtMvrF8Y0QceMo7QqnSPc7+uWjUIAbvCQ5WIKlMVdQ==" "integrity": "sha512-c7CZADjRcl6j0PlvFy0ZqXQ67qSEZfrVPynmnL+2zPc+NtMvrF8Y0QceMo7QqnSPc7+uWjUIAbvCQ5WIKlMVdQ=="
}, },
"node_modules/foreground-child": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/foreground-child/-/foreground-child-3.3.0.tgz",
"integrity": "sha512-Ld2g8rrAyMYFXBhEqMz8ZAHBi4J4uS1i/CxGMDnjyFWddMXLVcDp051DZfu+t7+ab7Wv6SMqpWmyFIj5UbfFvg==",
"dependencies": {
"cross-spawn": "^7.0.0",
"signal-exit": "^4.0.1"
},
"engines": {
"node": ">=14"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/foreground-child/node_modules/signal-exit": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-4.1.0.tgz",
"integrity": "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==",
"engines": {
"node": ">=14"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/form-data": { "node_modules/form-data": {
"version": "4.0.1", "version": "4.0.5",
"resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.1.tgz", "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.5.tgz",
"integrity": "sha512-tzN8e4TX8+kkxGPK8D5u0FNmjPUjw3lwC9lSLxxoB/+GtsJG91CO8bSWy73APlgAZzZbXEYZJuxjkHH2w+Ezhw==", "integrity": "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w==",
"license": "MIT",
"dependencies": { "dependencies": {
"asynckit": "^0.4.0", "asynckit": "^0.4.0",
"combined-stream": "^1.0.8", "combined-stream": "^1.0.8",
"es-set-tostringtag": "^2.1.0",
"hasown": "^2.0.2",
"mime-types": "^2.1.12" "mime-types": "^2.1.12"
}, },
"engines": { "engines": {
@@ -2684,7 +2649,6 @@
"version": "1.1.2", "version": "1.1.2",
"resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
"integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
"dev": true,
"funding": { "funding": {
"url": "https://github.com/sponsors/ljharb" "url": "https://github.com/sponsors/ljharb"
} }
@@ -2707,6 +2671,30 @@
"node": "6.* || 8.* || >= 10.*" "node": "6.* || 8.* || >= 10.*"
} }
}, },
"node_modules/get-intrinsic": {
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz",
"integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==",
"license": "MIT",
"dependencies": {
"call-bind-apply-helpers": "^1.0.2",
"es-define-property": "^1.0.1",
"es-errors": "^1.3.0",
"es-object-atoms": "^1.1.1",
"function-bind": "^1.1.2",
"get-proto": "^1.0.1",
"gopd": "^1.2.0",
"has-symbols": "^1.1.0",
"hasown": "^2.0.2",
"math-intrinsics": "^1.1.0"
},
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/get-package-type": { "node_modules/get-package-type": {
"version": "0.1.0", "version": "0.1.0",
"resolved": "https://registry.npmjs.org/get-package-type/-/get-package-type-0.1.0.tgz", "resolved": "https://registry.npmjs.org/get-package-type/-/get-package-type-0.1.0.tgz",
@@ -2716,6 +2704,19 @@
"node": ">=8.0.0" "node": ">=8.0.0"
} }
}, },
"node_modules/get-proto": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz",
"integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==",
"license": "MIT",
"dependencies": {
"dunder-proto": "^1.0.1",
"es-object-atoms": "^1.0.0"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/get-stream": { "node_modules/get-stream": {
"version": "6.0.1", "version": "6.0.1",
"resolved": "https://registry.npmjs.org/get-stream/-/get-stream-6.0.1.tgz", "resolved": "https://registry.npmjs.org/get-stream/-/get-stream-6.0.1.tgz",
@@ -2758,6 +2759,18 @@
"node": ">=4" "node": ">=4"
} }
}, },
"node_modules/gopd": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
"integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/graceful-fs": { "node_modules/graceful-fs": {
"version": "4.2.11", "version": "4.2.11",
"resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz", "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz",
@@ -2778,11 +2791,37 @@
"node": ">=8" "node": ">=8"
} }
}, },
"node_modules/has-symbols": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz",
"integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/has-tostringtag": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/has-tostringtag/-/has-tostringtag-1.0.2.tgz",
"integrity": "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==",
"license": "MIT",
"dependencies": {
"has-symbols": "^1.0.3"
},
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/hasown": { "node_modules/hasown": {
"version": "2.0.2", "version": "2.0.2",
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz", "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
"integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==", "integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==",
"dev": true,
"dependencies": { "dependencies": {
"function-bind": "^1.1.2" "function-bind": "^1.1.2"
}, },
@@ -2882,6 +2921,7 @@
"version": "3.0.0", "version": "3.0.0",
"resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz",
"integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==",
"dev": true,
"engines": { "engines": {
"node": ">=8" "node": ">=8"
} }
@@ -2919,7 +2959,8 @@
"node_modules/isexe": { "node_modules/isexe": {
"version": "2.0.0", "version": "2.0.0",
"resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz",
"integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==" "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==",
"dev": true
}, },
"node_modules/istanbul-lib-coverage": { "node_modules/istanbul-lib-coverage": {
"version": "3.2.2", "version": "3.2.2",
@@ -2987,20 +3028,6 @@
"node": ">=8" "node": ">=8"
} }
}, },
"node_modules/jackspeak": {
"version": "3.4.3",
"resolved": "https://registry.npmjs.org/jackspeak/-/jackspeak-3.4.3.tgz",
"integrity": "sha512-OGlZQpz2yfahA/Rd1Y8Cd9SIEsqvXkLVoSw/cgwhnhFMDbsQFeZYoJJ7bIZBS9BcamUW96asq/npPWugM+RQBw==",
"dependencies": {
"@isaacs/cliui": "^8.0.2"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
},
"optionalDependencies": {
"@pkgjs/parseargs": "^0.11.0"
}
},
"node_modules/jake": { "node_modules/jake": {
"version": "10.9.2", "version": "10.9.2",
"resolved": "https://registry.npmjs.org/jake/-/jake-10.9.2.tgz", "resolved": "https://registry.npmjs.org/jake/-/jake-10.9.2.tgz",
@@ -3605,10 +3632,11 @@
"dev": true "dev": true
}, },
"node_modules/js-yaml": { "node_modules/js-yaml": {
"version": "3.14.1", "version": "3.14.2",
"resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.14.1.tgz", "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.14.2.tgz",
"integrity": "sha512-okMH7OXXJ7YrN9Ok3/SXrnu4iX9yOk+25nqX4imS2npuvTYDmo/QEZoqwZkYaIDk3jVvBOTOIEgEhaLOynBS9g==", "integrity": "sha512-PMSmkqxr106Xa156c2M265Z+FTrPl+oxd/rgOQy2tijQeK5TxQ43psO1ZCwhVOSdnn+RzkzlRz/eY4BgJBYVpg==",
"dev": true, "dev": true,
"license": "MIT",
"dependencies": { "dependencies": {
"argparse": "^1.0.7", "argparse": "^1.0.7",
"esprima": "^4.0.0" "esprima": "^4.0.0"
@@ -3728,6 +3756,15 @@
"tmpl": "1.0.5" "tmpl": "1.0.5"
} }
}, },
"node_modules/math-intrinsics": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
"integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
}
},
"node_modules/merge-stream": { "node_modules/merge-stream": {
"version": "2.0.0", "version": "2.0.0",
"resolved": "https://registry.npmjs.org/merge-stream/-/merge-stream-2.0.0.tgz", "resolved": "https://registry.npmjs.org/merge-stream/-/merge-stream-2.0.0.tgz",
@@ -3776,10 +3813,11 @@
} }
}, },
"node_modules/minimatch": { "node_modules/minimatch": {
"version": "3.1.2", "version": "3.1.5",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.2.tgz", "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.5.tgz",
"integrity": "sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw==", "integrity": "sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==",
"dev": true, "dev": true,
"license": "ISC",
"dependencies": { "dependencies": {
"brace-expansion": "^1.1.7" "brace-expansion": "^1.1.7"
}, },
@@ -3796,31 +3834,17 @@
} }
}, },
"node_modules/minizlib": { "node_modules/minizlib": {
"version": "3.0.1", "version": "3.1.0",
"resolved": "https://registry.npmjs.org/minizlib/-/minizlib-3.0.1.tgz", "resolved": "https://registry.npmjs.org/minizlib/-/minizlib-3.1.0.tgz",
"integrity": "sha512-umcy022ILvb5/3Djuu8LWeqUa8D68JaBzlttKeMWen48SjabqS3iY5w/vzeMzMUNhLDifyhbOwKDSznB1vvrwg==", "integrity": "sha512-KZxYo1BUkWD2TVFLr0MQoM8vUUigWD3LlD83a/75BqC+4qE0Hb1Vo5v1FgcfaNXvfXzr+5EhQ6ing/CaBijTlw==",
"license": "MIT",
"dependencies": { "dependencies": {
"minipass": "^7.0.4", "minipass": "^7.1.2"
"rimraf": "^5.0.5"
}, },
"engines": { "engines": {
"node": ">= 18" "node": ">= 18"
} }
}, },
"node_modules/mkdirp": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-3.0.1.tgz",
"integrity": "sha512-+NsyUUAZDmo6YVHzL/stxSu3t9YS1iljliy3BSDrXJ/dkn1KYdmtZODGGjLcc9XLgVVpH4KshHB8XmZgMhaBXg==",
"bin": {
"mkdirp": "dist/cjs/src/bin.js"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/ms": { "node_modules/ms": {
"version": "2.1.3", "version": "2.1.3",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
@@ -4010,11 +4034,6 @@
"node": ">=6" "node": ">=6"
} }
}, },
"node_modules/package-json-from-dist": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/package-json-from-dist/-/package-json-from-dist-1.0.1.tgz",
"integrity": "sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw=="
},
"node_modules/parse-json": { "node_modules/parse-json": {
"version": "5.2.0", "version": "5.2.0",
"resolved": "https://registry.npmjs.org/parse-json/-/parse-json-5.2.0.tgz", "resolved": "https://registry.npmjs.org/parse-json/-/parse-json-5.2.0.tgz",
@@ -4055,6 +4074,7 @@
"version": "3.1.1", "version": "3.1.1",
"resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz",
"integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==", "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==",
"dev": true,
"engines": { "engines": {
"node": ">=8" "node": ">=8"
} }
@@ -4065,26 +4085,6 @@
"integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==", "integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==",
"dev": true "dev": true
}, },
"node_modules/path-scurry": {
"version": "1.11.1",
"resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-1.11.1.tgz",
"integrity": "sha512-Xa4Nw17FS9ApQFJ9umLiJS4orGjm7ZzwUrwamcGQuHSzDyth9boKDaycYdDcZDuqYATXw4HFXgaqWTctW/v1HA==",
"dependencies": {
"lru-cache": "^10.2.0",
"minipass": "^5.0.0 || ^6.0.2 || ^7.0.0"
},
"engines": {
"node": ">=16 || 14 >=14.18"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/path-scurry/node_modules/lru-cache": {
"version": "10.4.3",
"resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-10.4.3.tgz",
"integrity": "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="
},
"node_modules/picocolors": { "node_modules/picocolors": {
"version": "1.1.1", "version": "1.1.1",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
@@ -4246,61 +4246,6 @@
"node": ">=10" "node": ">=10"
} }
}, },
"node_modules/rimraf": {
"version": "5.0.10",
"resolved": "https://registry.npmjs.org/rimraf/-/rimraf-5.0.10.tgz",
"integrity": "sha512-l0OE8wL34P4nJH/H2ffoaniAokM2qSmrtXHmlpvYr5AVVX8msAyW0l8NVJFDxlSK4u3Uh/f41cQheDVdnYijwQ==",
"dependencies": {
"glob": "^10.3.7"
},
"bin": {
"rimraf": "dist/esm/bin.mjs"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/rimraf/node_modules/brace-expansion": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.0.1.tgz",
"integrity": "sha512-XnAIvQ8eM+kC6aULx6wuQiwVsnzsi9d3WxzV3FpWTGA19F621kwdbsAcFKXgKUHZWsy+mY6iL1sHTxWEFCytDA==",
"dependencies": {
"balanced-match": "^1.0.0"
}
},
"node_modules/rimraf/node_modules/glob": {
"version": "10.4.5",
"resolved": "https://registry.npmjs.org/glob/-/glob-10.4.5.tgz",
"integrity": "sha512-7Bv8RF0k6xjo7d4A/PxYLbUCfb6c+Vpd2/mB2yRDlew7Jb5hEXiCD9ibfO7wpk8i4sevK6DFny9h7EYbM3/sHg==",
"dependencies": {
"foreground-child": "^3.1.0",
"jackspeak": "^3.1.2",
"minimatch": "^9.0.4",
"minipass": "^7.1.2",
"package-json-from-dist": "^1.0.0",
"path-scurry": "^1.11.1"
},
"bin": {
"glob": "dist/esm/bin.mjs"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/rimraf/node_modules/minimatch": {
"version": "9.0.5",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.5.tgz",
"integrity": "sha512-G6T0ZX48xgozx7587koeX9Ys2NYy6Gmv//P89sEte9V9whIapMNF4idKxnW2QtCcLiTWlb/wfCabAtAFWhhBow==",
"dependencies": {
"brace-expansion": "^2.0.1"
},
"engines": {
"node": ">=16 || 14 >=14.17"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/semver": { "node_modules/semver": {
"version": "7.6.3", "version": "7.6.3",
"resolved": "https://registry.npmjs.org/semver/-/semver-7.6.3.tgz", "resolved": "https://registry.npmjs.org/semver/-/semver-7.6.3.tgz",
@@ -4354,6 +4299,7 @@
"version": "2.0.0", "version": "2.0.0",
"resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz",
"integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==",
"dev": true,
"dependencies": { "dependencies": {
"shebang-regex": "^3.0.0" "shebang-regex": "^3.0.0"
}, },
@@ -4365,6 +4311,7 @@
"version": "3.0.0", "version": "3.0.0",
"resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz", "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz",
"integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==", "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==",
"dev": true,
"engines": { "engines": {
"node": ">=8" "node": ">=8"
} }
@@ -4452,20 +4399,7 @@
"version": "4.2.3", "version": "4.2.3",
"resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz",
"integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==",
"dependencies": { "dev": true,
"emoji-regex": "^8.0.0",
"is-fullwidth-code-point": "^3.0.0",
"strip-ansi": "^6.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/string-width-cjs": {
"name": "string-width",
"version": "4.2.3",
"resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz",
"integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==",
"dependencies": { "dependencies": {
"emoji-regex": "^8.0.0", "emoji-regex": "^8.0.0",
"is-fullwidth-code-point": "^3.0.0", "is-fullwidth-code-point": "^3.0.0",
@@ -4479,18 +4413,7 @@
"version": "6.0.1", "version": "6.0.1",
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz",
"integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==",
"dependencies": { "dev": true,
"ansi-regex": "^5.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/strip-ansi-cjs": {
"name": "strip-ansi",
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz",
"integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==",
"dependencies": { "dependencies": {
"ansi-regex": "^5.0.1" "ansi-regex": "^5.0.1"
}, },
@@ -4541,15 +4464,15 @@
} }
}, },
"node_modules/tar": { "node_modules/tar": {
"version": "7.4.3", "version": "7.5.10",
"resolved": "https://registry.npmjs.org/tar/-/tar-7.4.3.tgz", "resolved": "https://registry.npmjs.org/tar/-/tar-7.5.10.tgz",
"integrity": "sha512-5S7Va8hKfV7W5U6g3aYxXmlPoZVAwUMy9AOKyF2fVuZa2UD3qZjg578OrLRt8PcNN1PleVaL/5/yYATNL0ICUw==", "integrity": "sha512-8mOPs1//5q/rlkNSPcCegA6hiHJYDmSLEI8aMH/CdSQJNWztHC9WHNam5zdQlfpTwB9Xp7IBEsHfV5LKMJGVAw==",
"license": "BlueOak-1.0.0",
"dependencies": { "dependencies": {
"@isaacs/fs-minipass": "^4.0.0", "@isaacs/fs-minipass": "^4.0.0",
"chownr": "^3.0.0", "chownr": "^3.0.0",
"minipass": "^7.1.2", "minipass": "^7.1.2",
"minizlib": "^3.0.1", "minizlib": "^3.1.0",
"mkdirp": "^3.0.1",
"yallist": "^5.0.0" "yallist": "^5.0.0"
}, },
"engines": { "engines": {
@@ -4782,6 +4705,7 @@
"version": "2.0.2", "version": "2.0.2",
"resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz",
"integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==",
"dev": true,
"dependencies": { "dependencies": {
"isexe": "^2.0.0" "isexe": "^2.0.0"
}, },
@@ -4809,23 +4733,6 @@
"url": "https://github.com/chalk/wrap-ansi?sponsor=1" "url": "https://github.com/chalk/wrap-ansi?sponsor=1"
} }
}, },
"node_modules/wrap-ansi-cjs": {
"name": "wrap-ansi",
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz",
"integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==",
"dependencies": {
"ansi-styles": "^4.0.0",
"string-width": "^4.1.0",
"strip-ansi": "^6.0.0"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
}
},
"node_modules/wrappy": { "node_modules/wrappy": {
"version": "1.0.2", "version": "1.0.2",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",

View File

@@ -20,6 +20,8 @@ import {
Float32, Float32,
Float64, Float64,
Int, Int,
Int8,
Int16,
Int32, Int32,
Int64, Int64,
LargeBinary, LargeBinary,
@@ -35,6 +37,8 @@ import {
Timestamp, Timestamp,
Type, Type,
Uint8, Uint8,
Uint16,
Uint32,
Utf8, Utf8,
Vector, Vector,
makeVector as arrowMakeVector, makeVector as arrowMakeVector,
@@ -113,8 +117,9 @@ export type TableLike =
export type IntoVector = export type IntoVector =
| Float32Array | Float32Array
| Float64Array | Float64Array
| Uint8Array
| number[] | number[]
| Promise<Float32Array | Float64Array | number[]>; | Promise<Float32Array | Float64Array | Uint8Array | number[]>;
export type MultiVector = IntoVector[]; export type MultiVector = IntoVector[];
@@ -122,14 +127,48 @@ export function isMultiVector(value: unknown): value is MultiVector {
return Array.isArray(value) && isIntoVector(value[0]); return Array.isArray(value) && isIntoVector(value[0]);
} }
// Float16Array is not in TypeScript's standard lib yet; access dynamically
type Float16ArrayCtor = new (
...args: unknown[]
) => { buffer: ArrayBuffer; byteOffset: number; byteLength: number };
const float16ArrayCtor = (globalThis as unknown as Record<string, unknown>)
.Float16Array as Float16ArrayCtor | undefined;
export function isIntoVector(value: unknown): value is IntoVector { export function isIntoVector(value: unknown): value is IntoVector {
return ( return (
value instanceof Float32Array || value instanceof Float32Array ||
value instanceof Float64Array || value instanceof Float64Array ||
value instanceof Uint8Array ||
(float16ArrayCtor !== undefined && value instanceof float16ArrayCtor) ||
(Array.isArray(value) && !Array.isArray(value[0])) (Array.isArray(value) && !Array.isArray(value[0]))
); );
} }
/**
* Extract the underlying byte buffer and data type from a typed array
* for passing to the Rust NAPI layer without precision loss.
*/
export function extractVectorBuffer(
vector: Float32Array | Float64Array | Uint8Array,
): { data: Uint8Array; dtype: string } | null {
if (float16ArrayCtor !== undefined && vector instanceof float16ArrayCtor) {
return {
data: new Uint8Array(vector.buffer, vector.byteOffset, vector.byteLength),
dtype: "float16",
};
}
if (vector instanceof Float64Array) {
return {
data: new Uint8Array(vector.buffer, vector.byteOffset, vector.byteLength),
dtype: "float64",
};
}
if (vector instanceof Uint8Array && !(vector instanceof Float32Array)) {
return { data: vector, dtype: "uint8" };
}
return null;
}
export function isArrowTable(value: object): value is TableLike { export function isArrowTable(value: object): value is TableLike {
if (value instanceof ArrowTable) return true; if (value instanceof ArrowTable) return true;
return "schema" in value && "batches" in value; return "schema" in value && "batches" in value;
@@ -529,7 +568,8 @@ function isObject(value: unknown): value is Record<string, unknown> {
!(value instanceof Date) && !(value instanceof Date) &&
!(value instanceof Set) && !(value instanceof Set) &&
!(value instanceof Map) && !(value instanceof Map) &&
!(value instanceof Buffer) !(value instanceof Buffer) &&
!ArrayBuffer.isView(value)
); );
} }
@@ -588,6 +628,13 @@ function inferType(
return new Bool(); return new Bool();
} else if (value instanceof Buffer) { } else if (value instanceof Buffer) {
return new Binary(); return new Binary();
} else if (ArrayBuffer.isView(value) && !(value instanceof DataView)) {
const info = typedArrayToArrowType(value);
if (info !== undefined) {
const child = new Field("item", info.elementType, true);
return new FixedSizeList(info.length, child);
}
return undefined;
} else if (Array.isArray(value)) { } else if (Array.isArray(value)) {
if (value.length === 0) { if (value.length === 0) {
return undefined; // Without any values we can't infer the type return undefined; // Without any values we can't infer the type
@@ -746,6 +793,32 @@ function makeListVector(lists: unknown[][]): Vector<unknown> {
return listBuilder.finish().toVector(); return listBuilder.finish().toVector();
} }
/**
* Map a JS TypedArray instance to the corresponding Arrow element DataType
* and its length. Returns undefined if the value is not a recognized TypedArray.
*/
function typedArrayToArrowType(
value: ArrayBufferView,
): { elementType: DataType; length: number } | undefined {
if (value instanceof Float32Array)
return { elementType: new Float32(), length: value.length };
if (value instanceof Float64Array)
return { elementType: new Float64(), length: value.length };
if (value instanceof Uint8Array)
return { elementType: new Uint8(), length: value.length };
if (value instanceof Uint16Array)
return { elementType: new Uint16(), length: value.length };
if (value instanceof Uint32Array)
return { elementType: new Uint32(), length: value.length };
if (value instanceof Int8Array)
return { elementType: new Int8(), length: value.length };
if (value instanceof Int16Array)
return { elementType: new Int16(), length: value.length };
if (value instanceof Int32Array)
return { elementType: new Int32(), length: value.length };
return undefined;
}
/** Helper function to convert an Array of JS values to an Arrow Vector */ /** Helper function to convert an Array of JS values to an Arrow Vector */
function makeVector( function makeVector(
values: unknown[], values: unknown[],
@@ -814,6 +887,16 @@ function makeVector(
"makeVector cannot infer the type if all values are null or undefined", "makeVector cannot infer the type if all values are null or undefined",
); );
} }
if (ArrayBuffer.isView(sampleValue) && !(sampleValue instanceof DataView)) {
const info = typedArrayToArrowType(sampleValue);
if (info !== undefined) {
const fslType = new FixedSizeList(
info.length,
new Field("item", info.elementType, true),
);
return vectorFromArray(values, fslType);
}
}
if (Array.isArray(sampleValue)) { if (Array.isArray(sampleValue)) {
// Default Arrow inference doesn't handle list types // Default Arrow inference doesn't handle list types
return makeListVector(values as unknown[][]); return makeListVector(values as unknown[][]);

View File

@@ -166,25 +166,25 @@ export abstract class Connection {
* List all the table names in this database. * List all the table names in this database.
* *
* Tables will be returned in lexicographical order. * Tables will be returned in lexicographical order.
* @param {string[]} namespace - The namespace to list tables from (defaults to root namespace) * @param {string[]} namespacePath - The namespace path to list tables from (defaults to root namespace)
* @param {Partial<TableNamesOptions>} options - options to control the * @param {Partial<TableNamesOptions>} options - options to control the
* paging / start point * paging / start point
* *
*/ */
abstract tableNames( abstract tableNames(
namespace?: string[], namespacePath?: string[],
options?: Partial<TableNamesOptions>, options?: Partial<TableNamesOptions>,
): Promise<string[]>; ): Promise<string[]>;
/** /**
* Open a table in the database. * Open a table in the database.
* @param {string} name - The name of the table * @param {string} name - The name of the table
* @param {string[]} namespace - The namespace of the table (defaults to root namespace) * @param {string[]} namespacePath - The namespace path of the table (defaults to root namespace)
* @param {Partial<OpenTableOptions>} options - Additional options * @param {Partial<OpenTableOptions>} options - Additional options
*/ */
abstract openTable( abstract openTable(
name: string, name: string,
namespace?: string[], namespacePath?: string[],
options?: Partial<OpenTableOptions>, options?: Partial<OpenTableOptions>,
): Promise<Table>; ): Promise<Table>;
@@ -193,7 +193,7 @@ export abstract class Connection {
* @param {object} options - The options object. * @param {object} options - The options object.
* @param {string} options.name - The name of the table. * @param {string} options.name - The name of the table.
* @param {Data} options.data - Non-empty Array of Records to be inserted into the table * @param {Data} options.data - Non-empty Array of Records to be inserted into the table
* @param {string[]} namespace - The namespace to create the table in (defaults to root namespace) * @param {string[]} namespacePath - The namespace path to create the table in (defaults to root namespace)
* *
*/ */
abstract createTable( abstract createTable(
@@ -201,7 +201,7 @@ export abstract class Connection {
name: string; name: string;
data: Data; data: Data;
} & Partial<CreateTableOptions>, } & Partial<CreateTableOptions>,
namespace?: string[], namespacePath?: string[],
): Promise<Table>; ): Promise<Table>;
/** /**
* Creates a new Table and initialize it with new data. * Creates a new Table and initialize it with new data.
@@ -220,13 +220,13 @@ export abstract class Connection {
* @param {string} name - The name of the table. * @param {string} name - The name of the table.
* @param {Record<string, unknown>[] | TableLike} data - Non-empty Array of Records * @param {Record<string, unknown>[] | TableLike} data - Non-empty Array of Records
* to be inserted into the table * to be inserted into the table
* @param {string[]} namespace - The namespace to create the table in (defaults to root namespace) * @param {string[]} namespacePath - The namespace path to create the table in (defaults to root namespace)
* @param {Partial<CreateTableOptions>} options - Additional options * @param {Partial<CreateTableOptions>} options - Additional options
*/ */
abstract createTable( abstract createTable(
name: string, name: string,
data: Record<string, unknown>[] | TableLike, data: Record<string, unknown>[] | TableLike,
namespace?: string[], namespacePath?: string[],
options?: Partial<CreateTableOptions>, options?: Partial<CreateTableOptions>,
): Promise<Table>; ): Promise<Table>;
@@ -245,28 +245,28 @@ export abstract class Connection {
* Creates a new empty Table * Creates a new empty Table
* @param {string} name - The name of the table. * @param {string} name - The name of the table.
* @param {Schema} schema - The schema of the table * @param {Schema} schema - The schema of the table
* @param {string[]} namespace - The namespace to create the table in (defaults to root namespace) * @param {string[]} namespacePath - The namespace path to create the table in (defaults to root namespace)
* @param {Partial<CreateTableOptions>} options - Additional options * @param {Partial<CreateTableOptions>} options - Additional options
*/ */
abstract createEmptyTable( abstract createEmptyTable(
name: string, name: string,
schema: import("./arrow").SchemaLike, schema: import("./arrow").SchemaLike,
namespace?: string[], namespacePath?: string[],
options?: Partial<CreateTableOptions>, options?: Partial<CreateTableOptions>,
): Promise<Table>; ): Promise<Table>;
/** /**
* Drop an existing table. * Drop an existing table.
* @param {string} name The name of the table to drop. * @param {string} name The name of the table to drop.
* @param {string[]} namespace The namespace of the table (defaults to root namespace). * @param {string[]} namespacePath The namespace path of the table (defaults to root namespace).
*/ */
abstract dropTable(name: string, namespace?: string[]): Promise<void>; abstract dropTable(name: string, namespacePath?: string[]): Promise<void>;
/** /**
* Drop all tables in the database. * Drop all tables in the database.
* @param {string[]} namespace The namespace to drop tables from (defaults to root namespace). * @param {string[]} namespacePath The namespace path to drop tables from (defaults to root namespace).
*/ */
abstract dropAllTables(namespace?: string[]): Promise<void>; abstract dropAllTables(namespacePath?: string[]): Promise<void>;
/** /**
* Clone a table from a source table. * Clone a table from a source table.
@@ -279,7 +279,7 @@ export abstract class Connection {
* @param {string} targetTableName - The name of the target table to create. * @param {string} targetTableName - The name of the target table to create.
* @param {string} sourceUri - The URI of the source table to clone from. * @param {string} sourceUri - The URI of the source table to clone from.
* @param {object} options - Clone options. * @param {object} options - Clone options.
* @param {string[]} options.targetNamespace - The namespace for the target table (defaults to root namespace). * @param {string[]} options.targetNamespacePath - The namespace path for the target table (defaults to root namespace).
* @param {number} options.sourceVersion - The version of the source table to clone. * @param {number} options.sourceVersion - The version of the source table to clone.
* @param {string} options.sourceTag - The tag of the source table to clone. * @param {string} options.sourceTag - The tag of the source table to clone.
* @param {boolean} options.isShallow - Whether to perform a shallow clone (defaults to true). * @param {boolean} options.isShallow - Whether to perform a shallow clone (defaults to true).
@@ -288,7 +288,7 @@ export abstract class Connection {
targetTableName: string, targetTableName: string,
sourceUri: string, sourceUri: string,
options?: { options?: {
targetNamespace?: string[]; targetNamespacePath?: string[];
sourceVersion?: number; sourceVersion?: number;
sourceTag?: string; sourceTag?: string;
isShallow?: boolean; isShallow?: boolean;
@@ -319,25 +319,25 @@ export class LocalConnection extends Connection {
} }
async tableNames( async tableNames(
namespaceOrOptions?: string[] | Partial<TableNamesOptions>, namespacePathOrOptions?: string[] | Partial<TableNamesOptions>,
options?: Partial<TableNamesOptions>, options?: Partial<TableNamesOptions>,
): Promise<string[]> { ): Promise<string[]> {
// Detect if first argument is namespace array or options object // Detect if first argument is namespacePath array or options object
let namespace: string[] | undefined; let namespacePath: string[] | undefined;
let tableNamesOptions: Partial<TableNamesOptions> | undefined; let tableNamesOptions: Partial<TableNamesOptions> | undefined;
if (Array.isArray(namespaceOrOptions)) { if (Array.isArray(namespacePathOrOptions)) {
// First argument is namespace array // First argument is namespacePath array
namespace = namespaceOrOptions; namespacePath = namespacePathOrOptions;
tableNamesOptions = options; tableNamesOptions = options;
} else { } else {
// First argument is options object (backwards compatibility) // First argument is options object (backwards compatibility)
namespace = undefined; namespacePath = undefined;
tableNamesOptions = namespaceOrOptions; tableNamesOptions = namespacePathOrOptions;
} }
return this.inner.tableNames( return this.inner.tableNames(
namespace ?? [], namespacePath ?? [],
tableNamesOptions?.startAfter, tableNamesOptions?.startAfter,
tableNamesOptions?.limit, tableNamesOptions?.limit,
); );
@@ -345,12 +345,12 @@ export class LocalConnection extends Connection {
async openTable( async openTable(
name: string, name: string,
namespace?: string[], namespacePath?: string[],
options?: Partial<OpenTableOptions>, options?: Partial<OpenTableOptions>,
): Promise<Table> { ): Promise<Table> {
const innerTable = await this.inner.openTable( const innerTable = await this.inner.openTable(
name, name,
namespace ?? [], namespacePath ?? [],
cleanseStorageOptions(options?.storageOptions), cleanseStorageOptions(options?.storageOptions),
options?.indexCacheSize, options?.indexCacheSize,
); );
@@ -362,7 +362,7 @@ export class LocalConnection extends Connection {
targetTableName: string, targetTableName: string,
sourceUri: string, sourceUri: string,
options?: { options?: {
targetNamespace?: string[]; targetNamespacePath?: string[];
sourceVersion?: number; sourceVersion?: number;
sourceTag?: string; sourceTag?: string;
isShallow?: boolean; isShallow?: boolean;
@@ -371,7 +371,7 @@ export class LocalConnection extends Connection {
const innerTable = await this.inner.cloneTable( const innerTable = await this.inner.cloneTable(
targetTableName, targetTableName,
sourceUri, sourceUri,
options?.targetNamespace ?? [], options?.targetNamespacePath ?? [],
options?.sourceVersion ?? null, options?.sourceVersion ?? null,
options?.sourceTag ?? null, options?.sourceTag ?? null,
options?.isShallow ?? true, options?.isShallow ?? true,
@@ -406,42 +406,42 @@ export class LocalConnection extends Connection {
nameOrOptions: nameOrOptions:
| string | string
| ({ name: string; data: Data } & Partial<CreateTableOptions>), | ({ name: string; data: Data } & Partial<CreateTableOptions>),
dataOrNamespace?: Record<string, unknown>[] | TableLike | string[], dataOrNamespacePath?: Record<string, unknown>[] | TableLike | string[],
namespaceOrOptions?: string[] | Partial<CreateTableOptions>, namespacePathOrOptions?: string[] | Partial<CreateTableOptions>,
options?: Partial<CreateTableOptions>, options?: Partial<CreateTableOptions>,
): Promise<Table> { ): Promise<Table> {
if (typeof nameOrOptions !== "string" && "name" in nameOrOptions) { if (typeof nameOrOptions !== "string" && "name" in nameOrOptions) {
// First overload: createTable(options, namespace?) // First overload: createTable(options, namespacePath?)
const { name, data, ...createOptions } = nameOrOptions; const { name, data, ...createOptions } = nameOrOptions;
const namespace = dataOrNamespace as string[] | undefined; const namespacePath = dataOrNamespacePath as string[] | undefined;
return this._createTableImpl(name, data, namespace, createOptions); return this._createTableImpl(name, data, namespacePath, createOptions);
} }
// Second overload: createTable(name, data, namespace?, options?) // Second overload: createTable(name, data, namespacePath?, options?)
const name = nameOrOptions; const name = nameOrOptions;
const data = dataOrNamespace as Record<string, unknown>[] | TableLike; const data = dataOrNamespacePath as Record<string, unknown>[] | TableLike;
// Detect if third argument is namespace array or options object // Detect if third argument is namespacePath array or options object
let namespace: string[] | undefined; let namespacePath: string[] | undefined;
let createOptions: Partial<CreateTableOptions> | undefined; let createOptions: Partial<CreateTableOptions> | undefined;
if (Array.isArray(namespaceOrOptions)) { if (Array.isArray(namespacePathOrOptions)) {
// Third argument is namespace array // Third argument is namespacePath array
namespace = namespaceOrOptions; namespacePath = namespacePathOrOptions;
createOptions = options; createOptions = options;
} else { } else {
// Third argument is options object (backwards compatibility) // Third argument is options object (backwards compatibility)
namespace = undefined; namespacePath = undefined;
createOptions = namespaceOrOptions; createOptions = namespacePathOrOptions;
} }
return this._createTableImpl(name, data, namespace, createOptions); return this._createTableImpl(name, data, namespacePath, createOptions);
} }
private async _createTableImpl( private async _createTableImpl(
name: string, name: string,
data: Data, data: Data,
namespace?: string[], namespacePath?: string[],
options?: Partial<CreateTableOptions>, options?: Partial<CreateTableOptions>,
): Promise<Table> { ): Promise<Table> {
if (data === undefined) { if (data === undefined) {
@@ -455,7 +455,7 @@ export class LocalConnection extends Connection {
name, name,
buf, buf,
mode, mode,
namespace ?? [], namespacePath ?? [],
storageOptions, storageOptions,
); );
@@ -465,21 +465,21 @@ export class LocalConnection extends Connection {
async createEmptyTable( async createEmptyTable(
name: string, name: string,
schema: import("./arrow").SchemaLike, schema: import("./arrow").SchemaLike,
namespaceOrOptions?: string[] | Partial<CreateTableOptions>, namespacePathOrOptions?: string[] | Partial<CreateTableOptions>,
options?: Partial<CreateTableOptions>, options?: Partial<CreateTableOptions>,
): Promise<Table> { ): Promise<Table> {
// Detect if third argument is namespace array or options object // Detect if third argument is namespacePath array or options object
let namespace: string[] | undefined; let namespacePath: string[] | undefined;
let createOptions: Partial<CreateTableOptions> | undefined; let createOptions: Partial<CreateTableOptions> | undefined;
if (Array.isArray(namespaceOrOptions)) { if (Array.isArray(namespacePathOrOptions)) {
// Third argument is namespace array // Third argument is namespacePath array
namespace = namespaceOrOptions; namespacePath = namespacePathOrOptions;
createOptions = options; createOptions = options;
} else { } else {
// Third argument is options object (backwards compatibility) // Third argument is options object (backwards compatibility)
namespace = undefined; namespacePath = undefined;
createOptions = namespaceOrOptions; createOptions = namespacePathOrOptions;
} }
let mode: string = createOptions?.mode ?? "create"; let mode: string = createOptions?.mode ?? "create";
@@ -502,18 +502,18 @@ export class LocalConnection extends Connection {
name, name,
buf, buf,
mode, mode,
namespace ?? [], namespacePath ?? [],
storageOptions, storageOptions,
); );
return new LocalTable(innerTable); return new LocalTable(innerTable);
} }
async dropTable(name: string, namespace?: string[]): Promise<void> { async dropTable(name: string, namespacePath?: string[]): Promise<void> {
return this.inner.dropTable(name, namespace ?? []); return this.inner.dropTable(name, namespacePath ?? []);
} }
async dropAllTables(namespace?: string[]): Promise<void> { async dropAllTables(namespacePath?: string[]): Promise<void> {
return this.inner.dropAllTables(namespace ?? []); return this.inner.dropAllTables(namespacePath ?? []);
} }
} }

View File

@@ -5,6 +5,7 @@ import {
Table as ArrowTable, Table as ArrowTable,
type IntoVector, type IntoVector,
RecordBatch, RecordBatch,
extractVectorBuffer,
fromBufferToRecordBatch, fromBufferToRecordBatch,
fromRecordBatchToBuffer, fromRecordBatchToBuffer,
tableFromIPC, tableFromIPC,
@@ -661,10 +662,8 @@ export class VectorQuery extends StandardQueryBase<NativeVectorQuery> {
const res = (async () => { const res = (async () => {
try { try {
const v = await vector; const v = await vector;
const arr = Float32Array.from(v);
//
// biome-ignore lint/suspicious/noExplicitAny: we need to get the `inner`, but js has no package scoping // biome-ignore lint/suspicious/noExplicitAny: we need to get the `inner`, but js has no package scoping
const value: any = this.addQueryVector(arr); const value: any = this.addQueryVector(v);
const inner = value.inner as const inner = value.inner as
| NativeVectorQuery | NativeVectorQuery
| Promise<NativeVectorQuery>; | Promise<NativeVectorQuery>;
@@ -676,7 +675,12 @@ export class VectorQuery extends StandardQueryBase<NativeVectorQuery> {
return new VectorQuery(res); return new VectorQuery(res);
} else { } else {
super.doCall((inner) => { super.doCall((inner) => {
inner.addQueryVector(Float32Array.from(vector)); const raw = Array.isArray(vector) ? null : extractVectorBuffer(vector);
if (raw) {
inner.addQueryVectorRaw(raw.data, raw.dtype);
} else {
inner.addQueryVector(Float32Array.from(vector as number[]));
}
}); });
return this; return this;
} }
@@ -765,14 +769,23 @@ export class Query extends StandardQueryBase<NativeQuery> {
* a default `limit` of 10 will be used. @see {@link Query#limit} * a default `limit` of 10 will be used. @see {@link Query#limit}
*/ */
nearestTo(vector: IntoVector): VectorQuery { nearestTo(vector: IntoVector): VectorQuery {
const callNearestTo = (
inner: NativeQuery,
resolved: Float32Array | Float64Array | Uint8Array | number[],
): NativeVectorQuery => {
const raw = Array.isArray(resolved)
? null
: extractVectorBuffer(resolved);
if (raw) {
return inner.nearestToRaw(raw.data, raw.dtype);
}
return inner.nearestTo(Float32Array.from(resolved as number[]));
};
if (this.inner instanceof Promise) { if (this.inner instanceof Promise) {
const nativeQuery = this.inner.then(async (inner) => { const nativeQuery = this.inner.then(async (inner) => {
if (vector instanceof Promise) { const resolved = vector instanceof Promise ? await vector : vector;
const arr = await vector.then((v) => Float32Array.from(v)); return callNearestTo(inner, resolved);
return inner.nearestTo(arr);
} else {
return inner.nearestTo(Float32Array.from(vector));
}
}); });
return new VectorQuery(nativeQuery); return new VectorQuery(nativeQuery);
} }
@@ -780,10 +793,8 @@ export class Query extends StandardQueryBase<NativeQuery> {
const res = (async () => { const res = (async () => {
try { try {
const v = await vector; const v = await vector;
const arr = Float32Array.from(v);
//
// biome-ignore lint/suspicious/noExplicitAny: we need to get the `inner`, but js has no package scoping // biome-ignore lint/suspicious/noExplicitAny: we need to get the `inner`, but js has no package scoping
const value: any = this.nearestTo(arr); const value: any = this.nearestTo(v);
const inner = value.inner as const inner = value.inner as
| NativeVectorQuery | NativeVectorQuery
| Promise<NativeVectorQuery>; | Promise<NativeVectorQuery>;
@@ -794,7 +805,7 @@ export class Query extends StandardQueryBase<NativeQuery> {
})(); })();
return new VectorQuery(res); return new VectorQuery(res);
} else { } else {
const vectorQuery = this.inner.nearestTo(Float32Array.from(vector)); const vectorQuery = callNearestTo(this.inner, vector);
return new VectorQuery(vectorQuery); return new VectorQuery(vectorQuery);
} }
} }

View File

@@ -5,12 +5,15 @@ import {
Table as ArrowTable, Table as ArrowTable,
Data, Data,
DataType, DataType,
Field,
IntoVector, IntoVector,
MultiVector, MultiVector,
Schema, Schema,
dataTypeToJson, dataTypeToJson,
fromDataToBuffer, fromDataToBuffer,
fromTableToBuffer,
isMultiVector, isMultiVector,
makeEmptyTable,
tableFromIPC, tableFromIPC,
} from "./arrow"; } from "./arrow";
@@ -84,6 +87,16 @@ export interface OptimizeOptions {
* tbl.optimize({cleanupOlderThan: new Date()}); * tbl.optimize({cleanupOlderThan: new Date()});
*/ */
cleanupOlderThan: Date; cleanupOlderThan: Date;
/**
* Because they may be part of an in-progress transaction, files newer than
* 7 days old are not deleted by default. If you are sure that there are no
* in-progress transactions, then you can set this to true to delete all
* files older than `cleanupOlderThan`.
*
* **WARNING**: This should only be set to true if you can guarantee that
* no other process is currently working on this dataset. Otherwise the
* dataset could be put into a corrupted state.
*/
deleteUnverified: boolean; deleteUnverified: boolean;
} }
@@ -381,15 +394,16 @@ export abstract class Table {
abstract vectorSearch(vector: IntoVector | MultiVector): VectorQuery; abstract vectorSearch(vector: IntoVector | MultiVector): VectorQuery;
/** /**
* Add new columns with defined values. * Add new columns with defined values.
* @param {AddColumnsSql[]} newColumnTransforms pairs of column names and * @param {AddColumnsSql[] | Field | Field[] | Schema} newColumnTransforms Either:
* the SQL expression to use to calculate the value of the new column. These * - An array of objects with column names and SQL expressions to calculate values
* expressions will be evaluated for each row in the table, and can * - A single Arrow Field defining one column with its data type (column will be initialized with null values)
* reference existing columns in the table. * - An array of Arrow Fields defining columns with their data types (columns will be initialized with null values)
* - An Arrow Schema defining columns with their data types (columns will be initialized with null values)
* @returns {Promise<AddColumnsResult>} A promise that resolves to an object * @returns {Promise<AddColumnsResult>} A promise that resolves to an object
* containing the new version number of the table after adding the columns. * containing the new version number of the table after adding the columns.
*/ */
abstract addColumns( abstract addColumns(
newColumnTransforms: AddColumnsSql[], newColumnTransforms: AddColumnsSql[] | Field | Field[] | Schema,
): Promise<AddColumnsResult>; ): Promise<AddColumnsResult>;
/** /**
@@ -501,19 +515,7 @@ export abstract class Table {
* - Index: Optimizes the indices, adding new data to existing indices * - Index: Optimizes the indices, adding new data to existing indices
* *
* *
* Experimental API * The frequency an application should call optimize is based on the frequency of
* ----------------
*
* The optimization process is undergoing active development and may change.
* Our goal with these changes is to improve the performance of optimization and
* reduce the complexity.
*
* That being said, it is essential today to run optimize if you want the best
* performance. It should be stable and safe to use in production, but it our
* hope that the API may be simplified (or not even need to be called) in the
* future.
*
* The frequency an application shoudl call optimize is based on the frequency of
* data modifications. If data is frequently added, deleted, or updated then * data modifications. If data is frequently added, deleted, or updated then
* optimize should be run frequently. A good rule of thumb is to run optimize if * optimize should be run frequently. A good rule of thumb is to run optimize if
* you have added or modified 100,000 or more records or run more than 20 data * you have added or modified 100,000 or more records or run more than 20 data
@@ -806,9 +808,40 @@ export class LocalTable extends Table {
// TODO: Support BatchUDF // TODO: Support BatchUDF
async addColumns( async addColumns(
newColumnTransforms: AddColumnsSql[], newColumnTransforms: AddColumnsSql[] | Field | Field[] | Schema,
): Promise<AddColumnsResult> { ): Promise<AddColumnsResult> {
return await this.inner.addColumns(newColumnTransforms); // Handle single Field -> convert to array of Fields
if (newColumnTransforms instanceof Field) {
newColumnTransforms = [newColumnTransforms];
}
// Handle array of Fields -> convert to Schema
if (
Array.isArray(newColumnTransforms) &&
newColumnTransforms.length > 0 &&
newColumnTransforms[0] instanceof Field
) {
const fields = newColumnTransforms as Field[];
newColumnTransforms = new Schema(fields);
}
// Handle Schema -> use schema-based approach
if (newColumnTransforms instanceof Schema) {
const schema = newColumnTransforms;
// Convert schema to buffer using Arrow IPC format
const emptyTable = makeEmptyTable(schema);
const schemaBuf = await fromTableToBuffer(emptyTable);
return await this.inner.addColumnsWithSchema(schemaBuf);
}
// Handle SQL expressions (existing functionality)
if (Array.isArray(newColumnTransforms)) {
return await this.inner.addColumns(
newColumnTransforms as AddColumnsSql[],
);
}
throw new Error("Invalid input type for addColumns");
} }
async alterColumns( async alterColumns(

View File

@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-darwin-arm64", "name": "@lancedb/lancedb-darwin-arm64",
"version": "0.27.0-beta.2", "version": "0.27.2",
"os": ["darwin"], "os": ["darwin"],
"cpu": ["arm64"], "cpu": ["arm64"],
"main": "lancedb.darwin-arm64.node", "main": "lancedb.darwin-arm64.node",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-arm64-gnu", "name": "@lancedb/lancedb-linux-arm64-gnu",
"version": "0.27.0-beta.2", "version": "0.27.2",
"os": ["linux"], "os": ["linux"],
"cpu": ["arm64"], "cpu": ["arm64"],
"main": "lancedb.linux-arm64-gnu.node", "main": "lancedb.linux-arm64-gnu.node",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-arm64-musl", "name": "@lancedb/lancedb-linux-arm64-musl",
"version": "0.27.0-beta.2", "version": "0.27.2",
"os": ["linux"], "os": ["linux"],
"cpu": ["arm64"], "cpu": ["arm64"],
"main": "lancedb.linux-arm64-musl.node", "main": "lancedb.linux-arm64-musl.node",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-x64-gnu", "name": "@lancedb/lancedb-linux-x64-gnu",
"version": "0.27.0-beta.2", "version": "0.27.2",
"os": ["linux"], "os": ["linux"],
"cpu": ["x64"], "cpu": ["x64"],
"main": "lancedb.linux-x64-gnu.node", "main": "lancedb.linux-x64-gnu.node",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-x64-musl", "name": "@lancedb/lancedb-linux-x64-musl",
"version": "0.27.0-beta.2", "version": "0.27.2",
"os": ["linux"], "os": ["linux"],
"cpu": ["x64"], "cpu": ["x64"],
"main": "lancedb.linux-x64-musl.node", "main": "lancedb.linux-x64-musl.node",

View File

@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-win32-arm64-msvc", "name": "@lancedb/lancedb-win32-arm64-msvc",
"version": "0.27.0-beta.2", "version": "0.27.2",
"os": [ "os": [
"win32" "win32"
], ],

View File

@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-win32-x64-msvc", "name": "@lancedb/lancedb-win32-x64-msvc",
"version": "0.27.0-beta.2", "version": "0.27.2",
"os": ["win32"], "os": ["win32"],
"cpu": ["x64"], "cpu": ["x64"],
"main": "lancedb.win32-x64-msvc.node", "main": "lancedb.win32-x64-msvc.node",

4289
nodejs/package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -11,7 +11,7 @@
"ann" "ann"
], ],
"private": false, "private": false,
"version": "0.27.0-beta.2", "version": "0.27.2",
"main": "dist/index.js", "main": "dist/index.js",
"exports": { "exports": {
".": "./dist/index.js", ".": "./dist/index.js",

View File

@@ -8,10 +8,10 @@ use lancedb::database::{CreateTableMode, Database};
use napi::bindgen_prelude::*; use napi::bindgen_prelude::*;
use napi_derive::*; use napi_derive::*;
use crate::ConnectionOptions;
use crate::error::NapiErrorExt; use crate::error::NapiErrorExt;
use crate::header::JsHeaderProvider; use crate::header::JsHeaderProvider;
use crate::table::Table; use crate::table::Table;
use crate::ConnectionOptions;
use lancedb::connection::{ConnectBuilder, Connection as LanceDBConnection}; use lancedb::connection::{ConnectBuilder, Connection as LanceDBConnection};
use lancedb::ipc::{ipc_file_to_batches, ipc_file_to_schema}; use lancedb::ipc::{ipc_file_to_batches, ipc_file_to_schema};
@@ -119,12 +119,12 @@ impl Connection {
#[napi(catch_unwind)] #[napi(catch_unwind)]
pub async fn table_names( pub async fn table_names(
&self, &self,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
start_after: Option<String>, start_after: Option<String>,
limit: Option<u32>, limit: Option<u32>,
) -> napi::Result<Vec<String>> { ) -> napi::Result<Vec<String>> {
let mut op = self.get_inner()?.table_names(); let mut op = self.get_inner()?.table_names();
op = op.namespace(namespace); op = op.namespace(namespace_path.unwrap_or_default());
if let Some(start_after) = start_after { if let Some(start_after) = start_after {
op = op.start_after(start_after); op = op.start_after(start_after);
} }
@@ -146,7 +146,7 @@ impl Connection {
name: String, name: String,
buf: Buffer, buf: Buffer,
mode: String, mode: String,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
storage_options: Option<HashMap<String, String>>, storage_options: Option<HashMap<String, String>>,
) -> napi::Result<Table> { ) -> napi::Result<Table> {
let batches = ipc_file_to_batches(buf.to_vec()) let batches = ipc_file_to_batches(buf.to_vec())
@@ -154,7 +154,7 @@ impl Connection {
let mode = Self::parse_create_mode_str(&mode)?; let mode = Self::parse_create_mode_str(&mode)?;
let mut builder = self.get_inner()?.create_table(&name, batches).mode(mode); let mut builder = self.get_inner()?.create_table(&name, batches).mode(mode);
builder = builder.namespace(namespace); builder = builder.namespace(namespace_path.unwrap_or_default());
if let Some(storage_options) = storage_options { if let Some(storage_options) = storage_options {
for (key, value) in storage_options { for (key, value) in storage_options {
@@ -171,7 +171,7 @@ impl Connection {
name: String, name: String,
schema_buf: Buffer, schema_buf: Buffer,
mode: String, mode: String,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
storage_options: Option<HashMap<String, String>>, storage_options: Option<HashMap<String, String>>,
) -> napi::Result<Table> { ) -> napi::Result<Table> {
let schema = ipc_file_to_schema(schema_buf.to_vec()).map_err(|e| { let schema = ipc_file_to_schema(schema_buf.to_vec()).map_err(|e| {
@@ -183,7 +183,7 @@ impl Connection {
.create_empty_table(&name, schema) .create_empty_table(&name, schema)
.mode(mode); .mode(mode);
builder = builder.namespace(namespace); builder = builder.namespace(namespace_path.unwrap_or_default());
if let Some(storage_options) = storage_options { if let Some(storage_options) = storage_options {
for (key, value) in storage_options { for (key, value) in storage_options {
@@ -198,13 +198,13 @@ impl Connection {
pub async fn open_table( pub async fn open_table(
&self, &self,
name: String, name: String,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
storage_options: Option<HashMap<String, String>>, storage_options: Option<HashMap<String, String>>,
index_cache_size: Option<u32>, index_cache_size: Option<u32>,
) -> napi::Result<Table> { ) -> napi::Result<Table> {
let mut builder = self.get_inner()?.open_table(&name); let mut builder = self.get_inner()?.open_table(&name);
builder = builder.namespace(namespace); builder = builder.namespace(namespace_path.unwrap_or_default());
if let Some(storage_options) = storage_options { if let Some(storage_options) = storage_options {
for (key, value) in storage_options { for (key, value) in storage_options {
@@ -223,7 +223,7 @@ impl Connection {
&self, &self,
target_table_name: String, target_table_name: String,
source_uri: String, source_uri: String,
target_namespace: Vec<String>, target_namespace_path: Option<Vec<String>>,
source_version: Option<i64>, source_version: Option<i64>,
source_tag: Option<String>, source_tag: Option<String>,
is_shallow: bool, is_shallow: bool,
@@ -232,7 +232,7 @@ impl Connection {
.get_inner()? .get_inner()?
.clone_table(&target_table_name, &source_uri); .clone_table(&target_table_name, &source_uri);
builder = builder.target_namespace(target_namespace); builder = builder.target_namespace(target_namespace_path.unwrap_or_default());
if let Some(version) = source_version { if let Some(version) = source_version {
builder = builder.source_version(version as u64); builder = builder.source_version(version as u64);
@@ -250,18 +250,21 @@ impl Connection {
/// Drop table with the name. Or raise an error if the table does not exist. /// Drop table with the name. Or raise an error if the table does not exist.
#[napi(catch_unwind)] #[napi(catch_unwind)]
pub async fn drop_table(&self, name: String, namespace: Vec<String>) -> napi::Result<()> { pub async fn drop_table(
&self,
name: String,
namespace_path: Option<Vec<String>>,
) -> napi::Result<()> {
let ns = namespace_path.unwrap_or_default();
self.get_inner()? self.get_inner()?
.drop_table(&name, &namespace) .drop_table(&name, &ns)
.await .await
.default_error() .default_error()
} }
#[napi(catch_unwind)] #[napi(catch_unwind)]
pub async fn drop_all_tables(&self, namespace: Vec<String>) -> napi::Result<()> { pub async fn drop_all_tables(&self, namespace_path: Option<Vec<String>>) -> napi::Result<()> {
self.get_inner()? let ns = namespace_path.unwrap_or_default();
.drop_all_tables(&namespace) self.get_inner()?.drop_all_tables(&ns).await.default_error()
.await
.default_error()
} }
} }

View File

@@ -3,12 +3,12 @@
use std::sync::Mutex; use std::sync::Mutex;
use lancedb::index::Index as LanceDbIndex;
use lancedb::index::scalar::{BTreeIndexBuilder, FtsIndexBuilder}; use lancedb::index::scalar::{BTreeIndexBuilder, FtsIndexBuilder};
use lancedb::index::vector::{ use lancedb::index::vector::{
IvfFlatIndexBuilder, IvfHnswPqIndexBuilder, IvfHnswSqIndexBuilder, IvfPqIndexBuilder, IvfFlatIndexBuilder, IvfHnswPqIndexBuilder, IvfHnswSqIndexBuilder, IvfPqIndexBuilder,
IvfRqIndexBuilder, IvfRqIndexBuilder,
}; };
use lancedb::index::Index as LanceDbIndex;
use napi_derive::napi; use napi_derive::napi;
use crate::util::parse_distance_type; use crate::util::parse_distance_type;

View File

@@ -3,6 +3,12 @@
use std::sync::Arc; use std::sync::Arc;
use arrow_array::{
Array, Float16Array as ArrowFloat16Array, Float32Array as ArrowFloat32Array,
Float64Array as ArrowFloat64Array, UInt8Array as ArrowUInt8Array,
};
use arrow_buffer::ScalarBuffer;
use half::f16;
use lancedb::index::scalar::{ use lancedb::index::scalar::{
BooleanQuery, BoostQuery, FtsQuery, FullTextSearchQuery, MatchQuery, MultiMatchQuery, Occur, BooleanQuery, BoostQuery, FtsQuery, FullTextSearchQuery, MatchQuery, MultiMatchQuery, Occur,
Operator, PhraseQuery, Operator, PhraseQuery,
@@ -17,13 +23,40 @@ use lancedb::query::VectorQuery as LanceDbVectorQuery;
use napi::bindgen_prelude::*; use napi::bindgen_prelude::*;
use napi_derive::napi; use napi_derive::napi;
use crate::error::convert_error;
use crate::error::NapiErrorExt; use crate::error::NapiErrorExt;
use crate::error::convert_error;
use crate::iterator::RecordBatchIterator; use crate::iterator::RecordBatchIterator;
use crate::rerankers::RerankHybridCallbackArgs; use crate::rerankers::RerankHybridCallbackArgs;
use crate::rerankers::Reranker; use crate::rerankers::Reranker;
use crate::util::{parse_distance_type, schema_to_buffer}; use crate::util::{parse_distance_type, schema_to_buffer};
fn bytes_to_arrow_array(data: Uint8Array, dtype: String) -> napi::Result<Arc<dyn Array>> {
let buf = arrow_buffer::Buffer::from(data.to_vec());
let num_bytes = buf.len();
match dtype.as_str() {
"float16" => {
let scalar_buf = ScalarBuffer::<f16>::new(buf, 0, num_bytes / 2);
Ok(Arc::new(ArrowFloat16Array::new(scalar_buf, None)))
}
"float32" => {
let scalar_buf = ScalarBuffer::<f32>::new(buf, 0, num_bytes / 4);
Ok(Arc::new(ArrowFloat32Array::new(scalar_buf, None)))
}
"float64" => {
let scalar_buf = ScalarBuffer::<f64>::new(buf, 0, num_bytes / 8);
Ok(Arc::new(ArrowFloat64Array::new(scalar_buf, None)))
}
"uint8" => {
let scalar_buf = ScalarBuffer::<u8>::new(buf, 0, num_bytes);
Ok(Arc::new(ArrowUInt8Array::new(scalar_buf, None)))
}
_ => Err(napi::Error::from_reason(format!(
"Unsupported vector dtype: {}. Expected one of: float16, float32, float64, uint8",
dtype
))),
}
}
#[napi] #[napi]
pub struct Query { pub struct Query {
inner: LanceDbQuery, inner: LanceDbQuery,
@@ -78,6 +111,13 @@ impl Query {
Ok(VectorQuery { inner }) Ok(VectorQuery { inner })
} }
#[napi]
pub fn nearest_to_raw(&mut self, data: Uint8Array, dtype: String) -> Result<VectorQuery> {
let array = bytes_to_arrow_array(data, dtype)?;
let inner = self.inner.clone().nearest_to(array).default_error()?;
Ok(VectorQuery { inner })
}
#[napi] #[napi]
pub fn fast_search(&mut self) { pub fn fast_search(&mut self) {
self.inner = self.inner.clone().fast_search(); self.inner = self.inner.clone().fast_search();
@@ -163,6 +203,13 @@ impl VectorQuery {
Ok(()) Ok(())
} }
#[napi]
pub fn add_query_vector_raw(&mut self, data: Uint8Array, dtype: String) -> Result<()> {
let array = bytes_to_arrow_array(data, dtype)?;
self.inner = self.inner.clone().add_query_vector(array).default_error()?;
Ok(())
}
#[napi] #[napi]
pub fn distance_type(&mut self, distance_type: String) -> napi::Result<()> { pub fn distance_type(&mut self, distance_type: String) -> napi::Result<()> {
let distance_type = parse_distance_type(distance_type)?; let distance_type = parse_distance_type(distance_type)?;
@@ -551,15 +598,12 @@ fn parse_fts_query(query: Object) -> napi::Result<FullTextSearchQuery> {
} }
}; };
let mut query = FullTextSearchQuery::new_query(query); let mut query = FullTextSearchQuery::new_query(query);
if let Some(cols) = columns { if let Some(cols) = columns
if !cols.is_empty() { && !cols.is_empty()
query = query.with_columns(&cols).map_err(|e| { {
napi::Error::from_reason(format!( query = query.with_columns(&cols).map_err(|e| {
"Failed to set full text search columns: {}", napi::Error::from_reason(format!("Failed to set full text search columns: {}", e))
e })?;
))
})?;
}
} }
Ok(query) Ok(query)
} else { } else {

View File

@@ -95,7 +95,7 @@ impl napi::bindgen_prelude::FromNapiValue for Session {
napi_val: napi::sys::napi_value, napi_val: napi::sys::napi_value,
) -> napi::Result<Self> { ) -> napi::Result<Self> {
let object: napi::bindgen_prelude::ClassInstance<Self> = let object: napi::bindgen_prelude::ClassInstance<Self> =
napi::bindgen_prelude::ClassInstance::from_napi_value(env, napi_val)?; unsafe { napi::bindgen_prelude::ClassInstance::from_napi_value(env, napi_val)? };
Ok((*object).clone()) Ok((*object).clone())
} }
} }

View File

@@ -3,7 +3,7 @@
use std::collections::HashMap; use std::collections::HashMap;
use lancedb::ipc::ipc_file_to_batches; use lancedb::ipc::{ipc_file_to_batches, ipc_file_to_schema};
use lancedb::table::{ use lancedb::table::{
AddDataMode, ColumnAlteration as LanceColumnAlteration, Duration, NewColumnTransform, AddDataMode, ColumnAlteration as LanceColumnAlteration, Duration, NewColumnTransform,
OptimizeAction, OptimizeOptions, Table as LanceDbTable, OptimizeAction, OptimizeOptions, Table as LanceDbTable,
@@ -279,6 +279,23 @@ impl Table {
Ok(res.into()) Ok(res.into())
} }
#[napi(catch_unwind)]
pub async fn add_columns_with_schema(
&self,
schema_buf: Buffer,
) -> napi::Result<AddColumnsResult> {
let schema = ipc_file_to_schema(schema_buf.to_vec())
.map_err(|e| napi::Error::from_reason(format!("Failed to read IPC schema: {}", e)))?;
let transforms = NewColumnTransform::AllNulls(schema);
let res = self
.inner_ref()?
.add_columns(transforms, None)
.await
.default_error()?;
Ok(res.into())
}
#[napi(catch_unwind)] #[napi(catch_unwind)]
pub async fn alter_columns( pub async fn alter_columns(
&self, &self,
@@ -753,12 +770,14 @@ impl From<lancedb::table::AddResult> for AddResult {
#[napi(object)] #[napi(object)]
pub struct DeleteResult { pub struct DeleteResult {
pub num_deleted_rows: i64,
pub version: i64, pub version: i64,
} }
impl From<lancedb::table::DeleteResult> for DeleteResult { impl From<lancedb::table::DeleteResult> for DeleteResult {
fn from(value: lancedb::table::DeleteResult) -> Self { fn from(value: lancedb::table::DeleteResult) -> Self {
Self { Self {
num_deleted_rows: value.num_deleted_rows as i64,
version: value.version as i64, version: value.version as i64,
} }
} }

View File

@@ -1,5 +1,5 @@
[tool.bumpversion] [tool.bumpversion]
current_version = "0.30.0-beta.3" current_version = "0.31.0-beta.0"
parse = """(?x) parse = """(?x)
(?P<major>0|[1-9]\\d*)\\. (?P<major>0|[1-9]\\d*)\\.
(?P<minor>0|[1-9]\\d*)\\. (?P<minor>0|[1-9]\\d*)\\.

2
python/.gitignore vendored
View File

@@ -1,3 +1,5 @@
# Test data created by some example tests # Test data created by some example tests
data/ data/
_lancedb.pyd _lancedb.pyd
# macOS debug symbols bundle generated during build
*.dSYM/

View File

@@ -1,6 +1,6 @@
[package] [package]
name = "lancedb-python" name = "lancedb-python"
version = "0.30.0-beta.3" version = "0.31.0-beta.0"
edition.workspace = true edition.workspace = true
description = "Python bindings for LanceDB" description = "Python bindings for LanceDB"
license.workspace = true license.workspace = true
@@ -16,11 +16,14 @@ crate-type = ["cdylib"]
[dependencies] [dependencies]
arrow = { version = "57.2", features = ["pyarrow"] } arrow = { version = "57.2", features = ["pyarrow"] }
async-trait = "0.1" async-trait = "0.1"
bytes = "1"
lancedb = { path = "../rust/lancedb", default-features = false } lancedb = { path = "../rust/lancedb", default-features = false }
lance-core.workspace = true lance-core.workspace = true
lance-namespace.workspace = true lance-namespace.workspace = true
lance-namespace-impls.workspace = true
lance-io.workspace = true lance-io.workspace = true
env_logger.workspace = true env_logger.workspace = true
log.workspace = true
pyo3 = { version = "0.26", features = ["extension-module", "abi3-py39"] } pyo3 = { version = "0.26", features = ["extension-module", "abi3-py39"] }
pyo3-async-runtimes = { version = "0.26", features = [ pyo3-async-runtimes = { version = "0.26", features = [
"attributes", "attributes",
@@ -28,6 +31,8 @@ pyo3-async-runtimes = { version = "0.26", features = [
] } ] }
pin-project = "1.1.5" pin-project = "1.1.5"
futures.workspace = true futures.workspace = true
serde = "1"
serde_json = "1"
snafu.workspace = true snafu.workspace = true
tokio = { version = "1.40", features = ["sync"] } tokio = { version = "1.40", features = ["sync"] }

View File

@@ -1,4 +1,4 @@
# LanceDB # LanceDB Python SDK
A Python library for [LanceDB](https://github.com/lancedb/lancedb). A Python library for [LanceDB](https://github.com/lancedb/lancedb).

View File

@@ -3,10 +3,10 @@ name = "lancedb"
# version in Cargo.toml # version in Cargo.toml
dynamic = ["version"] dynamic = ["version"]
dependencies = [ dependencies = [
"deprecation", "deprecation>=2.1.0",
"numpy", "numpy>=1.24.0",
"overrides>=0.7; python_version<'3.12'", "overrides>=0.7; python_version<'3.12'",
"packaging", "packaging>=23.0",
"pyarrow>=16", "pyarrow>=16",
"pydantic>=1.10", "pydantic>=1.10",
"tqdm>=4.27.0", "tqdm>=4.27.0",
@@ -45,51 +45,51 @@ repository = "https://github.com/lancedb/lancedb"
[project.optional-dependencies] [project.optional-dependencies]
pylance = [ pylance = [
"pylance>=1.0.0b14", "pylance>=5.0.0b3",
] ]
tests = [ tests = [
"aiohttp", "aiohttp>=3.9.0",
"boto3", "boto3>=1.28.57",
"pandas>=1.4", "pandas>=1.4",
"pytest", "pytest>=7.0",
"pytest-mock", "pytest-mock>=3.10",
"pytest-asyncio", "pytest-asyncio>=0.21",
"duckdb", "duckdb>=0.9.0",
"pytz", "pytz>=2023.3",
"polars>=0.19, <=1.3.0", "polars>=0.19, <=1.3.0",
"tantivy", "tantivy>=0.20.0",
"pyarrow-stubs", "pyarrow-stubs>=16.0",
"pylance>=1.0.0b14,<3.0.0", "pylance>=5.0.0b3",
"requests", "requests>=2.31.0",
"datafusion<52", "datafusion>=52,<53",
] ]
dev = [ dev = [
"ruff", "ruff>=0.3.0",
"pre-commit", "pre-commit>=3.5.0",
"pyright", "pyright>=1.1.350",
'typing-extensions>=4.0.0; python_version < "3.11"', 'typing-extensions>=4.0.0; python_version < "3.11"',
] ]
docs = ["mkdocs", "mkdocs-jupyter", "mkdocs-material", "mkdocstrings-python"] docs = ["mkdocs", "mkdocs-jupyter", "mkdocs-material", "mkdocstrings-python"]
clip = ["torch", "pillow", "open-clip-torch"] clip = ["torch", "pillow>=12.1.1", "open-clip-torch"]
siglip = ["torch", "pillow", "transformers>=4.41.0","sentencepiece"] siglip = ["torch", "pillow>=12.1.1", "transformers>=4.41.0","sentencepiece"]
embeddings = [ embeddings = [
"requests>=2.31.0", "requests>=2.31.0",
"openai>=1.6.1", "openai>=1.6.1",
"sentence-transformers", "sentence-transformers>=2.2.0",
"torch", "torch>=2.0.0",
"pillow", "pillow>=12.1.1",
"open-clip-torch", "open-clip-torch>=2.20.0",
"cohere", "cohere>=4.0",
"colpali-engine>=0.3.10", "colpali-engine>=0.3.10",
"huggingface_hub", "huggingface_hub>=0.19.0",
"InstructorEmbedding", "InstructorEmbedding>=1.0.1",
"google.generativeai", "google.generativeai>=0.3.0",
"boto3>=1.28.57", "boto3>=1.28.57",
"awscli>=1.29.57", "awscli>=1.44.38",
"botocore>=1.31.57", "botocore>=1.31.57",
'ibm-watsonx-ai>=1.1.2; python_version >= "3.10"', 'ibm-watsonx-ai>=1.1.2; python_version >= "3.10"',
"ollama>=0.3.0", "ollama>=0.3.0",
"sentencepiece" "sentencepiece>=0.1.99"
] ]
azure = ["adlfs>=2024.2.0"] azure = ["adlfs>=2024.2.0"]

View File

@@ -6,7 +6,7 @@ import importlib.metadata
import os import os
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from datetime import timedelta from datetime import timedelta
from typing import Dict, Optional, Union, Any from typing import Dict, Optional, Union, Any, List
import warnings import warnings
__version__ = importlib.metadata.version("lancedb") __version__ = importlib.metadata.version("lancedb")
@@ -15,9 +15,9 @@ from ._lancedb import connect as lancedb_connect
from .common import URI, sanitize_uri from .common import URI, sanitize_uri
from urllib.parse import urlparse from urllib.parse import urlparse
from .db import AsyncConnection, DBConnection, LanceDBConnection from .db import AsyncConnection, DBConnection, LanceDBConnection
from .io import StorageOptionsProvider
from .remote import ClientConfig from .remote import ClientConfig
from .remote.db import RemoteDBConnection from .remote.db import RemoteDBConnection
from .expr import Expr, col, lit, func
from .schema import vector from .schema import vector
from .table import AsyncTable, Table from .table import AsyncTable, Table
from ._lancedb import Session from ._lancedb import Session
@@ -63,7 +63,7 @@ def _check_s3_bucket_with_dots(
def connect( def connect(
uri: URI, uri: Optional[URI] = None,
*, *,
api_key: Optional[str] = None, api_key: Optional[str] = None,
region: str = "us-east-1", region: str = "us-east-1",
@@ -73,14 +73,18 @@ def connect(
client_config: Union[ClientConfig, Dict[str, Any], None] = None, client_config: Union[ClientConfig, Dict[str, Any], None] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
session: Optional[Session] = None, session: Optional[Session] = None,
namespace_client_impl: Optional[str] = None,
namespace_client_properties: Optional[Dict[str, str]] = None,
namespace_client_pushdown_operations: Optional[List[str]] = None,
**kwargs: Any, **kwargs: Any,
) -> DBConnection: ) -> DBConnection:
"""Connect to a LanceDB database. """Connect to a LanceDB database.
Parameters Parameters
---------- ----------
uri: str or Path uri: str or Path, optional
The uri of the database. The uri of the database. When ``namespace_client_impl`` is provided you may
omit ``uri`` and connect through a namespace client instead.
api_key: str, optional api_key: str, optional
If presented, connect to LanceDB cloud. If presented, connect to LanceDB cloud.
Otherwise, connect to a database on file system or cloud storage. Otherwise, connect to a database on file system or cloud storage.
@@ -113,6 +117,18 @@ def connect(
cache sizes for index and metadata caches, which can significantly cache sizes for index and metadata caches, which can significantly
impact memory use and performance. They can also be re-used across impact memory use and performance. They can also be re-used across
multiple connections to share the same cache state. multiple connections to share the same cache state.
namespace_client_impl : str, optional
When provided along with ``namespace_client_properties``, ``connect``
returns a namespace-backed connection by delegating to
:func:`connect_namespace`. The value identifies which namespace
implementation to load (e.g., ``"dir"`` or ``"rest"``).
namespace_client_properties : dict, optional
Configuration to pass to the namespace client implementation. Required
when ``namespace_client_impl`` is set.
namespace_client_pushdown_operations : list[str], optional
Only used when ``namespace_client_properties`` is provided. Forwards to
:func:`connect_namespace` to control which operations are executed on the
namespace service (e.g., ``["QueryTable", "CreateTable"]``).
Examples Examples
-------- --------
@@ -132,11 +148,42 @@ def connect(
>>> db = lancedb.connect("db://my_database", api_key="ldb_...", >>> db = lancedb.connect("db://my_database", api_key="ldb_...",
... client_config={"retry_config": {"retries": 5}}) ... client_config={"retry_config": {"retries": 5}})
Connect to a namespace-backed database:
>>> db = lancedb.connect(namespace_client_impl="dir",
... namespace_client_properties={"root": "/tmp/ns"})
Returns Returns
------- -------
conn : DBConnection conn : DBConnection
A connection to a LanceDB database. A connection to a LanceDB database.
""" """
if namespace_client_impl is not None or namespace_client_properties is not None:
if namespace_client_impl is None or namespace_client_properties is None:
raise ValueError(
"Both namespace_client_impl and "
"namespace_client_properties must be provided"
)
if kwargs:
raise ValueError(f"Unknown keyword arguments: {kwargs}")
return connect_namespace(
namespace_client_impl,
namespace_client_properties,
read_consistency_interval=read_consistency_interval,
storage_options=storage_options,
session=session,
namespace_client_pushdown_operations=namespace_client_pushdown_operations,
)
if namespace_client_pushdown_operations is not None:
raise ValueError(
"namespace_client_pushdown_operations is only valid when "
"connecting through a namespace"
)
if uri is None:
raise ValueError(
"uri is required when not connecting through a namespace client"
)
if isinstance(uri, str) and uri.startswith("db://"): if isinstance(uri, str) and uri.startswith("db://"):
if api_key is None: if api_key is None:
api_key = os.environ.get("LANCEDB_API_KEY") api_key = os.environ.get("LANCEDB_API_KEY")
@@ -271,6 +318,10 @@ __all__ = [
"AsyncConnection", "AsyncConnection",
"AsyncLanceNamespaceDBConnection", "AsyncLanceNamespaceDBConnection",
"AsyncTable", "AsyncTable",
"col",
"Expr",
"func",
"lit",
"URI", "URI",
"sanitize_uri", "sanitize_uri",
"vector", "vector",
@@ -279,7 +330,6 @@ __all__ = [
"LanceNamespaceDBConnection", "LanceNamespaceDBConnection",
"RemoteDBConnection", "RemoteDBConnection",
"Session", "Session",
"StorageOptionsProvider",
"Table", "Table",
"__version__", "__version__",
] ]

View File

@@ -14,7 +14,6 @@ from .index import (
HnswSq, HnswSq,
FTS, FTS,
) )
from .io import StorageOptionsProvider
from lance_namespace import ( from lance_namespace import (
ListNamespacesResponse, ListNamespacesResponse,
CreateNamespaceResponse, CreateNamespaceResponse,
@@ -27,6 +26,32 @@ from .remote import ClientConfig
IvfHnswPq: type[HnswPq] = HnswPq IvfHnswPq: type[HnswPq] = HnswPq
IvfHnswSq: type[HnswSq] = HnswSq IvfHnswSq: type[HnswSq] = HnswSq
class PyExpr:
"""A type-safe DataFusion expression node (Rust-side handle)."""
def eq(self, other: "PyExpr") -> "PyExpr": ...
def ne(self, other: "PyExpr") -> "PyExpr": ...
def lt(self, other: "PyExpr") -> "PyExpr": ...
def lte(self, other: "PyExpr") -> "PyExpr": ...
def gt(self, other: "PyExpr") -> "PyExpr": ...
def gte(self, other: "PyExpr") -> "PyExpr": ...
def and_(self, other: "PyExpr") -> "PyExpr": ...
def or_(self, other: "PyExpr") -> "PyExpr": ...
def not_(self) -> "PyExpr": ...
def add(self, other: "PyExpr") -> "PyExpr": ...
def sub(self, other: "PyExpr") -> "PyExpr": ...
def mul(self, other: "PyExpr") -> "PyExpr": ...
def div(self, other: "PyExpr") -> "PyExpr": ...
def lower(self) -> "PyExpr": ...
def upper(self) -> "PyExpr": ...
def contains(self, substr: "PyExpr") -> "PyExpr": ...
def cast(self, data_type: pa.DataType) -> "PyExpr": ...
def to_sql(self) -> str: ...
def expr_col(name: str) -> PyExpr: ...
def expr_lit(value: Union[bool, int, float, str]) -> PyExpr: ...
def expr_func(name: str, args: List[PyExpr]) -> PyExpr: ...
class Session: class Session:
def __init__( def __init__(
self, self,
@@ -46,35 +71,35 @@ class Connection(object):
async def close(self): ... async def close(self): ...
async def list_namespaces( async def list_namespaces(
self, self,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> ListNamespacesResponse: ... ) -> ListNamespacesResponse: ...
async def create_namespace( async def create_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
mode: Optional[str] = None, mode: Optional[str] = None,
properties: Optional[Dict[str, str]] = None, properties: Optional[Dict[str, str]] = None,
) -> CreateNamespaceResponse: ... ) -> CreateNamespaceResponse: ...
async def drop_namespace( async def drop_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
mode: Optional[str] = None, mode: Optional[str] = None,
behavior: Optional[str] = None, behavior: Optional[str] = None,
) -> DropNamespaceResponse: ... ) -> DropNamespaceResponse: ...
async def describe_namespace( async def describe_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
) -> DescribeNamespaceResponse: ... ) -> DescribeNamespaceResponse: ...
async def list_tables( async def list_tables(
self, self,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> ListTablesResponse: ... ) -> ListTablesResponse: ...
async def table_names( async def table_names(
self, self,
namespace: Optional[List[str]], namespace_path: Optional[List[str]],
start_after: Optional[str], start_after: Optional[str],
limit: Optional[int], limit: Optional[int],
) -> list[str]: ... # Deprecated: Use list_tables instead ) -> list[str]: ... # Deprecated: Use list_tables instead
@@ -83,9 +108,8 @@ class Connection(object):
name: str, name: str,
mode: str, mode: str,
data: pa.RecordBatchReader, data: pa.RecordBatchReader,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional[StorageOptionsProvider] = None,
location: Optional[str] = None, location: Optional[str] = None,
) -> Table: ... ) -> Table: ...
async def create_empty_table( async def create_empty_table(
@@ -93,17 +117,15 @@ class Connection(object):
name: str, name: str,
mode: str, mode: str,
schema: pa.Schema, schema: pa.Schema,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional[StorageOptionsProvider] = None,
location: Optional[str] = None, location: Optional[str] = None,
) -> Table: ... ) -> Table: ...
async def open_table( async def open_table(
self, self,
name: str, name: str,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional[StorageOptionsProvider] = None,
index_cache_size: Optional[int] = None, index_cache_size: Optional[int] = None,
location: Optional[str] = None, location: Optional[str] = None,
) -> Table: ... ) -> Table: ...
@@ -111,7 +133,7 @@ class Connection(object):
self, self,
target_table_name: str, target_table_name: str,
source_uri: str, source_uri: str,
target_namespace: Optional[List[str]] = None, target_namespace_path: Optional[List[str]] = None,
source_version: Optional[int] = None, source_version: Optional[int] = None,
source_tag: Optional[str] = None, source_tag: Optional[str] = None,
is_shallow: bool = True, is_shallow: bool = True,
@@ -120,13 +142,15 @@ class Connection(object):
self, self,
cur_name: str, cur_name: str,
new_name: str, new_name: str,
cur_namespace: Optional[List[str]] = None, cur_namespace_path: Optional[List[str]] = None,
new_namespace: Optional[List[str]] = None, new_namespace_path: Optional[List[str]] = None,
) -> None: ... ) -> None: ...
async def drop_table( async def drop_table(
self, name: str, namespace: Optional[List[str]] = None self, name: str, namespace_path: Optional[List[str]] = None
) -> None: ...
async def drop_all_tables(
self, namespace_path: Optional[List[str]] = None
) -> None: ... ) -> None: ...
async def drop_all_tables(self, namespace: Optional[List[str]] = None) -> None: ...
class Table: class Table:
def name(self) -> str: ... def name(self) -> str: ...
@@ -135,7 +159,10 @@ class Table:
def close(self) -> None: ... def close(self) -> None: ...
async def schema(self) -> pa.Schema: ... async def schema(self) -> pa.Schema: ...
async def add( async def add(
self, data: pa.RecordBatchReader, mode: Literal["append", "overwrite"] self,
data: pa.RecordBatchReader,
mode: Literal["append", "overwrite"],
progress: Optional[Any] = None,
) -> AddResult: ... ) -> AddResult: ...
async def update( async def update(
self, updates: Dict[str, str], where: Optional[str] self, updates: Dict[str, str], where: Optional[str]
@@ -166,6 +193,8 @@ class Table:
async def checkout(self, version: Union[int, str]): ... async def checkout(self, version: Union[int, str]): ...
async def checkout_latest(self): ... async def checkout_latest(self): ...
async def restore(self, version: Optional[Union[int, str]] = None): ... async def restore(self, version: Optional[Union[int, str]] = None): ...
async def prewarm_index(self, index_name: str) -> None: ...
async def prewarm_data(self, columns: Optional[List[str]] = None) -> None: ...
async def list_indices(self) -> list[IndexConfig]: ... async def list_indices(self) -> list[IndexConfig]: ...
async def delete(self, filter: str) -> DeleteResult: ... async def delete(self, filter: str) -> DeleteResult: ...
async def add_columns(self, columns: list[tuple[str, str]]) -> AddColumnsResult: ... async def add_columns(self, columns: list[tuple[str, str]]) -> AddColumnsResult: ...
@@ -220,7 +249,9 @@ class RecordBatchStream:
class Query: class Query:
def where(self, filter: str): ... def where(self, filter: str): ...
def select(self, columns: Tuple[str, str]): ... def where_expr(self, expr: PyExpr): ...
def select(self, columns: List[Tuple[str, str]]): ...
def select_expr(self, columns: List[Tuple[str, PyExpr]]): ...
def select_columns(self, columns: List[str]): ... def select_columns(self, columns: List[str]): ...
def limit(self, limit: int): ... def limit(self, limit: int): ...
def offset(self, offset: int): ... def offset(self, offset: int): ...
@@ -246,7 +277,9 @@ class TakeQuery:
class FTSQuery: class FTSQuery:
def where(self, filter: str): ... def where(self, filter: str): ...
def select(self, columns: List[str]): ... def where_expr(self, expr: PyExpr): ...
def select(self, columns: List[Tuple[str, str]]): ...
def select_expr(self, columns: List[Tuple[str, PyExpr]]): ...
def limit(self, limit: int): ... def limit(self, limit: int): ...
def offset(self, offset: int): ... def offset(self, offset: int): ...
def fast_search(self): ... def fast_search(self): ...
@@ -265,7 +298,9 @@ class VectorQuery:
async def output_schema(self) -> pa.Schema: ... async def output_schema(self) -> pa.Schema: ...
async def execute(self) -> RecordBatchStream: ... async def execute(self) -> RecordBatchStream: ...
def where(self, filter: str): ... def where(self, filter: str): ...
def select(self, columns: List[str]): ... def where_expr(self, expr: PyExpr): ...
def select(self, columns: List[Tuple[str, str]]): ...
def select_expr(self, columns: List[Tuple[str, PyExpr]]): ...
def select_with_projection(self, columns: Tuple[str, str]): ... def select_with_projection(self, columns: Tuple[str, str]): ...
def limit(self, limit: int): ... def limit(self, limit: int): ...
def offset(self, offset: int): ... def offset(self, offset: int): ...
@@ -282,7 +317,9 @@ class VectorQuery:
class HybridQuery: class HybridQuery:
def where(self, filter: str): ... def where(self, filter: str): ...
def select(self, columns: List[str]): ... def where_expr(self, expr: PyExpr): ...
def select(self, columns: List[Tuple[str, str]]): ...
def select_expr(self, columns: List[Tuple[str, PyExpr]]): ...
def limit(self, limit: int): ... def limit(self, limit: int): ...
def offset(self, offset: int): ... def offset(self, offset: int): ...
def fast_search(self): ... def fast_search(self): ...

View File

@@ -8,7 +8,7 @@ from abc import abstractmethod
from datetime import timedelta from datetime import timedelta
from pathlib import Path from pathlib import Path
import sys import sys
from typing import TYPE_CHECKING, Dict, Iterable, List, Literal, Optional, Union from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Literal, Optional, Union
if sys.version_info >= (3, 12): if sys.version_info >= (3, 12):
from typing import override from typing import override
@@ -52,7 +52,6 @@ if TYPE_CHECKING:
from ._lancedb import Connection as LanceDbConnection from ._lancedb import Connection as LanceDbConnection
from .common import DATA, URI from .common import DATA, URI
from .embeddings import EmbeddingFunctionConfig from .embeddings import EmbeddingFunctionConfig
from .io import StorageOptionsProvider
from ._lancedb import Session from ._lancedb import Session
from .namespace_utils import ( from .namespace_utils import (
@@ -67,7 +66,7 @@ class DBConnection(EnforceOverrides):
def list_namespaces( def list_namespaces(
self, self,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> ListNamespacesResponse: ) -> ListNamespacesResponse:
@@ -75,7 +74,7 @@ class DBConnection(EnforceOverrides):
Parameters Parameters
---------- ----------
namespace: List[str], default [] namespace_path: List[str], default []
The parent namespace to list namespaces in. The parent namespace to list namespaces in.
Empty list represents root namespace. Empty list represents root namespace.
page_token: str, optional page_token: str, optional
@@ -89,13 +88,13 @@ class DBConnection(EnforceOverrides):
ListNamespacesResponse ListNamespacesResponse
Response containing namespace names and optional page_token for pagination. Response containing namespace names and optional page_token for pagination.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
return ListNamespacesResponse(namespaces=[], page_token=None) return ListNamespacesResponse(namespaces=[], page_token=None)
def create_namespace( def create_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
mode: Optional[str] = None, mode: Optional[str] = None,
properties: Optional[Dict[str, str]] = None, properties: Optional[Dict[str, str]] = None,
) -> CreateNamespaceResponse: ) -> CreateNamespaceResponse:
@@ -103,7 +102,7 @@ class DBConnection(EnforceOverrides):
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to create. The namespace identifier to create.
mode: str, optional mode: str, optional
Creation mode - "create" (fail if exists), "exist_ok" (skip if exists), Creation mode - "create" (fail if exists), "exist_ok" (skip if exists),
@@ -122,7 +121,7 @@ class DBConnection(EnforceOverrides):
def drop_namespace( def drop_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
mode: Optional[str] = None, mode: Optional[str] = None,
behavior: Optional[str] = None, behavior: Optional[str] = None,
) -> DropNamespaceResponse: ) -> DropNamespaceResponse:
@@ -130,7 +129,7 @@ class DBConnection(EnforceOverrides):
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to drop. The namespace identifier to drop.
mode: str, optional mode: str, optional
Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive. Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.
@@ -147,12 +146,14 @@ class DBConnection(EnforceOverrides):
"Namespace operations are not supported for this connection type" "Namespace operations are not supported for this connection type"
) )
def describe_namespace(self, namespace: List[str]) -> DescribeNamespaceResponse: def describe_namespace(
self, namespace_path: List[str]
) -> DescribeNamespaceResponse:
"""Describe a namespace. """Describe a namespace.
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to describe. The namespace identifier to describe.
Returns Returns
@@ -166,7 +167,7 @@ class DBConnection(EnforceOverrides):
def list_tables( def list_tables(
self, self,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> ListTablesResponse: ) -> ListTablesResponse:
@@ -174,7 +175,7 @@ class DBConnection(EnforceOverrides):
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The namespace to list tables in. The namespace to list tables in.
None or empty list represents root namespace. None or empty list represents root namespace.
page_token: str, optional page_token: str, optional
@@ -198,13 +199,13 @@ class DBConnection(EnforceOverrides):
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: int = 10, limit: int = 10,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
) -> Iterable[str]: ) -> Iterable[str]:
"""List all tables in this database, in sorted order """List all tables in this database, in sorted order
Parameters Parameters
---------- ----------
namespace: List[str], default [] namespace_path: List[str], default []
The namespace to list tables in. The namespace to list tables in.
Empty list represents root namespace. Empty list represents root namespace.
page_token: str, optional page_token: str, optional
@@ -231,9 +232,8 @@ class DBConnection(EnforceOverrides):
fill_value: float = 0.0, fill_value: float = 0.0,
embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None, embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional["StorageOptionsProvider"] = None,
data_storage_version: Optional[str] = None, data_storage_version: Optional[str] = None,
enable_v2_manifest_paths: Optional[bool] = None, enable_v2_manifest_paths: Optional[bool] = None,
) -> Table: ) -> Table:
@@ -243,7 +243,7 @@ class DBConnection(EnforceOverrides):
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], default [] namespace_path: List[str], default []
The namespace to create the table in. The namespace to create the table in.
Empty list represents root namespace. Empty list represents root namespace.
data: The data to initialize the table, *optional* data: The data to initialize the table, *optional*
@@ -401,9 +401,8 @@ class DBConnection(EnforceOverrides):
self, self,
name: str, name: str,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional["StorageOptionsProvider"] = None,
index_cache_size: Optional[int] = None, index_cache_size: Optional[int] = None,
) -> Table: ) -> Table:
"""Open a Lance Table in the database. """Open a Lance Table in the database.
@@ -412,7 +411,7 @@ class DBConnection(EnforceOverrides):
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], optional namespace_path: List[str], optional
The namespace to open the table from. The namespace to open the table from.
None or empty list represents root namespace. None or empty list represents root namespace.
index_cache_size: int, default 256 index_cache_size: int, default 256
@@ -440,27 +439,27 @@ class DBConnection(EnforceOverrides):
""" """
raise NotImplementedError raise NotImplementedError
def drop_table(self, name: str, namespace: Optional[List[str]] = None): def drop_table(self, name: str, namespace_path: Optional[List[str]] = None):
"""Drop a table from the database. """Drop a table from the database.
Parameters Parameters
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], default [] namespace_path: List[str], default []
The namespace to drop the table from. The namespace to drop the table from.
Empty list represents root namespace. Empty list represents root namespace.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
raise NotImplementedError raise NotImplementedError
def rename_table( def rename_table(
self, self,
cur_name: str, cur_name: str,
new_name: str, new_name: str,
cur_namespace: Optional[List[str]] = None, cur_namespace_path: Optional[List[str]] = None,
new_namespace: Optional[List[str]] = None, new_namespace_path: Optional[List[str]] = None,
): ):
"""Rename a table in the database. """Rename a table in the database.
@@ -470,17 +469,17 @@ class DBConnection(EnforceOverrides):
The current name of the table. The current name of the table.
new_name: str new_name: str
The new name of the table. The new name of the table.
cur_namespace: List[str], optional cur_namespace_path: List[str], optional
The namespace of the current table. The namespace of the current table.
None or empty list represents root namespace. None or empty list represents root namespace.
new_namespace: List[str], optional new_namespace_path: List[str], optional
The namespace to move the table to. The namespace to move the table to.
If not specified, defaults to the same as cur_namespace. If not specified, defaults to the same as cur_namespace.
""" """
if cur_namespace is None: if cur_namespace_path is None:
cur_namespace = [] cur_namespace_path = []
if new_namespace is None: if new_namespace_path is None:
new_namespace = [] new_namespace_path = []
raise NotImplementedError raise NotImplementedError
def drop_database(self): def drop_database(self):
@@ -490,18 +489,18 @@ class DBConnection(EnforceOverrides):
""" """
raise NotImplementedError raise NotImplementedError
def drop_all_tables(self, namespace: Optional[List[str]] = None): def drop_all_tables(self, namespace_path: Optional[List[str]] = None):
""" """
Drop all tables from the database Drop all tables from the database
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The namespace to drop all tables from. The namespace to drop all tables from.
None or empty list represents root namespace. None or empty list represents root namespace.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
raise NotImplementedError raise NotImplementedError
@property @property
@@ -642,7 +641,7 @@ class LanceDBConnection(DBConnection):
@override @override
def list_namespaces( def list_namespaces(
self, self,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> ListNamespacesResponse: ) -> ListNamespacesResponse:
@@ -650,7 +649,7 @@ class LanceDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The parent namespace to list namespaces in. The parent namespace to list namespaces in.
None or empty list represents root namespace. None or empty list represents root namespace.
page_token: str, optional page_token: str, optional
@@ -664,18 +663,18 @@ class LanceDBConnection(DBConnection):
ListNamespacesResponse ListNamespacesResponse
Response containing namespace names and optional page_token for pagination. Response containing namespace names and optional page_token for pagination.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
return LOOP.run( return LOOP.run(
self._conn.list_namespaces( self._conn.list_namespaces(
namespace=namespace, page_token=page_token, limit=limit namespace_path=namespace_path, page_token=page_token, limit=limit
) )
) )
@override @override
def create_namespace( def create_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
mode: Optional[str] = None, mode: Optional[str] = None,
properties: Optional[Dict[str, str]] = None, properties: Optional[Dict[str, str]] = None,
) -> CreateNamespaceResponse: ) -> CreateNamespaceResponse:
@@ -683,7 +682,7 @@ class LanceDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to create. The namespace identifier to create.
mode: str, optional mode: str, optional
Creation mode - "create" (fail if exists), "exist_ok" (skip if exists), Creation mode - "create" (fail if exists), "exist_ok" (skip if exists),
@@ -698,14 +697,14 @@ class LanceDBConnection(DBConnection):
""" """
return LOOP.run( return LOOP.run(
self._conn.create_namespace( self._conn.create_namespace(
namespace=namespace, mode=mode, properties=properties namespace_path=namespace_path, mode=mode, properties=properties
) )
) )
@override @override
def drop_namespace( def drop_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
mode: Optional[str] = None, mode: Optional[str] = None,
behavior: Optional[str] = None, behavior: Optional[str] = None,
) -> DropNamespaceResponse: ) -> DropNamespaceResponse:
@@ -713,7 +712,7 @@ class LanceDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to drop. The namespace identifier to drop.
mode: str, optional mode: str, optional
Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive. Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.
@@ -727,16 +726,20 @@ class LanceDBConnection(DBConnection):
Response containing properties and transaction_id if applicable. Response containing properties and transaction_id if applicable.
""" """
return LOOP.run( return LOOP.run(
self._conn.drop_namespace(namespace=namespace, mode=mode, behavior=behavior) self._conn.drop_namespace(
namespace_path=namespace_path, mode=mode, behavior=behavior
)
) )
@override @override
def describe_namespace(self, namespace: List[str]) -> DescribeNamespaceResponse: def describe_namespace(
self, namespace_path: List[str]
) -> DescribeNamespaceResponse:
"""Describe a namespace. """Describe a namespace.
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to describe. The namespace identifier to describe.
Returns Returns
@@ -744,12 +747,12 @@ class LanceDBConnection(DBConnection):
DescribeNamespaceResponse DescribeNamespaceResponse
Response containing the namespace properties. Response containing the namespace properties.
""" """
return LOOP.run(self._conn.describe_namespace(namespace=namespace)) return LOOP.run(self._conn.describe_namespace(namespace_path=namespace_path))
@override @override
def list_tables( def list_tables(
self, self,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> ListTablesResponse: ) -> ListTablesResponse:
@@ -757,7 +760,7 @@ class LanceDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The namespace to list tables in. The namespace to list tables in.
None or empty list represents root namespace. None or empty list represents root namespace.
page_token: str, optional page_token: str, optional
@@ -771,11 +774,11 @@ class LanceDBConnection(DBConnection):
ListTablesResponse ListTablesResponse
Response containing table names and optional page_token for pagination. Response containing table names and optional page_token for pagination.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
return LOOP.run( return LOOP.run(
self._conn.list_tables( self._conn.list_tables(
namespace=namespace, page_token=page_token, limit=limit namespace_path=namespace_path, page_token=page_token, limit=limit
) )
) )
@@ -785,7 +788,7 @@ class LanceDBConnection(DBConnection):
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: int = 10, limit: int = 10,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
) -> Iterable[str]: ) -> Iterable[str]:
"""Get the names of all tables in the database. The names are sorted. """Get the names of all tables in the database. The names are sorted.
@@ -794,7 +797,7 @@ class LanceDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The namespace to list tables in. The namespace to list tables in.
page_token: str, optional page_token: str, optional
The token to use for pagination. The token to use for pagination.
@@ -813,11 +816,11 @@ class LanceDBConnection(DBConnection):
DeprecationWarning, DeprecationWarning,
stacklevel=2, stacklevel=2,
) )
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
return LOOP.run( return LOOP.run(
self._conn.table_names( self._conn.table_names(
namespace=namespace, start_after=page_token, limit=limit namespace_path=namespace_path, start_after=page_token, limit=limit
) )
) )
@@ -839,9 +842,8 @@ class LanceDBConnection(DBConnection):
fill_value: float = 0.0, fill_value: float = 0.0,
embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None, embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional["StorageOptionsProvider"] = None,
data_storage_version: Optional[str] = None, data_storage_version: Optional[str] = None,
enable_v2_manifest_paths: Optional[bool] = None, enable_v2_manifest_paths: Optional[bool] = None,
) -> LanceTable: ) -> LanceTable:
@@ -849,15 +851,15 @@ class LanceDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The namespace to create the table in. The namespace to create the table in.
See See
--- ---
DBConnection.create_table DBConnection.create_table
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
if mode.lower() not in ["create", "overwrite"]: if mode.lower() not in ["create", "overwrite"]:
raise ValueError("mode must be either 'create' or 'overwrite'") raise ValueError("mode must be either 'create' or 'overwrite'")
validate_table_name(name) validate_table_name(name)
@@ -872,9 +874,8 @@ class LanceDBConnection(DBConnection):
on_bad_vectors=on_bad_vectors, on_bad_vectors=on_bad_vectors,
fill_value=fill_value, fill_value=fill_value,
embedding_functions=embedding_functions, embedding_functions=embedding_functions,
namespace=namespace, namespace_path=namespace_path,
storage_options=storage_options, storage_options=storage_options,
storage_options_provider=storage_options_provider,
) )
return tbl return tbl
@@ -883,9 +884,8 @@ class LanceDBConnection(DBConnection):
self, self,
name: str, name: str,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional["StorageOptionsProvider"] = None,
index_cache_size: Optional[int] = None, index_cache_size: Optional[int] = None,
) -> LanceTable: ) -> LanceTable:
"""Open a table in the database. """Open a table in the database.
@@ -894,15 +894,15 @@ class LanceDBConnection(DBConnection):
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], optional namespace_path: List[str], optional
The namespace to open the table from. The namespace to open the table from.
Returns Returns
------- -------
A LanceTable object representing the table. A LanceTable object representing the table.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
if index_cache_size is not None: if index_cache_size is not None:
import warnings import warnings
@@ -917,9 +917,8 @@ class LanceDBConnection(DBConnection):
return LanceTable.open( return LanceTable.open(
self, self,
name, name,
namespace=namespace, namespace_path=namespace_path,
storage_options=storage_options, storage_options=storage_options,
storage_options_provider=storage_options_provider,
index_cache_size=index_cache_size, index_cache_size=index_cache_size,
) )
@@ -928,7 +927,7 @@ class LanceDBConnection(DBConnection):
target_table_name: str, target_table_name: str,
source_uri: str, source_uri: str,
*, *,
target_namespace: Optional[List[str]] = None, target_namespace_path: Optional[List[str]] = None,
source_version: Optional[int] = None, source_version: Optional[int] = None,
source_tag: Optional[str] = None, source_tag: Optional[str] = None,
is_shallow: bool = True, is_shallow: bool = True,
@@ -946,7 +945,7 @@ class LanceDBConnection(DBConnection):
The name of the target table to create. The name of the target table to create.
source_uri: str source_uri: str
The URI of the source table to clone from. The URI of the source table to clone from.
target_namespace: List[str], optional target_namespace_path: List[str], optional
The namespace for the target table. The namespace for the target table.
None or empty list represents root namespace. None or empty list represents root namespace.
source_version: int, optional source_version: int, optional
@@ -961,13 +960,13 @@ class LanceDBConnection(DBConnection):
------- -------
A LanceTable object representing the cloned table. A LanceTable object representing the cloned table.
""" """
if target_namespace is None: if target_namespace_path is None:
target_namespace = [] target_namespace_path = []
LOOP.run( LOOP.run(
self._conn.clone_table( self._conn.clone_table(
target_table_name, target_table_name,
source_uri, source_uri,
target_namespace=target_namespace, target_namespace_path=target_namespace_path,
source_version=source_version, source_version=source_version,
source_tag=source_tag, source_tag=source_tag,
is_shallow=is_shallow, is_shallow=is_shallow,
@@ -976,14 +975,14 @@ class LanceDBConnection(DBConnection):
return LanceTable.open( return LanceTable.open(
self, self,
target_table_name, target_table_name,
namespace=target_namespace, namespace_path=target_namespace_path,
) )
@override @override
def drop_table( def drop_table(
self, self,
name: str, name: str,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
ignore_missing: bool = False, ignore_missing: bool = False,
): ):
"""Drop a table from the database. """Drop a table from the database.
@@ -992,32 +991,32 @@ class LanceDBConnection(DBConnection):
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], optional namespace_path: List[str], optional
The namespace to drop the table from. The namespace to drop the table from.
ignore_missing: bool, default False ignore_missing: bool, default False
If True, ignore if the table does not exist. If True, ignore if the table does not exist.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
LOOP.run( LOOP.run(
self._conn.drop_table( self._conn.drop_table(
name, namespace=namespace, ignore_missing=ignore_missing name, namespace_path=namespace_path, ignore_missing=ignore_missing
) )
) )
@override @override
def drop_all_tables(self, namespace: Optional[List[str]] = None): def drop_all_tables(self, namespace_path: Optional[List[str]] = None):
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
LOOP.run(self._conn.drop_all_tables(namespace=namespace)) LOOP.run(self._conn.drop_all_tables(namespace_path=namespace_path))
@override @override
def rename_table( def rename_table(
self, self,
cur_name: str, cur_name: str,
new_name: str, new_name: str,
cur_namespace: Optional[List[str]] = None, cur_namespace_path: Optional[List[str]] = None,
new_namespace: Optional[List[str]] = None, new_namespace_path: Optional[List[str]] = None,
): ):
"""Rename a table in the database. """Rename a table in the database.
@@ -1027,21 +1026,21 @@ class LanceDBConnection(DBConnection):
The current name of the table. The current name of the table.
new_name: str new_name: str
The new name of the table. The new name of the table.
cur_namespace: List[str], optional cur_namespace_path: List[str], optional
The namespace of the current table. The namespace of the current table.
new_namespace: List[str], optional new_namespace_path: List[str], optional
The namespace to move the table to. The namespace to move the table to.
""" """
if cur_namespace is None: if cur_namespace_path is None:
cur_namespace = [] cur_namespace_path = []
if new_namespace is None: if new_namespace_path is None:
new_namespace = [] new_namespace_path = []
LOOP.run( LOOP.run(
self._conn.rename_table( self._conn.rename_table(
cur_name, cur_name,
new_name, new_name,
cur_namespace=cur_namespace, cur_namespace_path=cur_namespace_path,
new_namespace=new_namespace, new_namespace_path=new_namespace_path,
) )
) )
@@ -1125,7 +1124,7 @@ class AsyncConnection(object):
async def list_namespaces( async def list_namespaces(
self, self,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> ListNamespacesResponse: ) -> ListNamespacesResponse:
@@ -1133,7 +1132,7 @@ class AsyncConnection(object):
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The parent namespace to list namespaces in. The parent namespace to list namespaces in.
None or empty list represents root namespace. None or empty list represents root namespace.
page_token: str, optional page_token: str, optional
@@ -1146,16 +1145,16 @@ class AsyncConnection(object):
ListNamespacesResponse ListNamespacesResponse
Response containing namespace names and optional pagination token Response containing namespace names and optional pagination token
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
result = await self._inner.list_namespaces( result = await self._inner.list_namespaces(
namespace=namespace, page_token=page_token, limit=limit namespace_path=namespace_path, page_token=page_token, limit=limit
) )
return ListNamespacesResponse(**result) return ListNamespacesResponse(**result)
async def create_namespace( async def create_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
mode: Optional[str] = None, mode: Optional[str] = None,
properties: Optional[Dict[str, str]] = None, properties: Optional[Dict[str, str]] = None,
) -> CreateNamespaceResponse: ) -> CreateNamespaceResponse:
@@ -1163,7 +1162,7 @@ class AsyncConnection(object):
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to create. The namespace identifier to create.
mode: str, optional mode: str, optional
Creation mode - "create", "exist_ok", or "overwrite". Case insensitive. Creation mode - "create", "exist_ok", or "overwrite". Case insensitive.
@@ -1176,7 +1175,7 @@ class AsyncConnection(object):
Response containing namespace properties Response containing namespace properties
""" """
result = await self._inner.create_namespace( result = await self._inner.create_namespace(
namespace, namespace_path,
mode=_normalize_create_namespace_mode(mode), mode=_normalize_create_namespace_mode(mode),
properties=properties, properties=properties,
) )
@@ -1184,7 +1183,7 @@ class AsyncConnection(object):
async def drop_namespace( async def drop_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
mode: Optional[str] = None, mode: Optional[str] = None,
behavior: Optional[str] = None, behavior: Optional[str] = None,
) -> DropNamespaceResponse: ) -> DropNamespaceResponse:
@@ -1192,7 +1191,7 @@ class AsyncConnection(object):
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to drop. The namespace identifier to drop.
mode: str, optional mode: str, optional
Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive. Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.
@@ -1206,20 +1205,20 @@ class AsyncConnection(object):
Response containing properties and transaction_id if applicable. Response containing properties and transaction_id if applicable.
""" """
result = await self._inner.drop_namespace( result = await self._inner.drop_namespace(
namespace, namespace_path,
mode=_normalize_drop_namespace_mode(mode), mode=_normalize_drop_namespace_mode(mode),
behavior=_normalize_drop_namespace_behavior(behavior), behavior=_normalize_drop_namespace_behavior(behavior),
) )
return DropNamespaceResponse(**result) return DropNamespaceResponse(**result)
async def describe_namespace( async def describe_namespace(
self, namespace: List[str] self, namespace_path: List[str]
) -> DescribeNamespaceResponse: ) -> DescribeNamespaceResponse:
"""Describe a namespace. """Describe a namespace.
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to describe. The namespace identifier to describe.
Returns Returns
@@ -1227,12 +1226,12 @@ class AsyncConnection(object):
DescribeNamespaceResponse DescribeNamespaceResponse
Response containing the namespace properties. Response containing the namespace properties.
""" """
result = await self._inner.describe_namespace(namespace) result = await self._inner.describe_namespace(namespace_path)
return DescribeNamespaceResponse(**result) return DescribeNamespaceResponse(**result)
async def list_tables( async def list_tables(
self, self,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> ListTablesResponse: ) -> ListTablesResponse:
@@ -1240,7 +1239,7 @@ class AsyncConnection(object):
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The namespace to list tables in. The namespace to list tables in.
None or empty list represents root namespace. None or empty list represents root namespace.
page_token: str, optional page_token: str, optional
@@ -1254,17 +1253,17 @@ class AsyncConnection(object):
ListTablesResponse ListTablesResponse
Response containing table names and optional page_token for pagination. Response containing table names and optional page_token for pagination.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
result = await self._inner.list_tables( result = await self._inner.list_tables(
namespace=namespace, page_token=page_token, limit=limit namespace_path=namespace_path, page_token=page_token, limit=limit
) )
return ListTablesResponse(**result) return ListTablesResponse(**result)
async def table_names( async def table_names(
self, self,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
start_after: Optional[str] = None, start_after: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> Iterable[str]: ) -> Iterable[str]:
@@ -1275,7 +1274,7 @@ class AsyncConnection(object):
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The namespace to list tables in. The namespace to list tables in.
None or empty list represents root namespace. None or empty list represents root namespace.
start_after: str, optional start_after: str, optional
@@ -1298,10 +1297,10 @@ class AsyncConnection(object):
DeprecationWarning, DeprecationWarning,
stacklevel=2, stacklevel=2,
) )
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
return await self._inner.table_names( return await self._inner.table_names(
namespace=namespace, start_after=start_after, limit=limit namespace_path=namespace_path, start_after=start_after, limit=limit
) )
async def create_table( async def create_table(
@@ -1314,9 +1313,8 @@ class AsyncConnection(object):
on_bad_vectors: Optional[str] = None, on_bad_vectors: Optional[str] = None,
fill_value: Optional[float] = None, fill_value: Optional[float] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional["StorageOptionsProvider"] = None,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None, embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None,
location: Optional[str] = None, location: Optional[str] = None,
) -> AsyncTable: ) -> AsyncTable:
@@ -1326,7 +1324,7 @@ class AsyncConnection(object):
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], default [] namespace_path: List[str], default []
The namespace to create the table in. The namespace to create the table in.
Empty list represents root namespace. Empty list represents root namespace.
data: The data to initialize the table, *optional* data: The data to initialize the table, *optional*
@@ -1477,8 +1475,8 @@ class AsyncConnection(object):
... await db.create_table("table4", make_batches(), schema=schema) ... await db.create_table("table4", make_batches(), schema=schema)
>>> asyncio.run(iterable_example()) >>> asyncio.run(iterable_example())
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
metadata = None metadata = None
if embedding_functions is not None: if embedding_functions is not None:
@@ -1513,9 +1511,8 @@ class AsyncConnection(object):
name, name,
mode, mode,
schema, schema,
namespace=namespace, namespace_path=namespace_path,
storage_options=storage_options, storage_options=storage_options,
storage_options_provider=storage_options_provider,
location=location, location=location,
) )
else: else:
@@ -1524,9 +1521,8 @@ class AsyncConnection(object):
name, name,
mode, mode,
data, data,
namespace=namespace, namespace_path=namespace_path,
storage_options=storage_options, storage_options=storage_options,
storage_options_provider=storage_options_provider,
location=location, location=location,
) )
@@ -1536,11 +1532,12 @@ class AsyncConnection(object):
self, self,
name: str, name: str,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional["StorageOptionsProvider"] = None,
index_cache_size: Optional[int] = None, index_cache_size: Optional[int] = None,
location: Optional[str] = None, location: Optional[str] = None,
namespace_client: Optional[Any] = None,
managed_versioning: Optional[bool] = None,
) -> AsyncTable: ) -> AsyncTable:
"""Open a Lance Table in the database. """Open a Lance Table in the database.
@@ -1548,7 +1545,7 @@ class AsyncConnection(object):
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], optional namespace_path: List[str], optional
The namespace to open the table from. The namespace to open the table from.
None or empty list represents root namespace. None or empty list represents root namespace.
storage_options: dict, optional storage_options: dict, optional
@@ -1573,20 +1570,24 @@ class AsyncConnection(object):
The explicit location (URI) of the table. If provided, the table will be The explicit location (URI) of the table. If provided, the table will be
opened from this location instead of deriving it from the database URI opened from this location instead of deriving it from the database URI
and table name. and table name.
managed_versioning: bool, optional
Whether managed versioning is enabled for this table. If provided,
avoids a redundant describe_table call when namespace_client is set.
Returns Returns
------- -------
A LanceTable object representing the table. A LanceTable object representing the table.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
table = await self._inner.open_table( table = await self._inner.open_table(
name, name,
namespace=namespace, namespace_path=namespace_path,
storage_options=storage_options, storage_options=storage_options,
storage_options_provider=storage_options_provider,
index_cache_size=index_cache_size, index_cache_size=index_cache_size,
location=location, location=location,
namespace_client=namespace_client,
managed_versioning=managed_versioning,
) )
return AsyncTable(table) return AsyncTable(table)
@@ -1595,7 +1596,7 @@ class AsyncConnection(object):
target_table_name: str, target_table_name: str,
source_uri: str, source_uri: str,
*, *,
target_namespace: Optional[List[str]] = None, target_namespace_path: Optional[List[str]] = None,
source_version: Optional[int] = None, source_version: Optional[int] = None,
source_tag: Optional[str] = None, source_tag: Optional[str] = None,
is_shallow: bool = True, is_shallow: bool = True,
@@ -1613,7 +1614,7 @@ class AsyncConnection(object):
The name of the target table to create. The name of the target table to create.
source_uri: str source_uri: str
The URI of the source table to clone from. The URI of the source table to clone from.
target_namespace: List[str], optional target_namespace_path: List[str], optional
The namespace for the target table. The namespace for the target table.
None or empty list represents root namespace. None or empty list represents root namespace.
source_version: int, optional source_version: int, optional
@@ -1628,12 +1629,12 @@ class AsyncConnection(object):
------- -------
An AsyncTable object representing the cloned table. An AsyncTable object representing the cloned table.
""" """
if target_namespace is None: if target_namespace_path is None:
target_namespace = [] target_namespace_path = []
table = await self._inner.clone_table( table = await self._inner.clone_table(
target_table_name, target_table_name,
source_uri, source_uri,
target_namespace=target_namespace, target_namespace_path=target_namespace_path,
source_version=source_version, source_version=source_version,
source_tag=source_tag, source_tag=source_tag,
is_shallow=is_shallow, is_shallow=is_shallow,
@@ -1644,8 +1645,8 @@ class AsyncConnection(object):
self, self,
cur_name: str, cur_name: str,
new_name: str, new_name: str,
cur_namespace: Optional[List[str]] = None, cur_namespace_path: Optional[List[str]] = None,
new_namespace: Optional[List[str]] = None, new_namespace_path: Optional[List[str]] = None,
): ):
"""Rename a table in the database. """Rename a table in the database.
@@ -1655,26 +1656,29 @@ class AsyncConnection(object):
The current name of the table. The current name of the table.
new_name: str new_name: str
The new name of the table. The new name of the table.
cur_namespace: List[str], optional cur_namespace_path: List[str], optional
The namespace of the current table. The namespace of the current table.
None or empty list represents root namespace. None or empty list represents root namespace.
new_namespace: List[str], optional new_namespace_path: List[str], optional
The namespace to move the table to. The namespace to move the table to.
If not specified, defaults to the same as cur_namespace. If not specified, defaults to the same as cur_namespace.
""" """
if cur_namespace is None: if cur_namespace_path is None:
cur_namespace = [] cur_namespace_path = []
if new_namespace is None: if new_namespace_path is None:
new_namespace = [] new_namespace_path = []
await self._inner.rename_table( await self._inner.rename_table(
cur_name, new_name, cur_namespace=cur_namespace, new_namespace=new_namespace cur_name,
new_name,
cur_namespace_path=cur_namespace_path,
new_namespace_path=new_namespace_path,
) )
async def drop_table( async def drop_table(
self, self,
name: str, name: str,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
ignore_missing: bool = False, ignore_missing: bool = False,
): ):
"""Drop a table from the database. """Drop a table from the database.
@@ -1683,34 +1687,34 @@ class AsyncConnection(object):
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], default [] namespace_path: List[str], default []
The namespace to drop the table from. The namespace to drop the table from.
Empty list represents root namespace. Empty list represents root namespace.
ignore_missing: bool, default False ignore_missing: bool, default False
If True, ignore if the table does not exist. If True, ignore if the table does not exist.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
try: try:
await self._inner.drop_table(name, namespace=namespace) await self._inner.drop_table(name, namespace_path=namespace_path)
except ValueError as e: except ValueError as e:
if not ignore_missing: if not ignore_missing:
raise e raise e
if f"Table '{name}' was not found" not in str(e): if f"Table '{name}' was not found" not in str(e):
raise e raise e
async def drop_all_tables(self, namespace: Optional[List[str]] = None): async def drop_all_tables(self, namespace_path: Optional[List[str]] = None):
"""Drop all tables from the database. """Drop all tables from the database.
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The namespace to drop all tables from. The namespace to drop all tables from.
None or empty list represents root namespace. None or empty list represents root namespace.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
await self._inner.drop_all_tables(namespace=namespace) await self._inner.drop_all_tables(namespace_path=namespace_path)
@deprecation.deprecated( @deprecation.deprecated(
deprecated_in="0.15.1", deprecated_in="0.15.1",

View File

@@ -10,6 +10,7 @@ import sys
import threading import threading
import time import time
import urllib.error import urllib.error
import urllib.request
import weakref import weakref
import logging import logging
from functools import wraps from functools import wraps

View File

@@ -0,0 +1,298 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
"""Type-safe expression builder for filters and projections.
Instead of writing raw SQL strings you can build expressions with Python
operators::
from lancedb.expr import col, lit
# filter: age > 18 AND status = 'active'
filt = (col("age") > lit(18)) & (col("status") == lit("active"))
# projection: compute a derived column
proj = {"score": col("raw_score") * lit(1.5)}
table.search().where(filt).select(proj).to_list()
"""
from __future__ import annotations
from typing import Union
import pyarrow as pa
from lancedb._lancedb import PyExpr, expr_col, expr_lit, expr_func
__all__ = ["Expr", "col", "lit", "func"]
_STR_TO_PA_TYPE: dict = {
"bool": pa.bool_(),
"boolean": pa.bool_(),
"int8": pa.int8(),
"int16": pa.int16(),
"int32": pa.int32(),
"int64": pa.int64(),
"uint8": pa.uint8(),
"uint16": pa.uint16(),
"uint32": pa.uint32(),
"uint64": pa.uint64(),
"float16": pa.float16(),
"float32": pa.float32(),
"float": pa.float32(),
"float64": pa.float64(),
"double": pa.float64(),
"string": pa.string(),
"utf8": pa.string(),
"str": pa.string(),
"large_string": pa.large_utf8(),
"large_utf8": pa.large_utf8(),
"date32": pa.date32(),
"date": pa.date32(),
"date64": pa.date64(),
}
def _coerce(value: "ExprLike") -> "Expr":
"""Return *value* as an :class:`Expr`, wrapping plain Python values via
:func:`lit` if needed."""
if isinstance(value, Expr):
return value
return lit(value)
# Type alias used in annotations.
ExprLike = Union["Expr", bool, int, float, str]
class Expr:
"""A type-safe expression node.
Construct instances with :func:`col` and :func:`lit`, then combine them
using Python operators or the named methods below.
Examples
--------
>>> from lancedb.expr import col, lit
>>> filt = (col("age") > lit(18)) & (col("name").lower() == lit("alice"))
>>> proj = {"double": col("x") * lit(2)}
"""
# Make Expr unhashable so that == returns an Expr rather than being used
# for dict keys / set membership.
__hash__ = None # type: ignore[assignment]
def __init__(self, inner: PyExpr) -> None:
self._inner = inner
# ── comparisons ──────────────────────────────────────────────────────────
def __eq__(self, other: ExprLike) -> "Expr": # type: ignore[override]
"""Equal to (``col("x") == 1``)."""
return Expr(self._inner.eq(_coerce(other)._inner))
def __ne__(self, other: ExprLike) -> "Expr": # type: ignore[override]
"""Not equal to (``col("x") != 1``)."""
return Expr(self._inner.ne(_coerce(other)._inner))
def __lt__(self, other: ExprLike) -> "Expr":
"""Less than (``col("x") < 1``)."""
return Expr(self._inner.lt(_coerce(other)._inner))
def __le__(self, other: ExprLike) -> "Expr":
"""Less than or equal to (``col("x") <= 1``)."""
return Expr(self._inner.lte(_coerce(other)._inner))
def __gt__(self, other: ExprLike) -> "Expr":
"""Greater than (``col("x") > 1``)."""
return Expr(self._inner.gt(_coerce(other)._inner))
def __ge__(self, other: ExprLike) -> "Expr":
"""Greater than or equal to (``col("x") >= 1``)."""
return Expr(self._inner.gte(_coerce(other)._inner))
# ── logical ──────────────────────────────────────────────────────────────
def __and__(self, other: "Expr") -> "Expr":
"""Logical AND (``expr_a & expr_b``)."""
return Expr(self._inner.and_(_coerce(other)._inner))
def __or__(self, other: "Expr") -> "Expr":
"""Logical OR (``expr_a | expr_b``)."""
return Expr(self._inner.or_(_coerce(other)._inner))
def __invert__(self) -> "Expr":
"""Logical NOT (``~expr``)."""
return Expr(self._inner.not_())
# ── arithmetic ───────────────────────────────────────────────────────────
def __add__(self, other: ExprLike) -> "Expr":
"""Add (``col("x") + 1``)."""
return Expr(self._inner.add(_coerce(other)._inner))
def __radd__(self, other: ExprLike) -> "Expr":
"""Right-hand add (``1 + col("x")``)."""
return Expr(_coerce(other)._inner.add(self._inner))
def __sub__(self, other: ExprLike) -> "Expr":
"""Subtract (``col("x") - 1``)."""
return Expr(self._inner.sub(_coerce(other)._inner))
def __rsub__(self, other: ExprLike) -> "Expr":
"""Right-hand subtract (``1 - col("x")``)."""
return Expr(_coerce(other)._inner.sub(self._inner))
def __mul__(self, other: ExprLike) -> "Expr":
"""Multiply (``col("x") * 2``)."""
return Expr(self._inner.mul(_coerce(other)._inner))
def __rmul__(self, other: ExprLike) -> "Expr":
"""Right-hand multiply (``2 * col("x")``)."""
return Expr(_coerce(other)._inner.mul(self._inner))
def __truediv__(self, other: ExprLike) -> "Expr":
"""Divide (``col("x") / 2``)."""
return Expr(self._inner.div(_coerce(other)._inner))
def __rtruediv__(self, other: ExprLike) -> "Expr":
"""Right-hand divide (``1 / col("x")``)."""
return Expr(_coerce(other)._inner.div(self._inner))
# ── string methods ───────────────────────────────────────────────────────
def lower(self) -> "Expr":
"""Convert string column values to lowercase."""
return Expr(self._inner.lower())
def upper(self) -> "Expr":
"""Convert string column values to uppercase."""
return Expr(self._inner.upper())
def contains(self, substr: "ExprLike") -> "Expr":
"""Return True where the string contains *substr*."""
return Expr(self._inner.contains(_coerce(substr)._inner))
# ── type cast ────────────────────────────────────────────────────────────
def cast(self, data_type: Union[str, "pa.DataType"]) -> "Expr":
"""Cast values to *data_type*.
Parameters
----------
data_type:
A PyArrow ``DataType`` (e.g. ``pa.int32()``) or one of the type
name strings: ``"bool"``, ``"int8"``, ``"int16"``, ``"int32"``,
``"int64"``, ``"uint8"````"uint64"``, ``"float32"``,
``"float64"``, ``"string"``, ``"date32"``, ``"date64"``.
"""
if isinstance(data_type, str):
try:
data_type = _STR_TO_PA_TYPE[data_type]
except KeyError:
raise ValueError(
f"unsupported data type: '{data_type}'. Supported: "
f"{', '.join(_STR_TO_PA_TYPE)}"
)
return Expr(self._inner.cast(data_type))
# ── named comparison helpers (alternative to operators) ──────────────────
def eq(self, other: ExprLike) -> "Expr":
"""Equal to."""
return self.__eq__(other)
def ne(self, other: ExprLike) -> "Expr":
"""Not equal to."""
return self.__ne__(other)
def lt(self, other: ExprLike) -> "Expr":
"""Less than."""
return self.__lt__(other)
def lte(self, other: ExprLike) -> "Expr":
"""Less than or equal to."""
return self.__le__(other)
def gt(self, other: ExprLike) -> "Expr":
"""Greater than."""
return self.__gt__(other)
def gte(self, other: ExprLike) -> "Expr":
"""Greater than or equal to."""
return self.__ge__(other)
def and_(self, other: "Expr") -> "Expr":
"""Logical AND."""
return self.__and__(other)
def or_(self, other: "Expr") -> "Expr":
"""Logical OR."""
return self.__or__(other)
# ── utilities ────────────────────────────────────────────────────────────
def to_sql(self) -> str:
"""Render the expression as a SQL string (useful for debugging)."""
return self._inner.to_sql()
def __repr__(self) -> str:
return f"Expr({self._inner.to_sql()})"
# ── free functions ────────────────────────────────────────────────────────────
def col(name: str) -> Expr:
"""Reference a table column by name.
Parameters
----------
name:
The column name.
Examples
--------
>>> from lancedb.expr import col, lit
>>> col("age") > lit(18)
Expr((age > 18))
"""
return Expr(expr_col(name))
def lit(value: Union[bool, int, float, str]) -> Expr:
"""Create a literal (constant) value expression.
Parameters
----------
value:
A Python ``bool``, ``int``, ``float``, or ``str``.
Examples
--------
>>> from lancedb.expr import col, lit
>>> col("price") * lit(1.1)
Expr((price * 1.1))
"""
return Expr(expr_lit(value))
def func(name: str, *args: ExprLike) -> Expr:
"""Call an arbitrary SQL function by name.
Parameters
----------
name:
The SQL function name (e.g. ``"lower"``, ``"upper"``).
*args:
The function arguments as :class:`Expr` or plain Python literals.
Examples
--------
>>> from lancedb.expr import col, func
>>> func("lower", col("name"))
Expr(lower(name))
"""
inner_args = [_coerce(a)._inner for a in args]
return Expr(expr_func(name, inner_args))

View File

@@ -2,70 +2,3 @@
# SPDX-FileCopyrightText: Copyright The LanceDB Authors # SPDX-FileCopyrightText: Copyright The LanceDB Authors
"""I/O utilities and interfaces for LanceDB.""" """I/O utilities and interfaces for LanceDB."""
from abc import ABC, abstractmethod
from typing import Dict
class StorageOptionsProvider(ABC):
"""Abstract base class for providing storage options to LanceDB tables.
Storage options providers enable automatic credential refresh for cloud
storage backends (e.g., AWS S3, Azure Blob Storage, GCS). When credentials
have an expiration time, the provider's fetch_storage_options() method will
be called periodically to get fresh credentials before they expire.
Example
-------
>>> class MyProvider(StorageOptionsProvider):
... def fetch_storage_options(self) -> Dict[str, str]:
... # Fetch fresh credentials from your credential manager
... return {
... "aws_access_key_id": "...",
... "aws_secret_access_key": "...",
... "expires_at_millis": "1234567890000" # Optional
... }
"""
@abstractmethod
def fetch_storage_options(self) -> Dict[str, str]:
"""Fetch fresh storage credentials.
This method is called by LanceDB when credentials need to be refreshed.
If the returned dictionary contains an "expires_at_millis" key with a
Unix timestamp in milliseconds, LanceDB will automatically refresh the
credentials before that time. If the key is not present, credentials
are assumed to not expire.
Returns
-------
Dict[str, str]
Dictionary containing cloud storage credentials and optionally an
expiration time:
- "expires_at_millis" (optional): Unix timestamp in milliseconds when
credentials expire
- Provider-specific credential keys (e.g., aws_access_key_id,
aws_secret_access_key, etc.)
Raises
------
RuntimeError
If credentials cannot be fetched or are invalid
"""
pass
def provider_id(self) -> str:
"""Return a human-readable unique identifier for this provider instance.
This identifier is used for caching and equality comparison. Two providers
with the same ID will share the same cached object store connection.
The default implementation uses the class name and string representation.
Override this method if you need custom identification logic.
Returns
-------
str
A unique identifier for this provider instance
"""
return f"{self.__class__.__name__} {{ repr: {str(self)!r} }}"

File diff suppressed because it is too large Load Diff

View File

@@ -38,6 +38,7 @@ from .rerankers.base import Reranker
from .rerankers.rrf import RRFReranker from .rerankers.rrf import RRFReranker
from .rerankers.util import check_reranker_result from .rerankers.util import check_reranker_result
from .util import flatten_columns from .util import flatten_columns
from .expr import Expr
from lancedb._lancedb import fts_query_to_json from lancedb._lancedb import fts_query_to_json
from typing_extensions import Annotated from typing_extensions import Annotated
@@ -70,7 +71,7 @@ def ensure_vector_query(
) -> Union[List[float], List[List[float]], pa.Array, List[pa.Array]]: ) -> Union[List[float], List[List[float]], pa.Array, List[pa.Array]]:
if isinstance(val, list): if isinstance(val, list):
if len(val) == 0: if len(val) == 0:
return ValueError("Vector query must be a non-empty list") raise ValueError("Vector query must be a non-empty list")
sample = val[0] sample = val[0]
else: else:
if isinstance(val, float): if isinstance(val, float):
@@ -83,7 +84,7 @@ def ensure_vector_query(
return val return val
if isinstance(sample, list): if isinstance(sample, list):
if len(sample) == 0: if len(sample) == 0:
return ValueError("Vector query must be a non-empty list") raise ValueError("Vector query must be a non-empty list")
if isinstance(sample[0], float): if isinstance(sample[0], float):
# val is list of list of floats # val is list of list of floats
return val return val
@@ -449,8 +450,8 @@ class Query(pydantic.BaseModel):
ensure_vector_query, ensure_vector_query,
] = None ] = None
# sql filter to refine the query with # sql filter or type-safe Expr to refine the query with
filter: Optional[str] = None filter: Optional[Union[str, Expr]] = None
# if True then apply the filter after vector search # if True then apply the filter after vector search
postfilter: Optional[bool] = None postfilter: Optional[bool] = None
@@ -464,8 +465,8 @@ class Query(pydantic.BaseModel):
# distance type to use for vector search # distance type to use for vector search
distance_type: Optional[str] = None distance_type: Optional[str] = None
# which columns to return in the results # which columns to return in the results (dict values may be str or Expr)
columns: Optional[Union[List[str], Dict[str, str]]] = None columns: Optional[Union[List[str], Dict[str, Union[str, Expr]]]] = None
# minimum number of IVF partitions to search # minimum number of IVF partitions to search
# #
@@ -606,6 +607,7 @@ class LanceQueryBuilder(ABC):
query, query,
ordering_field_name=ordering_field_name, ordering_field_name=ordering_field_name,
fts_columns=fts_columns, fts_columns=fts_columns,
fast_search=fast_search,
) )
if isinstance(query, list): if isinstance(query, list):
@@ -855,14 +857,15 @@ class LanceQueryBuilder(ABC):
self._offset = offset self._offset = offset
return self return self
def select(self, columns: Union[list[str], dict[str, str]]) -> Self: def select(self, columns: Union[list[str], dict[str, Union[str, Expr]]]) -> Self:
"""Set the columns to return. """Set the columns to return.
Parameters Parameters
---------- ----------
columns: list of str, or dict of str to str default None columns: list of str, or dict of str to str or Expr
List of column names to be fetched. List of column names to be fetched.
Or a dictionary of column names to SQL expressions. Or a dictionary of column names to SQL expressions or
:class:`~lancedb.expr.Expr` objects.
All columns are fetched if None or unspecified. All columns are fetched if None or unspecified.
Returns Returns
@@ -876,15 +879,15 @@ class LanceQueryBuilder(ABC):
raise ValueError("columns must be a list or a dictionary") raise ValueError("columns must be a list or a dictionary")
return self return self
def where(self, where: str, prefilter: bool = True) -> Self: def where(self, where: Union[str, Expr], prefilter: bool = True) -> Self:
"""Set the where clause. """Set the where clause.
Parameters Parameters
---------- ----------
where: str where: str or :class:`~lancedb.expr.Expr`
The where clause which is a valid SQL where clause. See The filter condition. Can be a SQL string or a type-safe
`Lance filter pushdown <https://lance.org/guide/read_and_write#filter-push-down>`_ :class:`~lancedb.expr.Expr` built with :func:`~lancedb.expr.col`
for valid SQL expressions. and :func:`~lancedb.expr.lit`.
prefilter: bool, default True prefilter: bool, default True
If True, apply the filter before vector search, otherwise the If True, apply the filter before vector search, otherwise the
filter is applied on the result of vector search. filter is applied on the result of vector search.
@@ -1354,15 +1357,17 @@ class LanceVectorQueryBuilder(LanceQueryBuilder):
return result_set return result_set
def where(self, where: str, prefilter: bool = None) -> LanceVectorQueryBuilder: def where(
self, where: Union[str, Expr], prefilter: bool = None
) -> LanceVectorQueryBuilder:
"""Set the where clause. """Set the where clause.
Parameters Parameters
---------- ----------
where: str where: str or :class:`~lancedb.expr.Expr`
The where clause which is a valid SQL where clause. See The filter condition. Can be a SQL string or a type-safe
`Lance filter pushdown <https://lance.org/guide/read_and_write#filter-push-down>`_ :class:`~lancedb.expr.Expr` built with :func:`~lancedb.expr.col`
for valid SQL expressions. and :func:`~lancedb.expr.lit`.
prefilter: bool, default True prefilter: bool, default True
If True, apply the filter before vector search, otherwise the If True, apply the filter before vector search, otherwise the
filter is applied on the result of vector search. filter is applied on the result of vector search.
@@ -1456,13 +1461,14 @@ class LanceFtsQueryBuilder(LanceQueryBuilder):
query: str | FullTextQuery, query: str | FullTextQuery,
ordering_field_name: Optional[str] = None, ordering_field_name: Optional[str] = None,
fts_columns: Optional[Union[str, List[str]]] = None, fts_columns: Optional[Union[str, List[str]]] = None,
fast_search: bool = None,
): ):
super().__init__(table) super().__init__(table)
self._query = query self._query = query
self._phrase_query = False self._phrase_query = False
self.ordering_field_name = ordering_field_name self.ordering_field_name = ordering_field_name
self._reranker = None self._reranker = None
self._fast_search = None self._fast_search = fast_search
if isinstance(fts_columns, str): if isinstance(fts_columns, str):
fts_columns = [fts_columns] fts_columns = [fts_columns]
self._fts_columns = fts_columns self._fts_columns = fts_columns
@@ -2203,8 +2209,8 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
self._vector_query.select(self._columns) self._vector_query.select(self._columns)
self._fts_query.select(self._columns) self._fts_query.select(self._columns)
if self._where: if self._where:
self._vector_query.where(self._where, self._postfilter) self._vector_query.where(self._where, not self._postfilter)
self._fts_query.where(self._where, self._postfilter) self._fts_query.where(self._where, not self._postfilter)
if self._with_row_id: if self._with_row_id:
self._vector_query.with_row_id(True) self._vector_query.with_row_id(True)
self._fts_query.with_row_id(True) self._fts_query.with_row_id(True)
@@ -2284,10 +2290,20 @@ class AsyncQueryBase(object):
""" """
if isinstance(columns, list) and all(isinstance(c, str) for c in columns): if isinstance(columns, list) and all(isinstance(c, str) for c in columns):
self._inner.select_columns(columns) self._inner.select_columns(columns)
elif isinstance(columns, dict) and all( elif isinstance(columns, dict) and all(isinstance(k, str) for k in columns):
isinstance(k, str) and isinstance(v, str) for k, v in columns.items() if any(isinstance(v, Expr) for v in columns.values()):
): # At least one value is an Expr — use the type-safe path.
self._inner.select(list(columns.items())) from .expr import _coerce
pairs = [(k, _coerce(v)._inner) for k, v in columns.items()]
self._inner.select_expr(pairs)
elif all(isinstance(v, str) for v in columns.values()):
self._inner.select(list(columns.items()))
else:
raise TypeError(
"dict values must be str or Expr, got "
+ str({k: type(v) for k, v in columns.items()})
)
else: else:
raise TypeError("columns must be a list of column names or a dict") raise TypeError("columns must be a list of column names or a dict")
return self return self
@@ -2527,11 +2543,13 @@ class AsyncStandardQuery(AsyncQueryBase):
""" """
super().__init__(inner) super().__init__(inner)
def where(self, predicate: str) -> Self: def where(self, predicate: Union[str, Expr]) -> Self:
""" """
Only return rows matching the given predicate Only return rows matching the given predicate
The predicate should be supplied as an SQL query string. The predicate can be a SQL string or a type-safe
:class:`~lancedb.expr.Expr` built with :func:`~lancedb.expr.col`
and :func:`~lancedb.expr.lit`.
Examples Examples
-------- --------
@@ -2543,7 +2561,10 @@ class AsyncStandardQuery(AsyncQueryBase):
Filtering performance can often be improved by creating a scalar index Filtering performance can often be improved by creating a scalar index
on the filter column(s). on the filter column(s).
""" """
self._inner.where(predicate) if isinstance(predicate, Expr):
self._inner.where_expr(predicate._inner)
else:
self._inner.where(predicate)
return self return self
def limit(self, limit: int) -> Self: def limit(self, limit: int) -> Self:

View File

@@ -111,7 +111,7 @@ class RemoteDBConnection(DBConnection):
@override @override
def list_namespaces( def list_namespaces(
self, self,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> ListNamespacesResponse: ) -> ListNamespacesResponse:
@@ -119,7 +119,7 @@ class RemoteDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The parent namespace to list namespaces in. The parent namespace to list namespaces in.
None or empty list represents root namespace. None or empty list represents root namespace.
page_token: str, optional page_token: str, optional
@@ -133,18 +133,18 @@ class RemoteDBConnection(DBConnection):
ListNamespacesResponse ListNamespacesResponse
Response containing namespace names and optional page_token for pagination. Response containing namespace names and optional page_token for pagination.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
return LOOP.run( return LOOP.run(
self._conn.list_namespaces( self._conn.list_namespaces(
namespace=namespace, page_token=page_token, limit=limit namespace_path=namespace_path, page_token=page_token, limit=limit
) )
) )
@override @override
def create_namespace( def create_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
mode: Optional[str] = None, mode: Optional[str] = None,
properties: Optional[Dict[str, str]] = None, properties: Optional[Dict[str, str]] = None,
) -> CreateNamespaceResponse: ) -> CreateNamespaceResponse:
@@ -152,7 +152,7 @@ class RemoteDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to create. The namespace identifier to create.
mode: str, optional mode: str, optional
Creation mode - "create" (fail if exists), "exist_ok" (skip if exists), Creation mode - "create" (fail if exists), "exist_ok" (skip if exists),
@@ -167,14 +167,14 @@ class RemoteDBConnection(DBConnection):
""" """
return LOOP.run( return LOOP.run(
self._conn.create_namespace( self._conn.create_namespace(
namespace=namespace, mode=mode, properties=properties namespace_path=namespace_path, mode=mode, properties=properties
) )
) )
@override @override
def drop_namespace( def drop_namespace(
self, self,
namespace: List[str], namespace_path: List[str],
mode: Optional[str] = None, mode: Optional[str] = None,
behavior: Optional[str] = None, behavior: Optional[str] = None,
) -> DropNamespaceResponse: ) -> DropNamespaceResponse:
@@ -182,7 +182,7 @@ class RemoteDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to drop. The namespace identifier to drop.
mode: str, optional mode: str, optional
Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive. Whether to skip if not exists ("SKIP") or fail ("FAIL"). Case insensitive.
@@ -196,16 +196,20 @@ class RemoteDBConnection(DBConnection):
Response containing properties and transaction_id if applicable. Response containing properties and transaction_id if applicable.
""" """
return LOOP.run( return LOOP.run(
self._conn.drop_namespace(namespace=namespace, mode=mode, behavior=behavior) self._conn.drop_namespace(
namespace_path=namespace_path, mode=mode, behavior=behavior
)
) )
@override @override
def describe_namespace(self, namespace: List[str]) -> DescribeNamespaceResponse: def describe_namespace(
self, namespace_path: List[str]
) -> DescribeNamespaceResponse:
"""Describe a namespace. """Describe a namespace.
Parameters Parameters
---------- ----------
namespace: List[str] namespace_path: List[str]
The namespace identifier to describe. The namespace identifier to describe.
Returns Returns
@@ -213,12 +217,12 @@ class RemoteDBConnection(DBConnection):
DescribeNamespaceResponse DescribeNamespaceResponse
Response containing the namespace properties. Response containing the namespace properties.
""" """
return LOOP.run(self._conn.describe_namespace(namespace=namespace)) return LOOP.run(self._conn.describe_namespace(namespace_path=namespace_path))
@override @override
def list_tables( def list_tables(
self, self,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: Optional[int] = None, limit: Optional[int] = None,
) -> ListTablesResponse: ) -> ListTablesResponse:
@@ -226,7 +230,7 @@ class RemoteDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str], optional namespace_path: List[str], optional
The namespace to list tables in. The namespace to list tables in.
None or empty list represents root namespace. None or empty list represents root namespace.
page_token: str, optional page_token: str, optional
@@ -240,11 +244,11 @@ class RemoteDBConnection(DBConnection):
ListTablesResponse ListTablesResponse
Response containing table names and optional page_token for pagination. Response containing table names and optional page_token for pagination.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
return LOOP.run( return LOOP.run(
self._conn.list_tables( self._conn.list_tables(
namespace=namespace, page_token=page_token, limit=limit namespace_path=namespace_path, page_token=page_token, limit=limit
) )
) )
@@ -254,7 +258,7 @@ class RemoteDBConnection(DBConnection):
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: int = 10, limit: int = 10,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
) -> Iterable[str]: ) -> Iterable[str]:
"""List the names of all tables in the database. """List the names of all tables in the database.
@@ -263,7 +267,7 @@ class RemoteDBConnection(DBConnection):
Parameters Parameters
---------- ----------
namespace: List[str], default [] namespace_path: List[str], default []
The namespace to list tables in. The namespace to list tables in.
Empty list represents root namespace. Empty list represents root namespace.
page_token: str page_token: str
@@ -282,11 +286,11 @@ class RemoteDBConnection(DBConnection):
DeprecationWarning, DeprecationWarning,
stacklevel=2, stacklevel=2,
) )
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
return LOOP.run( return LOOP.run(
self._conn.table_names( self._conn.table_names(
namespace=namespace, start_after=page_token, limit=limit namespace_path=namespace_path, start_after=page_token, limit=limit
) )
) )
@@ -295,7 +299,7 @@ class RemoteDBConnection(DBConnection):
self, self,
name: str, name: str,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
index_cache_size: Optional[int] = None, index_cache_size: Optional[int] = None,
) -> Table: ) -> Table:
@@ -305,7 +309,7 @@ class RemoteDBConnection(DBConnection):
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], optional namespace_path: List[str], optional
The namespace to open the table from. The namespace to open the table from.
None or empty list represents root namespace. None or empty list represents root namespace.
@@ -315,15 +319,15 @@ class RemoteDBConnection(DBConnection):
""" """
from .table import RemoteTable from .table import RemoteTable
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
if index_cache_size is not None: if index_cache_size is not None:
logging.info( logging.info(
"index_cache_size is ignored in LanceDb Cloud" "index_cache_size is ignored in LanceDb Cloud"
" (there is no local cache to configure)" " (there is no local cache to configure)"
) )
table = LOOP.run(self._conn.open_table(name, namespace=namespace)) table = LOOP.run(self._conn.open_table(name, namespace_path=namespace_path))
return RemoteTable(table, self.db_name) return RemoteTable(table, self.db_name)
def clone_table( def clone_table(
@@ -331,7 +335,7 @@ class RemoteDBConnection(DBConnection):
target_table_name: str, target_table_name: str,
source_uri: str, source_uri: str,
*, *,
target_namespace: Optional[List[str]] = None, target_namespace_path: Optional[List[str]] = None,
source_version: Optional[int] = None, source_version: Optional[int] = None,
source_tag: Optional[str] = None, source_tag: Optional[str] = None,
is_shallow: bool = True, is_shallow: bool = True,
@@ -344,7 +348,7 @@ class RemoteDBConnection(DBConnection):
The name of the target table to create. The name of the target table to create.
source_uri: str source_uri: str
The URI of the source table to clone from. The URI of the source table to clone from.
target_namespace: List[str], optional target_namespace_path: List[str], optional
The namespace for the target table. The namespace for the target table.
None or empty list represents root namespace. None or empty list represents root namespace.
source_version: int, optional source_version: int, optional
@@ -361,13 +365,13 @@ class RemoteDBConnection(DBConnection):
""" """
from .table import RemoteTable from .table import RemoteTable
if target_namespace is None: if target_namespace_path is None:
target_namespace = [] target_namespace_path = []
table = LOOP.run( table = LOOP.run(
self._conn.clone_table( self._conn.clone_table(
target_table_name, target_table_name,
source_uri, source_uri,
target_namespace=target_namespace, target_namespace_path=target_namespace_path,
source_version=source_version, source_version=source_version,
source_tag=source_tag, source_tag=source_tag,
is_shallow=is_shallow, is_shallow=is_shallow,
@@ -387,7 +391,7 @@ class RemoteDBConnection(DBConnection):
exist_ok: bool = False, exist_ok: bool = False,
embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None, embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
) -> Table: ) -> Table:
"""Create a [Table][lancedb.table.Table] in the database. """Create a [Table][lancedb.table.Table] in the database.
@@ -395,7 +399,7 @@ class RemoteDBConnection(DBConnection):
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], optional namespace_path: List[str], optional
The namespace to create the table in. The namespace to create the table in.
None or empty list represents root namespace. None or empty list represents root namespace.
data: The data to initialize the table, *optional* data: The data to initialize the table, *optional*
@@ -495,8 +499,8 @@ class RemoteDBConnection(DBConnection):
mode = "exist_ok" mode = "exist_ok"
elif not mode: elif not mode:
mode = "exist_ok" mode = "exist_ok"
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
validate_table_name(name) validate_table_name(name)
if embedding_functions is not None: if embedding_functions is not None:
logging.warning( logging.warning(
@@ -511,7 +515,7 @@ class RemoteDBConnection(DBConnection):
self._conn.create_table( self._conn.create_table(
name, name,
data, data,
namespace=namespace, namespace_path=namespace_path,
mode=mode, mode=mode,
schema=schema, schema=schema,
on_bad_vectors=on_bad_vectors, on_bad_vectors=on_bad_vectors,
@@ -521,28 +525,28 @@ class RemoteDBConnection(DBConnection):
return RemoteTable(table, self.db_name) return RemoteTable(table, self.db_name)
@override @override
def drop_table(self, name: str, namespace: Optional[List[str]] = None): def drop_table(self, name: str, namespace_path: Optional[List[str]] = None):
"""Drop a table from the database. """Drop a table from the database.
Parameters Parameters
---------- ----------
name: str name: str
The name of the table. The name of the table.
namespace: List[str], optional namespace_path: List[str], optional
The namespace to drop the table from. The namespace to drop the table from.
None or empty list represents root namespace. None or empty list represents root namespace.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
LOOP.run(self._conn.drop_table(name, namespace=namespace)) LOOP.run(self._conn.drop_table(name, namespace_path=namespace_path))
@override @override
def rename_table( def rename_table(
self, self,
cur_name: str, cur_name: str,
new_name: str, new_name: str,
cur_namespace: Optional[List[str]] = None, cur_namespace_path: Optional[List[str]] = None,
new_namespace: Optional[List[str]] = None, new_namespace_path: Optional[List[str]] = None,
): ):
"""Rename a table in the database. """Rename a table in the database.
@@ -553,19 +557,19 @@ class RemoteDBConnection(DBConnection):
new_name: str new_name: str
The new name of the table. The new name of the table.
""" """
if cur_namespace is None: if cur_namespace_path is None:
cur_namespace = [] cur_namespace_path = []
if new_namespace is None: if new_namespace_path is None:
new_namespace = [] new_namespace_path = []
LOOP.run( LOOP.run(
self._conn.rename_table( self._conn.rename_table(
cur_name, cur_name,
new_name, new_name,
cur_namespace=cur_namespace, cur_namespace_path=cur_namespace_path,
new_namespace=new_namespace, new_namespace_path=new_namespace_path,
) )
) )
async def close(self): async def close(self):
"""Close the connection to the database.""" """Close the connection to the database."""
self._client.close() self._conn.close()

View File

@@ -4,7 +4,7 @@
from datetime import timedelta from datetime import timedelta
import logging import logging
from functools import cached_property from functools import cached_property
from typing import Dict, Iterable, List, Optional, Union, Literal from typing import Any, Callable, Dict, Iterable, List, Optional, Union, Literal
import warnings import warnings
from lancedb._lancedb import ( from lancedb._lancedb import (
@@ -35,6 +35,7 @@ import pyarrow as pa
from lancedb.common import DATA, VEC, VECTOR_COLUMN_NAME from lancedb.common import DATA, VEC, VECTOR_COLUMN_NAME
from lancedb.merge import LanceMergeInsertBuilder from lancedb.merge import LanceMergeInsertBuilder
from lancedb.embeddings import EmbeddingFunctionRegistry from lancedb.embeddings import EmbeddingFunctionRegistry
from lancedb.table import _normalize_progress
from ..query import LanceVectorQueryBuilder, LanceQueryBuilder, LanceTakeQueryBuilder from ..query import LanceVectorQueryBuilder, LanceQueryBuilder, LanceTakeQueryBuilder
from ..table import AsyncTable, IndexStatistics, Query, Table, Tags from ..table import AsyncTable, IndexStatistics, Query, Table, Tags
@@ -218,8 +219,6 @@ class RemoteTable(Table):
train: bool = True, train: bool = True,
): ):
"""Create an index on the table. """Create an index on the table.
Currently, the only parameters that matter are
the metric and the vector column name.
Parameters Parameters
---------- ----------
@@ -250,11 +249,6 @@ class RemoteTable(Table):
>>> table.create_index("l2", "vector") # doctest: +SKIP >>> table.create_index("l2", "vector") # doctest: +SKIP
""" """
if num_sub_vectors is not None:
logging.warning(
"num_sub_vectors is not supported on LanceDB cloud."
"This parameter will be tuned automatically."
)
if accelerator is not None: if accelerator is not None:
logging.warning( logging.warning(
"GPU accelerator is not yet supported on LanceDB cloud." "GPU accelerator is not yet supported on LanceDB cloud."
@@ -315,6 +309,7 @@ class RemoteTable(Table):
mode: str = "append", mode: str = "append",
on_bad_vectors: str = "error", on_bad_vectors: str = "error",
fill_value: float = 0.0, fill_value: float = 0.0,
progress: Optional[Union[bool, Callable, Any]] = None,
) -> AddResult: ) -> AddResult:
"""Add more data to the [Table](Table). It has the same API signature as """Add more data to the [Table](Table). It has the same API signature as
the OSS version. the OSS version.
@@ -337,17 +332,29 @@ class RemoteTable(Table):
One of "error", "drop", "fill". One of "error", "drop", "fill".
fill_value: float, default 0. fill_value: float, default 0.
The value to use when filling vectors. Only used if on_bad_vectors="fill". The value to use when filling vectors. Only used if on_bad_vectors="fill".
progress: bool, callable, or tqdm-like, optional
A callback or tqdm-compatible progress bar. See
:meth:`Table.add` for details.
Returns Returns
------- -------
AddResult AddResult
An object containing the new version number of the table after adding data. An object containing the new version number of the table after adding data.
""" """
return LOOP.run( progress, owns = _normalize_progress(progress)
self._table.add( try:
data, mode=mode, on_bad_vectors=on_bad_vectors, fill_value=fill_value return LOOP.run(
self._table.add(
data,
mode=mode,
on_bad_vectors=on_bad_vectors,
fill_value=fill_value,
progress=progress,
)
) )
) finally:
if owns:
progress.close()
def search( def search(
self, self,
@@ -647,6 +654,45 @@ class RemoteTable(Table):
def drop_index(self, index_name: str): def drop_index(self, index_name: str):
return LOOP.run(self._table.drop_index(index_name)) return LOOP.run(self._table.drop_index(index_name))
def prewarm_index(self, name: str) -> None:
"""Prewarm an index in the table.
This is a hint to the database that the index will be accessed in the
future and should be loaded into memory if possible. This can reduce
cold-start latency for subsequent queries.
This call initiates prewarming and returns once the request is accepted.
It is idempotent and safe to call from multiple clients concurrently.
Parameters
----------
name: str
The name of the index to prewarm
"""
return LOOP.run(self._table.prewarm_index(name))
def prewarm_data(self, columns: Optional[List[str]] = None) -> None:
"""Prewarm data for the table.
This is a hint to the database that the given columns will be accessed
in the future and the database should prefetch the data if possible.
Currently only supported on remote tables.
This call initiates prewarming and returns once the request is accepted.
It is idempotent and safe to call from multiple clients concurrently.
This operation has a large upfront cost but can speed up future queries
that need to fetch the given columns. Large columns such as embeddings
or binary data may not be practical to prewarm. This feature is intended
for workloads that issue many queries against the same columns.
Parameters
----------
columns: list of str, optional
The columns to prewarm. If None, all columns are prewarmed.
"""
return LOOP.run(self._table.prewarm_data(columns))
def wait_for_index( def wait_for_index(
self, index_names: Iterable[str], timeout: timedelta = timedelta(seconds=300) self, index_names: Iterable[str], timeout: timedelta = timedelta(seconds=300)
): ):

View File

@@ -14,6 +14,7 @@ from functools import cached_property
from typing import ( from typing import (
TYPE_CHECKING, TYPE_CHECKING,
Any, Any,
Callable,
Dict, Dict,
Iterable, Iterable,
List, List,
@@ -88,7 +89,6 @@ from .index import lang_mapping
if TYPE_CHECKING: if TYPE_CHECKING:
from .db import LanceDBConnection from .db import LanceDBConnection
from .io import StorageOptionsProvider
from ._lancedb import ( from ._lancedb import (
Table as LanceDBTable, Table as LanceDBTable,
OptimizeStats, OptimizeStats,
@@ -277,7 +277,7 @@ def _sanitize_data(
if metadata: if metadata:
new_metadata = target_schema.metadata or {} new_metadata = target_schema.metadata or {}
new_metadata = new_metadata.update(metadata) new_metadata.update(metadata)
target_schema = target_schema.with_metadata(new_metadata) target_schema = target_schema.with_metadata(new_metadata)
_validate_schema(target_schema) _validate_schema(target_schema)
@@ -556,6 +556,21 @@ def _table_uri(base: str, table_name: str) -> str:
return join_uri(base, f"{table_name}.lance") return join_uri(base, f"{table_name}.lance")
def _normalize_progress(progress):
"""Normalize a ``progress`` parameter for :meth:`Table.add`.
Returns ``(progress_obj, owns)`` where *owns* is True when we created a
tqdm bar that the caller must close.
"""
if progress is True:
from tqdm.auto import tqdm
return tqdm(unit=" rows"), True
if progress is False or progress is None:
return None, False
return progress, False
class Table(ABC): class Table(ABC):
""" """
A Table is a collection of Records in a LanceDB Database. A Table is a collection of Records in a LanceDB Database.
@@ -974,6 +989,7 @@ class Table(ABC):
mode: AddMode = "append", mode: AddMode = "append",
on_bad_vectors: OnBadVectorsType = "error", on_bad_vectors: OnBadVectorsType = "error",
fill_value: float = 0.0, fill_value: float = 0.0,
progress: Optional[Union[bool, Callable, Any]] = None,
) -> AddResult: ) -> AddResult:
"""Add more data to the [Table](Table). """Add more data to the [Table](Table).
@@ -995,6 +1011,29 @@ class Table(ABC):
One of "error", "drop", "fill". One of "error", "drop", "fill".
fill_value: float, default 0. fill_value: float, default 0.
The value to use when filling vectors. Only used if on_bad_vectors="fill". The value to use when filling vectors. Only used if on_bad_vectors="fill".
progress: bool, callable, or tqdm-like, optional
Progress reporting during the add operation. Can be:
- ``True`` to automatically create and display a tqdm progress
bar (requires ``tqdm`` to be installed)::
table.add(data, progress=True)
- A **callable** that receives a dict with keys ``output_rows``,
``output_bytes``, ``total_rows``, ``elapsed_seconds``,
``active_tasks``, ``total_tasks``, and ``done``::
def on_progress(p):
print(f"{p['output_rows']}/{p['total_rows']} rows, "
f"{p['active_tasks']}/{p['total_tasks']} workers")
table.add(data, progress=on_progress)
- A **tqdm-compatible** progress bar whose ``total`` and
``update()`` will be called automatically. The postfix shows
write throughput (MB/s) and active worker count::
with tqdm() as pbar:
table.add(data, progress=pbar)
Returns Returns
------- -------
@@ -1331,7 +1370,7 @@ class Table(ABC):
1 2 [3.0, 4.0] 1 2 [3.0, 4.0]
2 3 [5.0, 6.0] 2 3 [5.0, 6.0]
>>> table.delete("x = 2") >>> table.delete("x = 2")
DeleteResult(version=2) DeleteResult(num_deleted_rows=1, version=2)
>>> table.to_pandas() >>> table.to_pandas()
x vector x vector
0 1 [1.0, 2.0] 0 1 [1.0, 2.0]
@@ -1345,7 +1384,7 @@ class Table(ABC):
>>> to_remove >>> to_remove
'1, 5' '1, 5'
>>> table.delete(f"x IN ({to_remove})") >>> table.delete(f"x IN ({to_remove})")
DeleteResult(version=3) DeleteResult(num_deleted_rows=1, version=3)
>>> table.to_pandas() >>> table.to_pandas()
x vector x vector
0 3 [5.0, 6.0] 0 3 [5.0, 6.0]
@@ -1506,22 +1545,17 @@ class Table(ABC):
in-progress operation (e.g. appending new data) and these files will not in-progress operation (e.g. appending new data) and these files will not
be deleted unless they are at least 7 days old. If delete_unverified is True be deleted unless they are at least 7 days old. If delete_unverified is True
then these files will be deleted regardless of their age. then these files will be deleted regardless of their age.
.. warning::
This should only be set to True if you can guarantee that no other
process is currently working on this dataset. Otherwise the dataset
could be put into a corrupted state.
retrain: bool, default False retrain: bool, default False
This parameter is no longer used and is deprecated. This parameter is no longer used and is deprecated.
Experimental API The frequency an application should call optimize is based on the frequency of
----------------
The optimization process is undergoing active development and may change.
Our goal with these changes is to improve the performance of optimization and
reduce the complexity.
That being said, it is essential today to run optimize if you want the best
performance. It should be stable and safe to use in production, but it our
hope that the API may be simplified (or not even need to be called) in the
future.
The frequency an application shoudl call optimize is based on the frequency of
data modifications. If data is frequently added, deleted, or updated then data modifications. If data is frequently added, deleted, or updated then
optimize should be run frequently. A good rule of thumb is to run optimize if optimize should be run frequently. A good rule of thumb is to run optimize if
you have added or modified 100,000 or more records or run more than 20 data you have added or modified 100,000 or more records or run more than 20 data
@@ -1741,29 +1775,34 @@ class LanceTable(Table):
connection: "LanceDBConnection", connection: "LanceDBConnection",
name: str, name: str,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional["StorageOptionsProvider"] = None,
index_cache_size: Optional[int] = None, index_cache_size: Optional[int] = None,
location: Optional[str] = None, location: Optional[str] = None,
namespace_client: Optional[Any] = None,
managed_versioning: Optional[bool] = None,
pushdown_operations: Optional[set] = None,
_async: AsyncTable = None, _async: AsyncTable = None,
): ):
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
self._conn = connection self._conn = connection
self._namespace = namespace self._namespace_path = namespace_path
self._location = location # Store location for use in _dataset_path self._location = location # Store location for use in _dataset_path
self._namespace_client = namespace_client
self._pushdown_operations = pushdown_operations or set()
if _async is not None: if _async is not None:
self._table = _async self._table = _async
else: else:
self._table = LOOP.run( self._table = LOOP.run(
connection._conn.open_table( connection._conn.open_table(
name, name,
namespace=namespace, namespace_path=namespace_path,
storage_options=storage_options, storage_options=storage_options,
storage_options_provider=storage_options_provider,
index_cache_size=index_cache_size, index_cache_size=index_cache_size,
location=location, location=location,
namespace_client=namespace_client,
managed_versioning=managed_versioning,
) )
) )
@@ -1774,13 +1813,13 @@ class LanceTable(Table):
@property @property
def namespace(self) -> List[str]: def namespace(self) -> List[str]:
"""Return the namespace path of the table.""" """Return the namespace path of the table."""
return self._namespace return self._namespace_path
@property @property
def id(self) -> str: def id(self) -> str:
"""Return the full identifier of the table (namespace$name).""" """Return the full identifier of the table (namespace$name)."""
if self._namespace: if self._namespace_path:
return "$".join(self._namespace + [self.name]) return "$".join(self._namespace_path + [self.name])
return self.name return self.name
@classmethod @classmethod
@@ -1801,22 +1840,26 @@ class LanceTable(Table):
db, db,
name, name,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None, storage_options: Optional[Dict[str, str]] = None,
storage_options_provider: Optional["StorageOptionsProvider"] = None,
index_cache_size: Optional[int] = None, index_cache_size: Optional[int] = None,
location: Optional[str] = None, location: Optional[str] = None,
namespace_client: Optional[Any] = None,
managed_versioning: Optional[bool] = None,
pushdown_operations: Optional[set] = None,
): ):
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
tbl = cls( tbl = cls(
db, db,
name, name,
namespace=namespace, namespace_path=namespace_path,
storage_options=storage_options, storage_options=storage_options,
storage_options_provider=storage_options_provider,
index_cache_size=index_cache_size, index_cache_size=index_cache_size,
location=location, location=location,
namespace_client=namespace_client,
managed_versioning=managed_versioning,
pushdown_operations=pushdown_operations,
) )
# check the dataset exists # check the dataset exists
@@ -1848,6 +1891,16 @@ class LanceTable(Table):
"Please install with `pip install pylance`." "Please install with `pip install pylance`."
) )
if self._namespace_client is not None:
table_id = self._namespace_path + [self.name]
return lance.dataset(
version=self.version,
storage_options=self._conn.storage_options,
namespace_client=self._namespace_client,
table_id=table_id,
**kwargs,
)
return lance.dataset( return lance.dataset(
self._dataset_path, self._dataset_path,
version=self.version, version=self.version,
@@ -2200,12 +2253,18 @@ class LanceTable(Table):
def prewarm_index(self, name: str) -> None: def prewarm_index(self, name: str) -> None:
""" """
Prewarms an index in the table Prewarm an index in the table.
This loads the entire index into memory This is a hint to the database that the index will be accessed in the
future and should be loaded into memory if possible. This can reduce
cold-start latency for subsequent queries.
If the index does not fit into the available cache this call This call initiates prewarming and returns once the request is accepted.
may be wasteful It is idempotent and safe to call from multiple clients concurrently.
It is generally wasteful to call this if the index does not fit into the
available cache. Not all index types support prewarming; unsupported
indices will silently ignore the request.
Parameters Parameters
---------- ----------
@@ -2214,6 +2273,29 @@ class LanceTable(Table):
""" """
return LOOP.run(self._table.prewarm_index(name)) return LOOP.run(self._table.prewarm_index(name))
def prewarm_data(self, columns: Optional[List[str]] = None) -> None:
"""
Prewarm data for the table.
This is a hint to the database that the given columns will be accessed
in the future and the database should prefetch the data if possible.
Currently only supported on remote tables.
This call initiates prewarming and returns once the request is accepted.
It is idempotent and safe to call from multiple clients concurrently.
This operation has a large upfront cost but can speed up future queries
that need to fetch the given columns. Large columns such as embeddings
or binary data may not be practical to prewarm. This feature is intended
for workloads that issue many queries against the same columns.
Parameters
----------
columns: list of str, optional
The columns to prewarm. If None, all columns are prewarmed.
"""
return LOOP.run(self._table.prewarm_data(columns))
def wait_for_index( def wait_for_index(
self, index_names: Iterable[str], timeout: timedelta = timedelta(seconds=300) self, index_names: Iterable[str], timeout: timedelta = timedelta(seconds=300)
) -> None: ) -> None:
@@ -2449,6 +2531,7 @@ class LanceTable(Table):
mode: AddMode = "append", mode: AddMode = "append",
on_bad_vectors: OnBadVectorsType = "error", on_bad_vectors: OnBadVectorsType = "error",
fill_value: float = 0.0, fill_value: float = 0.0,
progress: Optional[Union[bool, Callable, Any]] = None,
) -> AddResult: ) -> AddResult:
"""Add data to the table. """Add data to the table.
If vector columns are missing and the table If vector columns are missing and the table
@@ -2467,17 +2550,29 @@ class LanceTable(Table):
One of "error", "drop", "fill", "null". One of "error", "drop", "fill", "null".
fill_value: float, default 0. fill_value: float, default 0.
The value to use when filling vectors. Only used if on_bad_vectors="fill". The value to use when filling vectors. Only used if on_bad_vectors="fill".
progress: bool, callable, or tqdm-like, optional
A callback or tqdm-compatible progress bar. See
:meth:`Table.add` for details.
Returns Returns
------- -------
int int
The number of vectors in the table. The number of vectors in the table.
""" """
return LOOP.run( progress, owns = _normalize_progress(progress)
self._table.add( try:
data, mode=mode, on_bad_vectors=on_bad_vectors, fill_value=fill_value return LOOP.run(
self._table.add(
data,
mode=mode,
on_bad_vectors=on_bad_vectors,
fill_value=fill_value,
progress=progress,
)
) )
) finally:
if owns:
progress.close()
def merge( def merge(
self, self,
@@ -2707,12 +2802,13 @@ class LanceTable(Table):
fill_value: float = 0.0, fill_value: float = 0.0,
embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None, embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None,
*, *,
namespace: Optional[List[str]] = None, namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str | bool]] = None, storage_options: Optional[Dict[str, str | bool]] = None,
storage_options_provider: Optional["StorageOptionsProvider"] = None,
data_storage_version: Optional[str] = None, data_storage_version: Optional[str] = None,
enable_v2_manifest_paths: Optional[bool] = None, enable_v2_manifest_paths: Optional[bool] = None,
location: Optional[str] = None, location: Optional[str] = None,
namespace_client: Optional[Any] = None,
pushdown_operations: Optional[set] = None,
): ):
""" """
Create a new table. Create a new table.
@@ -2767,12 +2863,14 @@ class LanceTable(Table):
Deprecated. Set `storage_options` when connecting to the database and set Deprecated. Set `storage_options` when connecting to the database and set
`new_table_enable_v2_manifest_paths` in the options. `new_table_enable_v2_manifest_paths` in the options.
""" """
if namespace is None: if namespace_path is None:
namespace = [] namespace_path = []
self = cls.__new__(cls) self = cls.__new__(cls)
self._conn = db self._conn = db
self._namespace = namespace self._namespace_path = namespace_path
self._location = location self._location = location
self._namespace_client = namespace_client
self._pushdown_operations = pushdown_operations or set()
if data_storage_version is not None: if data_storage_version is not None:
warnings.warn( warnings.warn(
@@ -2805,9 +2903,8 @@ class LanceTable(Table):
on_bad_vectors=on_bad_vectors, on_bad_vectors=on_bad_vectors,
fill_value=fill_value, fill_value=fill_value,
embedding_functions=embedding_functions, embedding_functions=embedding_functions,
namespace=namespace, namespace_path=namespace_path,
storage_options=storage_options, storage_options=storage_options,
storage_options_provider=storage_options_provider,
location=location, location=location,
) )
) )
@@ -2876,6 +2973,15 @@ class LanceTable(Table):
batch_size: Optional[int] = None, batch_size: Optional[int] = None,
timeout: Optional[timedelta] = None, timeout: Optional[timedelta] = None,
) -> pa.RecordBatchReader: ) -> pa.RecordBatchReader:
if (
"QueryTable" in self._pushdown_operations
and self._namespace_client is not None
):
from lancedb.namespace import _execute_server_side_query
table_id = self._namespace_path + [self.name]
return _execute_server_side_query(self._namespace_client, table_id, query)
async_iter = LOOP.run( async_iter = LOOP.run(
self._table._execute_query(query, batch_size=batch_size, timeout=timeout) self._table._execute_query(query, batch_size=batch_size, timeout=timeout)
) )
@@ -2997,22 +3103,17 @@ class LanceTable(Table):
in-progress operation (e.g. appending new data) and these files will not in-progress operation (e.g. appending new data) and these files will not
be deleted unless they are at least 7 days old. If delete_unverified is True be deleted unless they are at least 7 days old. If delete_unverified is True
then these files will be deleted regardless of their age. then these files will be deleted regardless of their age.
.. warning::
This should only be set to True if you can guarantee that no other
process is currently working on this dataset. Otherwise the dataset
could be put into a corrupted state.
retrain: bool, default False retrain: bool, default False
This parameter is no longer used and is deprecated. This parameter is no longer used and is deprecated.
Experimental API The frequency an application should call optimize is based on the frequency of
----------------
The optimization process is undergoing active development and may change.
Our goal with these changes is to improve the performance of optimization and
reduce the complexity.
That being said, it is essential today to run optimize if you want the best
performance. It should be stable and safe to use in production, but it our
hope that the API may be simplified (or not even need to be called) in the
future.
The frequency an application shoudl call optimize is based on the frequency of
data modifications. If data is frequently added, deleted, or updated then data modifications. If data is frequently added, deleted, or updated then
optimize should be run frequently. A good rule of thumb is to run optimize if optimize should be run frequently. A good rule of thumb is to run optimize if
you have added or modified 100,000 or more records or run more than 20 data you have added or modified 100,000 or more records or run more than 20 data
@@ -3613,19 +3714,47 @@ class AsyncTable:
""" """
Prewarm an index in the table. Prewarm an index in the table.
This is a hint to the database that the index will be accessed in the
future and should be loaded into memory if possible. This can reduce
cold-start latency for subsequent queries.
This call initiates prewarming and returns once the request is accepted.
It is idempotent and safe to call from multiple clients concurrently.
It is generally wasteful to call this if the index does not fit into the
available cache. Not all index types support prewarming; unsupported
indices will silently ignore the request.
Parameters Parameters
---------- ----------
name: str name: str
The name of the index to prewarm The name of the index to prewarm
Notes
-----
This will load the index into memory. This may reduce the cold-start time for
future queries. If the index does not fit in the cache then this call may be
wasteful.
""" """
await self._inner.prewarm_index(name) await self._inner.prewarm_index(name)
async def prewarm_data(self, columns: Optional[List[str]] = None) -> None:
"""
Prewarm data for the table.
This is a hint to the database that the given columns will be accessed
in the future and the database should prefetch the data if possible.
Currently only supported on remote tables.
This call initiates prewarming and returns once the request is accepted.
It is idempotent and safe to call from multiple clients concurrently.
This operation has a large upfront cost but can speed up future queries
that need to fetch the given columns. Large columns such as embeddings
or binary data may not be practical to prewarm. This feature is intended
for workloads that issue many queries against the same columns.
Parameters
----------
columns: list of str, optional
The columns to prewarm. If None, all columns are prewarmed.
"""
await self._inner.prewarm_data(columns)
async def wait_for_index( async def wait_for_index(
self, index_names: Iterable[str], timeout: timedelta = timedelta(seconds=300) self, index_names: Iterable[str], timeout: timedelta = timedelta(seconds=300)
) -> None: ) -> None:
@@ -3701,6 +3830,7 @@ class AsyncTable:
mode: Optional[Literal["append", "overwrite"]] = "append", mode: Optional[Literal["append", "overwrite"]] = "append",
on_bad_vectors: Optional[OnBadVectorsType] = None, on_bad_vectors: Optional[OnBadVectorsType] = None,
fill_value: Optional[float] = None, fill_value: Optional[float] = None,
progress: Optional[Union[bool, Callable, Any]] = None,
) -> AddResult: ) -> AddResult:
"""Add more data to the [Table](Table). """Add more data to the [Table](Table).
@@ -3722,6 +3852,9 @@ class AsyncTable:
One of "error", "drop", "fill", "null". One of "error", "drop", "fill", "null".
fill_value: float, default 0. fill_value: float, default 0.
The value to use when filling vectors. Only used if on_bad_vectors="fill". The value to use when filling vectors. Only used if on_bad_vectors="fill".
progress: callable or tqdm-like, optional
A callback or tqdm-compatible progress bar. See
:meth:`Table.add` for details.
""" """
schema = await self.schema() schema = await self.schema()
@@ -3732,7 +3865,13 @@ class AsyncTable:
# _santitize_data is an old code path, but we will use it until the # _santitize_data is an old code path, but we will use it until the
# new code path is ready. # new code path is ready.
if on_bad_vectors != "error" or ( if mode == "overwrite":
# For overwrite, apply the same preprocessing as create_table
# so vector columns are inferred as FixedSizeList.
data, _ = sanitize_create_table(
data, None, on_bad_vectors=on_bad_vectors, fill_value=fill_value
)
elif on_bad_vectors != "error" or (
schema.metadata is not None and b"embedding_functions" in schema.metadata schema.metadata is not None and b"embedding_functions" in schema.metadata
): ):
data = _sanitize_data( data = _sanitize_data(
@@ -3745,8 +3884,9 @@ class AsyncTable:
) )
_register_optional_converters() _register_optional_converters()
data = to_scannable(data) data = to_scannable(data)
progress, owns = _normalize_progress(progress)
try: try:
return await self._inner.add(data, mode or "append") return await self._inner.add(data, mode or "append", progress=progress)
except RuntimeError as e: except RuntimeError as e:
if "Cast error" in str(e): if "Cast error" in str(e):
raise ValueError(e) raise ValueError(e)
@@ -3754,6 +3894,9 @@ class AsyncTable:
raise ValueError(e) raise ValueError(e)
else: else:
raise raise
finally:
if owns:
progress.close()
def merge_insert(self, on: Union[str, Iterable[str]]) -> LanceMergeInsertBuilder: def merge_insert(self, on: Union[str, Iterable[str]]) -> LanceMergeInsertBuilder:
""" """
@@ -4076,7 +4219,7 @@ class AsyncTable:
async_query = async_query.offset(query.offset) async_query = async_query.offset(query.offset)
if query.columns: if query.columns:
async_query = async_query.select(query.columns) async_query = async_query.select(query.columns)
if query.filter: if query.filter is not None:
async_query = async_query.where(query.filter) async_query = async_query.where(query.filter)
if query.fast_search: if query.fast_search:
async_query = async_query.fast_search() async_query = async_query.fast_search()
@@ -4215,7 +4358,7 @@ class AsyncTable:
1 2 [3.0, 4.0] 1 2 [3.0, 4.0]
2 3 [5.0, 6.0] 2 3 [5.0, 6.0]
>>> table.delete("x = 2") >>> table.delete("x = 2")
DeleteResult(version=2) DeleteResult(num_deleted_rows=1, version=2)
>>> table.to_pandas() >>> table.to_pandas()
x vector x vector
0 1 [1.0, 2.0] 0 1 [1.0, 2.0]
@@ -4229,7 +4372,7 @@ class AsyncTable:
>>> to_remove >>> to_remove
'1, 5' '1, 5'
>>> table.delete(f"x IN ({to_remove})") >>> table.delete(f"x IN ({to_remove})")
DeleteResult(version=3) DeleteResult(num_deleted_rows=1, version=3)
>>> table.to_pandas() >>> table.to_pandas()
x vector x vector
0 3 [5.0, 6.0] 0 3 [5.0, 6.0]
@@ -4552,22 +4695,17 @@ class AsyncTable:
in-progress operation (e.g. appending new data) and these files will not in-progress operation (e.g. appending new data) and these files will not
be deleted unless they are at least 7 days old. If delete_unverified is True be deleted unless they are at least 7 days old. If delete_unverified is True
then these files will be deleted regardless of their age. then these files will be deleted regardless of their age.
.. warning::
This should only be set to True if you can guarantee that no other
process is currently working on this dataset. Otherwise the dataset
could be put into a corrupted state.
retrain: bool, default False retrain: bool, default False
This parameter is no longer used and is deprecated. This parameter is no longer used and is deprecated.
Experimental API The frequency an application should call optimize is based on the frequency of
----------------
The optimization process is undergoing active development and may change.
Our goal with these changes is to improve the performance of optimization and
reduce the complexity.
That being said, it is essential today to run optimize if you want the best
performance. It should be stable and safe to use in production, but it our
hope that the API may be simplified (or not even need to be called) in the
future.
The frequency an application shoudl call optimize is based on the frequency of
data modifications. If data is frequently added, deleted, or updated then data modifications. If data is frequently added, deleted, or updated then
optimize should be run frequently. A good rule of thumb is to run optimize if optimize should be run frequently. A good rule of thumb is to run optimize if
you have added or modified 100,000 or more records or run more than 20 data you have added or modified 100,000 or more records or run more than 20 data
@@ -4688,7 +4826,16 @@ class IndexStatistics:
num_indexed_rows: int num_indexed_rows: int
num_unindexed_rows: int num_unindexed_rows: int
index_type: Literal[ index_type: Literal[
"IVF_PQ", "IVF_HNSW_PQ", "IVF_HNSW_SQ", "FTS", "BTREE", "BITMAP", "LABEL_LIST" "IVF_FLAT",
"IVF_SQ",
"IVF_PQ",
"IVF_RQ",
"IVF_HNSW_SQ",
"IVF_HNSW_PQ",
"FTS",
"BTREE",
"BITMAP",
"LABEL_LIST",
] ]
distance_type: Optional[Literal["l2", "cosine", "dot"]] = None distance_type: Optional[Literal["l2", "cosine", "dot"]] = None
num_indices: Optional[int] = None num_indices: Optional[int] = None

View File

@@ -324,6 +324,16 @@ def _(value: list):
return "[" + ", ".join(map(value_to_sql, value)) + "]" return "[" + ", ".join(map(value_to_sql, value)) + "]"
@value_to_sql.register(dict)
def _(value: dict):
# https://datafusion.apache.org/user-guide/sql/scalar_functions.html#named-struct
return (
"named_struct("
+ ", ".join(f"'{k}', {value_to_sql(v)}" for k, v in value.items())
+ ")"
)
@value_to_sql.register(np.ndarray) @value_to_sql.register(np.ndarray)
def _(value: np.ndarray): def _(value: np.ndarray):
return value_to_sql(value.tolist()) return value_to_sql(value.tolist())

View File

@@ -183,8 +183,8 @@ def test_table_names(tmp_db: lancedb.DBConnection):
result = list(tmp_db.table_names("test2", limit=2)) result = list(tmp_db.table_names("test2", limit=2))
assert result == ["test3"], f"Expected ['test3'], got {result}" assert result == ["test3"], f"Expected ['test3'], got {result}"
# Test that namespace parameter can be passed as keyword # Test that namespace_path parameter can be passed as keyword
result = list(tmp_db.table_names(namespace=[])) result = list(tmp_db.table_names(namespace_path=[]))
assert len(result) == 3 assert len(result) == 3
@@ -909,7 +909,7 @@ def test_local_namespace_operations(tmp_path):
NotImplementedError, NotImplementedError,
match="Namespace operations are not supported for listing database", match="Namespace operations are not supported for listing database",
): ):
db.list_namespaces(namespace=["test"]) db.list_namespaces(namespace_path=["test"])
def test_local_create_namespace_not_supported(tmp_path): def test_local_create_namespace_not_supported(tmp_path):

View File

@@ -546,3 +546,24 @@ def test_openai_no_retry_on_401(mock_sleep):
assert mock_func.call_count == 1 assert mock_func.call_count == 1
# Verify that sleep was never called (no retries) # Verify that sleep was never called (no retries)
assert mock_sleep.call_count == 0 assert mock_sleep.call_count == 0
def test_url_retrieve_downloads_image():
"""
Embedding functions like open-clip, siglip, and jinaai use url_retrieve()
to download images from HTTP URLs. For example, open_clip._to_pil() calls:
PIL_Image.open(io.BytesIO(url_retrieve(image)))
Verify that url_retrieve() can download an image and open it as PIL Image,
matching the real usage pattern in embedding functions.
"""
import io
Image = pytest.importorskip("PIL.Image")
from lancedb.embeddings.utils import url_retrieve
image_url = "http://farm1.staticflickr.com/53/167798175_7c7845bbbd_z.jpg"
image_bytes = url_retrieve(image_url)
img = Image.open(io.BytesIO(image_bytes))
assert img.size[0] > 0 and img.size[1] > 0

View File

@@ -27,6 +27,7 @@ from lancedb.query import (
PhraseQuery, PhraseQuery,
BooleanQuery, BooleanQuery,
Occur, Occur,
LanceFtsQueryBuilder,
) )
import numpy as np import numpy as np
import pyarrow as pa import pyarrow as pa
@@ -920,6 +921,10 @@ def test_fts_fast_search(table):
assert query.limit == 5 assert query.limit == 5
assert query.columns == ["text"] assert query.columns == ["text"]
# fast_search should be enabled by keyword argument too
query = LanceFtsQueryBuilder(table, "xyz", fast_search=True).to_query_object()
assert query.fast_search is True
# Verify it executes without error and skips unindexed data # Verify it executes without error and skips unindexed data
results = table.search("xyz", query_type="fts").fast_search().limit(5).to_list() results = table.search("xyz", query_type="fts").fast_search().limit(5).to_list()
assert len(results) == 0 assert len(results) == 0

View File

@@ -177,6 +177,60 @@ async def test_analyze_plan(table: AsyncTable):
assert "metrics=" in res assert "metrics=" in res
@pytest.fixture
def table_with_id(tmpdir_factory) -> Table:
tmp_path = str(tmpdir_factory.mktemp("data"))
db = lancedb.connect(tmp_path)
data = pa.table(
{
"id": pa.array([1, 2, 3, 4], type=pa.int64()),
"text": pa.array(["a", "b", "cat", "dog"]),
"vector": pa.array(
[[0.1, 0.1], [2, 2], [-0.1, -0.1], [0.5, -0.5]],
type=pa.list_(pa.float32(), list_size=2),
),
}
)
table = db.create_table("test_with_id", data)
table.create_fts_index("text", with_position=False, use_tantivy=False)
return table
def test_hybrid_prefilter_explain_plan(table_with_id: Table):
"""
Verify that the prefilter logic is not inverted in LanceHybridQueryBuilder.
"""
plan_prefilter = (
table_with_id.search(query_type="hybrid")
.vector([0.0, 0.0])
.text("dog")
.where("id = 1", prefilter=True)
.limit(2)
.explain_plan(verbose=True)
)
plan_postfilter = (
table_with_id.search(query_type="hybrid")
.vector([0.0, 0.0])
.text("dog")
.where("id = 1", prefilter=False)
.limit(2)
.explain_plan(verbose=True)
)
# prefilter=True: filter is pushed into the LanceRead scan.
# The FTS sub-plan exposes this as "full_filter=id = Int64(1)" inside LanceRead.
assert "full_filter=id = Int64(1)" in plan_prefilter, (
f"Should push the filter into the scan.\nPlan:\n{plan_prefilter}"
)
# prefilter=False: filter is applied as a separate FilterExec after the search.
# The filter must NOT be embedded in the scan.
assert "full_filter=id = Int64(1)" not in plan_postfilter, (
f"Should NOT push the filter into the scan.\nPlan:\n{plan_postfilter}"
)
def test_normalize_scores(): def test_normalize_scores():
cases = [ cases = [
(pa.array([0.1, 0.4]), pa.array([0.0, 1.0])), (pa.array([0.1, 0.4]), pa.array([0.0, 1.0])),

View File

@@ -3,6 +3,7 @@
from datetime import timedelta from datetime import timedelta
import random import random
from typing import get_args, get_type_hints
import pyarrow as pa import pyarrow as pa
import pytest import pytest
@@ -22,6 +23,7 @@ from lancedb.index import (
HnswSq, HnswSq,
FTS, FTS,
) )
from lancedb.table import IndexStatistics
@pytest_asyncio.fixture @pytest_asyncio.fixture
@@ -283,3 +285,23 @@ async def test_create_index_with_binary_vectors(binary_table: AsyncTable):
for v in range(256): for v in range(256):
res = await binary_table.query().nearest_to([v] * 128).to_arrow() res = await binary_table.query().nearest_to([v] * 128).to_arrow()
assert res["id"][0].as_py() == v assert res["id"][0].as_py() == v
def test_index_statistics_index_type_lists_all_supported_values():
expected_index_types = {
"IVF_FLAT",
"IVF_SQ",
"IVF_PQ",
"IVF_RQ",
"IVF_HNSW_SQ",
"IVF_HNSW_PQ",
"FTS",
"BTREE",
"BITMAP",
"LABEL_LIST",
}
assert (
set(get_args(get_type_hints(IndexStatistics)["index_type"]))
== expected_index_types
)

View File

@@ -8,6 +8,7 @@ import shutil
import pytest import pytest
import pyarrow as pa import pyarrow as pa
import lancedb import lancedb
from lance_namespace.errors import NamespaceNotEmptyError, TableNotFoundError
class TestNamespaceConnection: class TestNamespaceConnection:
@@ -32,6 +33,16 @@ class TestNamespaceConnection:
# Initially no tables in root # Initially no tables in root
assert len(list(db.table_names())) == 0 assert len(list(db.table_names())) == 0
def test_connect_via_connect_helper(self):
"""Connecting via lancedb.connect should delegate to namespace connection."""
db = lancedb.connect(
namespace_client_impl="dir",
namespace_client_properties={"root": self.temp_dir},
)
assert isinstance(db, lancedb.LanceNamespaceDBConnection)
assert len(list(db.table_names())) == 0
def test_create_table_through_namespace(self): def test_create_table_through_namespace(self):
"""Test creating a table through namespace.""" """Test creating a table through namespace."""
db = lancedb.connect_namespace("dir", {"root": self.temp_dir}) db = lancedb.connect_namespace("dir", {"root": self.temp_dir})
@@ -49,14 +60,14 @@ class TestNamespaceConnection:
) )
# Create empty table in child namespace # Create empty table in child namespace
table = db.create_table("test_table", schema=schema, namespace=["test_ns"]) table = db.create_table("test_table", schema=schema, namespace_path=["test_ns"])
assert table is not None assert table is not None
assert table.name == "test_table" assert table.name == "test_table"
assert table.namespace == ["test_ns"] assert table.namespace == ["test_ns"]
assert table.id == "test_ns$test_table" assert table.id == "test_ns$test_table"
# Table should appear in child namespace # Table should appear in child namespace
table_names = list(db.table_names(namespace=["test_ns"])) table_names = list(db.table_names(namespace_path=["test_ns"]))
assert "test_table" in table_names assert "test_table" in table_names
assert len(table_names) == 1 assert len(table_names) == 1
@@ -79,10 +90,10 @@ class TestNamespaceConnection:
pa.field("vector", pa.list_(pa.float32(), 2)), pa.field("vector", pa.list_(pa.float32(), 2)),
] ]
) )
db.create_table("test_table", schema=schema, namespace=["test_ns"]) db.create_table("test_table", schema=schema, namespace_path=["test_ns"])
# Open the table # Open the table
table = db.open_table("test_table", namespace=["test_ns"]) table = db.open_table("test_table", namespace_path=["test_ns"])
assert table is not None assert table is not None
assert table.name == "test_table" assert table.name == "test_table"
assert table.namespace == ["test_ns"] assert table.namespace == ["test_ns"]
@@ -107,31 +118,31 @@ class TestNamespaceConnection:
pa.field("vector", pa.list_(pa.float32(), 2)), pa.field("vector", pa.list_(pa.float32(), 2)),
] ]
) )
db.create_table("table1", schema=schema, namespace=["test_ns"]) db.create_table("table1", schema=schema, namespace_path=["test_ns"])
db.create_table("table2", schema=schema, namespace=["test_ns"]) db.create_table("table2", schema=schema, namespace_path=["test_ns"])
# Verify both tables exist in child namespace # Verify both tables exist in child namespace
table_names = list(db.table_names(namespace=["test_ns"])) table_names = list(db.table_names(namespace_path=["test_ns"]))
assert "table1" in table_names assert "table1" in table_names
assert "table2" in table_names assert "table2" in table_names
assert len(table_names) == 2 assert len(table_names) == 2
# Drop one table # Drop one table
db.drop_table("table1", namespace=["test_ns"]) db.drop_table("table1", namespace_path=["test_ns"])
# Verify only table2 remains # Verify only table2 remains
table_names = list(db.table_names(namespace=["test_ns"])) table_names = list(db.table_names(namespace_path=["test_ns"]))
assert "table1" not in table_names assert "table1" not in table_names
assert "table2" in table_names assert "table2" in table_names
assert len(table_names) == 1 assert len(table_names) == 1
# Drop the second table # Drop the second table
db.drop_table("table2", namespace=["test_ns"]) db.drop_table("table2", namespace_path=["test_ns"])
assert len(list(db.table_names(namespace=["test_ns"]))) == 0 assert len(list(db.table_names(namespace_path=["test_ns"]))) == 0
# Should not be able to open dropped table # Should not be able to open dropped table
with pytest.raises(RuntimeError): with pytest.raises(TableNotFoundError):
db.open_table("table1", namespace=["test_ns"]) db.open_table("table1", namespace_path=["test_ns"])
def test_create_table_with_schema(self): def test_create_table_with_schema(self):
"""Test creating a table with explicit schema through namespace.""" """Test creating a table with explicit schema through namespace."""
@@ -150,7 +161,7 @@ class TestNamespaceConnection:
) )
# Create table with schema in child namespace # Create table with schema in child namespace
table = db.create_table("test_table", schema=schema, namespace=["test_ns"]) table = db.create_table("test_table", schema=schema, namespace_path=["test_ns"])
assert table is not None assert table is not None
assert table.namespace == ["test_ns"] assert table.namespace == ["test_ns"]
@@ -174,7 +185,7 @@ class TestNamespaceConnection:
pa.field("vector", pa.list_(pa.float32(), 2)), pa.field("vector", pa.list_(pa.float32(), 2)),
] ]
) )
db.create_table("old_name", schema=schema, namespace=["test_ns"]) db.create_table("old_name", schema=schema, namespace_path=["test_ns"])
# Rename should raise NotImplementedError # Rename should raise NotImplementedError
with pytest.raises(NotImplementedError, match="rename_table is not supported"): with pytest.raises(NotImplementedError, match="rename_table is not supported"):
@@ -195,20 +206,20 @@ class TestNamespaceConnection:
] ]
) )
for i in range(3): for i in range(3):
db.create_table(f"table{i}", schema=schema, namespace=["test_ns"]) db.create_table(f"table{i}", schema=schema, namespace_path=["test_ns"])
# Verify tables exist in child namespace # Verify tables exist in child namespace
assert len(list(db.table_names(namespace=["test_ns"]))) == 3 assert len(list(db.table_names(namespace_path=["test_ns"]))) == 3
# Drop all tables in child namespace # Drop all tables in child namespace
db.drop_all_tables(namespace=["test_ns"]) db.drop_all_tables(namespace_path=["test_ns"])
# Verify all tables are gone from child namespace # Verify all tables are gone from child namespace
assert len(list(db.table_names(namespace=["test_ns"]))) == 0 assert len(list(db.table_names(namespace_path=["test_ns"]))) == 0
# Test that table_names works with keyword-only namespace parameter # Test that table_names works with keyword-only namespace parameter
db.create_table("test_table", schema=schema, namespace=["test_ns"]) db.create_table("test_table", schema=schema, namespace_path=["test_ns"])
result = list(db.table_names(namespace=["test_ns"])) result = list(db.table_names(namespace_path=["test_ns"]))
assert "test_table" in result assert "test_table" in result
def test_table_operations(self): def test_table_operations(self):
@@ -226,7 +237,7 @@ class TestNamespaceConnection:
pa.field("text", pa.string()), pa.field("text", pa.string()),
] ]
) )
table = db.create_table("test_table", schema=schema, namespace=["test_ns"]) table = db.create_table("test_table", schema=schema, namespace_path=["test_ns"])
# Verify empty table was created # Verify empty table was created
result = table.to_pandas() result = table.to_pandas()
@@ -297,25 +308,25 @@ class TestNamespaceConnection:
] ]
) )
table = db.create_table( table = db.create_table(
"test_table", schema=schema, namespace=["test_namespace"] "test_table", schema=schema, namespace_path=["test_namespace"]
) )
assert table is not None assert table is not None
# Verify table exists in namespace # Verify table exists in namespace
tables_in_namespace = list(db.table_names(namespace=["test_namespace"])) tables_in_namespace = list(db.table_names(namespace_path=["test_namespace"]))
assert "test_table" in tables_in_namespace assert "test_table" in tables_in_namespace
assert len(tables_in_namespace) == 1 assert len(tables_in_namespace) == 1
# Open table from namespace # Open table from namespace
table = db.open_table("test_table", namespace=["test_namespace"]) table = db.open_table("test_table", namespace_path=["test_namespace"])
assert table is not None assert table is not None
assert table.name == "test_table" assert table.name == "test_table"
# Drop table from namespace # Drop table from namespace
db.drop_table("test_table", namespace=["test_namespace"]) db.drop_table("test_table", namespace_path=["test_namespace"])
# Verify table no longer exists in namespace # Verify table no longer exists in namespace
tables_in_namespace = list(db.table_names(namespace=["test_namespace"])) tables_in_namespace = list(db.table_names(namespace_path=["test_namespace"]))
assert len(tables_in_namespace) == 0 assert len(tables_in_namespace) == 0
# Drop namespace # Drop namespace
@@ -337,14 +348,14 @@ class TestNamespaceConnection:
pa.field("vector", pa.list_(pa.float32(), 2)), pa.field("vector", pa.list_(pa.float32(), 2)),
] ]
) )
db.create_table("test_table", schema=schema, namespace=["test_namespace"]) db.create_table("test_table", schema=schema, namespace_path=["test_namespace"])
# Try to drop namespace with tables - should fail # Try to drop namespace with tables - should fail
with pytest.raises(RuntimeError, match="is not empty"): with pytest.raises(NamespaceNotEmptyError):
db.drop_namespace(["test_namespace"]) db.drop_namespace(["test_namespace"])
# Drop table first # Drop table first
db.drop_table("test_table", namespace=["test_namespace"]) db.drop_table("test_table", namespace_path=["test_namespace"])
# Now dropping namespace should work # Now dropping namespace should work
db.drop_namespace(["test_namespace"]) db.drop_namespace(["test_namespace"])
@@ -367,10 +378,10 @@ class TestNamespaceConnection:
# Create table with same name in both namespaces # Create table with same name in both namespaces
table_a = db.create_table( table_a = db.create_table(
"same_name_table", schema=schema, namespace=["namespace_a"] "same_name_table", schema=schema, namespace_path=["namespace_a"]
) )
table_b = db.create_table( table_b = db.create_table(
"same_name_table", schema=schema, namespace=["namespace_b"] "same_name_table", schema=schema, namespace_path=["namespace_b"]
) )
# Add different data to each table # Add different data to each table
@@ -388,7 +399,9 @@ class TestNamespaceConnection:
table_b.add(data_b) table_b.add(data_b)
# Verify data in namespace_a table # Verify data in namespace_a table
opened_table_a = db.open_table("same_name_table", namespace=["namespace_a"]) opened_table_a = db.open_table(
"same_name_table", namespace_path=["namespace_a"]
)
result_a = opened_table_a.to_pandas().sort_values("id").reset_index(drop=True) result_a = opened_table_a.to_pandas().sort_values("id").reset_index(drop=True)
assert len(result_a) == 2 assert len(result_a) == 2
assert result_a["id"].tolist() == [1, 2] assert result_a["id"].tolist() == [1, 2]
@@ -399,7 +412,9 @@ class TestNamespaceConnection:
assert [v.tolist() for v in result_a["vector"]] == [[1.0, 2.0], [3.0, 4.0]] assert [v.tolist() for v in result_a["vector"]] == [[1.0, 2.0], [3.0, 4.0]]
# Verify data in namespace_b table # Verify data in namespace_b table
opened_table_b = db.open_table("same_name_table", namespace=["namespace_b"]) opened_table_b = db.open_table(
"same_name_table", namespace_path=["namespace_b"]
)
result_b = opened_table_b.to_pandas().sort_values("id").reset_index(drop=True) result_b = opened_table_b.to_pandas().sort_values("id").reset_index(drop=True)
assert len(result_b) == 3 assert len(result_b) == 3
assert result_b["id"].tolist() == [10, 20, 30] assert result_b["id"].tolist() == [10, 20, 30]
@@ -419,8 +434,8 @@ class TestNamespaceConnection:
assert "same_name_table" not in root_tables assert "same_name_table" not in root_tables
# Clean up # Clean up
db.drop_table("same_name_table", namespace=["namespace_a"]) db.drop_table("same_name_table", namespace_path=["namespace_a"])
db.drop_table("same_name_table", namespace=["namespace_b"]) db.drop_table("same_name_table", namespace_path=["namespace_b"])
db.drop_namespace(["namespace_a"]) db.drop_namespace(["namespace_a"])
db.drop_namespace(["namespace_b"]) db.drop_namespace(["namespace_b"])
@@ -448,6 +463,8 @@ class TestAsyncNamespaceConnection:
table_names = await db.table_names() table_names = await db.table_names()
assert len(list(table_names)) == 0 assert len(list(table_names)) == 0
# Async connect via namespace helper is not enabled yet.
async def test_create_table_async(self): async def test_create_table_async(self):
"""Test creating a table asynchronously through namespace.""" """Test creating a table asynchronously through namespace."""
db = lancedb.connect_namespace_async("dir", {"root": self.temp_dir}) db = lancedb.connect_namespace_async("dir", {"root": self.temp_dir})
@@ -466,13 +483,13 @@ class TestAsyncNamespaceConnection:
# Create empty table in child namespace # Create empty table in child namespace
table = await db.create_table( table = await db.create_table(
"test_table", schema=schema, namespace=["test_ns"] "test_table", schema=schema, namespace_path=["test_ns"]
) )
assert table is not None assert table is not None
assert isinstance(table, lancedb.AsyncTable) assert isinstance(table, lancedb.AsyncTable)
# Table should appear in child namespace # Table should appear in child namespace
table_names = await db.table_names(namespace=["test_ns"]) table_names = await db.table_names(namespace_path=["test_ns"])
assert "test_table" in list(table_names) assert "test_table" in list(table_names)
async def test_open_table_async(self): async def test_open_table_async(self):
@@ -489,10 +506,10 @@ class TestAsyncNamespaceConnection:
pa.field("vector", pa.list_(pa.float32(), 2)), pa.field("vector", pa.list_(pa.float32(), 2)),
] ]
) )
await db.create_table("test_table", schema=schema, namespace=["test_ns"]) await db.create_table("test_table", schema=schema, namespace_path=["test_ns"])
# Open the table # Open the table
table = await db.open_table("test_table", namespace=["test_ns"]) table = await db.open_table("test_table", namespace_path=["test_ns"])
assert table is not None assert table is not None
assert isinstance(table, lancedb.AsyncTable) assert isinstance(table, lancedb.AsyncTable)
@@ -546,20 +563,20 @@ class TestAsyncNamespaceConnection:
pa.field("vector", pa.list_(pa.float32(), 2)), pa.field("vector", pa.list_(pa.float32(), 2)),
] ]
) )
await db.create_table("table1", schema=schema, namespace=["test_ns"]) await db.create_table("table1", schema=schema, namespace_path=["test_ns"])
await db.create_table("table2", schema=schema, namespace=["test_ns"]) await db.create_table("table2", schema=schema, namespace_path=["test_ns"])
# Verify both tables exist in child namespace # Verify both tables exist in child namespace
table_names = list(await db.table_names(namespace=["test_ns"])) table_names = list(await db.table_names(namespace_path=["test_ns"]))
assert "table1" in table_names assert "table1" in table_names
assert "table2" in table_names assert "table2" in table_names
assert len(table_names) == 2 assert len(table_names) == 2
# Drop one table # Drop one table
await db.drop_table("table1", namespace=["test_ns"]) await db.drop_table("table1", namespace_path=["test_ns"])
# Verify only table2 remains # Verify only table2 remains
table_names = list(await db.table_names(namespace=["test_ns"])) table_names = list(await db.table_names(namespace_path=["test_ns"]))
assert "table1" not in table_names assert "table1" not in table_names
assert "table2" in table_names assert "table2" in table_names
assert len(table_names) == 1 assert len(table_names) == 1
@@ -588,20 +605,24 @@ class TestAsyncNamespaceConnection:
] ]
) )
table = await db.create_table( table = await db.create_table(
"test_table", schema=schema, namespace=["test_namespace"] "test_table", schema=schema, namespace_path=["test_namespace"]
) )
assert table is not None assert table is not None
# Verify table exists in namespace # Verify table exists in namespace
tables_in_namespace = list(await db.table_names(namespace=["test_namespace"])) tables_in_namespace = list(
await db.table_names(namespace_path=["test_namespace"])
)
assert "test_table" in tables_in_namespace assert "test_table" in tables_in_namespace
assert len(tables_in_namespace) == 1 assert len(tables_in_namespace) == 1
# Drop table from namespace # Drop table from namespace
await db.drop_table("test_table", namespace=["test_namespace"]) await db.drop_table("test_table", namespace_path=["test_namespace"])
# Verify table no longer exists in namespace # Verify table no longer exists in namespace
tables_in_namespace = list(await db.table_names(namespace=["test_namespace"])) tables_in_namespace = list(
await db.table_names(namespace_path=["test_namespace"])
)
assert len(tables_in_namespace) == 0 assert len(tables_in_namespace) == 0
# Drop namespace # Drop namespace
@@ -626,15 +647,98 @@ class TestAsyncNamespaceConnection:
] ]
) )
for i in range(3): for i in range(3):
await db.create_table(f"table{i}", schema=schema, namespace=["test_ns"]) await db.create_table(
f"table{i}", schema=schema, namespace_path=["test_ns"]
)
# Verify tables exist in child namespace # Verify tables exist in child namespace
table_names = await db.table_names(namespace=["test_ns"]) table_names = await db.table_names(namespace_path=["test_ns"])
assert len(list(table_names)) == 3 assert len(list(table_names)) == 3
# Drop all tables in child namespace # Drop all tables in child namespace
await db.drop_all_tables(namespace=["test_ns"]) await db.drop_all_tables(namespace_path=["test_ns"])
# Verify all tables are gone from child namespace # Verify all tables are gone from child namespace
table_names = await db.table_names(namespace=["test_ns"]) table_names = await db.table_names(namespace_path=["test_ns"])
assert len(list(table_names)) == 0 assert len(list(table_names)) == 0
class TestPushdownOperations:
"""Test pushdown operations on namespace connections."""
def setup_method(self):
"""Set up test fixtures."""
self.temp_dir = tempfile.mkdtemp()
def teardown_method(self):
"""Clean up test fixtures."""
shutil.rmtree(self.temp_dir, ignore_errors=True)
def test_query_table_pushdown_stored(self):
"""Test that QueryTable pushdown is stored on sync connection."""
db = lancedb.connect_namespace(
"dir",
{"root": self.temp_dir},
namespace_client_pushdown_operations=["QueryTable"],
)
assert "QueryTable" in db._pushdown_operations
def test_create_table_pushdown_stored(self):
"""Test that CreateTable pushdown is stored on sync connection."""
db = lancedb.connect_namespace(
"dir",
{"root": self.temp_dir},
namespace_client_pushdown_operations=["CreateTable"],
)
assert "CreateTable" in db._pushdown_operations
def test_both_pushdowns_stored(self):
"""Test that both pushdown operations can be set together."""
db = lancedb.connect_namespace(
"dir",
{"root": self.temp_dir},
namespace_client_pushdown_operations=["QueryTable", "CreateTable"],
)
assert "QueryTable" in db._pushdown_operations
assert "CreateTable" in db._pushdown_operations
def test_pushdown_defaults_to_empty(self):
"""Test that pushdown operations default to empty."""
db = lancedb.connect_namespace("dir", {"root": self.temp_dir})
assert len(db._pushdown_operations) == 0
@pytest.mark.asyncio
class TestAsyncPushdownOperations:
"""Test pushdown operations on async namespace connections."""
def setup_method(self):
"""Set up test fixtures."""
self.temp_dir = tempfile.mkdtemp()
def teardown_method(self):
"""Clean up test fixtures."""
shutil.rmtree(self.temp_dir, ignore_errors=True)
async def test_async_query_table_pushdown_stored(self):
"""Test that QueryTable pushdown is stored on async connection."""
db = lancedb.connect_namespace_async(
"dir",
{"root": self.temp_dir},
namespace_client_pushdown_operations=["QueryTable"],
)
assert "QueryTable" in db._pushdown_operations
async def test_async_create_table_pushdown_stored(self):
"""Test that CreateTable pushdown is stored on async connection."""
db = lancedb.connect_namespace_async(
"dir",
{"root": self.temp_dir},
namespace_client_pushdown_operations=["CreateTable"],
)
assert "CreateTable" in db._pushdown_operations
async def test_async_pushdown_defaults_to_empty(self):
"""Test that pushdown operations default to empty on async connection."""
db = lancedb.connect_namespace_async("dir", {"root": self.temp_dir})
assert len(db._pushdown_operations) == 0

View File

@@ -4,9 +4,11 @@
""" """
Integration tests for LanceDB Namespace with S3 and credential refresh. Integration tests for LanceDB Namespace with S3 and credential refresh.
This test simulates a namespace server that returns incrementing credentials This test uses DirectoryNamespace with native ops_metrics and vend_input_storage_options
and verifies that the credential refresh mechanism works correctly for both features to track API calls and test credential refresh mechanisms.
create_table and open_table operations.
Tests are parameterized to run with both DirectoryNamespace and a CustomNamespace
wrapper to verify Python-Rust binding works correctly for custom implementations.
Tests verify: Tests verify:
- Storage options provider is auto-created and used - Storage options provider is auto-created and used
@@ -18,22 +20,136 @@ Tests verify:
import copy import copy
import time import time
import uuid import uuid
from threading import Lock from typing import Dict, Optional
from typing import Dict
import pyarrow as pa import pyarrow as pa
import pytest import pytest
from lance_namespace import ( from lance.namespace import (
CreateEmptyTableRequest,
CreateEmptyTableResponse,
DeclareTableRequest, DeclareTableRequest,
DeclareTableResponse, DeclareTableResponse,
DescribeTableRequest, DescribeTableRequest,
DescribeTableResponse, DescribeTableResponse,
DirectoryNamespace,
LanceNamespace, LanceNamespace,
) )
from lance_namespace import (
CreateNamespaceRequest,
CreateNamespaceResponse,
CreateTableRequest,
CreateTableResponse,
CreateTableVersionRequest,
CreateTableVersionResponse,
DeregisterTableRequest,
DeregisterTableResponse,
DescribeNamespaceRequest,
DescribeNamespaceResponse,
DescribeTableVersionRequest,
DescribeTableVersionResponse,
DropNamespaceRequest,
DropNamespaceResponse,
DropTableRequest,
DropTableResponse,
ListNamespacesRequest,
ListNamespacesResponse,
ListTablesRequest,
ListTablesResponse,
ListTableVersionsRequest,
ListTableVersionsResponse,
NamespaceExistsRequest,
RegisterTableRequest,
RegisterTableResponse,
TableExistsRequest,
)
from lancedb.namespace import LanceNamespaceDBConnection from lancedb.namespace import LanceNamespaceDBConnection
class CustomNamespace(LanceNamespace):
"""A custom namespace wrapper that delegates to DirectoryNamespace.
This class verifies that the Python-Rust binding works correctly for
custom namespace implementations that wrap the native DirectoryNamespace.
All methods simply delegate to the underlying DirectoryNamespace instance.
"""
def __init__(self, inner: DirectoryNamespace):
self._inner = inner
def namespace_id(self) -> str:
return f"CustomNamespace[{self._inner.namespace_id()}]"
def create_namespace(
self, request: CreateNamespaceRequest
) -> CreateNamespaceResponse:
return self._inner.create_namespace(request)
def describe_namespace(
self, request: DescribeNamespaceRequest
) -> DescribeNamespaceResponse:
return self._inner.describe_namespace(request)
def namespace_exists(self, request: NamespaceExistsRequest) -> None:
return self._inner.namespace_exists(request)
def drop_namespace(self, request: DropNamespaceRequest) -> DropNamespaceResponse:
return self._inner.drop_namespace(request)
def list_namespaces(self, request: ListNamespacesRequest) -> ListNamespacesResponse:
return self._inner.list_namespaces(request)
def create_table(
self, request: CreateTableRequest, data: bytes
) -> CreateTableResponse:
return self._inner.create_table(request, data)
def declare_table(self, request: DeclareTableRequest) -> DeclareTableResponse:
return self._inner.declare_table(request)
def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse:
return self._inner.describe_table(request)
def table_exists(self, request: TableExistsRequest) -> None:
return self._inner.table_exists(request)
def drop_table(self, request: DropTableRequest) -> DropTableResponse:
return self._inner.drop_table(request)
def list_tables(self, request: ListTablesRequest) -> ListTablesResponse:
return self._inner.list_tables(request)
def register_table(self, request: RegisterTableRequest) -> RegisterTableResponse:
return self._inner.register_table(request)
def deregister_table(
self, request: DeregisterTableRequest
) -> DeregisterTableResponse:
return self._inner.deregister_table(request)
def list_table_versions(
self, request: ListTableVersionsRequest
) -> ListTableVersionsResponse:
return self._inner.list_table_versions(request)
def describe_table_version(
self, request: DescribeTableVersionRequest
) -> DescribeTableVersionResponse:
return self._inner.describe_table_version(request)
def create_table_version(
self, request: CreateTableVersionRequest
) -> CreateTableVersionResponse:
return self._inner.create_table_version(request)
def retrieve_ops_metrics(self) -> Optional[Dict[str, int]]:
return self._inner.retrieve_ops_metrics()
def _wrap_if_custom(ns_client: DirectoryNamespace, use_custom: bool):
"""Wrap namespace client in CustomNamespace if use_custom is True."""
if use_custom:
return CustomNamespace(ns_client)
return ns_client
# LocalStack S3 configuration # LocalStack S3 configuration
CONFIG = { CONFIG = {
"allow_http": "true", "allow_http": "true",
@@ -89,157 +205,88 @@ def delete_bucket(s3, bucket_name):
pass pass
class TrackingNamespace(LanceNamespace): def create_tracking_namespace(
bucket_name: str,
storage_options: dict,
credential_expires_in_seconds: int = 60,
use_custom: bool = False,
):
"""Create a DirectoryNamespace with ops metrics and credential vending enabled.
Uses native DirectoryNamespace features:
- ops_metrics_enabled=true: Tracks API call counts via retrieve_ops_metrics()
- vend_input_storage_options=true: Returns input storage options in responses
- vend_input_storage_options_refresh_interval_millis: Adds expires_at_millis
Args:
bucket_name: S3 bucket name or local path
storage_options: Storage options to pass through (credentials, endpoint, etc.)
credential_expires_in_seconds: Interval in seconds for credential expiration
use_custom: If True, wrap in CustomNamespace for testing custom implementations
Returns:
Tuple of (namespace_client, inner_namespace_client) where inner is always
the DirectoryNamespace (used for metrics retrieval)
""" """
Mock namespace that wraps DirectoryNamespace and tracks API calls. # Add refresh_offset_millis to storage options so that credentials are not
# considered expired immediately. Set to 1 second (1000ms) so that refresh
# checks work correctly with short-lived credentials in tests.
storage_options_with_refresh = dict(storage_options)
storage_options_with_refresh["refresh_offset_millis"] = "1000"
This namespace returns incrementing credentials with each API call to simulate dir_props = {f"storage.{k}": v for k, v in storage_options_with_refresh.items()}
credential rotation. It also tracks the number of times each API is called
to verify caching behavior.
"""
def __init__( if bucket_name.startswith("/") or bucket_name.startswith("file://"):
self, dir_props["root"] = f"{bucket_name}/namespace_root"
bucket_name: str, else:
storage_options: Dict[str, str], dir_props["root"] = f"s3://{bucket_name}/namespace_root"
credential_expires_in_seconds: int = 60,
):
from lance.namespace import DirectoryNamespace
self.bucket_name = bucket_name # Enable ops metrics tracking
self.base_storage_options = storage_options dir_props["ops_metrics_enabled"] = "true"
self.credential_expires_in_seconds = credential_expires_in_seconds # Enable storage options vending
self.describe_call_count = 0 dir_props["vend_input_storage_options"] = "true"
self.create_call_count = 0 # Set refresh interval in milliseconds
self.lock = Lock() dir_props["vend_input_storage_options_refresh_interval_millis"] = str(
credential_expires_in_seconds * 1000
)
# Create underlying DirectoryNamespace with storage options inner_ns_client = DirectoryNamespace(**dir_props)
dir_props = {f"storage.{k}": v for k, v in storage_options.items()} ns_client = _wrap_if_custom(inner_ns_client, use_custom)
return ns_client, inner_ns_client
# Use S3 path for bucket name, local path for file paths
if bucket_name.startswith("/") or bucket_name.startswith("file://"):
dir_props["root"] = f"{bucket_name}/namespace_root"
else:
dir_props["root"] = f"s3://{bucket_name}/namespace_root"
self.inner = DirectoryNamespace(**dir_props) def get_describe_call_count(namespace_client) -> int:
"""Get the number of describe_table calls made to the namespace client."""
return namespace_client.retrieve_ops_metrics().get("describe_table", 0)
def get_describe_call_count(self) -> int:
"""Thread-safe getter for describe call count."""
with self.lock:
return self.describe_call_count
def get_create_call_count(self) -> int: def get_declare_call_count(namespace_client) -> int:
"""Thread-safe getter for create call count.""" """Get the number of declare_table calls made to the namespace client."""
with self.lock: return namespace_client.retrieve_ops_metrics().get("declare_table", 0)
return self.create_call_count
def namespace_id(self) -> str:
"""Return namespace identifier."""
return f"TrackingNamespace {{ inner: {self.inner.namespace_id()} }}"
def _modify_storage_options(
self, storage_options: Dict[str, str], count: int
) -> Dict[str, str]:
"""
Add incrementing credentials with expiration timestamp.
This simulates a credential rotation system where each call returns
new credentials that expire after credential_expires_in_seconds.
"""
modified = copy.deepcopy(storage_options) if storage_options else {}
# Increment credentials to simulate rotation
modified["aws_access_key_id"] = f"AKID_{count}"
modified["aws_secret_access_key"] = f"SECRET_{count}"
modified["aws_session_token"] = f"TOKEN_{count}"
# Set expiration time
expires_at_millis = int(
(time.time() + self.credential_expires_in_seconds) * 1000
)
modified["expires_at_millis"] = str(expires_at_millis)
return modified
def declare_table(self, request: DeclareTableRequest) -> DeclareTableResponse:
"""Track declare_table calls and inject rotating credentials."""
with self.lock:
self.create_call_count += 1
count = self.create_call_count
response = self.inner.declare_table(request)
response.storage_options = self._modify_storage_options(
response.storage_options, count
)
return response
def create_empty_table(
self, request: CreateEmptyTableRequest
) -> CreateEmptyTableResponse:
"""Track create_empty_table calls and inject rotating credentials."""
with self.lock:
self.create_call_count += 1
count = self.create_call_count
response = self.inner.create_empty_table(request)
response.storage_options = self._modify_storage_options(
response.storage_options, count
)
return response
def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse:
"""Track describe_table calls and inject rotating credentials."""
with self.lock:
self.describe_call_count += 1
count = self.describe_call_count
response = self.inner.describe_table(request)
response.storage_options = self._modify_storage_options(
response.storage_options, count
)
return response
# Pass through other methods to inner namespace
def list_tables(self, request):
return self.inner.list_tables(request)
def drop_table(self, request):
return self.inner.drop_table(request)
def list_namespaces(self, request):
return self.inner.list_namespaces(request)
def create_namespace(self, request):
return self.inner.create_namespace(request)
def drop_namespace(self, request):
return self.inner.drop_namespace(request)
@pytest.mark.s3_test @pytest.mark.s3_test
def test_namespace_create_table_with_provider(s3_bucket: str): @pytest.mark.parametrize("use_custom", [False, True], ids=["DirectoryNS", "CustomNS"])
def test_namespace_create_table_with_provider(s3_bucket: str, use_custom: bool):
""" """
Test creating a table through namespace with storage options provider. Test creating a table through namespace with storage options provider.
Verifies: Verifies:
- create_empty_table is called once to reserve location - declare_table is called once to reserve location
- Storage options provider is auto-created - Storage options provider is auto-created
- Table can be written successfully - Table can be written successfully
- Credentials are cached during write operations - Credentials are cached during write operations
""" """
storage_options = copy.deepcopy(CONFIG) storage_options = copy.deepcopy(CONFIG)
namespace = TrackingNamespace( ns_client, inner_ns_client = create_tracking_namespace(
bucket_name=s3_bucket, bucket_name=s3_bucket,
storage_options=storage_options, storage_options=storage_options,
credential_expires_in_seconds=3600, # 1 hour credential_expires_in_seconds=3600, # 1 hour
use_custom=use_custom,
) )
db = LanceNamespaceDBConnection(namespace) db = LanceNamespaceDBConnection(ns_client)
# Create unique namespace for this test # Create unique namespace for this test
namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}" namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}"
@@ -249,8 +296,8 @@ def test_namespace_create_table_with_provider(s3_bucket: str):
namespace_path = [namespace_name] namespace_path = [namespace_name]
# Verify initial state # Verify initial state
assert namespace.get_create_call_count() == 0 assert get_declare_call_count(inner_ns_client) == 0
assert namespace.get_describe_call_count() == 0 assert get_describe_call_count(inner_ns_client) == 0
# Create table with data # Create table with data
data = pa.table( data = pa.table(
@@ -261,12 +308,12 @@ def test_namespace_create_table_with_provider(s3_bucket: str):
} }
) )
table = db.create_table(table_name, data, namespace=namespace_path) table = db.create_table(table_name, data, namespace_path=namespace_path)
# Verify create_empty_table was called exactly once # Verify declare_table was called exactly once
assert namespace.get_create_call_count() == 1 assert get_declare_call_count(inner_ns_client) == 1
# describe_table should NOT be called during create in create mode # describe_table should NOT be called during create in create mode
assert namespace.get_describe_call_count() == 0 assert get_describe_call_count(inner_ns_client) == 0
# Verify table was created successfully # Verify table was created successfully
assert table.name == table_name assert table.name == table_name
@@ -276,7 +323,8 @@ def test_namespace_create_table_with_provider(s3_bucket: str):
@pytest.mark.s3_test @pytest.mark.s3_test
def test_namespace_open_table_with_provider(s3_bucket: str): @pytest.mark.parametrize("use_custom", [False, True], ids=["DirectoryNS", "CustomNS"])
def test_namespace_open_table_with_provider(s3_bucket: str, use_custom: bool):
""" """
Test opening a table through namespace with storage options provider. Test opening a table through namespace with storage options provider.
@@ -288,13 +336,14 @@ def test_namespace_open_table_with_provider(s3_bucket: str):
""" """
storage_options = copy.deepcopy(CONFIG) storage_options = copy.deepcopy(CONFIG)
namespace = TrackingNamespace( ns_client, inner_ns_client = create_tracking_namespace(
bucket_name=s3_bucket, bucket_name=s3_bucket,
storage_options=storage_options, storage_options=storage_options,
credential_expires_in_seconds=3600, credential_expires_in_seconds=3600,
use_custom=use_custom,
) )
db = LanceNamespaceDBConnection(namespace) db = LanceNamespaceDBConnection(ns_client)
# Create unique namespace for this test # Create unique namespace for this test
namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}" namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}"
@@ -312,21 +361,21 @@ def test_namespace_open_table_with_provider(s3_bucket: str):
} }
) )
db.create_table(table_name, data, namespace=namespace_path) db.create_table(table_name, data, namespace_path=namespace_path)
initial_create_count = namespace.get_create_call_count() initial_declare_count = get_declare_call_count(inner_ns_client)
assert initial_create_count == 1 assert initial_declare_count == 1
# Open the table # Open the table
opened_table = db.open_table(table_name, namespace=namespace_path) opened_table = db.open_table(table_name, namespace_path=namespace_path)
# Verify describe_table was called exactly once # Verify describe_table was called exactly once
assert namespace.get_describe_call_count() == 1 assert get_describe_call_count(inner_ns_client) == 1
# create_empty_table should not be called again # declare_table should not be called again
assert namespace.get_create_call_count() == initial_create_count assert get_declare_call_count(inner_ns_client) == initial_declare_count
# Perform multiple read operations # Perform multiple read operations
describe_count_after_open = namespace.get_describe_call_count() describe_count_after_open = get_describe_call_count(inner_ns_client)
for _ in range(3): for _ in range(3):
result = opened_table.to_pandas() result = opened_table.to_pandas()
@@ -335,11 +384,12 @@ def test_namespace_open_table_with_provider(s3_bucket: str):
assert count == 5 assert count == 5
# Verify credentials were cached (no additional describe_table calls) # Verify credentials were cached (no additional describe_table calls)
assert namespace.get_describe_call_count() == describe_count_after_open assert get_describe_call_count(inner_ns_client) == describe_count_after_open
@pytest.mark.s3_test @pytest.mark.s3_test
def test_namespace_credential_refresh_on_read(s3_bucket: str): @pytest.mark.parametrize("use_custom", [False, True], ids=["DirectoryNS", "CustomNS"])
def test_namespace_credential_refresh_on_read(s3_bucket: str, use_custom: bool):
""" """
Test credential refresh when credentials expire during read operations. Test credential refresh when credentials expire during read operations.
@@ -350,13 +400,14 @@ def test_namespace_credential_refresh_on_read(s3_bucket: str):
""" """
storage_options = copy.deepcopy(CONFIG) storage_options = copy.deepcopy(CONFIG)
namespace = TrackingNamespace( ns_client, inner_ns_client = create_tracking_namespace(
bucket_name=s3_bucket, bucket_name=s3_bucket,
storage_options=storage_options, storage_options=storage_options,
credential_expires_in_seconds=3, # Short expiration for testing credential_expires_in_seconds=3, # Short expiration for testing
use_custom=use_custom,
) )
db = LanceNamespaceDBConnection(namespace) db = LanceNamespaceDBConnection(ns_client)
# Create unique namespace for this test # Create unique namespace for this test
namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}" namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}"
@@ -373,16 +424,16 @@ def test_namespace_credential_refresh_on_read(s3_bucket: str):
} }
) )
db.create_table(table_name, data, namespace=namespace_path) db.create_table(table_name, data, namespace_path=namespace_path)
# Open table (triggers describe_table) # Open table (triggers describe_table)
opened_table = db.open_table(table_name, namespace=namespace_path) opened_table = db.open_table(table_name, namespace_path=namespace_path)
# Perform an immediate read (should use credentials from open) # Perform an immediate read (should use credentials from open)
result = opened_table.to_pandas() result = opened_table.to_pandas()
assert len(result) == 3 assert len(result) == 3
describe_count_after_first_read = namespace.get_describe_call_count() describe_count_after_first_read = get_describe_call_count(inner_ns_client)
# Wait for credentials to expire (3 seconds + buffer) # Wait for credentials to expire (3 seconds + buffer)
time.sleep(5) time.sleep(5)
@@ -391,7 +442,7 @@ def test_namespace_credential_refresh_on_read(s3_bucket: str):
result = opened_table.to_pandas() result = opened_table.to_pandas()
assert len(result) == 3 assert len(result) == 3
describe_count_after_refresh = namespace.get_describe_call_count() describe_count_after_refresh = get_describe_call_count(inner_ns_client)
# Verify describe_table was called again (credential refresh) # Verify describe_table was called again (credential refresh)
refresh_delta = describe_count_after_refresh - describe_count_after_first_read refresh_delta = describe_count_after_refresh - describe_count_after_first_read
@@ -404,7 +455,8 @@ def test_namespace_credential_refresh_on_read(s3_bucket: str):
@pytest.mark.s3_test @pytest.mark.s3_test
def test_namespace_credential_refresh_on_write(s3_bucket: str): @pytest.mark.parametrize("use_custom", [False, True], ids=["DirectoryNS", "CustomNS"])
def test_namespace_credential_refresh_on_write(s3_bucket: str, use_custom: bool):
""" """
Test credential refresh when credentials expire during write operations. Test credential refresh when credentials expire during write operations.
@@ -415,13 +467,14 @@ def test_namespace_credential_refresh_on_write(s3_bucket: str):
""" """
storage_options = copy.deepcopy(CONFIG) storage_options = copy.deepcopy(CONFIG)
namespace = TrackingNamespace( ns_client, inner_ns_client = create_tracking_namespace(
bucket_name=s3_bucket, bucket_name=s3_bucket,
storage_options=storage_options, storage_options=storage_options,
credential_expires_in_seconds=3, # Short expiration credential_expires_in_seconds=3, # Short expiration
use_custom=use_custom,
) )
db = LanceNamespaceDBConnection(namespace) db = LanceNamespaceDBConnection(ns_client)
# Create unique namespace for this test # Create unique namespace for this test
namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}" namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}"
@@ -438,7 +491,7 @@ def test_namespace_credential_refresh_on_write(s3_bucket: str):
} }
) )
table = db.create_table(table_name, initial_data, namespace=namespace_path) table = db.create_table(table_name, initial_data, namespace_path=namespace_path)
# Add more data (should use cached credentials) # Add more data (should use cached credentials)
new_data = pa.table( new_data = pa.table(
@@ -466,24 +519,26 @@ def test_namespace_credential_refresh_on_write(s3_bucket: str):
@pytest.mark.s3_test @pytest.mark.s3_test
def test_namespace_overwrite_mode(s3_bucket: str): @pytest.mark.parametrize("use_custom", [False, True], ids=["DirectoryNS", "CustomNS"])
def test_namespace_overwrite_mode(s3_bucket: str, use_custom: bool):
""" """
Test creating table in overwrite mode with credential tracking. Test creating table in overwrite mode with credential tracking.
Verifies: Verifies:
- First create calls create_empty_table exactly once - First create calls declare_table exactly once
- Overwrite mode calls describe_table exactly once to check existence - Overwrite mode calls describe_table exactly once to check existence
- Storage options provider works in overwrite mode - Storage options provider works in overwrite mode
""" """
storage_options = copy.deepcopy(CONFIG) storage_options = copy.deepcopy(CONFIG)
namespace = TrackingNamespace( ns_client, inner_ns_client = create_tracking_namespace(
bucket_name=s3_bucket, bucket_name=s3_bucket,
storage_options=storage_options, storage_options=storage_options,
credential_expires_in_seconds=3600, credential_expires_in_seconds=3600,
use_custom=use_custom,
) )
db = LanceNamespaceDBConnection(namespace) db = LanceNamespaceDBConnection(ns_client)
# Create unique namespace for this test # Create unique namespace for this test
namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}" namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}"
@@ -500,11 +555,11 @@ def test_namespace_overwrite_mode(s3_bucket: str):
} }
) )
table = db.create_table(table_name, data1, namespace=namespace_path) table = db.create_table(table_name, data1, namespace_path=namespace_path)
# Exactly one create_empty_table call for initial create # Exactly one declare_table call for initial create
assert namespace.get_create_call_count() == 1 assert get_declare_call_count(inner_ns_client) == 1
# No describe_table calls in create mode # No describe_table calls in create mode
assert namespace.get_describe_call_count() == 0 assert get_describe_call_count(inner_ns_client) == 0
assert table.count_rows() == 2 assert table.count_rows() == 2
# Overwrite the table # Overwrite the table
@@ -516,14 +571,14 @@ def test_namespace_overwrite_mode(s3_bucket: str):
) )
table2 = db.create_table( table2 = db.create_table(
table_name, data2, namespace=namespace_path, mode="overwrite" table_name, data2, namespace_path=namespace_path, mode="overwrite"
) )
# Should still have only 1 create_empty_table call # Should still have only 1 declare_table call
# (overwrite reuses location from describe_table) # (overwrite reuses location from describe_table)
assert namespace.get_create_call_count() == 1 assert get_declare_call_count(inner_ns_client) == 1
# Should have called describe_table exactly once to get existing table location # Should have called describe_table exactly once to get existing table location
assert namespace.get_describe_call_count() == 1 assert get_describe_call_count(inner_ns_client) == 1
# Verify new data # Verify new data
assert table2.count_rows() == 3 assert table2.count_rows() == 3
@@ -532,7 +587,8 @@ def test_namespace_overwrite_mode(s3_bucket: str):
@pytest.mark.s3_test @pytest.mark.s3_test
def test_namespace_multiple_tables(s3_bucket: str): @pytest.mark.parametrize("use_custom", [False, True], ids=["DirectoryNS", "CustomNS"])
def test_namespace_multiple_tables(s3_bucket: str, use_custom: bool):
""" """
Test creating and opening multiple tables in the same namespace. Test creating and opening multiple tables in the same namespace.
@@ -543,13 +599,14 @@ def test_namespace_multiple_tables(s3_bucket: str):
""" """
storage_options = copy.deepcopy(CONFIG) storage_options = copy.deepcopy(CONFIG)
namespace = TrackingNamespace( ns_client, inner_ns_client = create_tracking_namespace(
bucket_name=s3_bucket, bucket_name=s3_bucket,
storage_options=storage_options, storage_options=storage_options,
credential_expires_in_seconds=3600, credential_expires_in_seconds=3600,
use_custom=use_custom,
) )
db = LanceNamespaceDBConnection(namespace) db = LanceNamespaceDBConnection(ns_client)
# Create unique namespace for this test # Create unique namespace for this test
namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}" namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}"
@@ -559,22 +616,22 @@ def test_namespace_multiple_tables(s3_bucket: str):
# Create first table # Create first table
table1_name = f"table1_{uuid.uuid4().hex}" table1_name = f"table1_{uuid.uuid4().hex}"
data1 = pa.table({"id": [1, 2], "value": [10, 20]}) data1 = pa.table({"id": [1, 2], "value": [10, 20]})
db.create_table(table1_name, data1, namespace=namespace_path) db.create_table(table1_name, data1, namespace_path=namespace_path)
# Create second table # Create second table
table2_name = f"table2_{uuid.uuid4().hex}" table2_name = f"table2_{uuid.uuid4().hex}"
data2 = pa.table({"id": [3, 4], "value": [30, 40]}) data2 = pa.table({"id": [3, 4], "value": [30, 40]})
db.create_table(table2_name, data2, namespace=namespace_path) db.create_table(table2_name, data2, namespace_path=namespace_path)
# Should have 2 create calls (one per table) # Should have 2 declare calls (one per table)
assert namespace.get_create_call_count() == 2 assert get_declare_call_count(inner_ns_client) == 2
# Open both tables # Open both tables
opened1 = db.open_table(table1_name, namespace=namespace_path) opened1 = db.open_table(table1_name, namespace_path=namespace_path)
opened2 = db.open_table(table2_name, namespace=namespace_path) opened2 = db.open_table(table2_name, namespace_path=namespace_path)
# Should have 2 describe calls (one per open) # Should have 2 describe calls (one per open)
assert namespace.get_describe_call_count() == 2 assert get_describe_call_count(inner_ns_client) == 2
# Verify both tables work independently # Verify both tables work independently
assert opened1.count_rows() == 2 assert opened1.count_rows() == 2
@@ -588,7 +645,8 @@ def test_namespace_multiple_tables(s3_bucket: str):
@pytest.mark.s3_test @pytest.mark.s3_test
def test_namespace_with_schema_only(s3_bucket: str): @pytest.mark.parametrize("use_custom", [False, True], ids=["DirectoryNS", "CustomNS"])
def test_namespace_with_schema_only(s3_bucket: str, use_custom: bool):
""" """
Test creating empty table with schema only (no data). Test creating empty table with schema only (no data).
@@ -599,13 +657,14 @@ def test_namespace_with_schema_only(s3_bucket: str):
""" """
storage_options = copy.deepcopy(CONFIG) storage_options = copy.deepcopy(CONFIG)
namespace = TrackingNamespace( ns_client, inner_ns_client = create_tracking_namespace(
bucket_name=s3_bucket, bucket_name=s3_bucket,
storage_options=storage_options, storage_options=storage_options,
credential_expires_in_seconds=3600, credential_expires_in_seconds=3600,
use_custom=use_custom,
) )
db = LanceNamespaceDBConnection(namespace) db = LanceNamespaceDBConnection(ns_client)
# Create unique namespace for this test # Create unique namespace for this test
namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}" namespace_name = f"test_ns_{uuid.uuid4().hex[:8]}"
@@ -623,12 +682,12 @@ def test_namespace_with_schema_only(s3_bucket: str):
] ]
) )
table = db.create_table(table_name, schema=schema, namespace=namespace_path) table = db.create_table(table_name, schema=schema, namespace_path=namespace_path)
# Should have called create_empty_table once # Should have called declare_table once
assert namespace.get_create_call_count() == 1 assert get_declare_call_count(inner_ns_client) == 1
# Should NOT have called describe_table in create mode # Should NOT have called describe_table in create mode
assert namespace.get_describe_call_count() == 0 assert get_describe_call_count(inner_ns_client) == 0
# Verify empty table # Verify empty table
assert table.count_rows() == 0 assert table.count_rows() == 0

View File

@@ -30,6 +30,7 @@ from lancedb.query import (
PhraseQuery, PhraseQuery,
Query, Query,
FullTextSearchQuery, FullTextSearchQuery,
ensure_vector_query,
) )
from lancedb.rerankers.cross_encoder import CrossEncoderReranker from lancedb.rerankers.cross_encoder import CrossEncoderReranker
from lancedb.table import AsyncTable, LanceTable from lancedb.table import AsyncTable, LanceTable
@@ -1501,6 +1502,18 @@ def test_search_empty_table(mem_db):
assert results == [] assert results == []
def test_ensure_vector_query_empty_list():
"""Regression: ensure_vector_query used to return instead of raise ValueError."""
with pytest.raises(ValueError, match="non-empty"):
ensure_vector_query([])
def test_ensure_vector_query_nested_empty_list():
"""Regression: ensure_vector_query used to return instead of raise ValueError."""
with pytest.raises(ValueError, match="non-empty"):
ensure_vector_query([[]])
def test_fast_search(tmp_path): def test_fast_search(tmp_path):
db = lancedb.connect(tmp_path) db = lancedb.connect(tmp_path)

View File

@@ -1201,6 +1201,18 @@ async def test_header_provider_overrides_static_headers():
await db.table_names() await db.table_names()
def test_close():
"""Test that close() works without AttributeError."""
import asyncio
def handler(req):
req.send_response(200)
req.end_headers()
with mock_lancedb_connection(handler) as db:
asyncio.run(db.close())
@pytest.mark.parametrize("exception", [KeyboardInterrupt, SystemExit, GeneratorExit]) @pytest.mark.parametrize("exception", [KeyboardInterrupt, SystemExit, GeneratorExit])
def test_background_loop_cancellation(exception): def test_background_loop_cancellation(exception):
"""Test that BackgroundEventLoop.run() cancels the future on interrupt.""" """Test that BackgroundEventLoop.run() cancels the future on interrupt."""

View File

@@ -326,6 +326,24 @@ def test_add_struct(mem_db: DBConnection):
table = mem_db.create_table("test2", schema=schema) table = mem_db.create_table("test2", schema=schema)
table.add(data) table.add(data)
struct_type = pa.struct(
[
("b", pa.int64()),
("a", pa.int64()),
]
)
expected = pa.table(
{
"s_list": [
[
pa.scalar({"b": 1, "a": 2}, type=struct_type),
pa.scalar({"b": 4, "a": None}, type=struct_type),
]
],
}
)
assert table.to_arrow() == expected
def test_add_subschema(mem_db: DBConnection): def test_add_subschema(mem_db: DBConnection):
schema = pa.schema( schema = pa.schema(
@@ -509,6 +527,132 @@ async def test_add_async(mem_db_async: AsyncConnection):
assert await table.count_rows() == 3 assert await table.count_rows() == 3
def test_add_overwrite_infers_vector_schema(mem_db: DBConnection):
"""Overwrite should infer vector columns the same way create_table does.
Regression test for https://github.com/lancedb/lancedb/issues/3183
"""
table = mem_db.create_table(
"test_overwrite_vec",
data=[
{"vector": [1.0, 2.0, 3.0, 4.0], "item": "foo"},
{"vector": [5.0, 6.0, 7.0, 8.0], "item": "bar"},
],
)
# create_table infers vector as fixed_size_list<float32, 4>
original_type = table.schema.field("vector").type
assert pa.types.is_fixed_size_list(original_type)
# overwrite with plain Python lists (PyArrow infers list<double>)
table.add(
[
{"vector": [10.0, 20.0, 30.0, 40.0], "item": "baz"},
],
mode="overwrite",
)
# overwrite should infer vector column the same way as create_table
new_type = table.schema.field("vector").type
assert pa.types.is_fixed_size_list(new_type), (
f"Expected fixed_size_list after overwrite, got {new_type}"
)
def test_add_progress_callback(mem_db: DBConnection):
table = mem_db.create_table(
"test",
data=[{"id": 1}, {"id": 2}],
)
updates = []
table.add([{"id": 3}, {"id": 4}], progress=lambda p: updates.append(dict(p)))
assert len(table) == 4
# The done callback always fires, so we should always get at least one.
assert len(updates) >= 1, "expected at least one progress callback"
for p in updates:
assert "output_rows" in p
assert "output_bytes" in p
assert "total_rows" in p
assert "elapsed_seconds" in p
assert "active_tasks" in p
assert "total_tasks" in p
assert "done" in p
# The last callback should have done=True.
assert updates[-1]["done"] is True
def test_add_progress_tqdm_like(mem_db: DBConnection):
"""Test that a tqdm-like object gets total set and update() called."""
class FakeBar:
def __init__(self):
self.total = None
self.n = 0
self.postfix = None
def update(self, n):
self.n += n
def set_postfix_str(self, s):
self.postfix = s
def refresh(self):
pass
table = mem_db.create_table(
"test",
data=[{"id": 1}, {"id": 2}],
)
bar = FakeBar()
table.add([{"id": 3}, {"id": 4}], progress=bar)
assert len(table) == 4
# Postfix should contain throughput and worker count
if bar.postfix is not None:
assert "MB/s" in bar.postfix
assert "workers" in bar.postfix
def test_add_progress_bool(mem_db: DBConnection):
"""Test that progress=True creates and closes a tqdm bar automatically."""
table = mem_db.create_table(
"test",
data=[{"id": 1}, {"id": 2}],
)
table.add([{"id": 3}, {"id": 4}], progress=True)
assert len(table) == 4
# progress=False should be the same as None
table.add([{"id": 5}], progress=False)
assert len(table) == 5
@pytest.mark.asyncio
async def test_add_progress_callback_async(mem_db_async: AsyncConnection):
"""Progress callbacks work through the async path too."""
table = await mem_db_async.create_table("test", data=[{"id": 1}, {"id": 2}])
updates = []
await table.add([{"id": 3}, {"id": 4}], progress=lambda p: updates.append(dict(p)))
assert await table.count_rows() == 4
assert len(updates) >= 1
assert updates[-1]["done"] is True
def test_add_progress_callback_error(mem_db: DBConnection):
"""A failing callback must not prevent the write from succeeding."""
table = mem_db.create_table("test", data=[{"id": 1}, {"id": 2}])
def bad_callback(p):
raise RuntimeError("boom")
table.add([{"id": 3}, {"id": 4}], progress=bad_callback)
assert len(table) == 4
def test_polars(mem_db: DBConnection): def test_polars(mem_db: DBConnection):
data = { data = {
"vector": [[3.1, 4.1], [5.9, 26.5]], "vector": [[3.1, 4.1], [5.9, 26.5]],
@@ -1964,7 +2108,7 @@ def test_stats(mem_db: DBConnection):
stats = table.stats() stats = table.stats()
print(f"{stats=}") print(f"{stats=}")
assert stats == { assert stats == {
"total_bytes": 38, "total_bytes": 60,
"num_rows": 2, "num_rows": 2,
"num_indices": 0, "num_indices": 0,
"fragment_stats": { "fragment_stats": {
@@ -2029,3 +2173,33 @@ def test_table_uri(tmp_path):
db = lancedb.connect(tmp_path) db = lancedb.connect(tmp_path)
table = db.create_table("my_table", data=[{"x": 0}]) table = db.create_table("my_table", data=[{"x": 0}])
assert table.uri == str(tmp_path / "my_table.lance") assert table.uri == str(tmp_path / "my_table.lance")
def test_sanitize_data_metadata_not_stripped():
"""Regression test: dict.update() returns None, so assigning its result
would silently replace metadata with None, causing with_metadata(None)
to strip all schema metadata from the target schema."""
from lancedb.table import _sanitize_data
schema = pa.schema(
[pa.field("x", pa.int64())],
metadata={b"existing_key": b"existing_value"},
)
batch = pa.record_batch([pa.array([1, 2, 3])], schema=schema)
# Use a different field type so the reader and target schemas differ,
# forcing _cast_to_target_schema to rebuild the schema with the
# target's metadata (instead of taking the fast-path).
target_schema = pa.schema(
[pa.field("x", pa.int32())],
metadata={b"existing_key": b"existing_value"},
)
reader = pa.RecordBatchReader.from_batches(schema, [batch])
metadata = {b"new_key": b"new_value"}
result = _sanitize_data(reader, target_schema=target_schema, metadata=metadata)
result_schema = result.schema
assert result_schema.metadata is not None
assert result_schema.metadata[b"existing_key"] == b"existing_value"
assert result_schema.metadata[b"new_key"] == b"new_value"

View File

@@ -121,6 +121,32 @@ def test_value_to_sql_string(tmp_path):
assert table.to_pandas().query("search == @value")["replace"].item() == value assert table.to_pandas().query("search == @value")["replace"].item() == value
def test_value_to_sql_dict():
# Simple flat struct
assert value_to_sql({"a": 1, "b": "hello"}) == "named_struct('a', 1, 'b', 'hello')"
# Nested struct
assert (
value_to_sql({"outer": {"inner": 1}})
== "named_struct('outer', named_struct('inner', 1))"
)
# List inside struct
assert value_to_sql({"a": [1, 2]}) == "named_struct('a', [1, 2])"
# Mixed types
assert (
value_to_sql({"name": "test", "count": 42, "rate": 3.14, "active": True})
== "named_struct('name', 'test', 'count', 42, 'rate', 3.14, 'active', TRUE)"
)
# Null value inside struct
assert value_to_sql({"a": None}) == "named_struct('a', NULL)"
# Empty dict
assert value_to_sql({}) == "named_struct()"
def test_append_vector_columns(): def test_append_vector_columns():
registry = EmbeddingFunctionRegistry.get_instance() registry = EmbeddingFunctionRegistry.get_instance()
registry.register("test")(MockTextEmbeddingFunction) registry.register("test")(MockTextEmbeddingFunction)

View File

@@ -10,7 +10,7 @@ use arrow::{
use futures::stream::StreamExt; use futures::stream::StreamExt;
use lancedb::arrow::SendableRecordBatchStream; use lancedb::arrow::SendableRecordBatchStream;
use pyo3::{ use pyo3::{
exceptions::PyStopAsyncIteration, pyclass, pymethods, Bound, Py, PyAny, PyRef, PyResult, Python, Bound, Py, PyAny, PyRef, PyResult, Python, exceptions::PyStopAsyncIteration, pyclass, pymethods,
}; };
use pyo3_async_runtimes::tokio::future_into_py; use pyo3_async_runtimes::tokio::future_into_py;

View File

@@ -9,15 +9,17 @@ use lancedb::{
database::{CreateTableMode, Database, ReadConsistency}, database::{CreateTableMode, Database, ReadConsistency},
}; };
use pyo3::{ use pyo3::{
Bound, FromPyObject, Py, PyAny, PyRef, PyResult, Python,
exceptions::{PyRuntimeError, PyValueError}, exceptions::{PyRuntimeError, PyValueError},
pyclass, pyfunction, pymethods, pyclass, pyfunction, pymethods,
types::{PyDict, PyDictMethods}, types::{PyDict, PyDictMethods},
Bound, FromPyObject, Py, PyAny, PyRef, PyResult, Python,
}; };
use pyo3_async_runtimes::tokio::future_into_py; use pyo3_async_runtimes::tokio::future_into_py;
use crate::{ use crate::{
error::PythonErrorExt, storage_options::py_object_to_storage_options_provider, table::Table, error::PythonErrorExt,
namespace::{create_namespace_storage_options_provider, extract_namespace_arc},
table::Table,
}; };
#[pyclass] #[pyclass]
@@ -86,16 +88,16 @@ impl Connection {
}) })
} }
#[pyo3(signature = (namespace=vec![], start_after=None, limit=None))] #[pyo3(signature = (namespace_path=None, start_after=None, limit=None))]
pub fn table_names( pub fn table_names(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
start_after: Option<String>, start_after: Option<String>,
limit: Option<u32>, limit: Option<u32>,
) -> PyResult<Bound<'_, PyAny>> { ) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.get_inner()?.clone(); let inner = self_.get_inner()?.clone();
let mut op = inner.table_names(); let mut op = inner.table_names();
op = op.namespace(namespace); op = op.namespace(namespace_path.unwrap_or_default());
if let Some(start_after) = start_after { if let Some(start_after) = start_after {
op = op.start_after(start_after); op = op.start_after(start_after);
} }
@@ -106,34 +108,43 @@ impl Connection {
} }
#[allow(clippy::too_many_arguments)] #[allow(clippy::too_many_arguments)]
#[pyo3(signature = (name, mode, data, namespace=vec![], storage_options=None, storage_options_provider=None, location=None))] #[pyo3(signature = (name, mode, data, namespace_path=None, storage_options=None, location=None, namespace_client=None))]
pub fn create_table<'a>( pub fn create_table<'a>(
self_: PyRef<'a, Self>, self_: PyRef<'a, Self>,
name: String, name: String,
mode: &str, mode: &str,
data: Bound<'_, PyAny>, data: Bound<'_, PyAny>,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
storage_options: Option<HashMap<String, String>>, storage_options: Option<HashMap<String, String>>,
storage_options_provider: Option<Py<PyAny>>,
location: Option<String>, location: Option<String>,
namespace_client: Option<Py<PyAny>>,
) -> PyResult<Bound<'a, PyAny>> { ) -> PyResult<Bound<'a, PyAny>> {
let inner = self_.get_inner()?.clone(); let inner = self_.get_inner()?.clone();
let py = self_.py();
let mode = Self::parse_create_mode_str(mode)?; let mode = Self::parse_create_mode_str(mode)?;
let batches: Box<dyn arrow::array::RecordBatchReader + Send> = let batches: Box<dyn arrow::array::RecordBatchReader + Send> =
Box::new(ArrowArrayStreamReader::from_pyarrow_bound(&data)?); Box::new(ArrowArrayStreamReader::from_pyarrow_bound(&data)?);
let mut builder = inner.create_table(name, batches).mode(mode); let ns_path = namespace_path.clone().unwrap_or_default();
let mut builder = inner.create_table(name.clone(), batches).mode(mode);
builder = builder.namespace(namespace); builder = builder.namespace(ns_path.clone());
if let Some(storage_options) = storage_options { if let Some(storage_options) = storage_options {
builder = builder.storage_options(storage_options); builder = builder.storage_options(storage_options);
} }
if let Some(provider_obj) = storage_options_provider {
let provider = py_object_to_storage_options_provider(provider_obj)?; // Auto-create storage options provider from namespace_client
if let Some(ns_obj) = namespace_client {
let ns_client = extract_namespace_arc(py, ns_obj)?;
// Create table_id by combining namespace_path with table name
let mut table_id = ns_path;
table_id.push(name);
let provider = create_namespace_storage_options_provider(ns_client, table_id);
builder = builder.storage_options_provider(provider); builder = builder.storage_options_provider(provider);
} }
if let Some(location) = location { if let Some(location) = location {
builder = builder.location(location); builder = builder.location(location);
} }
@@ -145,33 +156,44 @@ impl Connection {
} }
#[allow(clippy::too_many_arguments)] #[allow(clippy::too_many_arguments)]
#[pyo3(signature = (name, mode, schema, namespace=vec![], storage_options=None, storage_options_provider=None, location=None))] #[pyo3(signature = (name, mode, schema, namespace_path=None, storage_options=None, location=None, namespace_client=None))]
pub fn create_empty_table<'a>( pub fn create_empty_table<'a>(
self_: PyRef<'a, Self>, self_: PyRef<'a, Self>,
name: String, name: String,
mode: &str, mode: &str,
schema: Bound<'_, PyAny>, schema: Bound<'_, PyAny>,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
storage_options: Option<HashMap<String, String>>, storage_options: Option<HashMap<String, String>>,
storage_options_provider: Option<Py<PyAny>>,
location: Option<String>, location: Option<String>,
namespace_client: Option<Py<PyAny>>,
) -> PyResult<Bound<'a, PyAny>> { ) -> PyResult<Bound<'a, PyAny>> {
let inner = self_.get_inner()?.clone(); let inner = self_.get_inner()?.clone();
let py = self_.py();
let mode = Self::parse_create_mode_str(mode)?; let mode = Self::parse_create_mode_str(mode)?;
let schema = Schema::from_pyarrow_bound(&schema)?; let schema = Schema::from_pyarrow_bound(&schema)?;
let mut builder = inner.create_empty_table(name, Arc::new(schema)).mode(mode); let ns_path = namespace_path.clone().unwrap_or_default();
let mut builder = inner
.create_empty_table(name.clone(), Arc::new(schema))
.mode(mode);
builder = builder.namespace(namespace); builder = builder.namespace(ns_path.clone());
if let Some(storage_options) = storage_options { if let Some(storage_options) = storage_options {
builder = builder.storage_options(storage_options); builder = builder.storage_options(storage_options);
} }
if let Some(provider_obj) = storage_options_provider {
let provider = py_object_to_storage_options_provider(provider_obj)?; // Auto-create storage options provider from namespace_client
if let Some(ns_obj) = namespace_client {
let ns_client = extract_namespace_arc(py, ns_obj)?;
// Create table_id by combining namespace_path with table name
let mut table_id = ns_path;
table_id.push(name);
let provider = create_namespace_storage_options_provider(ns_client, table_id);
builder = builder.storage_options_provider(provider); builder = builder.storage_options_provider(provider);
} }
if let Some(location) = location { if let Some(location) = location {
builder = builder.location(location); builder = builder.location(location);
} }
@@ -182,33 +204,49 @@ impl Connection {
}) })
} }
#[pyo3(signature = (name, namespace=vec![], storage_options = None, storage_options_provider=None, index_cache_size = None, location=None))] #[allow(clippy::too_many_arguments)]
#[pyo3(signature = (name, namespace_path=None, storage_options=None, index_cache_size=None, location=None, namespace_client=None, managed_versioning=None))]
pub fn open_table( pub fn open_table(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
name: String, name: String,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
storage_options: Option<HashMap<String, String>>, storage_options: Option<HashMap<String, String>>,
storage_options_provider: Option<Py<PyAny>>,
index_cache_size: Option<u32>, index_cache_size: Option<u32>,
location: Option<String>, location: Option<String>,
namespace_client: Option<Py<PyAny>>,
managed_versioning: Option<bool>,
) -> PyResult<Bound<'_, PyAny>> { ) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.get_inner()?.clone(); let inner = self_.get_inner()?.clone();
let py = self_.py();
let mut builder = inner.open_table(name); let ns_path = namespace_path.clone().unwrap_or_default();
builder = builder.namespace(namespace); let mut builder = inner.open_table(name.clone());
builder = builder.namespace(ns_path.clone());
if let Some(storage_options) = storage_options { if let Some(storage_options) = storage_options {
builder = builder.storage_options(storage_options); builder = builder.storage_options(storage_options);
} }
if let Some(provider_obj) = storage_options_provider {
let provider = py_object_to_storage_options_provider(provider_obj)?; // Auto-create storage options provider from namespace_client
if let Some(ns_obj) = namespace_client {
let ns_client = extract_namespace_arc(py, ns_obj)?;
// Create table_id by combining namespace_path with table name
let mut table_id = ns_path;
table_id.push(name);
let provider = create_namespace_storage_options_provider(ns_client.clone(), table_id);
builder = builder.storage_options_provider(provider); builder = builder.storage_options_provider(provider);
builder = builder.namespace_client(ns_client);
} }
if let Some(index_cache_size) = index_cache_size { if let Some(index_cache_size) = index_cache_size {
builder = builder.index_cache_size(index_cache_size); builder = builder.index_cache_size(index_cache_size);
} }
if let Some(location) = location { if let Some(location) = location {
builder = builder.location(location); builder = builder.location(location);
} }
// Pass managed_versioning if provided to avoid redundant describe_table call
if let Some(enabled) = managed_versioning {
builder = builder.managed_versioning(enabled);
}
future_into_py(self_.py(), async move { future_into_py(self_.py(), async move {
let table = builder.execute().await.infer_error()?; let table = builder.execute().await.infer_error()?;
@@ -216,12 +254,12 @@ impl Connection {
}) })
} }
#[pyo3(signature = (target_table_name, source_uri, target_namespace=vec![], source_version=None, source_tag=None, is_shallow=true))] #[pyo3(signature = (target_table_name, source_uri, target_namespace_path=None, source_version=None, source_tag=None, is_shallow=true))]
pub fn clone_table( pub fn clone_table(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
target_table_name: String, target_table_name: String,
source_uri: String, source_uri: String,
target_namespace: Vec<String>, target_namespace_path: Option<Vec<String>>,
source_version: Option<u64>, source_version: Option<u64>,
source_tag: Option<String>, source_tag: Option<String>,
is_shallow: bool, is_shallow: bool,
@@ -229,7 +267,7 @@ impl Connection {
let inner = self_.get_inner()?.clone(); let inner = self_.get_inner()?.clone();
let mut builder = inner.clone_table(target_table_name, source_uri); let mut builder = inner.clone_table(target_table_name, source_uri);
builder = builder.target_namespace(target_namespace); builder = builder.target_namespace(target_namespace_path.unwrap_or_default());
if let Some(version) = source_version { if let Some(version) = source_version {
builder = builder.source_version(version); builder = builder.source_version(version);
} }
@@ -244,52 +282,56 @@ impl Connection {
}) })
} }
#[pyo3(signature = (cur_name, new_name, cur_namespace=vec![], new_namespace=vec![]))] #[pyo3(signature = (cur_name, new_name, cur_namespace_path=None, new_namespace_path=None))]
pub fn rename_table( pub fn rename_table(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
cur_name: String, cur_name: String,
new_name: String, new_name: String,
cur_namespace: Vec<String>, cur_namespace_path: Option<Vec<String>>,
new_namespace: Vec<String>, new_namespace_path: Option<Vec<String>>,
) -> PyResult<Bound<'_, PyAny>> { ) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.get_inner()?.clone(); let inner = self_.get_inner()?.clone();
let cur_ns_path = cur_namespace_path.unwrap_or_default();
let new_ns_path = new_namespace_path.unwrap_or_default();
future_into_py(self_.py(), async move { future_into_py(self_.py(), async move {
inner inner
.rename_table(cur_name, new_name, &cur_namespace, &new_namespace) .rename_table(cur_name, new_name, &cur_ns_path, &new_ns_path)
.await .await
.infer_error() .infer_error()
}) })
} }
#[pyo3(signature = (name, namespace=vec![]))] #[pyo3(signature = (name, namespace_path=None))]
pub fn drop_table( pub fn drop_table(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
name: String, name: String,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
) -> PyResult<Bound<'_, PyAny>> { ) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.get_inner()?.clone(); let inner = self_.get_inner()?.clone();
let ns_path = namespace_path.unwrap_or_default();
future_into_py(self_.py(), async move { future_into_py(self_.py(), async move {
inner.drop_table(name, &namespace).await.infer_error() inner.drop_table(name, &ns_path).await.infer_error()
}) })
} }
#[pyo3(signature = (namespace=vec![],))] #[pyo3(signature = (namespace_path=None,))]
pub fn drop_all_tables( pub fn drop_all_tables(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
) -> PyResult<Bound<'_, PyAny>> { ) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.get_inner()?.clone(); let inner = self_.get_inner()?.clone();
let ns_path = namespace_path.unwrap_or_default();
future_into_py(self_.py(), async move { future_into_py(self_.py(), async move {
inner.drop_all_tables(&namespace).await.infer_error() inner.drop_all_tables(&ns_path).await.infer_error()
}) })
} }
// Namespace management methods // Namespace management methods
#[pyo3(signature = (namespace=vec![], page_token=None, limit=None))] #[pyo3(signature = (namespace_path=None, page_token=None, limit=None))]
pub fn list_namespaces( pub fn list_namespaces(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
page_token: Option<String>, page_token: Option<String>,
limit: Option<u32>, limit: Option<u32>,
) -> PyResult<Bound<'_, PyAny>> { ) -> PyResult<Bound<'_, PyAny>> {
@@ -298,11 +340,7 @@ impl Connection {
future_into_py(py, async move { future_into_py(py, async move {
use lance_namespace::models::ListNamespacesRequest; use lance_namespace::models::ListNamespacesRequest;
let request = ListNamespacesRequest { let request = ListNamespacesRequest {
id: if namespace.is_empty() { id: namespace_path,
None
} else {
Some(namespace)
},
page_token, page_token,
limit: limit.map(|l| l as i32), limit: limit.map(|l| l as i32),
..Default::default() ..Default::default()
@@ -317,10 +355,10 @@ impl Connection {
}) })
} }
#[pyo3(signature = (namespace, mode=None, properties=None))] #[pyo3(signature = (namespace_path, mode=None, properties=None))]
pub fn create_namespace( pub fn create_namespace(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
namespace: Vec<String>, namespace_path: Vec<String>,
mode: Option<String>, mode: Option<String>,
properties: Option<std::collections::HashMap<String, String>>, properties: Option<std::collections::HashMap<String, String>>,
) -> PyResult<Bound<'_, PyAny>> { ) -> PyResult<Bound<'_, PyAny>> {
@@ -336,11 +374,7 @@ impl Connection {
_ => None, _ => None,
}); });
let request = CreateNamespaceRequest { let request = CreateNamespaceRequest {
id: if namespace.is_empty() { id: Some(namespace_path),
None
} else {
Some(namespace)
},
mode: mode_str, mode: mode_str,
properties, properties,
..Default::default() ..Default::default()
@@ -354,10 +388,10 @@ impl Connection {
}) })
} }
#[pyo3(signature = (namespace, mode=None, behavior=None))] #[pyo3(signature = (namespace_path, mode=None, behavior=None))]
pub fn drop_namespace( pub fn drop_namespace(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
namespace: Vec<String>, namespace_path: Vec<String>,
mode: Option<String>, mode: Option<String>,
behavior: Option<String>, behavior: Option<String>,
) -> PyResult<Bound<'_, PyAny>> { ) -> PyResult<Bound<'_, PyAny>> {
@@ -377,11 +411,7 @@ impl Connection {
_ => None, _ => None,
}); });
let request = DropNamespaceRequest { let request = DropNamespaceRequest {
id: if namespace.is_empty() { id: Some(namespace_path),
None
} else {
Some(namespace)
},
mode: mode_str, mode: mode_str,
behavior: behavior_str, behavior: behavior_str,
..Default::default() ..Default::default()
@@ -396,21 +426,17 @@ impl Connection {
}) })
} }
#[pyo3(signature = (namespace,))] #[pyo3(signature = (namespace_path,))]
pub fn describe_namespace( pub fn describe_namespace(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
namespace: Vec<String>, namespace_path: Vec<String>,
) -> PyResult<Bound<'_, PyAny>> { ) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.get_inner()?.clone(); let inner = self_.get_inner()?.clone();
let py = self_.py(); let py = self_.py();
future_into_py(py, async move { future_into_py(py, async move {
use lance_namespace::models::DescribeNamespaceRequest; use lance_namespace::models::DescribeNamespaceRequest;
let request = DescribeNamespaceRequest { let request = DescribeNamespaceRequest {
id: if namespace.is_empty() { id: Some(namespace_path),
None
} else {
Some(namespace)
},
..Default::default() ..Default::default()
}; };
let response = inner.describe_namespace(request).await.infer_error()?; let response = inner.describe_namespace(request).await.infer_error()?;
@@ -422,10 +448,10 @@ impl Connection {
}) })
} }
#[pyo3(signature = (namespace=vec![], page_token=None, limit=None))] #[pyo3(signature = (namespace_path=None, page_token=None, limit=None))]
pub fn list_tables( pub fn list_tables(
self_: PyRef<'_, Self>, self_: PyRef<'_, Self>,
namespace: Vec<String>, namespace_path: Option<Vec<String>>,
page_token: Option<String>, page_token: Option<String>,
limit: Option<u32>, limit: Option<u32>,
) -> PyResult<Bound<'_, PyAny>> { ) -> PyResult<Bound<'_, PyAny>> {
@@ -434,11 +460,7 @@ impl Connection {
future_into_py(py, async move { future_into_py(py, async move {
use lance_namespace::models::ListTablesRequest; use lance_namespace::models::ListTablesRequest;
let request = ListTablesRequest { let request = ListTablesRequest {
id: if namespace.is_empty() { id: namespace_path,
None
} else {
Some(namespace)
},
page_token, page_token,
limit: limit.map(|l| l as i32), limit: limit.map(|l| l as i32),
..Default::default() ..Default::default()

View File

@@ -2,10 +2,10 @@
// SPDX-FileCopyrightText: Copyright The LanceDB Authors // SPDX-FileCopyrightText: Copyright The LanceDB Authors
use pyo3::{ use pyo3::{
PyErr, PyResult, Python,
exceptions::{PyIOError, PyNotImplementedError, PyOSError, PyRuntimeError, PyValueError}, exceptions::{PyIOError, PyNotImplementedError, PyOSError, PyRuntimeError, PyValueError},
intern, intern,
types::{PyAnyMethods, PyNone}, types::{PyAnyMethods, PyNone},
PyErr, PyResult, Python,
}; };
use lancedb::error::Error as LanceError; use lancedb::error::Error as LanceError;

175
python/src/expr.rs Normal file
View File

@@ -0,0 +1,175 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
//! PyO3 bindings for the LanceDB expression builder API.
//!
//! This module exposes [`PyExpr`] and helper free functions so Python can
//! build type-safe filter / projection expressions that map directly to
//! DataFusion [`Expr`] nodes, bypassing SQL string parsing.
use arrow::{datatypes::DataType, pyarrow::PyArrowType};
use lancedb::expr::{DfExpr, col as ldb_col, contains, expr_cast, lit as df_lit, lower, upper};
use pyo3::{Bound, PyAny, PyResult, exceptions::PyValueError, prelude::*, pyfunction};
/// A type-safe DataFusion expression.
///
/// Instances are constructed via the free functions [`expr_col`] and
/// [`expr_lit`] and combined with the methods on this struct. On the Python
/// side a thin wrapper class (`lancedb.expr.Expr`) delegates to these methods
/// and adds Python operator overloads.
#[pyclass(name = "PyExpr")]
#[derive(Clone)]
pub struct PyExpr(pub DfExpr);
#[pymethods]
impl PyExpr {
// ── comparisons ──────────────────────────────────────────────────────────
fn eq(&self, other: &Self) -> Self {
Self(self.0.clone().eq(other.0.clone()))
}
fn ne(&self, other: &Self) -> Self {
Self(self.0.clone().not_eq(other.0.clone()))
}
fn lt(&self, other: &Self) -> Self {
Self(self.0.clone().lt(other.0.clone()))
}
fn lte(&self, other: &Self) -> Self {
Self(self.0.clone().lt_eq(other.0.clone()))
}
fn gt(&self, other: &Self) -> Self {
Self(self.0.clone().gt(other.0.clone()))
}
fn gte(&self, other: &Self) -> Self {
Self(self.0.clone().gt_eq(other.0.clone()))
}
// ── logical ──────────────────────────────────────────────────────────────
fn and_(&self, other: &Self) -> Self {
Self(self.0.clone().and(other.0.clone()))
}
fn or_(&self, other: &Self) -> Self {
Self(self.0.clone().or(other.0.clone()))
}
fn not_(&self) -> Self {
use std::ops::Not;
Self(self.0.clone().not())
}
// ── arithmetic ───────────────────────────────────────────────────────────
fn add(&self, other: &Self) -> Self {
use std::ops::Add;
Self(self.0.clone().add(other.0.clone()))
}
fn sub(&self, other: &Self) -> Self {
use std::ops::Sub;
Self(self.0.clone().sub(other.0.clone()))
}
fn mul(&self, other: &Self) -> Self {
use std::ops::Mul;
Self(self.0.clone().mul(other.0.clone()))
}
fn div(&self, other: &Self) -> Self {
use std::ops::Div;
Self(self.0.clone().div(other.0.clone()))
}
// ── string functions ─────────────────────────────────────────────────────
/// Convert string column to lowercase.
fn lower(&self) -> Self {
Self(lower(self.0.clone()))
}
/// Convert string column to uppercase.
fn upper(&self) -> Self {
Self(upper(self.0.clone()))
}
/// Test whether the string contains `substr`.
fn contains(&self, substr: &Self) -> Self {
Self(contains(self.0.clone(), substr.0.clone()))
}
// ── type cast ────────────────────────────────────────────────────────────
/// Cast the expression to `data_type`.
///
/// `data_type` must be a PyArrow `DataType` (e.g. `pa.int32()`).
/// On the Python side, `lancedb.expr.Expr.cast` also accepts type name
/// strings via `pa.lib.ensure_type` before forwarding here.
fn cast(&self, data_type: PyArrowType<DataType>) -> Self {
Self(expr_cast(self.0.clone(), data_type.0))
}
// ── utilities ────────────────────────────────────────────────────────────
/// Render the expression as a SQL string (useful for debugging).
fn to_sql(&self) -> PyResult<String> {
lancedb::expr::expr_to_sql_string(&self.0).map_err(|e| PyValueError::new_err(e.to_string()))
}
fn __repr__(&self) -> PyResult<String> {
let sql =
lancedb::expr::expr_to_sql_string(&self.0).unwrap_or_else(|_| "<expr>".to_string());
Ok(format!("PyExpr({})", sql))
}
}
// ── free functions ────────────────────────────────────────────────────────────
/// Create a column reference expression.
///
/// The column name is preserved exactly as given (case-sensitive), so
/// `col("firstName")` correctly references a field named `firstName`.
#[pyfunction]
pub fn expr_col(name: &str) -> PyExpr {
PyExpr(ldb_col(name))
}
/// Create a literal value expression.
///
/// Supported Python types: `bool`, `int`, `float`, `str`.
#[pyfunction]
pub fn expr_lit(value: Bound<'_, PyAny>) -> PyResult<PyExpr> {
// bool must be checked before int because bool is a subclass of int in Python
if let Ok(b) = value.extract::<bool>() {
return Ok(PyExpr(df_lit(b)));
}
if let Ok(i) = value.extract::<i64>() {
return Ok(PyExpr(df_lit(i)));
}
if let Ok(f) = value.extract::<f64>() {
return Ok(PyExpr(df_lit(f)));
}
if let Ok(s) = value.extract::<String>() {
return Ok(PyExpr(df_lit(s)));
}
Err(PyValueError::new_err(format!(
"unsupported literal type: {}. Supported: bool, int, float, str",
value.get_type().name()?
)))
}
/// Call an arbitrary registered SQL function by name.
///
/// See `lancedb::expr::func` for the list of supported function names.
#[pyfunction]
pub fn expr_func(name: &str, args: Vec<PyExpr>) -> PyResult<PyExpr> {
let df_args: Vec<DfExpr> = args.into_iter().map(|e| e.0).collect();
lancedb::expr::func(name, df_args)
.map(PyExpr)
.map_err(|e| PyValueError::new_err(e.to_string()))
}

View File

@@ -3,17 +3,17 @@
use lancedb::index::vector::{IvfFlatIndexBuilder, IvfRqIndexBuilder, IvfSqIndexBuilder}; use lancedb::index::vector::{IvfFlatIndexBuilder, IvfRqIndexBuilder, IvfSqIndexBuilder};
use lancedb::index::{ use lancedb::index::{
Index as LanceDbIndex,
scalar::{BTreeIndexBuilder, FtsIndexBuilder}, scalar::{BTreeIndexBuilder, FtsIndexBuilder},
vector::{IvfHnswPqIndexBuilder, IvfHnswSqIndexBuilder, IvfPqIndexBuilder}, vector::{IvfHnswPqIndexBuilder, IvfHnswSqIndexBuilder, IvfPqIndexBuilder},
Index as LanceDbIndex,
}; };
use pyo3::types::PyStringMethods;
use pyo3::IntoPyObject; use pyo3::IntoPyObject;
use pyo3::types::PyStringMethods;
use pyo3::{ use pyo3::{
Bound, FromPyObject, PyAny, PyResult, Python,
exceptions::{PyKeyError, PyValueError}, exceptions::{PyKeyError, PyValueError},
intern, pyclass, pymethods, intern, pyclass, pymethods,
types::PyAnyMethods, types::PyAnyMethods,
Bound, FromPyObject, PyAny, PyResult, Python,
}; };
use crate::util::parse_distance_type; use crate::util::parse_distance_type;
@@ -41,7 +41,12 @@ pub fn extract_index_params(source: &Option<Bound<'_, PyAny>>) -> PyResult<Lance
let inner_opts = FtsIndexBuilder::default() let inner_opts = FtsIndexBuilder::default()
.base_tokenizer(params.base_tokenizer) .base_tokenizer(params.base_tokenizer)
.language(&params.language) .language(&params.language)
.map_err(|_| PyValueError::new_err(format!("LanceDB does not support the requested language: '{}'", params.language)))? .map_err(|_| {
PyValueError::new_err(format!(
"LanceDB does not support the requested language: '{}'",
params.language
))
})?
.with_position(params.with_position) .with_position(params.with_position)
.lower_case(params.lower_case) .lower_case(params.lower_case)
.max_token_length(params.max_token_length) .max_token_length(params.max_token_length)
@@ -52,7 +57,7 @@ pub fn extract_index_params(source: &Option<Bound<'_, PyAny>>) -> PyResult<Lance
.ngram_max_length(params.ngram_max_length) .ngram_max_length(params.ngram_max_length)
.ngram_prefix_only(params.prefix_only); .ngram_prefix_only(params.prefix_only);
Ok(LanceDbIndex::FTS(inner_opts)) Ok(LanceDbIndex::FTS(inner_opts))
}, }
"IvfFlat" => { "IvfFlat" => {
let params = source.extract::<IvfFlatParams>()?; let params = source.extract::<IvfFlatParams>()?;
let distance_type = parse_distance_type(params.distance_type)?; let distance_type = parse_distance_type(params.distance_type)?;
@@ -64,10 +69,11 @@ pub fn extract_index_params(source: &Option<Bound<'_, PyAny>>) -> PyResult<Lance
ivf_flat_builder = ivf_flat_builder.num_partitions(num_partitions); ivf_flat_builder = ivf_flat_builder.num_partitions(num_partitions);
} }
if let Some(target_partition_size) = params.target_partition_size { if let Some(target_partition_size) = params.target_partition_size {
ivf_flat_builder = ivf_flat_builder.target_partition_size(target_partition_size); ivf_flat_builder =
ivf_flat_builder.target_partition_size(target_partition_size);
} }
Ok(LanceDbIndex::IvfFlat(ivf_flat_builder)) Ok(LanceDbIndex::IvfFlat(ivf_flat_builder))
}, }
"IvfPq" => { "IvfPq" => {
let params = source.extract::<IvfPqParams>()?; let params = source.extract::<IvfPqParams>()?;
let distance_type = parse_distance_type(params.distance_type)?; let distance_type = parse_distance_type(params.distance_type)?;
@@ -86,7 +92,7 @@ pub fn extract_index_params(source: &Option<Bound<'_, PyAny>>) -> PyResult<Lance
ivf_pq_builder = ivf_pq_builder.num_sub_vectors(num_sub_vectors); ivf_pq_builder = ivf_pq_builder.num_sub_vectors(num_sub_vectors);
} }
Ok(LanceDbIndex::IvfPq(ivf_pq_builder)) Ok(LanceDbIndex::IvfPq(ivf_pq_builder))
}, }
"IvfSq" => { "IvfSq" => {
let params = source.extract::<IvfSqParams>()?; let params = source.extract::<IvfSqParams>()?;
let distance_type = parse_distance_type(params.distance_type)?; let distance_type = parse_distance_type(params.distance_type)?;
@@ -101,7 +107,7 @@ pub fn extract_index_params(source: &Option<Bound<'_, PyAny>>) -> PyResult<Lance
ivf_sq_builder = ivf_sq_builder.target_partition_size(target_partition_size); ivf_sq_builder = ivf_sq_builder.target_partition_size(target_partition_size);
} }
Ok(LanceDbIndex::IvfSq(ivf_sq_builder)) Ok(LanceDbIndex::IvfSq(ivf_sq_builder))
}, }
"IvfRq" => { "IvfRq" => {
let params = source.extract::<IvfRqParams>()?; let params = source.extract::<IvfRqParams>()?;
let distance_type = parse_distance_type(params.distance_type)?; let distance_type = parse_distance_type(params.distance_type)?;
@@ -117,7 +123,7 @@ pub fn extract_index_params(source: &Option<Bound<'_, PyAny>>) -> PyResult<Lance
ivf_rq_builder = ivf_rq_builder.target_partition_size(target_partition_size); ivf_rq_builder = ivf_rq_builder.target_partition_size(target_partition_size);
} }
Ok(LanceDbIndex::IvfRq(ivf_rq_builder)) Ok(LanceDbIndex::IvfRq(ivf_rq_builder))
}, }
"HnswPq" => { "HnswPq" => {
let params = source.extract::<IvfHnswPqParams>()?; let params = source.extract::<IvfHnswPqParams>()?;
let distance_type = parse_distance_type(params.distance_type)?; let distance_type = parse_distance_type(params.distance_type)?;
@@ -138,7 +144,7 @@ pub fn extract_index_params(source: &Option<Bound<'_, PyAny>>) -> PyResult<Lance
hnsw_pq_builder = hnsw_pq_builder.num_sub_vectors(num_sub_vectors); hnsw_pq_builder = hnsw_pq_builder.num_sub_vectors(num_sub_vectors);
} }
Ok(LanceDbIndex::IvfHnswPq(hnsw_pq_builder)) Ok(LanceDbIndex::IvfHnswPq(hnsw_pq_builder))
}, }
"HnswSq" => { "HnswSq" => {
let params = source.extract::<IvfHnswSqParams>()?; let params = source.extract::<IvfHnswSqParams>()?;
let distance_type = parse_distance_type(params.distance_type)?; let distance_type = parse_distance_type(params.distance_type)?;
@@ -155,7 +161,7 @@ pub fn extract_index_params(source: &Option<Bound<'_, PyAny>>) -> PyResult<Lance
hnsw_sq_builder = hnsw_sq_builder.target_partition_size(target_partition_size); hnsw_sq_builder = hnsw_sq_builder.target_partition_size(target_partition_size);
} }
Ok(LanceDbIndex::IvfHnswSq(hnsw_sq_builder)) Ok(LanceDbIndex::IvfHnswSq(hnsw_sq_builder))
}, }
not_supported => Err(PyValueError::new_err(format!( not_supported => Err(PyValueError::new_err(format!(
"Invalid index type '{}'. Must be one of BTree, Bitmap, LabelList, FTS, IvfPq, IvfSq, IvfHnswPq, or IvfHnswSq", "Invalid index type '{}'. Must be one of BTree, Bitmap, LabelList, FTS, IvfPq, IvfSq, IvfHnswPq, or IvfHnswSq",
not_supported not_supported

View File

@@ -2,14 +2,15 @@
// SPDX-FileCopyrightText: Copyright The LanceDB Authors // SPDX-FileCopyrightText: Copyright The LanceDB Authors
use arrow::RecordBatchStream; use arrow::RecordBatchStream;
use connection::{connect, Connection}; use connection::{Connection, connect};
use env_logger::Env; use env_logger::Env;
use expr::{PyExpr, expr_col, expr_func, expr_lit};
use index::IndexConfig; use index::IndexConfig;
use permutation::{PyAsyncPermutationBuilder, PyPermutationReader}; use permutation::{PyAsyncPermutationBuilder, PyPermutationReader};
use pyo3::{ use pyo3::{
pymodule, Bound, PyResult, Python, pymodule,
types::{PyModule, PyModuleMethods}, types::{PyModule, PyModuleMethods},
wrap_pyfunction, Bound, PyResult, Python, wrap_pyfunction,
}; };
use query::{FTSQuery, HybridQuery, Query, VectorQuery}; use query::{FTSQuery, HybridQuery, Query, VectorQuery};
use session::Session; use session::Session;
@@ -21,12 +22,13 @@ use table::{
pub mod arrow; pub mod arrow;
pub mod connection; pub mod connection;
pub mod error; pub mod error;
pub mod expr;
pub mod header; pub mod header;
pub mod index; pub mod index;
pub mod namespace;
pub mod permutation; pub mod permutation;
pub mod query; pub mod query;
pub mod session; pub mod session;
pub mod storage_options;
pub mod table; pub mod table;
pub mod util; pub mod util;
@@ -54,10 +56,14 @@ pub fn _lancedb(_py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_class::<UpdateResult>()?; m.add_class::<UpdateResult>()?;
m.add_class::<PyAsyncPermutationBuilder>()?; m.add_class::<PyAsyncPermutationBuilder>()?;
m.add_class::<PyPermutationReader>()?; m.add_class::<PyPermutationReader>()?;
m.add_class::<PyExpr>()?;
m.add_function(wrap_pyfunction!(connect, m)?)?; m.add_function(wrap_pyfunction!(connect, m)?)?;
m.add_function(wrap_pyfunction!(permutation::async_permutation_builder, m)?)?; m.add_function(wrap_pyfunction!(permutation::async_permutation_builder, m)?)?;
m.add_function(wrap_pyfunction!(util::validate_table_name, m)?)?; m.add_function(wrap_pyfunction!(util::validate_table_name, m)?)?;
m.add_function(wrap_pyfunction!(query::fts_query_to_json, m)?)?; m.add_function(wrap_pyfunction!(query::fts_query_to_json, m)?)?;
m.add_function(wrap_pyfunction!(expr_col, m)?)?;
m.add_function(wrap_pyfunction!(expr_lit, m)?)?;
m.add_function(wrap_pyfunction!(expr_func, m)?)?;
m.add("__version__", env!("CARGO_PKG_VERSION"))?; m.add("__version__", env!("CARGO_PKG_VERSION"))?;
Ok(()) Ok(())
} }

715
python/src/namespace.rs Normal file
View File

@@ -0,0 +1,715 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
//! Namespace utilities for Python bindings
use std::collections::HashMap;
use std::sync::Arc;
use async_trait::async_trait;
use bytes::Bytes;
use lance_io::object_store::{LanceNamespaceStorageOptionsProvider, StorageOptionsProvider};
use lance_namespace::LanceNamespace as LanceNamespaceTrait;
use lance_namespace::models::*;
use pyo3::prelude::*;
use pyo3::types::PyDict;
/// Wrapper that allows any Python object implementing LanceNamespace protocol
/// to be used as a Rust LanceNamespace.
///
/// This is similar to PyLanceNamespace in lance's Python bindings - it wraps a Python
/// object and calls back into Python when namespace methods are invoked.
pub struct PyLanceNamespace {
py_namespace: Arc<Py<PyAny>>,
namespace_id: String,
}
impl PyLanceNamespace {
/// Create a new PyLanceNamespace wrapper around a Python namespace object.
pub fn new(_py: Python<'_>, py_namespace: &Bound<'_, PyAny>) -> PyResult<Self> {
let namespace_id = py_namespace
.call_method0("namespace_id")?
.extract::<String>()?;
Ok(Self {
py_namespace: Arc::new(py_namespace.clone().unbind()),
namespace_id,
})
}
/// Create an Arc<dyn LanceNamespace> from a Python namespace object.
pub fn create_arc(
py: Python<'_>,
py_namespace: &Bound<'_, PyAny>,
) -> PyResult<Arc<dyn LanceNamespaceTrait>> {
let wrapper = Self::new(py, py_namespace)?;
Ok(Arc::new(wrapper))
}
}
impl std::fmt::Debug for PyLanceNamespace {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "PyLanceNamespace {{ id: {} }}", self.namespace_id)
}
}
/// Get or create the DictWithModelDump class in Python.
/// This class acts like a dict but also has model_dump() method.
/// This allows it to work with both:
/// - depythonize (which expects a dict/Mapping)
/// - Python code that calls .model_dump() (like DirectoryNamespace wrapper)
fn get_dict_with_model_dump_class(py: Python<'_>) -> PyResult<Bound<'_, PyAny>> {
// Use a module-level cache via __builtins__
let builtins = py.import("builtins")?;
if builtins.hasattr("_DictWithModelDump")? {
return builtins.getattr("_DictWithModelDump");
}
// Create the class using exec
let locals = PyDict::new(py);
py.run(
c"class DictWithModelDump(dict):
def model_dump(self):
return dict(self)",
None,
Some(&locals),
)?;
let class = locals.get_item("DictWithModelDump")?.ok_or_else(|| {
pyo3::exceptions::PyRuntimeError::new_err("Failed to create DictWithModelDump class")
})?;
// Cache it
builtins.setattr("_DictWithModelDump", &class)?;
Ok(class)
}
/// Helper to call a Python namespace method with JSON serialization.
/// For methods that take a request and return a response.
/// Uses DictWithModelDump to pass a dict that also has model_dump() method,
/// making it compatible with both depythonize and Python wrappers.
async fn call_py_method<Req, Resp>(
py_namespace: Arc<Py<PyAny>>,
method_name: &'static str,
request: Req,
) -> lance_core::Result<Resp>
where
Req: serde::Serialize + Send + 'static,
Resp: serde::de::DeserializeOwned + Send + 'static,
{
let request_json = serde_json::to_string(&request).map_err(|e| {
lance_core::Error::io(format!(
"Failed to serialize request for {}: {}",
method_name, e
))
})?;
let response_json = tokio::task::spawn_blocking(move || {
Python::attach(|py| {
let json_module = py.import("json")?;
let request_dict = json_module.call_method1("loads", (&request_json,))?;
// Wrap dict in DictWithModelDump so it works with both depythonize and .model_dump()
let dict_class = get_dict_with_model_dump_class(py)?;
let request_arg = dict_class.call1((request_dict,))?;
// Call the Python method
let result = py_namespace.call_method1(py, method_name, (request_arg,))?;
// Convert response to dict, then to JSON
// Pydantic models have model_dump() method
let result_dict = if result.bind(py).hasattr("model_dump")? {
result.call_method0(py, "model_dump")?
} else {
result
};
let response_json: String = json_module
.call_method1("dumps", (result_dict,))?
.extract()?;
Ok::<_, PyErr>(response_json)
})
})
.await
.map_err(|e| lance_core::Error::io(format!("Task join error for {}: {}", method_name, e)))?
.map_err(|e: PyErr| lance_core::Error::io(format!("Python error in {}: {}", method_name, e)))?;
serde_json::from_str(&response_json).map_err(|e| {
lance_core::Error::io(format!(
"Failed to deserialize response from {}: {}",
method_name, e
))
})
}
/// Helper for methods that return () on success
async fn call_py_method_unit<Req>(
py_namespace: Arc<Py<PyAny>>,
method_name: &'static str,
request: Req,
) -> lance_core::Result<()>
where
Req: serde::Serialize + Send + 'static,
{
let request_json = serde_json::to_string(&request).map_err(|e| {
lance_core::Error::io(format!(
"Failed to serialize request for {}: {}",
method_name, e
))
})?;
tokio::task::spawn_blocking(move || {
Python::attach(|py| {
let json_module = py.import("json")?;
let request_dict = json_module.call_method1("loads", (&request_json,))?;
// Wrap dict in DictWithModelDump
let dict_class = get_dict_with_model_dump_class(py)?;
let request_arg = dict_class.call1((request_dict,))?;
// Call the Python method
py_namespace.call_method1(py, method_name, (request_arg,))?;
Ok::<_, PyErr>(())
})
})
.await
.map_err(|e| lance_core::Error::io(format!("Task join error for {}: {}", method_name, e)))?
.map_err(|e: PyErr| lance_core::Error::io(format!("Python error in {}: {}", method_name, e)))
}
/// Helper for methods that return a primitive type
async fn call_py_method_primitive<Req, Resp>(
py_namespace: Arc<Py<PyAny>>,
method_name: &'static str,
request: Req,
) -> lance_core::Result<Resp>
where
Req: serde::Serialize + Send + 'static,
Resp: for<'py> pyo3::FromPyObject<'py> + Send + 'static,
{
let request_json = serde_json::to_string(&request).map_err(|e| {
lance_core::Error::io(format!(
"Failed to serialize request for {}: {}",
method_name, e
))
})?;
tokio::task::spawn_blocking(move || {
Python::attach(|py| {
let json_module = py.import("json")?;
let request_dict = json_module.call_method1("loads", (&request_json,))?;
// Wrap dict in DictWithModelDump
let dict_class = get_dict_with_model_dump_class(py)?;
let request_arg = dict_class.call1((request_dict,))?;
// Call the Python method
let result = py_namespace.call_method1(py, method_name, (request_arg,))?;
let value: Resp = result.extract(py)?;
Ok::<_, PyErr>(value)
})
})
.await
.map_err(|e| lance_core::Error::io(format!("Task join error for {}: {}", method_name, e)))?
.map_err(|e: PyErr| lance_core::Error::io(format!("Python error in {}: {}", method_name, e)))
}
/// Helper for methods that return Bytes
async fn call_py_method_bytes<Req>(
py_namespace: Arc<Py<PyAny>>,
method_name: &'static str,
request: Req,
) -> lance_core::Result<Bytes>
where
Req: serde::Serialize + Send + 'static,
{
let request_json = serde_json::to_string(&request).map_err(|e| {
lance_core::Error::io(format!(
"Failed to serialize request for {}: {}",
method_name, e
))
})?;
tokio::task::spawn_blocking(move || {
Python::attach(|py| {
let json_module = py.import("json")?;
let request_dict = json_module.call_method1("loads", (&request_json,))?;
// Wrap dict in DictWithModelDump
let dict_class = get_dict_with_model_dump_class(py)?;
let request_arg = dict_class.call1((request_dict,))?;
// Call the Python method
let result = py_namespace.call_method1(py, method_name, (request_arg,))?;
let bytes_data: Vec<u8> = result.extract(py)?;
Ok::<_, PyErr>(Bytes::from(bytes_data))
})
})
.await
.map_err(|e| lance_core::Error::io(format!("Task join error for {}: {}", method_name, e)))?
.map_err(|e: PyErr| lance_core::Error::io(format!("Python error in {}: {}", method_name, e)))
}
/// Helper for methods that take request + data and return a response
async fn call_py_method_with_data<Req, Resp>(
py_namespace: Arc<Py<PyAny>>,
method_name: &'static str,
request: Req,
data: Bytes,
) -> lance_core::Result<Resp>
where
Req: serde::Serialize + Send + 'static,
Resp: serde::de::DeserializeOwned + Send + 'static,
{
let request_json = serde_json::to_string(&request).map_err(|e| {
lance_core::Error::io(format!(
"Failed to serialize request for {}: {}",
method_name, e
))
})?;
let response_json = tokio::task::spawn_blocking(move || {
Python::attach(|py| {
let json_module = py.import("json")?;
let request_dict = json_module.call_method1("loads", (&request_json,))?;
// Wrap dict in DictWithModelDump
let dict_class = get_dict_with_model_dump_class(py)?;
let request_arg = dict_class.call1((request_dict,))?;
// Pass request and bytes to Python method
let py_bytes = pyo3::types::PyBytes::new(py, &data);
let result = py_namespace.call_method1(py, method_name, (request_arg, py_bytes))?;
// Convert response dict to JSON
let response_json: String = json_module.call_method1("dumps", (result,))?.extract()?;
Ok::<_, PyErr>(response_json)
})
})
.await
.map_err(|e| lance_core::Error::io(format!("Task join error for {}: {}", method_name, e)))?
.map_err(|e: PyErr| lance_core::Error::io(format!("Python error in {}: {}", method_name, e)))?;
serde_json::from_str(&response_json).map_err(|e| {
lance_core::Error::io(format!(
"Failed to deserialize response from {}: {}",
method_name, e
))
})
}
#[async_trait]
impl LanceNamespaceTrait for PyLanceNamespace {
fn namespace_id(&self) -> String {
self.namespace_id.clone()
}
async fn list_namespaces(
&self,
request: ListNamespacesRequest,
) -> lance_core::Result<ListNamespacesResponse> {
call_py_method(self.py_namespace.clone(), "list_namespaces", request).await
}
async fn describe_namespace(
&self,
request: DescribeNamespaceRequest,
) -> lance_core::Result<DescribeNamespaceResponse> {
call_py_method(self.py_namespace.clone(), "describe_namespace", request).await
}
async fn create_namespace(
&self,
request: CreateNamespaceRequest,
) -> lance_core::Result<CreateNamespaceResponse> {
call_py_method(self.py_namespace.clone(), "create_namespace", request).await
}
async fn drop_namespace(
&self,
request: DropNamespaceRequest,
) -> lance_core::Result<DropNamespaceResponse> {
call_py_method(self.py_namespace.clone(), "drop_namespace", request).await
}
async fn namespace_exists(&self, request: NamespaceExistsRequest) -> lance_core::Result<()> {
call_py_method_unit(self.py_namespace.clone(), "namespace_exists", request).await
}
async fn list_tables(
&self,
request: ListTablesRequest,
) -> lance_core::Result<ListTablesResponse> {
call_py_method(self.py_namespace.clone(), "list_tables", request).await
}
async fn describe_table(
&self,
request: DescribeTableRequest,
) -> lance_core::Result<DescribeTableResponse> {
call_py_method(self.py_namespace.clone(), "describe_table", request).await
}
async fn register_table(
&self,
request: RegisterTableRequest,
) -> lance_core::Result<RegisterTableResponse> {
call_py_method(self.py_namespace.clone(), "register_table", request).await
}
async fn table_exists(&self, request: TableExistsRequest) -> lance_core::Result<()> {
call_py_method_unit(self.py_namespace.clone(), "table_exists", request).await
}
async fn drop_table(&self, request: DropTableRequest) -> lance_core::Result<DropTableResponse> {
call_py_method(self.py_namespace.clone(), "drop_table", request).await
}
async fn deregister_table(
&self,
request: DeregisterTableRequest,
) -> lance_core::Result<DeregisterTableResponse> {
call_py_method(self.py_namespace.clone(), "deregister_table", request).await
}
async fn count_table_rows(&self, request: CountTableRowsRequest) -> lance_core::Result<i64> {
call_py_method_primitive(self.py_namespace.clone(), "count_table_rows", request).await
}
async fn create_table(
&self,
request: CreateTableRequest,
request_data: Bytes,
) -> lance_core::Result<CreateTableResponse> {
call_py_method_with_data(
self.py_namespace.clone(),
"create_table",
request,
request_data,
)
.await
}
async fn declare_table(
&self,
request: DeclareTableRequest,
) -> lance_core::Result<DeclareTableResponse> {
call_py_method(self.py_namespace.clone(), "declare_table", request).await
}
async fn insert_into_table(
&self,
request: InsertIntoTableRequest,
request_data: Bytes,
) -> lance_core::Result<InsertIntoTableResponse> {
call_py_method_with_data(
self.py_namespace.clone(),
"insert_into_table",
request,
request_data,
)
.await
}
async fn merge_insert_into_table(
&self,
request: MergeInsertIntoTableRequest,
request_data: Bytes,
) -> lance_core::Result<MergeInsertIntoTableResponse> {
call_py_method_with_data(
self.py_namespace.clone(),
"merge_insert_into_table",
request,
request_data,
)
.await
}
async fn update_table(
&self,
request: UpdateTableRequest,
) -> lance_core::Result<UpdateTableResponse> {
call_py_method(self.py_namespace.clone(), "update_table", request).await
}
async fn delete_from_table(
&self,
request: DeleteFromTableRequest,
) -> lance_core::Result<DeleteFromTableResponse> {
call_py_method(self.py_namespace.clone(), "delete_from_table", request).await
}
async fn query_table(&self, request: QueryTableRequest) -> lance_core::Result<Bytes> {
call_py_method_bytes(self.py_namespace.clone(), "query_table", request).await
}
async fn create_table_index(
&self,
request: CreateTableIndexRequest,
) -> lance_core::Result<CreateTableIndexResponse> {
call_py_method(self.py_namespace.clone(), "create_table_index", request).await
}
async fn list_table_indices(
&self,
request: ListTableIndicesRequest,
) -> lance_core::Result<ListTableIndicesResponse> {
call_py_method(self.py_namespace.clone(), "list_table_indices", request).await
}
async fn describe_table_index_stats(
&self,
request: DescribeTableIndexStatsRequest,
) -> lance_core::Result<DescribeTableIndexStatsResponse> {
call_py_method(
self.py_namespace.clone(),
"describe_table_index_stats",
request,
)
.await
}
async fn describe_transaction(
&self,
request: DescribeTransactionRequest,
) -> lance_core::Result<DescribeTransactionResponse> {
call_py_method(self.py_namespace.clone(), "describe_transaction", request).await
}
async fn alter_transaction(
&self,
request: AlterTransactionRequest,
) -> lance_core::Result<AlterTransactionResponse> {
call_py_method(self.py_namespace.clone(), "alter_transaction", request).await
}
async fn create_table_scalar_index(
&self,
request: CreateTableIndexRequest,
) -> lance_core::Result<CreateTableScalarIndexResponse> {
call_py_method(
self.py_namespace.clone(),
"create_table_scalar_index",
request,
)
.await
}
async fn drop_table_index(
&self,
request: DropTableIndexRequest,
) -> lance_core::Result<DropTableIndexResponse> {
call_py_method(self.py_namespace.clone(), "drop_table_index", request).await
}
async fn list_all_tables(
&self,
request: ListTablesRequest,
) -> lance_core::Result<ListTablesResponse> {
call_py_method(self.py_namespace.clone(), "list_all_tables", request).await
}
async fn restore_table(
&self,
request: RestoreTableRequest,
) -> lance_core::Result<RestoreTableResponse> {
call_py_method(self.py_namespace.clone(), "restore_table", request).await
}
async fn rename_table(
&self,
request: RenameTableRequest,
) -> lance_core::Result<RenameTableResponse> {
call_py_method(self.py_namespace.clone(), "rename_table", request).await
}
async fn list_table_versions(
&self,
request: ListTableVersionsRequest,
) -> lance_core::Result<ListTableVersionsResponse> {
call_py_method(self.py_namespace.clone(), "list_table_versions", request).await
}
async fn create_table_version(
&self,
request: CreateTableVersionRequest,
) -> lance_core::Result<CreateTableVersionResponse> {
call_py_method(self.py_namespace.clone(), "create_table_version", request).await
}
async fn describe_table_version(
&self,
request: DescribeTableVersionRequest,
) -> lance_core::Result<DescribeTableVersionResponse> {
call_py_method(self.py_namespace.clone(), "describe_table_version", request).await
}
async fn batch_delete_table_versions(
&self,
request: BatchDeleteTableVersionsRequest,
) -> lance_core::Result<BatchDeleteTableVersionsResponse> {
call_py_method(
self.py_namespace.clone(),
"batch_delete_table_versions",
request,
)
.await
}
async fn update_table_schema_metadata(
&self,
request: UpdateTableSchemaMetadataRequest,
) -> lance_core::Result<UpdateTableSchemaMetadataResponse> {
call_py_method(
self.py_namespace.clone(),
"update_table_schema_metadata",
request,
)
.await
}
async fn get_table_stats(
&self,
request: GetTableStatsRequest,
) -> lance_core::Result<GetTableStatsResponse> {
call_py_method(self.py_namespace.clone(), "get_table_stats", request).await
}
async fn explain_table_query_plan(
&self,
request: ExplainTableQueryPlanRequest,
) -> lance_core::Result<String> {
call_py_method_primitive(
self.py_namespace.clone(),
"explain_table_query_plan",
request,
)
.await
}
async fn analyze_table_query_plan(
&self,
request: AnalyzeTableQueryPlanRequest,
) -> lance_core::Result<String> {
call_py_method_primitive(
self.py_namespace.clone(),
"analyze_table_query_plan",
request,
)
.await
}
async fn alter_table_add_columns(
&self,
request: AlterTableAddColumnsRequest,
) -> lance_core::Result<AlterTableAddColumnsResponse> {
call_py_method(
self.py_namespace.clone(),
"alter_table_add_columns",
request,
)
.await
}
async fn alter_table_alter_columns(
&self,
request: AlterTableAlterColumnsRequest,
) -> lance_core::Result<AlterTableAlterColumnsResponse> {
call_py_method(
self.py_namespace.clone(),
"alter_table_alter_columns",
request,
)
.await
}
async fn alter_table_drop_columns(
&self,
request: AlterTableDropColumnsRequest,
) -> lance_core::Result<AlterTableDropColumnsResponse> {
call_py_method(
self.py_namespace.clone(),
"alter_table_drop_columns",
request,
)
.await
}
async fn list_table_tags(
&self,
request: ListTableTagsRequest,
) -> lance_core::Result<ListTableTagsResponse> {
call_py_method(self.py_namespace.clone(), "list_table_tags", request).await
}
async fn create_table_tag(
&self,
request: CreateTableTagRequest,
) -> lance_core::Result<CreateTableTagResponse> {
call_py_method(self.py_namespace.clone(), "create_table_tag", request).await
}
async fn delete_table_tag(
&self,
request: DeleteTableTagRequest,
) -> lance_core::Result<DeleteTableTagResponse> {
call_py_method(self.py_namespace.clone(), "delete_table_tag", request).await
}
async fn update_table_tag(
&self,
request: UpdateTableTagRequest,
) -> lance_core::Result<UpdateTableTagResponse> {
call_py_method(self.py_namespace.clone(), "update_table_tag", request).await
}
async fn get_table_tag_version(
&self,
request: GetTableTagVersionRequest,
) -> lance_core::Result<GetTableTagVersionResponse> {
call_py_method(self.py_namespace.clone(), "get_table_tag_version", request).await
}
}
/// Convert Python dict to HashMap<String, String>
#[allow(dead_code)]
fn dict_to_hashmap(dict: &Bound<'_, PyDict>) -> PyResult<HashMap<String, String>> {
let mut map = HashMap::new();
for (key, value) in dict.iter() {
let key_str: String = key.extract()?;
let value_str: String = value.extract()?;
map.insert(key_str, value_str);
}
Ok(map)
}
/// Extract an Arc<dyn LanceNamespace> from a Python namespace object.
///
/// This function wraps any Python namespace object with PyLanceNamespace.
/// The PyLanceNamespace wrapper uses DictWithModelDump to pass requests,
/// which works with both:
/// - Native namespaces (DirectoryNamespace, RestNamespace) that use depythonize (expects dict)
/// - Custom Python implementations that call .model_dump() on the request
pub fn extract_namespace_arc(
py: Python<'_>,
ns: Py<PyAny>,
) -> PyResult<Arc<dyn LanceNamespaceTrait>> {
let ns_ref = ns.bind(py);
PyLanceNamespace::create_arc(py, ns_ref)
}
/// Create a LanceNamespaceStorageOptionsProvider from a namespace client and table ID.
///
/// This creates a Rust storage options provider that fetches credentials from the
/// namespace's describe_table() method, enabling automatic credential refresh.
///
/// # Arguments
/// * `namespace_client` - The namespace client (wrapped PyLanceNamespace)
/// * `table_id` - Full table identifier (namespace_path + table_name)
pub fn create_namespace_storage_options_provider(
namespace_client: Arc<dyn LanceNamespaceTrait>,
table_id: Vec<String>,
) -> Arc<dyn StorageOptionsProvider> {
Arc::new(LanceNamespaceStorageOptionsProvider::new(
namespace_client,
table_id,
))
}

View File

@@ -16,10 +16,10 @@ use lancedb::{
query::Select, query::Select,
}; };
use pyo3::{ use pyo3::{
Bound, PyAny, PyRef, PyRefMut, PyResult, Python,
exceptions::PyRuntimeError, exceptions::PyRuntimeError,
pyclass, pymethods, pyclass, pymethods,
types::{PyAnyMethods, PyDict, PyDictMethods, PyType}, types::{PyAnyMethods, PyDict, PyDictMethods, PyType},
Bound, PyAny, PyRef, PyRefMut, PyResult, Python,
}; };
use pyo3_async_runtimes::tokio::future_into_py; use pyo3_async_runtimes::tokio::future_into_py;

View File

@@ -4,9 +4,9 @@
use std::sync::Arc; use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
use arrow::array::make_array;
use arrow::array::Array; use arrow::array::Array;
use arrow::array::ArrayData; use arrow::array::ArrayData;
use arrow::array::make_array;
use arrow::pyarrow::FromPyArrow; use arrow::pyarrow::FromPyArrow;
use arrow::pyarrow::IntoPyArrow; use arrow::pyarrow::IntoPyArrow;
use arrow::pyarrow::ToPyArrow; use arrow::pyarrow::ToPyArrow;
@@ -22,25 +22,23 @@ use lancedb::query::{
VectorQuery as LanceDbVectorQuery, VectorQuery as LanceDbVectorQuery,
}; };
use lancedb::table::AnyQuery; use lancedb::table::AnyQuery;
use pyo3::prelude::{PyAnyMethods, PyDictMethods};
use pyo3::pyfunction;
use pyo3::pymethods;
use pyo3::types::PyList;
use pyo3::types::{PyDict, PyString};
use pyo3::Bound; use pyo3::Bound;
use pyo3::IntoPyObject; use pyo3::IntoPyObject;
use pyo3::PyAny; use pyo3::PyAny;
use pyo3::PyRef; use pyo3::PyRef;
use pyo3::PyResult; use pyo3::PyResult;
use pyo3::Python; use pyo3::Python;
use pyo3::{exceptions::PyRuntimeError, FromPyObject}; use pyo3::prelude::{PyAnyMethods, PyDictMethods};
use pyo3::{ use pyo3::pyfunction;
exceptions::{PyNotImplementedError, PyValueError}, use pyo3::pymethods;
intern, use pyo3::types::PyList;
}; use pyo3::types::{PyDict, PyString};
use pyo3::{pyclass, PyErr}; use pyo3::{FromPyObject, exceptions::PyRuntimeError};
use pyo3::{PyErr, pyclass};
use pyo3::{exceptions::PyValueError, intern};
use pyo3_async_runtimes::tokio::future_into_py; use pyo3_async_runtimes::tokio::future_into_py;
use crate::expr::PyExpr;
use crate::util::parse_distance_type; use crate::util::parse_distance_type;
use crate::{arrow::RecordBatchStream, util::PyLanceDB}; use crate::{arrow::RecordBatchStream, util::PyLanceDB};
use crate::{error::PythonErrorExt, index::class_name}; use crate::{error::PythonErrorExt, index::class_name};
@@ -316,6 +314,19 @@ impl<'py> IntoPyObject<'py> for PySelect {
Select::All => Ok(py.None().into_bound(py).into_any()), Select::All => Ok(py.None().into_bound(py).into_any()),
Select::Columns(columns) => Ok(columns.into_pyobject(py)?.into_any()), Select::Columns(columns) => Ok(columns.into_pyobject(py)?.into_any()),
Select::Dynamic(columns) => Ok(columns.into_pyobject(py)?.into_any()), Select::Dynamic(columns) => Ok(columns.into_pyobject(py)?.into_any()),
Select::Expr(pairs) => {
// Serialize DataFusion Expr -> SQL string so Python sees the same
// format as Select::Dynamic: a list of (name, sql_string) tuples.
let sql_pairs: PyResult<Vec<(String, String)>> = pairs
.into_iter()
.map(|(name, expr)| {
lancedb::expr::expr_to_sql_string(&expr)
.map(|sql| (name, sql))
.map_err(|e| PyRuntimeError::new_err(e.to_string()))
})
.collect();
Ok(sql_pairs?.into_pyobject(py)?.into_any())
}
} }
} }
} }
@@ -331,9 +342,13 @@ impl<'py> IntoPyObject<'py> for PyQueryFilter {
fn into_pyobject(self, py: pyo3::Python<'py>) -> PyResult<Self::Output> { fn into_pyobject(self, py: pyo3::Python<'py>) -> PyResult<Self::Output> {
match self.0 { match self.0 {
QueryFilter::Datafusion(_) => Err(PyNotImplementedError::new_err( QueryFilter::Datafusion(expr) => {
"Datafusion filter has no conversion to Python", // Serialize the DataFusion expression to a SQL string so that
)), // callers (e.g. remote tables) see the same format as Sql.
let sql = lancedb::expr::expr_to_sql_string(&expr)
.map_err(|e| PyRuntimeError::new_err(e.to_string()))?;
Ok(sql.into_pyobject(py)?.into_any())
}
QueryFilter::Sql(sql) => Ok(sql.into_pyobject(py)?.into_any()), QueryFilter::Sql(sql) => Ok(sql.into_pyobject(py)?.into_any()),
QueryFilter::Substrait(substrait) => Ok(substrait.into_pyobject(py)?.into_any()), QueryFilter::Substrait(substrait) => Ok(substrait.into_pyobject(py)?.into_any()),
} }
@@ -357,10 +372,20 @@ impl Query {
self.inner = self.inner.clone().only_if(predicate); self.inner = self.inner.clone().only_if(predicate);
} }
pub fn where_expr(&mut self, expr: PyExpr) {
self.inner = self.inner.clone().only_if_expr(expr.0);
}
pub fn select(&mut self, columns: Vec<(String, String)>) { pub fn select(&mut self, columns: Vec<(String, String)>) {
self.inner = self.inner.clone().select(Select::dynamic(&columns)); self.inner = self.inner.clone().select(Select::dynamic(&columns));
} }
pub fn select_expr(&mut self, columns: Vec<(String, PyExpr)>) {
let pairs: Vec<(String, lancedb::expr::DfExpr)> =
columns.into_iter().map(|(name, e)| (name, e.0)).collect();
self.inner = self.inner.clone().select(Select::Expr(pairs));
}
pub fn select_columns(&mut self, columns: Vec<String>) { pub fn select_columns(&mut self, columns: Vec<String>) {
self.inner = self.inner.clone().select(Select::columns(&columns)); self.inner = self.inner.clone().select(Select::columns(&columns));
} }
@@ -594,10 +619,20 @@ impl FTSQuery {
self.inner = self.inner.clone().only_if(predicate); self.inner = self.inner.clone().only_if(predicate);
} }
pub fn where_expr(&mut self, expr: PyExpr) {
self.inner = self.inner.clone().only_if_expr(expr.0);
}
pub fn select(&mut self, columns: Vec<(String, String)>) { pub fn select(&mut self, columns: Vec<(String, String)>) {
self.inner = self.inner.clone().select(Select::dynamic(&columns)); self.inner = self.inner.clone().select(Select::dynamic(&columns));
} }
pub fn select_expr(&mut self, columns: Vec<(String, PyExpr)>) {
let pairs: Vec<(String, lancedb::expr::DfExpr)> =
columns.into_iter().map(|(name, e)| (name, e.0)).collect();
self.inner = self.inner.clone().select(Select::Expr(pairs));
}
pub fn select_columns(&mut self, columns: Vec<String>) { pub fn select_columns(&mut self, columns: Vec<String>) {
self.inner = self.inner.clone().select(Select::columns(&columns)); self.inner = self.inner.clone().select(Select::columns(&columns));
} }
@@ -712,6 +747,10 @@ impl VectorQuery {
self.inner = self.inner.clone().only_if(predicate); self.inner = self.inner.clone().only_if(predicate);
} }
pub fn where_expr(&mut self, expr: PyExpr) {
self.inner = self.inner.clone().only_if_expr(expr.0);
}
pub fn add_query_vector(&mut self, vector: Bound<'_, PyAny>) -> PyResult<()> { pub fn add_query_vector(&mut self, vector: Bound<'_, PyAny>) -> PyResult<()> {
let data: ArrayData = ArrayData::from_pyarrow_bound(&vector)?; let data: ArrayData = ArrayData::from_pyarrow_bound(&vector)?;
let array = make_array(data); let array = make_array(data);
@@ -723,6 +762,12 @@ impl VectorQuery {
self.inner = self.inner.clone().select(Select::dynamic(&columns)); self.inner = self.inner.clone().select(Select::dynamic(&columns));
} }
pub fn select_expr(&mut self, columns: Vec<(String, PyExpr)>) {
let pairs: Vec<(String, lancedb::expr::DfExpr)> =
columns.into_iter().map(|(name, e)| (name, e.0)).collect();
self.inner = self.inner.clone().select(Select::Expr(pairs));
}
pub fn select_columns(&mut self, columns: Vec<String>) { pub fn select_columns(&mut self, columns: Vec<String>) {
self.inner = self.inner.clone().select(Select::columns(&columns)); self.inner = self.inner.clone().select(Select::columns(&columns));
} }
@@ -877,11 +922,21 @@ impl HybridQuery {
self.inner_fts.r#where(predicate); self.inner_fts.r#where(predicate);
} }
pub fn where_expr(&mut self, expr: PyExpr) {
self.inner_vec.where_expr(expr.clone());
self.inner_fts.where_expr(expr);
}
pub fn select(&mut self, columns: Vec<(String, String)>) { pub fn select(&mut self, columns: Vec<(String, String)>) {
self.inner_vec.select(columns.clone()); self.inner_vec.select(columns.clone());
self.inner_fts.select(columns); self.inner_fts.select(columns);
} }
pub fn select_expr(&mut self, columns: Vec<(String, PyExpr)>) {
self.inner_vec.select_expr(columns.clone());
self.inner_fts.select_expr(columns);
}
pub fn select_columns(&mut self, columns: Vec<String>) { pub fn select_columns(&mut self, columns: Vec<String>) {
self.inner_vec.select_columns(columns.clone()); self.inner_vec.select_columns(columns.clone());
self.inner_fts.select_columns(columns); self.inner_fts.select_columns(columns);

View File

@@ -4,7 +4,7 @@
use std::sync::Arc; use std::sync::Arc;
use lancedb::{ObjectStoreRegistry, Session as LanceSession}; use lancedb::{ObjectStoreRegistry, Session as LanceSession};
use pyo3::{pyclass, pymethods, PyResult}; use pyo3::{PyResult, pyclass, pymethods};
/// A session for managing caches and object stores across LanceDB operations. /// A session for managing caches and object stores across LanceDB operations.
/// ///

View File

@@ -1,150 +0,0 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
//! PyO3 bindings for StorageOptionsProvider
//!
//! This module provides the bridge between Python StorageOptionsProvider objects
//! and Rust's StorageOptionsProvider trait, enabling automatic credential refresh.
use std::collections::HashMap;
use std::sync::Arc;
use async_trait::async_trait;
use lance_io::object_store::StorageOptionsProvider;
use pyo3::prelude::*;
use pyo3::types::PyDict;
/// Internal wrapper around a Python object implementing StorageOptionsProvider
pub struct PyStorageOptionsProvider {
/// The Python object implementing fetch_storage_options()
inner: Py<PyAny>,
}
impl Clone for PyStorageOptionsProvider {
fn clone(&self) -> Self {
Python::attach(|py| Self {
inner: self.inner.clone_ref(py),
})
}
}
impl PyStorageOptionsProvider {
pub fn new(obj: Py<PyAny>) -> PyResult<Self> {
Python::attach(|py| {
// Verify the object has a fetch_storage_options method
if !obj.bind(py).hasattr("fetch_storage_options")? {
return Err(pyo3::exceptions::PyTypeError::new_err(
"StorageOptionsProvider must implement fetch_storage_options() method",
));
}
Ok(Self { inner: obj })
})
}
}
/// Wrapper that implements the Rust StorageOptionsProvider trait
pub struct PyStorageOptionsProviderWrapper {
py_provider: PyStorageOptionsProvider,
}
impl PyStorageOptionsProviderWrapper {
pub fn new(py_provider: PyStorageOptionsProvider) -> Self {
Self { py_provider }
}
}
#[async_trait]
impl StorageOptionsProvider for PyStorageOptionsProviderWrapper {
async fn fetch_storage_options(&self) -> lance_core::Result<Option<HashMap<String, String>>> {
// Call Python method from async context using spawn_blocking
let py_provider = self.py_provider.clone();
tokio::task::spawn_blocking(move || {
Python::attach(|py| {
// Call the Python fetch_storage_options method
let result = py_provider
.inner
.bind(py)
.call_method0("fetch_storage_options")
.map_err(|e| lance_core::Error::IO {
source: Box::new(std::io::Error::other(format!(
"Failed to call fetch_storage_options: {}",
e
))),
location: snafu::location!(),
})?;
// If result is None, return None
if result.is_none() {
return Ok(None);
}
// Extract the result dict - should be a flat Map<String, String>
let result_dict = result.downcast::<PyDict>().map_err(|_| {
lance_core::Error::InvalidInput {
source: "fetch_storage_options() must return None or a dict of string key-value pairs".into(),
location: snafu::location!(),
}
})?;
// Convert all entries to HashMap<String, String>
let mut storage_options = HashMap::new();
for (key, value) in result_dict.iter() {
let key_str: String = key.extract().map_err(|e| {
lance_core::Error::InvalidInput {
source: format!("Storage option key must be a string: {}", e).into(),
location: snafu::location!(),
}
})?;
let value_str: String = value.extract().map_err(|e| {
lance_core::Error::InvalidInput {
source: format!("Storage option value must be a string: {}", e).into(),
location: snafu::location!(),
}
})?;
storage_options.insert(key_str, value_str);
}
Ok(Some(storage_options))
})
})
.await
.map_err(|e| lance_core::Error::IO {
source: Box::new(std::io::Error::other(format!(
"Task join error: {}",
e
))),
location: snafu::location!(),
})?
}
fn provider_id(&self) -> String {
Python::attach(|py| {
// Call provider_id() method on the Python object
let obj = self.py_provider.inner.bind(py);
obj.call_method0("provider_id")
.and_then(|result| result.extract::<String>())
.unwrap_or_else(|e| {
// If provider_id() fails, construct a fallback ID
format!("PyStorageOptionsProvider(error: {})", e)
})
})
}
}
impl std::fmt::Debug for PyStorageOptionsProviderWrapper {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "PyStorageOptionsProviderWrapper({})", self.provider_id())
}
}
/// Convert a Python object to an Arc<dyn StorageOptionsProvider>
///
/// This is the main entry point for converting Python StorageOptionsProvider objects
/// to Rust trait objects that can be used by the Lance ecosystem.
pub fn py_object_to_storage_options_provider(
py_obj: Py<PyAny>,
) -> PyResult<Arc<dyn StorageOptionsProvider>> {
let py_provider = PyStorageOptionsProvider::new(py_obj)?;
Ok(Arc::new(PyStorageOptionsProviderWrapper::new(py_provider)))
}

View File

@@ -5,7 +5,7 @@ use std::{collections::HashMap, sync::Arc};
use crate::{ use crate::{
connection::Connection, connection::Connection,
error::PythonErrorExt, error::PythonErrorExt,
index::{extract_index_params, IndexConfig}, index::{IndexConfig, extract_index_params},
query::{Query, TakeQuery}, query::{Query, TakeQuery},
table::scannable::PyScannable, table::scannable::PyScannable,
}; };
@@ -19,10 +19,10 @@ use lancedb::table::{
Table as LanceDbTable, Table as LanceDbTable,
}; };
use pyo3::{ use pyo3::{
Bound, FromPyObject, Py, PyAny, PyRef, PyResult, Python,
exceptions::{PyKeyError, PyRuntimeError, PyValueError}, exceptions::{PyKeyError, PyRuntimeError, PyValueError},
pyclass, pymethods, pyclass, pymethods,
types::{IntoPyDict, PyAnyMethods, PyDict, PyDictMethods}, types::{IntoPyDict, PyAnyMethods, PyDict, PyDictMethods},
Bound, FromPyObject, PyAny, PyRef, PyResult, Python,
}; };
use pyo3_async_runtimes::tokio::future_into_py; use pyo3_async_runtimes::tokio::future_into_py;
@@ -112,19 +112,24 @@ impl From<lancedb::table::AddResult> for AddResult {
#[pyclass(get_all)] #[pyclass(get_all)]
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
pub struct DeleteResult { pub struct DeleteResult {
pub num_deleted_rows: u64,
pub version: u64, pub version: u64,
} }
#[pymethods] #[pymethods]
impl DeleteResult { impl DeleteResult {
pub fn __repr__(&self) -> String { pub fn __repr__(&self) -> String {
format!("DeleteResult(version={})", self.version) format!(
"DeleteResult(num_deleted_rows={}, version={})",
self.num_deleted_rows, self.version
)
} }
} }
impl From<lancedb::table::DeleteResult> for DeleteResult { impl From<lancedb::table::DeleteResult> for DeleteResult {
fn from(result: lancedb::table::DeleteResult) -> Self { fn from(result: lancedb::table::DeleteResult) -> Self {
Self { Self {
num_deleted_rows: result.num_deleted_rows,
version: result.version, version: result.version,
} }
} }
@@ -294,10 +299,12 @@ impl Table {
}) })
} }
#[pyo3(signature = (data, mode, progress=None))]
pub fn add<'a>( pub fn add<'a>(
self_: PyRef<'a, Self>, self_: PyRef<'a, Self>,
data: PyScannable, data: PyScannable,
mode: String, mode: String,
progress: Option<Py<PyAny>>,
) -> PyResult<Bound<'a, PyAny>> { ) -> PyResult<Bound<'a, PyAny>> {
let mut op = self_.inner_ref()?.add(data); let mut op = self_.inner_ref()?.add(data);
if mode == "append" { if mode == "append" {
@@ -307,6 +314,81 @@ impl Table {
} else { } else {
return Err(PyValueError::new_err(format!("Invalid mode: {}", mode))); return Err(PyValueError::new_err(format!("Invalid mode: {}", mode)));
} }
if let Some(progress_obj) = progress {
let is_callable = Python::attach(|py| progress_obj.bind(py).is_callable());
if is_callable {
// Callback: call with a dict of progress info.
op = op.progress(move |p| {
Python::attach(|py| {
let dict = PyDict::new(py);
if let Err(e) = dict
.set_item("output_rows", p.output_rows())
.and_then(|_| dict.set_item("output_bytes", p.output_bytes()))
.and_then(|_| dict.set_item("total_rows", p.total_rows()))
.and_then(|_| {
dict.set_item("elapsed_seconds", p.elapsed().as_secs_f64())
})
.and_then(|_| dict.set_item("active_tasks", p.active_tasks()))
.and_then(|_| dict.set_item("total_tasks", p.total_tasks()))
.and_then(|_| dict.set_item("done", p.done()))
{
log::warn!("progress dict error: {e}");
return;
}
if let Err(e) = progress_obj.call1(py, (dict,)) {
log::warn!("progress callback error: {e}");
}
});
});
} else {
// tqdm-like: has update() method.
let mut last_rows: usize = 0;
let mut total_set = false;
op = op.progress(move |p| {
let current = p.output_rows();
let prev = last_rows;
last_rows = current;
Python::attach(|py| {
if let Some(total) = p.total_rows()
&& !total_set
{
if let Err(e) = progress_obj.setattr(py, "total", total) {
log::warn!("progress setattr error: {e}");
}
total_set = true;
}
let delta = current.saturating_sub(prev);
if delta > 0 {
if let Err(e) = progress_obj.call_method1(py, "update", (delta,)) {
log::warn!("progress update error: {e}");
}
// Show throughput and active workers in tqdm postfix.
let elapsed = p.elapsed().as_secs_f64();
if elapsed > 0.0 {
let mb_per_sec = p.output_bytes() as f64 / elapsed / 1_000_000.0;
let postfix = format!(
"{:.1} MB/s | {}/{} workers",
mb_per_sec,
p.active_tasks(),
p.total_tasks()
);
if let Err(e) =
progress_obj.call_method1(py, "set_postfix_str", (postfix,))
{
log::warn!("progress set_postfix_str error: {e}");
}
}
}
if p.done() {
// Force a final refresh so the bar shows completion.
if let Err(e) = progress_obj.call_method0(py, "refresh") {
log::warn!("progress refresh error: {e}");
}
}
});
});
}
}
future_into_py(self_.py(), async move { future_into_py(self_.py(), async move {
let result = op.execute().await.infer_error()?; let result = op.execute().await.infer_error()?;
@@ -421,6 +503,17 @@ impl Table {
}) })
} }
pub fn prewarm_data(
self_: PyRef<'_, Self>,
columns: Option<Vec<String>>,
) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone();
future_into_py(self_.py(), async move {
inner.prewarm_data(columns).await.infer_error()?;
Ok(())
})
}
pub fn list_indices(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> { pub fn list_indices(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone(); let inner = self_.inner_ref()?.clone();
future_into_py(self_.py(), async move { future_into_py(self_.py(), async move {
@@ -537,7 +630,7 @@ impl Table {
let inner = self_.inner_ref()?.clone(); let inner = self_.inner_ref()?.clone();
future_into_py(self_.py(), async move { future_into_py(self_.py(), async move {
let versions = inner.list_versions().await.infer_error()?; let versions = inner.list_versions().await.infer_error()?;
let versions_as_dict = Python::attach(|py| { Python::attach(|py| {
versions versions
.iter() .iter()
.map(|v| { .map(|v| {
@@ -554,9 +647,7 @@ impl Table {
Ok(dict.unbind()) Ok(dict.unbind())
}) })
.collect::<PyResult<Vec<_>>>() .collect::<PyResult<Vec<_>>>()
}); })
versions_as_dict
}) })
} }

View File

@@ -10,11 +10,11 @@ use arrow::{
}; };
use futures::StreamExt; use futures::StreamExt;
use lancedb::{ use lancedb::{
Error,
arrow::{SendableRecordBatchStream, SimpleRecordBatchStream}, arrow::{SendableRecordBatchStream, SimpleRecordBatchStream},
data::scannable::Scannable, data::scannable::Scannable,
Error,
}; };
use pyo3::{types::PyAnyMethods, FromPyObject, Py, PyAny, Python}; use pyo3::{FromPyObject, Py, PyAny, Python, types::PyAnyMethods};
/// Adapter that implements Scannable for a Python reader factory callable. /// Adapter that implements Scannable for a Python reader factory callable.
/// ///
@@ -99,15 +99,15 @@ impl Scannable for PyScannable {
// Channel closed. Check if the task panicked — a panic // Channel closed. Check if the task panicked — a panic
// drops the sender without sending an error, so without // drops the sender without sending an error, so without
// this check we'd silently return a truncated stream. // this check we'd silently return a truncated stream.
if let Some(handle) = join_handle { if let Some(handle) = join_handle
if let Err(join_err) = handle.await { && let Err(join_err) = handle.await
return Some(( {
Err(Error::Runtime { return Some((
message: format!("Reader task panicked: {}", join_err), Err(Error::Runtime {
}), message: format!("Reader task panicked: {}", join_err),
(rx, None), }),
)); (rx, None),
} ));
} }
None None
} }

View File

@@ -5,8 +5,9 @@ use std::sync::Mutex;
use lancedb::DistanceType; use lancedb::DistanceType;
use pyo3::{ use pyo3::{
PyResult,
exceptions::{PyRuntimeError, PyValueError}, exceptions::{PyRuntimeError, PyValueError},
pyfunction, PyResult, pyfunction,
}; };
/// A wrapper around a rust builder /// A wrapper around a rust builder

387
python/tests/test_expr.py Normal file
View File

@@ -0,0 +1,387 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
"""Tests for the type-safe expression builder API."""
import pytest
import pyarrow as pa
import lancedb
from lancedb.expr import Expr, col, lit, func
# ── unit tests for Expr construction ─────────────────────────────────────────
class TestExprConstruction:
def test_col_returns_expr(self):
e = col("age")
assert isinstance(e, Expr)
def test_lit_int(self):
e = lit(42)
assert isinstance(e, Expr)
def test_lit_float(self):
e = lit(3.14)
assert isinstance(e, Expr)
def test_lit_str(self):
e = lit("hello")
assert isinstance(e, Expr)
def test_lit_bool(self):
e = lit(True)
assert isinstance(e, Expr)
def test_lit_unsupported_type_raises(self):
with pytest.raises(Exception):
lit([1, 2, 3])
def test_func(self):
e = func("lower", col("name"))
assert isinstance(e, Expr)
assert e.to_sql() == "lower(name)"
def test_func_unknown_raises(self):
with pytest.raises(Exception):
func("not_a_real_function", col("x"))
class TestExprOperators:
def test_eq_operator(self):
e = col("x") == lit(1)
assert isinstance(e, Expr)
assert e.to_sql() == "(x = 1)"
def test_ne_operator(self):
e = col("x") != lit(1)
assert isinstance(e, Expr)
assert e.to_sql() == "(x <> 1)"
def test_lt_operator(self):
e = col("age") < lit(18)
assert isinstance(e, Expr)
assert e.to_sql() == "(age < 18)"
def test_le_operator(self):
e = col("age") <= lit(18)
assert isinstance(e, Expr)
assert e.to_sql() == "(age <= 18)"
def test_gt_operator(self):
e = col("age") > lit(18)
assert isinstance(e, Expr)
assert e.to_sql() == "(age > 18)"
def test_ge_operator(self):
e = col("age") >= lit(18)
assert isinstance(e, Expr)
assert e.to_sql() == "(age >= 18)"
def test_and_operator(self):
e = (col("age") > lit(18)) & (col("status") == lit("active"))
assert isinstance(e, Expr)
assert e.to_sql() == "((age > 18) AND (status = 'active'))"
def test_or_operator(self):
e = (col("a") == lit(1)) | (col("b") == lit(2))
assert isinstance(e, Expr)
assert e.to_sql() == "((a = 1) OR (b = 2))"
def test_invert_operator(self):
e = ~(col("active") == lit(True))
assert isinstance(e, Expr)
assert e.to_sql() == "NOT (active = true)"
def test_add_operator(self):
e = col("x") + lit(1)
assert isinstance(e, Expr)
assert e.to_sql() == "(x + 1)"
def test_sub_operator(self):
e = col("x") - lit(1)
assert isinstance(e, Expr)
assert e.to_sql() == "(x - 1)"
def test_mul_operator(self):
e = col("price") * lit(1.1)
assert isinstance(e, Expr)
assert e.to_sql() == "(price * 1.1)"
def test_div_operator(self):
e = col("total") / lit(2)
assert isinstance(e, Expr)
assert e.to_sql() == "(total / 2)"
def test_radd(self):
e = lit(1) + col("x")
assert isinstance(e, Expr)
assert e.to_sql() == "(1 + x)"
def test_rmul(self):
e = lit(2) * col("x")
assert isinstance(e, Expr)
assert e.to_sql() == "(2 * x)"
def test_coerce_plain_int(self):
# Operators should auto-wrap plain Python values via lit()
e = col("age") > 18
assert isinstance(e, Expr)
assert e.to_sql() == "(age > 18)"
def test_coerce_plain_str(self):
e = col("name") == "alice"
assert isinstance(e, Expr)
assert e.to_sql() == "(name = 'alice')"
class TestExprStringMethods:
def test_lower(self):
e = col("name").lower()
assert isinstance(e, Expr)
assert e.to_sql() == "lower(name)"
def test_upper(self):
e = col("name").upper()
assert isinstance(e, Expr)
assert e.to_sql() == "upper(name)"
def test_contains(self):
e = col("text").contains(lit("hello"))
assert isinstance(e, Expr)
assert e.to_sql() == "contains(text, 'hello')"
def test_contains_with_str_coerce(self):
e = col("text").contains("hello")
assert isinstance(e, Expr)
assert e.to_sql() == "contains(text, 'hello')"
def test_chained_lower_eq(self):
e = col("name").lower() == lit("alice")
assert isinstance(e, Expr)
assert e.to_sql() == "(lower(name) = 'alice')"
class TestExprCast:
def test_cast_string(self):
e = col("id").cast("string")
assert isinstance(e, Expr)
assert e.to_sql() == "CAST(id AS VARCHAR)"
def test_cast_int32(self):
e = col("score").cast("int32")
assert isinstance(e, Expr)
assert e.to_sql() == "CAST(score AS INTEGER)"
def test_cast_float64(self):
e = col("val").cast("float64")
assert isinstance(e, Expr)
assert e.to_sql() == "CAST(val AS DOUBLE)"
def test_cast_pyarrow_type(self):
e = col("score").cast(pa.int32())
assert isinstance(e, Expr)
assert e.to_sql() == "CAST(score AS INTEGER)"
def test_cast_pyarrow_float64(self):
e = col("val").cast(pa.float64())
assert isinstance(e, Expr)
assert e.to_sql() == "CAST(val AS DOUBLE)"
def test_cast_pyarrow_string(self):
e = col("id").cast(pa.string())
assert isinstance(e, Expr)
assert e.to_sql() == "CAST(id AS VARCHAR)"
def test_cast_pyarrow_and_string_equivalent(self):
# pa.int32() and "int32" should produce equivalent SQL
sql_str = col("x").cast("int32").to_sql()
sql_pa = col("x").cast(pa.int32()).to_sql()
assert sql_str == sql_pa
class TestExprNamedMethods:
def test_eq_method(self):
e = col("x").eq(lit(1))
assert isinstance(e, Expr)
assert e.to_sql() == "(x = 1)"
def test_gt_method(self):
e = col("x").gt(lit(0))
assert isinstance(e, Expr)
assert e.to_sql() == "(x > 0)"
def test_and_method(self):
e = col("x").gt(lit(0)).and_(col("y").lt(lit(10)))
assert isinstance(e, Expr)
assert e.to_sql() == "((x > 0) AND (y < 10))"
def test_or_method(self):
e = col("x").eq(lit(1)).or_(col("x").eq(lit(2)))
assert isinstance(e, Expr)
assert e.to_sql() == "((x = 1) OR (x = 2))"
class TestExprRepr:
def test_repr(self):
e = col("age") > lit(18)
assert repr(e) == "Expr((age > 18))"
def test_to_sql(self):
e = col("age") > 18
assert e.to_sql() == "(age > 18)"
def test_unhashable(self):
e = col("x")
with pytest.raises(TypeError):
{e: 1}
# ── integration tests: end-to-end query against a real table ─────────────────
@pytest.fixture
def simple_table(tmp_path):
db = lancedb.connect(str(tmp_path))
data = pa.table(
{
"id": [1, 2, 3, 4, 5],
"name": ["Alice", "Bob", "Charlie", "alice", "BOB"],
"age": [25, 17, 30, 22, 15],
"score": [1.5, 2.0, 3.5, 4.0, 0.5],
}
)
return db.create_table("test", data)
class TestExprFilter:
def test_simple_gt_filter(self, simple_table):
result = simple_table.search().where(col("age") > lit(20)).to_arrow()
assert result.num_rows == 3 # ages 25, 30, 22
def test_compound_and_filter(self, simple_table):
result = (
simple_table.search()
.where((col("age") > lit(18)) & (col("score") > lit(2.0)))
.to_arrow()
)
assert result.num_rows == 2 # (30, 3.5) and (22, 4.0)
def test_string_equality_filter(self, simple_table):
result = simple_table.search().where(col("name") == lit("Bob")).to_arrow()
assert result.num_rows == 1
def test_or_filter(self, simple_table):
result = (
simple_table.search()
.where((col("age") < lit(18)) | (col("age") > lit(28)))
.to_arrow()
)
assert result.num_rows == 3 # ages 17, 30, 15
def test_coercion_no_lit(self, simple_table):
# Python values should be auto-coerced
result = simple_table.search().where(col("age") > 20).to_arrow()
assert result.num_rows == 3
def test_string_sql_still_works(self, simple_table):
# Backwards compatibility: plain strings still accepted
result = simple_table.search().where("age > 20").to_arrow()
assert result.num_rows == 3
class TestExprProjection:
def test_select_with_expr(self, simple_table):
result = (
simple_table.search()
.select({"double_score": col("score") * lit(2)})
.to_arrow()
)
assert "double_score" in result.schema.names
def test_select_mixed_str_and_expr(self, simple_table):
result = (
simple_table.search()
.select({"id": "id", "double_score": col("score") * lit(2)})
.to_arrow()
)
assert "id" in result.schema.names
assert "double_score" in result.schema.names
def test_select_list_of_columns(self, simple_table):
# Plain list of str still works
result = simple_table.search().select(["id", "name"]).to_arrow()
assert result.schema.names == ["id", "name"]
# ── column name edge cases ────────────────────────────────────────────────────
class TestColNaming:
"""Unit tests verifying that col() preserves identifiers exactly.
Identifiers that need quoting (camelCase, spaces, leading digits, unicode)
are wrapped in backticks to match the lance SQL parser's dialect.
"""
def test_camel_case_preserved_in_sql(self):
# camelCase is quoted with backticks so the case round-trips correctly.
assert col("firstName").to_sql() == "`firstName`"
def test_camel_case_in_expression(self):
assert (col("firstName") > lit(18)).to_sql() == "(`firstName` > 18)"
def test_space_in_name_quoted(self):
assert col("first name").to_sql() == "`first name`"
def test_space_in_expression(self):
assert (col("first name") == lit("A")).to_sql() == "(`first name` = 'A')"
def test_leading_digit_quoted(self):
assert col("2fast").to_sql() == "`2fast`"
def test_unicode_quoted(self):
assert col("名前").to_sql() == "`名前`"
def test_snake_case_unquoted(self):
# Plain snake_case needs no quoting.
assert col("first_name").to_sql() == "first_name"
@pytest.fixture
def special_col_table(tmp_path):
db = lancedb.connect(str(tmp_path))
data = pa.table(
{
"firstName": ["Alice", "Bob", "Charlie"],
"first name": ["A", "B", "C"],
"score": [10, 20, 30],
}
)
return db.create_table("special", data)
class TestColNamingIntegration:
def test_camel_case_filter(self, special_col_table):
result = (
special_col_table.search()
.where(col("firstName") == lit("Alice"))
.to_arrow()
)
assert result.num_rows == 1
assert result["firstName"][0].as_py() == "Alice"
def test_space_in_col_filter(self, special_col_table):
result = (
special_col_table.search().where(col("first name") == lit("B")).to_arrow()
)
assert result.num_rows == 1
def test_camel_case_projection(self, special_col_table):
result = (
special_col_table.search()
.select({"upper_name": col("firstName").upper()})
.to_arrow()
)
assert "upper_name" in result.schema.names
assert sorted(result["upper_name"].to_pylist()) == ["ALICE", "BOB", "CHARLIE"]

301
python/uv.lock generated
View File

@@ -247,7 +247,7 @@ wheels = [
[[package]] [[package]]
name = "awscli" name = "awscli"
version = "1.44.35" version = "1.44.70"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
dependencies = [ dependencies = [
{ name = "botocore" }, { name = "botocore" },
@@ -257,9 +257,9 @@ dependencies = [
{ name = "rsa" }, { name = "rsa" },
{ name = "s3transfer" }, { name = "s3transfer" },
] ]
sdist = { url = "https://files.pythonhosted.org/packages/f3/42/58705761bce0d24c4496aac146d724a8caf20a33d906ec954729c934088b/awscli-1.44.35.tar.gz", hash = "sha256:bc38774bfc71fd33112fd283522b010c2f5b606e57b28a85884d96e8051c58e7", size = 1888844, upload-time = "2026-02-09T21:50:10.697Z" } sdist = { url = "https://files.pythonhosted.org/packages/76/b2/0f522e76ca173ac06949883f00994ec173f9336c8f8146f982458ebc6ce7/awscli-1.44.70.tar.gz", hash = "sha256:25eafa6237a2ff9ad98c8bb486c40f904996db5fb3e9facc8cba9108caa7859c", size = 1886989, upload-time = "2026-03-31T19:33:41.249Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/cd/94/df482d7f36ffc0f8b973258aa3fc2cd33deef0c06a1ec0f228e55d79ed9a/awscli-1.44.35-py3-none-any.whl", hash = "sha256:0823c1af8926a3bd10db652d8b64d61cfbf34268be845aca332ea7aea0c1ac15", size = 4641343, upload-time = "2026-02-09T21:50:06.323Z" }, { url = "https://files.pythonhosted.org/packages/c1/eb/98028053ab43a723ddcfd0322b5d5929f413211ae6af834852e064493e68/awscli-1.44.70-py3-none-any.whl", hash = "sha256:eb742517feca3be3b6567c3f302de6b5a3a12b18b61e530509d6e098e243771f", size = 4624874, upload-time = "2026-03-31T19:33:37.912Z" },
] ]
[[package]] [[package]]
@@ -408,16 +408,16 @@ wheels = [
[[package]] [[package]]
name = "botocore" name = "botocore"
version = "1.42.45" version = "1.42.80"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
dependencies = [ dependencies = [
{ name = "jmespath" }, { name = "jmespath" },
{ name = "python-dateutil" }, { name = "python-dateutil" },
{ name = "urllib3" }, { name = "urllib3" },
] ]
sdist = { url = "https://files.pythonhosted.org/packages/7a/b1/c36ad705d67bb935eac3085052b5dc03ec22d5ac12e7aedf514f3d76cac8/botocore-1.42.45.tar.gz", hash = "sha256:40b577d07b91a0ed26879da9e4658d82d3a400382446af1014d6ad3957497545", size = 14941217, upload-time = "2026-02-09T21:50:01.966Z" } sdist = { url = "https://files.pythonhosted.org/packages/2e/42/d0ce09fe5b494e2a9de513206dec90fbe72bcb101457a60f526a6b1c300b/botocore-1.42.80.tar.gz", hash = "sha256:fe32af53dc87f5f4d61879bc231e2ca2cc0719b19b8f6d268e82a34f713a8a09", size = 15110373, upload-time = "2026-03-31T19:33:33.82Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/7e/ec/6681b8e4884f8663d7650220e702c503e4ba6bd09a5b91d44803b0b1d0a8/botocore-1.42.45-py3-none-any.whl", hash = "sha256:a5ea5d1b7c46c2d5d113879e45b21eaf7d60dc865f4bcb46dfcf0703fe3429f4", size = 14615557, upload-time = "2026-02-09T21:49:57.066Z" }, { url = "https://files.pythonhosted.org/packages/17/b0/c03f2ed8e7817db1c22d70720636a1b22a2a4d3aa3c09da0257072b30bc5/botocore-1.42.80-py3-none-any.whl", hash = "sha256:7291632b2ede71b7c69e6e366480bb6e2a5d2fae8f7d2d2eb49215e32b7c7a12", size = 14787168, upload-time = "2026-03-31T19:33:29.396Z" },
] ]
[[package]] [[package]]
@@ -750,19 +750,19 @@ wheels = [
[[package]] [[package]]
name = "datafusion" name = "datafusion"
version = "51.0.0" version = "52.3.0"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
dependencies = [ dependencies = [
{ name = "pyarrow" }, { name = "pyarrow" },
{ name = "typing-extensions", marker = "python_full_version < '3.13'" }, { name = "typing-extensions", marker = "python_full_version < '3.13'" },
] ]
sdist = { url = "https://files.pythonhosted.org/packages/2c/6d/d0e2632c93bbcca0687eeda672af3f92042ecd349df7be55da86253594a9/datafusion-51.0.0.tar.gz", hash = "sha256:1887c7d5ed3ae5d9f389e62ba869864afad4006a3f7c99ef0ca4707782a7838f", size = 193751, upload-time = "2026-01-09T13:23:41.562Z" } sdist = { url = "https://files.pythonhosted.org/packages/db/d4/a5ad7b665a80008901892fde61dc667318db0652a955d706ddca3a224b5a/datafusion-52.3.0.tar.gz", hash = "sha256:2e8b02ad142b1a0d673f035d96a0944a640ac78275003d7e453cee4afe4a20a4", size = 205026, upload-time = "2026-03-16T10:54:07.739Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/cf/a9/7717cec053a3309be3020fe3147e3f76e5bf21295fa8adf9b52dd44ea3ff/datafusion-51.0.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:0c0d265fe3ee0dcbfa7cc3c64c7cd94fc493f38418bd79debb7ec29f29b7176e", size = 30389413, upload-time = "2026-01-09T13:23:23.266Z" }, { url = "https://files.pythonhosted.org/packages/55/63/1bb0737988cefa77274b459d64fa4b57ba4cf755639a39733e9581b5d599/datafusion-52.3.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:a73f02406b2985b9145dd97f8221a929c9ef3289a8ba64c6b52043e240938528", size = 31503230, upload-time = "2026-03-16T10:53:50.312Z" },
{ url = "https://files.pythonhosted.org/packages/55/45/72c9874fd3740a4cb9d55049fdbae0df512dc5433e9f1176f3cfd970f1a1/datafusion-51.0.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:43e6011db86e950bf9a21ed73cc089c2346b340a41a4f1044268af6c3a357acc", size = 26982206, upload-time = "2026-01-09T13:23:27.437Z" }, { url = "https://files.pythonhosted.org/packages/d6/e3/ea3b79239953c3044d19d8e9581015da025b6640796db03799e435b17910/datafusion-52.3.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:118a1f0add6a3f91fcbc90c71819fe08750e2981637d5e7b346e099e94a20b8b", size = 28159497, upload-time = "2026-03-16T10:53:54.032Z" },
{ url = "https://files.pythonhosted.org/packages/21/ac/b32ba1f25d38fc16e7623cc4bfb7bd68db61be2ef27b2d9969ea5c865765/datafusion-51.0.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e76803907150159aa059d5cc9291645bbaac1b6a46d07e56035118d327b741ae", size = 33246117, upload-time = "2026-01-09T13:23:30.981Z" }, { url = "https://files.pythonhosted.org/packages/24/c8/7d325feb4b7509ae03857fd7e164e95ec72e8c9f3dfd3178ec7f80d53977/datafusion-52.3.0-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:253ce7aee5fe84bd6ee290c20608114114bdb5115852617f97d3855d36ad9341", size = 30769154, upload-time = "2026-03-16T10:53:57.835Z" },
{ url = "https://files.pythonhosted.org/packages/0b/4e/437121422ef010690fc3cdd7f080203e986ba00e0e3c3b577e03f5b54ca2/datafusion-51.0.0-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:9d0cfabfe1853994adc2e6e9da5f36c1eb061102e34a2f1101fa935c6991c9e1", size = 31421867, upload-time = "2026-01-09T13:23:34.436Z" }, { url = "https://files.pythonhosted.org/packages/37/ee/478689c69b3cb1ccabb2d52feac0c181f6cdf20b51a81df35344b1dab9a6/datafusion-52.3.0-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:2af3469d2f06959bec88579ab107a72f965de18b32e607069bbdd0b859ed8dbb", size = 33060335, upload-time = "2026-03-16T10:54:01.715Z" },
{ url = "https://files.pythonhosted.org/packages/db/fc/58cf27fcb85b2fd2a698253ae46213b1cbda784407e205c148f4006c1429/datafusion-51.0.0-cp310-abi3-win_amd64.whl", hash = "sha256:fd5f9abfd6669062debf0658d13e4583234c89d4df95faf381927b11cea411f5", size = 32517679, upload-time = "2026-01-09T13:23:39.615Z" }, { url = "https://files.pythonhosted.org/packages/1c/48/01906ab5c1a70373c6874ac5192d03646fa7b94d9ff06e3f676cb6b0f43f/datafusion-52.3.0-cp310-abi3-win_amd64.whl", hash = "sha256:9fb35738cf4dbff672dbcfffc7332813024cb0ad2ab8cda1fb90b9054277ab0c", size = 33765807, upload-time = "2026-03-16T10:54:05.728Z" },
] ]
[[package]] [[package]]
@@ -1251,6 +1251,7 @@ dependencies = [
{ name = "griffecli" }, { name = "griffecli" },
{ name = "griffelib" }, { name = "griffelib" },
] ]
sdist = { url = "https://files.pythonhosted.org/packages/04/56/28a0accac339c164b52a92c6cfc45a903acc0c174caa5c1713803467b533/griffe-2.0.0.tar.gz", hash = "sha256:c68979cd8395422083a51ea7cf02f9c119d889646d99b7b656ee43725de1b80f", size = 293906, upload-time = "2026-03-23T21:06:53.402Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/8b/94/ee21d41e7eb4f823b94603b9d40f86d3c7fde80eacc2c3c71845476dddaa/griffe-2.0.0-py3-none-any.whl", hash = "sha256:5418081135a391c3e6e757a7f3f156f1a1a746cc7b4023868ff7d5e2f9a980aa", size = 5214, upload-time = "2026-02-09T19:09:44.105Z" }, { url = "https://files.pythonhosted.org/packages/8b/94/ee21d41e7eb4f823b94603b9d40f86d3c7fde80eacc2c3c71845476dddaa/griffe-2.0.0-py3-none-any.whl", hash = "sha256:5418081135a391c3e6e757a7f3f156f1a1a746cc7b4023868ff7d5e2f9a980aa", size = 5214, upload-time = "2026-02-09T19:09:44.105Z" },
] ]
@@ -1263,6 +1264,7 @@ dependencies = [
{ name = "colorama" }, { name = "colorama" },
{ name = "griffelib" }, { name = "griffelib" },
] ]
sdist = { url = "https://files.pythonhosted.org/packages/a4/f8/2e129fd4a86e52e58eefe664de05e7d502decf766e7316cc9e70fdec3e18/griffecli-2.0.0.tar.gz", hash = "sha256:312fa5ebb4ce6afc786356e2d0ce85b06c1c20d45abc42d74f0cda65e159f6ef", size = 56213, upload-time = "2026-03-23T21:06:54.8Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/e6/ed/d93f7a447bbf7a935d8868e9617cbe1cadf9ee9ee6bd275d3040fbf93d60/griffecli-2.0.0-py3-none-any.whl", hash = "sha256:9f7cd9ee9b21d55e91689358978d2385ae65c22f307a63fb3269acf3f21e643d", size = 9345, upload-time = "2026-02-09T19:09:42.554Z" }, { url = "https://files.pythonhosted.org/packages/e6/ed/d93f7a447bbf7a935d8868e9617cbe1cadf9ee9ee6bd275d3040fbf93d60/griffecli-2.0.0-py3-none-any.whl", hash = "sha256:9f7cd9ee9b21d55e91689358978d2385ae65c22f307a63fb3269acf3f21e643d", size = 9345, upload-time = "2026-02-09T19:09:42.554Z" },
] ]
@@ -1271,6 +1273,7 @@ wheels = [
name = "griffelib" name = "griffelib"
version = "2.0.0" version = "2.0.0"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/ad/06/eccbd311c9e2b3ca45dbc063b93134c57a1ccc7607c5e545264ad092c4a9/griffelib-2.0.0.tar.gz", hash = "sha256:e504d637a089f5cab9b5daf18f7645970509bf4f53eda8d79ed71cce8bd97934", size = 166312, upload-time = "2026-03-23T21:06:55.954Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/4d/51/c936033e16d12b627ea334aaaaf42229c37620d0f15593456ab69ab48161/griffelib-2.0.0-py3-none-any.whl", hash = "sha256:01284878c966508b6d6f1dbff9b6fa607bc062d8261c5c7253cb285b06422a7f", size = 142004, upload-time = "2026-02-09T19:09:40.561Z" }, { url = "https://files.pythonhosted.org/packages/4d/51/c936033e16d12b627ea334aaaaf42229c37620d0f15593456ab69ab48161/griffelib-2.0.0-py3-none-any.whl", hash = "sha256:01284878c966508b6d6f1dbff9b6fa607bc062d8261c5c7253cb285b06422a7f", size = 142004, upload-time = "2026-02-09T19:09:40.561Z" },
] ]
@@ -1888,19 +1891,19 @@ wheels = [
[[package]] [[package]]
name = "lance-namespace" name = "lance-namespace"
version = "0.4.5" version = "0.6.1"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
dependencies = [ dependencies = [
{ name = "lance-namespace-urllib3-client" }, { name = "lance-namespace-urllib3-client" },
] ]
sdist = { url = "https://files.pythonhosted.org/packages/b4/b5/0c3c55cf336b1e90392c2e24ac833551659e8bb3c61644b2d94825eb31bd/lance_namespace-0.4.5.tar.gz", hash = "sha256:0aee0abed3a1fa762c2955c7d12bb3004cea5c82ba28f6fcb9fe79d0cc19e317", size = 9827, upload-time = "2026-01-07T19:20:23.005Z" } sdist = { url = "https://files.pythonhosted.org/packages/28/9f/7906ba4117df8d965510285eaf07264a77de2fd283b9d44ec7fc63a4a57a/lance_namespace-0.6.1.tar.gz", hash = "sha256:f0deea442bd3f1056a8e2fed056ae2778e3356517ec2e680db049058b824d131", size = 10666, upload-time = "2026-03-17T17:55:44.977Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/34/88/173687dad72baf819223e3b506898e386bc88c26ff8da5e8013291e02daf/lance_namespace-0.4.5-py3-none-any.whl", hash = "sha256:cd1a4f789de03ba23a0c16f100b1464cca572a5d04e428917a54d09db912d548", size = 11703, upload-time = "2026-01-07T19:20:25.394Z" }, { url = "https://files.pythonhosted.org/packages/d1/91/aee1c0a04d17f2810173bd304bd444eb78332045df1b0c1b07cebd01f530/lance_namespace-0.6.1-py3-none-any.whl", hash = "sha256:9699c9e3f12236e5e08ea979cc4e036a8e3c67ed2f37ae6f25c5353ab908e1be", size = 12498, upload-time = "2026-03-17T17:55:44.062Z" },
] ]
[[package]] [[package]]
name = "lance-namespace-urllib3-client" name = "lance-namespace-urllib3-client"
version = "0.4.5" version = "0.6.1"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
dependencies = [ dependencies = [
{ name = "pydantic" }, { name = "pydantic" },
@@ -1908,9 +1911,9 @@ dependencies = [
{ name = "typing-extensions" }, { name = "typing-extensions" },
{ name = "urllib3" }, { name = "urllib3" },
] ]
sdist = { url = "https://files.pythonhosted.org/packages/97/a9/4e527c2f05704565618b239b0965f829d1a194837f01234af3f8e2f33d92/lance_namespace_urllib3_client-0.4.5.tar.gz", hash = "sha256:184deda8cf8700926d994618187053c644eb1f2866a4479e7b80843cacc92b1c", size = 159726, upload-time = "2026-01-07T19:20:24.025Z" } sdist = { url = "https://files.pythonhosted.org/packages/63/a1/8706a2be25bd184acccc411e48f1a42a4cbf3b6556cba15b9fcf4c15cfcc/lance_namespace_urllib3_client-0.6.1.tar.gz", hash = "sha256:31fbd058ce1ea0bf49045cdeaa756360ece0bc61e9e10276f41af6d217debe87", size = 182567, upload-time = "2026-03-17T17:55:46.87Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/ca/86/0adee7190408a28dcc5a0562c674537457e3de59ee51d1c724ecdc4a9930/lance_namespace_urllib3_client-0.4.5-py3-none-any.whl", hash = "sha256:2ee154d616ba4721f0bfdf043d33c4fef2e79d380653e2f263058ab00fb4adf4", size = 277969, upload-time = "2026-01-07T19:20:26.597Z" }, { url = "https://files.pythonhosted.org/packages/cd/c7/cb9580602dec25f0fdd6005c1c9ba1d4c8c0c3dc8d543107e5a9f248bba8/lance_namespace_urllib3_client-0.6.1-py3-none-any.whl", hash = "sha256:b9c103e1377ad46d2bd70eec894bfec0b1e2133dae0964d7e4de543c6e16293b", size = 317111, upload-time = "2026-03-17T17:55:45.546Z" },
] ]
[[package]] [[package]]
@@ -1999,57 +2002,57 @@ tests = [
[package.metadata] [package.metadata]
requires-dist = [ requires-dist = [
{ name = "adlfs", marker = "extra == 'azure'", specifier = ">=2024.2.0" }, { name = "adlfs", marker = "extra == 'azure'", specifier = ">=2024.2.0" },
{ name = "aiohttp", marker = "extra == 'tests'" }, { name = "aiohttp", marker = "extra == 'tests'", specifier = ">=3.9.0" },
{ name = "awscli", marker = "extra == 'embeddings'", specifier = ">=1.29.57" }, { name = "awscli", marker = "extra == 'embeddings'", specifier = ">=1.44.38" },
{ name = "boto3", marker = "extra == 'embeddings'", specifier = ">=1.28.57" }, { name = "boto3", marker = "extra == 'embeddings'", specifier = ">=1.28.57" },
{ name = "boto3", marker = "extra == 'tests'" }, { name = "boto3", marker = "extra == 'tests'", specifier = ">=1.28.57" },
{ name = "botocore", marker = "extra == 'embeddings'", specifier = ">=1.31.57" }, { name = "botocore", marker = "extra == 'embeddings'", specifier = ">=1.31.57" },
{ name = "cohere", marker = "extra == 'embeddings'" }, { name = "cohere", marker = "extra == 'embeddings'", specifier = ">=4.0" },
{ name = "colpali-engine", marker = "extra == 'embeddings'", specifier = ">=0.3.10" }, { name = "colpali-engine", marker = "extra == 'embeddings'", specifier = ">=0.3.10" },
{ name = "datafusion", marker = "extra == 'tests'" }, { name = "datafusion", marker = "extra == 'tests'", specifier = ">=52,<53" },
{ name = "deprecation" }, { name = "deprecation", specifier = ">=2.1.0" },
{ name = "duckdb", marker = "extra == 'tests'" }, { name = "duckdb", marker = "extra == 'tests'", specifier = ">=0.9.0" },
{ name = "google-generativeai", marker = "extra == 'embeddings'" }, { name = "google-generativeai", marker = "extra == 'embeddings'", specifier = ">=0.3.0" },
{ name = "huggingface-hub", marker = "extra == 'embeddings'" }, { name = "huggingface-hub", marker = "extra == 'embeddings'", specifier = ">=0.19.0" },
{ name = "ibm-watsonx-ai", marker = "python_full_version >= '3.10' and extra == 'embeddings'", specifier = ">=1.1.2" }, { name = "ibm-watsonx-ai", marker = "python_full_version >= '3.10' and extra == 'embeddings'", specifier = ">=1.1.2" },
{ name = "instructorembedding", marker = "extra == 'embeddings'" }, { name = "instructorembedding", marker = "extra == 'embeddings'", specifier = ">=1.0.1" },
{ name = "lance-namespace", specifier = ">=0.3.2" }, { name = "lance-namespace", specifier = ">=0.3.2" },
{ name = "mkdocs", marker = "extra == 'docs'" }, { name = "mkdocs", marker = "extra == 'docs'" },
{ name = "mkdocs-jupyter", marker = "extra == 'docs'" }, { name = "mkdocs-jupyter", marker = "extra == 'docs'" },
{ name = "mkdocs-material", marker = "extra == 'docs'" }, { name = "mkdocs-material", marker = "extra == 'docs'" },
{ name = "mkdocstrings-python", marker = "extra == 'docs'" }, { name = "mkdocstrings-python", marker = "extra == 'docs'" },
{ name = "numpy" }, { name = "numpy", specifier = ">=1.24.0" },
{ name = "ollama", marker = "extra == 'embeddings'", specifier = ">=0.3.0" }, { name = "ollama", marker = "extra == 'embeddings'", specifier = ">=0.3.0" },
{ name = "open-clip-torch", marker = "extra == 'clip'" }, { name = "open-clip-torch", marker = "extra == 'clip'" },
{ name = "open-clip-torch", marker = "extra == 'embeddings'" }, { name = "open-clip-torch", marker = "extra == 'embeddings'", specifier = ">=2.20.0" },
{ name = "openai", marker = "extra == 'embeddings'", specifier = ">=1.6.1" }, { name = "openai", marker = "extra == 'embeddings'", specifier = ">=1.6.1" },
{ name = "overrides", marker = "python_full_version < '3.12'", specifier = ">=0.7" }, { name = "overrides", marker = "python_full_version < '3.12'", specifier = ">=0.7" },
{ name = "packaging" }, { name = "packaging", specifier = ">=23.0" },
{ name = "pandas", marker = "extra == 'tests'", specifier = ">=1.4" }, { name = "pandas", marker = "extra == 'tests'", specifier = ">=1.4" },
{ name = "pillow", marker = "extra == 'clip'" }, { name = "pillow", marker = "extra == 'clip'", specifier = ">=12.1.1" },
{ name = "pillow", marker = "extra == 'embeddings'" }, { name = "pillow", marker = "extra == 'embeddings'", specifier = ">=12.1.1" },
{ name = "pillow", marker = "extra == 'siglip'" }, { name = "pillow", marker = "extra == 'siglip'", specifier = ">=12.1.1" },
{ name = "polars", marker = "extra == 'tests'", specifier = ">=0.19,<=1.3.0" }, { name = "polars", marker = "extra == 'tests'", specifier = ">=0.19,<=1.3.0" },
{ name = "pre-commit", marker = "extra == 'dev'" }, { name = "pre-commit", marker = "extra == 'dev'", specifier = ">=3.5.0" },
{ name = "pyarrow", specifier = ">=16" }, { name = "pyarrow", specifier = ">=16" },
{ name = "pyarrow-stubs", marker = "extra == 'tests'" }, { name = "pyarrow-stubs", marker = "extra == 'tests'", specifier = ">=16.0" },
{ name = "pydantic", specifier = ">=1.10" }, { name = "pydantic", specifier = ">=1.10" },
{ name = "pylance", marker = "extra == 'pylance'", specifier = ">=1.0.0b14" }, { name = "pylance", marker = "extra == 'pylance'", specifier = ">=4.0.0b7" },
{ name = "pylance", marker = "extra == 'tests'", specifier = ">=1.0.0b14" }, { name = "pylance", marker = "extra == 'tests'", specifier = ">=4.0.0b7" },
{ name = "pyright", marker = "extra == 'dev'" }, { name = "pyright", marker = "extra == 'dev'", specifier = ">=1.1.350" },
{ name = "pytest", marker = "extra == 'tests'" }, { name = "pytest", marker = "extra == 'tests'", specifier = ">=7.0" },
{ name = "pytest-asyncio", marker = "extra == 'tests'" }, { name = "pytest-asyncio", marker = "extra == 'tests'", specifier = ">=0.21" },
{ name = "pytest-mock", marker = "extra == 'tests'" }, { name = "pytest-mock", marker = "extra == 'tests'", specifier = ">=3.10" },
{ name = "pytz", marker = "extra == 'tests'" }, { name = "pytz", marker = "extra == 'tests'", specifier = ">=2023.3" },
{ name = "requests", marker = "extra == 'embeddings'", specifier = ">=2.31.0" }, { name = "requests", marker = "extra == 'embeddings'", specifier = ">=2.31.0" },
{ name = "requests", marker = "extra == 'tests'" }, { name = "requests", marker = "extra == 'tests'", specifier = ">=2.31.0" },
{ name = "ruff", marker = "extra == 'dev'" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.3.0" },
{ name = "sentence-transformers", marker = "extra == 'embeddings'" }, { name = "sentence-transformers", marker = "extra == 'embeddings'", specifier = ">=2.2.0" },
{ name = "sentencepiece", marker = "extra == 'embeddings'" }, { name = "sentencepiece", marker = "extra == 'embeddings'", specifier = ">=0.1.99" },
{ name = "sentencepiece", marker = "extra == 'siglip'" }, { name = "sentencepiece", marker = "extra == 'siglip'" },
{ name = "tantivy", marker = "extra == 'tests'" }, { name = "tantivy", marker = "extra == 'tests'", specifier = ">=0.20.0" },
{ name = "torch", marker = "extra == 'clip'" }, { name = "torch", marker = "extra == 'clip'" },
{ name = "torch", marker = "extra == 'embeddings'" }, { name = "torch", marker = "extra == 'embeddings'", specifier = ">=2.0.0" },
{ name = "torch", marker = "extra == 'siglip'" }, { name = "torch", marker = "extra == 'siglip'" },
{ name = "tqdm", specifier = ">=4.27.0" }, { name = "tqdm", specifier = ">=4.27.0" },
{ name = "transformers", marker = "extra == 'siglip'", specifier = ">=4.41.0" }, { name = "transformers", marker = "extra == 'siglip'", specifier = ">=4.41.0" },
@@ -3169,100 +3172,100 @@ wheels = [
[[package]] [[package]]
name = "pillow" name = "pillow"
version = "12.1.0" version = "12.1.1"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/d0/02/d52c733a2452ef1ffcc123b68e6606d07276b0e358db70eabad7e40042b7/pillow-12.1.0.tar.gz", hash = "sha256:5c5ae0a06e9ea030ab786b0251b32c7e4ce10e58d983c0d5c56029455180b5b9", size = 46977283, upload-time = "2026-01-02T09:13:29.892Z" } sdist = { url = "https://files.pythonhosted.org/packages/1f/42/5c74462b4fd957fcd7b13b04fb3205ff8349236ea74c7c375766d6c82288/pillow-12.1.1.tar.gz", hash = "sha256:9ad8fa5937ab05218e2b6a4cff30295ad35afd2f83ac592e68c0d871bb0fdbc4", size = 46980264, upload-time = "2026-02-11T04:23:07.146Z" }
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/fe/41/f73d92b6b883a579e79600d391f2e21cb0df767b2714ecbd2952315dfeef/pillow-12.1.0-cp310-cp310-macosx_10_10_x86_64.whl", hash = "sha256:fb125d860738a09d363a88daa0f59c4533529a90e564785e20fe875b200b6dbd", size = 5304089, upload-time = "2026-01-02T09:10:24.953Z" }, { url = "https://files.pythonhosted.org/packages/1d/30/5bd3d794762481f8c8ae9c80e7b76ecea73b916959eb587521358ef0b2f9/pillow-12.1.1-cp310-cp310-macosx_10_10_x86_64.whl", hash = "sha256:1f1625b72740fdda5d77b4def688eb8fd6490975d06b909fd19f13f391e077e0", size = 5304099, upload-time = "2026-02-11T04:20:06.13Z" },
{ url = "https://files.pythonhosted.org/packages/94/55/7aca2891560188656e4a91ed9adba305e914a4496800da6b5c0a15f09edf/pillow-12.1.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:cad302dc10fac357d3467a74a9561c90609768a6f73a1923b0fd851b6486f8b0", size = 4657815, upload-time = "2026-01-02T09:10:27.063Z" }, { url = "https://files.pythonhosted.org/packages/bd/c1/aab9e8f3eeb4490180e357955e15c2ef74b31f64790ff356c06fb6cf6d84/pillow-12.1.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:178aa072084bd88ec759052feca8e56cbb14a60b39322b99a049e58090479713", size = 4657880, upload-time = "2026-02-11T04:20:09.291Z" },
{ url = "https://files.pythonhosted.org/packages/e9/d2/b28221abaa7b4c40b7dba948f0f6a708bd7342c4d47ce342f0ea39643974/pillow-12.1.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a40905599d8079e09f25027423aed94f2823adaf2868940de991e53a449e14a8", size = 6222593, upload-time = "2026-01-02T09:10:29.115Z" }, { url = "https://files.pythonhosted.org/packages/f1/0a/9879e30d56815ad529d3985aeff5af4964202425c27261a6ada10f7cbf53/pillow-12.1.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b66e95d05ba806247aaa1561f080abc7975daf715c30780ff92a20e4ec546e1b", size = 6222587, upload-time = "2026-02-11T04:20:10.82Z" },
{ url = "https://files.pythonhosted.org/packages/71/b8/7a61fb234df6a9b0b479f69e66901209d89ff72a435b49933f9122f94cac/pillow-12.1.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:92a7fe4225365c5e3a8e598982269c6d6698d3e783b3b1ae979e7819f9cd55c1", size = 8027579, upload-time = "2026-01-02T09:10:31.182Z" }, { url = "https://files.pythonhosted.org/packages/5a/5f/a1b72ff7139e4f89014e8d451442c74a774d5c43cd938fb0a9f878576b37/pillow-12.1.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:89c7e895002bbe49cdc5426150377cbbc04767d7547ed145473f496dfa40408b", size = 8027678, upload-time = "2026-02-11T04:20:12.455Z" },
{ url = "https://files.pythonhosted.org/packages/ea/51/55c751a57cc524a15a0e3db20e5cde517582359508d62305a627e77fd295/pillow-12.1.0-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f10c98f49227ed8383d28174ee95155a675c4ed7f85e2e573b04414f7e371bda", size = 6335760, upload-time = "2026-01-02T09:10:33.02Z" }, { url = "https://files.pythonhosted.org/packages/e2/c2/c7cb187dac79a3d22c3ebeae727abee01e077c8c7d930791dc592f335153/pillow-12.1.1-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a5cbdcddad0af3da87cb16b60d23648bc3b51967eb07223e9fed77a82b457c4", size = 6335777, upload-time = "2026-02-11T04:20:14.441Z" },
{ url = "https://files.pythonhosted.org/packages/dc/7c/60e3e6f5e5891a1a06b4c910f742ac862377a6fe842f7184df4a274ce7bf/pillow-12.1.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8637e29d13f478bc4f153d8daa9ffb16455f0a6cb287da1b432fdad2bfbd66c7", size = 7027127, upload-time = "2026-01-02T09:10:35.009Z" }, { url = "https://files.pythonhosted.org/packages/0c/7b/f9b09a7804ec7336effb96c26d37c29d27225783dc1501b7d62dcef6ae25/pillow-12.1.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9f51079765661884a486727f0729d29054242f74b46186026582b4e4769918e4", size = 7027140, upload-time = "2026-02-11T04:20:16.387Z" },
{ url = "https://files.pythonhosted.org/packages/06/37/49d47266ba50b00c27ba63a7c898f1bb41a29627ced8c09e25f19ebec0ff/pillow-12.1.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:21e686a21078b0f9cb8c8a961d99e6a4ddb88e0fc5ea6e130172ddddc2e5221a", size = 6449896, upload-time = "2026-01-02T09:10:36.793Z" }, { url = "https://files.pythonhosted.org/packages/98/b2/2fa3c391550bd421b10849d1a2144c44abcd966daadd2f7c12e19ea988c4/pillow-12.1.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:99c1506ea77c11531d75e3a412832a13a71c7ebc8192ab9e4b2e355555920e3e", size = 6449855, upload-time = "2026-02-11T04:20:18.554Z" },
{ url = "https://files.pythonhosted.org/packages/f9/e5/67fd87d2913902462cd9b79c6211c25bfe95fcf5783d06e1367d6d9a741f/pillow-12.1.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:2415373395a831f53933c23ce051021e79c8cd7979822d8cc478547a3f4da8ef", size = 7151345, upload-time = "2026-01-02T09:10:39.064Z" }, { url = "https://files.pythonhosted.org/packages/96/ff/9caf4b5b950c669263c39e96c78c0d74a342c71c4f43fd031bb5cb7ceac9/pillow-12.1.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:36341d06738a9f66c8287cf8b876d24b18db9bd8740fa0672c74e259ad408cff", size = 7151329, upload-time = "2026-02-11T04:20:20.646Z" },
{ url = "https://files.pythonhosted.org/packages/bd/15/f8c7abf82af68b29f50d77c227e7a1f87ce02fdc66ded9bf603bc3b41180/pillow-12.1.0-cp310-cp310-win32.whl", hash = "sha256:e75d3dba8fc1ddfec0cd752108f93b83b4f8d6ab40e524a95d35f016b9683b09", size = 6325568, upload-time = "2026-01-02T09:10:41.035Z" }, { url = "https://files.pythonhosted.org/packages/7b/f8/4b24841f582704da675ca535935bccb32b00a6da1226820845fac4a71136/pillow-12.1.1-cp310-cp310-win32.whl", hash = "sha256:6c52f062424c523d6c4db85518774cc3d50f5539dd6eed32b8f6229b26f24d40", size = 6325574, upload-time = "2026-02-11T04:20:22.43Z" },
{ url = "https://files.pythonhosted.org/packages/d4/24/7d1c0e160b6b5ac2605ef7d8be537e28753c0db5363d035948073f5513d7/pillow-12.1.0-cp310-cp310-win_amd64.whl", hash = "sha256:64efdf00c09e31efd754448a383ea241f55a994fd079866b92d2bbff598aad91", size = 7032367, upload-time = "2026-01-02T09:10:43.09Z" }, { url = "https://files.pythonhosted.org/packages/f8/f9/9f6b01c0881d7036063aa6612ef04c0e2cad96be21325a1e92d0203f8e91/pillow-12.1.1-cp310-cp310-win_amd64.whl", hash = "sha256:c6008de247150668a705a6338156efb92334113421ceecf7438a12c9a12dab23", size = 7032347, upload-time = "2026-02-11T04:20:23.932Z" },
{ url = "https://files.pythonhosted.org/packages/f4/03/41c038f0d7a06099254c60f618d0ec7be11e79620fc23b8e85e5b31d9a44/pillow-12.1.0-cp310-cp310-win_arm64.whl", hash = "sha256:f188028b5af6b8fb2e9a76ac0f841a575bd1bd396e46ef0840d9b88a48fdbcea", size = 2452345, upload-time = "2026-01-02T09:10:44.795Z" }, { url = "https://files.pythonhosted.org/packages/79/13/c7922edded3dcdaf10c59297540b72785620abc0538872c819915746757d/pillow-12.1.1-cp310-cp310-win_arm64.whl", hash = "sha256:1a9b0ee305220b392e1124a764ee4265bd063e54a751a6b62eff69992f457fa9", size = 2453457, upload-time = "2026-02-11T04:20:25.392Z" },
{ url = "https://files.pythonhosted.org/packages/43/c4/bf8328039de6cc22182c3ef007a2abfbbdab153661c0a9aa78af8d706391/pillow-12.1.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:a83e0850cb8f5ac975291ebfc4170ba481f41a28065277f7f735c202cd8e0af3", size = 5304057, upload-time = "2026-01-02T09:10:46.627Z" }, { url = "https://files.pythonhosted.org/packages/2b/46/5da1ec4a5171ee7bf1a0efa064aba70ba3d6e0788ce3f5acd1375d23c8c0/pillow-12.1.1-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:e879bb6cd5c73848ef3b2b48b8af9ff08c5b71ecda8048b7dd22d8a33f60be32", size = 5304084, upload-time = "2026-02-11T04:20:27.501Z" },
{ url = "https://files.pythonhosted.org/packages/43/06/7264c0597e676104cc22ca73ee48f752767cd4b1fe084662620b17e10120/pillow-12.1.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:b6e53e82ec2db0717eabb276aa56cf4e500c9a7cec2c2e189b55c24f65a3e8c0", size = 4657811, upload-time = "2026-01-02T09:10:49.548Z" }, { url = "https://files.pythonhosted.org/packages/78/93/a29e9bc02d1cf557a834da780ceccd54e02421627200696fcf805ebdc3fb/pillow-12.1.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:365b10bb9417dd4498c0e3b128018c4a624dc11c7b97d8cc54effe3b096f4c38", size = 4657866, upload-time = "2026-02-11T04:20:29.827Z" },
{ url = "https://files.pythonhosted.org/packages/72/64/f9189e44474610daf83da31145fa56710b627b5c4c0b9c235e34058f6b31/pillow-12.1.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:40a8e3b9e8773876d6e30daed22f016509e3987bab61b3b7fe309d7019a87451", size = 6232243, upload-time = "2026-01-02T09:10:51.62Z" }, { url = "https://files.pythonhosted.org/packages/13/84/583a4558d492a179d31e4aae32eadce94b9acf49c0337c4ce0b70e0a01f2/pillow-12.1.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d4ce8e329c93845720cd2014659ca67eac35f6433fd3050393d85f3ecef0dad5", size = 6232148, upload-time = "2026-02-11T04:20:31.329Z" },
{ url = "https://files.pythonhosted.org/packages/ef/30/0df458009be6a4caca4ca2c52975e6275c387d4e5c95544e34138b41dc86/pillow-12.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:800429ac32c9b72909c671aaf17ecd13110f823ddb7db4dfef412a5587c2c24e", size = 8037872, upload-time = "2026-01-02T09:10:53.446Z" }, { url = "https://files.pythonhosted.org/packages/d5/e2/53c43334bbbb2d3b938978532fbda8e62bb6e0b23a26ce8592f36bcc4987/pillow-12.1.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fc354a04072b765eccf2204f588a7a532c9511e8b9c7f900e1b64e3e33487090", size = 8038007, upload-time = "2026-02-11T04:20:34.225Z" },
{ url = "https://files.pythonhosted.org/packages/e4/86/95845d4eda4f4f9557e25381d70876aa213560243ac1a6d619c46caaedd9/pillow-12.1.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b022eaaf709541b391ee069f0022ee5b36c709df71986e3f7be312e46f42c84", size = 6345398, upload-time = "2026-01-02T09:10:55.426Z" }, { url = "https://files.pythonhosted.org/packages/b8/a6/3d0e79c8a9d58150dd98e199d7c1c56861027f3829a3a60b3c2784190180/pillow-12.1.1-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7e7976bf1910a8116b523b9f9f58bf410f3e8aa330cd9a2bb2953f9266ab49af", size = 6345418, upload-time = "2026-02-11T04:20:35.858Z" },
{ url = "https://files.pythonhosted.org/packages/5c/1f/8e66ab9be3aaf1435bc03edd1ebdf58ffcd17f7349c1d970cafe87af27d9/pillow-12.1.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1f345e7bc9d7f368887c712aa5054558bad44d2a301ddf9248599f4161abc7c0", size = 7034667, upload-time = "2026-01-02T09:10:57.11Z" }, { url = "https://files.pythonhosted.org/packages/a2/c8/46dfeac5825e600579157eea177be43e2f7ff4a99da9d0d0a49533509ac5/pillow-12.1.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:597bd9c8419bc7c6af5604e55847789b69123bbe25d65cc6ad3012b4f3c98d8b", size = 7034590, upload-time = "2026-02-11T04:20:37.91Z" },
{ url = "https://files.pythonhosted.org/packages/f9/f6/683b83cb9b1db1fb52b87951b1c0b99bdcfceaa75febf11406c19f82cb5e/pillow-12.1.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d70347c8a5b7ccd803ec0c85c8709f036e6348f1e6a5bf048ecd9c64d3550b8b", size = 6458743, upload-time = "2026-01-02T09:10:59.331Z" }, { url = "https://files.pythonhosted.org/packages/af/bf/e6f65d3db8a8bbfeaf9e13cc0417813f6319863a73de934f14b2229ada18/pillow-12.1.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:2c1fc0f2ca5f96a3c8407e41cca26a16e46b21060fe6d5b099d2cb01412222f5", size = 6458655, upload-time = "2026-02-11T04:20:39.496Z" },
{ url = "https://files.pythonhosted.org/packages/9a/7d/de833d63622538c1d58ce5395e7c6cb7e7dce80decdd8bde4a484e095d9f/pillow-12.1.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1fcc52d86ce7a34fd17cb04e87cfdb164648a3662a6f20565910a99653d66c18", size = 7159342, upload-time = "2026-01-02T09:11:01.82Z" }, { url = "https://files.pythonhosted.org/packages/f9/c2/66091f3f34a25894ca129362e510b956ef26f8fb67a0e6417bc5744e56f1/pillow-12.1.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:578510d88c6229d735855e1f278aa305270438d36a05031dfaae5067cc8eb04d", size = 7159286, upload-time = "2026-02-11T04:20:41.139Z" },
{ url = "https://files.pythonhosted.org/packages/8c/40/50d86571c9e5868c42b81fe7da0c76ca26373f3b95a8dd675425f4a92ec1/pillow-12.1.0-cp311-cp311-win32.whl", hash = "sha256:3ffaa2f0659e2f740473bcf03c702c39a8d4b2b7ffc629052028764324842c64", size = 6328655, upload-time = "2026-01-02T09:11:04.556Z" }, { url = "https://files.pythonhosted.org/packages/7b/5a/24bc8eb526a22f957d0cec6243146744966d40857e3d8deb68f7902ca6c1/pillow-12.1.1-cp311-cp311-win32.whl", hash = "sha256:7311c0a0dcadb89b36b7025dfd8326ecfa36964e29913074d47382706e516a7c", size = 6328663, upload-time = "2026-02-11T04:20:43.184Z" },
{ url = "https://files.pythonhosted.org/packages/6c/af/b1d7e301c4cd26cd45d4af884d9ee9b6fab893b0ad2450d4746d74a6968c/pillow-12.1.0-cp311-cp311-win_amd64.whl", hash = "sha256:806f3987ffe10e867bab0ddad45df1148a2b98221798457fa097ad85d6e8bc75", size = 7031469, upload-time = "2026-01-02T09:11:06.538Z" }, { url = "https://files.pythonhosted.org/packages/31/03/bef822e4f2d8f9d7448c133d0a18185d3cce3e70472774fffefe8b0ed562/pillow-12.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:fbfa2a7c10cc2623f412753cddf391c7f971c52ca40a3f65dc5039b2939e8563", size = 7031448, upload-time = "2026-02-11T04:20:44.696Z" },
{ url = "https://files.pythonhosted.org/packages/48/36/d5716586d887fb2a810a4a61518a327a1e21c8b7134c89283af272efe84b/pillow-12.1.0-cp311-cp311-win_arm64.whl", hash = "sha256:9f5fefaca968e700ad1a4a9de98bf0869a94e397fe3524c4c9450c1445252304", size = 2452515, upload-time = "2026-01-02T09:11:08.226Z" }, { url = "https://files.pythonhosted.org/packages/49/70/f76296f53610bd17b2e7d31728b8b7825e3ac3b5b3688b51f52eab7c0818/pillow-12.1.1-cp311-cp311-win_arm64.whl", hash = "sha256:b81b5e3511211631b3f672a595e3221252c90af017e399056d0faabb9538aa80", size = 2453651, upload-time = "2026-02-11T04:20:46.243Z" },
{ url = "https://files.pythonhosted.org/packages/20/31/dc53fe21a2f2996e1b7d92bf671cdb157079385183ef7c1ae08b485db510/pillow-12.1.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:a332ac4ccb84b6dde65dbace8431f3af08874bf9770719d32a635c4ef411b18b", size = 5262642, upload-time = "2026-01-02T09:11:10.138Z" }, { url = "https://files.pythonhosted.org/packages/07/d3/8df65da0d4df36b094351dce696f2989bec731d4f10e743b1c5f4da4d3bf/pillow-12.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ab323b787d6e18b3d91a72fc99b1a2c28651e4358749842b8f8dfacd28ef2052", size = 5262803, upload-time = "2026-02-11T04:20:47.653Z" },
{ url = "https://files.pythonhosted.org/packages/ab/c1/10e45ac9cc79419cedf5121b42dcca5a50ad2b601fa080f58c22fb27626e/pillow-12.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:907bfa8a9cb790748a9aa4513e37c88c59660da3bcfffbd24a7d9e6abf224551", size = 4657464, upload-time = "2026-01-02T09:11:12.319Z" }, { url = "https://files.pythonhosted.org/packages/d6/71/5026395b290ff404b836e636f51d7297e6c83beceaa87c592718747e670f/pillow-12.1.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:adebb5bee0f0af4909c30db0d890c773d1a92ffe83da908e2e9e720f8edf3984", size = 4657601, upload-time = "2026-02-11T04:20:49.328Z" },
{ url = "https://files.pythonhosted.org/packages/ad/26/7b82c0ab7ef40ebede7a97c72d473bda5950f609f8e0c77b04af574a0ddb/pillow-12.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:efdc140e7b63b8f739d09a99033aa430accce485ff78e6d311973a67b6bf3208", size = 6234878, upload-time = "2026-01-02T09:11:14.096Z" }, { url = "https://files.pythonhosted.org/packages/b1/2e/1001613d941c67442f745aff0f7cc66dd8df9a9c084eb497e6a543ee6f7e/pillow-12.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:bb66b7cc26f50977108790e2456b7921e773f23db5630261102233eb355a3b79", size = 6234995, upload-time = "2026-02-11T04:20:51.032Z" },
{ url = "https://files.pythonhosted.org/packages/76/25/27abc9792615b5e886ca9411ba6637b675f1b77af3104710ac7353fe5605/pillow-12.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bef9768cab184e7ae6e559c032e95ba8d07b3023c289f79a2bd36e8bf85605a5", size = 8044868, upload-time = "2026-01-02T09:11:15.903Z" }, { url = "https://files.pythonhosted.org/packages/07/26/246ab11455b2549b9233dbd44d358d033a2f780fa9007b61a913c5b2d24e/pillow-12.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:aee2810642b2898bb187ced9b349e95d2a7272930796e022efaf12e99dccd293", size = 8045012, upload-time = "2026-02-11T04:20:52.882Z" },
{ url = "https://files.pythonhosted.org/packages/0a/ea/f200a4c36d836100e7bc738fc48cd963d3ba6372ebc8298a889e0cfc3359/pillow-12.1.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:742aea052cf5ab5034a53c3846165bc3ce88d7c38e954120db0ab867ca242661", size = 6349468, upload-time = "2026-01-02T09:11:17.631Z" }, { url = "https://files.pythonhosted.org/packages/b2/8b/07587069c27be7535ac1fe33874e32de118fbd34e2a73b7f83436a88368c/pillow-12.1.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a0b1cd6232e2b618adcc54d9882e4e662a089d5768cd188f7c245b4c8c44a397", size = 6349638, upload-time = "2026-02-11T04:20:54.444Z" },
{ url = "https://files.pythonhosted.org/packages/11/8f/48d0b77ab2200374c66d344459b8958c86693be99526450e7aee714e03e4/pillow-12.1.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a6dfc2af5b082b635af6e08e0d1f9f1c4e04d17d4e2ca0ef96131e85eda6eb17", size = 7041518, upload-time = "2026-01-02T09:11:19.389Z" }, { url = "https://files.pythonhosted.org/packages/ff/79/6df7b2ee763d619cda2fb4fea498e5f79d984dae304d45a8999b80d6cf5c/pillow-12.1.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7aac39bcf8d4770d089588a2e1dd111cbaa42df5a94be3114222057d68336bd0", size = 7041540, upload-time = "2026-02-11T04:20:55.97Z" },
{ url = "https://files.pythonhosted.org/packages/1d/23/c281182eb986b5d31f0a76d2a2c8cd41722d6fb8ed07521e802f9bba52de/pillow-12.1.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:609e89d9f90b581c8d16358c9087df76024cf058fa693dd3e1e1620823f39670", size = 6462829, upload-time = "2026-01-02T09:11:21.28Z" }, { url = "https://files.pythonhosted.org/packages/2c/5e/2ba19e7e7236d7529f4d873bdaf317a318896bac289abebd4bb00ef247f0/pillow-12.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ab174cd7d29a62dd139c44bf74b698039328f45cb03b4596c43473a46656b2f3", size = 6462613, upload-time = "2026-02-11T04:20:57.542Z" },
{ url = "https://files.pythonhosted.org/packages/25/ef/7018273e0faac099d7b00982abdcc39142ae6f3bd9ceb06de09779c4a9d6/pillow-12.1.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:43b4899cfd091a9693a1278c4982f3e50f7fb7cff5153b05174b4afc9593b616", size = 7166756, upload-time = "2026-01-02T09:11:23.559Z" }, { url = "https://files.pythonhosted.org/packages/03/03/31216ec124bb5c3dacd74ce8efff4cc7f52643653bad4825f8f08c697743/pillow-12.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:339ffdcb7cbeaa08221cd401d517d4b1fe7a9ed5d400e4a8039719238620ca35", size = 7166745, upload-time = "2026-02-11T04:20:59.196Z" },
{ url = "https://files.pythonhosted.org/packages/8f/c8/993d4b7ab2e341fe02ceef9576afcf5830cdec640be2ac5bee1820d693d4/pillow-12.1.0-cp312-cp312-win32.whl", hash = "sha256:aa0c9cc0b82b14766a99fbe6084409972266e82f459821cd26997a488a7261a7", size = 6328770, upload-time = "2026-01-02T09:11:25.661Z" }, { url = "https://files.pythonhosted.org/packages/1f/e7/7c4552d80052337eb28653b617eafdef39adfb137c49dd7e831b8dc13bc5/pillow-12.1.1-cp312-cp312-win32.whl", hash = "sha256:5d1f9575a12bed9e9eedd9a4972834b08c97a352bd17955ccdebfeca5913fa0a", size = 6328823, upload-time = "2026-02-11T04:21:01.385Z" },
{ url = "https://files.pythonhosted.org/packages/a7/87/90b358775a3f02765d87655237229ba64a997b87efa8ccaca7dd3e36e7a7/pillow-12.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:d70534cea9e7966169ad29a903b99fc507e932069a881d0965a1a84bb57f6c6d", size = 7033406, upload-time = "2026-01-02T09:11:27.474Z" }, { url = "https://files.pythonhosted.org/packages/3d/17/688626d192d7261bbbf98846fc98995726bddc2c945344b65bec3a29d731/pillow-12.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:21329ec8c96c6e979cd0dfd29406c40c1d52521a90544463057d2aaa937d66a6", size = 7033367, upload-time = "2026-02-11T04:21:03.536Z" },
{ url = "https://files.pythonhosted.org/packages/5d/cf/881b457eccacac9e5b2ddd97d5071fb6d668307c57cbf4e3b5278e06e536/pillow-12.1.0-cp312-cp312-win_arm64.whl", hash = "sha256:65b80c1ee7e14a87d6a068dd3b0aea268ffcabfe0498d38661b00c5b4b22e74c", size = 2452612, upload-time = "2026-01-02T09:11:29.309Z" }, { url = "https://files.pythonhosted.org/packages/ed/fe/a0ef1f73f939b0eca03ee2c108d0043a87468664770612602c63266a43c4/pillow-12.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:af9a332e572978f0218686636610555ae3defd1633597be015ed50289a03c523", size = 2453811, upload-time = "2026-02-11T04:21:05.116Z" },
{ url = "https://files.pythonhosted.org/packages/dd/c7/2530a4aa28248623e9d7f27316b42e27c32ec410f695929696f2e0e4a778/pillow-12.1.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:7b5dd7cbae20285cdb597b10eb5a2c13aa9de6cde9bb64a3c1317427b1db1ae1", size = 4062543, upload-time = "2026-01-02T09:11:31.566Z" }, { url = "https://files.pythonhosted.org/packages/d5/11/6db24d4bd7685583caeae54b7009584e38da3c3d4488ed4cd25b439de486/pillow-12.1.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:d242e8ac078781f1de88bf823d70c1a9b3c7950a44cdf4b7c012e22ccbcd8e4e", size = 4062689, upload-time = "2026-02-11T04:21:06.804Z" },
{ url = "https://files.pythonhosted.org/packages/8f/1f/40b8eae823dc1519b87d53c30ed9ef085506b05281d313031755c1705f73/pillow-12.1.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:29a4cef9cb672363926f0470afc516dbf7305a14d8c54f7abbb5c199cd8f8179", size = 4138373, upload-time = "2026-01-02T09:11:33.367Z" }, { url = "https://files.pythonhosted.org/packages/33/c0/ce6d3b1fe190f0021203e0d9b5b99e57843e345f15f9ef22fcd43842fd21/pillow-12.1.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:02f84dfad02693676692746df05b89cf25597560db2857363a208e393429f5e9", size = 4138535, upload-time = "2026-02-11T04:21:08.452Z" },
{ url = "https://files.pythonhosted.org/packages/d4/77/6fa60634cf06e52139fd0e89e5bbf055e8166c691c42fb162818b7fda31d/pillow-12.1.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:681088909d7e8fa9e31b9799aaa59ba5234c58e5e4f1951b4c4d1082a2e980e0", size = 3601241, upload-time = "2026-01-02T09:11:35.011Z" }, { url = "https://files.pythonhosted.org/packages/a0/c6/d5eb6a4fb32a3f9c21a8c7613ec706534ea1cf9f4b3663e99f0d83f6fca8/pillow-12.1.1-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:e65498daf4b583091ccbb2556c7000abf0f3349fcd57ef7adc9a84a394ed29f6", size = 3601364, upload-time = "2026-02-11T04:21:10.194Z" },
{ url = "https://files.pythonhosted.org/packages/4f/bf/28ab865de622e14b747f0cd7877510848252d950e43002e224fb1c9ababf/pillow-12.1.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:983976c2ab753166dc66d36af6e8ec15bb511e4a25856e2227e5f7e00a160587", size = 5262410, upload-time = "2026-01-02T09:11:36.682Z" }, { url = "https://files.pythonhosted.org/packages/14/a1/16c4b823838ba4c9c52c0e6bbda903a3fe5a1bdbf1b8eb4fff7156f3e318/pillow-12.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:6c6db3b84c87d48d0088943bf33440e0c42370b99b1c2a7989216f7b42eede60", size = 5262561, upload-time = "2026-02-11T04:21:11.742Z" },
{ url = "https://files.pythonhosted.org/packages/1c/34/583420a1b55e715937a85bd48c5c0991598247a1fd2eb5423188e765ea02/pillow-12.1.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:db44d5c160a90df2d24a24760bbd37607d53da0b34fb546c4c232af7192298ac", size = 4657312, upload-time = "2026-01-02T09:11:38.535Z" }, { url = "https://files.pythonhosted.org/packages/bb/ad/ad9dc98ff24f485008aa5cdedaf1a219876f6f6c42a4626c08bc4e80b120/pillow-12.1.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8b7e5304e34942bf62e15184219a7b5ad4ff7f3bb5cca4d984f37df1a0e1aee2", size = 4657460, upload-time = "2026-02-11T04:21:13.786Z" },
{ url = "https://files.pythonhosted.org/packages/1d/fd/f5a0896839762885b3376ff04878f86ab2b097c2f9a9cdccf4eda8ba8dc0/pillow-12.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6b7a9d1db5dad90e2991645874f708e87d9a3c370c243c2d7684d28f7e133e6b", size = 6232605, upload-time = "2026-01-02T09:11:40.602Z" }, { url = "https://files.pythonhosted.org/packages/9e/1b/f1a4ea9a895b5732152789326202a82464d5254759fbacae4deea3069334/pillow-12.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:18e5bddd742a44b7e6b1e773ab5db102bd7a94c32555ba656e76d319d19c3850", size = 6232698, upload-time = "2026-02-11T04:21:15.949Z" },
{ url = "https://files.pythonhosted.org/packages/98/aa/938a09d127ac1e70e6ed467bd03834350b33ef646b31edb7452d5de43792/pillow-12.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6258f3260986990ba2fa8a874f8b6e808cf5abb51a94015ca3dc3c68aa4f30ea", size = 8041617, upload-time = "2026-01-02T09:11:42.721Z" }, { url = "https://files.pythonhosted.org/packages/95/f4/86f51b8745070daf21fd2e5b1fe0eb35d4db9ca26e6d58366562fb56a743/pillow-12.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fc44ef1f3de4f45b50ccf9136999d71abb99dca7706bc75d222ed350b9fd2289", size = 8041706, upload-time = "2026-02-11T04:21:17.723Z" },
{ url = "https://files.pythonhosted.org/packages/17/e8/538b24cb426ac0186e03f80f78bc8dc7246c667f58b540bdd57c71c9f79d/pillow-12.1.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e115c15e3bc727b1ca3e641a909f77f8ca72a64fff150f666fcc85e57701c26c", size = 6346509, upload-time = "2026-01-02T09:11:44.955Z" }, { url = "https://files.pythonhosted.org/packages/29/9b/d6ecd956bb1266dd1045e995cce9b8d77759e740953a1c9aad9502a0461e/pillow-12.1.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5a8eb7ed8d4198bccbd07058416eeec51686b498e784eda166395a23eb99138e", size = 6346621, upload-time = "2026-02-11T04:21:19.547Z" },
{ url = "https://files.pythonhosted.org/packages/01/9a/632e58ec89a32738cabfd9ec418f0e9898a2b4719afc581f07c04a05e3c9/pillow-12.1.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6741e6f3074a35e47c77b23a4e4f2d90db3ed905cb1c5e6e0d49bff2045632bc", size = 7038117, upload-time = "2026-01-02T09:11:46.736Z" }, { url = "https://files.pythonhosted.org/packages/71/24/538bff45bde96535d7d998c6fed1a751c75ac7c53c37c90dc2601b243893/pillow-12.1.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:47b94983da0c642de92ced1702c5b6c292a84bd3a8e1d1702ff923f183594717", size = 7038069, upload-time = "2026-02-11T04:21:21.378Z" },
{ url = "https://files.pythonhosted.org/packages/c7/a2/d40308cf86eada842ca1f3ffa45d0ca0df7e4ab33c83f81e73f5eaed136d/pillow-12.1.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:935b9d1aed48fcfb3f838caac506f38e29621b44ccc4f8a64d575cb1b2a88644", size = 6460151, upload-time = "2026-01-02T09:11:48.625Z" }, { url = "https://files.pythonhosted.org/packages/94/0e/58cb1a6bc48f746bc4cb3adb8cabff73e2742c92b3bf7a220b7cf69b9177/pillow-12.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:518a48c2aab7ce596d3bf79d0e275661b846e86e4d0e7dec34712c30fe07f02a", size = 6460040, upload-time = "2026-02-11T04:21:23.148Z" },
{ url = "https://files.pythonhosted.org/packages/f1/88/f5b058ad6453a085c5266660a1417bdad590199da1b32fb4efcff9d33b05/pillow-12.1.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:5fee4c04aad8932da9f8f710af2c1a15a83582cfb884152a9caa79d4efcdbf9c", size = 7164534, upload-time = "2026-01-02T09:11:50.445Z" }, { url = "https://files.pythonhosted.org/packages/6c/57/9045cb3ff11eeb6c1adce3b2d60d7d299d7b273a2e6c8381a524abfdc474/pillow-12.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a550ae29b95c6dc13cf69e2c9dc5747f814c54eeb2e32d683e5e93af56caa029", size = 7164523, upload-time = "2026-02-11T04:21:25.01Z" },
{ url = "https://files.pythonhosted.org/packages/19/ce/c17334caea1db789163b5d855a5735e47995b0b5dc8745e9a3605d5f24c0/pillow-12.1.0-cp313-cp313-win32.whl", hash = "sha256:a786bf667724d84aa29b5db1c61b7bfdde380202aaca12c3461afd6b71743171", size = 6332551, upload-time = "2026-01-02T09:11:52.234Z" }, { url = "https://files.pythonhosted.org/packages/73/f2/9be9cb99f2175f0d4dbadd6616ce1bf068ee54a28277ea1bf1fbf729c250/pillow-12.1.1-cp313-cp313-win32.whl", hash = "sha256:a003d7422449f6d1e3a34e3dd4110c22148336918ddbfc6a32581cd54b2e0b2b", size = 6332552, upload-time = "2026-02-11T04:21:27.238Z" },
{ url = "https://files.pythonhosted.org/packages/e5/07/74a9d941fa45c90a0d9465098fe1ec85de3e2afbdc15cc4766622d516056/pillow-12.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:461f9dfdafa394c59cd6d818bdfdbab4028b83b02caadaff0ffd433faf4c9a7a", size = 7040087, upload-time = "2026-01-02T09:11:54.822Z" }, { url = "https://files.pythonhosted.org/packages/3f/eb/b0834ad8b583d7d9d42b80becff092082a1c3c156bb582590fcc973f1c7c/pillow-12.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:344cf1e3dab3be4b1fa08e449323d98a2a3f819ad20f4b22e77a0ede31f0faa1", size = 7040108, upload-time = "2026-02-11T04:21:29.462Z" },
{ url = "https://files.pythonhosted.org/packages/88/09/c99950c075a0e9053d8e880595926302575bc742b1b47fe1bbcc8d388d50/pillow-12.1.0-cp313-cp313-win_arm64.whl", hash = "sha256:9212d6b86917a2300669511ed094a9406888362e085f2431a7da985a6b124f45", size = 2452470, upload-time = "2026-01-02T09:11:56.522Z" }, { url = "https://files.pythonhosted.org/packages/d5/7d/fc09634e2aabdd0feabaff4a32f4a7d97789223e7c2042fd805ea4b4d2c2/pillow-12.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:5c0dd1636633e7e6a0afe7bf6a51a14992b7f8e60de5789018ebbdfae55b040a", size = 2453712, upload-time = "2026-02-11T04:21:31.072Z" },
{ url = "https://files.pythonhosted.org/packages/b5/ba/970b7d85ba01f348dee4d65412476321d40ee04dcb51cd3735b9dc94eb58/pillow-12.1.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:00162e9ca6d22b7c3ee8e61faa3c3253cd19b6a37f126cad04f2f88b306f557d", size = 5264816, upload-time = "2026-01-02T09:11:58.227Z" }, { url = "https://files.pythonhosted.org/packages/19/2a/b9d62794fc8a0dd14c1943df68347badbd5511103e0d04c035ffe5cf2255/pillow-12.1.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:0330d233c1a0ead844fc097a7d16c0abff4c12e856c0b325f231820fee1f39da", size = 5264880, upload-time = "2026-02-11T04:21:32.865Z" },
{ url = "https://files.pythonhosted.org/packages/10/60/650f2fb55fdba7a510d836202aa52f0baac633e50ab1cf18415d332188fb/pillow-12.1.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:7d6daa89a00b58c37cb1747ec9fb7ac3bc5ffd5949f5888657dfddde6d1312e0", size = 4660472, upload-time = "2026-01-02T09:12:00.798Z" }, { url = "https://files.pythonhosted.org/packages/26/9d/e03d857d1347fa5ed9247e123fcd2a97b6220e15e9cb73ca0a8d91702c6e/pillow-12.1.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:5dae5f21afb91322f2ff791895ddd8889e5e947ff59f71b46041c8ce6db790bc", size = 4660616, upload-time = "2026-02-11T04:21:34.97Z" },
{ url = "https://files.pythonhosted.org/packages/2b/c0/5273a99478956a099d533c4f46cbaa19fd69d606624f4334b85e50987a08/pillow-12.1.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e2479c7f02f9d505682dc47df8c0ea1fc5e264c4d1629a5d63fe3e2334b89554", size = 6268974, upload-time = "2026-01-02T09:12:02.572Z" }, { url = "https://files.pythonhosted.org/packages/f7/ec/8a6d22afd02570d30954e043f09c32772bfe143ba9285e2fdb11284952cd/pillow-12.1.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2e0c664be47252947d870ac0d327fea7e63985a08794758aa8af5b6cb6ec0c9c", size = 6269008, upload-time = "2026-02-11T04:21:36.623Z" },
{ url = "https://files.pythonhosted.org/packages/b4/26/0bf714bc2e73d5267887d47931d53c4ceeceea6978148ed2ab2a4e6463c4/pillow-12.1.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f188d580bd870cda1e15183790d1cc2fa78f666e76077d103edf048eed9c356e", size = 8073070, upload-time = "2026-01-02T09:12:04.75Z" }, { url = "https://files.pythonhosted.org/packages/3d/1d/6d875422c9f28a4a361f495a5f68d9de4a66941dc2c619103ca335fa6446/pillow-12.1.1-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:691ab2ac363b8217f7d31b3497108fb1f50faab2f75dfb03284ec2f217e87bf8", size = 8073226, upload-time = "2026-02-11T04:21:38.585Z" },
{ url = "https://files.pythonhosted.org/packages/43/cf/1ea826200de111a9d65724c54f927f3111dc5ae297f294b370a670c17786/pillow-12.1.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0fde7ec5538ab5095cc02df38ee99b0443ff0e1c847a045554cf5f9af1f4aa82", size = 6380176, upload-time = "2026-01-02T09:12:06.626Z" }, { url = "https://files.pythonhosted.org/packages/a1/cd/134b0b6ee5eda6dc09e25e24b40fdafe11a520bc725c1d0bbaa5e00bf95b/pillow-12.1.1-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e9e8064fb1cc019296958595f6db671fba95209e3ceb0c4734c9baf97de04b20", size = 6380136, upload-time = "2026-02-11T04:21:40.562Z" },
{ url = "https://files.pythonhosted.org/packages/03/e0/7938dd2b2013373fd85d96e0f38d62b7a5a262af21ac274250c7ca7847c9/pillow-12.1.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0ed07dca4a8464bada6139ab38f5382f83e5f111698caf3191cb8dbf27d908b4", size = 7067061, upload-time = "2026-01-02T09:12:08.624Z" }, { url = "https://files.pythonhosted.org/packages/7a/a9/7628f013f18f001c1b98d8fffe3452f306a70dc6aba7d931019e0492f45e/pillow-12.1.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:472a8d7ded663e6162dafdf20015c486a7009483ca671cece7a9279b512fcb13", size = 7067129, upload-time = "2026-02-11T04:21:42.521Z" },
{ url = "https://files.pythonhosted.org/packages/86/ad/a2aa97d37272a929a98437a8c0ac37b3cf012f4f8721e1bd5154699b2518/pillow-12.1.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:f45bd71d1fa5e5749587613037b172e0b3b23159d1c00ef2fc920da6f470e6f0", size = 6491824, upload-time = "2026-01-02T09:12:10.488Z" }, { url = "https://files.pythonhosted.org/packages/1e/f8/66ab30a2193b277785601e82ee2d49f68ea575d9637e5e234faaa98efa4c/pillow-12.1.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:89b54027a766529136a06cfebeecb3a04900397a3590fd252160b888479517bf", size = 6491807, upload-time = "2026-02-11T04:21:44.22Z" },
{ url = "https://files.pythonhosted.org/packages/a4/44/80e46611b288d51b115826f136fb3465653c28f491068a72d3da49b54cd4/pillow-12.1.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:277518bf4fe74aa91489e1b20577473b19ee70fb97c374aa50830b279f25841b", size = 7190911, upload-time = "2026-01-02T09:12:12.772Z" }, { url = "https://files.pythonhosted.org/packages/da/0b/a877a6627dc8318fdb84e357c5e1a758c0941ab1ddffdafd231983788579/pillow-12.1.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:86172b0831b82ce4f7877f280055892b31179e1576aa00d0df3bb1bbf8c3e524", size = 7190954, upload-time = "2026-02-11T04:21:46.114Z" },
{ url = "https://files.pythonhosted.org/packages/86/77/eacc62356b4cf81abe99ff9dbc7402750044aed02cfd6a503f7c6fc11f3e/pillow-12.1.0-cp313-cp313t-win32.whl", hash = "sha256:7315f9137087c4e0ee73a761b163fc9aa3b19f5f606a7fc08d83fd3e4379af65", size = 6336445, upload-time = "2026-01-02T09:12:14.775Z" }, { url = "https://files.pythonhosted.org/packages/83/43/6f732ff85743cf746b1361b91665d9f5155e1483817f693f8d57ea93147f/pillow-12.1.1-cp313-cp313t-win32.whl", hash = "sha256:44ce27545b6efcf0fdbdceb31c9a5bdea9333e664cda58a7e674bb74608b3986", size = 6336441, upload-time = "2026-02-11T04:21:48.22Z" },
{ url = "https://files.pythonhosted.org/packages/e7/3c/57d81d0b74d218706dafccb87a87ea44262c43eef98eb3b164fd000e0491/pillow-12.1.0-cp313-cp313t-win_amd64.whl", hash = "sha256:0ddedfaa8b5f0b4ffbc2fa87b556dc59f6bb4ecb14a53b33f9189713ae8053c0", size = 7045354, upload-time = "2026-01-02T09:12:16.599Z" }, { url = "https://files.pythonhosted.org/packages/3b/44/e865ef3986611bb75bfabdf94a590016ea327833f434558801122979cd0e/pillow-12.1.1-cp313-cp313t-win_amd64.whl", hash = "sha256:a285e3eb7a5a45a2ff504e31f4a8d1b12ef62e84e5411c6804a42197c1cf586c", size = 7045383, upload-time = "2026-02-11T04:21:50.015Z" },
{ url = "https://files.pythonhosted.org/packages/ac/82/8b9b97bba2e3576a340f93b044a3a3a09841170ab4c1eb0d5c93469fd32f/pillow-12.1.0-cp313-cp313t-win_arm64.whl", hash = "sha256:80941e6d573197a0c28f394753de529bb436b1ca990ed6e765cf42426abc39f8", size = 2454547, upload-time = "2026-01-02T09:12:18.704Z" }, { url = "https://files.pythonhosted.org/packages/a8/c6/f4fb24268d0c6908b9f04143697ea18b0379490cb74ba9e8d41b898bd005/pillow-12.1.1-cp313-cp313t-win_arm64.whl", hash = "sha256:cc7d296b5ea4d29e6570dabeaed58d31c3fea35a633a69679fb03d7664f43fb3", size = 2456104, upload-time = "2026-02-11T04:21:51.633Z" },
{ url = "https://files.pythonhosted.org/packages/8c/87/bdf971d8bbcf80a348cc3bacfcb239f5882100fe80534b0ce67a784181d8/pillow-12.1.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:5cb7bc1966d031aec37ddb9dcf15c2da5b2e9f7cc3ca7c54473a20a927e1eb91", size = 4062533, upload-time = "2026-01-02T09:12:20.791Z" }, { url = "https://files.pythonhosted.org/packages/03/d0/bebb3ffbf31c5a8e97241476c4cf8b9828954693ce6744b4a2326af3e16b/pillow-12.1.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:417423db963cb4be8bac3fc1204fe61610f6abeed1580a7a2cbb2fbda20f12af", size = 4062652, upload-time = "2026-02-11T04:21:53.19Z" },
{ url = "https://files.pythonhosted.org/packages/ff/4f/5eb37a681c68d605eb7034c004875c81f86ec9ef51f5be4a63eadd58859a/pillow-12.1.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:97e9993d5ed946aba26baf9c1e8cf18adbab584b99f452ee72f7ee8acb882796", size = 4138546, upload-time = "2026-01-02T09:12:23.664Z" }, { url = "https://files.pythonhosted.org/packages/2d/c0/0e16fb0addda4851445c28f8350d8c512f09de27bbb0d6d0bbf8b6709605/pillow-12.1.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:b957b71c6b2387610f556a7eb0828afbe40b4a98036fc0d2acfa5a44a0c2036f", size = 4138823, upload-time = "2026-02-11T04:22:03.088Z" },
{ url = "https://files.pythonhosted.org/packages/11/6d/19a95acb2edbace40dcd582d077b991646b7083c41b98da4ed7555b59733/pillow-12.1.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:414b9a78e14ffeb98128863314e62c3f24b8a86081066625700b7985b3f529bd", size = 3601163, upload-time = "2026-01-02T09:12:26.338Z" }, { url = "https://files.pythonhosted.org/packages/6b/fb/6170ec655d6f6bb6630a013dd7cf7bc218423d7b5fa9071bf63dc32175ae/pillow-12.1.1-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:097690ba1f2efdeb165a20469d59d8bb03c55fb6621eb2041a060ae8ea3e9642", size = 3601143, upload-time = "2026-02-11T04:22:04.909Z" },
{ url = "https://files.pythonhosted.org/packages/fc/36/2b8138e51cb42e4cc39c3297713455548be855a50558c3ac2beebdc251dd/pillow-12.1.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:e6bdb408f7c9dd2a5ff2b14a3b0bb6d4deb29fb9961e6eb3ae2031ae9a5cec13", size = 5266086, upload-time = "2026-01-02T09:12:28.782Z" }, { url = "https://files.pythonhosted.org/packages/59/04/dc5c3f297510ba9a6837cbb318b87dd2b8f73eb41a43cc63767f65cb599c/pillow-12.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:2815a87ab27848db0321fb78c7f0b2c8649dee134b7f2b80c6a45c6831d75ccd", size = 5266254, upload-time = "2026-02-11T04:22:07.656Z" },
{ url = "https://files.pythonhosted.org/packages/53/4b/649056e4d22e1caa90816bf99cef0884aed607ed38075bd75f091a607a38/pillow-12.1.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:3413c2ae377550f5487991d444428f1a8ae92784aac79caa8b1e3b89b175f77e", size = 4657344, upload-time = "2026-01-02T09:12:31.117Z" }, { url = "https://files.pythonhosted.org/packages/05/30/5db1236b0d6313f03ebf97f5e17cda9ca060f524b2fcc875149a8360b21c/pillow-12.1.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:f7ed2c6543bad5a7d5530eb9e78c53132f93dfa44a28492db88b41cdab885202", size = 4657499, upload-time = "2026-02-11T04:22:09.613Z" },
{ url = "https://files.pythonhosted.org/packages/6c/6b/c5742cea0f1ade0cd61485dc3d81f05261fc2276f537fbdc00802de56779/pillow-12.1.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e5dcbe95016e88437ecf33544ba5db21ef1b8dd6e1b434a2cb2a3d605299e643", size = 6232114, upload-time = "2026-01-02T09:12:32.936Z" }, { url = "https://files.pythonhosted.org/packages/6f/18/008d2ca0eb612e81968e8be0bbae5051efba24d52debf930126d7eaacbba/pillow-12.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:652a2c9ccfb556235b2b501a3a7cf3742148cd22e04b5625c5fe057ea3e3191f", size = 6232137, upload-time = "2026-02-11T04:22:11.434Z" },
{ url = "https://files.pythonhosted.org/packages/bf/8f/9f521268ce22d63991601aafd3d48d5ff7280a246a1ef62d626d67b44064/pillow-12.1.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d0a7735df32ccbcc98b98a1ac785cc4b19b580be1bdf0aeb5c03223220ea09d5", size = 8042708, upload-time = "2026-01-02T09:12:34.78Z" }, { url = "https://files.pythonhosted.org/packages/70/f1/f14d5b8eeb4b2cd62b9f9f847eb6605f103df89ef619ac68f92f748614ea/pillow-12.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d6e4571eedf43af33d0fc233a382a76e849badbccdf1ac438841308652a08e1f", size = 8042721, upload-time = "2026-02-11T04:22:13.321Z" },
{ url = "https://files.pythonhosted.org/packages/1a/eb/257f38542893f021502a1bbe0c2e883c90b5cff26cc33b1584a841a06d30/pillow-12.1.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0c27407a2d1b96774cbc4a7594129cc027339fd800cd081e44497722ea1179de", size = 6347762, upload-time = "2026-01-02T09:12:36.748Z" }, { url = "https://files.pythonhosted.org/packages/5a/d6/17824509146e4babbdabf04d8171491fa9d776f7061ff6e727522df9bd03/pillow-12.1.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b574c51cf7d5d62e9be37ba446224b59a2da26dc4c1bb2ecbe936a4fb1a7cb7f", size = 6347798, upload-time = "2026-02-11T04:22:15.449Z" },
{ url = "https://files.pythonhosted.org/packages/c4/5a/8ba375025701c09b309e8d5163c5a4ce0102fa86bbf8800eb0d7ac87bc51/pillow-12.1.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:15c794d74303828eaa957ff8070846d0efe8c630901a1c753fdc63850e19ecd9", size = 7039265, upload-time = "2026-01-02T09:12:39.082Z" }, { url = "https://files.pythonhosted.org/packages/d1/ee/c85a38a9ab92037a75615aba572c85ea51e605265036e00c5b67dfafbfe2/pillow-12.1.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a37691702ed687799de29a518d63d4682d9016932db66d4e90c345831b02fb4e", size = 7039315, upload-time = "2026-02-11T04:22:17.24Z" },
{ url = "https://files.pythonhosted.org/packages/cf/dc/cf5e4cdb3db533f539e88a7bbf9f190c64ab8a08a9bc7a4ccf55067872e4/pillow-12.1.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c990547452ee2800d8506c4150280757f88532f3de2a58e3022e9b179107862a", size = 6462341, upload-time = "2026-01-02T09:12:40.946Z" }, { url = "https://files.pythonhosted.org/packages/ec/f3/bc8ccc6e08a148290d7523bde4d9a0d6c981db34631390dc6e6ec34cacf6/pillow-12.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f95c00d5d6700b2b890479664a06e754974848afaae5e21beb4d83c106923fd0", size = 6462360, upload-time = "2026-02-11T04:22:19.111Z" },
{ url = "https://files.pythonhosted.org/packages/d0/47/0291a25ac9550677e22eda48510cfc4fa4b2ef0396448b7fbdc0a6946309/pillow-12.1.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:b63e13dd27da389ed9475b3d28510f0f954bca0041e8e551b2a4eb1eab56a39a", size = 7165395, upload-time = "2026-01-02T09:12:42.706Z" }, { url = "https://files.pythonhosted.org/packages/f6/ab/69a42656adb1d0665ab051eec58a41f169ad295cf81ad45406963105408f/pillow-12.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:559b38da23606e68681337ad74622c4dbba02254fc9cb4488a305dd5975c7eeb", size = 7165438, upload-time = "2026-02-11T04:22:21.041Z" },
{ url = "https://files.pythonhosted.org/packages/4f/4c/e005a59393ec4d9416be06e6b45820403bb946a778e39ecec62f5b2b991e/pillow-12.1.0-cp314-cp314-win32.whl", hash = "sha256:1a949604f73eb07a8adab38c4fe50791f9919344398bdc8ac6b307f755fc7030", size = 6431413, upload-time = "2026-01-02T09:12:44.944Z" }, { url = "https://files.pythonhosted.org/packages/02/46/81f7aa8941873f0f01d4b55cc543b0a3d03ec2ee30d617a0448bf6bd6dec/pillow-12.1.1-cp314-cp314-win32.whl", hash = "sha256:03edcc34d688572014ff223c125a3f77fb08091e4607e7745002fc214070b35f", size = 6431503, upload-time = "2026-02-11T04:22:22.833Z" },
{ url = "https://files.pythonhosted.org/packages/1c/af/f23697f587ac5f9095d67e31b81c95c0249cd461a9798a061ed6709b09b5/pillow-12.1.0-cp314-cp314-win_amd64.whl", hash = "sha256:4f9f6a650743f0ddee5593ac9e954ba1bdbc5e150bc066586d4f26127853ab94", size = 7176779, upload-time = "2026-01-02T09:12:46.727Z" }, { url = "https://files.pythonhosted.org/packages/40/72/4c245f7d1044b67affc7f134a09ea619d4895333d35322b775b928180044/pillow-12.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:50480dcd74fa63b8e78235957d302d98d98d82ccbfac4c7e12108ba9ecbdba15", size = 7176748, upload-time = "2026-02-11T04:22:24.64Z" },
{ url = "https://files.pythonhosted.org/packages/b3/36/6a51abf8599232f3e9afbd16d52829376a68909fe14efe29084445db4b73/pillow-12.1.0-cp314-cp314-win_arm64.whl", hash = "sha256:808b99604f7873c800c4840f55ff389936ef1948e4e87645eaf3fccbc8477ac4", size = 2543105, upload-time = "2026-01-02T09:12:49.243Z" }, { url = "https://files.pythonhosted.org/packages/e4/ad/8a87bdbe038c5c698736e3348af5c2194ffb872ea52f11894c95f9305435/pillow-12.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:5cb1785d97b0c3d1d1a16bc1d710c4a0049daefc4935f3a8f31f827f4d3d2e7f", size = 2544314, upload-time = "2026-02-11T04:22:26.685Z" },
{ url = "https://files.pythonhosted.org/packages/82/54/2e1dd20c8749ff225080d6ba465a0cab4387f5db0d1c5fb1439e2d99923f/pillow-12.1.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:bc11908616c8a283cf7d664f77411a5ed2a02009b0097ff8abbba5e79128ccf2", size = 5268571, upload-time = "2026-01-02T09:12:51.11Z" }, { url = "https://files.pythonhosted.org/packages/6c/9d/efd18493f9de13b87ede7c47e69184b9e859e4427225ea962e32e56a49bc/pillow-12.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:1f90cff8aa76835cba5769f0b3121a22bd4eb9e6884cfe338216e557a9a548b8", size = 5268612, upload-time = "2026-02-11T04:22:29.884Z" },
{ url = "https://files.pythonhosted.org/packages/57/61/571163a5ef86ec0cf30d265ac2a70ae6fc9e28413d1dc94fa37fae6bda89/pillow-12.1.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:896866d2d436563fa2a43a9d72f417874f16b5545955c54a64941e87c1376c61", size = 4660426, upload-time = "2026-01-02T09:12:52.865Z" }, { url = "https://files.pythonhosted.org/packages/f8/f1/4f42eb2b388eb2ffc660dcb7f7b556c1015c53ebd5f7f754965ef997585b/pillow-12.1.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1f1be78ce9466a7ee64bfda57bdba0f7cc499d9794d518b854816c41bf0aa4e9", size = 4660567, upload-time = "2026-02-11T04:22:31.799Z" },
{ url = "https://files.pythonhosted.org/packages/5e/e1/53ee5163f794aef1bf84243f755ee6897a92c708505350dd1923f4afec48/pillow-12.1.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8e178e3e99d3c0ea8fc64b88447f7cac8ccf058af422a6cedc690d0eadd98c51", size = 6269908, upload-time = "2026-01-02T09:12:54.884Z" }, { url = "https://files.pythonhosted.org/packages/01/54/df6ef130fa43e4b82e32624a7b821a2be1c5653a5fdad8469687a7db4e00/pillow-12.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:42fc1f4677106188ad9a55562bbade416f8b55456f522430fadab3cef7cd4e60", size = 6269951, upload-time = "2026-02-11T04:22:33.921Z" },
{ url = "https://files.pythonhosted.org/packages/bc/0b/b4b4106ff0ee1afa1dc599fde6ab230417f800279745124f6c50bcffed8e/pillow-12.1.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:079af2fb0c599c2ec144ba2c02766d1b55498e373b3ac64687e43849fbbef5bc", size = 8074733, upload-time = "2026-01-02T09:12:56.802Z" }, { url = "https://files.pythonhosted.org/packages/a9/48/618752d06cc44bb4aae8ce0cd4e6426871929ed7b46215638088270d9b34/pillow-12.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:98edb152429ab62a1818039744d8fbb3ccab98a7c29fc3d5fcef158f3f1f68b7", size = 8074769, upload-time = "2026-02-11T04:22:35.877Z" },
{ url = "https://files.pythonhosted.org/packages/19/9f/80b411cbac4a732439e629a26ad3ef11907a8c7fc5377b7602f04f6fe4e7/pillow-12.1.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bdec5e43377761c5dbca620efb69a77f6855c5a379e32ac5b158f54c84212b14", size = 6381431, upload-time = "2026-01-02T09:12:58.823Z" }, { url = "https://files.pythonhosted.org/packages/c3/bd/f1d71eb39a72fa088d938655afba3e00b38018d052752f435838961127d8/pillow-12.1.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d470ab1178551dd17fdba0fef463359c41aaa613cdcd7ff8373f54be629f9f8f", size = 6381358, upload-time = "2026-02-11T04:22:37.698Z" },
{ url = "https://files.pythonhosted.org/packages/8f/b7/d65c45db463b66ecb6abc17c6ba6917a911202a07662247e1355ce1789e7/pillow-12.1.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:565c986f4b45c020f5421a4cea13ef294dde9509a8577f29b2fc5edc7587fff8", size = 7068529, upload-time = "2026-01-02T09:13:00.885Z" }, { url = "https://files.pythonhosted.org/packages/64/ef/c784e20b96674ed36a5af839305f55616f8b4f8aa8eeccf8531a6e312243/pillow-12.1.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6408a7b064595afcab0a49393a413732a35788f2a5092fdc6266952ed67de586", size = 7068558, upload-time = "2026-02-11T04:22:39.597Z" },
{ url = "https://files.pythonhosted.org/packages/50/96/dfd4cd726b4a45ae6e3c669fc9e49deb2241312605d33aba50499e9d9bd1/pillow-12.1.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:43aca0a55ce1eefc0aefa6253661cb54571857b1a7b2964bd8a1e3ef4b729924", size = 6492981, upload-time = "2026-01-02T09:13:03.314Z" }, { url = "https://files.pythonhosted.org/packages/73/cb/8059688b74422ae61278202c4e1ad992e8a2e7375227be0a21c6b87ca8d5/pillow-12.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:5d8c41325b382c07799a3682c1c258469ea2ff97103c53717b7893862d0c98ce", size = 6493028, upload-time = "2026-02-11T04:22:42.73Z" },
{ url = "https://files.pythonhosted.org/packages/4d/1c/b5dc52cf713ae46033359c5ca920444f18a6359ce1020dd3e9c553ea5bc6/pillow-12.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:0deedf2ea233722476b3a81e8cdfbad786f7adbed5d848469fa59fe52396e4ef", size = 7191878, upload-time = "2026-01-02T09:13:05.276Z" }, { url = "https://files.pythonhosted.org/packages/c6/da/e3c008ed7d2dd1f905b15949325934510b9d1931e5df999bb15972756818/pillow-12.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:c7697918b5be27424e9ce568193efd13d925c4481dd364e43f5dff72d33e10f8", size = 7191940, upload-time = "2026-02-11T04:22:44.543Z" },
{ url = "https://files.pythonhosted.org/packages/53/26/c4188248bd5edaf543864fe4834aebe9c9cb4968b6f573ce014cc42d0720/pillow-12.1.0-cp314-cp314t-win32.whl", hash = "sha256:b17fbdbe01c196e7e159aacb889e091f28e61020a8abeac07b68079b6e626988", size = 6438703, upload-time = "2026-01-02T09:13:07.491Z" }, { url = "https://files.pythonhosted.org/packages/01/4a/9202e8d11714c1fc5951f2e1ef362f2d7fbc595e1f6717971d5dd750e969/pillow-12.1.1-cp314-cp314t-win32.whl", hash = "sha256:d2912fd8114fc5545aa3a4b5576512f64c55a03f3ebcca4c10194d593d43ea36", size = 6438736, upload-time = "2026-02-11T04:22:46.347Z" },
{ url = "https://files.pythonhosted.org/packages/b8/0e/69ed296de8ea05cb03ee139cee600f424ca166e632567b2d66727f08c7ed/pillow-12.1.0-cp314-cp314t-win_amd64.whl", hash = "sha256:27b9baecb428899db6c0de572d6d305cfaf38ca1596b5c0542a5182e3e74e8c6", size = 7182927, upload-time = "2026-01-02T09:13:09.841Z" }, { url = "https://files.pythonhosted.org/packages/f3/ca/cbce2327eb9885476b3957b2e82eb12c866a8b16ad77392864ad601022ce/pillow-12.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:4ceb838d4bd9dab43e06c363cab2eebf63846d6a4aeaea283bbdfd8f1a8ed58b", size = 7182894, upload-time = "2026-02-11T04:22:48.114Z" },
{ url = "https://files.pythonhosted.org/packages/fc/f5/68334c015eed9b5cff77814258717dec591ded209ab5b6fb70e2ae873d1d/pillow-12.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:f61333d817698bdcdd0f9d7793e365ac3d2a21c1f1eb02b32ad6aefb8d8ea831", size = 2545104, upload-time = "2026-01-02T09:13:12.068Z" }, { url = "https://files.pythonhosted.org/packages/ec/d2/de599c95ba0a973b94410477f8bf0b6f0b5e67360eb89bcb1ad365258beb/pillow-12.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:7b03048319bfc6170e93bd60728a1af51d3dd7704935feb228c4d4faab35d334", size = 2546446, upload-time = "2026-02-11T04:22:50.342Z" },
{ url = "https://files.pythonhosted.org/packages/8b/bc/224b1d98cffd7164b14707c91aac83c07b047fbd8f58eba4066a3e53746a/pillow-12.1.0-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:ca94b6aac0d7af2a10ba08c0f888b3d5114439b6b3ef39968378723622fed377", size = 5228605, upload-time = "2026-01-02T09:13:14.084Z" }, { url = "https://files.pythonhosted.org/packages/56/11/5d43209aa4cb58e0cc80127956ff1796a68b928e6324bbf06ef4db34367b/pillow-12.1.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:600fd103672b925fe62ed08e0d874ea34d692474df6f4bf7ebe148b30f89f39f", size = 5228606, upload-time = "2026-02-11T04:22:52.106Z" },
{ url = "https://files.pythonhosted.org/packages/0c/ca/49ca7769c4550107de049ed85208240ba0f330b3f2e316f24534795702ce/pillow-12.1.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:351889afef0f485b84078ea40fe33727a0492b9af3904661b0abbafee0355b72", size = 4622245, upload-time = "2026-01-02T09:13:15.964Z" }, { url = "https://files.pythonhosted.org/packages/5f/d5/3b005b4e4fda6698b371fa6c21b097d4707585d7db99e98d9b0b87ac612a/pillow-12.1.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:665e1b916b043cef294bc54d47bf02d87e13f769bc4bc5fa225a24b3a6c5aca9", size = 4622321, upload-time = "2026-02-11T04:22:53.827Z" },
{ url = "https://files.pythonhosted.org/packages/73/48/fac807ce82e5955bcc2718642b94b1bd22a82a6d452aea31cbb678cddf12/pillow-12.1.0-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:bb0984b30e973f7e2884362b7d23d0a348c7143ee559f38ef3eaab640144204c", size = 5247593, upload-time = "2026-01-02T09:13:17.913Z" }, { url = "https://files.pythonhosted.org/packages/df/36/ed3ea2d594356fd8037e5a01f6156c74bc8d92dbb0fa60746cc96cabb6e8/pillow-12.1.1-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:495c302af3aad1ca67420ddd5c7bd480c8867ad173528767d906428057a11f0e", size = 5247579, upload-time = "2026-02-11T04:22:56.094Z" },
{ url = "https://files.pythonhosted.org/packages/d2/95/3e0742fe358c4664aed4fd05d5f5373dcdad0b27af52aa0972568541e3f4/pillow-12.1.0-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:84cabc7095dd535ca934d57e9ce2a72ffd216e435a84acb06b2277b1de2689bd", size = 6989008, upload-time = "2026-01-02T09:13:20.083Z" }, { url = "https://files.pythonhosted.org/packages/54/9a/9cc3e029683cf6d20ae5085da0dafc63148e3252c2f13328e553aaa13cfb/pillow-12.1.1-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8fd420ef0c52c88b5a035a0886f367748c72147b2b8f384c9d12656678dfdfa9", size = 6989094, upload-time = "2026-02-11T04:22:58.288Z" },
{ url = "https://files.pythonhosted.org/packages/5a/74/fe2ac378e4e202e56d50540d92e1ef4ff34ed687f3c60f6a121bcf99437e/pillow-12.1.0-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:53d8b764726d3af1a138dd353116f774e3862ec7e3794e0c8781e30db0f35dfc", size = 5313824, upload-time = "2026-01-02T09:13:22.405Z" }, { url = "https://files.pythonhosted.org/packages/00/98/fc53ab36da80b88df0967896b6c4b4cd948a0dc5aa40a754266aa3ae48b3/pillow-12.1.1-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f975aa7ef9684ce7e2c18a3aa8f8e2106ce1e46b94ab713d156b2898811651d3", size = 5313850, upload-time = "2026-02-11T04:23:00.554Z" },
{ url = "https://files.pythonhosted.org/packages/f3/77/2a60dee1adee4e2655ac328dd05c02a955c1cd683b9f1b82ec3feb44727c/pillow-12.1.0-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5da841d81b1a05ef940a8567da92decaa15bc4d7dedb540a8c219ad83d91808a", size = 5963278, upload-time = "2026-01-02T09:13:24.706Z" }, { url = "https://files.pythonhosted.org/packages/30/02/00fa585abfd9fe9d73e5f6e554dc36cc2b842898cbfc46d70353dae227f8/pillow-12.1.1-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8089c852a56c2966cf18835db62d9b34fef7ba74c726ad943928d494fa7f4735", size = 5963343, upload-time = "2026-02-11T04:23:02.934Z" },
{ url = "https://files.pythonhosted.org/packages/2d/71/64e9b1c7f04ae0027f788a248e6297d7fcc29571371fe7d45495a78172c0/pillow-12.1.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:75af0b4c229ac519b155028fa1be632d812a519abba9b46b20e50c6caa184f19", size = 7029809, upload-time = "2026-01-02T09:13:26.541Z" }, { url = "https://files.pythonhosted.org/packages/f2/26/c56ce33ca856e358d27fda9676c055395abddb82c35ac0f593877ed4562e/pillow-12.1.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:cb9bb857b2d057c6dfc72ac5f3b44836924ba15721882ef103cecb40d002d80e", size = 7029880, upload-time = "2026-02-11T04:23:04.783Z" },
] ]
[[package]] [[package]]
@@ -3767,7 +3770,7 @@ crypto = [
[[package]] [[package]]
name = "pylance" name = "pylance"
version = "2.0.0" version = "4.0.0"
source = { registry = "https://pypi.org/simple" } source = { registry = "https://pypi.org/simple" }
dependencies = [ dependencies = [
{ name = "lance-namespace" }, { name = "lance-namespace" },
@@ -3776,12 +3779,12 @@ dependencies = [
{ name = "pyarrow" }, { name = "pyarrow" },
] ]
wheels = [ wheels = [
{ url = "https://files.pythonhosted.org/packages/89/1e/7cba63f641e25243521a73c85d9f198c970546904bd32d86a74d8a5503b4/pylance-2.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ecfc291cace1aae2faeac9b329ee9b42674e6cad505fafcfe223b7fcbbc15a34", size = 51673048, upload-time = "2026-02-05T19:53:58.676Z" }, { url = "https://files.pythonhosted.org/packages/19/29/5152da1261a628c293876917b6185538bd68f4cf1420da6265b5be79d09b/pylance-4.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:7310892f3089eeddb1af1fe5c398b71cc483a3015646caceaa2f62fc92b227b2", size = 54420876, upload-time = "2026-03-30T18:18:37.525Z" },
{ url = "https://files.pythonhosted.org/packages/3d/b7/0674bea6e33a3edf466afa6d28271c495996a6f287f4426dd20d3cc08fcc/pylance-2.0.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c0397d7b9e7da2bbcc15c13edc52698a988f10e30ddb7577bebe82ec5deb82eb", size = 54124374, upload-time = "2026-02-05T20:01:43.278Z" }, { url = "https://files.pythonhosted.org/packages/99/ae/7edbbfc18c3be43eedb886e74a17826c09fdf35588b35912f2733779ea43/pylance-4.0.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:57f6a521b1b4b77a62d791850213a854093719c7d76b9641e8abcd445eb73e56", size = 56752552, upload-time = "2026-03-30T18:24:21.331Z" },
{ url = "https://files.pythonhosted.org/packages/f5/16/43ddd4dab5ae785eb6b6fea10c747ef757edebd702d8cdd2f7c451c82810/pylance-2.0.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:25be16d2797d7b684365f44e2ccdc85da210a1763cf7abb9382fbb1b132a605f", size = 57604350, upload-time = "2026-02-05T20:10:03.402Z" }, { url = "https://files.pythonhosted.org/packages/ef/88/6d8bda83224bac52806f09d3e211d8886b81500384948a753c4b24c11f35/pylance-4.0.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e433d6bddd66de99c58e472bc3e8ed1590c7ff4ff7948479254c1c2111a601a8", size = 60305704, upload-time = "2026-03-30T18:35:23.425Z" },
{ url = "https://files.pythonhosted.org/packages/ab/91/94bd6e88cc59e9a3642479a448c636307cbe3919cfbb03a2894fe40004d7/pylance-2.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:63fcedecb88ff0ab18d538b32ed5d285a814f2bab0776a75ef9f3bd42d5b6d7d", size = 54139864, upload-time = "2026-02-05T20:02:07.957Z" }, { url = "https://files.pythonhosted.org/packages/52/f3/8d8369c756c4173ea070f6964213f9b622ac278bd04a058c48d00a549177/pylance-4.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:f36dce83c11cd5d598cb0f64bad7c51fc21ed43df868b9029184a385c6bf4d84", size = 56771233, upload-time = "2026-03-30T18:25:40.012Z" },
{ url = "https://files.pythonhosted.org/packages/b1/ac/4cf5c2529cf7f10d1ed1195745c75e0817a09862297ad705ab539abab830/pylance-2.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:a3792af7bb4e77aa80436d7553b8567a3ac63be9199a0ece786a9ef2438f7930", size = 57575193, upload-time = "2026-02-05T20:10:27.163Z" }, { url = "https://files.pythonhosted.org/packages/66/e6/53e0713440685b1c76e20d72755eca2e531cc182ea9a612b4cb6a15abe50/pylance-4.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:9ca03f97f22e0b75f06378c4006d587aba26408122fd066f0e43e2b7a019c67e", size = 60260813, upload-time = "2026-03-30T18:36:07.976Z" },
{ url = "https://files.pythonhosted.org/packages/45/a3/05fd03f25c417e55f5f141e08585da8a5e5d0b17c71882b446388f203584/pylance-2.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:f08d9f87c6d6ac2d2dea6898a4364faef57d3c6a802f8faf3b62fe756fb6834b", size = 61682039, upload-time = "2026-02-05T20:30:48.272Z" }, { url = "https://files.pythonhosted.org/packages/1e/04/5f22b88c8965d3982f68f67bfe24d756e7b788e10392d2bec6f97f5eb0e3/pylance-4.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:9261c32d3bd6aaab33025a45b20c2f2554804e1bc2a1ec2bfcb06f0c9d2e59b9", size = 65137830, upload-time = "2026-03-30T18:37:33.048Z" },
] ]
[[package]] [[package]]

View File

@@ -1,6 +1,6 @@
[package] [package]
name = "lancedb" name = "lancedb"
version = "0.27.0-beta.2" version = "0.27.2"
edition.workspace = true edition.workspace = true
description = "LanceDB: A serverless, low-latency vector database for AI applications" description = "LanceDB: A serverless, low-latency vector database for AI applications"
license.workspace = true license.workspace = true

View File

@@ -1,4 +1,4 @@
# LanceDB Rust # LanceDB Rust SDK
<a href="https://crates.io/crates/vectordb">![img](https://img.shields.io/crates/v/vectordb)</a> <a href="https://crates.io/crates/vectordb">![img](https://img.shields.io/crates/v/vectordb)</a>
<a href="https://docs.rs/vectordb/latest/vectordb/">![Docs.rs](https://img.shields.io/docsrs/vectordb)</a> <a href="https://docs.rs/vectordb/latest/vectordb/">![Docs.rs](https://img.shields.io/docsrs/vectordb)</a>

View File

@@ -9,10 +9,9 @@ use aws_config::Region;
use aws_sdk_bedrockruntime::Client; use aws_sdk_bedrockruntime::Client;
use futures::StreamExt; use futures::StreamExt;
use lancedb::{ use lancedb::{
connect, Result, connect,
embeddings::{bedrock::BedrockEmbeddingFunction, EmbeddingDefinition, EmbeddingFunction}, embeddings::{EmbeddingDefinition, EmbeddingFunction, bedrock::BedrockEmbeddingFunction},
query::{ExecutableQuery, QueryBase}, query::{ExecutableQuery, QueryBase},
Result,
}; };
#[tokio::main] #[tokio::main]

Some files were not shown because too many files have changed in this diff Show More