rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-10-27 14:06:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into xuanwo/pr3851-merge

Browse Source
This commit is contained in:
Xuanwo
2026-10-26 23:35:36 +08:00
parent 1b6e9bcd61 7952ee2a71
commit 6bfd5ef0c6
137 changed files with 6687 additions and 1967 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/lancedb-update-lance-dependency/SKILL.md
+27
View File
@@ -48,6 +48,23 @@ Inputs can be a version (`7.2.0-beta.1`), a tag (`v7.2.0-beta.1`), a tag ref (`r
This updates the Rust workspace Lance dependencies through `ci/set_lance_version.py`, updates `java/pom.xml`, refreshes Cargo metadata, and prints JSON metadata containing `branch_name`, `commit_message`, and `pr_title`.
Also synchronize the Python requirements in `python/pyproject.toml`: set the
`tests` extra's exact `pylance` pin to the matching published Python release,
and update the `pylance` extra's minimum when needed to exclude older bundled
Lance majors. The minimum must still accept the test pin. Python prerelease
spelling differs from Rust (for example, `15.0.0-beta.1` becomes `15.0.0b1`).
Confirm the matching Python release is published before changing the pin.
Refresh and check `python/uv.lock` from the repository root:
```bash
uv lock --project python --index https://pypi.fury.io/lancedb/ --index https://pypi.fury.io/lance-format/ --index-strategy unsafe-best-match
uv lock --project python --check --index https://pypi.fury.io/lancedb/ --index https://pypi.fury.io/lance-format/ --index-strategy unsafe-best-match
```
This strategy considers versions across indexes; use it only with trusted
indexes. Commit the Python requirements and lockfile together with the bump.
5. Run validation:
```bash
@@ -57,6 +74,16 @@ Inputs can be a version (`7.2.0-beta.1`), a tag (`v7.2.0-beta.1`), a tag ref (`r
Fix real diagnostics and rerun clippy until it succeeds. Do not skip warnings.
Run the Python dependency guards with Python 3.11+ and `packaging` installed:
```bash
python3 -m unittest discover -s ci/tests -p test_python_dependencies.py -v
```
These checks reject older Lance majors and ensure the `pylance` and `tests`
extras remain compatible. The full CI-script suite uses
`python3 -m unittest discover -s ci/tests -v`.
6. Inspect `git status --short` and `git diff` to ensure only the Lance dependency update and required compatibility fixes are present.
7. If the task only asks to prepare local changes, stop here and report the changed files and validation result.
.bumpversion.toml
+1 -1
View File
@@ -1,5 +1,5 @@
[tool.bumpversion]
current_version = "0.41.0-beta.13"
current_version = "0.41.0-beta.14"
parse = """(?x)
(?P<major>0|[1-9]\\d*)\\.
(?P<minor>0|[1-9]\\d*)\\.
.github/workflows/ci-scripts.yml
+6
View File
@@ -5,6 +5,8 @@ on:
branches:
- main
paths:
- Cargo.toml
- python/pyproject.toml
- ci/check_cargo_lock.py
- ci/check_sdist.py
- ci/set_lance_version.py
@@ -12,6 +14,8 @@ on:
- .github/workflows/ci-scripts.yml
pull_request:
paths:
- Cargo.toml
- python/pyproject.toml
- ci/check_cargo_lock.py
- ci/check_sdist.py
- ci/set_lance_version.py
@@ -30,5 +34,7 @@ jobs:
- uses: actions/setup-python@v6
with:
python-version: "3.13"
- name: Install test dependencies
run: python -m pip install packaging
- name: Run tests
run: python -m unittest discover -s ci/tests -v
.github/workflows/docs.yml
+1
View File
@@ -38,6 +38,7 @@ jobs:
uses: actions/checkout@v6
- name: Install dependencies needed for ubuntu
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
rustup update && rustup default
- name: Set up Python
.github/workflows/pypi-publish.yml
+3
View File
@@ -15,6 +15,9 @@ on:
- ci/check_cargo_lock.py
- ci/check_sdist.py
- python/pyproject.toml
- python/Cargo.toml
- python/vendor/**
- rust/lancedb/Cargo.toml
- Cargo.toml # Change in dependency frequently breaks builds
- Cargo.lock
.github/workflows/rust.yml
+94 -6
View File
@@ -29,6 +29,63 @@ env:
RUST_BACKTRACE: "1"
jobs:
dependency-overrides:
name: Downstream Lance override
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
- uses: actions-rust-lang/setup-rust-toolchain@v1
- name: Read the pinned Lance tag
id: lance
run: |
python3 - <<'PYTHON'
import os
import tomllib
from pathlib import Path
manifest = tomllib.loads(Path("Cargo.toml").read_text())
tag = manifest["workspace"]["dependencies"]["lance"]["tag"]
with open(os.environ["GITHUB_OUTPUT"], "a") as output:
output.write(f"tag={tag}\n")
PYTHON
- uses: actions/checkout@v6
with:
repository: lance-format/lance
ref: ${{ steps.lance.outputs.tag }}
path: local-lance
- name: Resolve a downstream workspace with a local Lance override
run: |
mkdir -p downstream/src
touch downstream/src/lib.rs
cat > downstream/Cargo.toml <<'TOML'
[package]
name = "lancedb-local-lance-consumer"
version = "0.1.0"
edition = "2024"
[workspace]
[dependencies]
lancedb = { path = "../rust/lancedb", default-features = false }
lance-arrow = { path = "../local-lance/rust/lance-arrow" }
[patch."https://github.com/lance-format/lance.git"]
lance-arrow = { path = "../local-lance/rust/lance-arrow" }
TOML
cargo metadata --manifest-path downstream/Cargo.toml --format-version 1 > downstream/graph.json
python3 - <<'PYTHON'
import json
from pathlib import Path
graph = json.loads(Path("downstream/graph.json").read_text())
packages = [p for p in graph["packages"] if p["name"] == "lance-arrow"]
assert len(packages) == 1, [p["id"] for p in packages]
assert Path(packages[0]["manifest_path"]).resolve() == Path(
"local-lance/rust/lance-arrow/Cargo.toml"
).resolve()
PYTHON
lint:
timeout-minutes: 30
runs-on: ubuntu-24.04
@@ -245,12 +302,6 @@ jobs:
include:
- target: x86_64-pc-windows-msvc
runner: windows-2022
# windows-11-arm is a standard runner, so it is free on public repos.
# Running natively lets the aarch64 tests actually execute -- this
# job used to cross-compile them and then skip the test step, paying
# full codegen and link cost for a compile check.
- target: aarch64-pc-windows-msvc
runner: windows-11-arm
runs-on: ${{ matrix.runner }}
defaults:
run:
@@ -279,6 +330,43 @@ jobs:
# dependency graph a second time.
cargo test --profile ci --features aws,remote --locked --target ${{ matrix.target }}
# Run the aarch64 Windows tests only on main so PRs wait for a single
# Windows platform. windows-11-arm is a standard runner, so it is free on
# public repos. Running natively lets the aarch64 tests actually execute --
# this job used to cross-compile them and then skip the test step, paying
# full codegen and link cost for a compile check.
windows-arm:
# Keep the historical name so the check shows up as before on main.
name: windows (aarch64-pc-windows-msvc, windows-11-arm)
if: github.ref == 'refs/heads/main'
runs-on: windows-11-arm
defaults:
run:
working-directory: rust/lancedb
steps:
- uses: actions/checkout@v6
- name: Set target
run: rustup target add aarch64-pc-windows-msvc
- uses: Swatinem/rust-cache@v2
with:
# Restore everywhere, but only save from main. Per-PR saves are
# unreadable outside their own branch anyway, since GitHub scopes
# caches to the creating ref.
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install Protoc v21.12
run: choco install --no-progress protoc
- name: Build
run: |
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
cargo build --profile ci --features aws,remote --tests --locked --target aarch64-pc-windows-msvc
- name: Run tests
run: |
$env:VCPKG_ROOT = $env:VCPKG_INSTALLATION_ROOT
# `--target` has to match the build step above. Without it cargo uses
# target/ci/ rather than target/<triple>/ci/ and rebuilds the entire
# dependency graph a second time.
cargo test --profile ci --features aws,remote --locked --target aarch64-pc-windows-msvc
msrv:
# Check the minimum supported Rust version
name: MSRV Check - Rust v${{ matrix.msrv }}
Cargo.lock
Generated
+7 -7
View File
@@ -2230,9 +2230,9 @@ dependencies = [
[[package]]
name = "ctutils"
version = "0.4.2"
version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e"
checksum = "12218662ccac37d9714f2191252553d67585b97df8c839cbe49e0bde278dc865"
dependencies = [
"cmov",
]
@@ -5700,7 +5700,7 @@ dependencies = [
[[package]]
name = "lancedb"
version = "0.41.0-beta.13"
version = "0.41.0-beta.14"
dependencies = [
"ahash",
"anyhow",
@@ -5798,7 +5798,7 @@ dependencies = [
[[package]]
name = "lancedb-nodejs"
version = "0.41.0-beta.13"
version = "0.41.0-beta.14"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -5823,7 +5823,7 @@ dependencies = [
[[package]]
name = "lancedb-python"
version = "0.41.0-beta.13"
version = "0.41.0-beta.14"
dependencies = [
"arc-swap",
"arrow",
@@ -10446,9 +10446,9 @@ dependencies = [
[[package]]
name = "tokio"
version = "1.53.1"
version = "1.53.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed"
checksum = "ce3335fa71841cda333a58d7615b03901380ecf09d59b3296d21f8bbac0dde4e"
dependencies = [
"bytes",
"libc",
ci/set_lance_version.py
+14
View File
@@ -2,6 +2,8 @@ import argparse
import re
import sys
import json
import tomllib
from pathlib import Path
LANCE_GIT_URL = "https://github.com/lance-format/lance.git"
@@ -148,6 +150,18 @@ def update_cargo_toml(line_updater):
with open("Cargo.toml", "w") as f:
f.writelines(new_lines)
core_manifest = Path("rust/lancedb/Cargo.toml")
if core_manifest.exists():
lines = core_manifest.read_text().splitlines(keepends=True)
for index, line in enumerate(lines):
if line.startswith("lance-arrow ="):
updated = tomllib.loads(line_updater(line))["lance-arrow"]
config = {"version": updated} if isinstance(updated, str) else updated
if "path" in config:
config["path"] = "../../" + config["path"]
lines[index] = dict_to_toml_line("lance-arrow", config)
core_manifest.write_text("".join(lines))
def set_stable_version(version: str):
"""
ci/tests/test_python_dependencies.py
+47
View File
@@ -0,0 +1,47 @@
import tomllib
import unittest
from pathlib import Path
from packaging.requirements import Requirement
from packaging.version import Version
REPO_ROOT = Path(__file__).resolve().parents[2]
class PythonDependenciesTest(unittest.TestCase):
def setUp(self):
with (REPO_ROOT / "python/pyproject.toml").open("rb") as f:
self.project = tomllib.load(f)["project"]
self.pylance = Requirement(self.project["optional-dependencies"]["pylance"][0])
def test_pylance_extra_excludes_older_lance_majors(self):
with (REPO_ROOT / "Cargo.toml").open("rb") as f:
lance = tomllib.load(f)["workspace"]["dependencies"]["lance"]
if "version" not in lance:
self.skipTest("Local Lance checkout has no version requirement")
major = Version(lance["version"].lstrip("=")).major
for older_major in range(major):
for suffix in (".0.0", ".0.0b6", ".999.999"):
version = f"{older_major}{suffix}"
with self.subTest(version=version):
self.assertFalse(
self.pylance.specifier.contains(version, prereleases=True),
f"The pylance extra must exclude Lance {version}; "
f"the bundled Lance is major {major}",
)
def test_pylance_extra_accepts_tested_version(self):
requirements = [
Requirement(dep) for dep in self.project["optional-dependencies"]["tests"]
]
tested = next(dep for dep in requirements if dep.name == "pylance")
pin = next(spec for spec in tested.specifier if spec.operator == "==")
self.assertTrue(
self.pylance.specifier.contains(pin.version, prereleases=True),
f"The tests and pylance extras must be installable together: {tested}",
)
if __name__ == "__main__":
unittest.main()
ci/tests/test_set_lance_version.py
+28
View File
@@ -4,6 +4,7 @@ import subprocess
import sys
import tempfile
import textwrap
import tomllib
import unittest
from pathlib import Path
@@ -83,6 +84,13 @@ lance_datafusion = {{ "version" = "=8.2.0-beta.4", "features" = ["substrait"], "
with self.subTest(version=version), tempfile.TemporaryDirectory() as tmp:
workdir = Path(tmp)
(workdir / "Cargo.toml").write_text(CARGO_TOML)
core_manifest = workdir / "rust/lancedb/Cargo.toml"
core_manifest.parent.mkdir(parents=True)
core_manifest.write_text(
"[dependencies]\nlance-arrow = { "
'version = "=1.0.0", git = "https://github.com/lance-format/lance.git", '
'tag = "v1.0.0" }\narrow.workspace = true\n'
)
command_log = workdir / "commands.log"
fake_bin = workdir / "bin"
fake_bin.mkdir()
@@ -111,6 +119,26 @@ lance_datafusion = {{ "version" = "=8.2.0-beta.4", "features" = ["substrait"], "
+ updated_dependencies
+ UNTOUCHED_DEPENDENCIES,
)
core = tomllib.loads(core_manifest.read_text())["dependencies"]
dependency = core["lance-arrow"]
if isinstance(dependency, str):
dependency = {"version": dependency}
if version == "local":
self.assertEqual(
dependency["path"], "../../../lance/rust/lance-arrow"
)
else:
expected_version = {
"stable": "9.9.9",
"preview": "10.0.0-beta.3",
}.get(version, version.removeprefix("v"))
self.assertEqual(dependency["version"], f"={expected_version}")
if "beta" in expected_version:
self.assertEqual(dependency["git"], LANCE_GIT_URL)
self.assertEqual(dependency["tag"], f"v{expected_version}")
else:
self.assertNotIn("git", dependency)
self.assertEqual(core["arrow"], {"workspace": True})
commands = command_log.read_text().splitlines()
for command in expected_commands:
self.assertTrue(
deny.toml
+13 -1
View File
@@ -110,8 +110,18 @@ ignore = [
# compact_str/smol_str, so clearing this requires polars to migrate.
# https://rustsec.org/advisories/RUSTSEC-2026-0249
{ id = "RUSTSEC-2026-0249", reason = "smartstring unmaintained via polars; no fixed upstream release" },
# crypto-bigint < 0.7.6: timing side-channels in constant-time ops,
# incl. subtraction, zero counting, division and Montgomery multiplication.
# Reached only through aws-sdk-s3 sigv4a -> aws-sigv4 (p256/ecdsa); signed
# requests carry the SigV4a credential-derived key. The patched 0.7.6 was
# published 2026-10-23, but no published aws-sigv4 release uses it yet
# (latest 1.6.3 pins crypto-bigint ^0.5), so clearing requires smithy-rs to
# ship an updated aws-sigv4.
# https://rustsec.org/advisories/RUSTSEC-2026-0387
{ id = "RUSTSEC-2026-0387", reason = "crypto-bigint via aws-sigv4 (sigv4a); no fixed upstream aws-sigv4 release yet" },
]
# ---------------------------------------------------------------------------
# Licenses: only allow licenses we've reviewed as compatible with Apache-2.0.
# ---------------------------------------------------------------------------
@@ -167,7 +177,9 @@ wildcards = "deny"
# `[workspace.dependencies]` table: any crate used by more than one member must
# go through `workspace = true`, and entries nothing uses are an error. This
# keeps versions from drifting between the core crate and the bindings.
workspace-dependencies = { duplicates = "deny", unused = "deny" }
# Python bundles a patched path crate while the Rust SDK uses the overridable
# upstream source. Check shared external declarations, not these path bindings.
workspace-dependencies = { duplicates = "deny", unused = "deny", include-path-dependencies = false }
# Internal workspace crates reference each other via `path = "..."`, which
# cargo-deny sees as a wildcard version. That's fine for private workspace
# members (not published to crates.io), so allow it specifically for paths.
docs/src/java/java.md
+1 -1
View File
@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
<dependency>
<groupId>com.lancedb</groupId>
<artifactId>lancedb-core</artifactId>
<version>0.41.0-beta.13</version>
<version>0.41.0-beta.14</version>
</dependency>
```
docs/src/js/classes/Connection.md
+17 -63
View File
@@ -653,94 +653,48 @@ See [Listing](Listing.md) for pagination state and error behavior.
### listNamespaces()
```ts
abstract listNamespaces(namespacePath?, options?): Promise<ListNamespacesResponse>
abstract listNamespaces(namespacePath?, options?): Listing<string>
```
List the immediate child namespaces under the given parent.
Results may be paginated. To retrieve subsequent pages, pass the
`pageToken` returned by a previous call.
Lazily list immediate child namespaces, 100 per page by default.
#### Parameters
* **namespacePath?**: `string`[]
The parent namespace path. Defaults
to the root namespace if omitted.
* **options?**: `Partial`&lt;[`ListNamespacesOptions`](../interfaces/ListNamespacesOptions.md)&gt;
Pagination options
(`pageToken`, `limit`).
* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)
#### Returns
`Promise`&lt;[`ListNamespacesResponse`](../interfaces/ListNamespacesResponse.md)&gt;
Child namespace names and
an optional token for fetching the next page.
[`Listing`](Listing.md)&lt;`string`&gt;
***
### listTables()
#### listTables(options)
```ts
abstract listTables(options?): Promise<ListTablesResponse>
abstract listTables(namespacePath?, options?): Listing<string>
```
List a page of the tables in this database.
Lazily list tables in a namespace, requesting 100 per page by default.
To retrieve the tables after the page, pass the `pageToken` the response
carries back in. A page can be shorter than `limit` without being the last
one, so walk until a response carries no page token:
```ts
const names = [];
let pageToken = undefined;
do {
const page = await conn.listTables({ pageToken, limit: 100 });
names.push(...page.tables);
pageToken = page.pageToken;
} while (pageToken);
```
##### Parameters
* **options?**: `Partial`&lt;[`ListTablesOptions`](../interfaces/ListTablesOptions.md)&gt;
Pagination options
(`pageToken`, `limit`).
##### Returns
`Promise`&lt;[`ListTablesResponse`](../interfaces/ListTablesResponse.md)&gt;
A page of table names and an
optional token for the tables after it.
#### listTables(namespacePath, options)
```ts
abstract listTables(namespacePath?, options?): Promise<ListTablesResponse>
```
List a page of the tables in this database.
##### Parameters
#### Parameters
* **namespacePath?**: `string`[]
The namespace path to list tables from
(defaults to root namespace)
* **options?**: `Partial`&lt;[`ListTablesOptions`](../interfaces/ListTablesOptions.md)&gt;
Pagination options
(`pageToken`, `limit`).
* **options?**: [`ListingOptions`](../interfaces/ListingOptions.md)
##### Returns
#### Returns
`Promise`&lt;[`ListTablesResponse`](../interfaces/ListTablesResponse.md)&gt;
[`Listing`](Listing.md)&lt;`string`&gt;
A page of table names and an
optional token for the tables after it.
#### Example
```ts
for await (const name of conn.listTables([], { pageLimit: 20 })) {
console.log(name);
}
```
***
docs/src/js/classes/Job.md
+8 -5
View File
@@ -17,7 +17,8 @@ extra round trip on every call:
- [Job.refresh](Job.md#refresh) and [Job.status](Job.md#status) fetch the whole record.
- [Job.wait](Job.md#wait) records the terminal state it establishes, but not the
rest of the record.
- Everything is null until one of those runs.
- Registry details are null until one of those runs. Query handles report
jobType "query" immediately.
## Accessors
@@ -57,8 +58,10 @@ get id(): null | string
Identifies the operation on the server that is running it.
Operations that run in this process have no server id. The value is
opaque: parsing it or storing it to resume the job later is not supported.
Registry handles expose a registry job ID; query handles expose a
connection-scoped SQL query UUID string. Local handles have no ID.
Query handles report jobType "query". Their IDs are not registry IDs or
durable resume tokens.
#### Returns
@@ -72,8 +75,8 @@ opaque: parsing it or storing it to resume the job later is not supported.
get jobType(): null | string
```
The job's type, as the server names it. Null for an in-process job, which
has no server-side record.
"query" immediately for SQL-backed handles; otherwise the cached
server-reported job type. Null for in-process handles.
#### Returns
docs/src/js/globals.md
-4
View File
@@ -108,10 +108,6 @@
- [JobEventsOptions](interfaces/JobEventsOptions.md)
- [JobFailureInfo](interfaces/JobFailureInfo.md)
- [JobInfo](interfaces/JobInfo.md)
- [ListNamespacesOptions](interfaces/ListNamespacesOptions.md)
- [ListNamespacesResponse](interfaces/ListNamespacesResponse.md)
- [ListTablesOptions](interfaces/ListTablesOptions.md)
- [ListTablesResponse](interfaces/ListTablesResponse.md)
- [ListingOptions](interfaces/ListingOptions.md)
- [MaterializedViewDefinition](interfaces/MaterializedViewDefinition.md)
- [MemtableStats](interfaces/MemtableStats.md)
docs/src/js/interfaces/IvfRqOptions.md
+2 -1
View File
@@ -58,7 +58,8 @@ Number of bits per dimension for residual quantization.
This value controls how much each residual component is compressed. The more
bits, the more accurate the index will be but the slower search. Typical values
are small integers; the default is 1 bit per dimension.
are small integers. If unset, uses Lance's default (currently 5 bits per
dimension). Set to 1 for maximum compression.
***
docs/src/js/interfaces/ListNamespacesOptions.md
-27
View File
@@ -1,27 +0,0 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / ListNamespacesOptions
# Interface: ListNamespacesOptions
## Properties
### limit?
```ts
optional limit: number;
```
An optional limit to the number of results to return.
***
### pageToken?
```ts
optional pageToken: string;
```
Token from a previous response for pagination.
docs/src/js/interfaces/ListNamespacesResponse.md
-23
View File
@@ -1,23 +0,0 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / ListNamespacesResponse
# Interface: ListNamespacesResponse
## Properties
### namespaces
```ts
namespaces: string[];
```
***
### pageToken?
```ts
optional pageToken: string;
```
docs/src/js/interfaces/ListTablesOptions.md
-35
View File
@@ -1,35 +0,0 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / ListTablesOptions
# Interface: ListTablesOptions
## Properties
### limit?
```ts
optional limit: number;
```
An upper bound on how many tables to return.
Defaults to 100. Zero returns an empty page without a continuation token.
A page may hold fewer than this and still not be the last one, so keep
going while the response carries a page token rather than while pages are
full.
***
### pageToken?
```ts
optional pageToken: string;
```
Token from a previous response, to resume listing where it left off.
The token is opaque: it carries whatever the database needs to resume, and
callers should not construct or interpret one.
docs/src/js/interfaces/ListTablesResponse.md
-23
View File
@@ -1,23 +0,0 @@
[**@lancedb/lancedb**](../README.md) • **Docs**
***
[@lancedb/lancedb](../globals.md) / ListTablesResponse
# Interface: ListTablesResponse
## Properties
### pageToken?
```ts
optional pageToken: string;
```
***
### tables
```ts
tables: string[];
```
docs/src/js/interfaces/QueryExecutionOptions.md
+7 -6
View File
@@ -20,13 +20,14 @@ How to return blob v2 columns. Applies to `toArrow()` and `toArray()`.
- `"descriptions"` (default): each blob is a descriptor. Read the bytes
with [Table.fetchBlobs](../classes/Table.md#fetchblobs) or [Table.fetchBlobFiles](../classes/Table.md#fetchblobfiles).
- `"bytes"`: each blob column holds the blob bytes as `LargeBinary`,
including blobs nested in structs. The query also reads `_rowid` to
fetch the bytes, and leaves it out of the result unless
[QueryBase.withRowId](../classes/QueryBase.md#withrowid) was called. All the bytes are held in
memory.
- `"bytes"`: each blob column, including blobs nested in structs, holds
the bytes as `LargeBinary`. The query also reads `_rowid` to fetch them, and
leaves it out of the result unless [QueryBase.withRowId](../classes/QueryBase.md#withrowid) was
called. All the bytes are held in memory.
Blobs nested in a list keep their descriptors.
Selecting a parent struct or directly aliasing a blob or struct also returns
bytes. Blobs inside lists keep their descriptors. Arbitrary projection
expressions are not materialized as bytes.
The rows and the bytes come from the same table version. If the table
changes while the query runs, the query runs again, up to three times.
docs/src/js/interfaces/TableNamesOptions.md
+1 -1
View File
@@ -8,7 +8,7 @@
## Deprecated
Use [ListTablesOptions](ListTablesOptions.md) with [Connection.listTables](../classes/Connection.md#listtables)
Use [ListingOptions](ListingOptions.md) with [Connection.listTables](../classes/Connection.md#listtables)
instead.
## Properties
docs/src/python/python.md
+120 -8
View File
@@ -20,20 +20,34 @@ pip install lancedb
The following methods describe the synchronous API client. There
is also an [asynchronous API client](#connections-asynchronous).
Connections to `db://` databases reject local-only options with
`NotImplementedError`. These include `session` and `manifest_enabled=True` on
connect, per-table storage and write options on create, and `index_cache_size`,
`storage_options`, or `location` on open (where the method accepts them).
Omit these options to use server-managed storage and caching. Connection-level
`storage_options` supported by an Enterprise deployment remain available.
The synchronous remote `create_table` uses the same positional order as the
local API: `name, data, schema, mode, exist_ok, on_bad_vectors, fill_value,
embedding_functions`. Prefer keyword arguments for optional parameters.
## Connections (Synchronous)
::: lancedb.connect
::: lancedb.db.DBConnection
Resource listings (`list_secrets`, `list_views`, `list_jobs`, `list_functions`, and
`list_materialized_views`) return lazy iterators. Use `list(db.list_views())` to
collect synchronous results, or `[name async for name in db.list_views()]` with
an asynchronous connection. `page_limit` controls each request, and `page_token`
resumes a listing. Drain the iterator's cached results before saving its token.
Resource listings return iterators. Use `list(db.list_tables())` to collect all
table names. `db.list_tables().tables` is a list of the first page; `limit`
controls its page size and `page_token` resumes at a saved token. The same
patterns apply to `list_namespaces()`. Other resource listings use `page_limit`.
::: lancedb.listing.Listing
::: lancedb.listing.TableListing
::: lancedb.listing.NamespaceListing
::: lancedb.Session
## Catalogs (Synchronous)
@@ -218,10 +232,19 @@ The connected database and `default_namespace_path=["public"]` are used for
unqualified tables. Fully qualified references can still query other databases
and namespaces available to the same deployment. `execute_query` returns a
reader as soon as its initial result stream is available. `execute_query_async`
returns a query handle immediately; use it to inspect progress, open a reader,
returns a query handle once submission is accepted; use it to inspect progress, open a reader,
or cancel the query. The SQL client is initialized by the first query and
retained for the lifetime of the remote connection. Query ids are random,
connection-scoped references rather than encoded SQL or durable resume tokens:
connection-scoped references rather than encoded SQL or durable resume tokens.
Both APIs start with one Flight call. The server can stream results on that
call or return a polling continuation that the SDK follows automatically.
This behavior is the same with and without parameters. Open the reader directly;
a directly streamed query needs its reader to make progress. Dropping an unread
handle or its reader stops direct streaming. Deferred execution continues until
explicitly cancelled.
For example:
```python
import lancedb
@@ -285,6 +308,25 @@ listing a storage directory.
## Tables (Synchronous)
Local and remote tables use the same arguments for index creation, deletion,
index statistics, and search. For example, `table.delete(where="id = 1")`,
`table.create_index("cosine", 2)` (the deprecated vector API), and
`table.search([1.0, 1.0], fast_search=True)` work with either backend.
`fast_search=True` excludes rows that have not been indexed.
`create_scalar_index` defaults to `replace=True` on both backends. Both scalar
and full-text index creation accept `wait_timeout`. Remote maintenance methods
accept the local options: cleanup and compaction warn that they are no-ops,
while `optimize` reports an unsupported operation. Removed Tantivy options
produce the same explanatory errors on both backends.
The old remote keyword names `predicate` for `delete`, `index_name` for
`drop_index`, `index_uuid` for `index_stats`, and `column` for `create_fts_index`
remain available with deprecation warnings. Use `where`, `name`, `index_name`,
and `field_names`, respectively. For positional calls, use the argument order
documented by `Table` below; specify index options by keyword when migrating
code that used the former remote-only argument order.
::: lancedb.table.Table
::: lancedb.table.FragmentStatistics
@@ -301,6 +343,14 @@ listing a storage directory.
## Functions and Jobs
Annotate a blob input with `lancedb.BlobReader` for lazy, seekable access, or
with `bytes` to materialize its contents before the Function runs. Reader
inputs infer the Blob V2 type without an explicit input schema. Readers are
scoped to each invocation and support `read`, `readinto`, `seek`, `tell`,
`size`, and cursor-independent `read_range`.
::: lancedb.BlobReader
::: lancedb.functions.FunctionArtifact
::: lancedb.functions.FunctionParameter
@@ -518,12 +568,64 @@ instead of being materialized with the rest of the row.
`lancedb.BlobType` is `lance.blob.BlobType` when pylance is installed. Without
pylance, LanceDB uses a matching `lance.blob.v2` extension type so blob columns
still work. Queries return descriptors. Call
still work. Native table queries return descriptors. Call
[`fetch_blob_files`][lancedb.table.Table.fetch_blob_files] for lazy reads or
[`fetch_blobs`][lancedb.table.Table.fetch_blobs] for eager bytes.
One missing or deleted row ID rejects the entire blob-read batch; the APIs do
not return partial results.
Remote SQL returns blob bytes by default. Select `blob_descriptor(video)` to
return an openable reference instead, then use
[`open_blob_file`][lancedb.table.Table.open_blob_file] or the lazy iterable
[`open_blob_files`][lancedb.table.Table.open_blob_files]:
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
This example uses a remote SQL connection that produces source-bearing
descriptors. The descriptor contains `kind`, `position`, `size`, `blob_id`,
`blob_uri`, and a `source` struct with `uri`, `version`, `committed_at_micros`,
`branch`, `column`, `row_reference`, and `row_is_address`. The opener verifies
the source table and commit and reads the captured version without changing
the table handle's checkout. Raw native storage descriptors omit `source`;
use `fetch_blob_files(column, row_ids)` for those query results.
The singular API accepts a dictionary or Arrow struct scalar. The iterable
accepts Arrow arrays/chunked arrays or another iterable, opens handles as
iteration advances, and preserves order, duplicates, and nulls as `None`.
Close each non-null handle after use. Async tables provide
`await table.open_blob_file(...)` and
`async for file in table.open_blob_files(...)`.
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
For an async table, the iterable is consumed with `async for`:
```python
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
```
Payload bytes load on reads. Opening an external blob with unspecified size
can issue a metadata-only HEAD request. Remote reads use authenticated range
requests that validate the captured URI and commit timestamp.
Blobs inside lists can be written and queried as descriptors. They are omitted
from `blob_columns()` because row ids do not identify individual list elements,
so the fetch APIs cannot read their bytes yet.
@@ -631,6 +733,16 @@ can be used to create, list, or open tables.
::: lancedb.listing.AsyncListing
`async for name in db.list_tables()` loads pages as needed. To inspect the
first page, use `page = await db.list_tables()` and then `page.tables`. Reading
the page list does not consume the iterator. `page.page_token` is the token for
the next request; drain cached results before using it to resume in a new
listing. The same behavior applies to namespaces.
::: lancedb.listing.AsyncTableListing
::: lancedb.listing.AsyncNamespaceListing
## Namespaces (Asynchronous)
::: lancedb.connect_namespace_async
docs/web-tests/ts/connection.test.ts
+11 -5
View File
@@ -4,6 +4,12 @@ import { afterEach, describe, expect, jest, test } from "@jest/globals";
import * as lancedb from "@lancedb/lancedb";
import { withTempDirectory } from "./util.ts";
async function collectNames(items: AsyncIterable<string>): Promise<string[]> {
const names: string[] = [];
for await (const name of items) names.push(name);
return names;
}
test("connect to a local database", async () => {
await withTempDirectory(async (tempDir) => {
const cwd = process.cwd();
@@ -13,12 +19,12 @@ test("connect to a local database", async () => {
const db = await lancedb.connect("ex_lancedb");
// --8<-- [end:connect]
await db.createTable("probe", [{ id: 1 }]);
expect((await db.listTables()).tables).toEqual(["probe"]);
expect(await collectNames(db.listTables())).toEqual(["probe"]);
} finally {
process.chdir(cwd);
}
const reopened = await lancedb.connect(`${tempDir}/ex_lancedb`);
expect((await reopened.listTables()).tables).toEqual(["probe"]);
expect(await collectNames(reopened.listTables())).toEqual(["probe"]);
});
});
@@ -109,8 +115,8 @@ async function namespaceTableOpsExample() {
);
// Verify
console.log((await db.listNamespaces()).namespaces); // ["prod"]
console.log((await db.listNamespaces(["prod"])).namespaces); // ["recommendations", "search"]
console.log(await collectNames(db.listNamespaces())); // ["prod"]
console.log(await collectNames(db.listNamespaces(["prod"]))); // ["recommendations", "search"]
console.log(await db.tableNames(["prod", "search"])); // ["user"]
console.log(await db.tableNames(["prod", "recommendations"])); // ["user"]
// --8<-- [end:namespace_table_ops]
@@ -124,7 +130,7 @@ async function namespaceAdminOpsExample() {
await db.createNamespace(["prod"]);
await db.createNamespace(["prod", "search"]);
const childNamespaces = (await db.listNamespaces(["prod"])).namespaces;
const childNamespaces = await collectNames(db.listNamespaces(["prod"]));
console.log(`Child namespaces under ${JSON.stringify(namespace)}:`, childNamespaces);
// Child namespaces under ["prod","search"]: [ 'search' ]
docs/web-tests/ts/quickstart.test.ts
+1 -6
View File
@@ -20,12 +20,7 @@ const connections =
async function tableNames(db: lancedb.Connection) {
const names: string[] = [];
let pageToken: string | undefined;
do {
const page = await db.listTables({ pageToken });
names.push(...page.tables);
pageToken = page.pageToken;
} while (pageToken);
for await (const name of db.listTables()) names.push(name);
return names;
}
docs/web/basics/jobs.mdx
+31 -6
View File
@@ -20,19 +20,20 @@ failures.
Methods such as Python's `create_index_async` and TypeScript's
`createIndexAsync` return job handles. In OSS, the work runs in your application
process; in Enterprise, it runs on server workers. A returned handle can
represent an operation that has already finished, so check its status before
process; in Enterprise, it runs on server workers or through the SQL query
backend. A returned handle can represent an operation that has already finished,
so check its status before
waiting or requesting cancellation. See the operation's API reference for when
the method returns.
## Submit and wait for a job {#waiting}
1. **Submit the operation.** Keep the returned job handle. For an Enterprise
job, also keep its ID if another process needs to look it up later.
registry job, also keep its ID if another process needs to look it up later.
2. **Check progress.** Read the current status while the operation runs.
3. **Wait for completion.** Check that the job succeeded before using its
result. If it failed or was cancelled, inspect the failure details and any
results it committed before stopping.
result. If it failed or was cancelled, inspect the failure details and any
results it committed before stopping.
In Python, `job.status()` reports the current state and `job.wait()` waits for
completion, raising if the job fails or is cancelled. If `job.wait()` times out, the client stops waiting.
@@ -59,13 +60,37 @@ history.
## OSS and Enterprise execution {#in-process}
| | OSS | Enterprise |
The following table describes local operations and Enterprise registry jobs.
SQL-backed job handles follow the query lifecycle described below.
| | OSS | Enterprise registry job |
| --- | --- | --- |
| Execution | In your application process | On server workers |
| Job identity | No server ID | A server ID for looking up the job |
| Event history | No server event history | Available through the jobs API |
| Reattaching after a client restart | No server job to reopen | Reopen by ID while the record is retained |
### SQL-backed job handles
Remote materialized-view CREATE and REFRESH use the SQL query backend. Python's
`create_materialized_view_async()` and `mv.refresh_async()` still return job
handles with `status()`, `wait()` and `cancel()`. Synchronous creation and refresh
wait on the same SQL execution before returning.
The existing `job.job_type` identifies query-backed handles as `"query"` from
submission, and `job.id` contains the connection-scoped SQL query UUID string.
Use `describe_query(job.id)` on the submitting connection to inspect the query.
Registry handles retain their server-reported job types (available after
`status()` or `refresh()`) and registry IDs. Local handles have no ID or job type.
SQL-backed handles do not expose registry records or event history and cannot
be reopened through `open_job()` or reattached after a client restart. Their IDs
are not durable resume tokens. Cancellation follows the SQL server's
capabilities and can be unsupported.
The server decides whether SQL executes on its serving fleet or starts SQL job
workers. That placement does not change the handle's query-backed identity.
## Learn more {#next}
- [Build and repair indexes](/indexing/reindexing): submit index work.
docs/web/basics/sdk.mdx
+7
View File
@@ -92,6 +92,13 @@ manages background maintenance.
[background maintenance](/maintenance/background-jobs) rather than calling
the embedded table's `optimize()` method.
{/* Draft for the remote option validation release.
- **Remove local storage and cache settings.** `db://` connections reject local
sessions, table cache settings, and per-table storage and write options. Omit
these options so Enterprise manages storage and caching. Connection-level
storage options supported by your Enterprise deployment remain available.
*/}
For SQL connection details, see [SQL](/basics/sql). For the operating model and
deployment options, see [LanceDB Enterprise](/enterprise).
docs/web/basics/sql.mdx
+2
View File
@@ -82,6 +82,8 @@ In application code, load credentials from environment variables or a secrets
manager. The [connection guide](/basics/connect#enterprise) shows that setup.
`execute_query()` returns a blocking Arrow reader when results start arriving.
The server can stream rows on the first call or return result tickets that the
SDK follows automatically, with the same behavior for parameterized queries.
The example uses `read_all()` to collect the small result in memory. For larger
results, consume the reader in batches. To submit a query and check its status
separately, use `execute_query_async()`. See [Queries](/basics/query) for query
docs/web/docs.json
+1
View File
@@ -350,6 +350,7 @@
"pages": [
"enterprise/index",
"enterprise/architecture",
"enterprise/distributed-search",
"enterprise/deployment/index",
"deployment/storage",
"deployment/monitoring",
docs/web/enterprise/architecture.mdx
+3
View File
@@ -18,3 +18,6 @@ run on machines that can be replaced without any of them owning the data.
[Architecture](/architecture) covers this alongside the embedded form, which is
the comparison most readers want first.
See [Distributed search](/enterprise/distributed-search) for how Enterprise
distributes index execution and table-data reads.
docs/web/enterprise/distributed-search.mdx
+33
View File
@@ -0,0 +1,33 @@
---
title: "Distributed search"
sidebarTitle: "Distributed search"
description: "How LanceDB Enterprise distributes index execution and table-data reads."
icon: "network-wired"
availability:
oss: unavailable
enterprise: available
summary: >-
LanceDB Enterprise distributes index execution and table-data reads across
plan executors. Embedded LanceDB executes searches in the application process.
---
Distributed search spreads index execution, table-data reads, and caching across
multiple serving machines in LanceDB Enterprise. Query nodes plan queries and
combine results. Plan executors cache index structures and table data and execute
the remote work. See [Enterprise architecture](/enterprise/architecture).
## Embedded search {#embedded-search}
LanceDB OSS runs inside your application process. It searches indexes and reads
table data using that process's resources, including its index cache. Reading a
table from object storage does not by itself distribute query execution across
machines.
## Search interfaces {#search-interfaces}
The shared search guides describe the query APIs and index settings:
- [Vector search](/search/vector-search) and [vector indexes](/indexing/vector-index).
- [Full-text search](/search/full-text-search) and [FTS indexes](/indexing/fts-index).
- [Filtering](/search/filtering) and [scalar indexes](/indexing/scalar-index).
- [Query plans](/search/explain) for inspecting execution and runtime measurements.
docs/web/indexing/vector-index.mdx
+12 -7
View File
@@ -58,7 +58,7 @@ Lance with `index_type`.
| If your priority is... | Use this index | Why | Approx. compression ratio | Python config class |
| :--- | :--- | :--- | :--- | :--- |
| Maximum compression | `IVF_RQ` | IVF indexing with RaBitQ quantization | Around `1/32` of raw size | `IvfRq` |
| Maximum compression | `IVF_RQ` | IVF indexing with RaBitQ quantization | Around `1/32` of raw size with `num_bits=1` | `IvfRq` |
| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | IVF indexing with product quantization | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | `IvfPq` |
| Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar | Typically a little larger than `1/4` of raw size | `IvfHnswSq` |
| Highest recall / no quantization | `IVF_HNSW_FLAT` | IVF-HNSW indexing with no quantization; embedded library only | Around raw vector size plus HNSW graph overhead | `IvfHnswFlat` |
@@ -377,22 +377,27 @@ which has been compressed to $4$ chunks $\times 8$-bit integers $= 32$ bits of q
RaBitQ is an advanced quantization technique that outperforms PQ in several ways.
It needs no codebook to train, estimates distances very quickly at query-time,
and crucially, quantizes each vector in (with some small overhead) just **one bit per dimension!**
and can quantize each vector in (with some small overhead) as little as **one bit per dimension!**
In practice, RaBitQ compresses a $1024$-dimensional `float32` vector into just a few thousand bits, while maintaining good recall.
The inner workings of RaBitQ quantization are rather mathematically dense. It generates a quantization codebook by
applying a uniformly random, approximately distance-preserving orthogonal transformation of the vertices of the $d$-dimensional hypercube,
where $d$ is the dimensionality of the dataset. We defer the details, and an elegant theoretical error bound, to [the original paper.](https://arxiv.org/pdf/2405.12497)
##### Using RaBitQ
Use RaBitQ quantization by selecting quantized index types ending in the suffix `RQ`. For example, call `create_index` with `index_type="IVF_RQ"`.
##### Using RaBitQ {#using-rabitq}
Use RaBitQ quantization by selecting quantized index types ending in the suffix `RQ`. For example, import `IvfRq` from `lancedb.index` and call `table.create_index("vector", config=IvfRq())`.
Note that when using `IVF_RQ`, the dimension of the dataset must be a multiple of `8`.
`num_bits` determines how many bits are used to quantize each dimension.
`1` is the standard RaBitQ setting. Increase to `2`, `4`, or `8` bits to achieve better recall for additional storage and query-time compute.
By default, Python, Rust, and Node leave this unset and inherit Lance's default, currently `5` bits.
Higher values improve recall at the cost of additional storage and query-time compute.
To retain the previous one-bit default and maximum compression, explicitly set `num_bits=1` (for example, `IvfRq(num_bits=1)` in Python).
Five-bit quantized vector codes use roughly five times the storage of one-bit codes; total index size also includes other overhead.
<Warning title="Reading multi-bit indexes across versions">
RaBitQ-quantized indexes computed with `num_bits >= 2` use a newer on-disk layout, and cannot be read by some older LanceDB versions.
RaBitQ-quantized indexes computed with `num_bits >= 2` use a newer on-disk layout and cannot be read by some older LanceDB versions.
The new five-bit default therefore affects reader compatibility as well as index size.
Upgrade readers to versions that support multi-bit IVF_RQ before creating indexes with the default settings, or explicitly set `num_bits=1` when sharing indexes with readers that support only one-bit IVF_RQ.
</Warning>
See this [blog post](https://lancedb.com/blog/feature-rabitq-quantization/) for further discussion and benchmarking of LanceDB's RaBitQ implementation.
@@ -410,7 +415,7 @@ among all integral multiples $\{j \times 10^5: -128 \leq j \leq 127 \}$.
| Parameter | Description |
| :--- | :--- |
| `num_bits` | Bits per dimension for quantization. Only applies to `IVF_PQ`/`IVF_HNSW_PQ` (default `8`) and `IVF_RQ` (default `1`, RaBitQ); not used by `IVF_FLAT`/`IVF_SQ`/`IVF_HNSW_FLAT`/`IVF_HNSW_SQ`. Higher values improve accuracy at the cost of additional storage. |
| `num_bits` | Bits per dimension for quantization. Only applies to `IVF_PQ`/`IVF_HNSW_PQ` (default `8`) and `IVF_RQ` (`IvfRq()` inherits Lance's default, currently `5`, RaBitQ); not used by `IVF_FLAT`/`IVF_SQ`/`IVF_HNSW_FLAT`/`IVF_HNSW_SQ`. Higher values improve accuracy at the cost of additional storage. |
`max_iterations` and `sample_rate` also affect quantizer training, but since they apply to every IVF/HNSW index type (not just quantized ones), they're documented as general [Build-time Parameters](/indexing/vector-index#build-time-parameters) instead. All three are passed as keyword arguments to `create_index`, alongside `index_type`:
docs/web/ingestion/create-table.mdx
+5 -3
View File
@@ -148,9 +148,11 @@ non-nullable column, or provide actual nulls for it, ingestion fails; nullable c
omitted or written with null values. Without an explicit schema, Python infers list-like vector
values as fixed-size `float32` vector fields from the observed dimension.
For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
`fill_value`, and `"null"` writes nulls.
For Python ingestion, malformed vectors fail by default (`on_bad_vectors="error"`). This includes
vectors with incorrect sizes, NaNs, or null components, including pandas NaNs converted to Arrow
nulls. Use `"drop"` to remove affected rows, or `"null"` to replace affected vectors with nulls.
Use `"fill"` to replace NaN or null components with `fill_value` while preserving valid components.
This also pads short vectors with `fill_value` and truncates long vectors to the expected dimension.
### From an Arrow Table {#from-an-arrow-table}
You can also create LanceDB tables directly from Arrow tables.
docs/web/integrations/reranking/openai.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ credentials from environment variables or a secrets manager.
## Accepted Arguments {#accepted-arguments}
| Argument | Type | Default | Description |
| --- | --- | --- | --- |
| `model_name` | `str` | `"gpt-4-turbo-preview"` | The name of the reranker model to use.|
| `model_name` | `str` | `"gpt-5.6-sol"` | The name of the reranker model to use.|
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
| `api_key` | `str` | `None` | The API key to use. If None, will use the OPENAI_API_KEY environment variable.
docs/web/management/function.mdx
+34
View File
@@ -205,6 +205,40 @@ Function output blobs follow the same compaction and cleanup lifecycle as
ingested Managed blobs. Retained versions continue to own their payloads;
cleanup reclaims an object only after its retained references are gone.
### Draft: incremental blob inputs (upcoming)
This feature requires an SDK that exports `BlobReader` and a compatible
Enterprise runtime with reader input support enabled. Coordinate the runtime
upgrade before registering or executing reader Functions.
Annotate a row Function parameter with `BlobReader` to read a Blob v2 input
incrementally. The signature selects the access mode without another
registration option. For example, read the first MP4 box type without loading
the whole video:
```python
from lancedb import BlobReader, udf
@udf
def first_box_type(video: BlobReader) -> bytes:
video.seek(4)
return video.read(4)
```
The worker passes a seekable reader backed by the stored blob. `read(size)`
reads from the current cursor; `read_range(offset, length)` reads a byte range
without moving it. `size()` returns the payload size without reading it.
Each row call receives its own reader, which closes when the call returns or
raises. Do not retain it for another invocation.
Use `video: bytes` when the Function needs the complete payload. Bytes inputs
materialize before invocation and remain subject to eager input limits.
Reader inputs keep the unread payload out of the input batch, but calling
`video.read()` without a size loads all remaining bytes in Python. Reader
inputs are supported in row mode. Return values remain ordinary values such
as `bytes`; this feature does not add streamed outputs.
## Declare the environment {#environment}
The function runs remotely, so its dependencies are part of its definition:
docs/web/quickstart.mdx
+27
View File
@@ -58,6 +58,33 @@ uv add --prerelease allow --index https://pypi.fury.io/lancedb/ lancedb
```
</CodeGroup>
If you also need the optional `lance` Python API, install the `pylance` extra and
include the Lance preview index:
<CodeGroup>
```bash Python (pip) icon="terminal"
pip install --pre \
--extra-index-url https://pypi.fury.io/lancedb/ \
--extra-index-url https://pypi.fury.io/lance-format/ \
"lancedb[pylance]"
```
```bash Python (uv) icon="terminal"
uv pip install --prerelease allow \
--index https://pypi.fury.io/lancedb/ \
--index https://pypi.fury.io/lance-format/ \
--index-strategy unsafe-best-match \
"lancedb[pylance]"
```
</CodeGroup>
The LanceDB preview index contains old `pylance` releases; current Lance previews
are published to the `lance-format` index. By default, uv only considers versions
from the first index containing a package, which can select an old `pylance` or
backtrack to an older LanceDB release. `unsafe-best-match` searches all configured
indexes, like pip; use it only with indexes you trust. Install the extra rather
than an unconstrained `pylance` so LanceDB's minimum supported version is enforced.
<Note>
Pre-release builds receive the same level of testing as stable releases, but their availability is not guaranteed
for more than 6 months after release. For real-world workloads, we recommend you use the latest stable release
docs/web/search/vector-search.mdx
+1 -1
View File
@@ -235,7 +235,7 @@ The table below summarizes the behavior of `_distance` in search results based o
See [vector indexes](/indexing/vector-index) for index configuration. The
following sections cover query-time settings.
### Tuning `approx_mode`
### Tuning `approx_mode` {#tuning-approx-mode}
Use `approx_mode` when you want to adjust the speed/recall trade-off for approximate vector search at query time. This setting currently applies only to RQ-quantized indexes, such as `IVF_RQ`; other index types ignore it.
java/lancedb-core/pom.xml
+1 -1
View File
@@ -8,7 +8,7 @@
<parent>
<groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId>
<version>0.41.0-beta.13</version>
<version>0.41.0-beta.14</version>
<relativePath>../pom.xml</relativePath>
</parent>
java/pom.xml
+1 -1
View File
@@ -6,7 +6,7 @@
<groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId>
<version>0.41.0-beta.13</version>
<version>0.41.0-beta.14</version>
<packaging>pom</packaging>
<name>${project.artifactId}</name>
<description>LanceDB Java SDK Parent POM</description>
nodejs/Cargo.toml
+1 -1
View File
@@ -1,7 +1,7 @@
[package]
name = "lancedb-nodejs"
edition.workspace = true
version = "0.41.0-beta.13"
version = "0.41.0-beta.14"
publish = false
license.workspace = true
description.workspace = true
nodejs/__test__/connection.test.ts
+82 -74
View File
@@ -5,15 +5,15 @@ import { mkdirSync, readdirSync } from "fs";
import { join } from "path";
import { Field, Float64, Schema } from "apache-arrow";
import * as tmp from "tmp";
import {
Connection,
ListTablesResponse,
Table,
connect,
connectNamespace,
} from "../lancedb";
import { Connection, Table, connect, connectNamespace } from "../lancedb";
import { LocalTable } from "../lancedb/table";
async function collectNames(items: AsyncIterable<string>): Promise<string[]> {
const names: string[] = [];
for await (const name of items) names.push(name);
return names;
}
describe("when connecting", () => {
let tmpDir: tmp.DirResult;
beforeEach(() => {
@@ -135,7 +135,7 @@ describe("given a connection", () => {
await db.close();
expect(db.isOpen()).toBe(false);
await expect(db.tableNames()).rejects.toThrow("Connection is closed");
await expect(db.listTables()).rejects.toThrow("Connection is closed");
expect(() => db.listTables()).toThrow("Connection is closed");
await expect(db.renameTable("a", "b")).rejects.toThrow(
"Connection is closed",
);
@@ -247,15 +247,15 @@ describe("given a connection", () => {
await db.createTable(name, [{ id: 1 }]);
}
const first = await db.listTables(namespacePath, { limit: 1 });
expect(first.tables).toEqual(["a"]);
expect(first.pageToken).toBeDefined();
const first = db.listTables(namespacePath, { pageLimit: 1 });
expect(await first.next()).toEqual({ done: false, value: "a" });
expect(first.pageToken()).toBeDefined();
const second = await db.listTables(namespacePath, {
limit: 1,
pageToken: first.pageToken,
const second = db.listTables(namespacePath, {
pageLimit: 1,
pageToken: first.pageToken(),
});
expect(second.tables).toEqual(["b"]);
expect(await second.next()).toEqual({ done: false, value: "b" });
});
});
it("should keep the existing table when mode is exist_ok", async () => {
@@ -299,19 +299,18 @@ describe("given a connection", () => {
await db.createTable("a", [{ id: 1 }]);
await db.createTable("c", [{ id: 1 }]);
const all = await db.listTables();
expect(all.tables).toEqual(["a", "b", "c"]);
expect(all.pageToken).toBeUndefined();
expect(await collectNames(db.listTables())).toEqual(["a", "b", "c"]);
const first = await db.listTables({ limit: 1 });
expect(first.tables).toEqual(["a"]);
expect(first.pageToken).toBeDefined();
const first = db.listTables([], { pageLimit: 1 });
expect(first.numPageResults()).toBe(0);
expect(await first.next()).toEqual({ done: false, value: "a" });
expect(first.pageToken()).toBeDefined();
const second = await db.listTables({
limit: 1,
pageToken: first.pageToken,
const second = db.listTables([], {
pageLimit: 1,
pageToken: first.pageToken(),
});
expect(second.tables).toEqual(["b"]);
expect(await second.next()).toEqual({ done: false, value: "b" });
});
it("should visit every table exactly once when walking pages", async () => {
@@ -322,18 +321,9 @@ describe("given a connection", () => {
await db.createTable(name, [{ id: 1 }]);
}
const seen: string[] = [];
let pageToken: string | undefined = undefined;
do {
const page: ListTablesResponse = await db.listTables({
limit: 2,
pageToken,
});
seen.push(...page.tables);
pageToken = page.pageToken;
} while (pageToken);
expect(seen).toEqual(created);
expect(await collectNames(db.listTables([], { pageLimit: 2 }))).toEqual(
created,
);
});
it.each([2 ** 32, 2 ** 32 + 1, Number.MAX_SAFE_INTEGER])(
@@ -347,11 +337,10 @@ describe("given a connection", () => {
]) {
await expect(names).resolves.toEqual(["a", "b"]);
}
for (const page of [
db.listTables({ limit }),
db.listTables([], { limit }),
]) {
await expect(page).resolves.toEqual({ tables: ["a", "b"] });
for (const namespacePath of [undefined, []]) {
expect(() =>
db.listTables(namespacePath, { pageLimit: limit }),
).toThrow(/limit/);
}
},
);
@@ -366,8 +355,11 @@ describe("given a connection", () => {
])("should reject an invalid table listing limit %s", async (limit) => {
await expect(db.tableNames({ limit })).rejects.toThrow(/limit/);
await expect(db.tableNames([], { limit })).rejects.toThrow(/limit/);
await expect(db.listTables({ limit })).rejects.toThrow(/limit/);
await expect(db.listTables([], { limit })).rejects.toThrow(/limit/);
for (const namespacePath of [undefined, []]) {
expect(() => db.listTables(namespacePath, { pageLimit: limit })).toThrow(
/limit/,
);
}
});
it("should preserve zero, small and unsigned-32-bit table listing limits", async () => {
@@ -379,11 +371,21 @@ describe("given a connection", () => {
[2 ** 32 - 1, ["a", "b"]],
] as const) {
await expect(db.tableNames({ limit })).resolves.toEqual(names);
const page = await db.listTables({ limit });
expect(page.tables).toEqual(names);
}
await expect(db.tableNames()).resolves.toEqual(["a", "b"]);
await expect(db.listTables()).resolves.toEqual({ tables: ["a", "b"] });
for (const namespacePath of [undefined, []]) {
for (const pageLimit of [1, 2 ** 31 - 1]) {
expect(
await collectNames(db.listTables(namespacePath, { pageLimit })),
).toEqual(["a", "b"]);
}
for (const pageLimit of [0, 2 ** 31, 2 ** 32 - 1]) {
expect(() => db.listTables(namespacePath, { pageLimit })).toThrow(
/limit/,
);
}
}
expect(await collectNames(db.listTables())).toEqual(["a", "b"]);
});
it("should list all names and use a default page size of 100", async () => {
@@ -396,20 +398,17 @@ describe("given a connection", () => {
}
await expect(db.tableNames()).resolves.toEqual(names);
await expect(db.tableNames({ limit: 0 })).resolves.toEqual([]);
const first = await db.listTables();
expect(first.tables).toEqual(names.slice(0, 100));
expect(first.pageToken).toBeDefined();
const second = await db.listTables({ pageToken: first.pageToken });
expect(second.tables).toEqual(names.slice(100));
expect(second.pageToken).toBeUndefined();
await expect(db.listTables({ limit: 200 })).resolves.toEqual({
tables: names,
pageToken: undefined,
});
await expect(db.listTables({ limit: 0 })).resolves.toEqual({
tables: [],
pageToken: undefined,
});
const first = db.listTables();
const firstPage: string[] = [];
for (let i = 0; i < 100; i++) firstPage.push((await first.next()).value!);
expect(firstPage).toEqual(names.slice(0, 100));
expect(first.pageToken()).toBeDefined();
expect(first.numPageResults()).toBe(0);
expect(await collectNames(first)).toEqual(names.slice(100));
expect(await collectNames(db.listTables([], { pageLimit: 200 }))).toEqual(
names,
);
expect(() => db.listTables([], { pageLimit: 0 })).toThrow(/limit/);
});
it("should list tables in a namespace", async () => {
@@ -420,12 +419,8 @@ describe("given a connection", () => {
await db.createNamespace(["child"]);
await db.createTable("nested", [{ id: 1 }], ["child"]);
await expect(db.listTables(["child"])).resolves.toEqual(
expect.objectContaining({ tables: ["nested"] }),
);
await expect(db.listTables()).resolves.toEqual(
expect.objectContaining({ tables: [] }),
);
expect(await collectNames(db.listTables(["child"]))).toEqual(["nested"]);
expect(await collectNames(db.listTables())).toEqual([]);
});
it("should create tables in v2 mode", async () => {
@@ -719,22 +714,35 @@ describe("namespaces", () => {
it("should list namespaces created at the root", async () => {
await db.createNamespace(["alpha"]);
await db.createNamespace(["beta"]);
const list = await db.listNamespaces();
expect(list.namespaces).toEqual(expect.arrayContaining(["alpha", "beta"]));
const list = await collectNames(db.listNamespaces());
expect(list).toEqual(expect.arrayContaining(["alpha", "beta"]));
});
it("should list child namespaces under a parent", async () => {
await db.createNamespace(["parent"]);
await db.createNamespace(["parent", "child"]);
const list = await db.listNamespaces(["parent"]);
expect(list.namespaces).toContain("child");
const list = await collectNames(db.listNamespaces(["parent"]));
expect(list).toContain("child");
});
it("should resume a namespace listing from its page token", async () => {
await db.createNamespace(["alpha"]);
await db.createNamespace(["beta"]);
const first = db.listNamespaces([], { pageLimit: 1 });
expect(first.numPageResults()).toBe(0);
expect((await first.next()).value).toBe("alpha");
const second = db.listNamespaces([], {
pageLimit: 1,
pageToken: first.pageToken(),
});
expect(await collectNames(second)).toEqual(["beta"]);
});
it("should drop a namespace", async () => {
await db.createNamespace(["ephemeral"]);
await db.dropNamespace(["ephemeral"]);
const list = await db.listNamespaces();
expect(list.namespaces).not.toContain("ephemeral");
const list = await collectNames(db.listNamespaces());
expect(list).not.toContain("ephemeral");
});
it("should raise an error on any namespace op after close", async () => {
@@ -742,7 +750,7 @@ describe("namespaces", () => {
await expect(db.describeNamespace(["foo"])).rejects.toThrow(
"Connection is closed",
);
await expect(db.listNamespaces()).rejects.toThrow("Connection is closed");
expect(() => db.listNamespaces()).toThrow("Connection is closed");
await expect(db.createNamespace(["foo"])).rejects.toThrow(
"Connection is closed",
);
nodejs/__test__/materialized_view.test.ts
+1
View File
@@ -129,6 +129,7 @@ describe("materialized views", () => {
const job = await db.dropMaterializedViewAsync("adults");
expect(job.id).toBeNull();
expect(job.jobType).toBeNull();
await job.wait();
expect(await collect(db.listMaterializedViews())).toEqual([]);
});
nodejs/__test__/nested_blob.test.ts
+327
View File
@@ -0,0 +1,327 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
import * as tmp from "tmp";
import { type Table, blob, connect, isBlobField } from "../lancedb";
import {
DataType,
Field,
Int64,
List,
Schema,
Struct,
Utf8,
} from "../lancedb/arrow";
describe('nested blobs with blobMode "bytes"', () => {
let tmpDir: tmp.DirResult;
let table: Table;
const caption = new Field(
"caption",
new Utf8(),
false,
new Map([["label", "caption"]]),
);
const info = new Field(
"info",
new Struct([
blob("image"),
caption,
new Field("preview", new Struct([blob("image")]), true),
]),
true,
new Map([["label", "info"]]),
);
const schema = new Schema(
[new Field("id", new Int64(), false), info, blob("cover")],
new Map([["source", "nested-blobs"]]),
);
beforeEach(async () => {
tmpDir = tmp.dirSync({ unsafeCleanup: true });
const db = await connect(tmpDir.name);
table = await db.createTable(
"nested_blobs",
[
{
id: 1n,
info: {
image: Buffer.from("hello"),
caption: "first",
preview: { image: Buffer.from("preview") },
},
cover: Buffer.from("cover"),
},
{ id: 2n, info: null, cover: null },
{
id: 3n,
info: { image: null, caption: "third", preview: null },
cover: null,
},
{
id: 4n,
info: {
image: Buffer.alloc(0),
caption: "fourth",
preview: { image: null },
},
cover: Buffer.alloc(0),
},
],
{ schema },
);
});
afterEach(() => tmpDir.removeCallback());
it.each([false, true])(
"reads struct leaves across batches (select parent: %s)",
async (selectParent) => {
const query = table
.query()
.orderBy({ columnName: "id", ascending: true });
if (selectParent) {
query.select(["info"]);
}
const result = await query.toArrow({
blobMode: "bytes",
maxBatchLength: 1,
});
expect(result.batches.length).toBeGreaterThan(1);
expect(result.schema.fields.map((field) => field.name)).toEqual(
selectParent ? ["info"] : ["id", "info", "cover"],
);
expect(result.schema.metadata).toEqual(schema.metadata);
const infoField = result.schema.fields.find(
(field) => field.name === "info",
)!;
expect(infoField.nullable).toBe(true);
expect(infoField.metadata).toEqual(info.metadata);
const fields = (infoField.type as Struct).children;
expect(DataType.isLargeBinary(fields[0].type)).toBe(true);
expect(isBlobField(fields[0])).toBe(false);
expect(fields[0].metadata.size).toBe(0);
expect(fields[1]).toEqual(caption);
expect(
DataType.isLargeBinary((fields[2].type as Struct).children[0].type),
).toBe(true);
const rows = result.toArray();
expect(Buffer.from(rows[0].info.image).toString()).toBe("hello");
expect(rows[0].info.caption).toBe("first");
expect(Buffer.from(rows[0].info.preview.image).toString()).toBe(
"preview",
);
expect(rows[1].info).toBeNull();
expect(rows[2].info.image).toBeNull();
expect(rows[2].info.caption).toBe("third");
expect(rows[2].info.preview).toBeNull();
expect(Buffer.from(rows[3].info.image)).toEqual(Buffer.alloc(0));
expect(rows[3].info.preview.image).toBeNull();
if (!selectParent) {
expect(Buffer.from(rows[0].cover).toString()).toBe("cover");
expect(rows[1].cover).toBeNull();
}
},
);
it("reads direct leaf and struct aliases", async () => {
const rows = await table
.query()
.orderBy({ columnName: "id", ascending: true })
.select({
image: "info.image",
details: "info",
preview: "info.preview",
caption: "info.caption",
})
.toArray({ blobMode: "bytes", maxBatchLength: 1 });
expect(Object.keys(rows[0])).toEqual([
"image",
"details",
"preview",
"caption",
]);
expect(Buffer.from(rows[0].image).toString()).toBe("hello");
expect(Buffer.from(rows[0].details.image).toString()).toBe("hello");
expect(Buffer.from(rows[0].details.preview.image).toString()).toBe(
"preview",
);
expect(Buffer.from(rows[0].preview.image).toString()).toBe("preview");
expect(rows[0].caption).toBe("first");
expect(rows[1].details).toBeNull();
expect(rows[1].image).toBeNull();
expect(rows[2].image).toBeNull();
expect(Buffer.from(rows[3].image)).toEqual(Buffer.alloc(0));
});
it.each([false, true])(
"preserves a literal scalar alias matching a nested blob path (reverse: %s)",
async (reverse) => {
const projection: [string, string][] = [
["info", "info"],
["info.image", "id"],
];
if (reverse) {
projection.reverse();
}
const [row] = await table
.query()
.where("id = 1")
.select(new Map(projection))
.toArray({ blobMode: "bytes" });
expect(row["info.image"]).toBe(1n);
expect(Buffer.from(row.info.image).toString()).toBe("hello");
},
);
it.each([false, true])(
"materializes every overlapping blob projection (reverse: %s)",
async (reverse) => {
const columns = ["info.image", "info"];
if (reverse) {
columns.reverse();
}
const result = await table
.query()
.orderBy({ columnName: "id", ascending: true })
.select(columns)
.toArrow({ blobMode: "bytes", maxBatchLength: 1 });
expect(result.schema.fields.map((field) => field.name)).toEqual(columns);
expect(DataType.isLargeBinary(result.getChild("info.image")!.type)).toBe(
true,
);
expect(
DataType.isLargeBinary(
result.getChild("info")!.getChild("image")!.type,
),
).toBe(true);
const rows = result.toArray();
expect(Buffer.from(rows[0]["info.image"]).toString()).toBe("hello");
expect(Buffer.from(rows[0].info.image).toString()).toBe("hello");
expect(rows[0].info.caption).toBe("first");
expect(rows[1]["info.image"]).toBeNull();
expect(rows[1].info).toBeNull();
expect(rows[2]["info.image"]).toBeNull();
expect(rows[2].info.image).toBeNull();
expect(Buffer.from(rows[3]["info.image"])).toEqual(Buffer.alloc(0));
expect(Buffer.from(rows[3].info.image)).toEqual(Buffer.alloc(0));
},
);
it("materializes dotted aliases independently of nested fields", async () => {
const [row] = await table
.query()
.where("id = 1")
.select({
details: "info",
"details.image": "info.image",
"details.preview": "info.preview",
"details.preview.image": "info.preview.image",
})
.toArray({ blobMode: "bytes" });
expect(Buffer.from(row.details.image).toString()).toBe("hello");
expect(Buffer.from(row["details.image"]).toString()).toBe("hello");
expect(Buffer.from(row.details.preview.image).toString()).toBe("preview");
expect(Buffer.from(row["details.preview"].image).toString()).toBe(
"preview",
);
expect(Buffer.from(row["details.preview.image"]).toString()).toBe(
"preview",
);
});
it.each([false, true])(
"converts the empty result schema (alias: %s)",
async (alias) => {
const query = table.query().where("id < 0");
if (alias) {
query.select({ image: "info.image", details: "info" });
}
const result = await query.toArrow({ blobMode: "bytes" });
expect(result.numRows).toBe(0);
expect(result.schema.fields.map((field) => field.name)).toEqual(
alias ? ["image", "details"] : ["id", "info", "cover"],
);
const fields = (
result.schema.fields.find(
(field) => field.name === (alias ? "details" : "info"),
)!.type as Struct
).children;
expect(DataType.isLargeBinary(fields[0].type)).toBe(true);
expect(isBlobField(fields[0])).toBe(false);
expect(fields[1]).toEqual(caption);
expect(
DataType.isLargeBinary((fields[2].type as Struct).children[0].type),
).toBe(true);
if (alias) {
expect(DataType.isLargeBinary(result.schema.fields[0].type)).toBe(true);
}
},
);
it("preserves requested row IDs and leaves subsequent descriptors intact", async () => {
const query = table.query().where("id = 1").select(["info"]);
const [bytes] = await query.toArray({ blobMode: "bytes" });
expect(Object.keys(bytes)).toEqual(["info"]);
const [description] = await query.toArray();
expect(Object.keys(description)).toEqual(["info"]);
expect(Number(description.info.image.size)).toBe(5);
const [withRowId] = await query.withRowId().toArray({ blobMode: "bytes" });
expect(typeof withRowId._rowid).toBe("bigint");
expect(await table.fetchBlobs("info.image", [withRowId._rowid])).toEqual([
Buffer.from(withRowId.info.image),
]);
});
it("keeps descriptors in description mode", async () => {
const [description] = await table
.query()
.where("id = 1")
.toArray({ blobMode: "descriptions" });
expect(Number(description.info.image.size)).toBe(5);
});
it("keeps list elements as descriptors alongside struct blob bytes", async () => {
const db = await connect(tmpDir.name);
const table = await db.createTable(
"list_blobs",
[
{
info: { image: Buffer.from("hello"), images: [Buffer.from("list")] },
},
],
{
schema: new Schema([
new Field(
"info",
new Struct([
blob("image"),
new Field("images", new List(blob("item")), true),
]),
true,
),
]),
},
);
const [bytes] = await table.query().toArray({ blobMode: "bytes" });
expect(Buffer.from(bytes.info.image).toString()).toBe("hello");
expect(Number(bytes.info.images.get(0).size)).toBe(4);
const [row] = await table
.query()
.select(["info.images"])
.toArray({ blobMode: "bytes" });
expect(Number(row["info.images"].get(0).size)).toBe(4);
expect(Object.keys(row)).toEqual(["info.images"]);
const result = await table.query().toArrow({ blobMode: "bytes" });
const fields = (result.schema.fields[0].type as Struct).children;
const descriptions = await table.query().toArrow();
expect(fields[1]).toEqual(
(descriptions.schema.fields[0].type as Struct).children[1],
);
expect(DataType.isStruct((fields[1].type as List).children[0].type)).toBe(
true,
);
});
});
nodejs/__test__/query.test.ts
+11 -5
View File
@@ -538,11 +538,17 @@ describe("IVF search effort", () => {
const db = await connect(directory.name);
const rows = Array.from({ length: 512 }, (_, id) => ({
id,
vector: Array.from(
{ length: 16 },
(_, j) =>
(Math.floor(id / 32) === j ? 1 : 0) + Math.sin(id * 17 + j) * 0.001,
),
// Keep all eight exact neighbors together even if random IVF training
// splits the first cluster. This tests effort, not training recall.
vector:
id < 8
? [...vector]
: Array.from(
{ length: 16 },
(_, j) =>
(Math.floor(id / 32) === j ? 1 : 0) +
Math.sin(id * 17 + j) * 0.001,
),
}));
table = await db.createTable("effort", rows.slice(0, 256));
await table.add(rows.slice(256));
nodejs/__test__/remote.test.ts
+49
View File
@@ -98,6 +98,55 @@ async function withMockDatabase(
}
describe("remote connection", () => {
it.each([
{ dataStorageVersion: "2.0" },
{ enableV2ManifestPaths: false },
{ storageOptions: { bogusOptionXyz: "1" } },
])("rejects local create options %j before sending data", async (options) => {
const requests: string[] = [];
await withMockDatabase(
(req, res) => {
requests.push(req.url ?? "");
res.writeHead(200).end();
},
async (db) => {
await expect(
db.createTable("test", [{ value: 1 }], {
...options,
mode: "overwrite",
}),
).rejects.toThrow(/write_options.*remote/);
await expect(
db.createEmptyTable(
"test",
new Schema([new Field("value", new Int64(), true)]),
options,
),
).rejects.toThrow(/write_options.*remote/);
},
);
expect(requests).toEqual([]);
});
it.each([{ indexCacheSize: 5 }, { storageOptions: { x: "y" } }])(
"rejects local open options %j before sending a request",
async (options) => {
const requests: string[] = [];
await withMockDatabase(
(req, res) => {
requests.push(req.url ?? "");
res.writeHead(200).end();
},
async (db) => {
await expect(
db.openTable("test", undefined, options),
).rejects.toThrow(/not supported for remote/);
},
);
expect(requests).toEqual([]);
},
);
it.each([false, true])(
"normalizes remote blob writes (range target: %s)",
async (wide) => {
nodejs/__test__/table.test.ts
+78 -47
View File
@@ -3314,51 +3314,64 @@ describe("when dealing with blob columns", () => {
).rejects.toThrow(/blobMode must be/);
});
it("reruns the query when the table changes before the version check", async () => {
const { table, native, update } = await openVersionedBlobTable();
const liveVersion = native.version.bind(native);
native.version = async () => {
native.version = liveVersion;
await update(2n, "new");
return liveVersion();
};
const [row] = await table.query().toArray({ blobMode: "bytes" });
expect(row.generation).toBe(2n);
expect(Buffer.from(row.image).toString()).toBe("new");
});
it.each([false, true])(
"reruns the query when the table changes before the version check (nested: %s)",
async (nested) => {
const { table, native, update } = await openVersionedBlobTable(nested);
const liveVersion = native.version.bind(native);
native.version = async () => {
native.version = liveVersion;
await update(2n, "new");
return liveVersion();
};
const [row] = await table.query().toArray({ blobMode: "bytes" });
expect(row.generation).toBe(2n);
expect(Buffer.from(nested ? row.info.image : row.image).toString()).toBe(
"new",
);
},
);
it("reads bytes from the query's version when a write lands before the fetch", async () => {
const { table, native, update } = await openVersionedBlobTable();
const liveVersion = native.version.bind(native);
native.version = async () => {
native.version = liveVersion;
const version = await liveVersion();
await update(2n, "new");
return version;
};
const [row] = await table.query().toArray({ blobMode: "bytes" });
expect(row.generation).toBe(1n);
expect(Buffer.from(row.image).toString()).toBe("old");
});
it.each([false, true])(
"reads bytes from the query's version when a write lands before the fetch (nested: %s)",
async (nested) => {
const { table, native, update } = await openVersionedBlobTable(nested);
const liveVersion = native.version.bind(native);
native.version = async () => {
native.version = liveVersion;
const version = await liveVersion();
await update(2n, "new");
return version;
};
const [row] = await table.query().toArray({ blobMode: "bytes" });
expect(row.generation).toBe(1n);
expect(Buffer.from(nested ? row.info.image : row.image).toString()).toBe(
"old",
);
},
);
it("fails when the table changes during every attempt", async () => {
const { table, native, update } = await openVersionedBlobTable();
const liveVersion = native.version.bind(native);
let generation = 1n;
native.version = async () => {
generation += 1n;
await update(generation, `generation ${generation}`);
return liveVersion();
};
try {
await expect(
table.query().toArray({ blobMode: "bytes" }),
).rejects.toThrow(/changed during each of 3 attempts/);
expect(generation).toBe(4n);
} finally {
native.version = liveVersion;
}
});
it.each([false, true])(
"fails when the table changes during every attempt (nested: %s)",
async (nested) => {
const { table, native, update } = await openVersionedBlobTable(nested);
const liveVersion = native.version.bind(native);
let generation = 1n;
native.version = async () => {
generation += 1n;
await update(generation, `generation ${generation}`);
return liveVersion();
};
try {
await expect(
table.query().toArray({ blobMode: "bytes" }),
).rejects.toThrow(/changed during each of 3 attempts/);
expect(generation).toBe(4n);
} finally {
native.version = liveVersion;
}
},
);
it("rejects fetchBlobs on a non-blob column", async () => {
const { table, rowIds } = await openBlobTable();
@@ -3642,18 +3655,28 @@ describe("when dealing with blob columns", () => {
).map((value) => (value == null ? null : Number(value.size)));
}
async function openVersionedBlobTable() {
async function openVersionedBlobTable(nested = false) {
const db = await connect(tmpDir.name, {
storageOptions: { newTableEnableStableRowIds: "true" },
});
const schema = new Schema([
new Field("id", new Int64(), false),
new Field("generation", new Int64(), true),
blob("image"),
nested
? new Field("info", new Struct([blob("image")]), true)
: blob("image"),
]);
const table = await db.createTable(
"versions",
[{ id: 1n, generation: 1n, image: Buffer.from("old") }],
[
{
id: 1n,
generation: 1n,
...(nested
? { info: { image: Buffer.from("old") } }
: { image: Buffer.from("old") }),
},
],
{ schema },
);
const native = (
@@ -3663,7 +3686,15 @@ describe("when dealing with blob columns", () => {
await table
.mergeInsert("id")
.whenMatchedUpdateAll()
.execute([{ id: 1n, generation, image: Buffer.from(image) }]);
.execute([
{
id: 1n,
generation,
...(nested
? { info: { image: Buffer.from(image) } }
: { image: Buffer.from(image) }),
},
]);
};
return { table, native, update };
}
nodejs/lancedb/connection.ts
+31 -102
View File
@@ -30,16 +30,12 @@ import type {
DescribeNamespaceResponse,
DropNamespaceResponse,
JobInfo,
ListNamespacesResponse,
ListTablesResponse,
} from "./native";
import { ViewDescription, viewDescriptionFromNative } from "./view";
export type {
CreateNamespaceResponse,
DescribeNamespaceResponse,
DropNamespaceResponse,
ListNamespacesResponse,
ListTablesResponse,
};
import { sanitizeTable } from "./sanitize";
import { LocalTable, Table } from "./table";
@@ -141,7 +137,7 @@ export interface OpenTableOptions {
}
/**
* @deprecated Use {@link ListTablesOptions} with {@link Connection.listTables}
* @deprecated Use {@link ListingOptions} with {@link Connection.listTables}
* instead.
*/
export interface TableNamesOptions {
@@ -157,32 +153,6 @@ export interface TableNamesOptions {
limit?: number;
}
export interface ListTablesOptions {
/**
* Token from a previous response, to resume listing where it left off.
*
* The token is opaque: it carries whatever the database needs to resume, and
* callers should not construct or interpret one.
*/
pageToken?: string;
/**
* An upper bound on how many tables to return.
* Defaults to 100. Zero returns an empty page without a continuation token.
*
* A page may hold fewer than this and still not be the last one, so keep
* going while the response carries a page token rather than while pages are
* full.
*/
limit?: number;
}
export interface ListNamespacesOptions {
/** Token from a previous response for pagination. */
pageToken?: string;
/** An optional limit to the number of results to return. */
limit?: number;
}
export interface CreateNamespaceOptions {
/** Creation mode. */
mode?: "create" | "exist_ok" | "overwrite";
@@ -278,45 +248,18 @@ export abstract class Connection {
options?: Partial<TableNamesOptions>,
): Promise<string[]>;
/**
* List a page of the tables in this database.
*
* To retrieve the tables after the page, pass the `pageToken` the response
* carries back in. A page can be shorter than `limit` without being the last
* one, so walk until a response carries no page token:
*
/** Lazily list tables in a namespace, requesting 100 per page by default.
* @example
* ```ts
* const names = [];
* let pageToken = undefined;
* do {
* const page = await conn.listTables({ pageToken, limit: 100 });
* names.push(...page.tables);
* pageToken = page.pageToken;
* } while (pageToken);
* for await (const name of conn.listTables([], { pageLimit: 20 })) {
* console.log(name);
* }
* ```
*
* @param {Partial<ListTablesOptions>} options - Pagination options
* (`pageToken`, `limit`).
* @returns {Promise<ListTablesResponse>} A page of table names and an
* optional token for the tables after it.
*/
abstract listTables(
options?: Partial<ListTablesOptions>,
): Promise<ListTablesResponse>;
/**
* List a page of the tables in this database.
*
* @param {string[]} namespacePath - The namespace path to list tables from
* (defaults to root namespace)
* @param {Partial<ListTablesOptions>} options - Pagination options
* (`pageToken`, `limit`).
* @returns {Promise<ListTablesResponse>} A page of table names and an
* optional token for the tables after it.
*/
abstract listTables(
namespacePath?: string[],
options?: Partial<ListTablesOptions>,
): Promise<ListTablesResponse>;
options?: ListingOptions,
): Listing<string>;
/**
* Open a table in the database.
@@ -542,23 +485,11 @@ export abstract class Connection {
namespacePath: string[],
): Promise<DescribeNamespaceResponse>;
/**
* List the immediate child namespaces under the given parent.
*
* Results may be paginated. To retrieve subsequent pages, pass the
* `pageToken` returned by a previous call.
*
* @param {string[]} namespacePath - The parent namespace path. Defaults
* to the root namespace if omitted.
* @param {Partial<ListNamespacesOptions>} options - Pagination options
* (`pageToken`, `limit`).
* @returns {Promise<ListNamespacesResponse>} Child namespace names and
* an optional token for fetching the next page.
*/
/** Lazily list immediate child namespaces, 100 per page by default. */
abstract listNamespaces(
namespacePath?: string[],
options?: Partial<ListNamespacesOptions>,
): Promise<ListNamespacesResponse>;
options?: ListingOptions,
): Listing<string>;
/**
* Create a new namespace at the given path.
@@ -821,22 +752,17 @@ export class LocalConnection extends Connection {
);
}
async listTables(
namespacePathOrOptions?: string[] | Partial<ListTablesOptions>,
options?: Partial<ListTablesOptions>,
): Promise<ListTablesResponse> {
// Detect if first argument is namespacePath array or options object
const namespacePath = Array.isArray(namespacePathOrOptions)
? namespacePathOrOptions
: undefined;
const listTablesOptions = Array.isArray(namespacePathOrOptions)
? options
: (namespacePathOrOptions ?? options);
return this.inner.listTables(
namespacePath ?? [],
listTablesOptions?.pageToken,
normalizeTableListingLimit(listTablesOptions?.limit),
listTables(
namespacePath?: string[],
options: ListingOptions = {},
): Listing<string> {
validateListingOptions(options);
return new Listing(
this.inner.listTables(
namespacePath ?? [],
options.pageToken,
options.pageLimit,
),
);
}
@@ -1039,12 +965,15 @@ export class LocalConnection extends Connection {
listNamespaces(
namespacePath?: string[],
options?: Partial<ListNamespacesOptions>,
): Promise<ListNamespacesResponse> {
return this.inner.listNamespaces(
namespacePath ?? [],
options?.pageToken,
options?.limit,
options: ListingOptions = {},
): Listing<string> {
validateListingOptions(options);
return new Listing(
this.inner.listNamespaces(
namespacePath ?? [],
options.pageToken,
options.pageLimit,
),
);
}
nodejs/lancedb/index.ts
-4
View File
@@ -97,13 +97,9 @@ export {
Connection,
CreateTableOptions,
TableNamesOptions,
ListTablesOptions,
OpenTableOptions,
ListNamespacesOptions,
CreateNamespaceOptions,
DropNamespaceOptions,
ListNamespacesResponse,
ListTablesResponse,
CreateNamespaceResponse,
DropNamespaceResponse,
DescribeNamespaceResponse,
nodejs/lancedb/indices.ts
+2 -1
View File
@@ -131,7 +131,8 @@ export interface IvfRqOptions {
*
* This value controls how much each residual component is compressed. The more
* bits, the more accurate the index will be but the slower search. Typical values
* are small integers; the default is 1 bit per dimension.
* are small integers. If unset, uses Lance's default (currently 5 bits per
* dimension). Set to 1 for maximum compression.
*/
numBits?: number;
nodejs/lancedb/job.ts
+8 -5
View File
@@ -24,7 +24,8 @@ export interface JobEventsOptions {
* - {@link Job.refresh} and {@link Job.status} fetch the whole record.
* - {@link Job.wait} records the terminal state it establishes, but not the
* rest of the record.
* - Everything is null until one of those runs.
* - Registry details are null until one of those runs. Query handles report
* jobType "query" immediately.
*
* @hideconstructor
*/
@@ -38,8 +39,10 @@ export class Job {
/**
* Identifies the operation on the server that is running it.
*
* Operations that run in this process have no server id. The value is
* opaque: parsing it or storing it to resume the job later is not supported.
* Registry handles expose a registry job ID; query handles expose a
* connection-scoped SQL query UUID string. Local handles have no ID.
* Query handles report jobType "query". Their IDs are not registry IDs or
* durable resume tokens.
*/
get id(): string | null {
return this.inner.id ?? null;
@@ -51,8 +54,8 @@ export class Job {
}
/**
* The job's type, as the server names it. Null for an in-process job, which
* has no server-side record.
* "query" immediately for SQL-backed handles; otherwise the cached
* server-reported job type. Null for in-process handles.
*/
get jobType(): string | null {
return this.inner.jobType ?? null;
nodejs/lancedb/query.ts
+116 -143
View File
@@ -1,8 +1,8 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The LanceDB Authors
import type { Data as ArrowData } from "apache-arrow";
import {
type ArrowData,
Table as ArrowTable,
DataType,
Field,
@@ -14,7 +14,6 @@ import {
extractVectorBuffer,
fromBufferToRecordBatch,
fromRecordBatchToBuffer,
makeData,
tableFromIPC,
vectorFromArray,
} from "./arrow";
@@ -68,84 +67,42 @@ export interface QueryBlobContext {
const BLOB_BYTES_ATTEMPTS = 3;
interface BlobSource {
// Output names are literal: an alias containing dots is still one field.
output: string;
path: string[];
source: string;
}
function blobSources(
blobColumns: string[],
projection: [string, string][] | undefined,
): Map<string, string> {
const holds = (source: string) =>
blobColumns.some(
(column) => column === source || column.startsWith(`${source}.`),
);
const pairs =
projection ??
Array.from(new Set(blobColumns.map((column) => column.split(".")[0]))).map(
(name): [string, string] => [name, name],
);
return new Map(pairs.filter(([, source]) => holds(source)));
}
// Fetched bytes for the blobs under one output column. A blob column is a
// leaf; a struct holds the blobs nested in it, keyed by field name. A Map
// keeps struct field names from reaching shared JavaScript prototypes.
type BlobBytes = (Buffer | null)[] | Map<string, BlobBytes>;
function isLeaf(bytes: BlobBytes): bytes is (Buffer | null)[] {
return Array.isArray(bytes);
}
function nestedFieldType(type: Struct, bytes: Map<string, BlobBytes>): Struct {
return new Struct(
type.children.map((field) => {
const inner = bytes.get(field.name);
if (inner === undefined) {
return field;
): BlobSource[] {
if (projection === undefined) {
return blobColumns.map((source) => {
const [output, ...path] = source.split(".");
return { output, path, source };
});
}
const sources: BlobSource[] = [];
for (const [output, source] of projection) {
for (const column of blobColumns) {
if (column === source || column.startsWith(`${source}.`)) {
sources.push({
output,
path:
column === source ? [] : column.slice(source.length + 1).split("."),
source: column,
});
}
if (isLeaf(inner)) {
return new Field(field.name, new LargeBinary(), true);
}
return new Field(
field.name,
nestedFieldType(field.type as Struct, inner),
field.nullable,
);
}),
);
}
// Rebuilds a struct with the blobs under it as bytes. The struct keeps its own
// validity, so a null struct stays null.
function nestedData(
data: ArrowData,
bytes: Map<string, BlobBytes>,
offset: number,
): ArrowData {
const type = data.type as Struct;
const children = data.children.map((child, index) => {
const inner = bytes.get(type.children[index].name);
if (inner === undefined) {
return child;
}
if (isLeaf(inner)) {
return vectorFromArray(
inner.slice(offset, offset + data.length),
new LargeBinary(),
).data[0];
}
return nestedData(child, inner, offset);
});
return makeData({
type: nestedFieldType(type, bytes),
length: data.length,
nullCount: data.nullCount,
nullBitmap: data.nullBitmap,
children,
});
}
return sources;
}
async function readBlobBytes(
table: ArrowTable,
sources: Map<string, string>,
blobColumns: string[],
sources: BlobSource[],
snapshot: NativeTable,
dropRowId: boolean,
): Promise<ArrowTable> {
@@ -157,42 +114,27 @@ async function readBlobBytes(
rowIdColumn === null
? []
: Array.from(rowIdColumn.toArray() as BigUint64Array);
const fetch = async (column: string) =>
(rowIds.length === 0 ? [] : await snapshot.fetchBlobs(column, rowIds)).map(
(value) => value ?? null,
const outputFields = new Map(
table.schema.fields.map((field) => [field.name, field]),
);
const replacements = new Map<Field, (Buffer | null)[]>();
for (const { output, path, source } of sources) {
let field = outputFields.get(output);
for (const name of path) {
field =
field !== undefined && DataType.isStruct(field.type)
? field.type.children.find((child) => child.name === name)
: undefined;
}
if (field === undefined) {
continue;
}
const values =
rowIds.length === 0 ? [] : await snapshot.fetchBlobs(source, rowIds);
replacements.set(
field,
values.map((value) => value ?? null),
);
const replacements = new Map<number, BlobBytes>();
for (const [output, source] of sources) {
const index = table.schema.fields.findIndex(
(field) => field.name === output,
);
if (index < 0) {
continue;
}
if (blobColumns.includes(source)) {
replacements.set(index, await fetch(source));
continue;
}
if (!DataType.isStruct(table.schema.fields[index].type)) {
continue;
}
const nested = new Map<string, BlobBytes>();
for (const column of blobColumns.filter((c) =>
c.startsWith(`${source}.`),
)) {
const path = column.slice(source.length + 1).split(".");
let level = nested;
for (const name of path.slice(0, -1)) {
let next = level.get(name);
if (next === undefined) {
next = new Map<string, BlobBytes>();
level.set(name, next);
}
level = next as Map<string, BlobBytes>;
}
level.set(path[path.length - 1], await fetch(column));
}
replacements.set(index, nested);
}
const result = withBinaryColumns(table, replacements);
if (!dropRowId) {
@@ -205,48 +147,76 @@ async function readBlobBytes(
);
}
// A fresh field, not a clone: the descriptor field's metadata must not follow
// the bytes.
function withBinaryColumns(
table: ArrowTable,
replacements: Map<number, BlobBytes>,
replacements: Map<Field, (Buffer | null)[]>,
): ArrowTable {
if (replacements.size === 0) {
return table;
}
const fields = table.schema.fields.map((field, index) => {
const bytes = replacements.get(index);
if (bytes === undefined) {
return field;
const replaceField = (field: Field): Field => {
if (replacements.has(field)) {
// The descriptor's extension and encoding metadata must not follow bytes.
return new Field(field.name, new LargeBinary(), true);
}
return isLeaf(bytes)
? new Field(field.name, new LargeBinary(), true)
: new Field(
field.name,
nestedFieldType(field.type as Struct, bytes),
field.nullable,
);
});
if (DataType.isStruct(field.type)) {
const children = field.type.children.map(replaceField);
if (children.some((child, i) => child !== field.type.children[i])) {
return field.clone({ type: new Struct(children) });
}
}
return field;
};
const fields = table.schema.fields.map(replaceField);
const schema = new Schema(fields, table.schema.metadata);
let offset = 0;
const batches = table.batches.map((batch) => {
const children = batch.data.children.map((child, index) => {
const bytes = replacements.get(index);
if (bytes === undefined) {
return child;
const replaceData = (
data: ArrowData,
field: Field,
updated: Field,
): ArrowData => {
if (field === updated) {
return data;
}
if (!isLeaf(bytes)) {
return nestedData(child, bytes, offset);
const values = replacements.get(field);
if (values !== undefined) {
return vectorFromArray(
values.slice(offset, offset + batch.numRows),
new LargeBinary(),
).data[0];
}
return vectorFromArray(
bytes.slice(offset, offset + batch.numRows),
new LargeBinary(),
).data[0];
});
const children = data.children.map((child, index) =>
replaceData(
child,
field.type.children[index],
updated.type.children[index],
),
);
// Keep the struct's validity bitmap and offset, including null parents.
return data.clone(
updated.type,
undefined,
undefined,
undefined,
undefined,
children,
);
};
const children = batch.data.children.map((child, index) =>
replaceData(child, table.schema.fields[index], fields[index]),
);
offset += batch.numRows;
return new RecordBatch(
schema,
makeData({ type: new Struct(fields), length: batch.numRows, children }),
batch.data.clone(
new Struct(fields),
undefined,
undefined,
undefined,
undefined,
children,
),
);
});
return new ArrowTable(schema, batches);
@@ -274,13 +244,14 @@ export interface QueryExecutionOptions {
*
* - `"descriptions"` (default): each blob is a descriptor. Read the bytes
* with {@link Table.fetchBlobs} or {@link Table.fetchBlobFiles}.
* - `"bytes"`: each blob column holds the blob bytes as `LargeBinary`,
* including blobs nested in structs. The query also reads `_rowid` to
* fetch the bytes, and leaves it out of the result unless
* {@link QueryBase.withRowId} was called. All the bytes are held in
* memory.
* - `"bytes"`: each blob column, including blobs nested in structs, holds
* the bytes as `LargeBinary`. The query also reads `_rowid` to fetch them, and
* leaves it out of the result unless {@link QueryBase.withRowId} was
* called. All the bytes are held in memory.
*
* Blobs nested in a list keep their descriptors.
* Selecting a parent struct or directly aliasing a blob or struct also returns
* bytes. Blobs inside lists keep their descriptors. Arbitrary projection
* expressions are not materialized as bytes.
*
* The rows and the bytes come from the same table version. If the table
* changes while the query runs, the query runs again, up to three times.
@@ -529,15 +500,17 @@ export class QueryBase<
// unchanged version afterwards proves it read the snapshot's version.
const snapshot = await context.table.querySnapshot();
const version = await snapshot.version();
const blobColumns = await snapshot.blobColumns();
const sources = blobSources(blobColumns, context.projection);
if (sources.size === 0) {
const sources = blobSources(
await snapshot.blobColumns(),
context.projection,
);
if (sources.length === 0) {
return this.collect(options, false);
}
const addRowId = !context.withRowId;
const table = await this.collect(options, addRowId);
if ((await context.table.version()) === version) {
return readBlobBytes(table, sources, blobColumns, snapshot, addRowId);
return readBlobBytes(table, sources, snapshot, addRowId);
}
if (attempt === BLOB_BYTES_ATTEMPTS) {
throw new Error(
nodejs/npm/darwin-arm64/package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-darwin-arm64",
"version": "0.41.0-beta.13",
"version": "0.41.0-beta.14",
"os": ["darwin"],
"cpu": ["arm64"],
"main": "lancedb.darwin-arm64.node",
nodejs/npm/linux-arm64-gnu/package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-arm64-gnu",
"version": "0.41.0-beta.13",
"version": "0.41.0-beta.14",
"os": ["linux"],
"cpu": ["arm64"],
"main": "lancedb.linux-arm64-gnu.node",
nodejs/npm/linux-arm64-musl/package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-arm64-musl",
"version": "0.41.0-beta.13",
"version": "0.41.0-beta.14",
"os": ["linux"],
"cpu": ["arm64"],
"main": "lancedb.linux-arm64-musl.node",
nodejs/npm/linux-x64-gnu/package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-x64-gnu",
"version": "0.41.0-beta.13",
"version": "0.41.0-beta.14",
"os": ["linux"],
"cpu": ["x64"],
"main": "lancedb.linux-x64-gnu.node",
nodejs/npm/linux-x64-musl/package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-linux-x64-musl",
"version": "0.41.0-beta.13",
"version": "0.41.0-beta.14",
"os": ["linux"],
"cpu": ["x64"],
"main": "lancedb.linux-x64-musl.node",
nodejs/npm/win32-arm64-msvc/package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-win32-arm64-msvc",
"version": "0.41.0-beta.13",
"version": "0.41.0-beta.14",
"os": [
"win32"
],
nodejs/npm/win32-x64-msvc/package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lancedb/lancedb-win32-x64-msvc",
"version": "0.41.0-beta.13",
"version": "0.41.0-beta.14",
"os": ["win32"],
"cpu": ["x64"],
"main": "lancedb.win32-x64-msvc.node",
nodejs/package.json
+1 -1
View File
@@ -11,7 +11,7 @@
"ann"
],
"private": false,
"version": "0.41.0-beta.13",
"version": "0.41.0-beta.14",
"main": "dist/index.js",
"exports": {
".": "./dist/index.js",
nodejs/src/connection.rs
+19 -82
View File
@@ -16,8 +16,7 @@ use crate::table::Table;
use lancedb::connection::{ConnectBuilder, Connection as LanceDBConnection, connect_namespace};
use lance_namespace::models::{
CreateNamespaceRequest, DescribeNamespaceRequest, DropNamespaceRequest, ListNamespacesRequest,
ListTablesRequest,
CreateNamespaceRequest, DescribeNamespaceRequest, DropNamespaceRequest,
};
use lancedb::ipc::{ipc_file_to_batches, ipc_file_to_schema};
@@ -31,18 +30,6 @@ pub struct DescribeNamespaceResponse {
pub properties: Option<HashMap<String, String>>,
}
#[napi(object)]
pub struct ListNamespacesResponse {
pub namespaces: Vec<String>,
pub page_token: Option<String>,
}
#[napi(object)]
pub struct ListTablesResponse {
pub tables: Vec<String>,
pub page_token: Option<String>,
}
#[napi(object)]
pub struct CreateNamespaceResponse {
pub properties: Option<HashMap<String, String>>,
@@ -239,31 +226,20 @@ impl Connection {
op.execute().await.default_error()
}
/// List a page of tables in the database.
/// Lazily list tables in the database.
#[napi(catch_unwind)]
pub async fn list_tables(
pub fn list_tables(
&self,
namespace_path: Option<Vec<String>>,
page_token: Option<String>,
limit: Option<u32>,
) -> napi::Result<ListTablesResponse> {
let request = ListTablesRequest {
// The root namespace is an empty path, not an absent one: a namespace-backed
// database rejects a request that names no namespace.
id: Some(namespace_path.unwrap_or_default()),
page_token,
limit: limit.map(|limit| i32::try_from(limit).unwrap_or(i32::MAX)),
..Default::default()
};
let response = self
.get_inner()?
.list_tables(request)
.await
.default_error()?;
Ok(ListTablesResponse {
tables: response.tables,
page_token: response.page_token,
})
) -> napi::Result<crate::listing::NameListing> {
Ok(crate::listing::NameListing::new(
self.get_inner()?.list_tables(
&namespace_path.unwrap_or_default(),
crate::listing::options(page_token, limit),
),
))
}
/// Create table from a Apache Arrow IPC (file) buffer.
@@ -663,28 +639,19 @@ impl Connection {
}
#[napi(catch_unwind)]
/// List child namespaces under the given namespace path
pub async fn list_namespaces(
/// Lazily list child namespaces under the given namespace path
pub fn list_namespaces(
&self,
namespace_path: Option<Vec<String>>,
page_token: Option<String>,
limit: Option<u32>,
) -> napi::Result<ListNamespacesResponse> {
let req = ListNamespacesRequest {
id: namespace_path,
page_token,
limit: checked_limit(limit)?,
..Default::default()
};
let resp = self
.get_inner()?
.list_namespaces(req)
.await
.default_error()?;
Ok(ListNamespacesResponse {
namespaces: resp.namespaces,
page_token: resp.page_token,
})
) -> napi::Result<crate::listing::NameListing> {
Ok(crate::listing::NameListing::new(
self.get_inner()?.list_namespaces(
&namespace_path.unwrap_or_default(),
crate::listing::options(page_token, limit),
),
))
}
#[napi(catch_unwind)]
@@ -787,33 +754,3 @@ impl Connection {
.default_error()
}
}
fn checked_limit(limit: Option<u32>) -> napi::Result<Option<i32>> {
limit
.map(i32::try_from)
.transpose()
.map_err(|_| napi::Error::from_reason("limit must not exceed 2147483647"))
}
#[cfg(test)]
mod tests {
use super::checked_limit;
#[test]
fn checked_limit_rejects_values_above_i32_max() {
// u32 values above i32::MAX wrapped negative through `as i32`,
// sending a negative page size to the server.
assert!(checked_limit(Some(3_000_000_000)).is_err());
assert!(checked_limit(Some(u32::MAX)).is_err());
}
#[test]
fn checked_limit_passes_boundary_and_missing_values() {
assert_eq!(
checked_limit(Some(i32::MAX as u32)).unwrap(),
Some(i32::MAX)
);
assert_eq!(checked_limit(Some(0)).unwrap(), Some(0));
assert_eq!(checked_limit(None).unwrap(), None);
}
}
nodejs/src/job.rs
+6 -6
View File
@@ -29,8 +29,8 @@ impl Job {
#[napi]
impl Job {
/// Identifies the operation on the server that is running it. Operations
/// that run in this process have no server id. The value is opaque.
/// The operation's opaque ID. Interpret registry and query IDs using job_type().
/// Local handles have no ID.
#[napi(getter)]
pub fn id(&self) -> Option<String> {
self.inner.id().map(str::to_string)
@@ -62,8 +62,8 @@ impl Job {
/// Ask the backend for this job's current state, and for a server-side job
/// its full record, then cache it for the getters below.
///
/// They are all null until this runs, because submitting an operation
/// returns only a job id. {@link Job.status} fetches the whole record too;
/// Registry details are null until this runs; query handles already expose
/// job_type "query". {@link Job.status} fetches the whole registry record too;
/// {@link Job.wait} records only the terminal state it establishes.
#[napi(catch_unwind)]
pub async fn refresh(&self) -> napi::Result<()> {
@@ -76,8 +76,8 @@ impl Job {
self.inner.state()
}
/// The job's type, as the server names it. Null for an in-process job,
/// which has no server-side record.
/// "query" immediately for SQL-backed handles; otherwise the cached
/// server-reported job type. Null for in-process handles.
#[napi(getter)]
pub fn job_type(&self) -> Option<String> {
self.inner.job_type()
python/Cargo.toml
+1 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "lancedb-python"
version = "0.41.0-beta.13"
version = "0.41.0-beta.14"
publish = false
edition.workspace = true
description = "Python bindings for LanceDB"
@@ -22,7 +22,6 @@ bytes.workspace = true
lancedb.workspace = true
datafusion-common.workspace = true
lance-core.workspace = true
# The workspace path dependency includes the patched crate in Maturin sources.
lance-arrow.workspace = true
lance-namespace.workspace = true
lance-namespace-impls.workspace = true
python/README.md
+76
View File
@@ -40,6 +40,33 @@ Stable releases are created about every 2 weeks. For the latest features and bug
pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ lancedb
```
If you also need the optional `lance` Python API, install the `pylance` extra
and include the Lance preview index:
```bash
pip install --pre \
--extra-index-url https://pypi.fury.io/lancedb/ \
--extra-index-url https://pypi.fury.io/lance-format/ \
"lancedb[pylance]"
```
With uv, also select the best matching version across the indexes:
```bash
uv pip install --prerelease allow \
--index https://pypi.fury.io/lancedb/ \
--index https://pypi.fury.io/lance-format/ \
--index-strategy unsafe-best-match \
"lancedb[pylance]"
```
The LanceDB preview index contains old `pylance` releases; current Lance previews
are published to the `lance-format` index. By default, uv only considers versions
from the first index containing a package, which can select an old `pylance` or
backtrack to an older LanceDB release. `unsafe-best-match` searches all configured
indexes, like pip; use it only with indexes you trust. Install the extra rather
than an unconstrained `pylance` so LanceDB's minimum supported version is enforced.
### Threading in CPU-limited containers
LanceDB uses separate pools for compute work and storage I/O. On a container with
@@ -74,3 +101,52 @@ print(results)
### Development
See [CONTRIBUTING.md](./CONTRIBUTING.md) for information on how to contribute to LanceDB.
### Opening SQL blob descriptors
A SQL `blob_descriptor(video)` result keeps the Blob V2 storage metadata and a
`source` struct containing its table URI, version, commit timestamp, branch,
column and row identity. This example uses a remote SQL connection whose SQL
service produces those source-bearing descriptors. Open the reference on the
source table to read incrementally:
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
`open_blob_file` accepts the dictionary or an Arrow struct scalar directly. It
keeps the query's version and branch without changing the table handle's checkout.
Payload bytes load on reads. An external blob with an unspecified size may need a
metadata-only HEAD request to open.
`videos.open_blob_files(rows["video"])` returns an iterator over the same handles,
preserving order, duplicates and nulls (`None`). It accepts Arrow arrays/chunked
arrays or another iterable and opens each handle as iteration advances. Close each
non-null handle after use. Async tables offer `await videos.open_blob_file(...)`
and `async for video in videos.open_blob_files(...)`.
```python
for video in videos.open_blob_files(rows["video"]):
if video is not None:
with video:
header = video.read(4096)
```
To extract a frame while retaining each row's timestamp, iterate over the
handles and rows together:
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
python/pyproject.toml
+3 -2
View File
@@ -45,7 +45,8 @@ repository = "https://github.com/lancedb/lancedb"
[project.optional-dependencies]
pylance = [
"pylance>=5.0.0b5",
# Keep the minimum in the bundled Lance major and compatible with tests below.
"pylance>=15.0.0b1",
]
# A library only needs the OpenTelemetry API; the application supplies and
# configures the SDK (the actual exporter/reader). See
@@ -63,7 +64,7 @@ tests = [
"polars>=0.19, <=1.32.3",
"pyarrow<25",
"pyarrow-stubs>=16.0",
"pylance==14.0.0b12",
"pylance==15.0.0b1",
"requests>=2.31.0",
"datafusion>=54,<55",
"opentelemetry-sdk>=1.30.0",
python/python/lancedb/__init__.py
+16
View File
@@ -24,6 +24,7 @@ from .remote import ClientConfig
from .remote.db import RemoteDBConnection
from .expr import Expr, col, lit, func
from .schema import blob, vector
from ._blob_reader import BlobReader as BlobReader
from .job import AsyncJob, Job
from .sql import AsyncQuery as AsyncSqlQuery
from .sql import Query as SqlQuery
@@ -280,6 +281,21 @@ def connect(
"uri is required when not connecting through a namespace client"
)
if isinstance(uri, str) and uri.startswith("db://"):
# Preserve the deprecated remote timeout options, but validate all other
# unknown keywords just as the local connection does.
unknown_kwargs = {
key: value
for key, value in kwargs.items()
if key not in ("connection_timeout", "read_timeout")
}
if unknown_kwargs:
raise ValueError(f"Unknown keyword arguments: {unknown_kwargs}")
if session is not None:
raise NotImplementedError("session is not supported for remote connections")
if manifest_enabled:
raise NotImplementedError(
"manifest_enabled is not supported for remote connections"
)
if api_key is None:
api_key = os.environ.get("LANCEDB_API_KEY")
if api_key is None:
python/python/lancedb/_blob.py
+13 -1
View File
@@ -7,7 +7,7 @@ from __future__ import annotations
import io
import operator
from collections.abc import Awaitable, Callable, Iterable
from collections.abc import Awaitable, Callable, Iterable, Mapping
from typing import TYPE_CHECKING, Optional, Union
import pyarrow as pa
@@ -31,6 +31,18 @@ FetchBlobsSync = Callable[[str, list[int]], pa.Array | pa.ChunkedArray]
FetchBlobsAsync = Callable[[str, list[int]], Awaitable[pa.Array | pa.ChunkedArray]]
def _normalize_blob_descriptor(value):
if isinstance(value, pa.Scalar):
value = value.as_py()
if value is None:
return None
if not isinstance(value, Mapping):
raise TypeError(
"blob descriptor must be a mapping, an Arrow struct scalar, or None"
)
return dict(value)
class BlobFile(io.RawIOBase):
"""Seekable lazy handle from :meth:`~lancedb.table.Table.fetch_blob_files`.
python/python/lancedb/_blob_reader.py
+120
View File
@@ -0,0 +1,120 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
"""Seekable binary input for Functions that consume Blob V2 values lazily."""
import io
import operator
import urllib.request
from collections.abc import Callable
from http import HTTPStatus
_READ_CHUNK_BYTES = 1024 * 1024
_READ_TIMEOUT_SECONDS = 300
class BlobReader(io.RawIOBase):
"""Read a blob incrementally inside a Function.
Annotating a Function input with ``BlobReader`` requests a lazy, seekable
reader. Annotating it with ``bytes`` requests materialized bytes instead.
``read_range`` does not change the sequential cursor. Readers are scoped
to the invocation and must not be retained for a later call.
For local calls, construct a reader from a range-reading callable and the
blob's size. The callable accepts an offset and length and returns bytes.
"""
def __init__(self, read_range: Callable[[int, int], bytes], size: int):
super().__init__()
self._read_range = read_range
self._size = operator.index(size)
if self._size < 0:
raise ValueError("blob size must be nonnegative")
self._position = 0
@classmethod
def _from_uri(cls, uri: str, size: int):
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
def read_range(offset, length):
request = urllib.request.Request(
uri, headers={"Range": f"bytes={offset}-{offset + length - 1}"}
)
with opener.open(request, timeout=_READ_TIMEOUT_SECONDS) as response:
if response.status != HTTPStatus.PARTIAL_CONTENT:
raise OSError(
"blob range service did not return a partial response"
)
data = response.read(length + 1)
if len(data) != length:
raise OSError(
"blob range service returned an invalid payload length"
)
return data
return cls(read_range, size)
def readable(self) -> bool:
return not self.closed
def seekable(self) -> bool:
return not self.closed
def size(self) -> int:
"""Return the blob size without reading its payload."""
self._checkClosed()
return self._size
def tell(self) -> int:
self._checkClosed()
return self._position
def seek(self, offset: int, whence: int = io.SEEK_SET) -> int:
self._checkClosed()
offset = operator.index(offset)
if whence == io.SEEK_SET:
position = offset
elif whence == io.SEEK_CUR:
position = self._position + offset
elif whence == io.SEEK_END:
position = self._size + offset
else:
raise ValueError(f"invalid whence: {whence}")
if position < 0:
raise ValueError("negative seek position")
self._position = position
return position
def read_range(self, offset: int, length: int) -> bytes:
"""Read a byte range without changing the cursor; truncate at EOF."""
self._checkClosed()
offset, length = operator.index(offset), operator.index(length)
if offset < 0 or length < 0:
raise ValueError("offset and length must be nonnegative")
remaining = min(length, max(0, self._size - offset))
chunks = []
while remaining:
count = min(remaining, _READ_CHUNK_BYTES)
chunk = self._read_range(offset, count)
if len(chunk) != count:
raise OSError("blob range reader returned an invalid payload length")
chunks.append(chunk)
offset += count
remaining -= count
return b"".join(chunks)
def read(self, size: int = -1) -> bytes:
self._checkClosed()
size = -1 if size is None else operator.index(size)
if size < 0:
size = max(0, self._size - self._position)
data = self.read_range(self._position, size)
self._position += len(data)
return data
def readinto(self, buffer) -> int:
view = memoryview(buffer).cast("B")
data = self.read(len(view))
view[: len(data)] = data
return len(data)
python/python/lancedb/_lancedb.pyi
+8 -6
View File
@@ -19,11 +19,9 @@ from .index import (
FTS,
)
from lance_namespace import (
ListNamespacesResponse,
CreateNamespaceResponse,
DropNamespaceResponse,
DescribeNamespaceResponse,
ListTablesResponse,
)
from .remote import ClientConfig
from .types import BaseTokenizerType
@@ -114,12 +112,12 @@ class Connection(object):
uri: str
async def is_open(self): ...
async def close(self): ...
async def list_namespaces(
def list_namespaces(
self,
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListNamespacesResponse: ...
) -> NameListing: ...
async def create_namespace(
self,
namespace_path: List[str],
@@ -136,12 +134,12 @@ class Connection(object):
self,
namespace_path: List[str],
) -> DescribeNamespaceResponse: ...
async def list_tables(
def list_tables(
self,
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListTablesResponse: ...
) -> NameListing: ...
async def table_names(
self,
namespace_path: Optional[List[str]],
@@ -322,6 +320,8 @@ class Job:
@property
def id(self) -> Optional[str]: ...
@property
def job_type(self) -> Optional[str]: ...
@property
def _state(self) -> Optional[str]: ...
@property
def _description(self) -> Optional[JobDescription]: ...
@@ -580,6 +580,7 @@ class Table:
async def fetch_blob_files(
self, column: str, row_ids: list[int]
) -> list[Optional[BlobFile]]: ...
async def open_blob_file(self, descriptor: str) -> BlobFile: ...
def vector_search(self) -> VectorQuery: ...
class Tags:
@@ -1010,6 +1011,7 @@ async def connect_catalog(
class NameListing:
def num_page_results(self) -> int: ...
def page_token(self) -> Optional[str]: ...
async def next_page(self) -> Optional[dict[str, Any]]: ...
def __aiter__(self) -> NameListing: ...
async def __anext__(self) -> str: ...
python/python/lancedb/db.py
+116 -70
View File
@@ -32,18 +32,24 @@ if sys.version_info >= (3, 12):
else:
from overrides import EnforceOverrides, override # type: ignore
from lancedb.listing import AsyncListing, Listing
from lancedb.listing import (
AsyncListing,
AsyncNamespaceListing,
AsyncTableListing,
Listing,
NamespaceListing,
TableListing,
validate_page_limit,
)
from lancedb.embeddings.registry import EmbeddingFunctionRegistry
from lancedb.common import data_to_reader, sanitize_uri, validate_schema
from lancedb.background_loop import LOOP
from lance_namespace import (
LanceNamespace,
ListNamespacesResponse,
CreateNamespaceResponse,
DropNamespaceResponse,
DescribeNamespaceResponse,
ListTablesResponse,
connect as namespace_connect,
)
from lance_namespace.errors import NamespaceNotEmptyError, TableNotFoundError
@@ -171,7 +177,7 @@ class DBConnection(EnforceOverrides):
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListNamespacesResponse:
) -> NamespaceListing:
"""List immediate child namespace names in the given namespace.
Parameters
@@ -187,12 +193,12 @@ class DBConnection(EnforceOverrides):
Returns
-------
ListNamespacesResponse
NamespaceListing
Response containing namespace names and optional page_token for pagination.
"""
if namespace_path is None:
namespace_path = []
return ListNamespacesResponse(namespaces=[], page_token=None)
return NamespaceListing(AsyncNamespaceListing(None, empty=True))
def create_namespace(
self,
@@ -317,8 +323,8 @@ class DBConnection(EnforceOverrides):
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListTablesResponse:
"""List a page of tables in this database.
) -> TableListing:
"""List tables in this database, one page at a time.
Parameters
----------
@@ -334,7 +340,7 @@ class DBConnection(EnforceOverrides):
Returns
-------
ListTablesResponse
TableListing
Response containing table names and optional page_token for pagination.
"""
raise NotImplementedError(
@@ -422,10 +428,13 @@ class DBConnection(EnforceOverrides):
it will not add the provided data but will validate against any
schema that's specified.
on_bad_vectors: str, default "error"
What to do if any of the vectors are not the same size or contains NaNs.
One of "error", "drop", "fill".
How to handle vectors with incorrect sizes, NaNs, or null elements.
One of "error", "drop", "fill", "null".
fill_value: float
The value to use when filling vectors. Only used if on_bad_vectors="fill".
NaN or null elements and missing trailing elements are replaced with this
value. Valid elements are preserved, and vectors longer than the expected
dimension are truncated.
storage_options: dict, optional
Additional options for the storage backend. Options already set on the
connection will be inherited by the table, but can be overridden here.
@@ -542,7 +551,9 @@ class DBConnection(EnforceOverrides):
LanceTable(name='table4', ...)
"""
raise NotImplementedError
raise NotImplementedError(
"create_table is not supported for this connection type"
)
def __getitem__(self, name: str) -> LanceTable:
return self.open_table(name)
@@ -597,7 +608,9 @@ class DBConnection(EnforceOverrides):
-------
A LanceTable object representing the table.
"""
raise NotImplementedError
raise NotImplementedError(
"open_table is not supported for this connection type"
)
def create_materialized_view(
self,
@@ -664,10 +677,11 @@ class DBConnection(EnforceOverrides):
) -> Job[None]:
"""Submit materialized-view creation and return its job.
The job may already be complete for a local database. On LanceDB
Cloud and Enterprise, its ``id`` is the server job identifier from
the ``202 Accepted`` create response. Wait for the job before opening
or querying the view.
The job may already be complete for a local database. Remote creation
submits CREATE MATERIALIZED VIEW through the SQL server. Its job has
``job_type == "query"`` and ``id`` identifies the connection-scoped
query. Registry event history is unavailable.
Wait for the job before opening or querying the view.
"""
raise NotImplementedError(
"materialized views are not supported on this connection type"
@@ -737,13 +751,17 @@ class DBConnection(EnforceOverrides):
"""
if namespace_path is None:
namespace_path = []
raise NotImplementedError
raise NotImplementedError(
"drop_table is not supported for this connection type"
)
def drop_table_async(
self, name: str, namespace_path: Optional[List[str]] = None
) -> Job:
"""Start dropping a table and return its cleanup job."""
raise NotImplementedError
raise NotImplementedError(
"drop_table_async is not supported for this connection type"
)
def rename_table(
self,
@@ -771,19 +789,26 @@ class DBConnection(EnforceOverrides):
cur_namespace_path = []
if new_namespace_path is None:
new_namespace_path = []
raise NotImplementedError
raise NotImplementedError(
"rename_table is not supported for this connection type"
)
def drop_database(self):
"""
Drop database
This is the same thing as dropping all the tables
"""
raise NotImplementedError
raise NotImplementedError(
"drop_database is not supported for this connection type"
)
def drop_all_tables(self, namespace_path: Optional[List[str]] = None):
"""
Drop all tables from the database
Remote connections do not currently support this operation. Use
``list_tables()`` and ``drop_table()`` to drop tables individually.
Parameters
----------
namespace_path: List[str], optional
@@ -792,12 +817,32 @@ class DBConnection(EnforceOverrides):
"""
if namespace_path is None:
namespace_path = []
raise NotImplementedError
raise NotImplementedError(
"drop_all_tables is not supported for this connection type"
)
@property
def uri(self) -> str:
"""The URI of this database."""
return self._uri
def _all_table_names(self) -> Generator[str, None, None]:
page_token = None
while True:
response = self.list_tables(page_token=page_token)
yield from response.tables
page_token = response.page_token
if not page_token:
return
def __len__(self) -> int:
"""Return the number of tables in the root namespace across all pages."""
return sum(1 for _ in self._all_table_names())
def __contains__(self, name: str) -> bool:
"""Return whether the root namespace contains the named table."""
return name in self._all_table_names()
def namespace_client(self) -> LanceNamespace:
"""Get the equivalent namespace client for this connection.
@@ -1230,18 +1275,19 @@ class DBConnection(EnforceOverrides):
parameters=[np.array([0.1, 0.2, 0.3], dtype=np.float32), 10],
)
Parameters are supported in queries, not in DDL or DML. A
parameterized query runs on the call that returns its rows:
cancelling it, or dropping its handle before reading, stops it on the
server. Its status follows that stream as well: ``describe()``
reports ``running`` until the reader has received every row, so open
``reader()`` directly rather than waiting for ``finished``.
Parameters are supported in queries, not in DDL or DML. With or
without parameters, the server chooses whether to stream rows on the
initial call or return a polling continuation. The reader follows
either path transparently. Dropping an unread handle or its reader
stops a directly streamed query; deferred execution requires explicit
cancellation. Open ``reader()`` directly: a streamed query needs its
reader to make progress and reports ``running`` until the stream ends.
Local connections do not support SQL.
"""
raise NotImplementedError("SQL is not supported for this connection type")
def describe_query(self, query_id: UUID) -> QueryDescription:
def describe_query(self, query_id: Union[UUID, str]) -> QueryDescription:
"""Describe a submitted SQL query by its connection-scoped id."""
raise NotImplementedError("SQL is not supported for this connection type")
@@ -1436,7 +1482,7 @@ class LanceDBConnection(DBConnection):
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListNamespacesResponse:
) -> NamespaceListing:
"""List immediate child namespace names in the given namespace.
Parameters
@@ -1452,16 +1498,14 @@ class LanceDBConnection(DBConnection):
Returns
-------
ListNamespacesResponse
NamespaceListing
Response containing namespace names and optional page_token for pagination.
"""
if namespace_path is None:
namespace_path = []
return LOOP.run(
return NamespaceListing(
self._conn.list_namespaces(
namespace_path=namespace_path,
page_token=page_token,
limit=limit,
namespace_path=namespace_path, page_token=page_token, limit=limit
)
)
@@ -1512,8 +1556,8 @@ class LanceDBConnection(DBConnection):
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListTablesResponse:
"""List a page of tables in this database.
) -> TableListing:
"""List tables in this database, one page at a time.
Parameters
----------
@@ -1529,12 +1573,12 @@ class LanceDBConnection(DBConnection):
Returns
-------
ListTablesResponse
TableListing
Response containing table names and optional page_token for pagination.
"""
if namespace_path is None:
namespace_path = []
return LOOP.run(
return TableListing(
self._conn.list_tables(
namespace_path=namespace_path, page_token=page_token, limit=limit
)
@@ -1583,21 +1627,6 @@ class LanceDBConnection(DBConnection):
)
)
def _all_table_names(self) -> Generator[str, None, None]:
page_token = None
while True:
response = self.list_tables(page_token=page_token)
yield from response.tables
page_token = response.page_token
if not page_token:
return
def __len__(self) -> int:
return sum(1 for _ in self._all_table_names())
def __contains__(self, name: str) -> bool:
return name in self._all_table_names()
@override
def create_table(
self,
@@ -2286,12 +2315,12 @@ class AsyncConnection(object):
else:
return None
async def list_namespaces(
def list_namespaces(
self,
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListNamespacesResponse:
) -> AsyncNamespaceListing:
"""List immediate child namespace names in the given namespace.
Parameters
@@ -2306,15 +2335,18 @@ class AsyncConnection(object):
Returns
-------
ListNamespacesResponse
AsyncNamespaceListing
Response containing namespace names and optional pagination token
"""
if namespace_path is None:
namespace_path = []
result = await self._inner.list_namespaces(
validate_page_limit(limit)
inner = self._inner.list_namespaces(
namespace_path=namespace_path, page_token=page_token, limit=limit
)
return ListNamespacesResponse(**result)
if limit == 0:
return AsyncNamespaceListing(None, empty=True)
return AsyncNamespaceListing(inner, page_token)
async def create_namespace(
self,
@@ -2393,13 +2425,13 @@ class AsyncConnection(object):
result = await self._inner.describe_namespace(namespace_path)
return DescribeNamespaceResponse(**result)
async def list_tables(
def list_tables(
self,
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListTablesResponse:
"""List a page of tables in this database.
) -> AsyncTableListing:
"""List tables in this database, one page at a time.
Parameters
----------
@@ -2415,15 +2447,18 @@ class AsyncConnection(object):
Returns
-------
ListTablesResponse
AsyncTableListing
Response containing table names and optional page_token for pagination.
"""
if namespace_path is None:
namespace_path = []
result = await self._inner.list_tables(
validate_page_limit(limit)
inner = self._inner.list_tables(
namespace_path=namespace_path, page_token=page_token, limit=limit
)
return ListTablesResponse(**result)
if limit == 0:
return AsyncTableListing(None, empty=True)
return AsyncTableListing(inner, page_token)
async def table_names(
self,
@@ -2521,10 +2556,13 @@ class AsyncConnection(object):
it will not add the provided data but will validate against any
schema that's specified.
on_bad_vectors: str, default "error"
What to do if any of the vectors are not the same size or contains NaNs.
One of "error", "drop", "fill".
How to handle vectors with incorrect sizes, NaNs, or null elements.
One of "error", "drop", "fill", "null".
fill_value: float
The value to use when filling vectors. Only used if on_bad_vectors="fill".
NaN or null elements and missing trailing elements are replaced with this
value. Valid elements are preserved, and vectors longer than the expected
dimension are truncated.
storage_options: dict, optional
Additional options for the storage backend. Options already set on the
connection will be inherited by the table, but can be overridden here.
@@ -2841,7 +2879,9 @@ class AsyncConnection(object):
) -> AsyncJob[None]:
"""Submit materialized-view creation and return its job.
Wait for the returned job before opening or querying the view.
Wait for the returned job before opening or querying the view. Remote
creation uses SQL; ``job_type`` is ``"query"`` and ``id``
identifies that query on this connection.
"""
source, native_options = await self._materialized_view_source(
source, select, where, limit
@@ -3381,8 +3421,14 @@ class AsyncConnection(object):
)
)
async def describe_query(self, query_id: UUID) -> QueryDescription:
"""Describe a submitted SQL query by its connection-scoped id."""
async def describe_query(self, query_id: Union[UUID, str]) -> QueryDescription:
"""Describe a submitted SQL query by its UUID or string ID.
For a handle with ``job_type == "query"``, pass its ``id`` on the
connection that submitted it. Registry job IDs are not query IDs.
"""
if isinstance(query_id, str):
query_id = UUID(query_id)
return await self._inner.describe_query(query_id)
async def namespace_client(self) -> LanceNamespace:
python/python/lancedb/embeddings/open_clip.py
+3
View File
@@ -49,6 +49,9 @@ class OpenClipEmbeddings(EmbeddingFunction):
self.name, pretrained=self.pretrained
)
model.to(self.device)
# open_clip returns the model in training mode, which keeps dropout,
# stochastic depth and batch norm in their training behaviour.
model.eval()
self._model, self._preprocess = model, preprocess
self._tokenizer = open_clip.get_tokenizer(self.name)
self._ndims = None
python/python/lancedb/functions.py
+59 -21
View File
@@ -33,6 +33,7 @@ from typing import (
Annotated,
Any,
Callable,
Literal,
Optional,
Union,
get_args,
@@ -52,7 +53,9 @@ from pydantic import (
model_validator,
)
from .schema import blob as _blob_field
from .schema import is_blob_v2_field as _is_blob_v2_field
from ._blob_reader import BlobReader
from .secrets import EnvVarSecret
_Int32 = conint(strict=True, ge=-(2**31), le=2**31 - 1)
@@ -213,6 +216,7 @@ class FunctionParameter(_RemoteValue):
name: str
arrow_type: str
nullable: bool
blob_access: Optional[Literal["bytes", "reader"]] = None
class FunctionResultField(_OpenRemoteValue):
@@ -1096,6 +1100,43 @@ def _infer_signature(
if (input_schema is None) != (output_schema is None):
raise ValueError("input_schema and output_schema must be provided together")
try:
annotations = get_type_hints(function, include_extras=True)
except Exception as error:
raise TypeError(f"failed to resolve Function annotations: {error}") from error
def input_parameter(field):
annotation = annotations.get(field.name)
if get_origin(annotation) in (Union, types.UnionType):
choices = [
value for value in get_args(annotation) if value is not type(None)
]
if len(choices) == 1:
annotation = choices[0]
arrow_type = _canonical_arrow_field(field)
if annotation is BlobReader:
if arrow_type != _FUNCTION_BLOB_V2_TYPE:
raise TypeError(
f"BlobReader input '{field.name}' requires a Blob V2 field"
)
access = "reader"
elif annotation is bytes:
if arrow_type != _FUNCTION_BLOB_V2_TYPE and not (
pa.types.is_binary(field.type) or pa.types.is_large_binary(field.type)
):
raise TypeError(
f"bytes input '{field.name}' requires binary or Blob V2"
)
access = "bytes"
else:
access = "bytes" if arrow_type == _FUNCTION_BLOB_V2_TYPE else None
return FunctionParameter(
name=field.name,
arrow_type=arrow_type,
nullable=field.nullable,
blob_access=access,
)
if input_schema is not None:
if not isinstance(input_schema, pa.Schema):
raise TypeError("input_schema must be a PyArrow Schema")
@@ -1110,20 +1151,9 @@ def _infer_signature(
"input_schema fields must exactly match callable parameters in order: "
f"expected {expected!r}, got {actual!r}"
)
inputs = tuple(
FunctionParameter(
name=field.name,
arrow_type=_canonical_arrow_field(field),
nullable=field.nullable,
)
for field in input_schema
)
inputs = tuple(input_parameter(field) for field in input_schema)
return FunctionSignature(inputs=inputs, output=_function_output(output_schema))
try:
annotations = get_type_hints(function, include_extras=True)
except Exception as error:
raise TypeError(f"failed to resolve Function annotations: {error}") from error
missing = [
parameter.name for parameter in parameters if parameter.name not in annotations
]
@@ -1132,15 +1162,21 @@ def _infer_signature(
raise TypeError(f"missing Function annotations: {names!r}")
inputs = []
for parameter in parameters:
data_type, nullable = _annotation_type(annotations[parameter.name])
inputs.append(
FunctionParameter(
name=parameter.name,
arrow_type=_canonical_arrow_field(
pa.field(parameter.name, data_type, nullable=nullable)
),
nullable=nullable,
annotation = annotations[parameter.name]
base = annotation
nullable = False
if get_origin(base) in (Union, types.UnionType):
choices = [value for value in get_args(base) if value is not type(None)]
if len(choices) == 1:
base, nullable = choices[0], True
if base is BlobReader:
inputs.append(
input_parameter(_blob_field(parameter.name, nullable=nullable))
)
continue
data_type, nullable = _annotation_type(annotation)
inputs.append(
input_parameter(pa.field(parameter.name, data_type, nullable=nullable))
)
output_type, output_nullable = _annotation_type(annotations["return"])
if output_nullable:
@@ -1239,7 +1275,7 @@ _PYTHON_CALLABLE_ARTIFACT = "python_callable"
_PYTHON_BUNDLE_ARTIFACT = "python_bundle"
# Modules the Function image places beside user code.
_RESERVED_SOURCE_MODULES = frozenset(
{"user_function", "lance_source", "lance_callable"}
{"user_function", "lance_source", "lance_callable", "lance_blob"}
)
# The service accepts bundle paths made of these components only.
_SOURCE_PATH_COMPONENT = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
@@ -1499,6 +1535,8 @@ class _SourcePackager:
"""One module-level line that rebinds `name` to `value` in the artifact:
an import for modules and importable classes/functions, a literal
otherwise."""
if value is BlobReader:
return f"from lance_blob import BlobReader as {name}"
if isinstance(value, types.ModuleType):
if value.__name__.split(".")[0] in _NAMESPACE_MODULES:
raise ValueError(
python/python/lancedb/index.py
+6 -2
View File
@@ -882,8 +882,12 @@ class IvfRq:
num_partitions: int, default sqrt(num_rows)
Number of IVF partitions to create.
num_bits: int, default 1
num_bits: int, optional
Number of bits to encode each dimension in the RabitQ codebook.
If None, uses Lance's default (currently 5), matching Rust and Node.
Set to 1 to retain the previous Python default and maximum compression.
Five-bit vector codes use roughly five times the storage of one-bit codes,
in addition to other index overhead.
max_iterations: int, default 50
Max iterations to train kmeans when computing IVF partitions.
@@ -897,7 +901,7 @@ class IvfRq:
distance_type: Literal["l2", "cosine", "dot"] = "l2"
num_partitions: Optional[int] = None
num_bits: int = 1
num_bits: Optional[int] = None
max_iterations: int = 50
sample_rate: int = 256
target_partition_size: Optional[int] = None
python/python/lancedb/job.py
+17 -13
View File
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
"""Handles to operations a server may run asynchronously."""
"""Handles to asynchronous registry jobs, SQL queries, and local operations."""
import asyncio
import json
@@ -44,12 +44,12 @@ class AsyncJob(Generic[T]):
@property
def id(self) -> Optional[str]:
"""Identifies the operation on the server that is running it.
"""The operation's opaque ID, interpreted according to :attr:`job_type`.
Returned for correlating with server logs or the jobs API. Operations
that run in this process have no server id and return `None`. The value
is opaque: parsing it or storing it to resume the job later is not
supported.
Registry jobs expose a registry job ID; SQL-backed jobs expose a
connection-scoped query ID as a string. Local operations return ``None``.
Query IDs cannot be used with registry job APIs or to resume after a
client restart.
"""
return self._inner.id if self._inner is not None else None
@@ -94,9 +94,11 @@ class AsyncJob(Generic[T]):
"""Ask the backend for this job's current state, and for a server-side
job its full record, then cache it for the properties below.
The properties are all `None` until this runs, because submitting an
operation returns only a job id. `status` fetches the whole record too;
`wait` records only the terminal state it establishes.
Registry details are `None` until this runs, because submitting an
operation does not fetch its full registry record. Query-backed jobs
expose only lifecycle state through this method. For registry jobs,
`status` fetches the whole record too; `wait` records only the terminal
state it establishes.
"""
if self._inner is None:
return
@@ -114,11 +116,13 @@ class AsyncJob(Generic[T]):
@property
def job_type(self) -> Optional[str]:
"""The job's type, as the server names it.
"""The job's kind of work, without contacting the backend.
`None` for an in-process job, which has no server-side record.
SQL-backed handles report ``"query"`` immediately. Registry handles
report the server's job type after :meth:`refresh` or :meth:`status`.
In-process handles return ``None``.
"""
return self._field("job_type")
return self._inner.job_type if self._inner is not None else None
@property
def creation_ms(self) -> Optional[int]:
@@ -294,7 +298,7 @@ class Job(Generic[T]):
@property
def job_type(self) -> Optional[str]:
"""The job's type. See :attr:`AsyncJob.job_type`."""
return self._field("job_type")
return self._inner.job_type if self._inner is not None else None
@property
def creation_ms(self) -> Optional[int]:
python/python/lancedb/listing.py
+171
View File
@@ -5,6 +5,7 @@
from __future__ import annotations
import asyncio
from typing import Any, AsyncIterator, Callable, Generic, Iterator, Optional, TypeVar
from .background_loop import LOOP
@@ -83,3 +84,173 @@ class Listing(Iterator[T], Generic[T]):
for initial, cached, final-page, and error behavior.
"""
return self._inner.page_token()
def validate_page_limit(limit: Optional[int]) -> None:
"""Validate the legacy limit argument before passing it to the bindings."""
if limit is not None and (
not isinstance(limit, int) or limit < 0 or limit > 2**31 - 1
):
raise ValueError("limit must be between 0 and 2147483647")
class _AsyncPagedNameListing(AsyncIterator[str]):
"""An async listing with an inspectable copy of its current backend page."""
def __init__(
self, inner: Any, page_token: Optional[str] = None, *, empty: bool = False
):
self._inner = inner
self._token = page_token
self._page: Optional[list[str]] = [] if empty else None
self._position = 0
self._done = empty
self._failure: Optional[BaseException] = None
self._lock = asyncio.Lock()
@property
def page_token(self) -> Optional[str]:
"""The continuation token for the next backend request."""
return self._token
def num_page_results(self) -> int:
"""Count the remaining names in the current page."""
return 0 if self._page is None else len(self._page) - self._position
def _page_copy(self) -> list[str]:
if self._page is None:
raise RuntimeError("Await or iterate the listing before reading its page")
return self._page.copy()
async def _fetch_page(self) -> None:
try:
page = await self._inner.next_page()
except asyncio.CancelledError:
raise
except BaseException as exc:
self._failure = exc
self._done = True
raise
if page is None:
self._page = []
self._token = None
self._done = True
else:
self._page = page["items"]
self._token = page["page_token"]
self._done = self._token is None
self._position = 0
async def _load_initial(self):
async with self._lock:
if self._failure is not None:
raise self._failure
if self._page is None:
await self._fetch_page()
return self
def __await__(self):
return self._load_initial().__await__()
def __aiter__(self):
return self
async def __anext__(self) -> str:
async with self._lock:
if self._failure is not None:
raise StopAsyncIteration
if self._page is None:
await self._fetch_page()
assert self._page is not None
while self._position >= len(self._page):
if self._done:
raise StopAsyncIteration
await self._fetch_page()
assert self._page is not None
item = self._page[self._position]
self._position += 1
return item
class AsyncTableListing(_AsyncPagedNameListing):
"""Awaitable table iterator; awaiting loads one page.
For example::
page = await db.list_tables(limit=20)
print(page.tables, page.page_token)
async for name in db.list_tables(limit=20):
print(name)
"""
@property
def tables(self) -> list[str]:
"""A copy of every table name in the most recently fetched page."""
return self._page_copy()
class AsyncNamespaceListing(_AsyncPagedNameListing):
"""Awaitable namespace iterator; awaiting loads one page.
For example::
page = await db.list_namespaces(limit=20)
print(page.namespaces, page.page_token)
"""
@property
def namespaces(self) -> list[str]:
"""A copy of every name in the most recently fetched page."""
return self._page_copy()
class _PagedNameListing(Iterator[str]):
def __init__(self, inner: _AsyncPagedNameListing):
self._inner = inner
LOOP.run(inner._load_initial())
def __iter__(self):
return self
def __next__(self) -> str:
try:
return LOOP.run(self._inner.__anext__())
except StopAsyncIteration:
raise StopIteration from None
@property
def page_token(self) -> Optional[str]:
return self._inner.page_token
def num_page_results(self) -> int:
return self._inner.num_page_results()
class TableListing(_PagedNameListing):
"""Table iterator retaining the current page as a list.
For example::
page = db.list_tables(limit=20)
print(page.tables, page.page_token)
for name in db.list_tables(limit=20):
print(name)
"""
@property
def tables(self) -> list[str]:
return self._inner.tables
class NamespaceListing(_PagedNameListing):
"""Namespace iterator retaining the current page as a list.
For example::
page = db.list_namespaces(limit=20)
print(page.namespaces, page.page_token)
"""
@property
def namespaces(self) -> list[str]:
return self._inner.namespaces
python/python/lancedb/namespace.py
+23 -18
View File
@@ -52,7 +52,14 @@ from lancedb.background_loop import LOOP
from lancedb.arrow import AsyncRecordBatchReader
from lancedb.db import AsyncConnection, DBConnection
from lancedb.job import AsyncJob, Job
from lancedb.listing import AsyncListing, Listing
from lancedb.listing import (
AsyncListing,
AsyncNamespaceListing,
AsyncTableListing,
Listing,
NamespaceListing,
TableListing,
)
from lancedb.sql import AsyncQuery as AsyncSqlQuery
from lancedb.sql import QueryDescription, QueryParameters
from lance_namespace import (
@@ -61,8 +68,6 @@ from lance_namespace import (
CreateNamespaceResponse,
DescribeNamespaceResponse,
DropNamespaceResponse,
ListNamespacesResponse,
ListTablesResponse,
NamespaceExistsRequest,
TableExistsRequest,
)
@@ -875,7 +880,7 @@ class LanceNamespaceDBConnection(DBConnection):
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListNamespacesResponse:
) -> NamespaceListing:
"""
List child namespaces under the given namespace.
@@ -892,12 +897,12 @@ class LanceNamespaceDBConnection(DBConnection):
Returns
-------
ListNamespacesResponse
NamespaceListing
Response containing namespace names and optional page_token for pagination.
"""
if namespace_path is None:
namespace_path = []
return LOOP.run(
return NamespaceListing(
self._inner.list_namespaces(
namespace_path=namespace_path, page_token=page_token, limit=limit
)
@@ -1044,7 +1049,7 @@ class LanceNamespaceDBConnection(DBConnection):
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListTablesResponse:
) -> TableListing:
"""
List all tables in this database with pagination support.
@@ -1061,12 +1066,12 @@ class LanceNamespaceDBConnection(DBConnection):
Returns
-------
ListTablesResponse
TableListing
Response containing table names and optional page_token for pagination.
"""
if namespace_path is None:
namespace_path = []
return LOOP.run(
return TableListing(
self._inner.list_tables(
namespace_path=namespace_path, page_token=page_token, limit=limit
)
@@ -1483,12 +1488,12 @@ class AsyncLanceNamespaceDBConnection:
namespace_path = []
await self._inner.drop_all_tables(namespace_path=namespace_path)
async def list_namespaces(
def list_namespaces(
self,
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListNamespacesResponse:
) -> AsyncNamespaceListing:
"""
List child namespaces under the given namespace.
@@ -1505,12 +1510,12 @@ class AsyncLanceNamespaceDBConnection:
Returns
-------
ListNamespacesResponse
NamespaceListing
Response containing namespace names and optional page_token for pagination.
"""
if namespace_path is None:
namespace_path = []
return await self._inner.list_namespaces(
return self._inner.list_namespaces(
namespace_path=namespace_path, page_token=page_token, limit=limit
)
@@ -1640,12 +1645,12 @@ class AsyncLanceNamespaceDBConnection:
except TableNotFoundError:
return False
async def list_tables(
def list_tables(
self,
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListTablesResponse:
) -> AsyncTableListing:
"""
List all tables in this database with pagination support.
@@ -1662,12 +1667,12 @@ class AsyncLanceNamespaceDBConnection:
Returns
-------
ListTablesResponse
TableListing
Response containing table names and optional page_token for pagination.
"""
if namespace_path is None:
namespace_path = []
return await self._inner.list_tables(
return self._inner.list_tables(
namespace_path=namespace_path, page_token=page_token, limit=limit
)
@@ -1702,7 +1707,7 @@ class AsyncLanceNamespaceDBConnection:
parameters=parameters,
)
async def describe_query(self, query_id: UUID) -> QueryDescription:
async def describe_query(self, query_id: Union[UUID, str]) -> QueryDescription:
"""Describe a submitted SQL query when supported."""
return await self._inner.describe_query(query_id)
python/python/lancedb/permutation.py
+5 -3
View File
@@ -916,9 +916,11 @@ class Permutation:
f"Cannot rename column `{old_name}` to `{new_name}` because a column "
"with that name already exists"
)
new_selection = self.selection.copy()
new_selection[new_name] = new_selection[old_name]
del new_selection[old_name]
# Rebuild the mapping so the renamed column keeps its position
new_selection = {
(new_name if name == old_name else name): value
for name, value in self.selection.items()
}
return self._with_selection(new_selection)
def rename_columns(self, column_map: dict[str, str]) -> "Permutation":
python/python/lancedb/query.py
+17 -12
View File
@@ -1561,9 +1561,10 @@ class LanceQueryBuilder(ABC):
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], ...
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
<BLANKLINE>
Parameters
----------
@@ -1619,8 +1620,8 @@ class LanceQueryBuilder(ABC):
GlobalLimitExec: elapsed=..., skip=0, fetch=10,
metrics=[output_rows=..., elapsed_compute=..., output_bytes=...]
FilterExec: elapsed=..., _distance@2 IS NOT NULL, metrics=[...]
SortExec: elapsed=..., TopK(fetch=10), expr=[...],
preserve_partitioning=[...],
SortExec: elapsed=..., TopK(fetch=1), expr=[...],
preserve_partitioning=[...], filter=[...],
metrics=[output_rows=..., elapsed_compute=...,
output_bytes=..., row_replacements=...]
KNNVectorDistance: elapsed=..., metric=l2,
@@ -1633,6 +1634,7 @@ class LanceQueryBuilder(ABC):
metrics=[output_rows=..., elapsed_compute=..., output_bytes=...,
fragments_scanned=..., ranges_scanned=1, rows_scanned=1,
bytes_read=..., iops=..., requests=..., task_wait_time=...]
<BLANKLINE>
Parameters
----------
@@ -2959,9 +2961,10 @@ class LanceHybridQueryBuilder(LanceQueryBuilder):
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], ...
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
<BLANKLINE>
Parameters
----------
@@ -3601,9 +3604,10 @@ class AsyncQueryBase(object):
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], ...
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
<BLANKLINE>
<BLANKLINE>
Parameters
@@ -4521,9 +4525,9 @@ class AsyncHybridQuery(AsyncStandardQuery, AsyncVectorQueryBase):
LanceRead: uri=..., projection=[text], source=stream(_rowid)
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], ...
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
ProjectionExec: expr=[vector@2 as vector, text@3 as text, _score@1 as _score, _rowid@0 as _rowid]
LanceRead: uri=..., projection=[vector, text], source=stream(_rowid)
GlobalLimitExec: skip=0, fetch=10
@@ -4877,9 +4881,10 @@ class BaseQueryBuilder(object):
ProjectionExec: expr=[vector@0 as vector, _distance@2 as _distance]
GlobalLimitExec: skip=0, fetch=10
FilterExec: _distance@2 IS NOT NULL
SortExec: TopK(fetch=10), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
SortExec: TopK(fetch=1), expr=[_distance@2 ASC NULLS LAST, _rowid@1 ASC NULLS LAST], preserve_partitioning=[false]
KNNVectorDistance: metric=l2
LanceRead: uri=..., projection=[vector], ...
LanceRead: uri=..., projection=[vector], num_fragments=..., ...
<BLANKLINE>
<BLANKLINE>
Parameters
python/python/lancedb/remote/db.py
+67 -25
View File
@@ -7,7 +7,6 @@ from __future__ import annotations
from dataclasses import replace
from datetime import timedelta
import json
import logging
from concurrent.futures import ThreadPoolExecutor
import sys
from typing import (
@@ -36,7 +35,7 @@ from lancedb.remote import ClientConfig, RetryConfig, TimeoutConfig, TlsConfig
import pyarrow as pa
from ..common import DATA
from lancedb.listing import Listing
from lancedb.listing import Listing, NamespaceListing, TableListing
from ..db import DBConnection, LOOP
from ..functions import FunctionVersion, UdfDefinition
from ..job import AsyncJob, Job
@@ -47,15 +46,13 @@ from ..secrets import EnvVarSecret, SecretInfo
from ..view import ViewDescription
if TYPE_CHECKING:
from .._lancedb import JobInfo
from .._lancedb import JobInfo, Session
from ..embeddings import EmbeddingFunctionConfig
from lance_namespace import (
LanceNamespace,
CreateNamespaceResponse,
DescribeNamespaceResponse,
DropNamespaceResponse,
ListNamespacesResponse,
ListTablesResponse,
)
from ..pydantic import LanceModel
from ..table import Table
@@ -242,6 +239,23 @@ class RemoteDBConnection(DBConnection):
def __repr__(self) -> str:
return f"RemoteConnect(name={self.db_name})"
@property
@override
def uri(self) -> str:
return self._conn.uri
@property
def session(self) -> Optional["Session"]:
"""Remote connections do not expose a local cache session."""
raise NotImplementedError("session is not supported by remote connections")
@property
def read_consistency_interval(self) -> Optional[timedelta]:
"""Remote connections do not expose a local read consistency interval."""
raise NotImplementedError(
"read_consistency_interval is not supported by remote connections"
)
@override
def serialize(self) -> str:
if getattr(self, "_catalog_oauth", False):
@@ -268,7 +282,7 @@ class RemoteDBConnection(DBConnection):
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListNamespacesResponse:
) -> NamespaceListing:
"""List immediate child namespace names in the given namespace.
Parameters
@@ -284,12 +298,12 @@ class RemoteDBConnection(DBConnection):
Returns
-------
ListNamespacesResponse
NamespaceListing
Response containing namespace names and optional page_token for pagination.
"""
if namespace_path is None:
namespace_path = []
return LOOP.run(
return NamespaceListing(
self._conn.list_namespaces(
namespace_path=namespace_path, page_token=page_token, limit=limit
)
@@ -379,7 +393,7 @@ class RemoteDBConnection(DBConnection):
namespace_path: Optional[List[str]] = None,
page_token: Optional[str] = None,
limit: Optional[int] = None,
) -> ListTablesResponse:
) -> TableListing:
"""List a page of tables in this database.
Parameters
@@ -396,12 +410,12 @@ class RemoteDBConnection(DBConnection):
Returns
-------
ListTablesResponse
TableListing
Response containing table names and optional page_token for pagination.
"""
if namespace_path is None:
namespace_path = []
return LOOP.run(
return TableListing(
self._conn.list_tables(
namespace_path=namespace_path, page_token=page_token, limit=limit
)
@@ -487,18 +501,14 @@ class RemoteDBConnection(DBConnection):
if namespace_path is None:
namespace_path = []
if storage_options is not None:
logging.info(
"storage_options is ignored for remote tables"
" (storage is managed; set storage_options on connect() instead)"
table = LOOP.run(
self._conn.open_table(
name,
namespace_path=namespace_path,
storage_options=storage_options,
index_cache_size=index_cache_size,
)
if index_cache_size is not None:
logging.info(
"index_cache_size is ignored for remote tables"
" (there is no local cache to configure)"
)
table = LOOP.run(self._conn.open_table(name, namespace_path=namespace_path))
)
tbl = RemoteTable(
table,
self.db_name,
@@ -571,13 +581,16 @@ class RemoteDBConnection(DBConnection):
name: str,
data: DATA = None,
schema: Optional[Union[pa.Schema, LanceModel]] = None,
on_bad_vectors: str = "error",
fill_value: float = 0.0,
mode: Optional[str] = None,
exist_ok: bool = False,
on_bad_vectors: str = "error",
fill_value: float = 0.0,
embedding_functions: Optional[List[EmbeddingFunctionConfig]] = None,
*,
namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None,
data_storage_version: Optional[str] = None,
enable_v2_manifest_paths: Optional[bool] = None,
) -> Table:
"""Create a [Table][lancedb.table.Table] in the database.
@@ -617,6 +630,12 @@ class RemoteDBConnection(DBConnection):
embedding_functions: list of EmbeddingFunctionConfig, optional
The embedding functions to store in the table schema and use to
generate vectors when creating the table or adding data.
storage_options: dict, optional
Not supported for remote table creation. Raises NotImplementedError.
data_storage_version: str, optional
Not supported for remote table creation. Raises NotImplementedError.
enable_v2_manifest_paths: bool, optional
Not supported for remote table creation. Raises NotImplementedError.
Returns
-------
@@ -683,6 +702,14 @@ class RemoteDBConnection(DBConnection):
LanceTable(table4)
"""
for option, value in (
("data_storage_version", data_storage_version),
("enable_v2_manifest_paths", enable_v2_manifest_paths),
):
if value is not None:
raise NotImplementedError(
f"{option} is not supported for remote table creation"
)
if exist_ok:
if mode == "create":
mode = "exist_ok"
@@ -704,6 +731,7 @@ class RemoteDBConnection(DBConnection):
on_bad_vectors=on_bad_vectors,
fill_value=fill_value,
embedding_functions=embedding_functions,
storage_options=storage_options,
)
)
return RemoteTable(
@@ -843,6 +871,20 @@ class RemoteDBConnection(DBConnection):
job = LOOP.run(self._conn.drop_table_async(name, namespace_path=namespace_path))
return Job(job if isinstance(job, AsyncJob) else AsyncJob(job))
@override
def drop_all_tables(self, namespace_path: Optional[List[str]] = None):
"""Drop all tables from the database.
The remote API does not currently support this operation. Use
``list_tables()`` and ``drop_table()`` to drop tables individually.
"""
LOOP.run(self._conn.drop_all_tables(namespace_path=namespace_path))
@override
def drop_database(self):
"""Deprecated alias for ``drop_all_tables()``."""
LOOP.run(self._conn.drop_database())
@override
def rename_table(
self,
@@ -1122,7 +1164,7 @@ class RemoteDBConnection(DBConnection):
)
@override
def describe_query(self, query_id: UUID) -> QueryDescription:
def describe_query(self, query_id: Union[UUID, str]) -> QueryDescription:
"""Describe a submitted SQL query by its connection-scoped id."""
return LOOP.run(
self._conn.describe_query(
python/python/lancedb/remote/table.py
+255 -150
View File
@@ -4,9 +4,10 @@
from datetime import timedelta
import deprecation
import logging
from functools import cached_property
from functools import cached_property, wraps
import os
from typing import (
TYPE_CHECKING,
Any,
Callable,
Dict,
@@ -41,19 +42,12 @@ from lancedb.index import (
FTS,
BTree,
Bitmap,
HnswFlat,
HnswPq,
HnswSq,
IvfFlat,
IvfPq,
IvfRq,
IvfSq,
LabelList,
)
from lancedb.job import Job
from lancedb.functions import FunctionApplication, RefreshColumnResult
from lancedb.remote.db import LOOP
from lancedb.table import IndexConfigType, KNOWN_METRICS
from lancedb.table import IndexConfigType, _maybe_add_fts_error_note
import pyarrow as pa
from lancedb.common import DATA, VEC, VECTOR_COLUMN_NAME
@@ -68,7 +62,6 @@ from ..query import (
FullTextQuery,
LanceQueryBuilder,
LanceTakeQueryBuilder,
LanceVectorQueryBuilder,
)
from ..table import (
AsyncTable,
@@ -79,9 +72,35 @@ from ..table import (
Table,
Tags,
)
from ..types import BaseTokenizerType
from ..types import BaseTokenizerType, ScalarIndexType
from ..util import infer_vector_column_name
if TYPE_CHECKING:
import lance
import polars as pl
def _deprecated_keyword(old_name: str, new_name: str):
"""Keep old remote-only keyword names without changing the Table signature."""
def decorate(method):
@wraps(method)
def wrapped(self, *args, **kwargs):
if old_name in kwargs:
if new_name in kwargs:
raise TypeError(f"Cannot specify both {old_name} and {new_name}")
warnings.warn(
f"{old_name} is deprecated; use {new_name} instead",
DeprecationWarning,
stacklevel=2,
)
kwargs[new_name] = kwargs.pop(old_name)
return method(self, *args, **kwargs)
return wrapped
return decorate
class RemoteTable(Table):
def __init__(
@@ -173,6 +192,18 @@ class RemoteTable(Table):
"""The name of the table"""
return self._name
@property
def namespace(self) -> List[str]:
return self._namespace_path
@deprecation.deprecated(
deprecated_in="0.33.1",
current_version=__version__,
details="Use update_field_metadata() instead.",
)
def replace_field_metadata(self, field_name: str, new_metadata: Dict[str, str]):
LOOP.run(self._table.replace_field_metadata(field_name, new_metadata))
def __repr__(self) -> str:
return f"RemoteTable({self.db_name}.{self.name})"
@@ -243,12 +274,32 @@ class RemoteTable(Table):
def to_arrow(self, *, blob_mode: BlobMode = "lazy") -> pa.Table:
"""to_arrow() is not supported for remote tables."""
validate_blob_mode(blob_mode)
raise NotImplementedError("to_arrow() is not supported for remote tables.")
raise NotImplementedError(
"to_arrow() is not supported for remote tables. "
"Use table.search().to_arrow() instead."
)
def to_pandas(self, blob_mode: BlobMode = "lazy", **kwargs):
"""to_pandas() is not supported for remote tables."""
validate_blob_mode(blob_mode)
raise NotImplementedError("to_pandas() is not supported for remote tables.")
raise NotImplementedError(
"to_pandas() is not supported for remote tables. "
"Use table.search().to_pandas() instead."
)
def to_lance(self, **kwargs) -> "lance.LanceDataset":
"""Direct Lance dataset access is not supported for remote tables.
Use ``table.search().to_arrow()`` to read query results through the server.
"""
return LOOP.run(self._table.to_lance(**kwargs))
def to_polars(self, batch_size=None) -> "pl.LazyFrame":
"""Use ``table.search().to_polars()`` to read remote query results."""
raise NotImplementedError(
"to_polars() is not supported for remote tables. "
"Use table.search().to_polars() instead."
)
def checkout(self, version: Union[int, str]):
result = LOOP.run(self._table.checkout(version))
@@ -290,9 +341,10 @@ class RemoteTable(Table):
self._table.tokenize(query, column=column, index_name=index_name)
)
def index_stats(self, index_uuid: str) -> Optional[IndexStatistics]:
@_deprecated_keyword("index_uuid", "index_name")
def index_stats(self, index_name: str) -> Optional[IndexStatistics]:
"""List all the stats of a specified index"""
return LOOP.run(self._table.index_stats(index_uuid))
return LOOP.run(self._table.index_stats(index_name))
@deprecation.deprecated(
deprecated_in="0.25.0",
@@ -302,43 +354,38 @@ class RemoteTable(Table):
def create_scalar_index(
self,
column: str,
index_type: Literal["BTREE", "BITMAP", "LABEL_LIST", "scalar"] = "scalar",
*,
replace: bool = False,
replace: bool = True,
index_type: ScalarIndexType = "BTREE",
wait_timeout: Optional[timedelta] = None,
name: Optional[str] = None,
):
"""Creates a scalar index.
"""Create a scalar index on a column.
.. deprecated:: 0.25.0
Use :meth:`create_index` with a BTree, Bitmap, or LabelList config instead.
Example: ``table.create_index("column", config=BTree())``
Parameters
----------
column : str
The column to be indexed. Must be a boolean, integer, float,
or string column.
index_type : str
The index type of the scalar index. Must be "scalar" (BTREE),
"BTREE", "BITMAP", or "LABEL_LIST",
replace : bool
If True, replace the existing index with the new one.
"""
if index_type == "scalar" or index_type == "BTREE":
if index_type == "scalar":
warnings.warn(
'index_type="scalar" is deprecated; use "BTREE" instead',
DeprecationWarning,
stacklevel=2,
)
index_type = "BTREE"
if index_type == "BTREE":
config = BTree()
elif index_type == "BITMAP":
config = Bitmap()
elif index_type == "LABEL_LIST":
config = LabelList()
else:
raise ValueError(f"Unknown index type: {index_type}")
LOOP.run(
raise ValueError(f"Unknown index type {index_type}")
return LOOP.run(
self._table.create_index(
column,
config=config,
replace=replace,
config=config,
wait_timeout=wait_timeout,
name=name,
)
@@ -349,12 +396,16 @@ class RemoteTable(Table):
current_version=__version__,
details="Use create_index() with config=FTS() instead.",
)
@_deprecated_keyword("column", "field_names")
def create_fts_index(
self,
column: str,
field_names: Union[str, List[str]],
*,
ordering_field_names: Optional[Union[str, List[str]]] = None,
replace: bool = False,
wait_timeout: Optional[timedelta] = None,
writer_heap_size: Optional[int] = 1024 * 1024 * 1024,
use_tantivy: bool = False,
tokenizer_name: Optional[str] = None,
with_position: bool = False,
# tokenizer configs:
base_tokenizer: BaseTokenizerType = "simple",
@@ -370,6 +421,7 @@ class RemoteTable(Table):
prefix_only: bool = False,
block_size: int = 128,
document_granularity: DocumentGranularity = DocumentGranularity.ROW,
wait_timeout: Optional[timedelta] = None,
name: Optional[str] = None,
):
"""Create a full-text search index on a column.
@@ -378,31 +430,70 @@ class RemoteTable(Table):
Use :meth:`create_index` with an FTS config instead.
Example: ``table.create_index("text_column", config=FTS())``
"""
if use_tantivy:
raise ValueError(
"Tantivy-based FTS has been removed. "
"Remove use_tantivy and recreate the index with native FTS."
)
if ordering_field_names is not None:
raise ValueError(
"ordering_field_names was only supported by the removed "
"Tantivy-based FTS implementation."
)
if writer_heap_size != 1024 * 1024 * 1024:
raise ValueError(
"writer_heap_size was only supported by the removed "
"Tantivy-based FTS implementation."
)
if not isinstance(field_names, str):
raise ValueError(
"Native FTS indexes can only be created on a single field "
"at a time. To search over multiple text fields, create a "
"separate FTS index for each field."
)
if tokenizer_name is None:
tokenizer_configs = {
"base_tokenizer": base_tokenizer,
"language": language,
"with_position": with_position,
"max_token_length": max_token_length,
"lower_case": lower_case,
"stem": stem,
"remove_stop_words": remove_stop_words,
"custom_stop_words": custom_stop_words,
"ascii_folding": ascii_folding,
"ngram_min_length": ngram_min_length,
"ngram_max_length": ngram_max_length,
"prefix_only": prefix_only,
}
else:
tokenizer_configs = self.infer_tokenizer_configs(tokenizer_name)
tokenizer_configs["custom_stop_words"] = custom_stop_words
config = FTS(
with_position=with_position,
base_tokenizer=base_tokenizer,
language=language,
max_token_length=max_token_length,
lower_case=lower_case,
stem=stem,
remove_stop_words=remove_stop_words,
custom_stop_words=custom_stop_words,
ascii_folding=ascii_folding,
ngram_min_length=ngram_min_length,
ngram_max_length=ngram_max_length,
prefix_only=prefix_only,
block_size=block_size,
document_granularity=document_granularity,
**tokenizer_configs,
)
LOOP.run(
self._table.create_index(
column,
config=config,
replace=replace,
wait_timeout=wait_timeout,
name=name,
try:
LOOP.run(
self._table.create_index(
field_names,
replace=replace,
config=config,
name=name,
wait_timeout=wait_timeout,
)
)
)
except (ValueError, RuntimeError) as e:
_maybe_add_fts_error_note(
e,
base_tokenizer=config.base_tokenizer,
language=config.language,
)
raise e
# New unified API overload
@overload
@@ -412,6 +503,7 @@ class RemoteTable(Table):
/,
*,
config: IndexConfigType,
replace: bool = ...,
wait_timeout: Optional[timedelta] = ...,
name: Optional[str] = ...,
train: bool = ...,
@@ -422,38 +514,56 @@ class RemoteTable(Table):
def create_index(
self,
metric: Literal["l2", "cosine", "dot", "hamming"] = ...,
vector_column_name: str = ...,
index_cache_size: Optional[int] = ...,
num_partitions: Optional[int] = ...,
num_sub_vectors: Optional[int] = ...,
replace: Optional[bool] = ...,
vector_column_name: str = ...,
replace: bool = ...,
accelerator: Optional[str] = ...,
index_type: Literal[
"VECTOR", "IVF_FLAT", "IVF_SQ", "IVF_PQ", "IVF_HNSW_SQ", "IVF_HNSW_PQ"
] = ...,
wait_timeout: Optional[timedelta] = ...,
*,
index_cache_size: Optional[int] = ...,
num_bits: int = ...,
index_type: Literal[
"IVF_FLAT", "IVF_SQ", "IVF_PQ", "IVF_RQ", "IVF_HNSW_SQ", "IVF_HNSW_PQ"
] = ...,
max_iterations: int = ...,
sample_rate: int = ...,
m: int = ...,
ef_construction: int = ...,
*,
wait_timeout: Optional[timedelta] = ...,
name: Optional[str] = ...,
train: bool = ...,
target_partition_size: Optional[int] = ...,
) -> None: ...
def create_index(
self,
metric: str = "l2",
vector_column_name: str = VECTOR_COLUMN_NAME,
index_cache_size: Optional[int] = None,
num_partitions: Optional[int] = None,
num_sub_vectors: Optional[int] = None,
replace: Optional[bool] = None,
vector_column_name: str = VECTOR_COLUMN_NAME,
replace: bool = True,
accelerator: Optional[str] = None,
index_type="vector",
wait_timeout: Optional[timedelta] = None,
*,
index_cache_size: Optional[int] = None,
num_bits: int = 8,
index_type: Literal[
"IVF_FLAT",
"IVF_SQ",
"IVF_PQ",
"IVF_RQ",
"IVF_HNSW_SQ",
"IVF_HNSW_PQ",
"IVF_HNSW_FLAT",
] = "IVF_PQ",
max_iterations: int = 50,
sample_rate: int = 256,
m: int = 20,
ef_construction: int = 300,
*,
config: Optional[IndexConfigType] = None,
wait_timeout: Optional[timedelta] = None,
name: Optional[str] = None,
train: bool = True,
target_partition_size: Optional[int] = None,
):
"""Create an index on a column.
@@ -464,6 +574,24 @@ class RemoteTable(Table):
argument plus separate ``vector_column_name`` / ``num_partitions`` /
etc. parameters, and emits a ``DeprecationWarning``.
Parameters
----------
metric : str
For new API: the column name to index.
For legacy API: the distance metric ("l2", "cosine", "dot", "hamming").
config : IndexConfigType, optional
The index configuration object. If provided, uses the new unified API.
Can be one of: IvfFlat, IvfPq, IvfSq, IvfRq, HnswPq, HnswSq,
BTree, Bitmap, LabelList, Fm, FTS.
replace : bool, default True
Whether to replace an existing index on this column.
wait_timeout : timedelta, optional
Timeout to wait for async indexing to complete.
name : str, optional
Custom name for the index.
train : bool, default True
Whether to train the index with existing data.
Examples
--------
New API (recommended):
@@ -489,7 +617,6 @@ class RemoteTable(Table):
vector_column_name,
accelerator,
index_cache_size,
replace,
)
if is_legacy:
@@ -505,6 +632,7 @@ class RemoteTable(Table):
stacklevel=2,
)
# Legacy API: first arg is the distance metric
column = vector_column_name
if accelerator is not None:
@@ -513,45 +641,27 @@ class RemoteTable(Table):
"and will be ignored."
)
idx_type = index_type.upper()
if idx_type == "VECTOR" or idx_type == "IVF_PQ":
config = IvfPq(
distance_type=metric,
num_partitions=num_partitions,
num_sub_vectors=num_sub_vectors,
num_bits=num_bits,
)
elif idx_type == "IVF_RQ":
config = IvfRq(
distance_type=metric,
num_partitions=num_partitions,
num_bits=num_bits,
)
elif idx_type == "IVF_SQ":
config = IvfSq(distance_type=metric, num_partitions=num_partitions)
elif idx_type == "IVF_HNSW_PQ":
config = HnswPq(
distance_type=metric,
num_partitions=num_partitions,
num_sub_vectors=num_sub_vectors,
num_bits=num_bits,
)
elif idx_type == "IVF_HNSW_SQ":
config = HnswSq(distance_type=metric, num_partitions=num_partitions)
elif idx_type == "IVF_HNSW_FLAT":
config = HnswFlat(distance_type=metric, num_partitions=num_partitions)
elif idx_type == "IVF_FLAT":
config = IvfFlat(distance_type=metric, num_partitions=num_partitions)
else:
raise ValueError(
f"Unknown vector index type: {idx_type}. Valid options are"
" 'IVF_FLAT', 'IVF_PQ', 'IVF_RQ', 'IVF_SQ',"
" 'IVF_HNSW_PQ', 'IVF_HNSW_SQ', 'IVF_HNSW_FLAT'"
)
# Build config from legacy parameters
config = self._build_vector_config_from_legacy_params(
metric=metric,
index_type="IVF_PQ"
if index_type.upper() == "VECTOR"
else index_type.upper(),
num_partitions=num_partitions,
num_sub_vectors=num_sub_vectors,
num_bits=num_bits,
max_iterations=max_iterations,
sample_rate=sample_rate,
m=m,
ef_construction=ef_construction,
target_partition_size=target_partition_size,
accelerator=None,
)
else:
column = metric
LOOP.run(
return LOOP.run(
self._table.create_index(
column,
replace=replace,
@@ -590,37 +700,6 @@ class RemoteTable(Table):
)
)
def _is_legacy_create_index_call(
self,
first_arg: str,
config: Optional[IndexConfigType],
num_partitions: Optional[int],
num_sub_vectors: Optional[int],
vector_column_name: str,
accelerator: Optional[str],
index_cache_size: Optional[int],
replace: Optional[bool],
) -> bool:
"""Detect if this is a legacy create_index call."""
if config is not None:
return False
if any(
x is not None
for x in (
num_partitions,
num_sub_vectors,
accelerator,
index_cache_size,
replace,
)
):
return True
if vector_column_name != VECTOR_COLUMN_NAME:
return True
if first_arg.lower() in KNOWN_METRICS:
return True
return False
def add(
self,
data: DATA,
@@ -692,9 +771,10 @@ class RemoteTable(Table):
query: Union[VEC, str] = None,
vector_column_name: Optional[str] = None,
query_type="auto",
ordering_field_name: Optional[str] = None,
fts_columns: Optional[Union[str, List[str]]] = None,
fast_search: Optional[bool] = None,
) -> LanceVectorQueryBuilder:
) -> LanceQueryBuilder:
"""Create a search query to find the nearest neighbors
of the given query vector. We currently support
[vector search](https://lancedb.com/docs/search/vector-search/)
@@ -785,6 +865,7 @@ class RemoteTable(Table):
query,
query_type,
vector_column_name=vector_column_name,
ordering_field_name=ordering_field_name,
fts_columns=fts_columns or [],
fast_search=fast_search,
)
@@ -844,7 +925,8 @@ class RemoteTable(Table):
self._table._do_merge(merge, new_data, on_bad_vectors, fill_value)
)
def delete(self, predicate: str) -> DeleteResult:
@_deprecated_keyword("predicate", "where")
def delete(self, where: Union[str, Expr]) -> DeleteResult:
"""Delete rows from the table.
This can be used to delete a single row, many rows, all rows, or
@@ -852,7 +934,7 @@ class RemoteTable(Table):
Parameters
----------
predicate: str
where: Union[str, Expr]
The SQL where clause to use when deleting rows.
- For example, 'x = 2' or 'x IN (1, 2, 3)'.
@@ -896,7 +978,7 @@ class RemoteTable(Table):
x vector _distance # doctest: +SKIP
0 2 [3.0, 4.0] 85.0 # doctest: +SKIP
"""
return LOOP.run(self._table.delete(predicate))
return LOOP.run(self._table.delete(where))
def update(
self,
@@ -958,7 +1040,12 @@ class RemoteTable(Table):
self._table.update(where=where, updates=values, updates_sql=values_sql)
)
def cleanup_old_versions(self, *_):
def cleanup_old_versions(
self,
older_than: Optional[timedelta] = None,
*,
delete_unverified: bool = False,
):
"""
cleanup_old_versions() is a no-op for remote tables.
"""
@@ -968,7 +1055,7 @@ class RemoteTable(Table):
)
pass
def compact_files(self, *_):
def compact_files(self, *args, **kwargs):
"""
compact_files() is a no-op for remote tables.
"""
@@ -983,12 +1070,14 @@ class RemoteTable(Table):
*,
cleanup_older_than: Optional[timedelta] = None,
delete_unverified: bool = False,
retrain: bool = False,
):
"""optimize() is not supported for remote tables."""
return LOOP.run(
self._table.optimize(
cleanup_older_than=cleanup_older_than,
delete_unverified=delete_unverified,
retrain=retrain,
)
)
@@ -1096,8 +1185,9 @@ class RemoteTable(Table):
"""No-op for remote tables (no local shard writers)."""
return LOOP.run(self._table.close_rte_writers())
def drop_index(self, index_name: str):
return LOOP.run(self._table.drop_index(index_name))
@_deprecated_keyword("index_name", "name")
def drop_index(self, name: str) -> None:
return LOOP.run(self._table.drop_index(name))
def prewarm_index(self, name: str) -> None:
"""Prewarm an index in the table.
@@ -1154,11 +1244,21 @@ class RemoteTable(Table):
"""
return LOOP.run(self._table.uri())
def take_offsets(self, offsets: list[int]) -> LanceTakeQueryBuilder:
return LanceTakeQueryBuilder(self._table.take_offsets(offsets))
def take_offsets(
self, offsets: list[int], *, with_row_id: bool = False
) -> LanceTakeQueryBuilder:
builder = LanceTakeQueryBuilder(self._table.take_offsets(offsets))
if with_row_id:
builder.with_row_id()
return builder
def take_row_ids(self, row_ids: list[int]) -> LanceTakeQueryBuilder:
return LanceTakeQueryBuilder(self._table.take_row_ids(row_ids))
def take_row_ids(
self, row_ids: list[int], *, with_row_id: bool = False
) -> LanceTakeQueryBuilder:
builder = LanceTakeQueryBuilder(self._table.take_row_ids(row_ids))
if with_row_id:
builder.with_row_id()
return builder
def uses_v2_manifest_paths(self) -> bool:
return LOOP.run(self._table.uses_v2_manifest_paths())
@@ -1186,6 +1286,11 @@ class RemoteTable(Table):
) -> "list[Optional[BlobFile]]":
return LOOP.run(self._table.fetch_blob_files(column, row_ids))
def open_blob_file(
self, descriptor: Union[dict, pa.StructScalar, None]
) -> Optional[BlobFile]:
return LOOP.run(self._table.open_blob_file(descriptor))
def head(self, n=5) -> pa.Table:
"""
Return the first `n` rows of the table.
python/python/lancedb/rerankers/openai.py
+3 -4
View File
@@ -21,7 +21,7 @@ class OpenaiReranker(Reranker):
Parameters
----------
model_name : str, default "gpt-4-turbo-preview"
model_name : str, default "gpt-5.6-sol"
The name of the cross encoder model to use.
column : str, default "text"
The name of the column to use as input to the cross encoder model.
@@ -33,7 +33,7 @@ class OpenaiReranker(Reranker):
def __init__(
self,
model_name: str = "gpt-4-turbo-preview",
model_name: str = "gpt-5.6-sol",
column: str = "text",
return_score="relevance",
api_key: Optional[str] = None,
@@ -54,7 +54,6 @@ class OpenaiReranker(Reranker):
response = self._client.chat.completions.create(
model=self.model_name,
response_format={"type": "json_object"},
temperature=0,
messages=[
{
"role": "system",
@@ -132,4 +131,4 @@ class OpenaiReranker(Reranker):
"OPENAI_API_KEY not set. Either set it in your environment or \
pass it as `api_key` argument to the CohereReranker."
)
return openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY") or self.api_key)
return openai.OpenAI(api_key=self.api_key or os.environ.get("OPENAI_API_KEY"))
python/python/lancedb/schema.py
+38 -24
View File
@@ -4,6 +4,7 @@
"""Schema helpers for Lance blob columns."""
import ctypes
import importlib
from typing import TYPE_CHECKING, Optional
@@ -16,6 +17,7 @@ if TYPE_CHECKING:
_BLOB_EXTENSION_NAME = "lance.blob.v2"
_BLOB_V1_KEY = "lance-encoding:blob"
_ARROW_EXT_NAME_KEY = "ARROW:extension:name"
_MAX_RUST_USIZE = ctypes.c_size_t(-1).value
_BLOB_V2_STORAGE_TYPE = pa.struct(
[
pa.field("data", pa.large_binary(), nullable=True),
@@ -209,32 +211,44 @@ def blob(
Maximum size in bytes of a single packed blob sidecar file before
a new one is started. Must be positive.
The thresholds are forwarded to ``lance.blob_field``, which validates
them and stores them as field metadata, so pylance is required whenever
any threshold is set. When all thresholds are omitted the behavior is
unchanged.
Thresholds are validated and stored as Arrow field metadata, with the
same format as ``lance.blob_field``. They do not require pylance.
"""
if (
inline_size_threshold is None
and dedicated_size_threshold is None
and pack_file_size_threshold is None
metadata = {}
for threshold, value, allow_zero, key in (
(
"inline_size_threshold",
inline_size_threshold,
True,
"lance-encoding:blob-inline-size-threshold",
),
(
"dedicated_size_threshold",
dedicated_size_threshold,
False,
"lance-encoding:blob-dedicated-size-threshold",
),
(
"pack_file_size_threshold",
pack_file_size_threshold,
False,
"lance-encoding:blob-pack-file-size-threshold",
),
):
blob_type = _resolve_blob_type()
return pa.field(name, blob_type(), nullable=nullable)
try:
lance = importlib.import_module("lance")
except ModuleNotFoundError as err:
raise ValueError(
"Setting blob tier thresholds requires pylance to be installed"
) from err
_resolve_blob_type()
return lance.blob_field(
name,
nullable=nullable,
inline_size_threshold=inline_size_threshold,
dedicated_size_threshold=dedicated_size_threshold,
pack_file_size_threshold=pack_file_size_threshold,
)
if value is None:
continue
if isinstance(value, bool) or not isinstance(value, int):
raise TypeError(f"{threshold} must be an int, got {type(value).__name__}")
if value < 0 or (value == 0 and not allow_zero):
bound = "non-negative" if allow_zero else "positive"
raise ValueError(f"{threshold} must be {bound}")
if value > _MAX_RUST_USIZE:
raise OverflowError(f"{threshold} must fit in a Rust usize")
metadata[key] = str(value)
blob_type = _resolve_blob_type()
field = pa.field(name, blob_type(), nullable=nullable)
return field.with_metadata(metadata) if metadata else field
def vector(dimension: int, value_type: pa.DataType = pa.float32()) -> pa.DataType:
python/python/lancedb/table.py
+446 -272
View File
File diff suppressed because it is too large Load Diff
python/python/lancedb/util.py
+9
View File
@@ -452,6 +452,15 @@ def _(value: pa.Scalar):
return value_to_sql(value.as_py())
@value_to_sql.register(pa.MapScalar)
def _(value: pa.MapScalar):
if not value.is_valid:
return value_to_sql(None)
keys = value_to_sql(list(value.values.field(0)))
values = value_to_sql(list(value.values.field(1)))
return f"map({keys}, {values})"
@value_to_sql.register(np.ndarray)
def _(value: np.ndarray):
return value_to_sql(value.tolist())
python/python/tests/docs/test_basic.py
+2 -14
View File
@@ -91,19 +91,13 @@ def test_quickstart(tmp_path):
}
)
# --8<-- [end:alter_columns_vector]
# Change it back since we can get a panic with fp16
tbl.alter_columns(
{
"path": "vector",
"data_type": pa.list_(pa.float32(), list_size=2),
}
)
# --8<-- [start:drop_columns]
tbl.drop_columns(["dbl_price"])
# --8<-- [end:drop_columns]
# --8<-- [start:create_index]
tbl.create_index(num_sub_vectors=1)
# --8<-- [end:create_index]
tbl.search([100, 100]).limit(2).to_pandas()
# --8<-- [start:delete_rows]
tbl.delete("item = 'fizz'")
# --8<-- [end:delete_rows]
@@ -185,13 +179,6 @@ async def test_quickstart_async(tmp_path):
}
)
# --8<-- [end:alter_columns_async_vector]
# Change it back since we can get a panic with fp16
await tbl.alter_columns(
{
"path": "vector",
"data_type": pa.list_(pa.float32(), list_size=2),
}
)
# --8<-- [start:drop_columns_async]
await tbl.drop_columns(["dbl_price"])
# --8<-- [end:drop_columns_async]
@@ -200,6 +187,7 @@ async def test_quickstart_async(tmp_path):
# --8<-- [start:create_index_async]
await tbl.create_index("vector")
# --8<-- [end:create_index_async]
await tbl.vector_search([100, 100]).limit(2).to_pandas()
# --8<-- [start:delete_rows_async]
await tbl.delete("item = 'fizz'")
# --8<-- [end:delete_rows_async]
python/python/tests/test_blob.py
+169
View File
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
import ctypes
import io
import subprocess
import sys
@@ -14,6 +15,7 @@ import pytest
from lance.blob import Blob, BlobType as LanceBlobType
import lancedb
from lancedb import schema as schema_module
from lancedb._blob import (
_set_blob_column,
blob_v2_projection_sources,
@@ -139,6 +141,41 @@ def test_blob_without_thresholds_has_no_threshold_metadata():
assert not any(b"threshold" in key for key in metadata)
@pytest.mark.parametrize("nullable", [False, True])
@pytest.mark.parametrize(
"thresholds",
[
{},
{"inline_size_threshold": 0},
{"inline_size_threshold": 8192},
{"dedicated_size_threshold": 1 << 20},
{"pack_file_size_threshold": 1 << 26},
{
"inline_size_threshold": 8192,
"dedicated_size_threshold": 1 << 20,
"pack_file_size_threshold": 1 << 26,
},
{
"inline_size_threshold": ctypes.c_size_t(-1).value,
"dedicated_size_threshold": ctypes.c_size_t(-1).value,
"pack_file_size_threshold": ctypes.c_size_t(-1).value,
},
],
)
def test_blob_thresholds_match_pylance(monkeypatch, nullable, thresholds):
expected = lance.blob_field("payload", nullable=nullable, **thresholds)
actual = lancedb.blob("payload", nullable=nullable, **thresholds)
assert type(actual.type) is LanceBlobType
assert actual.equals(expected, check_metadata=True)
# Both extension classes must encode the same Arrow field on the wire.
monkeypatch.setattr(
schema_module, "_resolve_blob_type", lambda: schema_module._FallbackBlobType
)
fallback = lancedb.blob("payload", nullable=nullable, **thresholds)
assert pa.schema([fallback]).serialize() == pa.schema([expected]).serialize()
def test_blob_thresholds_are_validated():
with pytest.raises(ValueError, match="inline_size_threshold must be non-negative"):
lancedb.blob("payload", inline_size_threshold=-1)
@@ -2659,3 +2696,135 @@ def test_create_table_with_python_blob_dict_and_object():
b"from-dict",
b"from-blob",
]
def _openable_descriptors(table, *, by_address):
hits = (
table.search()
.select(["id", "image", "_rowaddr"])
.with_row_id(True)
.limit(100)
.to_arrow(blob_mode="descriptions")
)
version = table.version
committed_at = next(
item["timestamp"]
for item in table.list_versions()
if item["version"] == version
)
descriptors = {}
for row in hits.to_pylist():
descriptor = row["image"]
if descriptor is not None and descriptor.get("kind") is not None:
descriptor = dict(descriptor)
descriptor["source"] = {
"uri": table.to_lance().uri,
"version": version,
"committed_at_micros": round(committed_at.timestamp() * 1_000_000),
"branch": None,
"column": "image",
"row_reference": row["_rowaddr" if by_address else "_rowid"],
"row_is_address": by_address,
}
else:
descriptor = None
descriptors[row["id"]] = descriptor
return descriptors
@pytest.mark.parametrize("by_address", [False, True])
def test_open_blob_files_preserve_snapshot_order_nulls_and_empty_values(
tmp_path, by_address
):
db = lancedb.connect(str(tmp_path))
schema = pa.schema([pa.field("id", pa.int64()), lancedb.blob("image")])
table = db.create_table("videos", schema=schema)
table.add(
[
{"id": 1, "image": b"alpha"},
{"id": 2, "image": None},
{"id": 3, "image": b"beta"},
{"id": 4, "image": b""},
]
)
descriptors = _openable_descriptors(table, by_address=by_address)
table.add([{"id": 1, "image": b"replacement"}], mode="overwrite")
latest_version = table.version
values = pa.array(
[descriptors[3], descriptors[2], descriptors[1], descriptors[3], descriptors[4]]
)
files = table.open_blob_files(
pa.chunked_array([values.slice(0, 2), values.slice(2)])
)
assert iter(files) is files
actual = []
for file in files:
if file is None:
actual.append(None)
else:
with file:
actual.append(file.read())
assert file.closed
assert actual == [b"beta", None, b"alpha", b"beta", b""]
with table.open_blob_file(descriptors[1]) as file:
file.seek(2)
assert file.read_range(0, 2) == b"al"
assert file.tell() == 2
assert file.read(2) == b"ph"
assert table.version == latest_version
assert table.open_blob_file(pa.scalar(None, type=values.type)) is None
other = db.create_table("other", schema=schema)
with pytest.raises(ValueError, match="different table"):
other.open_blob_file(descriptors[1])
stale = dict(descriptors[1], source=dict(descriptors[1]["source"]))
stale["source"]["committed_at_micros"] += 1
with pytest.raises(ValueError, match="snapshot has been replaced"):
table.open_blob_file(stale)
raw = {key: value for key, value in descriptors[1].items() if key != "source"}
with pytest.raises(ValueError, match="source"):
table.open_blob_file(raw)
with pytest.raises(TypeError, match="descriptor"):
table.open_blob_file("not a descriptor")
def test_open_blob_files_does_not_open_ahead(monkeypatch):
table = _blob_table("lazy_descriptor_iteration", [{"id": 1, "image": b"x"}])
opened = []
def open_file(descriptor):
opened.append(descriptor)
return descriptor
monkeypatch.setattr(table, "open_blob_file", open_file)
files = table.open_blob_files(iter([1, None, 2]))
assert opened == []
assert next(files) == 1
assert opened == [1]
assert list(files) == [None, 2]
assert opened == [1, None, 2]
@pytest.mark.asyncio
async def test_open_blob_files_async_iterates_snapshot_handles(tmp_path):
db = lancedb.connect(str(tmp_path))
table = db.create_table(
"videos",
schema=pa.schema([pa.field("id", pa.int64()), lancedb.blob("image")]),
)
table.add([{"id": 1, "image": b"alpha"}, {"id": 2, "image": None}])
descriptors = _openable_descriptors(table, by_address=True)
async_db = await lancedb.connect_async(str(tmp_path))
videos = await async_db.open_table("videos")
file = await videos.open_blob_file(pa.scalar(descriptors[1]))
with file:
assert file.read(2) == b"al"
actual = []
async for file in videos.open_blob_files(
[descriptors[2], descriptors[1], descriptors[1]]
):
if file is None:
actual.append(None)
else:
with file:
actual.append(await file.aread())
assert actual == [None, b"alpha", b"alpha"]
python/python/tests/test_blob_reader.py
+59
View File
@@ -0,0 +1,59 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
import io
import pytest
from lancedb import BlobReader
def test_reader_is_lazy_and_seekable():
requests = []
size = 64 * 1024 * 1024
def read_range(offset, length):
requests.append((offset, length))
return bytes((index % 251 for index in range(offset, offset + length)))
with BlobReader(read_range, size) as reader:
assert requests == []
assert reader.size() == size
reader.seek(-4, io.SEEK_END)
assert reader.read(10) == bytes(index % 251 for index in range(size - 4, size))
assert reader.read(1) == b""
assert requests == [(size - 4, 4)]
assert reader.read_range(1, 3) == b"\x01\x02\x03"
assert reader.tell() == size
reader.seek(2)
target = bytearray(3)
assert reader.readinto(target) == 3
assert target == b"\x02\x03\x04"
reader.seek(-1, io.SEEK_CUR)
assert reader.tell() == 4
with pytest.raises(ValueError, match="closed"):
reader.read(1)
def test_reader_supports_buffered_io_and_bounds_range_requests():
requests = []
payload = b"x" * (3 * 1024 * 1024 + 7)
def read_range(offset, length):
requests.append((offset, length))
return payload[offset : offset + length]
with io.BufferedReader(BlobReader(read_range, len(payload))) as reader:
assert reader.read() == payload
assert requests
assert all(length <= 1024 * 1024 for _, length in requests)
def test_reader_rejects_invalid_ranges_and_short_reads():
reader = BlobReader(lambda offset, length: b"", 10)
with pytest.raises(ValueError):
reader.seek(-1)
with pytest.raises(ValueError):
reader.read_range(-1, 1)
with pytest.raises(OSError, match="payload length"):
reader.read(1)
python/python/tests/test_embeddings.py
+33
View File
@@ -1039,6 +1039,39 @@ def test_open_clip_opens_percent_encoded_file_uri(tmp_path):
assert image.size == (4, 4)
def test_open_clip_model_in_eval_mode():
"""
open_clip.create_model_and_transforms returns the model in training mode,
which leaves dropout, stochastic depth and batch norm in their training
behaviour. OpenCLIP must switch the model to eval mode before embedding.
"""
from lancedb.embeddings.open_clip import OpenClipEmbeddings
class FakeModel:
training = True
def to(self, device):
return self
def eval(self):
self.training = False
return self
open_clip = MagicMock()
open_clip.create_model_and_transforms.return_value = (FakeModel(), None, None)
with patch(
"lancedb.embeddings.open_clip.attempt_import_or_raise",
return_value=open_clip,
):
func = OpenClipEmbeddings(name="ViTamin-B", pretrained="datacomp1b")
open_clip.create_model_and_transforms.assert_called_once_with(
"ViTamin-B", pretrained="datacomp1b"
)
assert func._model.training is False
def test_jina_generate_image_input_dict_local_path(tmp_path):
"""
JinaEmbeddings._generate_image_input_dict must accept a local image path
python/python/tests/test_errors.py
+21
View File
@@ -3,7 +3,28 @@
import pickle
import pytest
from lancedb.db import DBConnection
from lancedb.remote.errors import HttpError, LanceDBClientError, RetryError
from lancedb.table import Table
@pytest.mark.parametrize(
"base, method",
[
(Table, "to_lance"),
(Table, "to_polars"),
(DBConnection, "drop_all_tables"),
(DBConnection, "drop_database"),
],
)
def test_base_unsupported_methods_have_messages(base, method):
with pytest.raises(NotImplementedError) as exc_info:
getattr(base, method)(None)
message = str(exc_info.value)
assert method in message
assert "not supported" in message
def test_pickle_lancedb_client_error():
python/python/tests/test_first_class_function_slice2.py
+59
View File
@@ -23,6 +23,7 @@ import pyarrow as pa
import pytest
import lancedb
from lancedb import BlobReader
from lancedb.functions import (
PythonRuntimeSpec,
SecretBinding,
@@ -966,6 +967,60 @@ def test_blob_fields_use_the_scalar_function_semantic_type():
assert signature.output.arrow_type == "blob_v2"
def test_blob_input_access_is_inferred_from_annotations():
@udf
def read_blob(video: BlobReader, timestamp_s: float) -> bytes:
video.seek(int(timestamp_s))
return video.read(4)
@udf
def read_bytes(video: bytes) -> int:
return len(video)
reader = read_blob.registration_request.signature
assert reader.inputs[0].arrow_type == "blob_v2"
assert reader.inputs[0].blob_access == "reader"
assert reader.inputs[1].blob_access is None
assert reader.output.arrow_type == "binary"
eager = read_bytes.registration_request.signature.inputs[0]
assert eager.arrow_type == "binary"
assert eager.blob_access == "bytes"
encoded = json.loads(read_blob.registration_request.to_canonical_json())
assert encoded["signature"]["inputs"][0]["blob_access"] == "reader"
def test_nullable_blob_reader_and_explicit_blob_bytes():
@udf
def nullable(video: Optional[BlobReader]) -> int:
return 0 if video is None else video.size()
@udf(
input_schema=pa.schema([lancedb.blob("video", nullable=False)]),
output_schema=pa.field("size", pa.int64(), nullable=False),
)
def eager(video: bytes) -> int:
return len(video)
parameter = nullable.registration_request.signature.inputs[0]
assert parameter.nullable is True
assert parameter.blob_access == "reader"
parameter = eager.registration_request.signature.inputs[0]
assert parameter.arrow_type == "blob_v2"
assert parameter.blob_access == "bytes"
def test_blob_reader_rejects_an_incompatible_explicit_schema():
def read(video: BlobReader) -> int:
return video.size()
with pytest.raises(TypeError, match="requires a Blob V2 field"):
udf(
read,
input_schema=pa.schema([pa.field("video", pa.int64(), nullable=False)]),
output_schema=pa.field("size", pa.int64(), nullable=False),
)
def test_whole_named_struct_function_can_include_a_blob_result_field():
@udf(
input_schema=pa.schema([lancedb.blob("image", nullable=False)]),
@@ -1674,7 +1729,11 @@ def test_remote_registration_job_and_exact_version_reopen_round_trip():
)
registration = db.create_function_async(normalize_score)
assert registration.id == "job-register"
assert registration.job_type is None
assert registration.status() == "finished"
assert registration.job_type == "create_function"
created = registration.wait()
assert registration.job_type == "create_function"
reopened = db.get_function("normalize_score", version=created.version)
assert created == reopened
python/python/tests/test_fts.py
+14
View File
@@ -391,6 +391,20 @@ def test_custom_stop_words_list(table):
)
def test_list_indices_with_missing_fts_files(table):
# Regression test for https://github.com/lancedb/lancedb/issues/4250
table.create_fts_index("text")
table.create_scalar_index("id")
fts = next(i for i in table.list_indices() if i.index_type == "FTS")
shutil.rmtree(Path(table.uri) / "_indices" / fts.index_uuid)
indices = {i.name: i for i in table.list_indices()}
assert set(indices) == {"text_idx", "id_idx"}
assert indices["text_idx"].index_type == "FTS"
assert indices["text_idx"].columns == ["text"]
def test_search_fts(table):
table.create_fts_index("text")
results = table.search("puppy").select(["id", "text"]).limit(5).to_list()
python/python/tests/test_index.py
+12 -2
View File
@@ -421,13 +421,23 @@ async def test_create_4bit_ivfpq_index(some_table: AsyncTable):
@pytest.mark.asyncio
async def test_create_ivfrq_index(some_table: AsyncTable):
await some_table.create_index("vector", config=IvfRq(num_bits=1))
@pytest.mark.parametrize(
"config, expected_bits",
[
pytest.param(IvfRq(), 5, id="default"),
pytest.param(IvfRq(num_bits=None), 5, id="none"),
pytest.param(IvfRq(num_bits=1), 1, id="one-bit"),
pytest.param(IvfRq(num_bits=4), 4, id="multi-bit"),
],
)
async def test_create_ivfrq_index(some_table: AsyncTable, config, expected_bits):
await some_table.create_index("vector", config=config)
indices = await some_table.list_indices()
assert len(indices) == 1
assert indices[0].index_type == "IvfRq"
assert indices[0].columns == ["vector"]
assert indices[0].name == "vector_idx"
assert indices[0].index_details["compression"]["num_bits"] == expected_bits
@pytest.mark.asyncio
python/python/tests/test_listing.py
+137
View File
@@ -88,6 +88,143 @@ async def advance(items, asynchronous):
KINDS = ["secrets", "views", "jobs", "functions", "materialized_views"]
@pytest.mark.asyncio
@pytest.mark.parametrize("kind", ["tables", "namespaces"])
@pytest.mark.parametrize("asynchronous", [False, True])
async def test_compatible_name_listing_pages(listing_server, kind, asynchronous):
endpoint, requests, responses = listing_server
responses.extend(
[
(200, response_page(kind, ["a", "b"], "next")),
(200, response_page(kind, ["c"], None)),
]
)
db = await connection(endpoint, asynchronous)
listing = getattr(db, "list_" + kind)(
namespace_path=["team"], page_token="start", limit=2
)
field = kind
assert len(requests) == (0 if asynchronous else 1)
if asynchronous:
with pytest.raises(RuntimeError, match="Await or iterate"):
getattr(listing, field)
assert listing.page_token == "start"
assert await listing is listing
assert len(requests) == 1
page_names = getattr(listing, field)
assert type(page_names) is list
assert page_names == ["a", "b"]
page_names.append("changed")
assert getattr(listing, field) == ["a", "b"]
assert listing.page_token == "next"
assert listing.num_page_results() == 2
assert await advance(listing, asynchronous) == "a"
assert getattr(listing, field) == ["a", "b"]
assert await advance(listing, asynchronous) == "b"
assert len(requests) == 1
assert await advance(listing, asynchronous) == "c"
assert getattr(listing, field) == ["c"]
assert listing.page_token is None
assert len(requests) == 2
assert ([item async for item in listing] if asynchronous else list(listing)) == []
if asynchronous:
assert await listing is listing
assert len(requests) == 2
@pytest.mark.asyncio
@pytest.mark.parametrize("kind", ["tables", "namespaces"])
async def test_name_listing_skips_empty_page_only_during_iteration(
listing_server, kind
):
endpoint, requests, responses = listing_server
responses.extend(
[
(200, response_page(kind, [], "next")),
(200, response_page(kind, ["a"], None)),
]
)
db = await connection(endpoint, True)
listing = getattr(db, "list_" + kind)()
await listing
assert getattr(listing, kind) == []
assert listing.page_token == "next"
assert len(requests) == 1
assert [name async for name in listing] == ["a"]
assert len(requests) == 2
@pytest.mark.asyncio
@pytest.mark.parametrize("kind", ["tables", "namespaces"])
@pytest.mark.parametrize("asynchronous", [False, True])
async def test_name_listing_zero_limit_uses_no_request(
listing_server, kind, asynchronous
):
endpoint, requests, _ = listing_server
db = await connection(endpoint, asynchronous)
listing = getattr(db, "list_" + kind)(limit=0)
if asynchronous:
assert await listing is listing
assert [name async for name in listing] == []
else:
assert list(listing) == []
assert getattr(listing, kind) == []
assert listing.page_token is None
assert requests == []
@pytest.mark.asyncio
@pytest.mark.parametrize("kind", ["tables", "namespaces"])
async def test_name_listing_failure_retains_token(listing_server, kind):
endpoint, requests, responses = listing_server
responses.append((401, {}))
db = await connection(endpoint, True)
listing = getattr(db, "list_" + kind)(page_token="retry")
with pytest.raises(HttpError):
await listing
assert listing.page_token == "retry"
with pytest.raises(HttpError):
await listing
assert [name async for name in listing] == []
assert len(requests) == 1
@pytest.mark.asyncio
@pytest.mark.parametrize("kind", ["tables", "namespaces"])
async def test_name_listing_direct_async_iteration(listing_server, kind):
endpoint, requests, responses = listing_server
responses.extend(
[
(200, response_page(kind, ["a"], "next")),
(200, response_page(kind, ["b"], None)),
]
)
db = await connection(endpoint, True)
listing = getattr(db, "list_" + kind)(limit=1)
assert requests == []
assert [name async for name in listing] == ["a", "b"]
assert getattr(listing, kind) == ["b"]
assert len(requests) == 2
@pytest.mark.asyncio
@pytest.mark.parametrize("kind", ["tables", "namespaces"])
async def test_name_listing_serializes_concurrent_advances(listing_server, kind):
endpoint, requests, responses = listing_server
responses.append((200, response_page(kind, ["a", "b"], None)))
db = await connection(endpoint, True)
listing = getattr(db, "list_" + kind)()
results = await asyncio.gather(
listing.__anext__(),
listing.__anext__(),
listing.__anext__(),
return_exceptions=True,
)
assert results[:2] == ["a", "b"]
assert isinstance(results[2], StopAsyncIteration)
assert len(requests) == 1
@pytest.mark.asyncio
@pytest.mark.parametrize("kind", KINDS)
@pytest.mark.parametrize("asynchronous", [False, True])
python/python/tests/test_materialized_views.py
+63 -24
View File
@@ -8,6 +8,7 @@ import threading
import lancedb
import pytest
from lancedb.db import AsyncConnection
from lancedb.materialized_view import MaterializedViewDefinition
from lancedb.remote.db import RemoteDBConnection
@@ -56,7 +57,7 @@ def mock_remote_materialized_views():
@contextlib.contextmanager
def mock_remote_materialized_view_create():
def mock_remote_materialized_view_drop():
requests = []
class Handler(http.server.BaseHTTPRequestHandler):
@@ -67,7 +68,8 @@ def mock_remote_materialized_view_create():
length = int(self.headers.get("Content-Length", 0))
body = json.loads(self.rfile.read(length) or b"{}")
requests.append((self.path, body))
job_id = "mv-drop-123" if self.path.endswith("/drop") else "mv-create-123"
assert self.path.endswith("/drop")
job_id = "mv-drop-123"
encoded = json.dumps({"job_id": job_id}).encode()
self.send_response(202)
self.send_header("Content-Type", "application/json")
@@ -98,7 +100,7 @@ def test_remote_list_uses_namespace_route():
def test_remote_drop_async_returns_server_job():
with mock_remote_materialized_view_create() as (host, requests):
with mock_remote_materialized_view_drop() as (host, requests):
db = lancedb.connect(
"db://dev",
api_key="fake",
@@ -192,11 +194,13 @@ def test_create_and_refresh_jobs(tmp_path):
"adults", "people", where="age >= 18", with_no_data=True
)
assert create_job.id is None
assert create_job.job_type is None
assert create_job.wait() is None
view = db.open_materialized_view("adults")
refresh_job = view.refresh_async()
assert refresh_job.id is None
assert refresh_job.job_type is None
result = refresh_job.wait()
assert result.mode == "rebuild"
assert result.rows_written == 2
@@ -337,11 +341,13 @@ async def test_async_create_and_refresh_jobs(tmp_path):
"adults", "people", with_no_data=True
)
assert create_job.id is None
assert create_job.job_type is None
assert await create_job.wait() is None
view = await db.open_materialized_view("adults")
refresh_job = await view.refresh_async()
assert refresh_job.id is None
assert refresh_job.job_type is None
result = await refresh_job.wait()
assert result.mode == "rebuild"
assert result.rows_written == 1
@@ -567,6 +573,27 @@ def test_local_materialized_view_listing_pages_and_resume(tmp_path):
assert list(resumed) == []
class QueryBackedCreateJob:
id = "0198f1b2-c3d4-7e5f-8123-456789abcdef"
job_type = "query"
async def wait(self):
return None
async def status(self):
return "finished"
class CapturingCreateConnection:
def __init__(self):
self.calls = []
async def create_materialized_view_async(self, name, source, **options):
self.calls.append((name, source, options))
return QueryBackedCreateJob()
@pytest.mark.parametrize(
"options",
[
@@ -576,16 +603,28 @@ def test_local_materialized_view_listing_pages_and_resume(tmp_path):
],
)
def test_remote_create_forwards_refresh_options_without_interval_default(options):
with mock_remote_materialized_view_create() as (host, requests):
db = lancedb.connect("db://dev", api_key="fake", host_override=host)
job = db.create_materialized_view_async("adults", "people", **options)
assert job.id == "mv-create-123"
assert requests == [
native = CapturingCreateConnection()
db = RemoteDBConnection.__new__(RemoteDBConnection)
db._conn = AsyncConnection(native)
job = db.create_materialized_view_async("adults", "people", **options)
assert job.id == QueryBackedCreateJob.id
assert job.job_type == "query"
assert not hasattr(job, "query_id")
assert job.wait() is None
assert job.status() == "finished"
assert native.calls == [
(
"/v1/materialized_view/adults/create",
"adults",
"people",
{
"query": 'SELECT * FROM "people"',
**options,
"projections": None,
"filter": None,
"limit": None,
"with_no_data": False,
"auto_refresh_enabled": options.get("auto_refresh_enabled"),
"auto_refresh_interval_seconds": options.get(
"auto_refresh_interval_seconds"
),
},
)
]
@@ -593,18 +632,18 @@ def test_remote_create_forwards_refresh_options_without_interval_default(options
@pytest.mark.asyncio
async def test_async_remote_create_keeps_interval_unspecified():
with mock_remote_materialized_view_create() as (host, requests):
db = await lancedb.connect_async("db://dev", api_key="fake", host_override=host)
job = await db.create_materialized_view_async(
"adults",
"people",
auto_refresh_enabled=True,
)
assert job.id == "mv-create-123"
assert requests[0][1] == {
"query": 'SELECT * FROM "people"',
"auto_refresh_enabled": True,
}
native = CapturingCreateConnection()
db = AsyncConnection(native)
job = await db.create_materialized_view_async(
"adults", "people", auto_refresh_enabled=True
)
assert job.id == QueryBackedCreateJob.id
assert job.job_type == "query"
assert not hasattr(job, "query_id")
assert await job.wait() is None
assert await job.status() == "finished"
assert native.calls[0][2]["auto_refresh_enabled"] is True
assert native.calls[0][2]["auto_refresh_interval_seconds"] is None
@pytest.mark.parametrize(
@@ -625,7 +664,7 @@ def test_embedded_create_refuses_automatic_refresh_before_creating_view(
@pytest.mark.parametrize("seconds", [0, -1, float("nan"), float("inf")])
def test_remote_create_rejects_invalid_interval_without_request(seconds):
with mock_remote_materialized_view_create() as (host, requests):
with mock_remote_materialized_view_drop() as (host, requests):
db = lancedb.connect("db://dev", api_key="fake", host_override=host)
with pytest.raises(ValueError, match="positive, finite"):
db.create_materialized_view_async(
python/python/tests/test_ollama_embeddings.py
+1 -1
View File
@@ -3,7 +3,6 @@
import json
import httpx
import numpy as np
import pyarrow as pa
import pytest
@@ -14,6 +13,7 @@ from lancedb.embeddings import get_registry
@pytest.fixture
def ollama_embeddings():
httpx = pytest.importorskip("httpx")
pytest.importorskip("ollama")
requests = []
python/python/tests/test_openai_reranker_credentials.py
+38
View File
@@ -0,0 +1,38 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
import pytest
from lancedb.rerankers import OpenaiReranker
@pytest.mark.parametrize(
"explicit_key, environment_key, expected_key",
[
("explicit-test-key", "environment-test-key", "explicit-test-key"),
("explicit-test-key", None, "explicit-test-key"),
(None, "environment-test-key", "environment-test-key"),
],
)
def test_openai_reranker_api_key_precedence(
monkeypatch, explicit_key, environment_key, expected_key
):
pytest.importorskip("openai")
if environment_key is None:
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
else:
monkeypatch.setenv("OPENAI_API_KEY", environment_key)
reranker = OpenaiReranker(api_key=explicit_key)
client = reranker._client
try:
assert client.api_key == expected_key
finally:
client.close()
def test_openai_reranker_requires_api_key(monkeypatch):
pytest.importorskip("openai")
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
with pytest.raises(ValueError, match="OPENAI_API_KEY not set"):
OpenaiReranker()._client
python/python/tests/test_permutation.py
+7
View File
@@ -811,6 +811,13 @@ def test_rename_column(some_permutation: Permutation):
some_permutation.rename_column("non_existent", "new_value")
def test_rename_column_keeps_position(some_permutation: Permutation):
renamed = some_permutation.rename_column("id", "new_id")
assert renamed.schema == pa.schema([("new_id", pa.int64()), ("value", pa.int64())])
batch = renamed.with_format("arrow").__getitems__([0])
assert batch.schema.names == ["new_id", "value"]
def test_rename_columns(some_permutation: Permutation):
assert some_permutation.rename_columns({"value": "new_value"}).schema == pa.schema(
[("id", pa.int64()), ("new_value", pa.int64())]
python/python/tests/test_query.py
+27
View File
@@ -2324,6 +2324,33 @@ def test_take_queries(tmp_path):
]
def test_take_queries_with_row_id(tmp_path):
db = lancedb.connect(tmp_path)
table = db.create_table("test", pa.table({"idx": range(100)}))
by_row_id = table.take_row_ids([5, 2, 17], with_row_id=True).to_arrow()
assert "_rowid" in by_row_id.column_names
by_offset = table.take_offsets([5, 2, 17], with_row_id=True).to_arrow()
assert "_rowid" in by_offset.column_names
# The column is opt-in and must stay out of the default projection.
assert "_rowid" not in table.take_row_ids([5, 2, 17]).to_arrow().column_names
@pytest.mark.asyncio
async def test_take_queries_with_row_id_async(tmp_path):
db = await lancedb.connect_async(tmp_path)
table = await db.create_table("test", pa.table({"idx": range(100)}))
by_row_id = await table.take_row_ids([5, 2, 17], with_row_id=True).to_arrow()
assert "_rowid" in by_row_id.column_names
by_offset = await table.take_offsets([5, 2, 17], with_row_id=True).to_arrow()
assert "_rowid" in by_offset.column_names
default = await table.take_row_ids([5, 2, 17]).to_arrow()
assert "_rowid" not in default.column_names
def test_take_queries_to_batches(tmp_path):
# Regression test for the sync take-query path: `to_batches` previously
# raised ``AttributeError: 'AsyncTakeQuery' object has no attribute
python/python/tests/test_remote_db.py
+217 -19
View File
@@ -5,6 +5,7 @@ import contextlib
from datetime import timedelta
import http.server
import io
import inspect
import json
import multiprocessing as mp
import pickle
@@ -765,6 +766,15 @@ def test_create_table_embedding_functions(with_schema):
]
def test_create_table_positional_parameters_match_base_contract():
from lancedb.db import DBConnection
from lancedb.remote.db import RemoteDBConnection
expected = list(inspect.signature(DBConnection.create_table).parameters)[:9]
actual = list(inspect.signature(RemoteDBConnection.create_table).parameters)[:9]
assert actual == expected
def test_create_table_exist_ok():
def handler(request):
if request.path == "/v1/table/test/create/?mode=exist_ok":
@@ -857,10 +867,116 @@ def test_table_unimplemented_functions():
with mock_lancedb_connection(handler) as db:
table = db.create_table("test", [{"id": 1}])
with pytest.raises(NotImplementedError):
table.to_arrow()
with pytest.raises(NotImplementedError):
table.to_pandas()
for method in ["to_arrow", "to_pandas", "to_polars"]:
with pytest.raises(NotImplementedError) as exc_info:
getattr(table, method)()
message = str(exc_info.value)
assert method in message
assert "not" in message and "supported" in message
assert "remote tables" in message
assert "cloud" not in message.lower()
assert f"search().{method}()" in message
@pytest.mark.parametrize("method", ["to_arrow", "to_pandas"])
@pytest.mark.parametrize("blob_mode", ["lazy", "descriptions", "bytes", "invalid"])
def test_remote_conversion_blob_mode_errors(method, blob_mode):
with blob_remote_table() as table:
if blob_mode == "invalid":
with pytest.raises(ValueError, match="blob_mode must be one of"):
getattr(table, method)(blob_mode=blob_mode)
else:
with pytest.raises(NotImplementedError) as error:
getattr(table, method)(blob_mode=blob_mode)
assert f"search().{method}()" in str(error.value)
@pytest.mark.asyncio
@pytest.mark.parametrize("version", [None, 1])
@pytest.mark.parametrize("expose_location", [False, True])
async def test_remote_to_lance_rejection(
tmp_path, version, expose_location, monkeypatch
):
import lance
location = tmp_path / "test.lance"
lance.write_dataset(pa.table({"id": [1]}), location)
lance.write_dataset(pa.table({"id": [2]}), location, mode="append")
requests = []
def fail_dataset(*args, **kwargs):
raise AssertionError("remote storage must not be opened")
monkeypatch.setattr(lance, "dataset", fail_dataset)
def handler(request):
requests.append(request.path)
assert request.path == "/v1/table/test/describe/"
content_len = int(request.headers.get("Content-Length", 0))
body = json.loads(request.rfile.read(content_len)) if content_len else {}
request.send_response(200)
request.send_header("Content-Type", "application/json")
request.end_headers()
response = {
"version": body.get("version") or 2,
"schema": {"fields": []},
}
if expose_location:
response["location"] = str(location)
request.wfile.write(json.dumps(response).encode())
options = {"default_scan_options": {"with_row_id": True}}
with mock_lancedb_connection(handler) as db:
table = db.open_table("test", version=version)
opened_requests = list(requests)
with pytest.raises(NotImplementedError) as sync_error:
table.to_lance(**options)
assert requests == opened_requests
async with mock_lancedb_connection_async(handler) as db:
table = await db.open_table("test", version=version)
opened_requests = list(requests)
with pytest.raises(NotImplementedError) as async_error:
await table.to_lance(**options)
assert requests == opened_requests
assert "to_lance() is not supported for remote tables" in str(sync_error.value)
assert "query the server" in str(sync_error.value)
assert str(sync_error.value) == str(async_error.value)
@pytest.mark.asyncio
@pytest.mark.parametrize(
"method, kwargs",
[
("drop_all_tables", {}),
("drop_all_tables", {"namespace_path": []}),
("drop_all_tables", {"namespace_path": ["team"]}),
("drop_database", {}),
],
)
async def test_remote_drop_all_tables_not_supported(method, kwargs):
requests = []
def handler(request):
requests.append(request.path)
request.send_response(500)
request.end_headers()
with mock_lancedb_connection(handler) as db:
with pytest.raises(NotImplementedError) as sync_error:
getattr(db, method)(**kwargs)
async with mock_lancedb_connection_async(handler) as db:
with pytest.raises(NotImplementedError) as async_error:
await getattr(db, method)(**kwargs)
assert "Dropping all tables is not currently supported in the remote API" in str(
sync_error.value
)
assert "drop_table" in str(sync_error.value)
assert str(sync_error.value) == str(async_error.value)
assert requests == []
def test_table_to_pandas_not_supported():
@@ -1039,8 +1155,9 @@ def test_table_create_indices():
# Test create_fts_index with custom name (legacy method)
with pytest.warns(DeprecationWarning, match="create_fts_index"):
table.create_fts_index(
"text",
field_names="text",
wait_timeout=timedelta(seconds=2),
tokenizer_name="en_stem",
block_size=256,
custom_stop_words=["cloud"],
name="custom_fts_idx",
@@ -1049,6 +1166,8 @@ def test_table_create_indices():
# Test create_index with custom name (legacy form: vector_column_name kwarg)
with pytest.warns(DeprecationWarning, match="create_index"):
table.create_index(
"cosine",
2,
vector_column_name="vector",
wait_timeout=timedelta(seconds=10),
name="custom_vector_idx",
@@ -1061,7 +1180,8 @@ def test_table_create_indices():
scalar_req = received_requests[0]
assert "name" in scalar_req
assert scalar_req["name"] == "custom_scalar_idx"
assert scalar_req["replace"] is False
# The wire protocol omits replace=True (the server default).
assert "replace" not in scalar_req
# Check FTS index request has custom name
fts_req = received_requests[1]
@@ -1070,18 +1190,39 @@ def test_table_create_indices():
assert fts_req["replace"] is False
assert fts_req["block_size"] == 256
assert fts_req["custom_stop_words"] == ["cloud"]
assert fts_req["language"] == "English"
assert fts_req["stem"] is True
# Check vector index request has custom name
vector_req = received_requests[2]
assert "name" in vector_req
assert vector_req["name"] == "custom_vector_idx"
assert "replace" not in vector_req
assert vector_req["metric_type"] == "cosine"
assert vector_req["num_partitions"] == 2
with pytest.warns(DeprecationWarning, match="create_index"):
table.create_index(
index_type="IVF_HNSW_SQ",
num_partitions=1,
m=10,
ef_construction=50,
max_iterations=12,
sample_rate=64,
)
tuned_req = received_requests[-1]
assert tuned_req["index_type"] == "IVF_HNSW_SQ"
assert tuned_req["num_partitions"] == 1
assert tuned_req["m"] == 10
assert tuned_req["ef_construction"] == 50
assert tuned_req["max_iterations"] == 12
assert tuned_req["sample_rate"] == 64
table.wait_for_index(["custom_scalar_idx"], timedelta(seconds=2))
table.wait_for_index(
["custom_fts_idx", "custom_vector_idx"], timedelta(seconds=2)
)
table.drop_index("custom_vector_idx")
table.drop_index(name="custom_vector_idx")
table.drop_index("custom_scalar_idx")
table.drop_index("custom_fts_idx")
@@ -1477,7 +1618,10 @@ def test_remote_json_fts_config(server_version):
assert request["block_size"] == 256
def test_remote_create_index_new_api(caplog):
@pytest.mark.parametrize(
"rq_kwargs", [{}, {"num_bits": None}, {"num_bits": 1}, {"num_bits": 4}]
)
def test_remote_create_index_new_api(caplog, rq_kwargs):
received_requests = []
def handler(request):
@@ -1550,7 +1694,7 @@ def test_remote_create_index_new_api(caplog):
table.create_index("category", config=BTree())
table.create_index("text", config=FTS(block_size=256))
# IvfRq via new API
table.create_index("vector", config=IvfRq(distance_type="l2"))
table.create_index("vector", config=IvfRq(distance_type="l2", **rq_kwargs))
table.create_index(
"vector", config=IvfPq(distance_type="l2"), replace=False
)
@@ -1575,6 +1719,10 @@ def test_remote_create_index_new_api(caplog):
"vector",
]
assert received_requests[2]["block_size"] == 256
if rq_kwargs.get("num_bits") is None:
assert "num_bits" not in received_requests[3]
else:
assert received_requests[3]["num_bits"] == rq_kwargs["num_bits"]
assert received_requests[4]["replace"] is False
assert received_requests[5]["replace"] is False
assert "replace is not supported" not in caplog.text
@@ -1675,16 +1823,19 @@ def test_remote_unsupported_legacy_hnsw_pq():
)
def test_remote_python_only_messages(caplog):
def test_remote_python_only_messages():
with mock_lancedb_connection(remote_unsupported_handler) as db:
with caplog.at_level("INFO"):
table = db.open_table("test", storage_options={}, index_cache_size=1)
assert "storage_options is ignored for remote tables" in caplog.text
assert "index_cache_size is ignored for remote tables" in caplog.text
for method in ["to_arrow", "to_pandas"]:
for options in [{"storage_options": {}}, {"index_cache_size": 1}]:
with pytest.raises(NotImplementedError, match=next(iter(options))):
db.open_table("test", **options)
table = db.open_table("test")
for method in ["to_arrow", "to_pandas", "to_polars"]:
with pytest.raises(NotImplementedError) as error:
getattr(table, method)()
assert str(error.value) == f"{method}() is not supported for remote tables."
assert str(error.value) == (
f"{method}() is not supported for remote tables. "
f"Use table.search().{method}() instead."
)
for method in ["compact_files", "cleanup_old_versions"]:
with pytest.warns(UserWarning) as warning:
getattr(table, method)()
@@ -2225,6 +2376,23 @@ def test_head():
assert data == pa.table({"id": [1, 2, 3]})
def test_remote_take_row_ids_with_row_id():
bodies = []
def handler(body):
bodies.append(body)
return pa.table({"id": [1, 2], "_rowid": pa.array([0, 1], pa.uint64())})
with query_test_table(handler) as table:
result = table.take_row_ids([0, 1], with_row_id=True).to_arrow()
assert "_rowid" in result.column_names
assert bodies[-1]["with_row_id"] is True
with query_test_table(handler) as table:
table.take_row_ids([0, 1]).to_arrow()
assert "with_row_id" not in bodies[-1]
def test_query_sync_minimal():
def handler(body):
assert body == {
@@ -3473,14 +3641,17 @@ def blob_descriptor_take_table():
@contextlib.contextmanager
def blob_remote_table(*, server_version=Version("0.5.0")):
def blob_remote_table(*, server_version=Version("0.5.0"), location=None):
def handler(request):
if request.path == "/v1/table/test/describe/":
request.send_response(200)
request.send_header("Content-Type", "application/json")
request.send_header("phalanx-version", str(server_version))
request.end_headers()
request.wfile.write(json.dumps(BLOB_DESCRIBE_RESPONSE).encode())
response = dict(BLOB_DESCRIBE_RESPONSE)
if location is not None:
response["location"] = str(location)
request.wfile.write(json.dumps(response).encode())
elif request.path.startswith("/v1/table/test/blob/image/"):
path = request.path.partition("?")[0]
row_id = int(path.split("/")[-2])
@@ -3515,7 +3686,6 @@ def blob_remote_table(*, server_version=Version("0.5.0")):
if columns == ["id"]:
response_table = blob_query_response_table().select(["id"])
else:
assert body["with_row_id"] is True
if columns == ["image"]:
# Blob handles are sized from a descriptor take.
assert body["filter"].startswith("_rowid IN")
@@ -3528,6 +3698,8 @@ def blob_remote_table(*, server_version=Version("0.5.0")):
)
else:
response_table = blob_query_response_table()
if not body.get("with_row_id", False):
response_table = response_table.drop(["_rowid"])
request.send_response(200)
request.send_header("Content-Type", "application/vnd.apache.arrow.file")
request.send_header("x-lancedb-version", "1")
@@ -3562,6 +3734,32 @@ def test_remote_blob_columns_and_fetch():
assert blobs.to_pylist() == [b"alpha", None, b"gamma"]
@pytest.mark.parametrize("blob_mode", ["descriptions", "bytes", "lazy"])
def test_sync_remote_blob_pandas_avoids_storage_access(
tmp_path, monkeypatch, blob_mode
):
import lance
location = tmp_path / "remote.lance"
lance.write_dataset(pa.table({"id": [7], "image": [b"server-owned"]}), location)
def fail_dataset(*args, **kwargs):
raise AssertionError("remote storage must not be opened")
monkeypatch.setattr(lance, "dataset", fail_dataset)
with blob_remote_table(location=location) as table:
monkeypatch.setattr(table, "to_lance", fail_dataset)
query = table.search().select(["id", "image"])
if blob_mode == "descriptions":
df = query.to_pandas(blob_mode=blob_mode)
assert df["id"].tolist() == [1, 2, 3]
assert "_rowid" not in df.columns
assert df["image"][0]["size"] == 5
else:
with pytest.raises(NotImplementedError, match="stable table snapshot"):
query.to_pandas(blob_mode=blob_mode)
@pytest.mark.asyncio
@pytest.mark.parametrize("use_async", [False, True])
async def test_remote_fetch_blob_ranges(use_async):
python/python/tests/test_remote_options.py
+154
View File
@@ -0,0 +1,154 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
import inspect
from urllib.parse import parse_qs, urlparse
import lancedb
import pyarrow as pa
import pytest
from test_remote_db import mock_lancedb_connection, mock_lancedb_connection_async
@pytest.mark.parametrize("uri", ["memory://", "db://dev"])
def test_unknown_connect_options(uri):
with pytest.raises(ValueError, match="Unknown keyword arguments.*bogus"):
lancedb.connect(uri, api_key="fake", bogus=1)
@pytest.mark.parametrize("option", ["session", "manifest_enabled"])
@pytest.mark.parametrize("asynchronous", [False, True])
@pytest.mark.asyncio
async def test_remote_connect_rejects_local_options(option, asynchronous):
value = lancedb.Session() if option == "session" else True
with pytest.raises(NotImplementedError, match=option):
if asynchronous:
await lancedb.connect_async("db://dev", api_key="fake", **{option: value})
else:
lancedb.connect("db://dev", api_key="fake", **{option: value})
def test_remote_create_signature():
from lancedb.db import DBConnection
from lancedb.remote.db import RemoteDBConnection
expected = inspect.signature(DBConnection.create_table).parameters
actual = inspect.signature(RemoteDBConnection.create_table).parameters
assert list(actual) == list(expected)
for name in expected:
assert actual[name].kind == expected[name].kind
if name == "mode":
# Remote connections retain their existing server-default mode.
assert actual[name].default is None
else:
assert actual[name].default == expected[name].default
def test_remote_create_positional_mode():
modes = []
def handler(request):
assert urlparse(request.path).path == "/v1/table/ct_pos/create/"
modes.append(parse_qs(urlparse(request.path).query)["mode"])
request.send_response(200)
request.end_headers()
with mock_lancedb_connection(handler) as db:
data = [{"value": 1}]
db.create_table("ct_pos", data, None, "overwrite")
db.create_table("ct_pos", data, None, "overwrite")
db.create_table("ct_pos", data, None, "create", True)
with pytest.raises(ValueError, match="Invalid mode"):
db.create_table("ct_pos", data, None, "drop")
assert modes == [["overwrite"], ["overwrite"], ["exist_ok"]]
CREATE_OPTIONS = [
{"new_table_data_storage_version": "2.0"},
{"new_table_enable_v2_manifest_paths": "false"},
{"bogus_option_xyz": "1"},
]
@pytest.mark.parametrize("storage_options", CREATE_OPTIONS)
@pytest.mark.parametrize("empty", [False, True])
@pytest.mark.asyncio
async def test_remote_create_rejects_write_options(storage_options, empty):
requests = []
def handler(request):
requests.append(request.path)
request.send_response(200)
request.end_headers()
kwargs = {"schema": pa.schema([("value", pa.int64())])}
if not empty:
kwargs["data"] = [{"value": 1}]
async with mock_lancedb_connection_async(handler) as db:
with pytest.raises(NotImplementedError, match="write_options.*remote"):
await db.create_table("t", storage_options=storage_options, **kwargs)
assert requests == []
@pytest.mark.parametrize(
"options",
[
{"data_storage_version": "2.0"},
{"enable_v2_manifest_paths": False},
{"storage_options": {"x": "1"}},
],
)
def test_sync_remote_create_rejects_write_options(options):
requests = []
def handler(request):
requests.append(request.path)
request.send_response(200)
request.end_headers()
with mock_lancedb_connection(handler) as db:
with pytest.raises(NotImplementedError, match="remote"):
db.create_table("t", [{"value": 1}], **options)
assert requests == []
@pytest.mark.parametrize("asynchronous", [False, True])
@pytest.mark.parametrize(
"options", [{"index_cache_size": 10}, {"storage_options": {"x": "y"}}]
)
@pytest.mark.asyncio
async def test_remote_open_rejects_local_options(options, asynchronous):
requests = []
def handler(request):
requests.append(request.path)
request.send_response(200)
request.end_headers()
request.wfile.write(b'{"version": 1, "schema": {"fields": []}}')
if asynchronous:
async with mock_lancedb_connection_async(handler) as db:
with pytest.raises(NotImplementedError, match="remote"):
await db.open_table("t", **options)
else:
with mock_lancedb_connection(handler) as db:
with pytest.raises(NotImplementedError, match="remote"):
db.open_table("t", **options)
assert requests == []
@pytest.mark.asyncio
async def test_remote_open_rejects_location():
requests = []
def handler(request):
requests.append(request.path)
request.send_response(200)
request.end_headers()
request.wfile.write(b'{"version": 1, "schema": {"fields": []}}')
async with mock_lancedb_connection_async(handler) as db:
with pytest.raises(NotImplementedError, match="location.*remote"):
await db.open_table("t", location="/tmp/zzz")
assert requests == []
python/python/tests/test_rerankers.py
+19
View File
@@ -2,6 +2,7 @@
# SPDX-FileCopyrightText: Copyright The LanceDB Authors
import os
from unittest.mock import MagicMock
import random
import lancedb
@@ -547,6 +548,24 @@ def test_openai_reranker(tmp_path):
_run_test_reranker(reranker, table, "single player experience", None, schema)
def test_openai_reranker_does_not_send_temperature():
reranker = OpenaiReranker(api_key="test-key")
create = MagicMock()
create.return_value.choices = [
MagicMock(
message=MagicMock(
content='{"documents": [{"content": "a", "relevance_score": 0.5}]}'
)
)
]
reranker.__dict__["_client"] = MagicMock()
reranker.__dict__["_client"].chat.completions.create = create
reranker._rerank(pa.table({"text": ["a"]}), "query")
assert "temperature" not in create.call_args.kwargs
@pytest.mark.skipif(
os.environ.get("JINA_API_KEY") is None, reason="JINA_API_KEY not set"
)

Some files were not shown because too many files have changed in this diff Show More