Compare commits

..

11 Commits

Author SHA1 Message Date
Jack Ye d940b9ade7 commit 2026-04-12 12:58:14 -07:00
Jack Ye d059447feb refactor: rename deserialize to deserialize_conn, consistent pushdown naming
- Rename deserialize() -> deserialize_conn() for clarity
- Rename internal _pushdown_operations -> _namespace_client_pushdown_operations
  for consistency with the parameter name
- Rename serialized key "pushdown_operations" ->
  "namespace_client_pushdown_operations"

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:58:14 -07:00
Jack Ye 6427024bcb fix: route list_namespaces through _namespace_conn for consistency
list_namespaces with empty path was going through Rust (ListingDatabase)
which doesn't see namespaces created via the directory namespace client.
Always delegate to _namespace_conn() so create/list are consistent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:58:14 -07:00
Jack Ye be19a880a9 fix: format, lint, and update tests for namespace delegation
- Run ruff format on all changed files
- Fix F821 forward reference in _namespace_conn return type
- Update test_local_namespace_operations to verify operations succeed
  instead of expecting NotImplementedError (namespace ops now work on
  LanceDBConnection via directory namespace delegation)
- Remove test_local_create_namespace_not_supported and
  test_local_drop_namespace_not_supported (no longer applicable)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:58:14 -07:00
Jack Ye e7ed3d5dab fix: merge user storage_options in server-side create_table
_create_table_server_side was only passing self.storage_options
(connection-level) to CreateTableRequest, ignoring the user-provided
storage_options parameter. This caused per-table options like
new_table_data_storage_version to be silently dropped.

Fix both sync and async paths to merge user options on top of
connection options.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:58:14 -07:00
Jack Ye 7ed4b059a6 refactor: rename serialize_to_json/from_serialized_json to serialize/deserialize
Simpler names, docs no longer reference JSON as the serialization format.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:58:14 -07:00
Jack Ye f36583d0c3 fix: keep original Rust path for root namespace operations
Only delegate to _namespace_conn() when namespace_path is non-empty.
Root namespace operations (list_namespaces, list_tables with empty
path) still go through the original Rust connection to avoid regression.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:58:14 -07:00
Jack Ye fea2ef6a0a refactor: generalize worker property overrides with _lancedb_worker_ prefix
Replace the special-cased worker_uri key with a generic mechanism:
any namespace_client_properties key starting with _lancedb_worker_
has the prefix stripped and overrides the corresponding property
when for_worker=True.

e.g. _lancedb_worker_uri overrides uri in worker context.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:58:14 -07:00
Jack Ye fc2a9726b2 refactor: delegate child namespace ops to LanceNamespaceDBConnection
Instead of reimplementing namespace logic (describe_table, merge
storage_options, etc.) in LanceDBConnection, delegate child namespace
operations to a LanceNamespaceDBConnection via _namespace_conn().

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:58:14 -07:00
Jack Ye e3893dacf8 feat: cache namespace_client and auto-delegate child namespace operations
LanceDBConnection now:
- Caches namespace_client() result to avoid repeated DirectoryNamespace builds
- Auto-delegates open_table/create_table with non-empty namespace_path
  through the directory namespace client
- Routes create_namespace/drop_namespace/describe_namespace/list_namespaces
  through the namespace client
- Routes list_tables/drop_table for child namespaces through namespace client

This enables local storage connections to transparently handle child
namespaces like ["__system"] without requiring a separate
LanceNamespaceDBConnection.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:58:14 -07:00
Jack Ye dd2a0ec48f feat: add serialize_to_json/from_serialized_json for DBConnection
Add serialization support to DBConnection classes so connections
can be reconstructed in remote workers without tracking namespace
params separately.

- DBConnection.serialize_to_json() base method
- LanceDBConnection: serializes uri, storage_options, read_consistency_interval
- LanceNamespaceDBConnection: stores namespace_client_impl/properties,
  serializes all connection params including pushdown_operations
- from_serialized_json() factory with for_worker flag for worker_uri swap
- connect_namespace() now passes impl/properties to connection for serialization

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:58:14 -07:00
38 changed files with 123 additions and 485 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
[tool.bumpversion] [tool.bumpversion]
current_version = "0.28.0-beta.5" current_version = "0.28.0-beta.4"
parse = """(?x) parse = """(?x)
(?P<major>0|[1-9]\\d*)\\. (?P<major>0|[1-9]\\d*)\\.
(?P<minor>0|[1-9]\\d*)\\. (?P<minor>0|[1-9]\\d*)\\.
+1 -1
View File
@@ -18,6 +18,6 @@ body:
label: Link label: Link
description: > description: >
Provide a link to the existing documentation, if applicable. Provide a link to the existing documentation, if applicable.
placeholder: ex. https://docs.lancedb.com/tables/... placeholder: ex. https://lancedb.com/docs/tables/...
validations: validations:
required: false required: false
Generated
+3 -3
View File
@@ -4633,7 +4633,7 @@ dependencies = [
[[package]] [[package]]
name = "lancedb" name = "lancedb"
version = "0.28.0-beta.5" version = "0.28.0-beta.4"
dependencies = [ dependencies = [
"ahash", "ahash",
"anyhow", "anyhow",
@@ -4715,7 +4715,7 @@ dependencies = [
[[package]] [[package]]
name = "lancedb-nodejs" name = "lancedb-nodejs"
version = "0.28.0-beta.5" version = "0.28.0-beta.4"
dependencies = [ dependencies = [
"arrow-array", "arrow-array",
"arrow-buffer", "arrow-buffer",
@@ -4737,7 +4737,7 @@ dependencies = [
[[package]] [[package]]
name = "lancedb-python" name = "lancedb-python"
version = "0.31.0-beta.5" version = "0.31.0-beta.4"
dependencies = [ dependencies = [
"arrow", "arrow",
"async-trait", "async-trait",
+2 -2
View File
@@ -15,7 +15,7 @@
# **The Multimodal AI Lakehouse** # **The Multimodal AI Lakehouse**
[**How to Install** ](#how-to-install) ✦ [**Detailed Documentation**](https://docs.lancedb.com) ✦ [**Tutorials and Recipes**](https://github.com/lancedb/vectordb-recipes/tree/main) ✦ [**Contributors**](#contributors) [**How to Install** ](#how-to-install) ✦ [**Detailed Documentation**](https://lancedb.com/docs) ✦ [**Tutorials and Recipes**](https://github.com/lancedb/vectordb-recipes/tree/main) ✦ [**Contributors**](#contributors)
**The ultimate multimodal data platform for AI/ML applications.** **The ultimate multimodal data platform for AI/ML applications.**
@@ -57,7 +57,7 @@ LanceDB is a central location where developers can build, train and analyze thei
## **How to Install**: ## **How to Install**:
Follow the [Quickstart](https://docs.lancedb.com/quickstart) doc to set up LanceDB locally. Follow the [Quickstart](https://lancedb.com/docs/quickstart/) doc to set up LanceDB locally.
**API & SDK:** We also support Python, Typescript and Rust SDKs **API & SDK:** We also support Python, Typescript and Rust SDKs
+1 -1
View File
@@ -1,6 +1,6 @@
# LanceDB Documentation # LanceDB Documentation
LanceDB docs are available at [docs.lancedb.com](https://docs.lancedb.com). LanceDB docs are available at [lancedb.com/docs](https://lancedb.com/docs).
The SDK docs are built and deployed automatically by [Github Actions](../.github/workflows/docs.yml) The SDK docs are built and deployed automatically by [Github Actions](../.github/workflows/docs.yml)
whenever a commit is pushed to the `main` branch. So it is possible for the docs to show whenever a commit is pushed to the `main` branch. So it is possible for the docs to show
+1 -1
View File
@@ -14,7 +14,7 @@ Add the following dependency to your `pom.xml`:
<dependency> <dependency>
<groupId>com.lancedb</groupId> <groupId>com.lancedb</groupId>
<artifactId>lancedb-core</artifactId> <artifactId>lancedb-core</artifactId>
<version>0.28.0-beta.5</version> <version>0.28.0-beta.4</version>
</dependency> </dependency>
``` ```
+1 -1
View File
@@ -34,7 +34,7 @@ const results = await table.vectorSearch([0.1, 0.3]).limit(20).toArray();
console.log(results); console.log(results);
``` ```
The [quickstart](https://docs.lancedb.com/quickstart/) contains more complete examples. The [quickstart](https://lancedb.com/docs/quickstart/basic-usage/) contains more complete examples.
## Development ## Development
+1 -1
View File
@@ -89,4 +89,4 @@ optional storageOptions: Record<string, string>;
(For LanceDB OSS only): configuration for object storage. (For LanceDB OSS only): configuration for object storage.
The available options are described at https://docs.lancedb.com/storage/ The available options are described at https://lancedb.com/docs/storage/
+1 -1
View File
@@ -97,4 +97,4 @@ Configuration for object storage.
Options already set on the connection will be inherited by the table, Options already set on the connection will be inherited by the table,
but can be overridden here. but can be overridden here.
The available options are described at https://docs.lancedb.com/storage/ The available options are described at https://lancedb.com/docs/storage/
+1 -1
View File
@@ -42,4 +42,4 @@ Configuration for object storage.
Options already set on the connection will be inherited by the table, Options already set on the connection will be inherited by the table,
but can be overridden here. but can be overridden here.
The available options are described at https://docs.lancedb.com/storage/ The available options are described at https://lancedb.com/docs/storage/
+1 -1
View File
@@ -8,7 +8,7 @@
<parent> <parent>
<groupId>com.lancedb</groupId> <groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId> <artifactId>lancedb-parent</artifactId>
<version>0.28.0-beta.5</version> <version>0.28.0-beta.4</version>
<relativePath>../pom.xml</relativePath> <relativePath>../pom.xml</relativePath>
</parent> </parent>
+1 -1
View File
@@ -6,7 +6,7 @@
<groupId>com.lancedb</groupId> <groupId>com.lancedb</groupId>
<artifactId>lancedb-parent</artifactId> <artifactId>lancedb-parent</artifactId>
<version>0.28.0-beta.5</version> <version>0.28.0-beta.4</version>
<packaging>pom</packaging> <packaging>pom</packaging>
<name>${project.artifactId}</name> <name>${project.artifactId}</name>
<description>LanceDB Java SDK Parent POM</description> <description>LanceDB Java SDK Parent POM</description>
+1 -1
View File
@@ -1,7 +1,7 @@
[package] [package]
name = "lancedb-nodejs" name = "lancedb-nodejs"
edition.workspace = true edition.workspace = true
version = "0.28.0-beta.5" version = "0.28.0-beta.4"
license.workspace = true license.workspace = true
description.workspace = true description.workspace = true
repository.workspace = true repository.workspace = true
+1 -1
View File
@@ -30,7 +30,7 @@ const results = await table.vectorSearch([0.1, 0.3]).limit(20).toArray();
console.log(results); console.log(results);
``` ```
The [quickstart](https://docs.lancedb.com/quickstart/) contains more complete examples. The [quickstart](https://lancedb.com/docs/quickstart/basic-usage/) contains more complete examples.
## Development ## Development
+2 -2
View File
@@ -42,7 +42,7 @@ export interface CreateTableOptions {
* Options already set on the connection will be inherited by the table, * Options already set on the connection will be inherited by the table,
* but can be overridden here. * but can be overridden here.
* *
* The available options are described at https://docs.lancedb.com/storage/ * The available options are described at https://lancedb.com/docs/storage/
*/ */
storageOptions?: Record<string, string>; storageOptions?: Record<string, string>;
@@ -78,7 +78,7 @@ export interface OpenTableOptions {
* Options already set on the connection will be inherited by the table, * Options already set on the connection will be inherited by the table,
* but can be overridden here. * but can be overridden here.
* *
* The available options are described at https://docs.lancedb.com/storage/ * The available options are described at https://lancedb.com/docs/storage/
*/ */
storageOptions?: Record<string, string>; storageOptions?: Record<string, string>;
/** /**
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-darwin-arm64", "name": "@lancedb/lancedb-darwin-arm64",
"version": "0.28.0-beta.5", "version": "0.28.0-beta.4",
"os": ["darwin"], "os": ["darwin"],
"cpu": ["arm64"], "cpu": ["arm64"],
"main": "lancedb.darwin-arm64.node", "main": "lancedb.darwin-arm64.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-arm64-gnu", "name": "@lancedb/lancedb-linux-arm64-gnu",
"version": "0.28.0-beta.5", "version": "0.28.0-beta.4",
"os": ["linux"], "os": ["linux"],
"cpu": ["arm64"], "cpu": ["arm64"],
"main": "lancedb.linux-arm64-gnu.node", "main": "lancedb.linux-arm64-gnu.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-arm64-musl", "name": "@lancedb/lancedb-linux-arm64-musl",
"version": "0.28.0-beta.5", "version": "0.28.0-beta.4",
"os": ["linux"], "os": ["linux"],
"cpu": ["arm64"], "cpu": ["arm64"],
"main": "lancedb.linux-arm64-musl.node", "main": "lancedb.linux-arm64-musl.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-x64-gnu", "name": "@lancedb/lancedb-linux-x64-gnu",
"version": "0.28.0-beta.5", "version": "0.28.0-beta.4",
"os": ["linux"], "os": ["linux"],
"cpu": ["x64"], "cpu": ["x64"],
"main": "lancedb.linux-x64-gnu.node", "main": "lancedb.linux-x64-gnu.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-linux-x64-musl", "name": "@lancedb/lancedb-linux-x64-musl",
"version": "0.28.0-beta.5", "version": "0.28.0-beta.4",
"os": ["linux"], "os": ["linux"],
"cpu": ["x64"], "cpu": ["x64"],
"main": "lancedb.linux-x64-musl.node", "main": "lancedb.linux-x64-musl.node",
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-win32-arm64-msvc", "name": "@lancedb/lancedb-win32-arm64-msvc",
"version": "0.28.0-beta.5", "version": "0.28.0-beta.4",
"os": [ "os": [
"win32" "win32"
], ],
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lancedb/lancedb-win32-x64-msvc", "name": "@lancedb/lancedb-win32-x64-msvc",
"version": "0.28.0-beta.5", "version": "0.28.0-beta.4",
"os": ["win32"], "os": ["win32"],
"cpu": ["x64"], "cpu": ["x64"],
"main": "lancedb.win32-x64-msvc.node", "main": "lancedb.win32-x64-msvc.node",
+2 -2
View File
@@ -1,12 +1,12 @@
{ {
"name": "@lancedb/lancedb", "name": "@lancedb/lancedb",
"version": "0.28.0-beta.5", "version": "0.28.0-beta.4",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "@lancedb/lancedb", "name": "@lancedb/lancedb",
"version": "0.28.0-beta.5", "version": "0.28.0-beta.4",
"cpu": [ "cpu": [
"x64", "x64",
"arm64" "arm64"
+1 -1
View File
@@ -11,7 +11,7 @@
"ann" "ann"
], ],
"private": false, "private": false,
"version": "0.28.0-beta.5", "version": "0.28.0-beta.4",
"main": "dist/index.js", "main": "dist/index.js",
"exports": { "exports": {
".": "./dist/index.js", ".": "./dist/index.js",
+1 -1
View File
@@ -35,7 +35,7 @@ pub struct ConnectionOptions {
pub read_consistency_interval: Option<f64>, pub read_consistency_interval: Option<f64>,
/// (For LanceDB OSS only): configuration for object storage. /// (For LanceDB OSS only): configuration for object storage.
/// ///
/// The available options are described at https://docs.lancedb.com/storage/ /// The available options are described at https://lancedb.com/docs/storage/
pub storage_options: Option<HashMap<String, String>>, pub storage_options: Option<HashMap<String, String>>,
/// (For LanceDB OSS only): the session to use for this connection. Holds /// (For LanceDB OSS only): the session to use for this connection. Holds
/// shared caches and other session-specific state. /// shared caches and other session-specific state.
+1 -1
View File
@@ -1,5 +1,5 @@
[tool.bumpversion] [tool.bumpversion]
current_version = "0.31.0-beta.5" current_version = "0.31.0-beta.4"
parse = """(?x) parse = """(?x)
(?P<major>0|[1-9]\\d*)\\. (?P<major>0|[1-9]\\d*)\\.
(?P<minor>0|[1-9]\\d*)\\. (?P<minor>0|[1-9]\\d*)\\.
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "lancedb-python" name = "lancedb-python"
version = "0.31.0-beta.5" version = "0.31.0-beta.4"
edition.workspace = true edition.workspace = true
description = "Python bindings for LanceDB" description = "Python bindings for LanceDB"
license.workspace = true license.workspace = true
+2 -2
View File
@@ -110,7 +110,7 @@ def connect(
default configuration is used. default configuration is used.
storage_options: dict, optional storage_options: dict, optional
Additional options for the storage backend. See available options at Additional options for the storage backend. See available options at
<https://docs.lancedb.com/storage/> <https://lancedb.com/docs/storage/>
session: Session, optional session: Session, optional
(For LanceDB OSS only) (For LanceDB OSS only)
A session to use for this connection. Sessions allow you to configure A session to use for this connection. Sessions allow you to configure
@@ -336,7 +336,7 @@ async def connect_async(
default configuration is used. default configuration is used.
storage_options: dict, optional storage_options: dict, optional
Additional options for the storage backend. See available options at Additional options for the storage backend. See available options at
<https://docs.lancedb.com/storage/> <https://lancedb.com/docs/storage/>
session: Session, optional session: Session, optional
(For LanceDB OSS only) (For LanceDB OSS only)
A session to use for this connection. Sessions allow you to configure A session to use for this connection. Sessions allow you to configure
+1 -1
View File
@@ -96,7 +96,7 @@ def data_to_reader(
f"Unknown data type {type(data)}. " f"Unknown data type {type(data)}. "
"Supported types: list of dicts, pandas DataFrame, polars DataFrame, " "Supported types: list of dicts, pandas DataFrame, polars DataFrame, "
"pyarrow Table/RecordBatch, or Pydantic models. " "pyarrow Table/RecordBatch, or Pydantic models. "
"See https://docs.lancedb.com/tables/ for examples." "See https://lancedb.com/docs/tables/ for examples."
) )
+4 -4
View File
@@ -282,7 +282,7 @@ class DBConnection(EnforceOverrides):
Additional options for the storage backend. Options already set on the Additional options for the storage backend. Options already set on the
connection will be inherited by the table, but can be overridden here. connection will be inherited by the table, but can be overridden here.
See available options at See available options at
<https://docs.lancedb.com/storage/> <https://lancedb.com/docs/storage/>
To enable stable row IDs (row IDs remain stable after compaction, To enable stable row IDs (row IDs remain stable after compaction,
update, delete, and merges), set `new_table_enable_stable_row_ids` update, delete, and merges), set `new_table_enable_stable_row_ids`
@@ -433,7 +433,7 @@ class DBConnection(EnforceOverrides):
Additional options for the storage backend. Options already set on the Additional options for the storage backend. Options already set on the
connection will be inherited by the table, but can be overridden here. connection will be inherited by the table, but can be overridden here.
See available options at See available options at
<https://docs.lancedb.com/storage/> <https://lancedb.com/docs/storage/>
Returns Returns
------- -------
@@ -1434,7 +1434,7 @@ class AsyncConnection(object):
Additional options for the storage backend. Options already set on the Additional options for the storage backend. Options already set on the
connection will be inherited by the table, but can be overridden here. connection will be inherited by the table, but can be overridden here.
See available options at See available options at
<https://docs.lancedb.com/storage/> <https://lancedb.com/docs/storage/>
To enable stable row IDs (row IDs remain stable after compaction, To enable stable row IDs (row IDs remain stable after compaction,
update, delete, and merges), set `new_table_enable_stable_row_ids` update, delete, and merges), set `new_table_enable_stable_row_ids`
@@ -1625,7 +1625,7 @@ class AsyncConnection(object):
Additional options for the storage backend. Options already set on the Additional options for the storage backend. Options already set on the
connection will be inherited by the table, but can be overridden here. connection will be inherited by the table, but can be overridden here.
See available options at See available options at
<https://docs.lancedb.com/storage/> <https://lancedb.com/docs/storage/>
index_cache_size: int, default 256 index_cache_size: int, default 256
**Deprecated**: Use session-level cache configuration instead. **Deprecated**: Use session-level cache configuration instead.
Create a Session with custom cache sizes and pass it to lancedb.connect(). Create a Session with custom cache sizes and pass it to lancedb.connect().
+1 -1
View File
@@ -191,7 +191,7 @@ def _into_pyarrow_reader(
f"Unknown data type {type(data)}. " f"Unknown data type {type(data)}. "
"Supported types: list of dicts, pandas DataFrame, polars DataFrame, " "Supported types: list of dicts, pandas DataFrame, polars DataFrame, "
"pyarrow Table/RecordBatch, or Pydantic models. " "pyarrow Table/RecordBatch, or Pydantic models. "
"See https://docs.lancedb.com/tables/ for examples." "See https://lancedb.com/docs/tables/ for examples."
) )
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "lancedb" name = "lancedb"
version = "0.28.0-beta.5" version = "0.28.0-beta.4"
edition.workspace = true edition.workspace = true
description = "LanceDB: A serverless, low-latency vector database for AI applications" description = "LanceDB: A serverless, low-latency vector database for AI applications"
license.workspace = true license.workspace = true
+7 -94
View File
@@ -171,7 +171,7 @@ impl OpenTableBuilder {
/// Options already set on the connection will be inherited by the table, /// Options already set on the connection will be inherited by the table,
/// but can be overridden here. /// but can be overridden here.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub fn storage_option(mut self, key: impl Into<String>, value: impl Into<String>) -> Self { pub fn storage_option(mut self, key: impl Into<String>, value: impl Into<String>) -> Self {
let store_params = self let store_params = self
.request .request
@@ -188,7 +188,7 @@ impl OpenTableBuilder {
/// Options already set on the connection will be inherited by the table, /// Options already set on the connection will be inherited by the table,
/// but can be overridden here. /// but can be overridden here.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub fn storage_options( pub fn storage_options(
mut self, mut self,
pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>, pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>,
@@ -582,14 +582,6 @@ pub struct ConnectRequest {
/// Database specific options /// Database specific options
pub options: HashMap<String, String>, pub options: HashMap<String, String>,
/// Extra properties for the equivalent namespace client.
///
/// For a local [`ListingDatabase`], these are merged into the backing
/// `DirectoryNamespace` properties. This is useful for namespace-specific
/// settings such as `table_version_tracking_enabled` that are distinct from
/// storage options.
pub namespace_client_properties: HashMap<String, String>,
/// The interval at which to check for updates from other processes. /// The interval at which to check for updates from other processes.
/// ///
/// If None, then consistency is not checked. For performance /// If None, then consistency is not checked. For performance
@@ -629,7 +621,6 @@ impl ConnectBuilder {
client_config: Default::default(), client_config: Default::default(),
read_consistency_interval: None, read_consistency_interval: None,
options: HashMap::new(), options: HashMap::new(),
namespace_client_properties: HashMap::new(),
session: None, session: None,
}, },
embedding_registry: None, embedding_registry: None,
@@ -747,7 +738,7 @@ impl ConnectBuilder {
/// Set an option for the storage layer. /// Set an option for the storage layer.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub fn storage_option(mut self, key: impl Into<String>, value: impl Into<String>) -> Self { pub fn storage_option(mut self, key: impl Into<String>, value: impl Into<String>) -> Self {
self.request.options.insert(key.into(), value.into()); self.request.options.insert(key.into(), value.into());
self self
@@ -755,7 +746,7 @@ impl ConnectBuilder {
/// Set multiple options for the storage layer. /// Set multiple options for the storage layer.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub fn storage_options( pub fn storage_options(
mut self, mut self,
pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>, pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>,
@@ -766,31 +757,6 @@ impl ConnectBuilder {
self self
} }
/// Set an additional property for the equivalent namespace client.
pub fn namespace_client_property(
mut self,
key: impl Into<String>,
value: impl Into<String>,
) -> Self {
self.request
.namespace_client_properties
.insert(key.into(), value.into());
self
}
/// Set multiple additional properties for the equivalent namespace client.
pub fn namespace_client_properties(
mut self,
pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>,
) -> Self {
for (key, value) in pairs {
self.request
.namespace_client_properties
.insert(key.into(), value.into());
}
self
}
/// The interval at which to check for updates from other processes. This /// The interval at which to check for updates from other processes. This
/// only affects LanceDB OSS. /// only affects LanceDB OSS.
/// ///
@@ -927,7 +893,6 @@ pub struct ConnectNamespaceBuilder {
ns_impl: String, ns_impl: String,
properties: HashMap<String, String>, properties: HashMap<String, String>,
storage_options: HashMap<String, String>, storage_options: HashMap<String, String>,
namespace_client_properties: HashMap<String, String>,
read_consistency_interval: Option<std::time::Duration>, read_consistency_interval: Option<std::time::Duration>,
embedding_registry: Option<Arc<dyn EmbeddingRegistry>>, embedding_registry: Option<Arc<dyn EmbeddingRegistry>>,
session: Option<Arc<lance::session::Session>>, session: Option<Arc<lance::session::Session>>,
@@ -940,7 +905,6 @@ impl ConnectNamespaceBuilder {
ns_impl: ns_impl.to_string(), ns_impl: ns_impl.to_string(),
properties, properties,
storage_options: HashMap::new(), storage_options: HashMap::new(),
namespace_client_properties: HashMap::new(),
read_consistency_interval: None, read_consistency_interval: None,
embedding_registry: None, embedding_registry: None,
session: None, session: None,
@@ -950,7 +914,7 @@ impl ConnectNamespaceBuilder {
/// Set an option for the storage layer. /// Set an option for the storage layer.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub fn storage_option(mut self, key: impl Into<String>, value: impl Into<String>) -> Self { pub fn storage_option(mut self, key: impl Into<String>, value: impl Into<String>) -> Self {
self.storage_options.insert(key.into(), value.into()); self.storage_options.insert(key.into(), value.into());
self self
@@ -958,7 +922,7 @@ impl ConnectNamespaceBuilder {
/// Set multiple options for the storage layer. /// Set multiple options for the storage layer.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub fn storage_options( pub fn storage_options(
mut self, mut self,
pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>, pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>,
@@ -969,29 +933,6 @@ impl ConnectNamespaceBuilder {
self self
} }
/// Set an additional namespace client property.
pub fn namespace_client_property(
mut self,
key: impl Into<String>,
value: impl Into<String>,
) -> Self {
self.namespace_client_properties
.insert(key.into(), value.into());
self
}
/// Set multiple additional namespace client properties.
pub fn namespace_client_properties(
mut self,
pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>,
) -> Self {
for (key, value) in pairs {
self.namespace_client_properties
.insert(key.into(), value.into());
}
self
}
/// The interval at which to check for updates from other processes. /// The interval at which to check for updates from other processes.
/// ///
/// If left unset, consistency is not checked. For maximum read /// If left unset, consistency is not checked. For maximum read
@@ -1053,13 +994,10 @@ impl ConnectNamespaceBuilder {
pub async fn execute(self) -> Result<Connection> { pub async fn execute(self) -> Result<Connection> {
use crate::database::namespace::LanceNamespaceDatabase; use crate::database::namespace::LanceNamespaceDatabase;
let mut properties = self.properties;
properties.extend(self.namespace_client_properties);
let internal = Arc::new( let internal = Arc::new(
LanceNamespaceDatabase::connect( LanceNamespaceDatabase::connect(
&self.ns_impl, &self.ns_impl,
properties, self.properties,
self.storage_options, self.storage_options,
self.read_consistency_interval, self.read_consistency_interval,
self.session, self.session,
@@ -1179,31 +1117,6 @@ mod tests {
assert_eq!(db.uri(), relative_uri.to_str().unwrap().to_string()); assert_eq!(db.uri(), relative_uri.to_str().unwrap().to_string());
} }
#[tokio::test]
async fn test_connect_with_namespace_client_properties() {
let tmp_dir = tempdir().unwrap();
let uri = tmp_dir.path().to_str().unwrap();
let db = connect(uri)
.namespace_client_property("table_version_tracking_enabled", "true")
.namespace_client_property("manifest_enabled", "true")
.execute()
.await
.unwrap();
let (ns_impl, properties) = db.namespace_client_config().await.unwrap();
assert_eq!(ns_impl, "dir");
assert_eq!(properties.get("root"), Some(&uri.to_string()));
assert_eq!(
properties.get("table_version_tracking_enabled"),
Some(&"true".to_string())
);
assert_eq!(
properties.get("manifest_enabled"),
Some(&"true".to_string())
);
}
#[tokio::test] #[tokio::test]
async fn test_table_names() { async fn test_table_names() {
let tc = new_test_connection().await.unwrap(); let tc = new_test_connection().await.unwrap();
+2 -2
View File
@@ -55,7 +55,7 @@ impl CreateTableBuilder {
/// Options already set on the connection will be inherited by the table, /// Options already set on the connection will be inherited by the table,
/// but can be overridden here. /// but can be overridden here.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub fn storage_option(mut self, key: impl Into<String>, value: impl Into<String>) -> Self { pub fn storage_option(mut self, key: impl Into<String>, value: impl Into<String>) -> Self {
let store_params = self let store_params = self
.request .request
@@ -73,7 +73,7 @@ impl CreateTableBuilder {
/// Options already set on the connection will be inherited by the table, /// Options already set on the connection will be inherited by the table,
/// but can be overridden here. /// but can be overridden here.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub fn storage_options( pub fn storage_options(
mut self, mut self,
pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>, pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>,
+71 -305
View File
@@ -20,7 +20,6 @@ use snafu::ResultExt;
use crate::connection::ConnectRequest; use crate::connection::ConnectRequest;
use crate::database::ReadConsistency; use crate::database::ReadConsistency;
use crate::database::namespace::LanceNamespaceDatabase;
use crate::error::{CreateDirSnafu, Error, Result}; use crate::error::{CreateDirSnafu, Error, Result};
use crate::io::object_store::MirroringObjectStoreWrapper; use crate::io::object_store::MirroringObjectStoreWrapper;
use crate::table::NativeTable; use crate::table::NativeTable;
@@ -74,7 +73,7 @@ pub struct ListingDatabaseOptions {
/// These are used to create/list tables and they are inherited by all tables /// These are used to create/list tables and they are inherited by all tables
/// opened by this database. /// opened by this database.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub storage_options: HashMap<String, String>, pub storage_options: HashMap<String, String>,
} }
@@ -186,7 +185,7 @@ impl ListingDatabaseOptionsBuilder {
/// Set an option for the storage layer. /// Set an option for the storage layer.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub fn storage_option(mut self, key: impl Into<String>, value: impl Into<String>) -> Self { pub fn storage_option(mut self, key: impl Into<String>, value: impl Into<String>) -> Self {
self.options self.options
.storage_options .storage_options
@@ -196,7 +195,7 @@ impl ListingDatabaseOptionsBuilder {
/// Set multiple options for the storage layer. /// Set multiple options for the storage layer.
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
pub fn storage_options( pub fn storage_options(
mut self, mut self,
pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>, pairs: impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>,
@@ -256,9 +255,6 @@ pub struct ListingDatabase {
// Session for object stores and caching // Session for object stores and caching
session: Arc<lance::session::Session>, session: Arc<lance::session::Session>,
// Namespace-backed database for child namespace operations
namespace_database: Arc<LanceNamespaceDatabase>,
} }
impl std::fmt::Display for ListingDatabase { impl std::fmt::Display for ListingDatabase {
@@ -285,44 +281,6 @@ const MIRRORED_STORE: &str = "mirroredStore";
/// A connection to LanceDB /// A connection to LanceDB
impl ListingDatabase { impl ListingDatabase {
fn build_namespace_client_properties(
uri: &str,
storage_options: &HashMap<String, String>,
namespace_client_properties: HashMap<String, String>,
) -> HashMap<String, String> {
let mut properties = namespace_client_properties;
properties.insert("root".to_string(), uri.to_string());
for (key, value) in storage_options {
properties.insert(format!("storage.{}", key), value.clone());
}
properties
}
async fn connect_namespace_database(
uri: &str,
storage_options: HashMap<String, String>,
namespace_client_properties: HashMap<String, String>,
read_consistency_interval: Option<std::time::Duration>,
session: Arc<lance::session::Session>,
) -> Result<Arc<LanceNamespaceDatabase>> {
let ns_properties = Self::build_namespace_client_properties(
uri,
&storage_options,
namespace_client_properties,
);
Ok(Arc::new(
LanceNamespaceDatabase::connect(
"dir",
ns_properties,
storage_options,
read_consistency_interval,
Some(session),
HashSet::new(),
)
.await?,
))
}
/// Connect to a listing database /// Connect to a listing database
/// ///
/// The URI should be a path to a directory where the tables are stored. /// The URI should be a path to a directory where the tables are stored.
@@ -342,7 +300,6 @@ impl ListingDatabase {
uri, uri,
request.read_consistency_interval, request.read_consistency_interval,
options.new_table_config, options.new_table_config,
request.namespace_client_properties.clone(),
request.session.clone(), request.session.clone(),
) )
.await .await
@@ -430,15 +387,6 @@ impl ListingDatabase {
None => None, None => None,
}; };
let namespace_database = Self::connect_namespace_database(
&table_base_uri,
options.storage_options.clone(),
request.namespace_client_properties.clone(),
request.read_consistency_interval,
session.clone(),
)
.await?;
Ok(Self { Ok(Self {
uri: table_base_uri, uri: table_base_uri,
query_string, query_string,
@@ -450,7 +398,6 @@ impl ListingDatabase {
storage_options_provider: None, storage_options_provider: None,
new_table_config: options.new_table_config, new_table_config: options.new_table_config,
session, session,
namespace_database,
}) })
} }
Err(_) => { Err(_) => {
@@ -458,7 +405,6 @@ impl ListingDatabase {
uri, uri,
request.read_consistency_interval, request.read_consistency_interval,
options.new_table_config, options.new_table_config,
request.namespace_client_properties.clone(),
request.session.clone(), request.session.clone(),
) )
.await .await
@@ -470,7 +416,6 @@ impl ListingDatabase {
path: &str, path: &str,
read_consistency_interval: Option<std::time::Duration>, read_consistency_interval: Option<std::time::Duration>,
new_table_config: NewTableConfig, new_table_config: NewTableConfig,
namespace_client_properties: HashMap<String, String>,
session: Option<Arc<lance::session::Session>>, session: Option<Arc<lance::session::Session>>,
) -> Result<Self> { ) -> Result<Self> {
let session = session.unwrap_or_else(|| Arc::new(lance::session::Session::default())); let session = session.unwrap_or_else(|| Arc::new(lance::session::Session::default()));
@@ -484,15 +429,6 @@ impl ListingDatabase {
Self::try_create_dir(path).context(CreateDirSnafu { path })?; Self::try_create_dir(path).context(CreateDirSnafu { path })?;
} }
let namespace_database = Self::connect_namespace_database(
path,
HashMap::new(),
namespace_client_properties,
read_consistency_interval,
session.clone(),
)
.await?;
Ok(Self { Ok(Self {
uri: path.to_string(), uri: path.to_string(),
query_string: None, query_string: None,
@@ -504,7 +440,6 @@ impl ListingDatabase {
storage_options_provider: None, storage_options_provider: None,
new_table_config, new_table_config,
session, session,
namespace_database,
}) })
} }
@@ -562,10 +497,6 @@ impl ListingDatabase {
Ok(uri) Ok(uri)
} }
fn namespace_database(&self) -> Arc<LanceNamespaceDatabase> {
self.namespace_database.clone()
}
async fn drop_tables(&self, names: Vec<String>) -> Result<()> { async fn drop_tables(&self, names: Vec<String>) -> Result<()> {
let object_store_params = ObjectStoreParams { let object_store_params = ObjectStoreParams {
storage_options_accessor: if self.storage_options.is_empty() { storage_options_accessor: if self.storage_options.is_empty() {
@@ -765,7 +696,16 @@ impl Database for ListingDatabase {
&self, &self,
request: ListNamespacesRequest, request: ListNamespacesRequest,
) -> Result<ListNamespacesResponse> { ) -> Result<ListNamespacesResponse> {
self.namespace_database().list_namespaces(request).await if request.id.as_ref().map(|v| !v.is_empty()).unwrap_or(false) {
return Err(Error::NotSupported {
message: "Namespace operations are not supported for listing database".into(),
});
}
Ok(ListNamespacesResponse {
namespaces: Vec::new(),
page_token: None,
})
} }
fn uri(&self) -> &str { fn uri(&self) -> &str {
@@ -786,26 +726,36 @@ impl Database for ListingDatabase {
async fn create_namespace( async fn create_namespace(
&self, &self,
request: CreateNamespaceRequest, _request: CreateNamespaceRequest,
) -> Result<CreateNamespaceResponse> { ) -> Result<CreateNamespaceResponse> {
self.namespace_database().create_namespace(request).await Err(Error::NotSupported {
message: "Namespace operations are not supported for listing database".into(),
})
} }
async fn drop_namespace(&self, request: DropNamespaceRequest) -> Result<DropNamespaceResponse> { async fn drop_namespace(
self.namespace_database().drop_namespace(request).await &self,
_request: DropNamespaceRequest,
) -> Result<DropNamespaceResponse> {
Err(Error::NotSupported {
message: "Namespace operations are not supported for listing database".into(),
})
} }
async fn describe_namespace( async fn describe_namespace(
&self, &self,
request: DescribeNamespaceRequest, _request: DescribeNamespaceRequest,
) -> Result<DescribeNamespaceResponse> { ) -> Result<DescribeNamespaceResponse> {
self.namespace_database().describe_namespace(request).await Err(Error::NotSupported {
message: "Namespace operations are not supported for listing database".into(),
})
} }
#[allow(deprecated)]
async fn table_names(&self, request: TableNamesRequest) -> Result<Vec<String>> { async fn table_names(&self, request: TableNamesRequest) -> Result<Vec<String>> {
if !request.namespace_path.is_empty() { if !request.namespace_path.is_empty() {
return self.namespace_database().table_names(request).await; return Err(Error::NotSupported {
message: "Namespace parameter is not supported for listing database. Only root namespace is supported.".into(),
});
} }
let mut f = self let mut f = self
.object_store .object_store
@@ -838,7 +788,9 @@ impl Database for ListingDatabase {
async fn list_tables(&self, request: ListTablesRequest) -> Result<ListTablesResponse> { async fn list_tables(&self, request: ListTablesRequest) -> Result<ListTablesResponse> {
if request.id.as_ref().map(|v| !v.is_empty()).unwrap_or(false) { if request.id.as_ref().map(|v| !v.is_empty()).unwrap_or(false) {
return self.namespace_database().list_tables(request).await; return Err(Error::NotSupported {
message: "Namespace parameter is not supported for listing database. Only root namespace is supported.".into(),
});
} }
let mut f = self let mut f = self
.object_store .object_store
@@ -886,8 +838,11 @@ impl Database for ListingDatabase {
} }
async fn create_table(&self, request: CreateTableRequest) -> Result<Arc<dyn BaseTable>> { async fn create_table(&self, request: CreateTableRequest) -> Result<Arc<dyn BaseTable>> {
if !request.namespace_path.is_empty() { // When namespace is not empty, location must be provided
return self.namespace_database().create_table(request).await; if !request.namespace_path.is_empty() && request.location.is_none() {
return Err(Error::InvalidInput {
message: "Location must be provided when namespace is not empty".into(),
});
} }
// Use provided location if available, otherwise derive from table name // Use provided location if available, otherwise derive from table name
let table_uri = request let table_uri = request
@@ -1004,8 +959,11 @@ impl Database for ListingDatabase {
} }
async fn open_table(&self, mut request: OpenTableRequest) -> Result<Arc<dyn BaseTable>> { async fn open_table(&self, mut request: OpenTableRequest) -> Result<Arc<dyn BaseTable>> {
if !request.namespace_path.is_empty() { // When namespace is not empty, location must be provided
return self.namespace_database().open_table(request).await; if !request.namespace_path.is_empty() && request.location.is_none() {
return Err(Error::InvalidInput {
message: "Location must be provided when namespace is not empty".into(),
});
} }
// Use provided location if available, otherwise derive from table name // Use provided location if available, otherwise derive from table name
let table_uri = request let table_uri = request
@@ -1101,10 +1059,9 @@ impl Database for ListingDatabase {
async fn drop_table(&self, name: &str, namespace_path: &[String]) -> Result<()> { async fn drop_table(&self, name: &str, namespace_path: &[String]) -> Result<()> {
if !namespace_path.is_empty() { if !namespace_path.is_empty() {
return self return Err(Error::NotSupported {
.namespace_database() message: "Namespace parameter is not supported for listing database.".into(),
.drop_table(name, namespace_path) });
.await;
} }
self.drop_tables(vec![name.to_string()]).await self.drop_tables(vec![name.to_string()]).await
} }
@@ -1113,10 +1070,9 @@ impl Database for ListingDatabase {
async fn drop_all_tables(&self, namespace_path: &[String]) -> Result<()> { async fn drop_all_tables(&self, namespace_path: &[String]) -> Result<()> {
// Check if namespace parameter is provided // Check if namespace parameter is provided
if !namespace_path.is_empty() { if !namespace_path.is_empty() {
return self return Err(Error::NotSupported {
.namespace_database() message: "Namespace parameter is not supported for listing database.".into(),
.drop_all_tables(namespace_path) });
.await;
} }
let tables = self.table_names(TableNamesRequest::default()).await?; let tables = self.table_names(TableNamesRequest::default()).await?;
self.drop_tables(tables).await self.drop_tables(tables).await
@@ -1127,11 +1083,30 @@ impl Database for ListingDatabase {
} }
async fn namespace_client(&self) -> Result<Arc<dyn lance_namespace::LanceNamespace>> { async fn namespace_client(&self) -> Result<Arc<dyn lance_namespace::LanceNamespace>> {
self.namespace_database.namespace_client().await // Create a DirectoryNamespace pointing to the same root with the same storage options
let mut builder = lance_namespace_impls::DirectoryNamespaceBuilder::new(&self.uri);
// Add storage options
if !self.storage_options.is_empty() {
builder = builder.storage_options(self.storage_options.clone());
}
// Use the same session
builder = builder.session(self.session.clone());
let namespace = builder.build().await.map_err(|e| Error::Runtime {
message: format!("Failed to create namespace client: {}", e),
})?;
Ok(Arc::new(namespace) as Arc<dyn lance_namespace::LanceNamespace>)
} }
async fn namespace_client_config(&self) -> Result<(String, HashMap<String, String>)> { async fn namespace_client_config(&self) -> Result<(String, HashMap<String, String>)> {
self.namespace_database.namespace_client_config().await let mut properties = HashMap::new();
properties.insert("root".to_string(), self.uri.clone());
for (key, value) in &self.storage_options {
properties.insert(format!("storage.{}", key), value.clone());
}
Ok(("dir".to_string(), properties))
} }
} }
@@ -1157,7 +1132,6 @@ mod tests {
#[cfg(feature = "remote")] #[cfg(feature = "remote")]
client_config: Default::default(), client_config: Default::default(),
options: Default::default(), options: Default::default(),
namespace_client_properties: Default::default(),
read_consistency_interval: None, read_consistency_interval: None,
session: None, session: None,
}; };
@@ -1291,7 +1265,6 @@ mod tests {
#[cfg(feature = "remote")] #[cfg(feature = "remote")]
client_config: Default::default(), client_config: Default::default(),
options: options.clone(), options: options.clone(),
namespace_client_properties: Default::default(),
read_consistency_interval: None, read_consistency_interval: None,
session: None, session: None,
}; };
@@ -1826,7 +1799,6 @@ mod tests {
#[cfg(feature = "remote")] #[cfg(feature = "remote")]
client_config: Default::default(), client_config: Default::default(),
options, options,
namespace_client_properties: Default::default(),
read_consistency_interval: None, read_consistency_interval: None,
session: None, session: None,
}; };
@@ -1932,7 +1904,6 @@ mod tests {
#[cfg(feature = "remote")] #[cfg(feature = "remote")]
client_config: Default::default(), client_config: Default::default(),
options, options,
namespace_client_properties: Default::default(),
read_consistency_interval: None, read_consistency_interval: None,
session: None, session: None,
}; };
@@ -2004,7 +1975,6 @@ mod tests {
#[cfg(feature = "remote")] #[cfg(feature = "remote")]
client_config: Default::default(), client_config: Default::default(),
options, options,
namespace_client_properties: Default::default(),
read_consistency_interval: None, read_consistency_interval: None,
session: None, session: None,
}; };
@@ -2138,208 +2108,4 @@ mod tests {
assert!(tables.contains(&"table1".to_string())); assert!(tables.contains(&"table1".to_string()));
assert!(tables.contains(&"table2".to_string())); assert!(tables.contains(&"table2".to_string()));
} }
#[tokio::test]
async fn test_listing_database_namespace_operations() {
let (_tempdir, db) = setup_database().await;
db.create_namespace(CreateNamespaceRequest {
id: Some(vec!["parent".to_string()]),
..Default::default()
})
.await
.unwrap();
db.create_namespace(CreateNamespaceRequest {
id: Some(vec!["parent".to_string(), "child".to_string()]),
..Default::default()
})
.await
.unwrap();
let root_namespaces = db
.list_namespaces(ListNamespacesRequest {
id: Some(vec![]),
..Default::default()
})
.await
.unwrap();
assert!(root_namespaces.namespaces.contains(&"parent".to_string()));
let child_namespaces = db
.list_namespaces(ListNamespacesRequest {
id: Some(vec!["parent".to_string()]),
..Default::default()
})
.await
.unwrap();
assert!(child_namespaces.namespaces.contains(&"child".to_string()));
db.describe_namespace(DescribeNamespaceRequest {
id: Some(vec!["parent".to_string(), "child".to_string()]),
..Default::default()
})
.await
.unwrap();
}
#[tokio::test]
async fn test_listing_database_with_namespace_client_properties() {
let tempdir = tempdir().unwrap();
let uri = tempdir.path().to_str().unwrap();
let mut namespace_client_properties = HashMap::new();
namespace_client_properties.insert(
"table_version_tracking_enabled".to_string(),
"true".to_string(),
);
namespace_client_properties.insert("manifest_enabled".to_string(), "true".to_string());
let request = ConnectRequest {
uri: uri.to_string(),
#[cfg(feature = "remote")]
client_config: Default::default(),
options: Default::default(),
namespace_client_properties,
read_consistency_interval: None,
session: None,
};
let db = ListingDatabase::connect_with_options(&request)
.await
.unwrap();
let namespace_path = vec!["test_ns".to_string()];
db.create_namespace(CreateNamespaceRequest {
id: Some(namespace_path.clone()),
..Default::default()
})
.await
.unwrap();
let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int32, false),
Field::new("name", DataType::Utf8, false),
]));
db.create_table(CreateTableRequest {
name: "managed_table".to_string(),
namespace_path: namespace_path.clone(),
data: Box::new(RecordBatch::new_empty(schema)) as Box<dyn Scannable>,
mode: CreateTableMode::Create,
write_options: Default::default(),
location: None,
namespace_client: None,
})
.await
.unwrap();
let namespace_client = db.namespace_client().await.unwrap();
let describe = namespace_client
.describe_table(lance_namespace::models::DescribeTableRequest {
id: Some(vec!["test_ns".to_string(), "managed_table".to_string()]),
..Default::default()
})
.await
.unwrap();
assert_eq!(describe.managed_versioning, Some(true));
}
#[tokio::test]
async fn test_listing_database_nested_namespace_table_ops() {
let (_tempdir, db) = setup_database().await;
let namespace_path = vec!["parent".to_string(), "child".to_string()];
db.create_namespace(CreateNamespaceRequest {
id: Some(vec!["parent".to_string()]),
..Default::default()
})
.await
.unwrap();
db.create_namespace(CreateNamespaceRequest {
id: Some(namespace_path.clone()),
..Default::default()
})
.await
.unwrap();
let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int32, false),
Field::new("name", DataType::Utf8, false),
]));
db.create_table(CreateTableRequest {
name: "nested_table".to_string(),
namespace_path: namespace_path.clone(),
data: Box::new(RecordBatch::new_empty(schema)) as Box<dyn Scannable>,
mode: CreateTableMode::Create,
write_options: Default::default(),
location: None,
namespace_client: None,
})
.await
.unwrap();
let namespace_client = db.namespace_client().await.unwrap();
let describe = namespace_client
.describe_table(lance_namespace::models::DescribeTableRequest {
id: Some(vec![
"parent".to_string(),
"child".to_string(),
"nested_table".to_string(),
]),
..Default::default()
})
.await
.unwrap();
assert!(describe.location.is_some());
let table = db
.open_table(OpenTableRequest {
name: "nested_table".to_string(),
namespace_path: namespace_path.clone(),
index_cache_size: None,
lance_read_params: None,
location: None,
namespace_client: None,
managed_versioning: None,
})
.await
.unwrap();
assert_eq!(table.name(), "nested_table");
#[allow(deprecated)]
let table_names = db
.table_names(TableNamesRequest {
namespace_path: namespace_path.clone(),
start_after: None,
limit: None,
})
.await
.unwrap();
assert_eq!(table_names, vec!["nested_table".to_string()]);
let list_tables = db
.list_tables(ListTablesRequest {
id: Some(namespace_path.clone()),
..Default::default()
})
.await
.unwrap();
assert_eq!(list_tables.tables, vec!["nested_table".to_string()]);
db.drop_table("nested_table", &namespace_path)
.await
.unwrap();
let post_drop = db
.list_tables(ListTablesRequest {
id: Some(namespace_path),
..Default::default()
})
.await
.unwrap();
assert!(post_drop.tables.is_empty());
}
} }
-41
View File
@@ -450,47 +450,6 @@ mod tests {
)); ));
} }
#[tokio::test]
async fn test_namespace_connection_with_namespace_client_properties() {
let tmp_dir = tempdir().unwrap();
let root_path = tmp_dir.path().to_str().unwrap().to_string();
let mut properties = HashMap::new();
properties.insert("root".to_string(), root_path);
let conn = connect_namespace("dir", properties)
.namespace_client_property("table_version_tracking_enabled", "true")
.namespace_client_property("manifest_enabled", "true")
.execute()
.await
.expect("Failed to connect to namespace");
conn.create_namespace(CreateNamespaceRequest {
id: Some(vec!["test_ns".into()]),
..Default::default()
})
.await
.expect("Failed to create namespace");
let test_data = create_test_data();
conn.create_table("test_table", test_data)
.namespace(vec!["test_ns".into()])
.execute()
.await
.expect("Failed to create table");
let namespace_client = conn.namespace_client().await.unwrap();
let describe = namespace_client
.describe_table(DescribeTableRequest {
id: Some(vec!["test_ns".into(), "test_table".into()]),
..Default::default()
})
.await
.expect("Failed to describe table");
assert_eq!(describe.managed_versioning, Some(true));
}
#[tokio::test] #[tokio::test]
async fn test_namespace_create_table_basic() { async fn test_namespace_create_table_basic() {
// Setup: Create a temporary directory for the namespace // Setup: Create a temporary directory for the namespace
+1 -1
View File
@@ -69,7 +69,7 @@
//! It treats [`FixedSizeList<Float16/Float32>`](https://docs.rs/arrow/latest/arrow/array/struct.FixedSizeListArray.html) //! It treats [`FixedSizeList<Float16/Float32>`](https://docs.rs/arrow/latest/arrow/array/struct.FixedSizeListArray.html)
//! columns as vector columns. //! columns as vector columns.
//! //!
//! For more details, please refer to the [LanceDB documentation](https://docs.lancedb.com). //! For more details, please refer to the [LanceDB documentation](https://lancedb.com/docs).
//! //!
//! #### Create a table //! #### Create a table
//! //!
+1 -1
View File
@@ -97,7 +97,7 @@ pub struct RemoteDatabaseOptions {
pub host_override: Option<String>, pub host_override: Option<String>,
/// Storage options configure the storage layer (e.g. S3, GCS, Azure, etc.) /// Storage options configure the storage layer (e.g. S3, GCS, Azure, etc.)
/// ///
/// See available options at <https://docs.lancedb.com/storage/> /// See available options at <https://lancedb.com/docs/storage/>
/// ///
/// These options are only used for LanceDB Enterprise and only a subset of options /// These options are only used for LanceDB Enterprise and only a subset of options
/// are supported. /// are supported.