rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-11-13 14:16:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add database storage integration APIs (#4618)

Browse Source 
Adds database-owned S3 storage integrations with Secret-style namespace
paths in a separate keyspace. Use a regular database under the cluster
root for bootstrap integrations and Secrets. `default` is an example
name, with no enforced or automatically provisioned database; `system`
remains read-only. Authentication uses either an IAM role or two
existing Secret references, including cross-database references;
configuration metadata never contains credential values.

### Python experience

```python
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
info = db.create_storage_integration(
    "raw", "s3://customer-bucket/raw/",
    aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
    aws_region="us-east-1",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("raw")
print(list(db.list_storage_integrations()))
db.drop_storage_integration("raw")
```

For static credentials, replace `aws_role_arn` with
`aws_access_key_id_secret=SecretReference("s3_key_id",
database="default")` and
`aws_secret_access_key_secret=SecretReference("s3_secret_key",
database="default")`, imported from `lancedb.storage_integration`. Pass
`namespace_path` to address nested namespaces. Async database
connections expose the same APIs; listing is lazy. All four client
operations execute SQL through the existing SQL transport, with typed
Arrow result decoding and name-based listing cursors.

### SQL experience

```sql
CREATE STORAGE INTEGRATION default.public.raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-bucket/raw/'
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
    STORAGE_AWS_REGION = 'us-east-1';
DESCRIBE STORAGE INTEGRATION default.public.raw;
SHOW STORAGE INTEGRATIONS IN default.public;
DROP STORAGE INTEGRATION IF EXISTS default.public.raw;

CREATE STORAGE INTEGRATION default.public.raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-bucket/raw/'
    STORAGE_AWS_ACCESS_KEY_ID_SECRET = default.public.s3_key_id
    STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = default.public.s3_secret_key;
```

CREATE and DESCRIBE return configuration and a single sentence of setup
guidance. IAM mode also returns the deployment role principal, stable
external ID, customer trust policy, and prefix-scoped permissions
policy. Registration does not verify access. Creation requires namespace
CREATE_STORAGE_INTEGRATION plus parent USAGE; Secret references require
source database, namespace, and Secret USAGE. Describe requires object
USAGE; drop requires OWNERSHIP. SQL supports GRANT/REVOKE for these
objects and their creation privilege.

Includes client documentation and Flight SQL regression coverage; no
dedicated storage-integration REST endpoints are used. Local
builds/tests are excluded by request.
This commit is contained in:
Jack Ye
2026-10-30 00:26:42 -07:00
committed by GitHub
parent a39a31807f
commit 6d126a3601
24 changed files with 1453 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/src/python/python.md
+81
View File
@@ -42,6 +42,12 @@ table names. `db.list_tables().tables` is a list of the first page; `limit`
controls its page size and `page_token` resumes at a saved token. The same
patterns apply to `list_namespaces()`. Other resource listings use `page_limit`.
Resource listings (`list_storage_integrations`, `list_secrets`, `list_views`, `list_jobs`, `list_functions`, and
`list_materialized_views`) return lazy iterators. Use `list(db.list_views())` to
collect synchronous results, or `[name async for name in db.list_views()]` with
an asynchronous connection. `page_limit` controls each request, and `page_token`
resumes a listing. Drain the iterator's cached results before saving its token.
::: lancedb.listing.Listing
::: lancedb.listing.TableListing
@@ -162,6 +168,11 @@ registered functions, for example,
namespace. These objects identify resources for ACL operations; a secret object
contains the credential's name, not its value.
Storage integration objects identify database-owned storage configuration, for
example `Object.storage_integration(database="default", name="raw")`. Pass
`namespace_path` for nested namespaces. Their keyspace is separate from Secrets,
Functions, and relations; the wire name is `storage_integration:default:public$raw`.
More object types may be added in the future.
### Access control lists
@@ -205,6 +216,27 @@ for privilege in [Privilege.CREATE_SECRET, Privilege.CREATE_FUNCTION]:
Both privileges also accept their string names in synchronous and asynchronous
ACL methods.
Storage integration creation requires `CREATE_STORAGE_INTEGRATION` on its containing
namespace, together with database and namespace `USAGE`:
```python
catalog.authz.add_acl(
object=Object.namespace(database="default"),
subject=Subject.role("developer"),
privilege=Privilege.CREATE_STORAGE_INTEGRATION,
)
catalog.authz.add_acl(
object=Object.storage_integration(database="default", name="raw"),
subject=Subject.role("reader"),
privilege=Privilege.USAGE,
)
```
Creation grants ownership to the creator. Describing an integration requires
`USAGE` on that integration; dropping it requires `OWNERSHIP`. Listing integrations
requires `USAGE` on the selected database and namespace. Describing and dropping
also require parent `USAGE` grants. Manage these grants through `catalog.authz`.
By default, the `list_acls` function lists all the access control entries in
the system. By adding a function parameter, the output is filtered to just ACLs
matching that parameter. For example, this code lists all the ACLs for the
@@ -851,3 +883,52 @@ rows nearest to a query vector and can be created with the
::: lancedb.query.AsyncTakeQuery
options:
inherited_members: true
## Storage integrations
::: lancedb.storage_integration.StorageIntegrationInfo
::: lancedb.storage_integration.SecretReference
Integrations belong to a database and follow the same namespace-path naming rules
as Secrets, with their own keyspace. Use a regular database under the cluster root for storage configuration needed
before opening an external database. `default` is a convention; no database with
that name is required or created automatically. The system database is read-only.
Client methods execute SQL through the existing SQL transport.
```python
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
integration = db.create_storage_integration(
"raw", "s3://customer-bucket/raw/",
aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
)
print(integration.configuration_instructions)
integration = db.describe_storage_integration("raw")
for name in db.list_storage_integrations():
print(name)
db.drop_storage_integration("raw")
```
Static credentials use two existing, single-value Secrets instead of a role:
```python
from lancedb.storage_integration import SecretReference
integration = db.create_storage_integration(
"raw", "s3://customer-bucket/raw/",
aws_access_key_id_secret=SecretReference("s3-key-id", database="default"),
aws_secret_access_key_secret=SecretReference("s3-secret-key", database="default"),
)
```
Pass `namespace_path` on the object APIs and Secret references to select a namespace.
A string Secret reference selects the root namespace of the integration's database.
Cross-database references require `USAGE` on the source database, namespace, and
Secret. Role and static-credential modes are mutually exclusive. Credential values
are never included in integration metadata.
`configuration_instructions` is one sentence of display guidance whose wording may
change. Use the structured identity and policy fields for programmatic access.
Registration does not verify AWS access. `AsyncConnection` provides the same methods;
await create, describe, and drop, and use `async for` for listing.