feat: add database storage integration APIs (#4618)

Adds database-owned S3 storage integrations with Secret-style namespace
paths in a separate keyspace. Use a regular database under the cluster
root for bootstrap integrations and Secrets. `default` is an example
name, with no enforced or automatically provisioned database; `system`
remains read-only. Authentication uses either an IAM role or two
existing Secret references, including cross-database references;
configuration metadata never contains credential values.

### Python experience

```python
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
info = db.create_storage_integration(
    "raw", "s3://customer-bucket/raw/",
    aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
    aws_region="us-east-1",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("raw")
print(list(db.list_storage_integrations()))
db.drop_storage_integration("raw")
```

For static credentials, replace `aws_role_arn` with
`aws_access_key_id_secret=SecretReference("s3_key_id",
database="default")` and
`aws_secret_access_key_secret=SecretReference("s3_secret_key",
database="default")`, imported from `lancedb.storage_integration`. Pass
`namespace_path` to address nested namespaces. Async database
connections expose the same APIs; listing is lazy. All four client
operations execute SQL through the existing SQL transport, with typed
Arrow result decoding and name-based listing cursors.

### SQL experience

```sql
CREATE STORAGE INTEGRATION default.public.raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-bucket/raw/'
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
    STORAGE_AWS_REGION = 'us-east-1';
DESCRIBE STORAGE INTEGRATION default.public.raw;
SHOW STORAGE INTEGRATIONS IN default.public;
DROP STORAGE INTEGRATION IF EXISTS default.public.raw;

CREATE STORAGE INTEGRATION default.public.raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-bucket/raw/'
    STORAGE_AWS_ACCESS_KEY_ID_SECRET = default.public.s3_key_id
    STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = default.public.s3_secret_key;
```

CREATE and DESCRIBE return configuration and a single sentence of setup
guidance. IAM mode also returns the deployment role principal, stable
external ID, customer trust policy, and prefix-scoped permissions
policy. Registration does not verify access. Creation requires namespace
CREATE_STORAGE_INTEGRATION plus parent USAGE; Secret references require
source database, namespace, and Secret USAGE. Describe requires object
USAGE; drop requires OWNERSHIP. SQL supports GRANT/REVOKE for these
objects and their creation privilege.

Includes client documentation and Flight SQL regression coverage; no
dedicated storage-integration REST endpoints are used. Local
builds/tests are excluded by request.
This commit is contained in:
Jack Ye
2026-10-30 00:26:42 -07:00
committed by GitHub
parent a39a31807f
commit 6d126a3601
24 changed files with 1453 additions and 11 deletions
+13
View File
@@ -262,6 +262,19 @@ impl AuthzObject {
}))
}
#[staticmethod]
fn storage_integration(
database: &str,
namespace: &Bound<'_, PyAny>,
name: &str,
) -> PyResult<Self> {
Self::validated(Object::StorageIntegration(StorageIntegrationObject {
database: database.into(),
namespace: namespace_path(namespace)?,
storage_integration: name.into(),
}))
}
#[staticmethod]
fn function(database: &str, namespace: &Bound<'_, PyAny>, name: &str) -> PyResult<Self> {
Self::validated(Object::Function(FunctionObject {