mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
feat: accept GCP credentials Secret in storage integration clients (#4771)
Adds GCP inputs to the existing database-owned storage integration
methods. GCP takes one Secret reference; the Secret stores the complete
application credentials JSON as an opaque string. The same methods and
SQL transport handle create, describe, list and drop.
```python
info = db.create_storage_integration(
"gcs", "gs://customer-data/raw/", storage_provider="GCP",
gcp_credentials_json_secret=SecretReference(
"gcp_json", database="shared", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("gcs")
names = list(db.list_storage_integrations())
db.drop_storage_integration("gcs")
```
```sql
CREATE STORAGE INTEGRATION gcs
STORAGE_PROVIDER = 'GCP'
STORAGE_PREFIX = 'gs://customer-data/raw/'
GCP_CREDENTIALS_JSON_SECRET = shared.credentials.gcp_json;
DESCRIBE STORAGE INTEGRATION gcs;
SHOW STORAGE INTEGRATIONS;
DROP STORAGE INTEGRATION gcs;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Existing Flight SQL transport]
SQL --> Config[Configuration and one Secret reference]
```
Adds Rust request construction, SQL rendering and typed metadata
decoding, plus sync/async Flight SQL regressions across the three Arrow
string encodings. Rejects missing or mixed-cloud credential options.
Local builds/tests are excluded by request; validation runs in CI.
This commit is contained in:
@@ -1023,3 +1023,41 @@ DROP STORAGE INTEGRATION customer_blobs;
|
||||
```
|
||||
|
||||
Registration returns setup instructions and leaves `access_verified` false. It stores an account-qualified prefix such as `az://customer-data@customeraccount.blob.core.windows.net/raw/`.
|
||||
|
||||
### GCP storage integrations
|
||||
|
||||
Create one Secret containing the complete GCP application credentials JSON file
|
||||
content, then reference its name when registering Google Cloud Storage access:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
db.create_secret("gcp_credentials", Path("credentials.json").read_text())
|
||||
integration = db.create_storage_integration(
|
||||
"customer_gcs",
|
||||
"gs://customer-data/raw/",
|
||||
storage_provider="GCP",
|
||||
gcp_credentials_json_secret="gcp_credentials",
|
||||
)
|
||||
print(integration.configuration_instructions)
|
||||
```
|
||||
|
||||
The equivalent SQL references the existing Secret:
|
||||
|
||||
```sql
|
||||
CREATE STORAGE INTEGRATION customer_gcs
|
||||
STORAGE_PROVIDER = 'GCP'
|
||||
STORAGE_PREFIX = 'gs://customer-data/raw/'
|
||||
GCP_CREDENTIALS_JSON_SECRET = public.gcp_credentials;
|
||||
DESCRIBE STORAGE INTEGRATION customer_gcs;
|
||||
SHOW STORAGE INTEGRATIONS;
|
||||
DROP STORAGE INTEGRATION customer_gcs;
|
||||
```
|
||||
|
||||
`gcp_credentials_json_secret` names a Secret; pass the file content to
|
||||
`create_secret`, not to `create_storage_integration`. Use `SecretReference` to
|
||||
qualify another database or namespace. Sync and async connections accept the same
|
||||
options. The service interprets the stored JSON when accessing storage; registration
|
||||
does not validate its credentials format or verify access, and leaves
|
||||
`access_verified` false. Integration metadata contains the Secret reference, never
|
||||
its JSON content. GCP options cannot be combined with AWS or Azure options.
|
||||
|
||||
Reference in New Issue
Block a user