mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 15:46:55 +00:00
Migrates SQL-backed catalog listing APIs to `LIMIT ... FROM '<page-token>'` and server-issued Arrow continuation metadata. Namespace-specific APIs send an explicit `IN` schema clause, including `IN "public"` for root, so raw SQL can default to all namespaces. ```sql SHOW VIEWS LIMIT 100; SHOW VIEWS IN team$ml LIMIT 100; SHOW VIEWS IN team$ml LIMIT 100 FROM '<page-token>'; SHOW VIEWS IN other_db.team$ml LIMIT 100; ``` SDK clients forward opaque tokens unchanged, preserve empty continuations, and decode identifier columns by name. Removes legacy cursor reconstruction and updates the SQL reference and Python Flight fixture. Public docs cover optional direct schema paths, all-namespace defaults, and the shared pagination syntax across every catalog listing.
263 lines
11 KiB
Plaintext
263 lines
11 KiB
Plaintext
---
|
|
title: "Storage Integration"
|
|
sidebarTitle: "Storage Integration"
|
|
description: "CREATE, DESCRIBE, SHOW and DROP STORAGE INTEGRATION: a namespace-scoped, reusable credential for reading or importing from external storage."
|
|
icon: cloud
|
|
keywords: ["storage integration", "create storage integration", "show storage integrations", "drop storage integration", "s3", "gcs", "azure", "huggingface", "sql"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
A storage integration is a namespace-scoped catalog object that lets a
|
|
query read from, or import from, a storage location outside the
|
|
deployment's own tables -- an S3, GCS or Azure prefix, or a HuggingFace
|
|
dataset repository -- without embedding a credential in the query itself.
|
|
It is addressed and scoped the same way a [Secret](/sql/ddl/secret) is: a
|
|
1-3 part identifier, `name`, `namespace.name` or `database.namespace.name`,
|
|
with a nested namespace path `$`-joined into that middle part.
|
|
|
|
## CREATE STORAGE INTEGRATION {#create}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
CREATE STORAGE INTEGRATION <name>
|
|
STORAGE_PROVIDER = 'S3' | 'AZURE' | 'GCP' | 'HUGGINGFACE'
|
|
STORAGE_PREFIX = '<uri-prefix>'
|
|
[ <provider option> = <value> [...] ]
|
|
```
|
|
</CodeGroup>
|
|
|
|
{/* Draft for the storage-integration exist_ok release.
|
|
|
|
`IF NOT EXISTS` returns an existing integration unchanged, including its provider,
|
|
storage prefix, credential references and external ID. In the Python SDK, pass
|
|
`exist_ok=True` to `create_storage_integration`. The default reports an error when
|
|
the name already exists.
|
|
Returning an existing integration requires `USAGE` on that integration.
|
|
|
|
*/}
|
|
|
|
`STORAGE_PROVIDER` and `STORAGE_PREFIX` are required for every provider.
|
|
`STORAGE_PREFIX` is the URI prefix the integration covers -- a query only
|
|
matches an integration whose prefix is a prefix of the URI it reads.
|
|
Options ending in `_SECRET` take a [Secret](/sql/ddl/secret) identifier, not
|
|
a quoted string: `name`, `namespace.name` or `database.namespace.name`, with a
|
|
nested namespace path `$`-joined in the middle part (`` db.`team$prod`.key ``).
|
|
Omitted parts default to the session's current database and namespace.
|
|
Creating the integration requires `USAGE` on each referenced Secret and on its
|
|
database and namespace. Passing an option a provider does not accept, such as
|
|
an Azure option on an `S3` integration, is refused.
|
|
|
|
### S3 {#s3}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
CREATE STORAGE INTEGRATION raw
|
|
STORAGE_PROVIDER = 'S3'
|
|
STORAGE_PREFIX = 's3://customer-data/raw/'
|
|
STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
|
|
|
|
CREATE STORAGE INTEGRATION raw
|
|
STORAGE_PROVIDER = 'S3'
|
|
STORAGE_PREFIX = 's3://customer-data/raw/'
|
|
STORAGE_AWS_ACCESS_KEY_ID_SECRET = bootstrap.credentials.aws_key_id
|
|
STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = bootstrap.credentials.aws_key_secret
|
|
```
|
|
</CodeGroup>
|
|
|
|
| Option | Type | Meaning |
|
|
| --- | --- | --- |
|
|
| `STORAGE_AWS_ROLE_ARN` | string | an IAM role to assume, by cross-account trust (see [below](#trust)) |
|
|
| `STORAGE_AWS_ACCESS_KEY_ID_SECRET` | Secret ref | a static access key ID, paired with the secret key below |
|
|
| `STORAGE_AWS_SECRET_ACCESS_KEY_SECRET` | Secret ref | the matching static secret access key |
|
|
| `STORAGE_AWS_REGION` | string, optional | the bucket's region |
|
|
| `STORAGE_AWS_ENDPOINT` | string, optional | an S3-compatible endpoint URL (MinIO, Tigris, ...) instead of AWS itself |
|
|
|
|
Give either `STORAGE_AWS_ROLE_ARN` alone, or both key-pair options together
|
|
-- mixing the two, or supplying only one half of the key pair, is refused.
|
|
|
|
### Azure {#azure}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
CREATE STORAGE INTEGRATION blobs
|
|
STORAGE_PROVIDER = 'AZURE'
|
|
STORAGE_PREFIX = 'az://customer-data/raw/'
|
|
STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
|
|
STORAGE_AZURE_TENANT_ID_SECRET = bootstrap.credentials.tenant
|
|
STORAGE_AZURE_CLIENT_ID_SECRET = bootstrap.credentials.client
|
|
STORAGE_AZURE_CLIENT_SECRET_SECRET = bootstrap.credentials.credential
|
|
```
|
|
</CodeGroup>
|
|
|
|
| Option | Type | Meaning |
|
|
| --- | --- | --- |
|
|
| `STORAGE_AZURE_STORAGE_ACCOUNT` | string | the storage account name |
|
|
| `STORAGE_AZURE_TENANT_ID_SECRET` | Secret ref | Entra ID tenant |
|
|
| `STORAGE_AZURE_CLIENT_ID_SECRET` | Secret ref | service principal client ID |
|
|
| `STORAGE_AZURE_CLIENT_SECRET_SECRET` | Secret ref | service principal client secret |
|
|
|
|
All four are required together; there is no role-based alternative for
|
|
Azure.
|
|
|
|
### GCP {#gcp}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
CREATE STORAGE INTEGRATION gcs
|
|
STORAGE_PROVIDER = 'GCP'
|
|
STORAGE_PREFIX = 'gs://customer-data/raw/'
|
|
GCP_CREDENTIALS_JSON_SECRET = bootstrap.credentials.gcp
|
|
```
|
|
</CodeGroup>
|
|
|
|
`GCP_CREDENTIALS_JSON_SECRET` -- note the option name has no `STORAGE_`
|
|
prefix, unlike every other provider's options -- names a Secret holding
|
|
the full service-account JSON key. It is required; GCP has no role-based
|
|
alternative here.
|
|
|
|
### HuggingFace {#huggingface}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
CREATE STORAGE INTEGRATION hub
|
|
STORAGE_PROVIDER = 'HUGGINGFACE'
|
|
STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
|
|
HUGGINGFACE_TOKEN_SECRET = bootstrap.credentials.hf_token
|
|
```
|
|
</CodeGroup>
|
|
|
|
`HUGGINGFACE_TOKEN_SECRET` is the only HuggingFace option, and it is
|
|
required: creating an integration without it is refused. For anonymous
|
|
access, at HuggingFace's default anonymous rate limits, register no
|
|
integration for that prefix at all. See
|
|
[Inspect Datasets](/curation/inspect-datasets) for the full
|
|
read and import workflow this integration unlocks.
|
|
|
|
### The S3 trust policy {#trust}
|
|
|
|
Creating an `STORAGE_AWS_ROLE_ARN` integration does not finish the AWS
|
|
side of the setup. The result row's `trust_policy`, `permissions_policy`
|
|
and `configuration_instructions` columns -- also visible from
|
|
[`DESCRIBE`](#describe) -- are the IAM policy documents your role needs:
|
|
attach `trust_policy` as the role's trust relationship, scoped to
|
|
`aws_account_id`, `aws_iam_principal_arn` and `aws_external_id` from the
|
|
same row, and attach `permissions_policy` to grant that role read access
|
|
to the bucket. `access_verified` reports whether the deployment already
|
|
confirmed it can assume the role and read the prefix; `false` typically
|
|
means the trust policy has not been attached yet.
|
|
|
|
## Reading and importing through the integration {#read-import}
|
|
|
|
A query or `CREATE TABLE ... AS SELECT` can reference a matching URI
|
|
directly, once an integration covers it -- the integration is never named
|
|
in the statement. What the URI reads depends on the provider:
|
|
|
|
- **HuggingFace** (`hf://datasets/...`) reads Parquet files from a dataset
|
|
repository. See [Inspect Datasets](/curation/inspect-datasets)
|
|
for the full workflow, including how a CTAS import distributes across
|
|
plan executors.
|
|
- **S3, GCS and Azure** (`s3://`, `gs://`, `az://`) read an existing Lance
|
|
dataset at that prefix, the same way as a managed table:
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
SELECT * FROM 's3://customer-data/raw/events.lance' LIMIT 10
|
|
|
|
CREATE TABLE events_copy AS
|
|
SELECT * FROM 's3://customer-data/raw/events.lance'
|
|
```
|
|
</CodeGroup>
|
|
|
|
A CTAS import distributes the same way regardless of provider: disjoint
|
|
file and row-group ranges go to plan-executor workers, each writes
|
|
uncommitted Lance files, and a coordinator commits the table once every
|
|
worker finishes. See [`CREATE TABLE`](/sql/ddl/table#create) for
|
|
`CREATE TABLE AS SELECT` syntax in general.
|
|
|
|
## DESCRIBE STORAGE INTEGRATION {#describe}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
DESCRIBE STORAGE INTEGRATION raw
|
|
DESC STORAGE INTEGRATION platform.hub
|
|
```
|
|
</CodeGroup>
|
|
|
|
Returns one row with every column `CREATE` returns: the provider, prefix,
|
|
the Secret references and plain-string options that apply to its
|
|
provider (the rest are null), `created_at_millis`, `access_verified`, and
|
|
-- for an `STORAGE_AWS_ROLE_ARN` integration -- the generated AWS policy
|
|
documents from [above](#trust). No call ever returns a Secret's value,
|
|
only the reference naming it.
|
|
|
|
## SHOW STORAGE INTEGRATIONS {#show}
|
|
|
|
```sql
|
|
SHOW STORAGE INTEGRATIONS IN analytics.`team$ml` LIMIT 100;
|
|
SHOW STORAGE INTEGRATIONS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
|
|
```
|
|
|
|
Returns two non-null string columns: `namespace_path`, a compact JSON array
|
|
of parent namespace components (`[]` for root), and `name`, the unqualified
|
|
object name. Omitting `IN` lists all authorized namespaces in the current database;
|
|
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
|
|
the current database's root. Listings require the relevant database and
|
|
namespace `USAGE` privileges.
|
|
|
|
`LIST STORAGE INTEGRATIONS` is an exact alias, including pagination.
|
|
|
|
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
|
|
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
|
|
unchanged with the same object type and scope. Stop when no token is returned;
|
|
a short or empty page can still carry a continuation. See
|
|
[catalog listing pagination](/sql/pagination).
|
|
|
|
## DROP STORAGE INTEGRATION {#drop}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
DROP STORAGE INTEGRATION raw
|
|
DROP STORAGE INTEGRATION IF EXISTS raw
|
|
```
|
|
</CodeGroup>
|
|
|
|
Returns `dropped`, `true` or `false`. There is no `ALTER STORAGE
|
|
INTEGRATION`: to rotate a credential, rotate the Secret it references
|
|
(see [Secret](/sql/ddl/secret)) rather than dropping and recreating the
|
|
integration. A storage integration an attached database still depends on
|
|
blocks the drop; detach the database first.
|
|
|
|
## GRANT and REVOKE {#grant}
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
GRANT CREATE STORAGE INTEGRATION ON SCHEMA platform TO ROLE platform_admins
|
|
GRANT USAGE ON STORAGE INTEGRATION platform.hub TO ROLE research
|
|
GRANT OWNERSHIP ON STORAGE INTEGRATION platform.hub TO ROLE platform_admins
|
|
REVOKE USAGE ON STORAGE INTEGRATION platform.hub FROM ROLE research
|
|
```
|
|
</CodeGroup>
|
|
|
|
`CREATE_STORAGE_INTEGRATION` on the namespace creates one; `USAGE` on the
|
|
integration itself lets a query match and read through it; `OWNERSHIP`
|
|
grants full control, including dropping it. See
|
|
[Privileges](/governance/privileges#sql) for the full statement syntax,
|
|
every grantee form, and how a query picks a matching integration without
|
|
naming one.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Inspect Datasets" icon="database" href="/curation/inspect-datasets">
|
|
Query and import HuggingFace or object-storage datasets, the most common use of a storage integration today.
|
|
</Card>
|
|
<Card title="Secret" icon="key" href="/sql/ddl/secret">
|
|
The credential storage integration options reference.
|
|
</Card>
|
|
<Card title="Privileges" icon="key" href="/governance/privileges">
|
|
Every privilege, and the grant and revoke syntax that covers each one.
|
|
</Card>
|
|
</CardGroup>
|