Files
lancedb/docs/web/sql/ddl/storage-integration.mdx
T
Jack Ye e8a8dd1b2a feat(sql): use server-issued catalog listing tokens (#5026)
Migrates SQL-backed catalog listing APIs to `LIMIT ... FROM
'<page-token>'` and server-issued Arrow continuation metadata.
Namespace-specific APIs send an explicit `IN` schema clause, including
`IN "public"` for root, so raw SQL can default to all namespaces.

```sql
SHOW VIEWS LIMIT 100;
SHOW VIEWS IN team$ml LIMIT 100;
SHOW VIEWS IN team$ml LIMIT 100 FROM '<page-token>';
SHOW VIEWS IN other_db.team$ml LIMIT 100;
```

SDK clients forward opaque tokens unchanged, preserve empty
continuations, and decode identifier columns by name. Removes legacy
cursor reconstruction and updates the SQL reference and Python Flight
fixture. Public docs cover optional direct schema paths, all-namespace
defaults, and the shared pagination syntax across every catalog listing.
2026-11-15 08:55:42 +08:00

263 lines
11 KiB
Plaintext

---
title: "Storage Integration"
sidebarTitle: "Storage Integration"
description: "CREATE, DESCRIBE, SHOW and DROP STORAGE INTEGRATION: a namespace-scoped, reusable credential for reading or importing from external storage."
icon: cloud
keywords: ["storage integration", "create storage integration", "show storage integrations", "drop storage integration", "s3", "gcs", "azure", "huggingface", "sql"]
availability:
embedded: unavailable
enterprise: available
---
A storage integration is a namespace-scoped catalog object that lets a
query read from, or import from, a storage location outside the
deployment's own tables -- an S3, GCS or Azure prefix, or a HuggingFace
dataset repository -- without embedding a credential in the query itself.
It is addressed and scoped the same way a [Secret](/sql/ddl/secret) is: a
1-3 part identifier, `name`, `namespace.name` or `database.namespace.name`,
with a nested namespace path `$`-joined into that middle part.
## CREATE STORAGE INTEGRATION {#create}
<CodeGroup>
```sql SQL icon="database"
CREATE STORAGE INTEGRATION <name>
STORAGE_PROVIDER = 'S3' | 'AZURE' | 'GCP' | 'HUGGINGFACE'
STORAGE_PREFIX = '<uri-prefix>'
[ <provider option> = <value> [...] ]
```
</CodeGroup>
{/* Draft for the storage-integration exist_ok release.
`IF NOT EXISTS` returns an existing integration unchanged, including its provider,
storage prefix, credential references and external ID. In the Python SDK, pass
`exist_ok=True` to `create_storage_integration`. The default reports an error when
the name already exists.
Returning an existing integration requires `USAGE` on that integration.
*/}
`STORAGE_PROVIDER` and `STORAGE_PREFIX` are required for every provider.
`STORAGE_PREFIX` is the URI prefix the integration covers -- a query only
matches an integration whose prefix is a prefix of the URI it reads.
Options ending in `_SECRET` take a [Secret](/sql/ddl/secret) identifier, not
a quoted string: `name`, `namespace.name` or `database.namespace.name`, with a
nested namespace path `$`-joined in the middle part (`` db.`team$prod`.key ``).
Omitted parts default to the session's current database and namespace.
Creating the integration requires `USAGE` on each referenced Secret and on its
database and namespace. Passing an option a provider does not accept, such as
an Azure option on an `S3` integration, is refused.
### S3 {#s3}
<CodeGroup>
```sql SQL icon="database"
CREATE STORAGE INTEGRATION raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-data/raw/'
STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
CREATE STORAGE INTEGRATION raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-data/raw/'
STORAGE_AWS_ACCESS_KEY_ID_SECRET = bootstrap.credentials.aws_key_id
STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = bootstrap.credentials.aws_key_secret
```
</CodeGroup>
| Option | Type | Meaning |
| --- | --- | --- |
| `STORAGE_AWS_ROLE_ARN` | string | an IAM role to assume, by cross-account trust (see [below](#trust)) |
| `STORAGE_AWS_ACCESS_KEY_ID_SECRET` | Secret ref | a static access key ID, paired with the secret key below |
| `STORAGE_AWS_SECRET_ACCESS_KEY_SECRET` | Secret ref | the matching static secret access key |
| `STORAGE_AWS_REGION` | string, optional | the bucket's region |
| `STORAGE_AWS_ENDPOINT` | string, optional | an S3-compatible endpoint URL (MinIO, Tigris, ...) instead of AWS itself |
Give either `STORAGE_AWS_ROLE_ARN` alone, or both key-pair options together
-- mixing the two, or supplying only one half of the key pair, is refused.
### Azure {#azure}
<CodeGroup>
```sql SQL icon="database"
CREATE STORAGE INTEGRATION blobs
STORAGE_PROVIDER = 'AZURE'
STORAGE_PREFIX = 'az://customer-data/raw/'
STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
STORAGE_AZURE_TENANT_ID_SECRET = bootstrap.credentials.tenant
STORAGE_AZURE_CLIENT_ID_SECRET = bootstrap.credentials.client
STORAGE_AZURE_CLIENT_SECRET_SECRET = bootstrap.credentials.credential
```
</CodeGroup>
| Option | Type | Meaning |
| --- | --- | --- |
| `STORAGE_AZURE_STORAGE_ACCOUNT` | string | the storage account name |
| `STORAGE_AZURE_TENANT_ID_SECRET` | Secret ref | Entra ID tenant |
| `STORAGE_AZURE_CLIENT_ID_SECRET` | Secret ref | service principal client ID |
| `STORAGE_AZURE_CLIENT_SECRET_SECRET` | Secret ref | service principal client secret |
All four are required together; there is no role-based alternative for
Azure.
### GCP {#gcp}
<CodeGroup>
```sql SQL icon="database"
CREATE STORAGE INTEGRATION gcs
STORAGE_PROVIDER = 'GCP'
STORAGE_PREFIX = 'gs://customer-data/raw/'
GCP_CREDENTIALS_JSON_SECRET = bootstrap.credentials.gcp
```
</CodeGroup>
`GCP_CREDENTIALS_JSON_SECRET` -- note the option name has no `STORAGE_`
prefix, unlike every other provider's options -- names a Secret holding
the full service-account JSON key. It is required; GCP has no role-based
alternative here.
### HuggingFace {#huggingface}
<CodeGroup>
```sql SQL icon="database"
CREATE STORAGE INTEGRATION hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
HUGGINGFACE_TOKEN_SECRET = bootstrap.credentials.hf_token
```
</CodeGroup>
`HUGGINGFACE_TOKEN_SECRET` is the only HuggingFace option, and it is
required: creating an integration without it is refused. For anonymous
access, at HuggingFace's default anonymous rate limits, register no
integration for that prefix at all. See
[Inspect Datasets](/curation/inspect-datasets) for the full
read and import workflow this integration unlocks.
### The S3 trust policy {#trust}
Creating an `STORAGE_AWS_ROLE_ARN` integration does not finish the AWS
side of the setup. The result row's `trust_policy`, `permissions_policy`
and `configuration_instructions` columns -- also visible from
[`DESCRIBE`](#describe) -- are the IAM policy documents your role needs:
attach `trust_policy` as the role's trust relationship, scoped to
`aws_account_id`, `aws_iam_principal_arn` and `aws_external_id` from the
same row, and attach `permissions_policy` to grant that role read access
to the bucket. `access_verified` reports whether the deployment already
confirmed it can assume the role and read the prefix; `false` typically
means the trust policy has not been attached yet.
## Reading and importing through the integration {#read-import}
A query or `CREATE TABLE ... AS SELECT` can reference a matching URI
directly, once an integration covers it -- the integration is never named
in the statement. What the URI reads depends on the provider:
- **HuggingFace** (`hf://datasets/...`) reads Parquet files from a dataset
repository. See [Inspect Datasets](/curation/inspect-datasets)
for the full workflow, including how a CTAS import distributes across
plan executors.
- **S3, GCS and Azure** (`s3://`, `gs://`, `az://`) read an existing Lance
dataset at that prefix, the same way as a managed table:
<CodeGroup>
```sql SQL icon="database"
SELECT * FROM 's3://customer-data/raw/events.lance' LIMIT 10
CREATE TABLE events_copy AS
SELECT * FROM 's3://customer-data/raw/events.lance'
```
</CodeGroup>
A CTAS import distributes the same way regardless of provider: disjoint
file and row-group ranges go to plan-executor workers, each writes
uncommitted Lance files, and a coordinator commits the table once every
worker finishes. See [`CREATE TABLE`](/sql/ddl/table#create) for
`CREATE TABLE AS SELECT` syntax in general.
## DESCRIBE STORAGE INTEGRATION {#describe}
<CodeGroup>
```sql SQL icon="database"
DESCRIBE STORAGE INTEGRATION raw
DESC STORAGE INTEGRATION platform.hub
```
</CodeGroup>
Returns one row with every column `CREATE` returns: the provider, prefix,
the Secret references and plain-string options that apply to its
provider (the rest are null), `created_at_millis`, `access_verified`, and
-- for an `STORAGE_AWS_ROLE_ARN` integration -- the generated AWS policy
documents from [above](#trust). No call ever returns a Secret's value,
only the reference naming it.
## SHOW STORAGE INTEGRATIONS {#show}
```sql
SHOW STORAGE INTEGRATIONS IN analytics.`team$ml` LIMIT 100;
SHOW STORAGE INTEGRATIONS IN analytics.`team$ml` LIMIT 100 FROM '<page-token>';
```
Returns two non-null string columns: `namespace_path`, a compact JSON array
of parent namespace components (`[]` for root), and `name`, the unqualified
object name. Omitting `IN` lists all authorized namespaces in the current database;
`IN team$ml` selects that exact path without a database qualifier; `IN public` selects
the current database's root. Listings require the relevant database and
namespace `USAGE` privileges.
`LIST STORAGE INTEGRATIONS` is an exact alias, including pagination.
Page through results using `LIMIT n [FROM '<page-token>']`. Read the opaque
`lancedb.next_page_token` from the Arrow result schema metadata and pass it back
unchanged with the same object type and scope. Stop when no token is returned;
a short or empty page can still carry a continuation. See
[catalog listing pagination](/sql/pagination).
## DROP STORAGE INTEGRATION {#drop}
<CodeGroup>
```sql SQL icon="database"
DROP STORAGE INTEGRATION raw
DROP STORAGE INTEGRATION IF EXISTS raw
```
</CodeGroup>
Returns `dropped`, `true` or `false`. There is no `ALTER STORAGE
INTEGRATION`: to rotate a credential, rotate the Secret it references
(see [Secret](/sql/ddl/secret)) rather than dropping and recreating the
integration. A storage integration an attached database still depends on
blocks the drop; detach the database first.
## GRANT and REVOKE {#grant}
<CodeGroup>
```sql SQL icon="database"
GRANT CREATE STORAGE INTEGRATION ON SCHEMA platform TO ROLE platform_admins
GRANT USAGE ON STORAGE INTEGRATION platform.hub TO ROLE research
GRANT OWNERSHIP ON STORAGE INTEGRATION platform.hub TO ROLE platform_admins
REVOKE USAGE ON STORAGE INTEGRATION platform.hub FROM ROLE research
```
</CodeGroup>
`CREATE_STORAGE_INTEGRATION` on the namespace creates one; `USAGE` on the
integration itself lets a query match and read through it; `OWNERSHIP`
grants full control, including dropping it. See
[Privileges](/governance/privileges#sql) for the full statement syntax,
every grantee form, and how a query picks a matching integration without
naming one.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Inspect Datasets" icon="database" href="/curation/inspect-datasets">
Query and import HuggingFace or object-storage datasets, the most common use of a storage integration today.
</Card>
<Card title="Secret" icon="key" href="/sql/ddl/secret">
The credential storage integration options reference.
</Card>
<Card title="Privileges" icon="key" href="/governance/privileges">
Every privilege, and the grant and revoke syntax that covers each one.
</Card>
</CardGroup>