Files
lancedb/docs/web/management/storage-integration.mdx
T
Jack YeandClaude Sonnet 5 de95ed3f6d docs: overhaul curation, feature engineering, security and deployment guides (#4748)
A broad documentation pass across the areas LanceDB Enterprise and
Embedded
diverge on most:

**Data Curation** -- GPU Function execution, Slurm scheduling (and what
Slurm is, for readers new to it), blob late materialization as its own
page, joins, aggregation/profiling, sampling, near-duplicate detection,
chunking, distributed shuffle, and IVF-partition clustering.

**Feature Engineering** -- computed columns and materialized views,
restructured to lead with the Function-based path and show the SQL form
side by side with Python throughout; a single Auto Refresh page instead
of
duplicating scheduling details on every refresh page; adaptive
execution;
checkpointing; GPU and Slurm placement for refresh workers; a Training
Cache page for the PyTorch data loader's automatic cache benefit.

**Security & Governance** -- an Audit Log reference (schema, access,
grants), Secrets promoted to an ordinary catalog object with their own
SQL
DDL and a vending/rotation guide, and privilege-table fixes verified
against the authorization source.

**Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry
moved
into a real Deployment area; Automated Cleanup folded into Intelligent
Auto-Maintenance (every other maintenance decision already lived there);
RTE Compaction and Advanced Conflict Resolution documented in depth.

**Landing page** -- the Enterprise feature comparison table reorganized
(several redundant or stale rows removed, several new areas added), and
the
"why Enterprise" bullets replaced with ones tied to a concrete page
(Performance, Scale, Fault tolerance, Multicloud deployment, Simplify
operations) instead of vague promises.

Plus a full pass addressing lancedb-gatekeeper's review: a real
test-safety
bug (a quickstart test could silently overwrite and delete a
deployment's
own `frames` table), a fabricated SQL statement with no backing
implementation, an actual dedup-guarantee bug in `take_row_ids`, and
dozens
of documentation-accuracy fixes verified against source rather than
taken
on faith -- each either fixed, or left with an explicit note on why it
needs a feature or deployment owner's sign-off instead of being guessed
at.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-11-03 16:17:24 -08:00

137 lines
5.7 KiB
Plaintext

---
title: "Storage Integration"
sidebarTitle: "Storage Integration"
description: "A namespace-scoped, reusable credential for reading or importing from storage outside the deployment's own tables."
icon: cloud
keywords: ["storage integration", "s3", "gcs", "azure", "huggingface", "external storage", "credentials"]
availability:
embedded: unavailable
enterprise: available
---
A **storage integration** is a namespace-scoped catalog object, like a
[Secret](/management/secret), that names a credential for reading or
importing from one external storage location: an S3, GCS or Azure prefix,
or a HuggingFace dataset repository. A query or import statement that
reads a matching URI uses the integration automatically -- it is never
named in the query itself.
## Create and manage {#manage}
Every provider takes a `storage_prefix` and `storage_provider`, plus
provider-specific credential options. Credential options take a
[Secret](/management/secret) reference, never a value directly -- either a
bare name (resolved in the caller's own database, root namespace) or a
`SecretReference` qualified with a database and namespace path:
<CodeGroup>
```sql SQL icon="database"
CREATE STORAGE INTEGRATION hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
HUGGINGFACE_TOKEN_SECRET = bootstrap.credentials.hf_token
DESCRIBE STORAGE INTEGRATION hub
SHOW STORAGE INTEGRATIONS
DROP STORAGE INTEGRATION hub
```
</CodeGroup>
{/*
Draft for the storage-integration SDK release (#4618, #4775). Keep this
section unpublished until create_storage_integration and its siblings ship
in a release; they are not in v0.41.0.
<CodeGroup>
```python Python icon="python"
from lancedb.storage_integration import SecretReference
info = db.create_storage_integration(
"hub", "hf://datasets/owner/private/data/",
storage_provider="HUGGINGFACE",
huggingface_token_secret=SecretReference(
"hf_token", database="bootstrap", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("hub")
```
</CodeGroup>
*/}
See [Storage Integration](/sql/ddl/storage-integration) for every
provider's exact options, S3's two credential modes, and the full output
schema -- this page covers the CRUD operations above at a glance; that one
is the complete reference.
| Operation | Python (not yet released) | SQL | Notes |
| --- | --- | --- | --- |
| Create | `create_storage_integration` | `CREATE STORAGE INTEGRATION` | Fails if the name is taken, so a create never silently replaces one. |
| Describe | `describe_storage_integration` | `DESCRIBE` / `DESC` | One row: provider, prefix, credential references, and (for an AWS role integration) the generated trust policy. |
| List | `list_storage_integrations` | `SHOW` / `LIST STORAGE INTEGRATIONS` | Optionally scoped to one namespace. |
| Drop | `drop_storage_integration` | `DROP [IF EXISTS]` | Blocked while an attached database still depends on it. |
The Python column names the methods landing in a future release (see the
draft note above); use SQL today. There is no TypeScript or REST
equivalent yet either.
There is no `ALTER`. To rotate a credential, rotate the
[Secret](/management/secret#manage) it references -- the integration's
reference does not change, so nothing that reads through it needs updating.
## The S3 trust workflow {#trust}
An S3 integration created with `STORAGE_AWS_ROLE_ARN` (instead of a static
access key pair) needs one more step outside LanceDB: `DESCRIBE` returns
`trust_policy` and `permissions_policy`, the IAM policy documents your role
needs, built from `aws_account_id`, `aws_iam_principal_arn` and
`aws_external_id` in the same row. Attach `trust_policy` as the role's
trust relationship and `permissions_policy` as its permissions, then check
`access_verified` on a later `DESCRIBE` to confirm LanceDB can assume the
role and read the prefix.
## Privileges {#privileges}
A storage integration is its own object in the authorization model, a
namespace child like a Secret: `CREATE_STORAGE_INTEGRATION` on the
namespace to create one, `USAGE` on the integration for a query to read
through it, `OWNERSHIP` for full control including dropping it. See
[Privileges](/governance/privileges#sql) for the privilege each operation
requires and the full `GRANT`/`REVOKE` syntax.
## Troubleshooting {#troubleshooting}
**A query does not use the integration you expected**: integrations match
by the most specific `STORAGE_PREFIX` covering the URI, not by name, and
are scoped to native databases under the catalog root. Two integrations
with equally specific prefixes make the query fail rather than guess; keep
prefixes for the same bucket or dataset family unambiguous.
**403 or access-denied reading a matching prefix**: confirm `USAGE` on the
integration, and for an S3 role integration, confirm `access_verified` is
`true` on `DESCRIBE` -- `false` usually means the trust policy from
[above](#trust) is missing or misconfigured on the AWS side, not a LanceDB
permission problem.
**`DROP` refuses with a dependent database**: detach the database that
still uses the integration first, then drop it.
## What to read next {#next}
<CardGroup cols={2}>
<Card title="Storage Integration" icon="cloud" href="/sql/ddl/storage-integration">
The complete SQL reference: every provider's options, output columns, and GRANT syntax.
</Card>
<Card title="Inspect Datasets" icon="database" href="/curation/inspect-datasets">
The most common use of a storage integration today: inspecting and importing HuggingFace datasets.
</Card>
<Card title="Secret" icon="key" href="/management/secret">
The credential a storage integration's `_SECRET` options reference.
</Card>
</CardGroup>