mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 15:46:55 +00:00
A broad documentation pass across the areas LanceDB Enterprise and Embedded diverge on most: **Data Curation** -- GPU Function execution, Slurm scheduling (and what Slurm is, for readers new to it), blob late materialization as its own page, joins, aggregation/profiling, sampling, near-duplicate detection, chunking, distributed shuffle, and IVF-partition clustering. **Feature Engineering** -- computed columns and materialized views, restructured to lead with the Function-based path and show the SQL form side by side with Python throughout; a single Auto Refresh page instead of duplicating scheduling details on every refresh page; adaptive execution; checkpointing; GPU and Slurm placement for refresh workers; a Training Cache page for the PyTorch data loader's automatic cache benefit. **Security & Governance** -- an Audit Log reference (schema, access, grants), Secrets promoted to an ordinary catalog object with their own SQL DDL and a vending/rotation guide, and privilege-table fixes verified against the authorization source. **Deployment & Maintenance** -- Horizontal Scaling and OpenTelemetry moved into a real Deployment area; Automated Cleanup folded into Intelligent Auto-Maintenance (every other maintenance decision already lived there); RTE Compaction and Advanced Conflict Resolution documented in depth. **Landing page** -- the Enterprise feature comparison table reorganized (several redundant or stale rows removed, several new areas added), and the "why Enterprise" bullets replaced with ones tied to a concrete page (Performance, Scale, Fault tolerance, Multicloud deployment, Simplify operations) instead of vague promises. Plus a full pass addressing lancedb-gatekeeper's review: a real test-safety bug (a quickstart test could silently overwrite and delete a deployment's own `frames` table), a fabricated SQL statement with no backing implementation, an actual dedup-guarantee bug in `take_row_ids`, and dozens of documentation-accuracy fixes verified against source rather than taken on faith -- each either fixed, or left with an explicit note on why it needs a feature or deployment owner's sign-off instead of being guessed at. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
137 lines
5.7 KiB
Plaintext
137 lines
5.7 KiB
Plaintext
---
|
|
title: "Storage Integration"
|
|
sidebarTitle: "Storage Integration"
|
|
description: "A namespace-scoped, reusable credential for reading or importing from storage outside the deployment's own tables."
|
|
icon: cloud
|
|
keywords: ["storage integration", "s3", "gcs", "azure", "huggingface", "external storage", "credentials"]
|
|
availability:
|
|
embedded: unavailable
|
|
enterprise: available
|
|
---
|
|
|
|
A **storage integration** is a namespace-scoped catalog object, like a
|
|
[Secret](/management/secret), that names a credential for reading or
|
|
importing from one external storage location: an S3, GCS or Azure prefix,
|
|
or a HuggingFace dataset repository. A query or import statement that
|
|
reads a matching URI uses the integration automatically -- it is never
|
|
named in the query itself.
|
|
|
|
## Create and manage {#manage}
|
|
|
|
Every provider takes a `storage_prefix` and `storage_provider`, plus
|
|
provider-specific credential options. Credential options take a
|
|
[Secret](/management/secret) reference, never a value directly -- either a
|
|
bare name (resolved in the caller's own database, root namespace) or a
|
|
`SecretReference` qualified with a database and namespace path:
|
|
|
|
<CodeGroup>
|
|
```sql SQL icon="database"
|
|
CREATE STORAGE INTEGRATION hub
|
|
STORAGE_PROVIDER = 'HUGGINGFACE'
|
|
STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
|
|
HUGGINGFACE_TOKEN_SECRET = bootstrap.credentials.hf_token
|
|
|
|
DESCRIBE STORAGE INTEGRATION hub
|
|
|
|
SHOW STORAGE INTEGRATIONS
|
|
|
|
DROP STORAGE INTEGRATION hub
|
|
```
|
|
</CodeGroup>
|
|
|
|
{/*
|
|
Draft for the storage-integration SDK release (#4618, #4775). Keep this
|
|
section unpublished until create_storage_integration and its siblings ship
|
|
in a release; they are not in v0.41.0.
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
from lancedb.storage_integration import SecretReference
|
|
|
|
info = db.create_storage_integration(
|
|
"hub", "hf://datasets/owner/private/data/",
|
|
storage_provider="HUGGINGFACE",
|
|
huggingface_token_secret=SecretReference(
|
|
"hf_token", database="bootstrap", namespace_path=("credentials",),
|
|
),
|
|
)
|
|
print(info.configuration_instructions)
|
|
|
|
info = db.describe_storage_integration("hub")
|
|
names = list(db.list_storage_integrations())
|
|
db.drop_storage_integration("hub")
|
|
```
|
|
</CodeGroup>
|
|
*/}
|
|
|
|
See [Storage Integration](/sql/ddl/storage-integration) for every
|
|
provider's exact options, S3's two credential modes, and the full output
|
|
schema -- this page covers the CRUD operations above at a glance; that one
|
|
is the complete reference.
|
|
|
|
| Operation | Python (not yet released) | SQL | Notes |
|
|
| --- | --- | --- | --- |
|
|
| Create | `create_storage_integration` | `CREATE STORAGE INTEGRATION` | Fails if the name is taken, so a create never silently replaces one. |
|
|
| Describe | `describe_storage_integration` | `DESCRIBE` / `DESC` | One row: provider, prefix, credential references, and (for an AWS role integration) the generated trust policy. |
|
|
| List | `list_storage_integrations` | `SHOW` / `LIST STORAGE INTEGRATIONS` | Optionally scoped to one namespace. |
|
|
| Drop | `drop_storage_integration` | `DROP [IF EXISTS]` | Blocked while an attached database still depends on it. |
|
|
|
|
The Python column names the methods landing in a future release (see the
|
|
draft note above); use SQL today. There is no TypeScript or REST
|
|
equivalent yet either.
|
|
|
|
There is no `ALTER`. To rotate a credential, rotate the
|
|
[Secret](/management/secret#manage) it references -- the integration's
|
|
reference does not change, so nothing that reads through it needs updating.
|
|
|
|
## The S3 trust workflow {#trust}
|
|
|
|
An S3 integration created with `STORAGE_AWS_ROLE_ARN` (instead of a static
|
|
access key pair) needs one more step outside LanceDB: `DESCRIBE` returns
|
|
`trust_policy` and `permissions_policy`, the IAM policy documents your role
|
|
needs, built from `aws_account_id`, `aws_iam_principal_arn` and
|
|
`aws_external_id` in the same row. Attach `trust_policy` as the role's
|
|
trust relationship and `permissions_policy` as its permissions, then check
|
|
`access_verified` on a later `DESCRIBE` to confirm LanceDB can assume the
|
|
role and read the prefix.
|
|
|
|
## Privileges {#privileges}
|
|
|
|
A storage integration is its own object in the authorization model, a
|
|
namespace child like a Secret: `CREATE_STORAGE_INTEGRATION` on the
|
|
namespace to create one, `USAGE` on the integration for a query to read
|
|
through it, `OWNERSHIP` for full control including dropping it. See
|
|
[Privileges](/governance/privileges#sql) for the privilege each operation
|
|
requires and the full `GRANT`/`REVOKE` syntax.
|
|
|
|
## Troubleshooting {#troubleshooting}
|
|
|
|
**A query does not use the integration you expected**: integrations match
|
|
by the most specific `STORAGE_PREFIX` covering the URI, not by name, and
|
|
are scoped to native databases under the catalog root. Two integrations
|
|
with equally specific prefixes make the query fail rather than guess; keep
|
|
prefixes for the same bucket or dataset family unambiguous.
|
|
|
|
**403 or access-denied reading a matching prefix**: confirm `USAGE` on the
|
|
integration, and for an S3 role integration, confirm `access_verified` is
|
|
`true` on `DESCRIBE` -- `false` usually means the trust policy from
|
|
[above](#trust) is missing or misconfigured on the AWS side, not a LanceDB
|
|
permission problem.
|
|
|
|
**`DROP` refuses with a dependent database**: detach the database that
|
|
still uses the integration first, then drop it.
|
|
|
|
## What to read next {#next}
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Storage Integration" icon="cloud" href="/sql/ddl/storage-integration">
|
|
The complete SQL reference: every provider's options, output columns, and GRANT syntax.
|
|
</Card>
|
|
<Card title="Inspect Datasets" icon="database" href="/curation/inspect-datasets">
|
|
The most common use of a storage integration today: inspecting and importing HuggingFace datasets.
|
|
</Card>
|
|
<Card title="Secret" icon="key" href="/management/secret">
|
|
The credential a storage integration's `_SECRET` options reference.
|
|
</Card>
|
|
</CardGroup>
|