feat: support Hugging Face storage integrations with optional token Secrets (#4775)

Adds `HUGGINGFACE` and an optional token Secret reference to the
existing storage integration methods. The same SQL transport handles
create, describe, list and drop. Only a Secret reference is stored in
integration metadata; mixed-provider credential options are rejected.

```python
from lancedb.storage_integration import SecretReference

info = db.create_storage_integration(
    "private_hub", "hf://datasets/acme/private/raw/",
    storage_provider="HUGGINGFACE",
    huggingface_token_secret=SecretReference(
        "hf_token", database="shared", namespace_path=("credentials",),
    ),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("private_hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("private_hub")
```

Omit the token Secret for anonymous access at Hugging Face's default
anonymous
rate limits. A tokenless integration does not use a cluster's Hugging
Face token:

```python
info = db.create_storage_integration(
    "public_hub", "hf://datasets/acme/public/raw/",
    storage_provider="HUGGINGFACE",
)
```

```sql
CREATE STORAGE INTEGRATION public_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';
```


```sql
CREATE STORAGE INTEGRATION private_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
    HUGGINGFACE_TOKEN_SECRET = shared.credentials.hf_token;
DESCRIBE STORAGE INTEGRATION private_hub;
SHOW STORAGE INTEGRATIONS;
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
DROP STORAGE INTEGRATION private_hub;
```

```mermaid
flowchart LR
    Python[Sync or async Python] --> SQL[Existing SQL transport]
    SQL --> Metadata[Integration and token Secret reference]
```

Adds Rust request construction, SQL rendering, typed metadata decoding,
sync/async Flight SQL regressions and public documentation. Local
builds/tests are excluded by request; CI provides execution validation.
This commit is contained in:
Jack Ye
2026-10-31 01:56:42 -07:00
committed by GitHub
parent 1fb2dd1551
commit 784d649380
7 changed files with 347 additions and 7 deletions
+69
View File
@@ -1061,3 +1061,72 @@ options. The service interprets the stored JSON when accessing storage; registra
does not validate its credentials format or verify access, and leaves
`access_verified` false. Integration metadata contains the Secret reference, never
its JSON content. GCP options cannot be combined with AWS or Azure options.
### Hugging Face storage integrations
Store a Hugging Face token as one Secret, then register the repository prefix
with the existing integration methods. Only the Secret name is part of the
integration configuration:
```python
import os
from lancedb.storage_integration import SecretReference
db.create_secret("hf_token", os.environ["HF_TOKEN"])
info = db.create_storage_integration(
"private_hub",
"hf://datasets/acme/private/raw/",
storage_provider="HUGGINGFACE",
huggingface_token_secret=SecretReference("hf_token"),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("private_hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("private_hub")
```
The equivalent SQL references the existing Secret:
```sql
CREATE STORAGE INTEGRATION private_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
HUGGINGFACE_TOKEN_SECRET = public.hf_token;
DESCRIBE STORAGE INTEGRATION private_hub;
SHOW STORAGE INTEGRATIONS;
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
DROP STORAGE INTEGRATION private_hub;
```
Omit the token Secret for anonymous access at Hugging Face's default anonymous
rate limits. A tokenless integration does not use a cluster's Hugging Face token:
```python
info = db.create_storage_integration(
"public_hub", "hf://datasets/acme/public/raw/",
storage_provider="HUGGINGFACE",
)
```
```sql
CREATE STORAGE INTEGRATION public_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';
```
When provided, the Secret must contain the token itself, with repository read
permission.
Cross-database and namespace references use the same `SecretReference` rules as
other integrations. Sync and async APIs accept the same options. Registration
returns one sentence of instructions and leaves `access_verified` false; it does
not contact Hugging Face. Metadata contains the Secret reference, never its value.
Hugging Face options cannot be combined with AWS, Azure or GCP options.
Repository prefixes use `hf://datasets/<owner>/<repo>[/<path>]` or
`hf://models/<owner>/<repo>[/<path>]`. An optional `@revision` follows the repository
name; omitting it allows matching reads of any revision. Prefixes must be concrete
paths without wildcards or traversal. This integration does not make Hugging Face
a writable database location for `ATTACH DATABASE`. Dataset SQL sources select a
matching integration automatically, with no integration argument in the query.