mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
feat: support Hugging Face storage integrations with optional token Secrets (#4775)
Adds `HUGGINGFACE` and an optional token Secret reference to the
existing storage integration methods. The same SQL transport handles
create, describe, list and drop. Only a Secret reference is stored in
integration metadata; mixed-provider credential options are rejected.
```python
from lancedb.storage_integration import SecretReference
info = db.create_storage_integration(
"private_hub", "hf://datasets/acme/private/raw/",
storage_provider="HUGGINGFACE",
huggingface_token_secret=SecretReference(
"hf_token", database="shared", namespace_path=("credentials",),
),
)
print(info.configuration_instructions)
info = db.describe_storage_integration("private_hub")
names = list(db.list_storage_integrations())
db.drop_storage_integration("private_hub")
```
Omit the token Secret for anonymous access at Hugging Face's default
anonymous
rate limits. A tokenless integration does not use a cluster's Hugging
Face token:
```python
info = db.create_storage_integration(
"public_hub", "hf://datasets/acme/public/raw/",
storage_provider="HUGGINGFACE",
)
```
```sql
CREATE STORAGE INTEGRATION public_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';
```
```sql
CREATE STORAGE INTEGRATION private_hub
STORAGE_PROVIDER = 'HUGGINGFACE'
STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
HUGGINGFACE_TOKEN_SECRET = shared.credentials.hf_token;
DESCRIBE STORAGE INTEGRATION private_hub;
SHOW STORAGE INTEGRATIONS;
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
DROP STORAGE INTEGRATION private_hub;
```
```mermaid
flowchart LR
Python[Sync or async Python] --> SQL[Existing SQL transport]
SQL --> Metadata[Integration and token Secret reference]
```
Adds Rust request construction, SQL rendering, typed metadata decoding,
sync/async Flight SQL regressions and public documentation. Local
builds/tests are excluded by request; CI provides execution validation.
This commit is contained in:
@@ -1061,3 +1061,72 @@ options. The service interprets the stored JSON when accessing storage; registra
|
||||
does not validate its credentials format or verify access, and leaves
|
||||
`access_verified` false. Integration metadata contains the Secret reference, never
|
||||
its JSON content. GCP options cannot be combined with AWS or Azure options.
|
||||
|
||||
|
||||
### Hugging Face storage integrations
|
||||
|
||||
Store a Hugging Face token as one Secret, then register the repository prefix
|
||||
with the existing integration methods. Only the Secret name is part of the
|
||||
integration configuration:
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from lancedb.storage_integration import SecretReference
|
||||
|
||||
db.create_secret("hf_token", os.environ["HF_TOKEN"])
|
||||
info = db.create_storage_integration(
|
||||
"private_hub",
|
||||
"hf://datasets/acme/private/raw/",
|
||||
storage_provider="HUGGINGFACE",
|
||||
huggingface_token_secret=SecretReference("hf_token"),
|
||||
)
|
||||
print(info.configuration_instructions)
|
||||
info = db.describe_storage_integration("private_hub")
|
||||
names = list(db.list_storage_integrations())
|
||||
db.drop_storage_integration("private_hub")
|
||||
```
|
||||
|
||||
The equivalent SQL references the existing Secret:
|
||||
|
||||
```sql
|
||||
CREATE STORAGE INTEGRATION private_hub
|
||||
STORAGE_PROVIDER = 'HUGGINGFACE'
|
||||
STORAGE_PREFIX = 'hf://datasets/acme/private/raw/'
|
||||
HUGGINGFACE_TOKEN_SECRET = public.hf_token;
|
||||
DESCRIBE STORAGE INTEGRATION private_hub;
|
||||
SHOW STORAGE INTEGRATIONS;
|
||||
SELECT * FROM 'hf://datasets/acme/private/raw/*.parquet' LIMIT 10;
|
||||
DROP STORAGE INTEGRATION private_hub;
|
||||
```
|
||||
|
||||
Omit the token Secret for anonymous access at Hugging Face's default anonymous
|
||||
rate limits. A tokenless integration does not use a cluster's Hugging Face token:
|
||||
|
||||
```python
|
||||
info = db.create_storage_integration(
|
||||
"public_hub", "hf://datasets/acme/public/raw/",
|
||||
storage_provider="HUGGINGFACE",
|
||||
)
|
||||
```
|
||||
|
||||
```sql
|
||||
CREATE STORAGE INTEGRATION public_hub
|
||||
STORAGE_PROVIDER = 'HUGGINGFACE'
|
||||
STORAGE_PREFIX = 'hf://datasets/acme/public/raw/';
|
||||
```
|
||||
|
||||
When provided, the Secret must contain the token itself, with repository read
|
||||
permission.
|
||||
Cross-database and namespace references use the same `SecretReference` rules as
|
||||
other integrations. Sync and async APIs accept the same options. Registration
|
||||
returns one sentence of instructions and leaves `access_verified` false; it does
|
||||
not contact Hugging Face. Metadata contains the Secret reference, never its value.
|
||||
Hugging Face options cannot be combined with AWS, Azure or GCP options.
|
||||
|
||||
Repository prefixes use `hf://datasets/<owner>/<repo>[/<path>]` or
|
||||
`hf://models/<owner>/<repo>[/<path>]`. An optional `@revision` follows the repository
|
||||
name; omitting it allows matching reads of any revision. Prefixes must be concrete
|
||||
paths without wildcards or traversal. This integration does not make Hugging Face
|
||||
a writable database location for `ATTACH DATABASE`. Dataset SQL sources select a
|
||||
matching integration automatically, with no integration argument in the query.
|
||||
|
||||
Reference in New Issue
Block a user