mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 15:22:30 +00:00
Jack Ye
b21fb2a4dc
The pages describing this code lived in lancedb/docs, a repository with ten stars that neither contributing guide mentioned. A user who hit a wrong page while reading the source had to find a separate, near-invisible repository to fix it. These 102 pages now sit beside the code, and a documentation fix is a pull request in the repository the reader already had open. `docs/web` is a complete Mintlify site: `cd docs/web && mint dev` serves it with no other checkout. Enterprise pages stay private and are merged in when the published site is assembled, so links into them do not resolve here — every open-source page does. `docs/web-tests` holds the tests the code examples are extracted from. They ran against a published wheel before; here they can run against this repository's own SDK, so an example that stops compiling fails the pull request that broke it rather than being noticed later. Both directories sit beside `docs/src` and are invisible to it: mkdocs reads `docs_dir: src`, so the SDK reference builds and deploys exactly as before.
342 lines
13 KiB
Plaintext
342 lines
13 KiB
Plaintext
---
|
|
title: "Configuring Cloud Storage in LanceDB"
|
|
sidebarTitle: "Configuring storage"
|
|
description: "Configure LanceDB to use S3, GCS, Azure Blob, and S3-compatible object stores with environment variables or storage options."
|
|
icon: "wrench"
|
|
---
|
|
import {
|
|
PyStorageAzureAccount,
|
|
PyStorageAzureSas,
|
|
PyStorageConnectAzure,
|
|
PyStorageConnectGcs,
|
|
PyStorageConnectS3,
|
|
PyStorageConnectTimeout,
|
|
PyStorageGcsServiceAccount,
|
|
PyStorageS3Express,
|
|
PyStorageS3Minio,
|
|
PyStorageS3SseKms,
|
|
PyStorageTableTimeout,
|
|
PyStorageTigrisConnect,
|
|
PyStorageCosConnect,
|
|
PyStorageGoosefsConnect,
|
|
TsStorageAzureAccount,
|
|
TsStorageAzureSas,
|
|
TsStorageConnectAzure,
|
|
TsStorageConnectGcs,
|
|
TsStorageConnectS3,
|
|
TsStorageConnectTimeout,
|
|
TsStorageGcsServiceAccount,
|
|
TsStorageS3Express,
|
|
TsStorageS3Minio,
|
|
TsStorageS3SseKms,
|
|
TsStorageTableTimeout,
|
|
TsStorageTigrisConnect,
|
|
TsStorageGoosefsConnect,
|
|
} from '/snippets/storage.mdx';
|
|
|
|
When using LanceDB OSS, you can choose where to store your data. The tradeoffs between storage options are covered in the [storage architecture guide](/storage). This page shows how to configure each backend.
|
|
|
|
<Note>
|
|
**LanceDB Enterprise storage configuration**
|
|
|
|
In LanceDB Enterprise, you connect with `db://...` and the cluster owns the storage credentials, so `storage_options` are not passed at runtime. Cloud auth is set at deployment time. For federated databases, the namespace service vends per-request credentials automatically. See the [quickstart](/quickstart), [Enterprise overview](/enterprise/), and [Azure deployment guide](/enterprise/deployment/azure) for the Enterprise flow.
|
|
</Note>
|
|
|
|
## Object stores {#object-stores}
|
|
|
|
LanceDB supports AWS S3 (and compatible stores), Azure Blob Storage, and Google Cloud Storage. The URI scheme in your `connect` call selects the backend.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageConnectS3}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageConnectS3}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageConnectGcs}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageConnectGcs}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageConnectAzure}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageConnectAzure}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
### Configuration options {#configuration-options}
|
|
|
|
When running inside the target cloud with correct IAM bindings, LanceDB often needs no extra configuration. When running elsewhere, provide credentials via environment variables or `storage_options`.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageConnectTimeout}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageConnectTimeout}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<Info>
|
|
**Storage option casing**
|
|
|
|
Keys are case-insensitive. Use lowercase in `storage_options` and uppercase in environment variables.
|
|
</Info>
|
|
|
|
Table-level `storage_options` inherit every key from the connection and override on a per-key basis. Pass them to `create_table` or `open_table` for options that should apply to a single table:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageTableTimeout}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageTableTimeout}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
<Tip>
|
|
**Inspect the effective options**
|
|
|
|
On `AsyncTable`, `await table.initial_storage_options()` returns the options the table was opened with, and `await table.latest_storage_options()` returns the current options after any provider-driven refresh. The deprecated `table.storage_options()` method will be removed in a future release.
|
|
</Tip>
|
|
|
|
#### General object store options {#general-object-store-options}
|
|
|
|
| Key | Description |
|
|
| :-- | :-- |
|
|
| `allow_http` | Allow non-TLS connections. |
|
|
| `allow_invalid_certificates` | Skip certificate validation for TLS connections. |
|
|
| `connect_timeout` | Timeout for the connect phase. |
|
|
| `timeout` | Timeout for the full request. |
|
|
| `user_agent` | User agent string sent with requests. |
|
|
| `proxy_url` | Proxy URL to route requests through. |
|
|
| `proxy_ca_certificate` | PEM-formatted CA certificate for proxy connections. |
|
|
| `proxy_excludes` | Comma-separated hosts that bypass the proxy (domains or CIDR). |
|
|
| `download_retry_count` | Number of retries when downloading objects. |
|
|
| `client_max_retries` | Maximum retries for object-store client requests. |
|
|
| `client_retry_timeout` | Total retry timeout (seconds) for object-store client requests. |
|
|
|
|
<Info>
|
|
**Option support varies by backend**
|
|
|
|
These are commonly used options. Cloud-specific keys (for example `region`, `endpoint`, `service_account`, and Azure credential keys) are backend-dependent and can be provided in `storage_options` as needed.
|
|
</Info>
|
|
|
|
#### New table configuration {#new-table-configuration}
|
|
|
|
These options control the Lance file format and features used when creating new tables. Pass them via `storage_options` at connection or table level. They are evaluated only at table creation; setting them on an existing connection does not rewrite or alter tables that already exist.
|
|
|
|
| Key | Values | Default | Description |
|
|
| :-- | :-- | :-- | :-- |
|
|
| `new_table_data_storage_version` | `legacy`, `stable` | `stable` | Lance file format version for new tables. Use `legacy` for backward compatibility with older clients, or `stable` for the current format with better performance. |
|
|
| `new_table_enable_v2_manifest_paths` | `true`, `false` | `false` | Use v2 manifest path naming. Requires LanceDB >= 0.10.0 to read. |
|
|
| `new_table_enable_stable_row_ids` | `true`, `false` | `false` | Keep row IDs stable across compaction, delete, and merge operations. |
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
A[Creating a new table] --> B{Need backward compatibility\nwith older LanceDB clients?}
|
|
B -->|Yes| C[new_table_data_storage_version: legacy]
|
|
B -->|No| D[new_table_data_storage_version: stable\nDefault — recommended]
|
|
D --> E{Need stable row IDs\nacross compaction and deletes?}
|
|
E -->|Yes| F[new_table_enable_stable_row_ids: true]
|
|
E -->|No| G[Default: false]
|
|
D --> H{All clients on LanceDB >= 0.10.0?}
|
|
H -->|Yes| I[new_table_enable_v2_manifest_paths: true]
|
|
H -->|No| J[Default: false]
|
|
```
|
|
|
|
<CodeGroup>
|
|
```python Python icon="python"
|
|
import lancedb
|
|
|
|
# Set the Lance file format version at connection level
|
|
db = lancedb.connect(
|
|
"s3://bucket/path",
|
|
storage_options={
|
|
"new_table_data_storage_version": "stable",
|
|
},
|
|
)
|
|
```
|
|
|
|
```typescript TypeScript icon="square-js"
|
|
import * as lancedb from "@lancedb/lancedb";
|
|
|
|
// Set the Lance file format version at connection level
|
|
const db = await lancedb.connect("s3://bucket/path", {
|
|
storageOptions: {
|
|
newTableDataStorageVersion: "stable",
|
|
},
|
|
});
|
|
```
|
|
</CodeGroup>
|
|
|
|
<Warning>
|
|
**Deprecated parameter**
|
|
|
|
The `data_storage_version` parameter on `create_table()` is deprecated. Use `new_table_data_storage_version` in `storage_options` instead.
|
|
</Warning>
|
|
|
|
## AWS S3 {#aws-s3}
|
|
|
|
|
|
|
|
Set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and optionally `AWS_SESSION_TOKEN` as environment variables or pass them in `storage_options`. Region is optional for AWS but required for most S3-compatible stores.
|
|
|
|
Minimum permissions usually include `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject`, `s3:ListBucket`, and `s3:GetBucketLocation` scoped to the relevant bucket/prefix.
|
|
|
|
### S3-compatible stores {#s3-compatible-stores}
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageS3Minio}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageS3Minio}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
If the endpoint is `http://` (common in local development), also set `ALLOW_HTTP=true` or pass `allow_http=True` in `storage_options`.
|
|
|
|
### S3 Express {#s3-express}
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageS3Express}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageS3Express}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Consult AWS networking requirements for S3 Express before enabling.
|
|
|
|
<Tip>
|
|
**Clean up failed multipart uploads**
|
|
|
|
LanceDB aborts multipart uploads on graceful shutdown, but crashes can leave incomplete uploads. Add an S3 lifecycle rule to delete in-progress uploads after a few days.
|
|
</Tip>
|
|
|
|
### Server-side encryption with KMS {#server-side-encryption-with-kms}
|
|
|
|
To encrypt at rest with an AWS KMS key, set `aws_server_side_encryption` to `aws:kms` and `aws_sse_kms_key_id` to the key ID or ARN. The same options apply at connection or table level and combine with bucket-level default encryption.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageS3SseKms}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageS3SseKms}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
The IAM principal needs `kms:Encrypt`, `kms:Decrypt`, and `kms:GenerateDataKey` on the configured KMS key.
|
|
|
|
## Google Cloud Storage {#google-cloud-storage}
|
|
|
|
|
|
|
|
Provide credentials via `GOOGLE_SERVICE_ACCOUNT` (path to JSON) or include the path in `storage_options`. GCS defaults to HTTP/1; set `HTTP1_ONLY=false` if you need HTTP/2.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageGcsServiceAccount}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageGcsServiceAccount}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
## Azure Blob Storage {#azure-blob-storage}
|
|
|
|
|
|
|
|
Set `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCOUNT_KEY` as environment variables, or pass them via `storage_options`.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageAzureAccount}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageAzureAccount}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
For SAS-token auth, set `azure_storage_account_name` and `azure_storage_sas_token`:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageAzureSas}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageAzureSas}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Other supported keys include service principal credentials (`azure_client_id`, `azure_client_secret`, `azure_tenant_id`), managed identities, and custom endpoints.
|
|
|
|
## Tigris Object Storage {#tigris-object-storage}
|
|
|
|
|
|
|
|
Tigris exposes an S3-compatible API. Configure the endpoint and region:
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageTigrisConnect}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageTigrisConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Environment variables `AWS_ENDPOINT=https://t3.storage.dev` and `AWS_DEFAULT_REGION=auto` achieve the same configuration.
|
|
|
|
## Tencent COS {#tencent-cos}
|
|
|
|
[Tencent Cloud Object Storage (COS)](https://www.tencentcloud.com/products/cos) is the primary object store for workloads running in the China region. Use the `cos://` URI scheme to connect directly to a COS bucket.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageCosConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
Supported keys include `secret_id`, `secret_key`, `region`, and `endpoint`. You can also authenticate via the `TENCENTCLOUD_SECRET_ID` and `TENCENTCLOUD_SECRET_KEY` environment variables.
|
|
|
|
<Note>
|
|
**Availability**
|
|
|
|
COS is bundled in the Python wheel by default. To use it from Rust or the Node binding, build with the `cos` Cargo feature enabled.
|
|
</Note>
|
|
|
|
## GooseFS {#goosefs}
|
|
|
|
[GooseFS](https://www.tencentcloud.com/document/product/1424) is Tencent Cloud's distributed cache acceleration layer for COS and S3. It is a common choice when the same hot dataset is read repeatedly, such as vector search and AI training workloads. Connect using the `goosefs://` URI scheme.
|
|
|
|
<CodeGroup>
|
|
<CodeBlock filename="Python" language="Python" icon="python">
|
|
{PyStorageGoosefsConnect}
|
|
</CodeBlock>
|
|
<CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
|
|
{TsStorageGoosefsConnect}
|
|
</CodeBlock>
|
|
</CodeGroup>
|
|
|
|
GooseFS reads credentials and endpoint configuration from the GooseFS client environment. See the [GooseFS documentation](https://www.tencentcloud.com/document/product/1424) for cluster setup.
|
|
|
|
<Note>
|
|
**Availability**
|
|
|
|
GooseFS is bundled by default in the Python wheel and the Node binding. To use it from Rust, build with the `goosefs` Cargo feature enabled.
|
|
</Note>
|
|
|