diff --git a/docs/web/storage/monitoring.mdx b/docs/web/deployment/monitoring.mdx similarity index 100% rename from docs/web/storage/monitoring.mdx rename to docs/web/deployment/monitoring.mdx diff --git a/docs/web/storage/configuration.mdx b/docs/web/deployment/storage.mdx similarity index 70% rename from docs/web/storage/configuration.mdx rename to docs/web/deployment/storage.mdx index 1d9074ca9..fd8550ea6 100644 --- a/docs/web/storage/configuration.mdx +++ b/docs/web/deployment/storage.mdx @@ -1,9 +1,81 @@ --- -title: "Configuring Cloud Storage in LanceDB" -sidebarTitle: "Configuration" -description: "Configure LanceDB to use S3, GCS, Azure Blob, and S3-compatible object stores with environment variables or storage options." -icon: "wrench" +title: "Storage" +sidebarTitle: "Storage" +description: "Choosing and configuring where LanceDB stores its data -- how you deploy an embedded database." +icon: database +keywords: ["storage", "s3", "gcs", "azure", "object store", "configuration", "deploy"] --- + +LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs). + +Choosing a backend is a balance between latency, scalability, cost, and operational complexity. Use this guide to pick the right fit for your workload. + +## Choosing a backend {#choosing} + +![](/static/assets/images/storage/lancedb_storage_tradeoffs.png) + +When architecting your system, ask yourself: + +- **Latency**: How fast do I need results? What do the p50 and p95 look like? +- **Scalability**: Can I scale data volume and QPS easily? +- **Cost**: What is the all-in cost of storage plus serving? +- **Reliability/Availability**: How will replication and disaster recovery work? + +### Comparison {#comparison} + +Below is a high-level comparison ordered from lowest cost to lowest latency. + +### 1\. Object storage (S3, GCS, Azure Blob) {#1-object-storage-s3-gcs-azure-blob} +- **Latency**: Highest; expect hundreds of milliseconds and higher p95. +- **Scalability**: Effectively unlimited storage; QPS bound by concurrency limits. +- **Cost**: Lowest overall. +- **Reliability/Availability**: Highly available, backed by cloud SLAs. + +LanceDB separates storage and compute and writes immutable fragments, making it a strong fit for stateless, horizontally scalable deployments. + + +**Concurrent writers on S3** + +S3 and S3 Express now support atomic writes natively, so LanceDB handles concurrent writers against the same table out-of-the-box — no external commit coordinator is required. Bucket-level [server-side encryption with KMS](/storage/configuration#server-side-encryption-with-kms) and [S3 Express One Zone](/storage/configuration#s3-express) are also supported on this tier. + + +### 2\. File storage (EFS, GCS Filestore, Azure File) {#2-file-storage-efs-gcs-filestore-azure-file} +- **Latency**: Better than object storage; p95 under ~<100ms is typical. +- **Scalability**: High, but limited by provisioned IOPS per volume. +- **Cost**: More than object storage but cheaper than in-memory options; cold data can tier down automatically. +- **Reliability/Availability**: Highly available; replication/backup must be managed separately. + +Keep a copy of data in object storage for disaster recovery. If zero downtime is required, provision a second network file system with replicated data. + +### 3\. Third-party storage (e.g., MinIO, WekaFS) {#3-third-party-storage-e-g--minio-wekafs} + +- **Latency**: Similar to EFS; typically under <100ms. +- **Scalability**: Determined by the chosen vendor’s cluster sizing. +- **Cost**: Higher than S3; may edge above EFS at larger scales. +- **Reliability/Availability**: Shareable across many nodes; replication depends on vendor capabilities. + +### 4\. Block storage (EBS, GCP Persistent Disk, Azure Managed Disk) {#4-block-storage-ebs-gcp-persistent-disk-azure-managed-disk} +- **Latency**: Near-local performance; often <30ms. +- **Scalability**: Not shareable across instances; shard or copy data when scaling. +- **Cost**: Higher than networked file systems, plus potential I/O charges. +- **Reliability/Availability**: Persists through instance restarts; backups and sharding must be managed. + +### 5\. Local storage (SSD, NVMe) {#5-local-storage-ssd-nvme} +- **Latency**: Fastest; p95 often under <10ms. +- **Scalability**: Hard to scale in cloud environments; requires sharding or additional copies for higher QPS. +- **Cost**: Highest; tightly coupling compute and storage makes horizontal scaling difficult. +- **Reliability/Availability**: Data is tied to the instance; backups must be rigorous. + +Use local disk only when you need extremely low latency and are comfortable owning the operational overhead. + +### File-format choices that interact with the backend {#file-format-choices} + +A few `storage_options` keys shape new tables in ways that depend on the backend you picked above. They are documented in full on [below](#new-table-configuration); the architecture-level summary is: + +- `new_table_enable_v2_manifest_paths` matters most on object stores, where opening a table with many versions is dominated by listing cost. Leave it off for backward compatibility with clients older than LanceDB 0.10.0. +- `new_table_enable_stable_row_ids` keeps row IDs stable across compaction, delete, and merge. The choice is independent of the backend but affects any system that joins on row ID. +- `new_table_data_storage_version` selects the on-disk format. The default `stable` is recommended for all new tables; pick `legacy` only when older readers must keep working. + import { PyStorageAzureAccount, PyStorageAzureSas, @@ -34,7 +106,7 @@ import { TsStorageGoosefsConnect, } from '/snippets/storage.mdx'; -When using LanceDB OSS, you can choose where to store your data. The tradeoffs between storage options are covered in the [storage architecture guide](/storage). This page shows how to configure each backend. +### Configuring a backend {#configuring} **LanceDB Enterprise storage configuration** @@ -42,7 +114,7 @@ When using LanceDB OSS, you can choose where to store your data. The tradeoffs b In LanceDB Enterprise, you connect with `db://...` and the cluster owns the storage credentials, so `storage_options` are not passed at runtime. Cloud auth is set at deployment time. For federated databases, the namespace service vends per-request credentials automatically. See the [quickstart](/quickstart), [Enterprise overview](/enterprise/), and [Azure deployment guide](/enterprise/deployment/azure) for the Enterprise flow. -## Object stores {#object-stores} +### Object stores {#object-stores} LanceDB supports AWS S3 (and compatible stores), Azure Blob Storage, and Google Cloud Storage. The URI scheme in your `connect` call selects the backend. @@ -185,7 +257,7 @@ const db = await lancedb.connect("s3://bucket/path", { The `data_storage_version` parameter on `create_table()` is deprecated. Use `new_table_data_storage_version` in `storage_options` instead. -## AWS S3 {#aws-s3} +### AWS S3 {#aws-s3} ![](/static/assets/images/storage/aws.jpg) @@ -240,7 +312,7 @@ To encrypt at rest with an AWS KMS key, set `aws_server_side_encryption` to `aws The IAM principal needs `kms:Encrypt`, `kms:Decrypt`, and `kms:GenerateDataKey` on the configured KMS key. -## Google Cloud Storage {#google-cloud-storage} +### Google Cloud Storage {#google-cloud-storage} ![](/static/assets/images/storage/gcp.jpg) @@ -255,7 +327,7 @@ Provide credentials via `GOOGLE_SERVICE_ACCOUNT` (path to JSON) or include the p -## Azure Blob Storage {#azure-blob-storage} +### Azure Blob Storage {#azure-blob-storage} ![](/static/assets/images/storage/azure.jpg) @@ -283,7 +355,7 @@ For SAS-token auth, set `azure_storage_account_name` and `azure_storage_sas_toke Other supported keys include service principal credentials (`azure_client_id`, `azure_client_secret`, `azure_tenant_id`), managed identities, and custom endpoints. -## Tigris Object Storage {#tigris-object-storage} +### Tigris Object Storage {#tigris-object-storage} ![](/static/assets/images/storage/tigris.jpg) @@ -300,7 +372,7 @@ Tigris exposes an S3-compatible API. Configure the endpoint and region: Environment variables `AWS_ENDPOINT=https://t3.storage.dev` and `AWS_DEFAULT_REGION=auto` achieve the same configuration. -## Tencent COS {#tencent-cos} +### Tencent COS {#tencent-cos} [Tencent Cloud Object Storage (COS)](https://www.tencentcloud.com/products/cos) is the primary object store for workloads running in the China region. Use the `cos://` URI scheme to connect directly to a COS bucket. @@ -318,7 +390,7 @@ Supported keys include `secret_id`, `secret_key`, `region`, and `endpoint`. You COS is bundled in the Python wheel by default. To use it from Rust or the Node binding, build with the `cos` Cargo feature enabled. -## GooseFS {#goosefs} +### GooseFS {#goosefs} [GooseFS](https://www.tencentcloud.com/document/product/1424) is Tencent Cloud's distributed cache acceleration layer for COS and S3. It is a common choice when the same hot dataset is read repeatedly, such as vector search and AI training workloads. Connect using the `goosefs://` URI scheme. @@ -338,4 +410,3 @@ GooseFS reads credentials and endpoint configuration from the GooseFS client env GooseFS is bundled by default in the Python wheel and the Node binding. To use it from Rust, build with the `goosefs` Cargo feature enabled. - diff --git a/docs/web/docs.json b/docs/web/docs.json index 02491a31c..8859494e8 100644 --- a/docs/web/docs.json +++ b/docs/web/docs.json @@ -146,6 +146,22 @@ { "source": "/core/update", "destination": "/ingestion/update" + }, + { + "source": "/storage", + "destination": "/deployment/storage" + }, + { + "source": "/storage/configuration", + "destination": "/deployment/storage" + }, + { + "source": "/storage/monitoring", + "destination": "/deployment/monitoring" + }, + { + "source": "/enterprise/deployment", + "destination": "/enterprise/deployment/index" } ], "navigation": { @@ -243,25 +259,14 @@ { "group": "Maintenance", "pages": [ - { - "group": "Indexing", - "pages": [ - "indexing/index", - "indexing/vector-index", - "indexing/fts-index", - "indexing/scalar-index", - "indexing/reindexing" - ] - }, + "indexing/index", + "indexing/vector-index", + "indexing/fts-index", + "indexing/scalar-index", + "indexing/reindexing", "maintenance/compaction", - { - "group": "Storage", - "pages": [ - "storage/index", - "storage/configuration", - "storage/monitoring" - ] - } + "maintenance/cleanup", + "maintenance/background-jobs" ] }, { @@ -277,13 +282,10 @@ "pages": [ "enterprise/index", "enterprise/architecture", - { - "group": "Cloud Providers", - "pages": [ - "enterprise/deployment/index", - "enterprise/deployment/azure" - ] - }, + "enterprise/deployment/index", + "deployment/storage", + "deployment/monitoring", + "enterprise/deployment/azure", "enterprise/benchmarks" ] }, diff --git a/docs/web/enterprise/deployment/index.mdx b/docs/web/enterprise/deployment/index.mdx index 5fe98eba5..c718cc32b 100644 --- a/docs/web/enterprise/deployment/index.mdx +++ b/docs/web/enterprise/deployment/index.mdx @@ -1,6 +1,6 @@ --- title: "Deployment guide" -sidebarTitle: "Overview" +sidebarTitle: "Deployment Models" description: "Standing up a LanceDB Enterprise deployment." icon: server keywords: ["deployment", "install", "cluster", "kubernetes"] diff --git a/docs/web/feature-engineering/materialized-views.mdx b/docs/web/feature-engineering/materialized-views.mdx index 034eaa028..3ad000ea9 100644 --- a/docs/web/feature-engineering/materialized-views.mdx +++ b/docs/web/feature-engineering/materialized-views.mdx @@ -78,6 +78,10 @@ Concurrent refreshes of one view do not duplicate rows. If two plan the same source rows, they conflict on commit and the loser raises rather than writing them twice — so a retry loop around `refresh()` is safe. + +**Enterprise job type:** `mat_view_refresh`. + + ## Against a deployment {#availability} Enterprise diff --git a/docs/web/feature-engineering/refresh-column.mdx b/docs/web/feature-engineering/refresh-column.mdx index db1480811..fab9e6fd1 100644 --- a/docs/web/feature-engineering/refresh-column.mdx +++ b/docs/web/feature-engineering/refresh-column.mdx @@ -59,6 +59,12 @@ committing to the whole table: REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01' ``` + +**Enterprise job type:** `computed_refresh` for a SQL-expression column, +`function_refresh` for one filled by a registered +[function](/management/function). + + ## As a job {#async} Enterprise diff --git a/docs/web/indexing/reindexing.mdx b/docs/web/indexing/reindexing.mdx index 5ebf94e5c..dd50f62fe 100644 --- a/docs/web/indexing/reindexing.mdx +++ b/docs/web/indexing/reindexing.mdx @@ -19,6 +19,13 @@ As data is being added and a reindex operation is running, LanceDB will combine Rather than dropping an existing index entirely and reindexing from scratch, LanceDB supports **incremental indexing**. + +**Enterprise job types:** building an index runs as `create_ivf_pq`, `create_btree`, +`create_fts`, and one more per index algorithm -- the type names the algorithm, +not a separate feature. Folding new rows into an existing one runs as the +matching `reindex_*` type instead. + + ## Incremental Reindexing {#incremental-reindexing} You can manually trigger an incremental indexing operation on updated data diff --git a/docs/web/maintenance/background-jobs.mdx b/docs/web/maintenance/background-jobs.mdx new file mode 100644 index 000000000..fc1c83767 --- /dev/null +++ b/docs/web/maintenance/background-jobs.mdx @@ -0,0 +1,65 @@ +--- +title: "Background jobs" +sidebarTitle: "Background Jobs" +description: "Jobs a deployment runs on its own — visible in SHOW JOBS, not started by a call you make." +icon: gears +keywords: ["background jobs", "job type", "wal", "spfresh", "prewarm", "archive"] +--- + +Enterprise + +Every job documented elsewhere in these docs -- [Build & Repair](/indexing/reindexing), +[Compaction](/maintenance/compaction), [Cleanup](/maintenance/cleanup), +[Column Refresh](/feature-engineering/refresh-column) -- is one you cause, +directly or by a schedule you set. This page is the rest: jobs the deployment +runs to keep itself correct and fast, which you may see named in +[`SHOW JOBS`](/sql/ddl#job) without having asked for any of them. + +There is no action to take on these beyond recognizing the name. None of them +has a parameter you set or a call you make. + +## Write-ahead log maintenance {#wal} + +For a table with a [write-ahead log](/sql/ddl#table) enabled, two jobs keep the +log from growing without bound: + +| Job type | What it does | +| --- | --- | +| `sstable_compaction` | Folds a WAL bucket's SSTables into the base table by primary key. Distinct from `compact`, which merges base-table fragments -- the two are unrelated. | +| `wal_index_catchup` | Extends a base-table index over the rows `sstable_compaction` just folded in, and records how far the index has caught up. Nothing else advances that record. | + +## Index maintenance {#index} + +| Job type | What it does | +| --- | --- | +| `index_remap` | Reassigns index entries after row ids shift underneath an index -- the kind of remap a delta compaction can require. | +| `spfresh_merge` | Merges SPFresh-managed segments for indexes that use it. | + +## Cache warming {#cache} + +| Job type | What it does | +| --- | --- | +| `prewarm_page_cache` | Loads table data into a query node's page cache ahead of a request needing it. | +| `prewarm_index_cache` | The same, for index structures. | +| `startup_prewarm_idx` | Runs cache prewarming as part of a query node's own startup, rather than on a schedule. | + +## Registry housekeeping {#registry} + +| Job type | What it does | +| --- | --- | +| `job_registry_archive` | Sweeps terminal jobs out of the active registry and into archival storage, on a per-database schedule. | +| `job_events_export` | Copies terminal jobs' event histories into a curated system table for metering. Registry-scoped, like the archive sweep. | + +These two are why an old job you completed weeks ago no longer shows in +`SHOW JOBS` -- it was archived, not deleted; its events survive in the exported +table. + +## Object teardown {#teardown} + +| Job type | What it does | +| --- | --- | +| `drop_object` | Deletes a dropped catalog object's content, then clears it from the database's cleanup backlog. Covers namespaces, views, and functions -- everything [`drop_table`](/management/table#drop) does not, since that one fences a directory with its own marker instead. | + +You trigger this indirectly, by dropping a [namespace](/management/namespace), +[view](/sql/ddl#view), or [function](/management/function#drop) -- the teardown +itself runs as this job rather than inline with the call. diff --git a/docs/web/maintenance/cleanup.mdx b/docs/web/maintenance/cleanup.mdx new file mode 100644 index 000000000..71447a7f7 --- /dev/null +++ b/docs/web/maintenance/cleanup.mdx @@ -0,0 +1,82 @@ +--- +title: "Cleanup" +sidebarTitle: "Cleanup" +description: "Pruning old versions and the files only they referenced." +icon: broom +keywords: ["cleanup", "retention", "prune", "delete_unverified", "vacuum"] +--- + +Every write keeps the version before it, which is what makes +[time travel](/management/table#versioning) possible. Cleanup is what ends +that -- pruning versions older than a retention window, and the files that only +those versions referenced. + + +**Enterprise job type:** `cleanup`. Distinct from `compact` -- compaction +merges files, cleanup removes them. `optimize()` in OSS runs both in one call; +Enterprise schedules them as separate jobs. + + +## Retention {#retention} + +```python +from datetime import timedelta + +table.optimize(cleanup_older_than=timedelta(days=30)) +``` + +`cleanup_older_than` defaults to **7 days**. Versions older than that are +removed, along with the files only they referenced. + +The latest version is never removed, whatever you pass -- `timedelta(0)` prunes +everything except it. + + +Pruning is what makes [time travel](/management/table#versioning) finite. A +version outside the retention window is gone, and a tag or branch pointing at +it no longer resolves. + +If you rely on reading old versions, set the window to cover the range you +need before running this, not after. + + +## Disk usage does not drop immediately {#disk} + +Compaction writes new, merged files before the old ones are removed, so disk +usage can rise before it falls. Space is reclaimed only when cleanup prunes the +versions that referenced the superseded files. + +If you need to reclaim space aggressively, shorten the retention window rather +than waiting for the default: + +```python +table.optimize(cleanup_older_than=timedelta(days=1)) +``` + +## Files from failed transactions {#unverified} + +A write that fails partway can leave files that look like an operation still +in progress. Cleanup will not remove them until they are at least seven days +old, because it cannot tell them apart from a live write. + +```python +table.optimize(delete_unverified=True) +``` + + +`delete_unverified=True` removes them regardless of age. Only set it when you +can guarantee **no other process is writing to the table** -- otherwise it can +delete files belonging to a write that is genuinely in progress, and corrupt +the table. + + +## What to read next {#next} + + + + Merging small files -- the other half of `optimize()`. + + + The Enterprise jobs that run without a call from you. + + diff --git a/docs/web/maintenance/compaction.mdx b/docs/web/maintenance/compaction.mdx index 602df31a0..db669c598 100644 --- a/docs/web/maintenance/compaction.mdx +++ b/docs/web/maintenance/compaction.mdx @@ -23,53 +23,18 @@ Three operations, in one call: | | | | --- | --- | | **Compaction** | merges small files into larger ones | -| **Prune** | removes versions older than the retention window | -| **Index** | folds newly added rows into existing indexes | +| **Prune** | removes versions older than the retention window -- see [Cleanup](/maintenance/cleanup) | +| **Index** | folds newly added rows into existing indexes -- see [Build & Repair](/indexing/reindexing) | -The third is the one people miss. An [index](/indexing) does not -automatically cover rows appended after it was built — those rows are still -searchable, but by scanning rather than through the index. Running `optimize()` -is what brings them under it. +Compaction itself is only the first of the three: merging small fragments back +into larger ones, which is what keeps a scan from opening more files than it +needs to. -## Retention {#retention} - -```python -from datetime import timedelta - -table.optimize(cleanup_older_than=timedelta(days=30)) -``` - -`cleanup_older_than` defaults to **7 days**. Versions older than that are -removed, along with the files only they referenced. - -The latest version is never removed, whatever you pass — `timedelta(0)` prunes -everything except it. - - -Pruning is what makes [time travel](/management/table#versioning) finite. A version -outside the retention window is gone, and a tag or branch pointing at it no -longer resolves. - -If you rely on reading old versions, set the window to cover the range you need -before running this, not after. - - -## Files from failed transactions {#unverified} - -A write that fails partway can leave files that look like an operation still in -progress. `optimize()` will not remove them until they are at least seven days -old, because it cannot tell them apart from a live write. - -```python -table.optimize(delete_unverified=True) -``` - - -`delete_unverified=True` removes them regardless of age. Only set it when you -can guarantee **no other process is writing to the table** — otherwise it can -delete files belonging to a write that is genuinely in progress, and corrupt the -table. - + +**Enterprise job type:** `compact`. Runs automatically as data accumulates, +separately from [`cleanup`](/maintenance/cleanup) -- the two are distinct jobs +even though `optimize()` bundles both in one OSS call. + ## Retraining indexes {#retrain} @@ -100,6 +65,10 @@ Compaction rewrites files, which conflicts with a concurrent version it replaces. Sequence them rather than running both at once. +See [Cleanup](/maintenance/cleanup) for pruning old versions, and +[Build & Repair](/indexing/reindexing) for the index-folding half of +`optimize()`. + ## Performance {#performance} Two things accumulate on a long-lived table as more and more data gets added to it: @@ -115,6 +84,4 @@ table.optimize() # 7-day default retention table.optimize(cleanup_older_than=timedelta(days=1)) # reclaim space sooner ``` -The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates — see [Reindexing](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed — rebuild the index after large update batches. - -LanceDB Enterprise handles both compaction and cleanup automatically. +The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates — see [Build & Repair](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed — rebuild the index after large update batches. diff --git a/docs/web/management/database.mdx b/docs/web/management/database.mdx index 33ade08a4..317a2aaec 100644 --- a/docs/web/management/database.mdx +++ b/docs/web/management/database.mdx @@ -113,7 +113,7 @@ to have them. A table created without them cannot later become a view's source without being recreated. -See [Storage](/storage) for the full set, including per-cloud credentials. +See [Storage](/deployment/storage) for the full set, including per-cloud credentials. ## Read consistency {#consistency} diff --git a/docs/web/management/function.mdx b/docs/web/management/function.mdx index b47e43e82..49d4b8b34 100644 --- a/docs/web/management/function.mdx +++ b/docs/web/management/function.mdx @@ -110,6 +110,12 @@ In practice this means a function should be self-contained: take its inputs as arguments, import what it needs at module level, and avoid capturing state from the surrounding scope. + +**Enterprise job type:** `create_function`. Registration returns a job because +the function is baked into an image before the version becomes usable -- see +below. + + ## Register it {#register} Registration returns a job; the immutable version exists once the job succeeds. diff --git a/docs/web/management/table.mdx b/docs/web/management/table.mdx index fff50e03b..ca4d6c767 100644 --- a/docs/web/management/table.mdx +++ b/docs/web/management/table.mdx @@ -206,6 +206,13 @@ the clone points at can be pruned. Use a deep clone (`is_shallow=False`) when the copy has to outlive the source's history. + +**Enterprise job type:** `drop_table`. Fences the table's directory with a +marker and deletes its content in the background -- see +[Background jobs](/maintenance/background-jobs#teardown) for the namespace, +view, and function equivalent. + + ## Dropping {#drop} diff --git a/docs/web/quickstart.mdx b/docs/web/quickstart.mdx index 43ba8c18c..50dc72945 100644 --- a/docs/web/quickstart.mdx +++ b/docs/web/quickstart.mdx @@ -175,7 +175,7 @@ You can also connect directly to object storage from the client SDK: For credentials, endpoints, and provider-specific options, see -[Configuring storage](/storage/configuration). +[Configuring storage](/deployment/storage). ### Connect to LanceDB Enterprise {#connect-to-lancedb-enterprise} diff --git a/docs/web/storage/index.mdx b/docs/web/storage/index.mdx deleted file mode 100644 index c3ad7c452..000000000 --- a/docs/web/storage/index.mdx +++ /dev/null @@ -1,77 +0,0 @@ ---- -title: "Storage Architecture in LanceDB" -sidebarTitle: "Storage Options" -description: "Understand LanceDB storage backends, tradeoffs, and how to pick the right option for your latency, scale, and cost goals." -icon: "database" ---- - -LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs). - -Choosing a backend is a balance between latency, scalability, cost, and operational complexity. Use this guide to pick the right fit for your workload. - -## Storage backend selection guide {#storage-backend-selection-guide} - -![](/static/assets/images/storage/lancedb_storage_tradeoffs.png) - -When architecting your system, ask yourself: - -- **Latency**: How fast do I need results? What do the p50 and p95 look like? -- **Scalability**: Can I scale data volume and QPS easily? -- **Cost**: What is the all-in cost of storage plus serving? -- **Reliability/Availability**: How will replication and disaster recovery work? - -## Storage backend comparison {#storage-backend-comparison} - -Below is a high-level comparison ordered from lowest cost to lowest latency. - -### 1\. Object storage (S3, GCS, Azure Blob) {#1-object-storage-s3-gcs-azure-blob} -- **Latency**: Highest; expect hundreds of milliseconds and higher p95. -- **Scalability**: Effectively unlimited storage; QPS bound by concurrency limits. -- **Cost**: Lowest overall. -- **Reliability/Availability**: Highly available, backed by cloud SLAs. - -LanceDB separates storage and compute and writes immutable fragments, making it a strong fit for stateless, horizontally scalable deployments. - - -**Concurrent writers on S3** - -S3 and S3 Express now support atomic writes natively, so LanceDB handles concurrent writers against the same table out-of-the-box — no external commit coordinator is required. Bucket-level [server-side encryption with KMS](/storage/configuration#server-side-encryption-with-kms) and [S3 Express One Zone](/storage/configuration#s3-express) are also supported on this tier. - - -### 2\. File storage (EFS, GCS Filestore, Azure File) {#2-file-storage-efs-gcs-filestore-azure-file} -- **Latency**: Better than object storage; p95 under ~<100ms is typical. -- **Scalability**: High, but limited by provisioned IOPS per volume. -- **Cost**: More than object storage but cheaper than in-memory options; cold data can tier down automatically. -- **Reliability/Availability**: Highly available; replication/backup must be managed separately. - -Keep a copy of data in object storage for disaster recovery. If zero downtime is required, provision a second network file system with replicated data. - -### 3\. Third-party storage (e.g., MinIO, WekaFS) {#3-third-party-storage-e-g--minio-wekafs} - -- **Latency**: Similar to EFS; typically under <100ms. -- **Scalability**: Determined by the chosen vendor’s cluster sizing. -- **Cost**: Higher than S3; may edge above EFS at larger scales. -- **Reliability/Availability**: Shareable across many nodes; replication depends on vendor capabilities. - -### 4\. Block storage (EBS, GCP Persistent Disk, Azure Managed Disk) {#4-block-storage-ebs-gcp-persistent-disk-azure-managed-disk} -- **Latency**: Near-local performance; often <30ms. -- **Scalability**: Not shareable across instances; shard or copy data when scaling. -- **Cost**: Higher than networked file systems, plus potential I/O charges. -- **Reliability/Availability**: Persists through instance restarts; backups and sharding must be managed. - -### 5\. Local storage (SSD, NVMe) {#5-local-storage-ssd-nvme} -- **Latency**: Fastest; p95 often under <10ms. -- **Scalability**: Hard to scale in cloud environments; requires sharding or additional copies for higher QPS. -- **Cost**: Highest; tightly coupling compute and storage makes horizontal scaling difficult. -- **Reliability/Availability**: Data is tied to the instance; backups must be rigorous. - -Use local disk only when you need extremely low latency and are comfortable owning the operational overhead. - -## File-format choices that interact with the backend {#file-format-choices-that-interact-with-the-backend} - -A few `storage_options` keys shape new tables in ways that depend on the backend you picked above. They are documented in full on the [configuration page](/storage/configuration#new-table-configuration); the architecture-level summary is: - -- `new_table_enable_v2_manifest_paths` matters most on object stores, where opening a table with many versions is dominated by listing cost. Leave it off for backward compatibility with clients older than LanceDB 0.10.0. -- `new_table_enable_stable_row_ids` keeps row IDs stable across compaction, delete, and merge. The choice is independent of the backend but affects any system that joins on row ID. -- `new_table_data_storage_version` selects the on-disk format. The default `stable` is recommended for all new tables; pick `legacy` only when older readers must keep working. -