mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
Jack Ye
734eb39dc3
Six groups were verbs -- Manage, Ingest, Train, Operate, Secure, Deploy -- and now name the thing rather than the action. Four more were sentence case. The page labels that were verb phrases follow: Add Computed Column becomes Computed Columns, Refresh Column becomes Column Refresh, Configuring Storage becomes Configuration, and Explain becomes Query Plans, which is also more accurate now that the page covers analyze as well. SQL statement pages take the keyword -- SELECT, DDL, DML, REFRESH, SHOW, EXPLAIN -- because a statement name is a noun and lowercase Select and Show read as instructions. Concept pages there keep noun phrases: Joins, Window Functions, Subqueries, Operators. Two duplicate labels are resolved on the way. Reranking sat in Semantic Search labelled Overview despite being the only page on its subject, and the Enterprise client page sat in SQL Reference labelled SQL next to the reference Overview; it is Client Connection now. Title case leaves prepositions and articles lowercase, so Needle in a Haystack and Hybrid Search on BEIR are already correct and unchanged.
78 lines
5.1 KiB
Plaintext
78 lines
5.1 KiB
Plaintext
---
|
||
title: "Storage Architecture in LanceDB"
|
||
sidebarTitle: "Storage Options"
|
||
description: "Understand LanceDB storage backends, tradeoffs, and how to pick the right option for your latency, scale, and cost goals."
|
||
icon: "database"
|
||
---
|
||
|
||
LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs).
|
||
|
||
Choosing a backend is a balance between latency, scalability, cost, and operational complexity. Use this guide to pick the right fit for your workload.
|
||
|
||
## Storage backend selection guide {#storage-backend-selection-guide}
|
||
|
||
|
||
|
||
When architecting your system, ask yourself:
|
||
|
||
- **Latency**: How fast do I need results? What do the p50 and p95 look like?
|
||
- **Scalability**: Can I scale data volume and QPS easily?
|
||
- **Cost**: What is the all-in cost of storage plus serving?
|
||
- **Reliability/Availability**: How will replication and disaster recovery work?
|
||
|
||
## Storage backend comparison {#storage-backend-comparison}
|
||
|
||
Below is a high-level comparison ordered from lowest cost to lowest latency.
|
||
|
||
### 1\. Object storage (S3, GCS, Azure Blob) {#1-object-storage-s3-gcs-azure-blob}
|
||
- **Latency**: Highest; expect hundreds of milliseconds and higher p95.
|
||
- **Scalability**: Effectively unlimited storage; QPS bound by concurrency limits.
|
||
- **Cost**: Lowest overall.
|
||
- **Reliability/Availability**: Highly available, backed by cloud SLAs.
|
||
|
||
LanceDB separates storage and compute and writes immutable fragments, making it a strong fit for stateless, horizontally scalable deployments.
|
||
|
||
<Note>
|
||
**Concurrent writers on S3**
|
||
|
||
S3 and S3 Express now support atomic writes natively, so LanceDB handles concurrent writers against the same table out-of-the-box — no external commit coordinator is required. Bucket-level [server-side encryption with KMS](/storage/configuration#server-side-encryption-with-kms) and [S3 Express One Zone](/storage/configuration#s3-express) are also supported on this tier.
|
||
</Note>
|
||
|
||
### 2\. File storage (EFS, GCS Filestore, Azure File) {#2-file-storage-efs-gcs-filestore-azure-file}
|
||
- **Latency**: Better than object storage; p95 under ~<100ms is typical.
|
||
- **Scalability**: High, but limited by provisioned IOPS per volume.
|
||
- **Cost**: More than object storage but cheaper than in-memory options; cold data can tier down automatically.
|
||
- **Reliability/Availability**: Highly available; replication/backup must be managed separately.
|
||
|
||
Keep a copy of data in object storage for disaster recovery. If zero downtime is required, provision a second network file system with replicated data.
|
||
|
||
### 3\. Third-party storage (e.g., MinIO, WekaFS) {#3-third-party-storage-e-g--minio-wekafs}
|
||
|
||
- **Latency**: Similar to EFS; typically under <100ms.
|
||
- **Scalability**: Determined by the chosen vendor’s cluster sizing.
|
||
- **Cost**: Higher than S3; may edge above EFS at larger scales.
|
||
- **Reliability/Availability**: Shareable across many nodes; replication depends on vendor capabilities.
|
||
|
||
### 4\. Block storage (EBS, GCP Persistent Disk, Azure Managed Disk) {#4-block-storage-ebs-gcp-persistent-disk-azure-managed-disk}
|
||
- **Latency**: Near-local performance; often <30ms.
|
||
- **Scalability**: Not shareable across instances; shard or copy data when scaling.
|
||
- **Cost**: Higher than networked file systems, plus potential I/O charges.
|
||
- **Reliability/Availability**: Persists through instance restarts; backups and sharding must be managed.
|
||
|
||
### 5\. Local storage (SSD, NVMe) {#5-local-storage-ssd-nvme}
|
||
- **Latency**: Fastest; p95 often under <10ms.
|
||
- **Scalability**: Hard to scale in cloud environments; requires sharding or additional copies for higher QPS.
|
||
- **Cost**: Highest; tightly coupling compute and storage makes horizontal scaling difficult.
|
||
- **Reliability/Availability**: Data is tied to the instance; backups must be rigorous.
|
||
|
||
Use local disk only when you need extremely low latency and are comfortable owning the operational overhead.
|
||
|
||
## File-format choices that interact with the backend {#file-format-choices-that-interact-with-the-backend}
|
||
|
||
A few `storage_options` keys shape new tables in ways that depend on the backend you picked above. They are documented in full on the [configuration page](/storage/configuration#new-table-configuration); the architecture-level summary is:
|
||
|
||
- `new_table_enable_v2_manifest_paths` matters most on object stores, where opening a table with many versions is dominated by listing cost. Leave it off for backward compatibility with clients older than LanceDB 0.10.0.
|
||
- `new_table_enable_stable_row_ids` keeps row IDs stable across compaction, delete, and merge. The choice is independent of the backend but affects any system that joins on row ID.
|
||
- `new_table_data_storage_version` selects the on-disk format. The default `stable` is recommended for all new tables; pick `legacy` only when older readers must keep working.
|
||
|