mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 14:52:26 +00:00
docs: flatten Maintenance and Deployment, add Cleanup and Background Jobs
Indexing loses its wrapper group -- Overview, Vector Index, FTS Index, Scalar Index and Build & Repair sit directly in Maintenance now, followed by Compaction, Cleanup and Background Jobs at the same level. Cleanup is new, split out of Compaction: pruning old versions is a distinct Enterprise job (cleanup) from merging files (compact), even though optimize() in OSS bundles both into one call. The retention, disk-usage and failed-transaction material that used to live under Compaction's umbrella moves with it. Background Jobs is new and covers what nothing else does: the job types a deployment runs on its own -- WAL/SPFresh maintenance, cache prewarming, registry archiving and metering export, and the object-teardown job that follows a namespace or view drop. Every entry is grounded against the job-coordinator's public job_type constants, not the internal two-letter wire codes used for registry storage. Five existing pages (Build & Repair, Table, Function, Column Refresh, Materialized View Refresh) gain a one-line job-type note rather than duplicating a page for job types they already document. Storage folds into Deployment: storage/index (choosing a backend) and storage/configuration (configuring one) merge into a single Storage page -- they never overlapped, one is the decision and one is the how-to -- and move under Deployment along with Monitoring. Cloud Providers loses its nesting; its two pages plus Azure sit flat, and the former Overview is retitled Deployment Models so it doesn't collide with the group's own Overview entry. Redirects cover every moved URL.
This commit is contained in:
Jack Ye
@@ -1,9 +1,81 @@
|
||||
---
|
||||
title: "Configuring Cloud Storage in LanceDB"
|
||||
sidebarTitle: "Configuration"
|
||||
description: "Configure LanceDB to use S3, GCS, Azure Blob, and S3-compatible object stores with environment variables or storage options."
|
||||
icon: "wrench"
|
||||
title: "Storage"
|
||||
sidebarTitle: "Storage"
|
||||
description: "Choosing and configuring where LanceDB stores its data -- how you deploy an embedded database."
|
||||
icon: database
|
||||
keywords: ["storage", "s3", "gcs", "azure", "object store", "configuration", "deploy"]
|
||||
---
|
||||
|
||||
LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs).
|
||||
|
||||
Choosing a backend is a balance between latency, scalability, cost, and operational complexity. Use this guide to pick the right fit for your workload.
|
||||
|
||||
## Choosing a backend {#choosing}
|
||||
|
||||
|
||||
|
||||
When architecting your system, ask yourself:
|
||||
|
||||
- **Latency**: How fast do I need results? What do the p50 and p95 look like?
|
||||
- **Scalability**: Can I scale data volume and QPS easily?
|
||||
- **Cost**: What is the all-in cost of storage plus serving?
|
||||
- **Reliability/Availability**: How will replication and disaster recovery work?
|
||||
|
||||
### Comparison {#comparison}
|
||||
|
||||
Below is a high-level comparison ordered from lowest cost to lowest latency.
|
||||
|
||||
### 1\. Object storage (S3, GCS, Azure Blob) {#1-object-storage-s3-gcs-azure-blob}
|
||||
- **Latency**: Highest; expect hundreds of milliseconds and higher p95.
|
||||
- **Scalability**: Effectively unlimited storage; QPS bound by concurrency limits.
|
||||
- **Cost**: Lowest overall.
|
||||
- **Reliability/Availability**: Highly available, backed by cloud SLAs.
|
||||
|
||||
LanceDB separates storage and compute and writes immutable fragments, making it a strong fit for stateless, horizontally scalable deployments.
|
||||
|
||||
<Note>
|
||||
**Concurrent writers on S3**
|
||||
|
||||
S3 and S3 Express now support atomic writes natively, so LanceDB handles concurrent writers against the same table out-of-the-box — no external commit coordinator is required. Bucket-level [server-side encryption with KMS](/storage/configuration#server-side-encryption-with-kms) and [S3 Express One Zone](/storage/configuration#s3-express) are also supported on this tier.
|
||||
</Note>
|
||||
|
||||
### 2\. File storage (EFS, GCS Filestore, Azure File) {#2-file-storage-efs-gcs-filestore-azure-file}
|
||||
- **Latency**: Better than object storage; p95 under ~<100ms is typical.
|
||||
- **Scalability**: High, but limited by provisioned IOPS per volume.
|
||||
- **Cost**: More than object storage but cheaper than in-memory options; cold data can tier down automatically.
|
||||
- **Reliability/Availability**: Highly available; replication/backup must be managed separately.
|
||||
|
||||
Keep a copy of data in object storage for disaster recovery. If zero downtime is required, provision a second network file system with replicated data.
|
||||
|
||||
### 3\. Third-party storage (e.g., MinIO, WekaFS) {#3-third-party-storage-e-g--minio-wekafs}
|
||||
|
||||
- **Latency**: Similar to EFS; typically under <100ms.
|
||||
- **Scalability**: Determined by the chosen vendor’s cluster sizing.
|
||||
- **Cost**: Higher than S3; may edge above EFS at larger scales.
|
||||
- **Reliability/Availability**: Shareable across many nodes; replication depends on vendor capabilities.
|
||||
|
||||
### 4\. Block storage (EBS, GCP Persistent Disk, Azure Managed Disk) {#4-block-storage-ebs-gcp-persistent-disk-azure-managed-disk}
|
||||
- **Latency**: Near-local performance; often <30ms.
|
||||
- **Scalability**: Not shareable across instances; shard or copy data when scaling.
|
||||
- **Cost**: Higher than networked file systems, plus potential I/O charges.
|
||||
- **Reliability/Availability**: Persists through instance restarts; backups and sharding must be managed.
|
||||
|
||||
### 5\. Local storage (SSD, NVMe) {#5-local-storage-ssd-nvme}
|
||||
- **Latency**: Fastest; p95 often under <10ms.
|
||||
- **Scalability**: Hard to scale in cloud environments; requires sharding or additional copies for higher QPS.
|
||||
- **Cost**: Highest; tightly coupling compute and storage makes horizontal scaling difficult.
|
||||
- **Reliability/Availability**: Data is tied to the instance; backups must be rigorous.
|
||||
|
||||
Use local disk only when you need extremely low latency and are comfortable owning the operational overhead.
|
||||
|
||||
### File-format choices that interact with the backend {#file-format-choices}
|
||||
|
||||
A few `storage_options` keys shape new tables in ways that depend on the backend you picked above. They are documented in full on [below](#new-table-configuration); the architecture-level summary is:
|
||||
|
||||
- `new_table_enable_v2_manifest_paths` matters most on object stores, where opening a table with many versions is dominated by listing cost. Leave it off for backward compatibility with clients older than LanceDB 0.10.0.
|
||||
- `new_table_enable_stable_row_ids` keeps row IDs stable across compaction, delete, and merge. The choice is independent of the backend but affects any system that joins on row ID.
|
||||
- `new_table_data_storage_version` selects the on-disk format. The default `stable` is recommended for all new tables; pick `legacy` only when older readers must keep working.
|
||||
|
||||
import {
|
||||
PyStorageAzureAccount,
|
||||
PyStorageAzureSas,
|
||||
@@ -34,7 +106,7 @@ import {
|
||||
TsStorageGoosefsConnect,
|
||||
} from '/snippets/storage.mdx';
|
||||
|
||||
When using LanceDB OSS, you can choose where to store your data. The tradeoffs between storage options are covered in the [storage architecture guide](/storage). This page shows how to configure each backend.
|
||||
### Configuring a backend {#configuring}
|
||||
|
||||
<Note>
|
||||
**LanceDB Enterprise storage configuration**
|
||||
@@ -42,7 +114,7 @@ When using LanceDB OSS, you can choose where to store your data. The tradeoffs b
|
||||
In LanceDB Enterprise, you connect with `db://...` and the cluster owns the storage credentials, so `storage_options` are not passed at runtime. Cloud auth is set at deployment time. For federated databases, the namespace service vends per-request credentials automatically. See the [quickstart](/quickstart), [Enterprise overview](/enterprise/), and [Azure deployment guide](/enterprise/deployment/azure) for the Enterprise flow.
|
||||
</Note>
|
||||
|
||||
## Object stores {#object-stores}
|
||||
### Object stores {#object-stores}
|
||||
|
||||
LanceDB supports AWS S3 (and compatible stores), Azure Blob Storage, and Google Cloud Storage. The URI scheme in your `connect` call selects the backend.
|
||||
|
||||
@@ -185,7 +257,7 @@ const db = await lancedb.connect("s3://bucket/path", {
|
||||
The `data_storage_version` parameter on `create_table()` is deprecated. Use `new_table_data_storage_version` in `storage_options` instead.
|
||||
</Warning>
|
||||
|
||||
## AWS S3 {#aws-s3}
|
||||
### AWS S3 {#aws-s3}
|
||||
|
||||
|
||||
|
||||
@@ -240,7 +312,7 @@ To encrypt at rest with an AWS KMS key, set `aws_server_side_encryption` to `aws
|
||||
|
||||
The IAM principal needs `kms:Encrypt`, `kms:Decrypt`, and `kms:GenerateDataKey` on the configured KMS key.
|
||||
|
||||
## Google Cloud Storage {#google-cloud-storage}
|
||||
### Google Cloud Storage {#google-cloud-storage}
|
||||
|
||||
|
||||
|
||||
@@ -255,7 +327,7 @@ Provide credentials via `GOOGLE_SERVICE_ACCOUNT` (path to JSON) or include the p
|
||||
</CodeBlock>
|
||||
</CodeGroup>
|
||||
|
||||
## Azure Blob Storage {#azure-blob-storage}
|
||||
### Azure Blob Storage {#azure-blob-storage}
|
||||
|
||||
|
||||
|
||||
@@ -283,7 +355,7 @@ For SAS-token auth, set `azure_storage_account_name` and `azure_storage_sas_toke
|
||||
|
||||
Other supported keys include service principal credentials (`azure_client_id`, `azure_client_secret`, `azure_tenant_id`), managed identities, and custom endpoints.
|
||||
|
||||
## Tigris Object Storage {#tigris-object-storage}
|
||||
### Tigris Object Storage {#tigris-object-storage}
|
||||
|
||||
|
||||
|
||||
@@ -300,7 +372,7 @@ Tigris exposes an S3-compatible API. Configure the endpoint and region:
|
||||
|
||||
Environment variables `AWS_ENDPOINT=https://t3.storage.dev` and `AWS_DEFAULT_REGION=auto` achieve the same configuration.
|
||||
|
||||
## Tencent COS {#tencent-cos}
|
||||
### Tencent COS {#tencent-cos}
|
||||
|
||||
[Tencent Cloud Object Storage (COS)](https://www.tencentcloud.com/products/cos) is the primary object store for workloads running in the China region. Use the `cos://` URI scheme to connect directly to a COS bucket.
|
||||
|
||||
@@ -318,7 +390,7 @@ Supported keys include `secret_id`, `secret_key`, `region`, and `endpoint`. You
|
||||
COS is bundled in the Python wheel by default. To use it from Rust or the Node binding, build with the `cos` Cargo feature enabled.
|
||||
</Note>
|
||||
|
||||
## GooseFS {#goosefs}
|
||||
### GooseFS {#goosefs}
|
||||
|
||||
[GooseFS](https://www.tencentcloud.com/document/product/1424) is Tencent Cloud's distributed cache acceleration layer for COS and S3. It is a common choice when the same hot dataset is read repeatedly, such as vector search and AI training workloads. Connect using the `goosefs://` URI scheme.
|
||||
|
||||
@@ -338,4 +410,3 @@ GooseFS reads credentials and endpoint configuration from the GooseFS client env
|
||||
|
||||
GooseFS is bundled by default in the Python wheel and the Node binding. To use it from Rust, build with the `goosefs` Cargo feature enabled.
|
||||
</Note>
|
||||
|
||||
+27
-25
@@ -146,6 +146,22 @@
|
||||
{
|
||||
"source": "/core/update",
|
||||
"destination": "/ingestion/update"
|
||||
},
|
||||
{
|
||||
"source": "/storage",
|
||||
"destination": "/deployment/storage"
|
||||
},
|
||||
{
|
||||
"source": "/storage/configuration",
|
||||
"destination": "/deployment/storage"
|
||||
},
|
||||
{
|
||||
"source": "/storage/monitoring",
|
||||
"destination": "/deployment/monitoring"
|
||||
},
|
||||
{
|
||||
"source": "/enterprise/deployment",
|
||||
"destination": "/enterprise/deployment/index"
|
||||
}
|
||||
],
|
||||
"navigation": {
|
||||
@@ -243,25 +259,14 @@
|
||||
{
|
||||
"group": "Maintenance",
|
||||
"pages": [
|
||||
{
|
||||
"group": "Indexing",
|
||||
"pages": [
|
||||
"indexing/index",
|
||||
"indexing/vector-index",
|
||||
"indexing/fts-index",
|
||||
"indexing/scalar-index",
|
||||
"indexing/reindexing"
|
||||
]
|
||||
},
|
||||
"indexing/index",
|
||||
"indexing/vector-index",
|
||||
"indexing/fts-index",
|
||||
"indexing/scalar-index",
|
||||
"indexing/reindexing",
|
||||
"maintenance/compaction",
|
||||
{
|
||||
"group": "Storage",
|
||||
"pages": [
|
||||
"storage/index",
|
||||
"storage/configuration",
|
||||
"storage/monitoring"
|
||||
]
|
||||
}
|
||||
"maintenance/cleanup",
|
||||
"maintenance/background-jobs"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -277,13 +282,10 @@
|
||||
"pages": [
|
||||
"enterprise/index",
|
||||
"enterprise/architecture",
|
||||
{
|
||||
"group": "Cloud Providers",
|
||||
"pages": [
|
||||
"enterprise/deployment/index",
|
||||
"enterprise/deployment/azure"
|
||||
]
|
||||
},
|
||||
"enterprise/deployment/index",
|
||||
"deployment/storage",
|
||||
"deployment/monitoring",
|
||||
"enterprise/deployment/azure",
|
||||
"enterprise/benchmarks"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Deployment guide"
|
||||
sidebarTitle: "Overview"
|
||||
sidebarTitle: "Deployment Models"
|
||||
description: "Standing up a LanceDB Enterprise deployment."
|
||||
icon: server
|
||||
keywords: ["deployment", "install", "cluster", "kubernetes"]
|
||||
|
||||
@@ -78,6 +78,10 @@ Concurrent refreshes of one view do not duplicate rows. If two plan the same
|
||||
source rows, they conflict on commit and the loser raises rather than writing
|
||||
them twice — so a retry loop around `refresh()` is safe.
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `mat_view_refresh`.
|
||||
</Note>
|
||||
|
||||
## Against a deployment {#availability}
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
@@ -59,6 +59,12 @@ committing to the whole table:
|
||||
REFRESH COLUMN embedding ON documents WHERE created_at > '2026-01-01'
|
||||
```
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `computed_refresh` for a SQL-expression column,
|
||||
`function_refresh` for one filled by a registered
|
||||
[function](/management/function).
|
||||
</Note>
|
||||
|
||||
## As a job {#async}
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
@@ -19,6 +19,13 @@ As data is being added and a reindex operation is running, LanceDB will combine
|
||||
|
||||
Rather than dropping an existing index entirely and reindexing from scratch, LanceDB supports **incremental indexing**.
|
||||
|
||||
<Note>
|
||||
**Enterprise job types:** building an index runs as `create_ivf_pq`, `create_btree`,
|
||||
`create_fts`, and one more per index algorithm -- the type names the algorithm,
|
||||
not a separate feature. Folding new rows into an existing one runs as the
|
||||
matching `reindex_*` type instead.
|
||||
</Note>
|
||||
|
||||
## Incremental Reindexing {#incremental-reindexing}
|
||||
|
||||
You can manually trigger an incremental indexing operation on updated data
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: "Background jobs"
|
||||
sidebarTitle: "Background Jobs"
|
||||
description: "Jobs a deployment runs on its own — visible in SHOW JOBS, not started by a call you make."
|
||||
icon: gears
|
||||
keywords: ["background jobs", "job type", "wal", "spfresh", "prewarm", "archive"]
|
||||
---
|
||||
|
||||
<Badge color="red">Enterprise</Badge>
|
||||
|
||||
Every job documented elsewhere in these docs -- [Build & Repair](/indexing/reindexing),
|
||||
[Compaction](/maintenance/compaction), [Cleanup](/maintenance/cleanup),
|
||||
[Column Refresh](/feature-engineering/refresh-column) -- is one you cause,
|
||||
directly or by a schedule you set. This page is the rest: jobs the deployment
|
||||
runs to keep itself correct and fast, which you may see named in
|
||||
[`SHOW JOBS`](/sql/ddl#job) without having asked for any of them.
|
||||
|
||||
There is no action to take on these beyond recognizing the name. None of them
|
||||
has a parameter you set or a call you make.
|
||||
|
||||
## Write-ahead log maintenance {#wal}
|
||||
|
||||
For a table with a [write-ahead log](/sql/ddl#table) enabled, two jobs keep the
|
||||
log from growing without bound:
|
||||
|
||||
| Job type | What it does |
|
||||
| --- | --- |
|
||||
| `sstable_compaction` | Folds a WAL bucket's SSTables into the base table by primary key. Distinct from `compact`, which merges base-table fragments -- the two are unrelated. |
|
||||
| `wal_index_catchup` | Extends a base-table index over the rows `sstable_compaction` just folded in, and records how far the index has caught up. Nothing else advances that record. |
|
||||
|
||||
## Index maintenance {#index}
|
||||
|
||||
| Job type | What it does |
|
||||
| --- | --- |
|
||||
| `index_remap` | Reassigns index entries after row ids shift underneath an index -- the kind of remap a delta compaction can require. |
|
||||
| `spfresh_merge` | Merges SPFresh-managed segments for indexes that use it. |
|
||||
|
||||
## Cache warming {#cache}
|
||||
|
||||
| Job type | What it does |
|
||||
| --- | --- |
|
||||
| `prewarm_page_cache` | Loads table data into a query node's page cache ahead of a request needing it. |
|
||||
| `prewarm_index_cache` | The same, for index structures. |
|
||||
| `startup_prewarm_idx` | Runs cache prewarming as part of a query node's own startup, rather than on a schedule. |
|
||||
|
||||
## Registry housekeeping {#registry}
|
||||
|
||||
| Job type | What it does |
|
||||
| --- | --- |
|
||||
| `job_registry_archive` | Sweeps terminal jobs out of the active registry and into archival storage, on a per-database schedule. |
|
||||
| `job_events_export` | Copies terminal jobs' event histories into a curated system table for metering. Registry-scoped, like the archive sweep. |
|
||||
|
||||
These two are why an old job you completed weeks ago no longer shows in
|
||||
`SHOW JOBS` -- it was archived, not deleted; its events survive in the exported
|
||||
table.
|
||||
|
||||
## Object teardown {#teardown}
|
||||
|
||||
| Job type | What it does |
|
||||
| --- | --- |
|
||||
| `drop_object` | Deletes a dropped catalog object's content, then clears it from the database's cleanup backlog. Covers namespaces, views, and functions -- everything [`drop_table`](/management/table#drop) does not, since that one fences a directory with its own marker instead. |
|
||||
|
||||
You trigger this indirectly, by dropping a [namespace](/management/namespace),
|
||||
[view](/sql/ddl#view), or [function](/management/function#drop) -- the teardown
|
||||
itself runs as this job rather than inline with the call.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: "Cleanup"
|
||||
sidebarTitle: "Cleanup"
|
||||
description: "Pruning old versions and the files only they referenced."
|
||||
icon: broom
|
||||
keywords: ["cleanup", "retention", "prune", "delete_unverified", "vacuum"]
|
||||
---
|
||||
|
||||
Every write keeps the version before it, which is what makes
|
||||
[time travel](/management/table#versioning) possible. Cleanup is what ends
|
||||
that -- pruning versions older than a retention window, and the files that only
|
||||
those versions referenced.
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `cleanup`. Distinct from `compact` -- compaction
|
||||
merges files, cleanup removes them. `optimize()` in OSS runs both in one call;
|
||||
Enterprise schedules them as separate jobs.
|
||||
</Note>
|
||||
|
||||
## Retention {#retention}
|
||||
|
||||
```python
|
||||
from datetime import timedelta
|
||||
|
||||
table.optimize(cleanup_older_than=timedelta(days=30))
|
||||
```
|
||||
|
||||
`cleanup_older_than` defaults to **7 days**. Versions older than that are
|
||||
removed, along with the files only they referenced.
|
||||
|
||||
The latest version is never removed, whatever you pass -- `timedelta(0)` prunes
|
||||
everything except it.
|
||||
|
||||
<Warning>
|
||||
Pruning is what makes [time travel](/management/table#versioning) finite. A
|
||||
version outside the retention window is gone, and a tag or branch pointing at
|
||||
it no longer resolves.
|
||||
|
||||
If you rely on reading old versions, set the window to cover the range you
|
||||
need before running this, not after.
|
||||
</Warning>
|
||||
|
||||
## Disk usage does not drop immediately {#disk}
|
||||
|
||||
Compaction writes new, merged files before the old ones are removed, so disk
|
||||
usage can rise before it falls. Space is reclaimed only when cleanup prunes the
|
||||
versions that referenced the superseded files.
|
||||
|
||||
If you need to reclaim space aggressively, shorten the retention window rather
|
||||
than waiting for the default:
|
||||
|
||||
```python
|
||||
table.optimize(cleanup_older_than=timedelta(days=1))
|
||||
```
|
||||
|
||||
## Files from failed transactions {#unverified}
|
||||
|
||||
A write that fails partway can leave files that look like an operation still
|
||||
in progress. Cleanup will not remove them until they are at least seven days
|
||||
old, because it cannot tell them apart from a live write.
|
||||
|
||||
```python
|
||||
table.optimize(delete_unverified=True)
|
||||
```
|
||||
|
||||
<Warning>
|
||||
`delete_unverified=True` removes them regardless of age. Only set it when you
|
||||
can guarantee **no other process is writing to the table** -- otherwise it can
|
||||
delete files belonging to a write that is genuinely in progress, and corrupt
|
||||
the table.
|
||||
</Warning>
|
||||
|
||||
## What to read next {#next}
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Compaction" icon="broom" href="/maintenance/compaction">
|
||||
Merging small files -- the other half of `optimize()`.
|
||||
</Card>
|
||||
<Card title="Background jobs" icon="gears" href="/maintenance/background-jobs">
|
||||
The Enterprise jobs that run without a call from you.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -23,53 +23,18 @@ Three operations, in one call:
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Compaction** | merges small files into larger ones |
|
||||
| **Prune** | removes versions older than the retention window |
|
||||
| **Index** | folds newly added rows into existing indexes |
|
||||
| **Prune** | removes versions older than the retention window -- see [Cleanup](/maintenance/cleanup) |
|
||||
| **Index** | folds newly added rows into existing indexes -- see [Build & Repair](/indexing/reindexing) |
|
||||
|
||||
The third is the one people miss. An [index](/indexing) does not
|
||||
automatically cover rows appended after it was built — those rows are still
|
||||
searchable, but by scanning rather than through the index. Running `optimize()`
|
||||
is what brings them under it.
|
||||
Compaction itself is only the first of the three: merging small fragments back
|
||||
into larger ones, which is what keeps a scan from opening more files than it
|
||||
needs to.
|
||||
|
||||
## Retention {#retention}
|
||||
|
||||
```python
|
||||
from datetime import timedelta
|
||||
|
||||
table.optimize(cleanup_older_than=timedelta(days=30))
|
||||
```
|
||||
|
||||
`cleanup_older_than` defaults to **7 days**. Versions older than that are
|
||||
removed, along with the files only they referenced.
|
||||
|
||||
The latest version is never removed, whatever you pass — `timedelta(0)` prunes
|
||||
everything except it.
|
||||
|
||||
<Warning>
|
||||
Pruning is what makes [time travel](/management/table#versioning) finite. A version
|
||||
outside the retention window is gone, and a tag or branch pointing at it no
|
||||
longer resolves.
|
||||
|
||||
If you rely on reading old versions, set the window to cover the range you need
|
||||
before running this, not after.
|
||||
</Warning>
|
||||
|
||||
## Files from failed transactions {#unverified}
|
||||
|
||||
A write that fails partway can leave files that look like an operation still in
|
||||
progress. `optimize()` will not remove them until they are at least seven days
|
||||
old, because it cannot tell them apart from a live write.
|
||||
|
||||
```python
|
||||
table.optimize(delete_unverified=True)
|
||||
```
|
||||
|
||||
<Warning>
|
||||
`delete_unverified=True` removes them regardless of age. Only set it when you
|
||||
can guarantee **no other process is writing to the table** — otherwise it can
|
||||
delete files belonging to a write that is genuinely in progress, and corrupt the
|
||||
table.
|
||||
</Warning>
|
||||
<Note>
|
||||
**Enterprise job type:** `compact`. Runs automatically as data accumulates,
|
||||
separately from [`cleanup`](/maintenance/cleanup) -- the two are distinct jobs
|
||||
even though `optimize()` bundles both in one OSS call.
|
||||
</Note>
|
||||
|
||||
## Retraining indexes {#retrain}
|
||||
|
||||
@@ -100,6 +65,10 @@ Compaction rewrites files, which conflicts with a concurrent
|
||||
version it replaces. Sequence them rather than running both at once.
|
||||
</Note>
|
||||
|
||||
See [Cleanup](/maintenance/cleanup) for pruning old versions, and
|
||||
[Build & Repair](/indexing/reindexing) for the index-folding half of
|
||||
`optimize()`.
|
||||
|
||||
## Performance {#performance}
|
||||
|
||||
Two things accumulate on a long-lived table as more and more data gets added to it:
|
||||
@@ -115,6 +84,4 @@ table.optimize() # 7-day default retention
|
||||
table.optimize(cleanup_older_than=timedelta(days=1)) # reclaim space sooner
|
||||
```
|
||||
|
||||
The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates — see [Reindexing](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed — rebuild the index after large update batches.
|
||||
|
||||
LanceDB Enterprise handles both compaction and cleanup automatically.
|
||||
The best practice is to run `optimize()` after large writes or on a schedule. It also bundles incremental index updates — see [Build & Repair](/indexing/reindexing) for the breakdown. Updates also move rows out of the vector index, so they remain searchable but unindexed — rebuild the index after large update batches.
|
||||
|
||||
@@ -113,7 +113,7 @@ to have them. A table created without them cannot later become a view's source
|
||||
without being recreated.
|
||||
</Warning>
|
||||
|
||||
See [Storage](/storage) for the full set, including per-cloud credentials.
|
||||
See [Storage](/deployment/storage) for the full set, including per-cloud credentials.
|
||||
|
||||
## Read consistency {#consistency}
|
||||
|
||||
|
||||
@@ -110,6 +110,12 @@ In practice this means a function should be self-contained: take its inputs as
|
||||
arguments, import what it needs at module level, and avoid capturing state from
|
||||
the surrounding scope.
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `create_function`. Registration returns a job because
|
||||
the function is baked into an image before the version becomes usable -- see
|
||||
below.
|
||||
</Note>
|
||||
|
||||
## Register it {#register}
|
||||
|
||||
Registration returns a job; the immutable version exists once the job succeeds.
|
||||
|
||||
@@ -206,6 +206,13 @@ the clone points at can be pruned. Use a deep clone (`is_shallow=False`) when
|
||||
the copy has to outlive the source's history.
|
||||
</Warning>
|
||||
|
||||
<Note>
|
||||
**Enterprise job type:** `drop_table`. Fences the table's directory with a
|
||||
marker and deletes its content in the background -- see
|
||||
[Background jobs](/maintenance/background-jobs#teardown) for the namespace,
|
||||
view, and function equivalent.
|
||||
</Note>
|
||||
|
||||
## Dropping {#drop}
|
||||
|
||||
<CodeGroup>
|
||||
|
||||
@@ -175,7 +175,7 @@ You can also connect directly to object storage from the client SDK:
|
||||
</CodeGroup >
|
||||
|
||||
For credentials, endpoints, and provider-specific options, see
|
||||
[Configuring storage](/storage/configuration).
|
||||
[Configuring storage](/deployment/storage).
|
||||
|
||||
### Connect to LanceDB Enterprise {#connect-to-lancedb-enterprise}
|
||||
|
||||
|
||||
@@ -1,77 +0,0 @@
|
||||
---
|
||||
title: "Storage Architecture in LanceDB"
|
||||
sidebarTitle: "Storage Options"
|
||||
description: "Understand LanceDB storage backends, tradeoffs, and how to pick the right option for your latency, scale, and cost goals."
|
||||
icon: "database"
|
||||
---
|
||||
|
||||
LanceDB's storage layer is built on modular, disk-first components. That design makes it flexible enough to run across local NVMe, EBS, EFS, and any object store that exposes an S3-compatible API. It also supports region-specific backends such as [Tencent COS](/storage/configuration#tencent-cos) and cache-acceleration layers such as [GooseFS](/storage/configuration#goosefs).
|
||||
|
||||
Choosing a backend is a balance between latency, scalability, cost, and operational complexity. Use this guide to pick the right fit for your workload.
|
||||
|
||||
## Storage backend selection guide {#storage-backend-selection-guide}
|
||||
|
||||
|
||||
|
||||
When architecting your system, ask yourself:
|
||||
|
||||
- **Latency**: How fast do I need results? What do the p50 and p95 look like?
|
||||
- **Scalability**: Can I scale data volume and QPS easily?
|
||||
- **Cost**: What is the all-in cost of storage plus serving?
|
||||
- **Reliability/Availability**: How will replication and disaster recovery work?
|
||||
|
||||
## Storage backend comparison {#storage-backend-comparison}
|
||||
|
||||
Below is a high-level comparison ordered from lowest cost to lowest latency.
|
||||
|
||||
### 1\. Object storage (S3, GCS, Azure Blob) {#1-object-storage-s3-gcs-azure-blob}
|
||||
- **Latency**: Highest; expect hundreds of milliseconds and higher p95.
|
||||
- **Scalability**: Effectively unlimited storage; QPS bound by concurrency limits.
|
||||
- **Cost**: Lowest overall.
|
||||
- **Reliability/Availability**: Highly available, backed by cloud SLAs.
|
||||
|
||||
LanceDB separates storage and compute and writes immutable fragments, making it a strong fit for stateless, horizontally scalable deployments.
|
||||
|
||||
<Note>
|
||||
**Concurrent writers on S3**
|
||||
|
||||
S3 and S3 Express now support atomic writes natively, so LanceDB handles concurrent writers against the same table out-of-the-box — no external commit coordinator is required. Bucket-level [server-side encryption with KMS](/storage/configuration#server-side-encryption-with-kms) and [S3 Express One Zone](/storage/configuration#s3-express) are also supported on this tier.
|
||||
</Note>
|
||||
|
||||
### 2\. File storage (EFS, GCS Filestore, Azure File) {#2-file-storage-efs-gcs-filestore-azure-file}
|
||||
- **Latency**: Better than object storage; p95 under ~<100ms is typical.
|
||||
- **Scalability**: High, but limited by provisioned IOPS per volume.
|
||||
- **Cost**: More than object storage but cheaper than in-memory options; cold data can tier down automatically.
|
||||
- **Reliability/Availability**: Highly available; replication/backup must be managed separately.
|
||||
|
||||
Keep a copy of data in object storage for disaster recovery. If zero downtime is required, provision a second network file system with replicated data.
|
||||
|
||||
### 3\. Third-party storage (e.g., MinIO, WekaFS) {#3-third-party-storage-e-g--minio-wekafs}
|
||||
|
||||
- **Latency**: Similar to EFS; typically under <100ms.
|
||||
- **Scalability**: Determined by the chosen vendor’s cluster sizing.
|
||||
- **Cost**: Higher than S3; may edge above EFS at larger scales.
|
||||
- **Reliability/Availability**: Shareable across many nodes; replication depends on vendor capabilities.
|
||||
|
||||
### 4\. Block storage (EBS, GCP Persistent Disk, Azure Managed Disk) {#4-block-storage-ebs-gcp-persistent-disk-azure-managed-disk}
|
||||
- **Latency**: Near-local performance; often <30ms.
|
||||
- **Scalability**: Not shareable across instances; shard or copy data when scaling.
|
||||
- **Cost**: Higher than networked file systems, plus potential I/O charges.
|
||||
- **Reliability/Availability**: Persists through instance restarts; backups and sharding must be managed.
|
||||
|
||||
### 5\. Local storage (SSD, NVMe) {#5-local-storage-ssd-nvme}
|
||||
- **Latency**: Fastest; p95 often under <10ms.
|
||||
- **Scalability**: Hard to scale in cloud environments; requires sharding or additional copies for higher QPS.
|
||||
- **Cost**: Highest; tightly coupling compute and storage makes horizontal scaling difficult.
|
||||
- **Reliability/Availability**: Data is tied to the instance; backups must be rigorous.
|
||||
|
||||
Use local disk only when you need extremely low latency and are comfortable owning the operational overhead.
|
||||
|
||||
## File-format choices that interact with the backend {#file-format-choices-that-interact-with-the-backend}
|
||||
|
||||
A few `storage_options` keys shape new tables in ways that depend on the backend you picked above. They are documented in full on the [configuration page](/storage/configuration#new-table-configuration); the architecture-level summary is:
|
||||
|
||||
- `new_table_enable_v2_manifest_paths` matters most on object stores, where opening a table with many versions is dominated by listing cost. Leave it off for backward compatibility with clients older than LanceDB 0.10.0.
|
||||
- `new_table_enable_stable_row_ids` keeps row IDs stable across compaction, delete, and merge. The choice is independent of the backend but affects any system that joins on row ID.
|
||||
- `new_table_data_storage_version` selects the on-disk format. The default `stable` is recommended for all new tables; pick `legacy` only when older readers must keep working.
|
||||
|
||||
Reference in New Issue
Block a user