rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-11-13 14:16:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: support remote table compaction via SQL OPTIMIZE TABLE statement (#4802)

Browse Source 
`Table::optimize()` previously returned `NotSupported` for remote
tables. This PR wires it to the LanceDB Enterprise server's `OPTIMIZE
TABLE ... REWRITE DATA` SQL statement, submitted over Flight SQL.

Only `OptimizeAction::Compact` without a custom `remap_options` or
non-default `CompactionOptions` is supported remotely.
`OptimizeAction::All`, `::Prune`, and `::Index` all return a clear
`NotSupported` error instead of silently doing the wrong thing. A
branch-scoped table handle submits a `WITH (branch = '<branch>')`
clause; the statement doesn't validate the branch exists itself, so an
unknown branch surfaces as an async job failure rather than an error
from this call.

The server's SQL job adapter doesn't expose compaction metrics, so
`OptimizeStats::compaction` is `Some(CompactionMetrics::default())`, a
placeholder rather than real numbers — it must still be `Some`, since
callers (including the Python/Node bindings) unwrap it unconditionally
on success. This mirrors how `refresh_materialized_view_async` already
handles the same limitation for its own metrics.

The Python and Node `optimize()` convenience methods call a new shared
`Table::optimize_full`, which runs `Compact` and then, on a remote
table, skips `Prune` and `Index` instead of attempting and failing on
them — returning zeroed prune stats rather than submitting a compaction
job and then raising. Python's `RemoteTable.optimize()` also now
forwards `compaction_options` to the Rust layer, so a non-default value
is rejected before submission instead of being silently dropped.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Bruno Ramirez
2026-11-04 11:25:49 -08:00
committed by GitHub
co-authored by Claude Sonnet 5
parent dcda5224ee
commit a9ad499781
14 changed files with 753 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/src/python/python.md
+8 -3
View File
@@ -348,9 +348,14 @@ index statistics, and search. For example, `table.delete(where="id = 1")`,
`create_scalar_index` defaults to `replace=True` on both backends. Both scalar
and full-text index creation accept `wait_timeout`. LanceDB Enterprise maintenance methods
accept the LanceDB Embedded options: cleanup and compaction warn that they are no-ops,
while `optimize` reports an unsupported operation. Removed Tantivy options
produce the same explanatory errors on both backends.
accept the LanceDB Embedded options: cleanup and compaction warn that they are no-ops.
`optimize` submits a durable compaction job and returns once that submission is
accepted, not once compaction itself finishes. Pruning and index optimization
are not yet supported remotely, so it silently skips them and returns with
zeroed pruning statistics rather than attempting and failing. A non-default
`compaction_options` raises `NotImplementedError` before anything is
submitted, since there is no way yet to forward those settings to the server.
Removed Tantivy options produce the same explanatory errors on both backends.
The old LanceDB Enterprise keyword names `predicate` for `delete`, `index_name` for
`drop_index`, `index_uuid` for `index_stats`, and `column` for `create_fts_index`