fix(rust): prevent oversized SQ remap panics (#3861)

## Summary

- preflight IVF-SQ and IVF-HNSW-SQ storage segments touched by
compaction
- preserve unknown legacy fragment coverage as conservatively affected
- cover both fixed-size vector and multivector SQ storage dimensions
- reject only affected partitions whose fixed-size-list child exceeds
Arrow's `u32` take-index limit
- return an actionable `InvalidInput` error before Arrow can panic

## Root cause

Arrow 58 builds fixed-size-list child indices with `u32`. For a large SQ
partition, `partition_rows * vector_dimension` can exceed `u32::MAX`;
the child range then wraps inside `take_fixed_size_list`, producing the
reported `ArrayData::slice` assertion during index remapping.

The preflight now reads raw segment metadata, mirrors Lance's
fragment-bitmap selection (including `None` as affected/unknown), opens
only affected vector segments, and checks their actual storage quantizer
and partition sizes. It derives the stored dimension for both
`FixedSizeList` vectors and flattened `List<FixedSizeList>`
multivectors. Stable-row-ID datasets, deferred remapping, custom
remappers, unaffected segments, non-SQ indexes, safe partitions, and
no-op compactions are unchanged.

Because the upstream Arrow kernel still has the limit, an affected index
must be recreated with default IVF partitioning (or enough partitions to
stay below the reported maximum) before compaction. This changes the
failure from a process panic to a targeted recovery error; it does not
silently alter index structure.

## Validation

- `cargo fmt --all -- --check`
- `cargo test --quiet --features aws,remote -p lancedb
table::optimize::tests --lib`
- `cargo test --quiet --features remote -p lancedb
table::create_index::tests::test_create_index_ivf_hnsw_sq`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples -- -A
clippy::redundant-pub-crate` (the allowance is for pre-existing Rust
1.97 warnings in remote-table code)

Fixes #2866

<!-- lance-gatekeeper-fix:v1 agent=ba2f07b1ee6f9e68a0de1961c164f70e
generation=1 -->

---------

Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>

docs/web/maintenance/compaction.mdx

@@ -23,6 +23,24 @@ three; a deployment handles them in the background and does not support
 Run optimization as regular table maintenance, particularly after many small
 writes or a large ingest.
 
+{/*
+Draft for the OSS compaction SQ partition check release. Keep this section
+unpublished until that behavior ships.
+
+### Recover from an oversized SQ partition {#oversized-sq-partition}
+
+Compacting a table with an IVF scalar-quantized (SQ) vector index can stop
+before rewriting files if an affected partition is too large to remap. The
+error identifies the index, segment, partition size, and vector dimension
+instead of letting the remap fail at Arrow's fixed-size-list `u32`
+child-array limit.
+
+Rebuild the affected vector index with enough IVF partitions to keep each
+partition at or below `floor(u32::MAX / dimension)` vectors, then retry the
+optimize. Rebuilding with the default IVF partitioning can also choose enough
+partitions.
+*/}
+
 ## Optimize a table {#what-it-does}
 
 ```python