greptimedb/docs/formats/mito-samples.md

Mito columnar samples format

Status: implemented as an opt-in internal backend. This document specifies a standalone samples writer and reader. Query-consumer integration, caches, and enabling samples by default are separate work.

The port is stacked on #9841, at commit 6b44040c0ad96fb06d592c7bba5a1744e2274718, and follows the presentation and artifact conventions in Mito columnar series. Its sample encodings and single-series blocks derive from the benchmark's columnar samples v2 specification and implementation at revision 8650c3003a861c8cf2c9f5037bfc81fde6a3bdc4. The benchmark variant name does not determine the Mito wire version.

The principal format choices are nullable Float64 fields, preservation of source rows and operation types, blocks bounded by rows and bytes without hourly splits, and Mito framing with sample-specific time bounds. This is an independent format; it does not read or write benchmark files.

Artifact manifest

The catalog identifier is columnar-samples. Its file and manifest version is 1, independent of the series format's version and both formats' codec identifiers. Existing parquet and columnar-series identifiers keep their meaning.

One artifact is derived from one immutable source SST. It owns these objects:

<region_id>/samples/<source_file_id>/<generation>/manifest.json
<region_id>/samples/<source_file_id>/<generation>/part-0.samples
<region_id>/samples/<source_file_id>/<generation>/part-1.samples
...

The source file ID and generation use existing FileId serialization. The generation is unique for each build. A manifest contains:

Field	Meaning
format	String columnar-samples
version	Integer 1
region_id	Physical region ID, u64
source_file_id	SST FileId matching SamplesEntry.file_id
generation	FileId matching the catalog artifact generation
timestamp	Object with column_id, name, and unit
fields	Ordered objects with column_id, name, type, and nullable
files	Ordered part descriptors, defined below

The timestamp's ID is u32, its name is the region's time-index name, and its unit is one of second, millisecond, microsecond, or nanosecond. Values are signed i64 counts since the Unix epoch in that unit. There is no dataset-relative origin, unit conversion, or timezone in the file. The timestamp is non-null.

The field list records the build-time region field schema in metadata order. Require 1..=1,024 fields, each with type = "float64" and nullable = true. A non-nullable source Float64 field is represented by this same nullable logical type with no null values. Sparse logical-table fields may be entirely null. Reject any other source field type; do not silently drop or cast it. Timestamp and field IDs must be distinct, and all output names must be unique, including the internal names below. Tags are not sample fields and are not stored here.

The stored logical Arrow schema is __table_id: UInt32, __tsid: UInt64, the timestamp column with its recorded unit, __sequence: UInt64, __op_type: UInt8, then nullable Float64 fields in manifest order. The first five columns are non-null. No __primary_key or tag columns are exposed by the format reader. Schema reconciliation with a later region schema belongs to a future consumer; the reader always exposes the stored schema.

Each files entry contains part_id: u32, file_size: u64, num_rows: u64, num_blocks: u64, first, last, min_timestamp: i64, and max_timestamp: i64. first and last are inclusive { table_id: u32, tsid: u64 } bounds. Time bounds are the minimum and maximum over all blocks in the part, not just its first and last rows. Every part is nonempty. Part IDs start at zero, increase by one, and derive part-<part_id>.samples names with decimal IDs and no leading zeros. No object path is stored in a part descriptor.

Parts follow input order. Adjacent identifier ranges satisfy previous.last <= next.first; equality is allowed because a series can continue in another part. The shared boundary series can also have equal timestamps on both sides. These ranges are not disjoint, unlike the series-index format. Readers consider every matching part rather than choosing a unique owner.

The manifest is written only after all parts close successfully. Existing samples catalog publication commits the artifact. ArtifactMetadata.objects records the manifest and every part as { path, file_size }; the artifact's file_size is their checked sum. Record sizes while writing, including the manifest's size. An empty writer produces a manifest with an empty file list and no part files; the existing builder may instead skip an SST for which it has no readable input.

On open, bound the manifest to 64 MiB before parsing. Validate its format, schema, part order, and sizes; match region, source SST, and generation to the pinned handle. Require the artifact object list to contain exactly the derived manifest and part paths, without duplicates, extra objects, or inconsistent size totals. Read order comes from the manifest, not the lifecycle object list.

Part file

┌──────────────────────────────────────────────┐
│ Header                              16 bytes │
├──────────────────────────────────────────────┤
│ Independently encoded/compressed chunks      │
│   Block 0: time, sequence, operation, fields  │
│   Block 1: ...                               │
├──────────────────────────────────────────────┤ ← payload_end
│ Leaf pages (one is the root at depth 1)       │
│ Optional internal root (depth 2)              │
├──────────────────────────────────────────────┤
│ Trailer                            112 bytes │
└──────────────────────────────────────────────┘

All fixed-width numbers are little-endian, without alignment padding. Byte ranges are half-open; offsets are absolute from the start of the part. Parse each field explicitly. Rust struct layout, enum order, and usize are not wire contracts. Reserved fields must be zero, and decoded structures must have no trailing bytes.

Header

Field	Bytes	Value
Magic	8	ASCII GTSAMPLE
File version	4	1
Reserved	4	Zero

Column payloads and row semantics

Each block contains exactly one (table_id, tsid). Identity is stored once in its descriptor. Timestamp, sequence, operation type, and each present value field are independent chunks; their references are in the metadata, not inline headers. Emit chunks in that order, with value fields following manifest order.

The input is the flat RecordBatch stream from the existing per-SST samples builder, after its FlatCompatBatch schema adaptation. Preserve every row delivered by that builder. The writer neither deduplicates timestamps nor chooses the highest sequence. It preserves delete markers, duplicate rows, and equal-key ties, including conflicting values at the same sequence. This is not a statement that an SST itself retains every historical write.

Require nondecreasing identifiers, nondecreasing timestamps within an identifier, and nonincreasing sequence within equal timestamps, across batches and parts. Equal ordering keys retain input order; operation type is not a sorting key. Reject decreasing input rather than sorting or buffering a whole SST. Future query adapters remain responsible for snapshot visibility, cross-SST merging, deduplication, and delete processing.

(table_id, tsid) is the complete identity under #9841's current uniqueness assumption. Equal TSIDs in different tables are distinct. Exact sparse primary keys and tags are not retained, and this format does not promise collision-safe identity. TODO: evaluate exact-key storage and collision handling with the series format before using these sidecars as a general query source.

TODO: evaluate adding dedicated job_id and instance_id components to the identifier for locality, following the VM native approach. The current (table_id, tsid) pair is the version-1 identity, not a permanent restriction on future formats. The extended identifier's component order, widths, derivation, and relationship to TSID uniqueness require a separate design and benchmarks. Coordinate that change with the series format: sample block identities, part/page bounds, sorting, and selected-series requests must agree. Changing the persisted identifier layout requires a new format version; do not repurpose reserved bytes or silently reinterpret existing keys. Ordinary job and instance tags do not become dedicated identifier components in this port.

Blocks split on an identity change or before the next row would exceed a packing limit. There is no hourly split and no padding of short blocks. The default row limit is 8,192, also the version-1 reader maximum; a writer may choose a smaller positive limit without changing the version. With F manifest fields and N rows, the decoded accounting size is:

N * (8 timestamp + 8 sequence + 1 operation + 8 * F values)
    + F * ceil(N / 8) validity bytes

Require this to be at most 262,144 bytes. Count all manifest fields, including omitted/all-null fields, and reserve validity bytes even for all-valid fields. This is a bound on decoded column buffers, not total allocator usage or RSS. The effective row cap is the largest integer satisfying both limits: 8,192 for one field and 31 for 1,024 fields. Encoded buffers and metadata have separate bounds below. Increasing the reader maximum requires an explicit compatibility decision; it is not merely a writer option in this version.

Materializing the two identity columns adds at most 12 * N bytes to an output batch. Thus a full-schema output batch has at most 262144 + 12 * 8192 = 360448 bytes of value/validity buffers under this accounting, plus Arrow and allocation overhead. Temporary encoded, compact non-null, and selected-output buffers can coexist; the block limit is not a promise that total reader memory is 256 KiB.

Payload reference

Every stored column and metadata page uses the same 32-byte reference layout as the series format:

Field	Bytes	Meaning
Offset	8	Absolute location of stored bytes
Stored length	4	Bytes to fetch
Decoded length	4	Bytes after decompression, before column/page decoding
Checksum	8	XXH64 of stored bytes, seed zero
Encoding ID	2	Sample column or metadata encoding, according to context
Compression ID	2	Compression of the encoded bytes
Reserved	4	Zero

Lengths are nonzero. Check offset + stored_length for u64 overflow before checking the enclosing area. u32 payload lengths do not limit the part or its offsets to 4 GiB. Verify the checksum before decompression; require exactly the declared decoded length, with bounded output even if a codec frame claims more. Checksums detect accidental corruption, not malicious modification.

Metadata pages

A leaf describes blocks. An internal page references leaves. The root is either the only leaf (depth 1) or an internal page (depth 2). Other depths are unsupported in version 1; roll parts before a third level would be needed.

As in the series format, the entire page, including its kind and count, is encoded and independently compressed. The reference in the parent or trailer supplies its storage location, lengths, checksum, and codecs. There is no separate stored page header. Decoded pages begin with kind: u32 and entry_count: u64:

Leaf (kind 0):                     Internal root (kind 1):
  kind                 u32          kind                 u32
  entry_count          u64          entry_count          u64
  block descriptor 0                page reference 0     72 bytes
  ...                               ...
  block descriptor N-1              page reference N-1   72 bytes

Pages are nonempty and limited to 65,536 decoded bytes, including the 12-byte prefix. Descriptors never straddle pages. Unlike series label descriptors, sample descriptors have no variable-size strings, and even the widest supported descriptor fits in one page. No oversized-leaf exception is needed.

Block descriptor

Field	Bytes	Meaning
Table ID	4	Constant for this block
TSID	8	Constant for this block
Minimum timestamp	8	Signed i64, first row's timestamp
Maximum timestamp	8	Signed i64, last row's timestamp
Block ordinal	8	Per-identifier ordinal across the artifact
Row count	4	Nonzero, subject to decoded-block limits
Column count	4	Number of following column descriptors
Column descriptors	44 each	Defined below

The fixed prefix is 44 bytes. Ordinals start at zero for each identity and increase by one across block and part boundaries. Blocks sort lexicographically by (table_id, tsid, min_timestamp, ordinal). Adjacent blocks of the same series satisfy previous.max_timestamp <= next.min_timestamp; equality is permitted.

Each column descriptor is:

Field	Bytes	Meaning
Role	4	1 timestamp, 2 sequence, 3 operation type, 4 value field
Column ID	4	Timestamp/field physical ID; zero for sequence/operation
Payload reference	32	Independently stored chunk
Null count	4	Number of null rows

Role and column ID together identify a chunk. The role namespace avoids reserving physical field IDs for internal columns. Time, sequence, and operation descriptors are mandatory, appear first in that order, and have zero null counts. Value descriptors follow manifest order with no repeated or unknown IDs. A value with all rows null is omitted and synthesized as null by the reader; present value chunks have null_count < row_count. No value statistics are stored in this port.

A block with P present fields has C = 3 + P chunks and a descriptor of 44 + 44 * C bytes. This is 220 bytes for one present field and 45,232 bytes for 1,024 present fields. The leaf prefix makes the latter 45,244 bytes. All chunks are physically ordered and disjoint in the column area, across blocks as well as within a block. There is no inline block header or block-wide compression.

Page reference and time bounds

Samples extend the series format's reference with two timestamp bounds:

Field	Bytes	Meaning
First table ID; first TSID	4 + 8	Inclusive first identifier
Last table ID; last TSID	4 + 8	Inclusive last identifier
Minimum timestamp	8	Minimum over all descendant blocks
Maximum timestamp	8	Maximum over all descendant blocks
Payload reference	32	Location and codecs of the child page

The reference is 72 bytes. Compare identifiers as integer pairs, not as serialized little-endian bytes. Time bounds use the manifest unit. The time range is an envelope over all identifiers in the page; it need not describe the first or last identifier alone. This can admit extra reads, but cannot exclude matching rows. Every part/page identifier interval requires first <= last, and all part/page/ block time intervals require min_timestamp <= max_timestamp, even when pruning would skip their payloads.

Child pages follow block order. Their identifier bounds may touch at a shared series, just like part bounds. Do not require strict disjointness or use an exclusive next-child lower bound. Validate each loaded page's derived identifier and time bounds against its reference. At depth 2 all children must be leaves; reject an internal child. Child byte ranges are ordered, disjoint, within the metadata area, and precede the parent. The root ends immediately before the trailer.

An internal root has 12 + 72 * N decoded bytes, so it holds at most floor((65536 - 12) / 72) = 910 leaf references. The writer must account for the next descriptor's leaf placement before writing its block. If the descriptor would require a 911th leaf, finish this part and start the block in a new part.

Navigation and equal-timestamp example

For a requested qualified identity and optional half-open interval [start, end), select every part/page whose identifier interval contains the identity and whose time envelope overlaps the request:

(start is absent || max_timestamp >= start)
    && (end is absent || min_timestamp < end)

For multiple selected identities, intersect the ordered ID set with the inclusive identifier range. A table restriction also intersects this selection. Visit each selected page once per request. Apply the same overlap test to leaf blocks, then filter decoded timestamps exactly. Metadata bounds are conservative candidates. No predecessor-only or unique-child lookup is used: a shared identity or timestamp can require several adjacent children, and even several parts.

Part 0, depth 2:
  Root R
    ├─ Leaf A: IDs [(7,42),(7,42)], times [0,100]
    │    block ordinal 0: (7,42), times [0,100]
    └─ Leaf B: IDs [(7,42),(7,43)], times [0,200]
         block ordinal 1: (7,42), times [100,200]
         block ordinal 0: (7,43), times [0,50]

Part 1:
  Leaf/root C: IDs [(7,43),(7,43)], times [50,80]
    block ordinal 1: (7,43), times [50,80]

For (7,42) and [100,101), visit both A and B, keeping all rows at timestamp 100 from blocks 0 and 1. Do not decode (7,43) payloads. For (7,43) and [50,51), visit B and part 1's C: both may contain rows at 50. A [51,60) request for (7,43) can skip B's (7,43) block even though B's time envelope overlaps. For (7,42) and [90,95), A is still visited although its block starts before the requested interval. Global time envelopes must not be computed from just the first and last blocks: B's range is [0,200], not [100,50].

This differs from #9841, which has no sample timestamps or time navigation.

Trailer

Field	Bytes	Meaning
Root page reference	72	Inclusive ID/time bounds and payload reference
Index depth	4	1 or 2
File version	4	1
File length	8	Includes header and trailer
payload_end	8	End of column area, start of metadata area
Magic	8	ASCII GTSAMPLE
Trailer checksum	8	XXH64 of the preceding 104 trailer bytes, seed zero

The trailer is 112 bytes. Locate it using the recorded object size without a stat request. Validate the checksum, magic, version, length, depth, area boundaries, root bounds against the manifest part descriptor, and root end position before traversal. Column references must lie in [16, payload_end); page references lie in [payload_end, file_length - 112). The header and trailer versions must agree.

Encodings and compression

Sample column encoding IDs below are distinct from series column encoding IDs, even where their numeric values coincide. Metadata and compression have separate namespaces as well. Never reuse an assigned ID with a different meaning.

Namespace	ID	Meaning
Column encoding	1	Timestamp constant step
Column encoding	2	Timestamp delta-of-delta
Column encoding	3	Constant sequence
Column encoding	4	Sequence delta
Column encoding	5	Nullable Float64 XOR
Column encoding	6	Constant operation type
Column encoding	7	Raw operation types
Metadata encoding	1	Leaf/internal page layout above
Compression	0	Raw encoded bytes
Compression	1	Zstd frame

Encode first, then compress each chunk/page independently. The default writer tries Zstd level 3 and chooses it only if strictly smaller than the raw encoded bytes, following the sample benchmark. Compression 0 requires equal stored and decoded lengths. Compression level is a writer choice; decoding information is in the frame. Codec selection adds no public configuration.

Unsigned varints use base-128/LEB128: seven data bits per byte and a high continuation bit. Writers emit the shortest representation. Readers reject truncation, more than ten bytes, a tenth byte greater than 1, and trailing bytes after the expected values. ZigZag maps signed i64 x to ((x as u64) << 1) ^ ((x >> 63) as u64) and reverses that mapping on decode. All wrapping arithmetic below is modulo 2^64, not signed overflowing arithmetic.

Timestamp encodings 1 and 2

Encoding 1 contains first: i64 and step: i64, exactly 16 bytes. Use it when every adjacent difference is the same nonnegative, representable i64. A singleton uses step zero. Reconstruct by checked addition; reject overflow. Zero step is valid for repeated timestamps.

Encoding 2 contains the first i64, followed for two or more rows by the first delta as a ZigZag varint, then ZigZag varints of differences between deltas. Compute deltas and delta differences with wrapping arithmetic; reconstruct in the same way. This permits differences across the full signed timestamp range. After either decoding, require nondecreasing signed timestamps and exact agreement of first/last values with the descriptor. Consume exactly the recorded row count. Use encoding 1 when applicable, otherwise encoding 2.

Sequence encodings 3 and 4

Encoding 3 contains one u64, repeated for all rows. Encoding 4 contains the first u64 followed by each wrapping u64 difference from its predecessor, reinterpreted as i64 and written as a ZigZag varint. Decode with wrapping addition. Use 3 if all sequences are equal, otherwise 4. Preserve all u64 bit patterns. Sequence need not be monotonic across different timestamps; equal-timestamp runs retain the input's nonincreasing order.

Nullable Float64 encoding 5

validity_kind <u8>                 # 0 = all valid, 1 = bitmap
validity <ceil(row_count/8) bytes>  # only for kind 1
first_non_null_bits <u64>
xor_deltas <one uvarint per remaining non-null value>

Bitmap bit i % 8 in byte i / 8 is one for a present value; unused high bits of the final byte are zero. Kind 0 has no bitmap and requires zero null count. Kind 1 requires 0 < null_count < row_count and an exactly matching count of unset in-range bits. All-null columns have no descriptor or payload, so every stored Float64 chunk contains at least one non-null value.

Process non-null values in row order. Write the first IEEE-754 bit pattern as u64, then varints of XOR with the previous non-null bit pattern. Nulls do not reset that previous value. Decode exactly row_count - null_count values and scatter according to validity. Preserve all bits, including signed zero, NaN payloads, infinities, and subnormals. Null and NaN are distinct. The unspecified value-buffer bits behind nulls are not part of the logical preservation contract.

Operation encodings 6 and 7

Operation values are u8: 0 delete, 1 put, matching Mito's current OpType values. Unknown operation values fail decoding. Encoding 6 stores one byte repeated for the whole block; encoding 7 stores exactly one byte per row. Use 6 when all values are equal, otherwise 7. Neither encoding filters delete rows or discards their field values.

Limits and evolution

An individual chunk has at most 1 MiB of encoded bytes after decompression and 1 MiB plus 64 KiB of stored bytes. Pages have at most 65,536 decoded and 131,072 stored bytes. Bound sizes before allocating, fetching, or decompressing. Validate counts against enclosing bytes before allocating descriptor vectors; row counts must also satisfy the decoded-block formula. Null output materialization remains subject to the same block bound even when chunks are omitted.

Unknown file versions, depths, reserved fields, roles, or metadata codecs fail open/read explicitly. Column role must agree with its encoding: time 1/2, sequence 3/4, Float64 5, operation 6/7. An unknown column encoding or compression can be skipped if its chunk is not needed, but attempting to decode it returns an unsupported-codec error. Known encodings assigned to a different role are invalid metadata, even for an unprojected column. Unknown fields or malformed references must not be interpreted as omitted all-null columns.

New column encodings or compression methods receive new IDs. Structural changes, changed field meanings, or incompatible limits require a new version. Deeper metadata trees require an explicit reader-support and rollout decision; an unchanged layout alone does not make them readable by version-1 implementations.

Writer, OpenDAL I/O, and lifecycle

Implement behind SamplesFormat::writer and FormatWriter, using the existing object_store::ObjectStore OpenDAL operator. Extend the internal writer factory with an explicit source SST FileId so the manifest does not infer identity by parsing an opaque prefix. Check that the supplied prefix agrees with metadata, source ID, and generation. The builder supplies these values from its pinned SST and captured region version; fake backends follow the same internal interface.

Validate supported sparse primary-key metadata and extract (table_id, tsid) with the same conventions as the series port. Accept the builder's binary or dictionary primary-key array representation. Validate column types and mandatory non-null values before writing; preserve ordering state across write calls. Retain only the active block and bounded per-part descriptors, not all SST rows. No external sort, benchmark manifest/schema file, or filesystem scratch spool is introduced.

Write chunks as each block completes. Finish a part at the next block boundary once its stored position reaches 512 MiB or its accumulated serialized block descriptors reach 8 MiB. Both are soft packing limits; one bounded block may overshoot. Root capacity is a separate hard limit checked before emitting that block. Write leaves, the internal root if needed, and the trailer, then close the part before opening its successor. Bound the manifest and lifecycle object-list growth as parts complete; fail rather than constructing a manifest over 64 MiB.

Use recorded sizes for reads; do not issue object stats just to locate trailers. Fetch metadata and selected column ranges only. Reads are sequential in this initial port, without speculative adjacent-field fetches or prefetch. Keep fetch/validate/decode entrypoints for metadata and columns separate so a later cache integration can use immutable generation, part, byte range, checksum, codec IDs, and representation without changing the format.

Writers own all partial outputs until finish succeeds. Write/close errors, explicit abort, and cancellation clean up every created object, including an unfinished manifest. Use the existing cleanup/runtime pattern from #9841; cleanup errors must not hide the original error. A process crash can leave orphan objects, but never publishes an unfinished generation as valid. Reconciliation and existing orphan handling remain responsible for recovery.

After finish, ownership transfers to the build/catalog lifecycle. Failed catalog publication retires unpublished outputs and preserves the prior catalog. Each reader and returned stream holds a SamplesFileHandle; superseded objects are purged only after those handles are released. Sample sidecars remain rebuildable, and authoritative data stays in Parquet SSTs.

Reader API and execution

Add a sample-specific reader factory to SamplesFormat, accepting the store and a pinned SamplesFileHandle. Provide access to the handle's region/source identity for manifest validation. Return a SamplesFormatReader exposing schema() and read(SamplesReadRequest) as an independent stream of projected Arrow batches. Do not reuse the series-only read request: it has no time semantics.

The internal request has these fields:

Field	Meaning
projection: Vec<String>	Ordered names from the stored logical schema
table_id: Option<u32>	Optional table restriction
ids: Option<BTreeSet<MetricSeriesId>>	Optional qualified identity set
start: Option<i64>	Inclusive timestamp in the stored unit
end: Option<i64>	Exclusive timestamp in the stored unit

ids = None means unrestricted; an explicitly empty set means zero rows. Intersect ids with table_id when both are supplied. Validate unknown or duplicate projection names before part I/O; preserve requested output order. An empty projection preserves selected row counts. Equal time endpoints produce an empty stream, and a reversed interval is an error. Unbounded endpoints avoid sentinel arithmetic and permit selecting i64::MAX with no end bound. Callers perform any unit conversion explicitly; the format API does not guess units.

1. The factory opens the bounded manifest, validates handle identity and recorded sizes, and exposes the stored schema. Each read validates its request even if the artifact or selection is empty. An empty selection needs no part reads.
2. Prune parts using inclusive ID ranges and time envelopes. Read and validate selected headers/trailers, then traverse only candidate pages in physical order. At depth 2, retain the root references and one active leaf.
3. Select blocks by exact identity and overlapping time bounds. Decode timestamp chunks for a constrained time request, validate their bounds, and select rows. With no time restriction, an unprojected timestamp chunk need not be fetched.
4. Fetch the remaining projected chunks only for blocks with surviving rows, reusing a timestamp chunk already decoded for selection. Decode each chunk once per block, synthesize identity constants and omitted null fields, and emit selected rows in input order. One output batch per selected block is the initial packing choice, with the additional identity-column accounting above.

Sequence and operation chunks are mandatory on disk but are read only if projected. The format reader does not silently apply visibility or delete rules. A future query adapter must request them when required for merging. There are no tag/value predicates in this initial request, no selector-result cache, and no cross-read page/data cache. All reads have independent traversal state and can outlive the factory. Validation is incremental over visited metadata and decoded chunks, not an eager whole-artifact integrity scan; unselected payload corruption is not guaranteed to be discovered by a selective query. Check block ordinals and row ordering across adjacent blocks when both are read; do not require the first selected block to have ordinal zero after pruning. A complete traversal also verifies the manifest's row and block totals. These totals are diagnostic summaries, not allocation sizes or a replacement for block row counts when producing output.

Compatibility, validation, and deferred work

The backend is opt-in through internal format selection, like #9841. Existing catalogs, Parquet artifacts, and series artifacts retain their meaning. An older binary is not promised to understand the new catalog identifier or sample bytes; do not claim downgrade compatibility merely because the samples are rebuildable. The new backend must not become necessary for reading authoritative SSTs. Query-consumer integration and a public enablement/rollout mechanism are separate work.

Implementation acceptance includes:

• Codec tests and round trips for every supported timestamp unit, repeated and negative timestamps, signed extremes, full-range sequences, both operations, null/NaN distinction, Float64 bit patterns, and nulls between XOR values.
• Empty input, singleton blocks, all-null/missing fields, mixed tables and sparse schemas, identical TSIDs in different tables, cross-batch continuation, duplicate/conflicting rows, and decreasing-input rejection.
• Row and decoded-byte caps with narrow/wide schemas; smaller row limits; blocks spanning hours; forced payload/metadata rollover; depths 1/2 and root-capacity rollover without creating depth 3. Reject depths 0/3 and internal children.
• Projection order and empty projection, all/empty/absent ID sets, table/ID intersection, exact time boundaries, unbounded endpoints, and equal timestamps spanning multiple leaves and parts. Verify selected physical ranges: no unused value chunks, duplicate page reads, or hidden stat requests.
• Truncation, corrupt checksums, invalid area bounds, u64 offset overflow, references beyond 4 GiB, wrong lengths/counts/roles/ordinals, inconsistent time bounds, malformed validity/varints, unsupported codecs, and bounded decompression. Test malformed metadata without allocating from untrusted counts.
• Manifest/schema/handle identity mismatches, wrong or duplicate object paths, size accounting, reopen and reconciliation, OpenDAL write/close failures, abort/cancellation, failed catalog publication, and retirement with pinned streams on memory and filesystem stores.
• A persisted-format compatibility case under tests/compatibility/, following its README and agent guide, covering the approved version-1 bytes and legacy catalog behavior. Do not substitute a SQL query test that never enables this internal backend or claim cross-binary coverage from a same-build round trip.
• cargo nextest run -p mito2, crate Clippy, formatting, and license checks for new sources. Any broader workspace validation and its limitations must be reported when the implementation PR is prepared.

The benchmark comparison changes materially: nullable fields and operation types add bytes; native timestamp units and preserved duplicate rows change codec inputs; 64-bit checksums, Mito manifests/references, and overlapping page bounds change framing; bounded two-level trees replace disk-spooled arbitrary-depth indexes; row/byte packing replaces the benchmark's hourly default. OpenDAL and the absence of caches also change performance. Measure the port independently; no performance equivalence is asserted.

The benchmark follow-ups from PR #9887 include:

• High-churn series with only a few rows per SST. Report stored bytes per row and separate metadata from column payloads across different rows-per-series distributions. Each identity change starts a new block, whose descriptor is 220 bytes before metadata compression with one present field, in addition to the mandatory timestamp, sequence, and operation chunks and the value chunk.
• Integer-valued Float64 counters. Compare the current XOR encoding with an exact first/step progression encoding like the prototype in PR #9556, measuring encoded and compressed sizes as well as encode/decode throughput. Include steady counters, resets, irregular increments, and nulls; any candidate must preserve Float64 bits exactly and fall back for values it cannot represent losslessly.

These measurements are deferred and do not change the version-1 wire contract.

Additional field types, value statistics/predicates, exact primary keys, dedicated job/instance identifier components, label reconstruction, cache integration, deeper trees, and query integration are deferred. The byte assignments above define the version-1 contract. The frozen sample fixture checks those bytes independently of writer/reader round trips.