Files
greptimedb/docs/formats/mito-samples.md
T
discord9 115d85b669 fix(mito2): identify series by their exact sparse key, not by tsid (#10330)
* fix(mito2): identify series by their exact sparse key, not by tsid

`(table_id, tsid)` is a hash, so distinct series can share it: 2000 label
values already produce 20 colliding pairs and a 36k-series SST keeps only 27856
distinct tsids. The columnar series writer silently merged such pairs (first
label set kept, row counts added) and the samples writer either reported
time_regressed or, with increasing timestamps, attached one series' samples to
another's labels. Every one of those decisions used the hash as identity.

The SST already stores the exact sparse primary key, so nothing on the storage
side changes: `tsid`, the hash function and the key layout are untouched. What
changes is that the artifacts, their readers and the prepared read path now
decide identity on the canonical sparse key bytes, and `(table_id, tsid)` is
only the coarse, order-preserving prefix used for ordering, pruning and
bucketing.

- `ExactSeriesKey` is the new identity type: equality, ordering and hashing are
  defined on the key bytes alone, `id()` is documented as a derived prefix that
  is never compared. `from_sparse_key` validates the entire key, not just the
  prefix, and bounds it at 8 KiB.
- Series format v2 keeps one row per exact key; the descriptor layout and the
  existing per-row tags already express this once the writer aggregates by key.
  Coarse page/block/part bounds may now touch, so readers prune conservatively.
- Samples format v2 records the owning key per block, appends a `__primary_key`
  column, and lets a key change restart timestamps and ordinals while the same
  key keeps the monotonic rules. Exact selection travels in
  `SamplesReadRequest.keys` next to the coarse `ids`.
- The prepared path keys candidates by `ExactSeriesKey`, re-encodes identity from
  the artifact's own tags, associates samples through `__primary_key`, and uses
  the exact key for the merge key and the direct-label path.
- Both manifests carry a mandatory `identity_encoding` discriminator. Artifacts
  without it are structurally valid and perfectly ordered yet may contain an
  already merged row, so they are rejected and rebuilt from SSTs; maintenance
  rotates them out instead of reusing them, and the local cache version drops
  the rebuildable store. There is no dual reader and no migration.

Verified: `cargo nextest run -p mito2` 1824 passed, 2 skipped (three times,
identical); `cargo nextest run -p mito-codec` 57 passed; clippy
`-D warnings` and rustfmt clean. Tests pin the real captured pair
(`host-020078` / `host-029838`, both retained with their own labels), the
silent-loss variant with globally increasing timestamps, sixteen distinct keys
under one forced tsid in both PK encodings, candidate isolation end to end, and
rejection of a frozen v1 fixture.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): close the exact-identity review findings

- The two-phase series scan assumed candidate discovery had already enforced
  tag predicates through an "exact assigned-series set". An assignment is a
  `(table_id, tsid)` hash, so a colliding partner shares it without matching
  the predicate, and skipping tag filters returned its rows. Apply the
  predicates the precise filter already holds instead of assuming them.
- `MAX_KEY_BYTES` was checked when reconstructing a key from an artifact but
  not when a writer built one, so the writer could publish a key the reader
  refused; a single 8 KiB label already encodes past the cap. Length is
  bounded by the containers that store a key (one key per block, descriptor,
  page and column limits), and long labels are real data, so drop the cap
  instead of enforcing it on both sides.
- The `__primary_key` column copies one key per output row, but the read
  reservation budgeted a fixed 32 bytes per column and metadata pages did not
  count the per-block key heap: 8192 rows of 8 KiB keys allocated ~64 MiB
  against a ~1 MiB estimate. Budget the real bytes and count the heap.
- `SamplesReadRequest` carried both `keys` and `ids`, which the only caller
  derived from the same candidates; selection is key-only now and the reader
  derives its coarse pruning set, so the two selectors cannot disagree.
- Restore the unknown-field-codec projection coverage that the frozen v1
  fixture used to carry, against a generated v2 artifact, and assert v1
  rejection through the compatibility property rather than a field name.
- Drop the redundant columnar increasing-timestamp test (that writer never
  reads timestamps) and correct the samples format doc: the five internal
  columns and `__primary_key` are non-null, value fields stay nullable.

Verified: `cargo nextest run -p mito2` 1830 passed, 2 skipped; clippy
`-D warnings` and rustfmt clean. Each new test was run with its fix reverted
first: the two-phase test returned the colliding partner's rows, the long-key
tests failed with "series key exceeds 8192 bytes", and the 4 MiB pool test read
8192 rows of 1 KiB keys while allocating ~8 MiB.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): budget the exact key column exactly and drop stale skip-tag claims

The read reservation charged the exact-key column its logical bytes, but the
column was built with `BinaryArray::from_iter_values`, whose value buffer grows
geometrically: measured on a 2135-row fixture it retained 4,989,432 bytes for
2,551,325 bytes of key data (1.956x), so a pool sized from the reservation was
oversubscribed. `BinaryArray::new_repeated` allocates the offsets once and
reserves the key bytes once (1.003x on the same fixture), which is what the
reservation already assumes. The new test asserts the pool covers what the
yielded batch retains, that the column keeps no growth slack, and that the
reservation is released on drop; 2135 rows sits just past a doubling step so a
growing builder fails it.

Also make the documentation match the mechanism and the encoder:

- `read/pruner.rs` still told readers the data phase skips tag filtering because
  candidate discovery had applied the predicates exactly once. Assignments are
  `(table_id, tsid)` hashes, so a colliding partner shares the identifier
  without matching the predicate, and the data phase does apply tag predicates.
- The initial-window fallback test anchored on `reads[5]`, which is the second
  payload window; it now identifies the initial window by its payload-header
  offset and asserts the window spans 128 blocks, so a traversal change cannot
  quietly point it at another read.
- The samples format descriptor sizes are recomputed from the encoder
  (`48 + 44 * C + key_length`): `224`, `45,236`, and with the leaf prefix
  `45,248`, each `+ key_length`; the page-fit sentence now states the enforced
  constraint (descriptor and leaf prefix inside one page) instead of claiming
  the widest descriptor always fits.

Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy
`-D warnings` and rustfmt clean.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): budget each projected sample array's fixed overhead

The read reservation charged per-row bytes plus the payload sizes, but not
the fixed cost of the arrays it builds: Arrow's `get_array_memory_size` is the
array struct plus its buffer capacities, so a one-row, key-only projection
reserved `key_length + 72` for the 1344 bytes it actually retained. Add
`ARRAY_HEADER_BYTES` (`size_of::<BinaryArray>()`) plus `BUFFER_ALIGNMENT` per
projected column, as their own term, and cover the one-row key-only case in
`exact_key_output_reservation_covers_the_retained_array`: the pool must cover
the batch's reported memory size while it is alive and be empty after drop.

Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy
`-D warnings` and rustfmt clean.

Signed-off-by: discord9 <discord9@163.com>

* chore(mito2): pin the series-index cache token to 1

The branch inherits `2` from an earlier layout change. This branch is
experimental and has no deployments, so it pins the token at `1`: whatever a
different build wrote into the rebuildable local store is discarded on startup
and rebuilt from the SSTs.

The token is not the compatibility boundary on its own. Artifacts are rejected
on open when they lack the mandatory `identity_encoding` discriminator, so a
hash-only artifact is never served even when the reset has not run yet (a read
that precedes it, or an artifact outside this cache).

Signed-off-by: discord9 <discord9@163.com>

* feat(mito2): keep exact series keys out of the samples descriptor page

A block descriptor used to embed the whole exact key, and a descriptor shares a
65,536-byte metadata page with its leaf prefix. That put an accidental ceiling on
key length: 65,344 bytes with no value fields, 65,300 with one, and 20,288 with
1,024 fields. Valid SST primary keys can be longer, so their samples builds failed
with "sample descriptor exceeds leaf limit" even though the key itself was
accepted.

The key now lives in a per-part key area and the descriptor carries a reference:

  part    = [header 16][column payloads][key area][leaf pages / root][trailer 120]
  trailer = root ref 72 | depth 4 | version 4 | file_length 8 | payload_end 8 |
            key_end 8 (body 104), magic 8, XXH64 over the preceding 112 bytes 8
  block   = table_id u32 | tsid u64 | key offset u64 | key length u32 | min i64 |
            max i64 | ordinal u64 | rows u32 | columns u32  (56 bytes, fixed)

Descriptor size no longer depends on key length (232 bytes for one present field,
45,244 for 1,024), so the ceilings are gone. Keys are written once per series,
back to back in block order and without padding, so an equal reference means
"same exact series" and a key may exceed any page limit; a reference is validated
against `key_end` with a checked add, an overlap between consecutive new series is
rejected, and a resolved key whose `(table_id, tsid)` disagrees with its descriptor
is refused before any row is emitted.

Reads stay lazy: only blocks that survive coarse selection fetch a key, at most
once per series, merged per page and bounded by `MAX_READ_BYTES`, reused from the
part's aligned cache when one is configured. Measured on the 260-series fixture:
260 per-series key reads before, 1 merged read after, with the fetched byte count
asserted to be exactly the key bytes.

The read contract also stops materializing one key per row: `SamplesBatch
{ batch, key }` carries the block's key once, the exact-key column is refused in
projections, and `prepared` attributes a whole batch with a single binary search.
For an 8,192-row block with a 1 KiB key the reservation drops from 8,388,608 B to
1,120 B (~7,490x), which is what lets a 4 MiB pool read it; `Candidate` keeps one
owned copy of the key instead of two, and `retained_rows` is gone in favour of the
`prepare_retained_rows` metric.

No version bump: the layout changes in place. Artifacts written by an earlier
shape of this branch (112-byte trailer, or keys inside descriptors) are
unreadable and must be rebuilt, which the cache token and the required
`identity_encoding` discriminator already bring about.

One user-visible cost: keys retained by a scan are charged to its memory pool, so
a scan that used to fit in 16 KiB needs ~14 KB more for 128 series. The fixture
that pinned 16 KiB is raised to 64 KiB with the arithmetic in a comment.

Verified: `cargo test -p mito2 --lib -- series_index::samples series_scan::prepared
series_index::identity` 72 passed / 0 failed / 2 ignored on six consecutive runs;
clippy `-D warnings` and rustfmt clean. Failing-before evidence, both produced by
reverting the relevant hunk:
- restoring `48 + 44 * C + key_len` fails the >64 KiB round-trip with
  "sample descriptor exceeds leaf limit";
- the 260-series coalescing assertion fails against per-series reads.

Signed-off-by: discord9 <discord9@163.com>

* test(mito2): assert the duplicate-drop on the preparation's own rows

`overlapping_buckets_retain_only_new_label_rows` compared the deltas of two
process-global counters (`prepare_unique_series`, `prepare_retained_rows`). Every
preparation in the process increments both, so under parallel test execution the
two deltas do not pair up and the test failed intermittently - it always passed in
isolation.

Assert the property the pairing was standing in for, on this preparation's own
rows: read every series of the prepared source and require that no series keeps
two rows at the same timestamp. A surviving duplicate from the overlapping bucket
would repeat a timestamp; the counters can no longer make the result depend on
what else the test binary happens to be running.

Signed-off-by: discord9 <discord9@163.com>

* test(mito2): drop the now-unused preparation counter helper

The duplicate-drop assertion no longer reads the process-global counters, so
`prepared_count` has no callers and `-D warnings` rejects the dead function in
`cargo check` and `cargo clippy` CI. Remove it.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): scope key references per part, bound key reads, checksum keys

Review of the key-area change found three defects that ordinary data can hit and
three smaller ones. All six are fixed with a test that fails beforehand.

Key references are file-local offsets, and the ordering state kept them across
parts: a series continuing into another part looked like an overlapping
reference (or a new series with a non-zero ordinal), and two different series in
structurally identical parts could be mistaken for one because their offsets and
lengths coincided. Reference ordering is now scoped to a part; continuation
across a boundary is decided on the exact key bytes plus time/ordinal, and the
loaded keys must be nondecreasing, a repeated key must continue its series.

A key prefetch fetched the whole window and only then grew the reservation, and
the per-key fallback charged after its read, so a small pool could read and
assemble a page of long keys before reporting exhaustion. The peak (fetched
entry + owned key + alignment/assembly) is now reserved before the read; on
refusal the reader drops alignment first, then halves the window, and a returned
key is cloned only after the block reserved for it. A test asserts that an
unpayable prefetch issues no key read at all.

The key area had no integrity check, so flipping one byte of a label - leaving
framing and `(table_id, tsid)` intact - could silently change which series a
block belongs to. Every stored key now carries its XXH64, excluded from the
reference length and verified when fetched.

The manifest discriminator now names the layout
(`exact-sparse-key-area` for samples; the series index keeps
`exact-sparse-key`), so an artifact with inline keys is refused at open instead
of being marked reusable by maintenance and failing later at a part read. No
version number changes anywhere. A key copy is also counted correctly when a
series spans a metadata rollover: the "first copy of this part" decision is
taken after the rollover, not before.

Verified: `series_index::samples` 51 passed, `series_scan::prepared` 27 passed,
`series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): charge the key peak on every path and chain the order check

Four follow-ups from the confirmation review of the key-area change.

The per-key fallback reserved only the retained and stored bytes and then
fetched through the aligned payload cache, whose aligned/assembly peak it never
charged, so a leftover key could read a whole aligned block the pool had not
paid for. It now reserves the same peak as the page prefetch: aligned budget
first, and on refusal an explicitly cache-less reader with the unaligned peak,
before any read.

The key carried across a part boundary was a deep copy taken while the per-part
reservation was being released, so it lived on uncharged until the next part.
It is now held by its own pool consumer, charged before the copy and released
when it is replaced or when the boundary falls behind the scan.

The content-order check was chained over the last *traversed* block, which an
unloaded block overwrote, so a block pruned by selection could hide reversed or
restarted keys - and could drop the boundary key entirely when a part ended on a
pruned block. It is now chained over the last *loaded* key, with adjacency taken
from "the last traversed block is the last loaded one", so a gap still requires
increasing order and a strictly greater ordinal but not exactly +1.

Finally, the key-area read counter in the tests only saw reads fully inside the
key area, missing aligned reads that start before it; it now counts reads that
intersect, and the cached-read test is documented as a cache-behaviour guard
rather than a failing-before proof for the budget work.

Verified: `series_index::samples` 55 passed, `series_scan::prepared` 27 passed,
`series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean.
Failing-before evidence: a 16384-byte key-area read issued with only 15080
reserved; a boundary key with 0 of its 1195 bytes charged; reversed keys hidden
behind a skipped block; and a pruned block hiding the boundary key.

Signed-off-by: discord9 <discord9@163.com>

* docs(mito2): qualify the ordinal rule to one exact key

The samples format said a loaded key's ordinal must still grow, which reads as if
ordinals had to increase across different series too. They do not: only an equal
exact key continues, an adjacent block of that key requires +1, and a different
key starts at zero.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-12 12:10:51 +08:00

32 KiB

Mito columnar samples format

This document specifies the version 1 on-disk layout, encodings, ordering rules, and validation limits for samples artifacts. The version number stays 1: the exact-identity change is enforced by the mandatory identity_encoding manifest discriminator, not by a version bump.

Every block names the exact series identity (the canonical sparse primary key), which the part stores once per series in its key area, followed by that copy's checksum. An artifact written before exact identity has the same version number and recorded only (table_id, tsid), a hash; it can be structurally valid and perfectly ordered while two distinct series were merged. Nothing in such an artifact proves otherwise, so a manifest without the identity_encoding discriminator is rejected, and its artifact rebuilt from SSTs; there is no dual reader. The same applies to an artifact of the earlier layout of this format, whose keys sat inside the block descriptors: its identity_encoding value is the older exact-sparse-key, so it is refused when the manifest is opened rather than when a part is first read. It shares artifact conventions with Mito columnar series format.

Samples store nullable Float64 fields, timestamps, sequences, and operation types in single-series blocks bounded by rows and decoded bytes.

Artifact manifest

The catalog identifier is columnar-samples. Its file and manifest version is 1, independent of the series format's version and both formats' codec identifiers. Existing parquet and columnar-series identifiers keep their meaning.

An artifact is derived either from one immutable source SST (legacy) or from a bounded group of snapshot SSTs contributing to one UTC day. It owns these objects:

<region_id>/samples/<generation>/manifest.json
<region_id>/samples/<generation>/part-0.samples
<region_id>/samples/<generation>/part-1.samples
...

Daily catalog entries add source_file_ids (the complete, sorted source SST set) and window (signed UTC day number since the Unix epoch). Missing fields retain legacy whole-SST semantics. The catalog key and manifest owner ID for a daily artifact are its unique generation ID, not an individual source SST. Each source group is merged once and rows are routed to nonempty days; sequence numbers, deletion markers, and duplicate versions are preserved. Blocks and parts retain their existing size bounds. An SST crossing midnight contributes to multiple artifacts.

Publication uses the existing atomic samples catalog, which also commits series and range indexes. Readers pin each generation once. A grouped artifact is ineligible if any source SST is absent from the query's data snapshot; exact sequence-range or SST-sequence-pruned scans fall back to the existing path. Maintenance retains complete visible groups and groups only uncovered sources, with at most 64 sources per merge. One anchor entry per group records source_windows, the actual populated days for every source in its group. Other daily entries share the same ordered source set without repeating the coverage map. Coverage checks use this record rather than enumerating the timestamp span. Earlier daily entries without this record require rebuilding. Publication commits all days together; retired objects remain pinned until readers release them. Source Parquet SSTs are unchanged.

The source file ID and generation use existing FileId serialization. The generation is unique for each build. A manifest contains:

Field Meaning
format String columnar-samples
version Integer 1
identity_encoding String exact-sparse-key-area; required, never defaulted. It names the identity and the layout that stores it, so exact-sparse-key (keys inline in the block descriptors) is different and not readable
region_id Physical region ID, u64
source_file_id Artifact owner ID matching SamplesEntry.file_id; the SST ID for legacy artifacts, or the generation ID for daily artifacts
generation FileId matching the catalog artifact generation
timestamp Object with column_id, name, and unit
fields Ordered objects with column_id, name, type, and nullable
files Ordered part descriptors, defined below

The timestamp's ID is u32, its name is the region's time-index name, and its unit is one of second, millisecond, microsecond, or nanosecond. Values are signed i64 counts since the Unix epoch in that unit. There is no dataset-relative origin, unit conversion, or timezone in the file. The timestamp is non-null.

The field list records the build-time region field schema in metadata order. Require 1..=1,024 fields, each with type = "float64" and nullable = true. A non-nullable source Float64 field is represented by this same nullable logical type with no null values. Sparse logical-table fields may be entirely null. Timestamp and field IDs must be distinct. All output names must be unique, including the internal names below. Tags are not sample fields and are not stored here.

The stored logical Arrow schema is __table_id: UInt32, __tsid: UInt64, the timestamp column with its recorded unit, __sequence: UInt64, __op_type: UInt8, then nullable Float64 fields in manifest order, and finally __primary_key: Binary naming the exact key of the block that produced each row. The five internal columns and __primary_key are non-null; value fields stay nullable, as the manifest declares them. __table_id/__tsid are the coarse, hash-based prefix of that key. Tag columns are not stored; the exact key is the identity. A reader returns that key once per output batch — each block belongs to exactly one series — instead of materializing the column per row, and rejects a request that projects it.

Each files entry contains part_id: u32, file_size: u64, num_rows: u64, num_blocks: u64, first, last, min_timestamp: i64, and max_timestamp: i64. first and last are inclusive { table_id: u32, tsid: u64 } bounds. Time bounds are the minimum and maximum over all blocks in the part, not just its first and last rows. Every part is nonempty. Part IDs start at zero, increase by one, and derive part-<part_id>.samples names with decimal IDs and no leading zeros. No object path is stored in a part descriptor.

Parts follow input order. Adjacent identifier ranges satisfy previous.last <= next.first; equality is allowed because a series can continue in another part. The shared boundary series can also have equal timestamps on both sides. These ranges are not disjoint, unlike the series-index format. Readers consider every matching part rather than choosing a unique owner.

ArtifactMetadata.objects records the manifest and every part as { path, file_size }; the artifact's file_size is their checked sum. An empty artifact has an empty file list and no part files. The manifest is at most 64 MiB.

Region, source SST, and generation must match the catalog entry. The artifact object list must contain exactly the derived manifest and part paths, without duplicates, extra objects, or inconsistent size totals. Part order comes from the manifest.

Part file

┌──────────────────────────────────────────────┐
│ Header                              16 bytes │
├──────────────────────────────────────────────┤
│ Independently encoded/compressed chunks      │
│   Block 0: time, sequence, operation, fields  │
│   Block 1: ...                               │
├──────────────────────────────────────────────┤ ← payload_end
│ Key area: one exact series key per series,   │
│   back to back, not page bounded             │
├──────────────────────────────────────────────┤ ← key_end
│ Leaf pages (one is the root at depth 1)       │
│ Optional internal root (depth 2)              │
├──────────────────────────────────────────────┤
│ Trailer                            120 bytes │
└──────────────────────────────────────────────┘

All fixed-width numbers are little-endian, without alignment padding. Byte ranges are half-open; offsets are absolute from the start of the part. Parse each field explicitly. Rust struct layout, enum order, and usize are not wire contracts. Reserved fields must be zero, and decoded structures must have no trailing bytes.

Header

Field Bytes Value
Magic 8 ASCII GTSAMPLE
File version 4 1
Reserved 4 Zero

Column payloads and row semantics

Each block contains exactly one series and names it by that series' exact canonical sparse primary key, which is stored once per series in the part's key area and referenced by the block descriptor. The key is the SST's own encoding of (table_id, tsid, label column-id/value pairs sorted by name, non-null only), and (table_id, tsid) is only its order-preserving prefix. Two distinct series may share a prefix. Timestamp, sequence, operation type, and each present value field are independent chunks; their references are in the metadata, not inline headers. Emit chunks in that order, with value fields following manifest order.

Rows preserve delete markers, duplicate timestamps, and equal-key ties, including conflicting values at the same sequence. The format does not encode a deduplicated or snapshot-filtered view.

Block keys are nondecreasing in byte-lexicographic order, timestamps are nondecreasing within one exact key, and sequences are nonincreasing within equal timestamps, including across parts. A change of exact key restarts timestamps and ordinals, even when the coarse (table_id, tsid) prefix is unchanged. Equal ordering keys retain input order; operation type is not a sorting key. The key area follows the same order, one copy per series. Block keys never decrease, so a key that is read must not sort before the key read before it; a repeated key must continue its timestamps and ordinals rather than restart them. Because key bytes are read lazily, the descriptor-only checks use the (table_id, tsid) prefix and the order, disjointness, and reuse of key references inside one part, and only the keys a request actually loads are checked for byte order and continuation.

Blocks split on an identity change or before the next row would exceed a packing limit. There is no hourly split and no padding of short blocks. The default row limit is 8,192, also the version-1 reader maximum; a writer may choose a smaller positive limit without changing the version. With F manifest fields and N rows, the decoded accounting size is:

N * (8 timestamp + 8 sequence + 1 operation + 8 * F values)
    + F * ceil(N / 8) validity bytes

Require this to be at most 262,144 bytes. Count all manifest fields, including omitted/all-null fields, and reserve validity bytes even for all-valid fields. This is a bound on decoded column buffers, not total allocator usage or RSS. The effective row cap is the largest integer satisfying both limits: 8,192 for one field and 31 for 1,024 fields. Encoded buffers and metadata have separate bounds below. Increasing the reader maximum requires an explicit compatibility decision; it is not merely a writer option in this version.

Payload reference

Every stored column and metadata page uses the same 32-byte reference layout as the series format:

Field Bytes Meaning
Offset 8 Absolute location of stored bytes
Stored length 4 Bytes to fetch
Decoded length 4 Bytes after decompression, before column/page decoding
Checksum 8 XXH64 of stored bytes, seed zero
Encoding ID 2 Sample column or metadata encoding, according to context
Compression ID 2 Compression of the encoded bytes
Reserved 4 Zero

Lengths are nonzero. Check offset + stored_length for u64 overflow before checking the enclosing area. u32 payload lengths do not limit the part or its offsets to 4 GiB. Verify the checksum before decompression; require exactly the declared decoded length, with bounded output even if a codec frame claims more. Checksums detect accidental corruption, not malicious modification.

Key area

The exact series keys occupy their own region between the column payloads and the metadata pages: [payload_end, key_end). Keys are written back to back with no padding and no alignment, in block order, and every block of one series shares one copy of that series' key. The area therefore holds exactly one key per distinct series of the part, ordered by exact key. A descriptor's key reference is unique per series, ordered, and disjoint from every other series' copy, which is why an equal reference means "the same exact series" without reading the bytes.

A key's length is bounded by the part, not by a page or a chunk. Version 1 puts no page inside the key area, so one key may be far longer than the 65,536-byte page limit; the u32 length in a descriptor is validated against key_end, so a corrupt length can never address bytes outside the part.

Each copy is [key bytes][checksum], where the checksum is the u64 XXH64 of those key bytes with seed zero, stored little-endian. A reference's length counts the key bytes only, so the checksum is always the eight bytes after offset + length and is never part of what the reference names. There is no per-key compression and no other framing: the key bytes between them are stored exactly as the SST encodes them, and their own primary-key framing (a fixed reserved prefix plus dense or sparse label encoding) is what decodes them. The checksum is what protects them. A key previously sat inside a metadata page, whose stored bytes the page reference already checksummed; without a per-key checksum, flipping one label byte would keep the framing and the (table_id, tsid) prefix valid and silently attribute the block's rows to another series.

Reading stays lazy. A reader reads only the byte ranges of the keys of the blocks a request selects, at most once per series, and validates a fetched key's checksum, its framing, and its (table_id, tsid) prefix against the descriptor that referenced it. A reference is rejected as soon as its descriptor is decoded — before any row is emitted — when it starts before payload_end, when offset + length + 8 exceeds key_end, when the length is zero, or when two consecutive descriptors of one part that do not name one series overlap.

A key reference is an offset into one part's key area, so the references of two parts are unrelated numbers and are never compared. Whether one exact series continues in another part is decided on the exact key bytes: the last key a request loaded from one part is compared with the first it loads from the next, and an equal key must continue timestamps and ordinals, while a greater key starts a new series at ordinal zero.

Because keys are only read for blocks a request selects, this content check covers the keys actually loaded: unvisited ranges are validated structurally, not by content. It is chained, though, so a selection that skips a block, a page, or a whole part cannot hide the two keys around the gap: the check compares each loaded key with the previous loaded key, not with the previous traversed block. Loaded keys must be nondecreasing. Within one exact key the ordinal must strictly increase, and only immediately adjacent blocks of that same key require it to be exactly one greater; a different key starts a new series at ordinal zero, so no ordering between the two series' ordinals is required.

Metadata pages

A leaf describes blocks. An internal page references leaves. The root is either the only leaf (depth 1) or an internal page (depth 2). Other depths are unsupported in version 1; roll parts before a third level would be needed.

As in the series format, the entire page, including its kind and count, is encoded and independently compressed. The reference in the parent or trailer supplies its storage location, lengths, checksum, and codecs. There is no separate stored page header. Decoded pages begin with kind: u32 and entry_count: u64:

Leaf (kind 0):                     Internal root (kind 1):
  kind                 u32          kind                 u32
  entry_count          u64          entry_count          u64
  block descriptor 0                page reference 0     72 bytes
  ...                               ...
  block descriptor N-1              page reference N-1   72 bytes

Pages are nonempty and limited to 65,536 decoded bytes, including the 12-byte prefix. Descriptors never straddle pages, and every descriptor is fixed-width because the key it names is not stored in it. The writer requires the 12-byte leaf prefix and one descriptor to fit in a configured page, which for the 1,024-field maximum is 12 + 56 + 44 * 1,027 = 45,256 decoded bytes. The page limit is a bound on descriptors only: it no longer bounds the length of a series key.

Block descriptor

Field Bytes Meaning
Table ID 4 Coarse prefix of the constant key in this block
TSID 8 Coarse prefix of the constant key in this block
Key offset 8 Absolute offset of the block's exact key in the part's key area
Key length 4 Nonzero; the key's (table_id, tsid) prefix must equal the two fields above
Minimum timestamp 8 Signed i64, first row's timestamp
Maximum timestamp 8 Signed i64, last row's timestamp
Block ordinal 8 Per-identifier ordinal across the artifact
Row count 4 Nonzero, subject to decoded-block limits
Column count 4 Number of following column descriptors
Column descriptors 44 each Defined below

The fixed prefix is 56 bytes and does not depend on the key: 4 table ID + 8 TSID + 8 key offset + 4 key length + 8 minimum timestamp + 8 maximum timestamp + 8 ordinal + 4 row count + 4 column count. Ordinals start at zero for each exact key and increase by one across block and part boundaries while that key continues. Blocks sort by exact key. Adjacent blocks of the same exact key share one key reference and satisfy previous.max_timestamp <= next.min_timestamp; equality is permitted. Blocks of different keys have no timestamp relation at all, and a new series' key copy starts at or after the end of the previous copy.

Each column descriptor is:

Field Bytes Meaning
Role 4 1 timestamp, 2 sequence, 3 operation type, 4 value field
Column ID 4 Timestamp/field physical ID; zero for sequence/operation
Payload reference 32 Independently stored chunk
Null count 4 Number of null rows

Role and column ID together identify a chunk. The role namespace avoids reserving physical field IDs for internal columns. Time, sequence, and operation descriptors are mandatory, appear first in that order, and have zero null counts. Value descriptors follow manifest order with no repeated or unknown IDs. A value with all rows null is omitted and synthesized as null by the reader; present value chunks have null_count < row_count. No value statistics are stored.

A block with P present fields has C = 3 + P chunks and a descriptor of 56 + 44 * C bytes, whatever its key's length. For one present field that is 56 + 44 * 4 = 232 bytes; at the 1,024-field maximum it is 56 + 44 * 1,027 = 45,244 bytes, and the 12-byte leaf prefix makes that 45,256. Key length is therefore not part of any page budget: a part whose exact key is far longer than a page produces the same descriptors, and the earlier base + key_length ceiling — 65,344 bytes with no value field, 65,300 with one, and 20,288 at the 1,024-field maximum — no longer exists. All chunks are physically ordered and disjoint in the column area, across blocks as well as within a block, and the key area follows the column area. There is no inline block header or block-wide compression.

Page reference and time bounds

Samples extend the series format's reference with two timestamp bounds:

Field Bytes Meaning
First table ID; first TSID 4 + 8 Inclusive first identifier
Last table ID; last TSID 4 + 8 Inclusive last identifier
Minimum timestamp 8 Minimum over all descendant blocks
Maximum timestamp 8 Maximum over all descendant blocks
Payload reference 32 Location and codecs of the child page

The reference is 72 bytes. Compare identifiers as integer pairs, not as serialized little-endian bytes. Time bounds use the manifest unit. The time range is an envelope over all identifiers in the page; it need not describe the first or last identifier alone. This can admit extra reads, but cannot exclude matching rows. Every part/page identifier interval requires first <= last, and all part/page/ block time intervals require min_timestamp <= max_timestamp, even when pruning would skip their payloads.

Child pages follow block order. Their identifier bounds may touch at a shared series, just like part bounds. Do not require strict disjointness or use an exclusive next-child lower bound. Validate each loaded page's derived identifier and time bounds against its reference. At depth 2 all children must be leaves; reject an internal child. Child byte ranges are ordered, disjoint, within the metadata area, and precede the parent. The root ends immediately before the trailer.

An internal root has 12 + 72 * N decoded bytes, so it holds at most floor((65536 - 12) / 72) = 910 leaf references. The writer must account for the next descriptor's leaf placement before writing its block. If the descriptor would require a 911th leaf, finish this part and start the block in a new part.

Navigation and equal-timestamp example

For a requested qualified identity and optional half-open interval [start, end), select every part/page whose identifier interval contains the identity and whose time envelope overlaps the request:

(start is absent || max_timestamp >= start)
    && (end is absent || min_timestamp < end)

For multiple selected identities, intersect the ordered ID set with the inclusive identifier range. A table restriction also intersects this selection. Visit each selected page once per request. Apply the same overlap test to leaf blocks, then filter decoded timestamps exactly. Metadata bounds are conservative candidates. No predecessor-only or unique-child lookup is used: a shared identity or timestamp can require several adjacent children, and even several parts.

Part 0, depth 2:
  Root R
    ├─ Leaf A: IDs [(7,42),(7,42)], times [0,100]
    │    block ordinal 0: (7,42), times [0,100]
    └─ Leaf B: IDs [(7,42),(7,43)], times [0,200]
         block ordinal 1: (7,42), times [100,200]
         block ordinal 0: (7,43), times [0,50]

Part 1:
  Leaf/root C: IDs [(7,43),(7,43)], times [50,80]
    block ordinal 1: (7,43), times [50,80]

For (7,42) and [100,101), visit both A and B, keeping all rows at timestamp 100 from blocks 0 and 1. Do not decode (7,43) payloads. For (7,43) and [50,51), visit B and part 1's C: both may contain rows at 50. A [51,60) request for (7,43) can skip B's (7,43) block even though B's time envelope overlaps. For (7,42) and [90,95), A is still visited although its block starts before the requested interval. Global time envelopes must not be computed from just the first and last blocks: B's range is [0,200], not [100,50].

The series-index format has no sample timestamps or time bounds.

Trailer

Field Bytes Meaning
Root page reference 72 Inclusive ID/time bounds and payload reference
Index depth 4 1 or 2
File version 4 1
File length 8 Includes header and trailer
payload_end 8 End of column area, start of key area
key_end 8 End of key area, start of metadata area
Magic 8 ASCII GTSAMPLE
Trailer checksum 8 XXH64 of the preceding 112 trailer bytes (the 104-byte body and the magic), seed zero

The trailer is 120 bytes. Locate it using the recorded object size without a stat request. Validate the checksum, magic, version, length, depth, area boundaries, root bounds against the manifest part descriptor, and root end position before traversal. Column references must lie in [16, payload_end); key references must lie in [payload_end, key_end); page references lie in [key_end, file_length - 120). The header and trailer versions must agree.

Encodings and compression

Sample column encoding IDs below are distinct from series column encoding IDs, even where their numeric values coincide. Metadata and compression have separate namespaces as well. Never reuse an assigned ID with a different meaning.

Namespace ID Meaning
Column encoding 1 Timestamp constant step
Column encoding 2 Timestamp delta-of-delta
Column encoding 3 Constant sequence
Column encoding 4 Sequence delta
Column encoding 5 Nullable Float64 XOR
Column encoding 6 Constant operation type
Column encoding 7 Raw operation types
Metadata encoding 1 Leaf/internal page layout above
Compression 0 Raw encoded bytes
Compression 1 Zstd frame

Encode first, then compress each chunk/page independently. The default writer tries Zstd level 3 and chooses it only if strictly smaller than the raw encoded bytes. Compression 0 requires equal stored and decoded lengths. Compression level is a writer choice; decoding information is in the frame.

Unsigned varints use base-128/LEB128: seven data bits per byte and a high continuation bit. Writers emit the shortest representation. Readers reject truncation, more than ten bytes, a tenth byte greater than 1, and trailing bytes after the expected values. ZigZag maps signed i64 x to ((x as u64) << 1) ^ ((x >> 63) as u64) and reverses that mapping on decode. All wrapping arithmetic below is modulo 2^64, not signed overflowing arithmetic.

Timestamp encodings 1 and 2

Encoding 1 contains first: i64 and step: i64, exactly 16 bytes. Use it when every adjacent difference is the same nonnegative, representable i64. A singleton uses step zero. Reconstruct by checked addition; reject overflow. Zero step is valid for repeated timestamps.

Encoding 2 contains the first i64, followed for two or more rows by the first delta as a ZigZag varint, then ZigZag varints of differences between deltas. Compute deltas and delta differences with wrapping arithmetic; reconstruct in the same way. This permits differences across the full signed timestamp range. After either decoding, require nondecreasing signed timestamps and exact agreement of first/last values with the descriptor. Consume exactly the recorded row count. Use encoding 1 when applicable, otherwise encoding 2.

Sequence encodings 3 and 4

Encoding 3 contains one u64, repeated for all rows. Encoding 4 contains the first u64 followed by each wrapping u64 difference from its predecessor, reinterpreted as i64 and written as a ZigZag varint. Decode with wrapping addition. Use 3 if all sequences are equal, otherwise 4. Preserve all u64 bit patterns. Sequence need not be monotonic across different timestamps; equal-timestamp runs retain the input's nonincreasing order.

Nullable Float64 encoding 5

validity_kind <u8>                 # 0 = all valid, 1 = bitmap
validity <ceil(row_count/8) bytes>  # only for kind 1
first_non_null_bits <u64>
xor_deltas <one uvarint per remaining non-null value>

Bitmap bit i % 8 in byte i / 8 is one for a present value; unused high bits of the final byte are zero. Kind 0 has no bitmap and requires zero null count. Kind 1 requires 0 < null_count < row_count and an exactly matching count of unset in-range bits. All-null columns have no descriptor or payload, so every stored Float64 chunk contains at least one non-null value.

Process non-null values in row order. Write the first IEEE-754 bit pattern as u64, then varints of XOR with the previous non-null bit pattern. Nulls do not reset that previous value. Decode exactly row_count - null_count values and scatter according to validity. Preserve all bits, including signed zero, NaN payloads, infinities, and subnormals. Null and NaN are distinct. The unspecified value-buffer bits behind nulls are not part of the logical preservation contract.

Operation encodings 6 and 7

Operation values are u8: 0 delete, 1 put, matching Mito's current OpType values. Unknown operation values fail decoding. Encoding 6 stores one byte repeated for the whole block; encoding 7 stores exactly one byte per row. Use 6 when all values are equal, otherwise 7. Neither encoding filters delete rows or discards their field values.

Limits and evolution

An individual chunk has at most 1 MiB of encoded bytes after decompression and 1 MiB plus 64 KiB of stored bytes. Pages have at most 65,536 decoded and 131,072 stored bytes. Series keys are stored in the key area rather than in a page or a chunk, so a key's length is bounded by the part and by its u32 reference length, not by PAGE_LIMIT; a key longer than a page is a supported part of this version. Bound sizes before allocating, fetching, or decompressing. Validate counts against enclosing bytes before allocating descriptor vectors; row counts must also satisfy the decoded-block formula. Null output materialization remains subject to the same block bound even when chunks are omitted.

Unknown file versions, depths, reserved fields, roles, or metadata codecs fail open/read explicitly. Column role must agree with its encoding: time 1/2, sequence 3/4, Float64 5, operation 6/7. An unknown column encoding or compression can be skipped if its chunk is not needed, but attempting to decode it returns an unsupported-codec error. Known encodings assigned to a different role are invalid metadata, even for an unprojected column. Unknown fields or malformed references must not be interpreted as omitted all-null columns.

New column encodings or compression methods receive new IDs. Structural changes, changed field meanings, or incompatible limits require a new version. Deeper metadata trees require an explicit reader-support and rollout decision; an unchanged layout alone does not make them readable by version-1 implementations.

The local cache root contains a layout-version marker. Startup clears the entire series-index cache when this marker is absent or incompatible, then rebuilds artifacts through existing maintenance. Source SST identity remains in metadata.

Local cache rollout

Daily consolidation is the default for the opt-in columnar samples backend, including manual reconciliation. Legacy-format defaults are unchanged. A day may contain several generations with disjoint source sets (at most 64 SSTs per merge), and each generation may contain multiple bounded part files. Source Parquet files are never rewritten.

This branch has no deployments, so the local series-index cache token is pinned to 1: a cache written by any other build is discarded on startup and rebuilt from the SSTs before regions open, and ordinary SST reads remain available meanwhile. That is the only cache migration this layout has. An artifact written by an earlier build of this branch — keys inside the block descriptors, a key area without stored checksums, a 112-byte trailer without key_end — carries a different identity_encoding value, exact-sparse-key instead of exact-sparse-key-area, so it is refused when its manifest is opened. Maintenance treats that refusal as "not reusable" and rebuilds the artifact, so an old artifact is never marked reusable and then fails on the first part read. A part reached without that check would also be rejected: the reader locates a trailer from the recorded object size, reads the last 120 bytes of such a part, and refuses the shifted frame instead of decoding it. Allow for rebuild time and I/O in either direction; do not reuse daily catalogs with an older reader outside this startup check. Existing per-SST catalog entries remain readable by the new reader.

A prepared scan uses a grouped artifact only when all of its sources belong to that scan's assignment or were excluded solely by its time filter. Partial coverage or a stale source set selects the ordinary SST path until reconciliation supplies complete valid coverage.