* fix(mito2): identify series by their exact sparse key, not by tsid `(table_id, tsid)` is a hash, so distinct series can share it: 2000 label values already produce 20 colliding pairs and a 36k-series SST keeps only 27856 distinct tsids. The columnar series writer silently merged such pairs (first label set kept, row counts added) and the samples writer either reported time_regressed or, with increasing timestamps, attached one series' samples to another's labels. Every one of those decisions used the hash as identity. The SST already stores the exact sparse primary key, so nothing on the storage side changes: `tsid`, the hash function and the key layout are untouched. What changes is that the artifacts, their readers and the prepared read path now decide identity on the canonical sparse key bytes, and `(table_id, tsid)` is only the coarse, order-preserving prefix used for ordering, pruning and bucketing. - `ExactSeriesKey` is the new identity type: equality, ordering and hashing are defined on the key bytes alone, `id()` is documented as a derived prefix that is never compared. `from_sparse_key` validates the entire key, not just the prefix, and bounds it at 8 KiB. - Series format v2 keeps one row per exact key; the descriptor layout and the existing per-row tags already express this once the writer aggregates by key. Coarse page/block/part bounds may now touch, so readers prune conservatively. - Samples format v2 records the owning key per block, appends a `__primary_key` column, and lets a key change restart timestamps and ordinals while the same key keeps the monotonic rules. Exact selection travels in `SamplesReadRequest.keys` next to the coarse `ids`. - The prepared path keys candidates by `ExactSeriesKey`, re-encodes identity from the artifact's own tags, associates samples through `__primary_key`, and uses the exact key for the merge key and the direct-label path. - Both manifests carry a mandatory `identity_encoding` discriminator. Artifacts without it are structurally valid and perfectly ordered yet may contain an already merged row, so they are rejected and rebuilt from SSTs; maintenance rotates them out instead of reusing them, and the local cache version drops the rebuildable store. There is no dual reader and no migration. Verified: `cargo nextest run -p mito2` 1824 passed, 2 skipped (three times, identical); `cargo nextest run -p mito-codec` 57 passed; clippy `-D warnings` and rustfmt clean. Tests pin the real captured pair (`host-020078` / `host-029838`, both retained with their own labels), the silent-loss variant with globally increasing timestamps, sixteen distinct keys under one forced tsid in both PK encodings, candidate isolation end to end, and rejection of a frozen v1 fixture. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): close the exact-identity review findings - The two-phase series scan assumed candidate discovery had already enforced tag predicates through an "exact assigned-series set". An assignment is a `(table_id, tsid)` hash, so a colliding partner shares it without matching the predicate, and skipping tag filters returned its rows. Apply the predicates the precise filter already holds instead of assuming them. - `MAX_KEY_BYTES` was checked when reconstructing a key from an artifact but not when a writer built one, so the writer could publish a key the reader refused; a single 8 KiB label already encodes past the cap. Length is bounded by the containers that store a key (one key per block, descriptor, page and column limits), and long labels are real data, so drop the cap instead of enforcing it on both sides. - The `__primary_key` column copies one key per output row, but the read reservation budgeted a fixed 32 bytes per column and metadata pages did not count the per-block key heap: 8192 rows of 8 KiB keys allocated ~64 MiB against a ~1 MiB estimate. Budget the real bytes and count the heap. - `SamplesReadRequest` carried both `keys` and `ids`, which the only caller derived from the same candidates; selection is key-only now and the reader derives its coarse pruning set, so the two selectors cannot disagree. - Restore the unknown-field-codec projection coverage that the frozen v1 fixture used to carry, against a generated v2 artifact, and assert v1 rejection through the compatibility property rather than a field name. - Drop the redundant columnar increasing-timestamp test (that writer never reads timestamps) and correct the samples format doc: the five internal columns and `__primary_key` are non-null, value fields stay nullable. Verified: `cargo nextest run -p mito2` 1830 passed, 2 skipped; clippy `-D warnings` and rustfmt clean. Each new test was run with its fix reverted first: the two-phase test returned the colliding partner's rows, the long-key tests failed with "series key exceeds 8192 bytes", and the 4 MiB pool test read 8192 rows of 1 KiB keys while allocating ~8 MiB. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): budget the exact key column exactly and drop stale skip-tag claims The read reservation charged the exact-key column its logical bytes, but the column was built with `BinaryArray::from_iter_values`, whose value buffer grows geometrically: measured on a 2135-row fixture it retained 4,989,432 bytes for 2,551,325 bytes of key data (1.956x), so a pool sized from the reservation was oversubscribed. `BinaryArray::new_repeated` allocates the offsets once and reserves the key bytes once (1.003x on the same fixture), which is what the reservation already assumes. The new test asserts the pool covers what the yielded batch retains, that the column keeps no growth slack, and that the reservation is released on drop; 2135 rows sits just past a doubling step so a growing builder fails it. Also make the documentation match the mechanism and the encoder: - `read/pruner.rs` still told readers the data phase skips tag filtering because candidate discovery had applied the predicates exactly once. Assignments are `(table_id, tsid)` hashes, so a colliding partner shares the identifier without matching the predicate, and the data phase does apply tag predicates. - The initial-window fallback test anchored on `reads[5]`, which is the second payload window; it now identifies the initial window by its payload-header offset and asserts the window spans 128 blocks, so a traversal change cannot quietly point it at another read. - The samples format descriptor sizes are recomputed from the encoder (`48 + 44 * C + key_length`): `224`, `45,236`, and with the leaf prefix `45,248`, each `+ key_length`; the page-fit sentence now states the enforced constraint (descriptor and leaf prefix inside one page) instead of claiming the widest descriptor always fits. Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy `-D warnings` and rustfmt clean. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): budget each projected sample array's fixed overhead The read reservation charged per-row bytes plus the payload sizes, but not the fixed cost of the arrays it builds: Arrow's `get_array_memory_size` is the array struct plus its buffer capacities, so a one-row, key-only projection reserved `key_length + 72` for the 1344 bytes it actually retained. Add `ARRAY_HEADER_BYTES` (`size_of::<BinaryArray>()`) plus `BUFFER_ALIGNMENT` per projected column, as their own term, and cover the one-row key-only case in `exact_key_output_reservation_covers_the_retained_array`: the pool must cover the batch's reported memory size while it is alive and be empty after drop. Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy `-D warnings` and rustfmt clean. Signed-off-by: discord9 <discord9@163.com> * chore(mito2): pin the series-index cache token to 1 The branch inherits `2` from an earlier layout change. This branch is experimental and has no deployments, so it pins the token at `1`: whatever a different build wrote into the rebuildable local store is discarded on startup and rebuilt from the SSTs. The token is not the compatibility boundary on its own. Artifacts are rejected on open when they lack the mandatory `identity_encoding` discriminator, so a hash-only artifact is never served even when the reset has not run yet (a read that precedes it, or an artifact outside this cache). Signed-off-by: discord9 <discord9@163.com> * feat(mito2): keep exact series keys out of the samples descriptor page A block descriptor used to embed the whole exact key, and a descriptor shares a 65,536-byte metadata page with its leaf prefix. That put an accidental ceiling on key length: 65,344 bytes with no value fields, 65,300 with one, and 20,288 with 1,024 fields. Valid SST primary keys can be longer, so their samples builds failed with "sample descriptor exceeds leaf limit" even though the key itself was accepted. The key now lives in a per-part key area and the descriptor carries a reference: part = [header 16][column payloads][key area][leaf pages / root][trailer 120] trailer = root ref 72 | depth 4 | version 4 | file_length 8 | payload_end 8 | key_end 8 (body 104), magic 8, XXH64 over the preceding 112 bytes 8 block = table_id u32 | tsid u64 | key offset u64 | key length u32 | min i64 | max i64 | ordinal u64 | rows u32 | columns u32 (56 bytes, fixed) Descriptor size no longer depends on key length (232 bytes for one present field, 45,244 for 1,024), so the ceilings are gone. Keys are written once per series, back to back in block order and without padding, so an equal reference means "same exact series" and a key may exceed any page limit; a reference is validated against `key_end` with a checked add, an overlap between consecutive new series is rejected, and a resolved key whose `(table_id, tsid)` disagrees with its descriptor is refused before any row is emitted. Reads stay lazy: only blocks that survive coarse selection fetch a key, at most once per series, merged per page and bounded by `MAX_READ_BYTES`, reused from the part's aligned cache when one is configured. Measured on the 260-series fixture: 260 per-series key reads before, 1 merged read after, with the fetched byte count asserted to be exactly the key bytes. The read contract also stops materializing one key per row: `SamplesBatch { batch, key }` carries the block's key once, the exact-key column is refused in projections, and `prepared` attributes a whole batch with a single binary search. For an 8,192-row block with a 1 KiB key the reservation drops from 8,388,608 B to 1,120 B (~7,490x), which is what lets a 4 MiB pool read it; `Candidate` keeps one owned copy of the key instead of two, and `retained_rows` is gone in favour of the `prepare_retained_rows` metric. No version bump: the layout changes in place. Artifacts written by an earlier shape of this branch (112-byte trailer, or keys inside descriptors) are unreadable and must be rebuilt, which the cache token and the required `identity_encoding` discriminator already bring about. One user-visible cost: keys retained by a scan are charged to its memory pool, so a scan that used to fit in 16 KiB needs ~14 KB more for 128 series. The fixture that pinned 16 KiB is raised to 64 KiB with the arithmetic in a comment. Verified: `cargo test -p mito2 --lib -- series_index::samples series_scan::prepared series_index::identity` 72 passed / 0 failed / 2 ignored on six consecutive runs; clippy `-D warnings` and rustfmt clean. Failing-before evidence, both produced by reverting the relevant hunk: - restoring `48 + 44 * C + key_len` fails the >64 KiB round-trip with "sample descriptor exceeds leaf limit"; - the 260-series coalescing assertion fails against per-series reads. Signed-off-by: discord9 <discord9@163.com> * test(mito2): assert the duplicate-drop on the preparation's own rows `overlapping_buckets_retain_only_new_label_rows` compared the deltas of two process-global counters (`prepare_unique_series`, `prepare_retained_rows`). Every preparation in the process increments both, so under parallel test execution the two deltas do not pair up and the test failed intermittently - it always passed in isolation. Assert the property the pairing was standing in for, on this preparation's own rows: read every series of the prepared source and require that no series keeps two rows at the same timestamp. A surviving duplicate from the overlapping bucket would repeat a timestamp; the counters can no longer make the result depend on what else the test binary happens to be running. Signed-off-by: discord9 <discord9@163.com> * test(mito2): drop the now-unused preparation counter helper The duplicate-drop assertion no longer reads the process-global counters, so `prepared_count` has no callers and `-D warnings` rejects the dead function in `cargo check` and `cargo clippy` CI. Remove it. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): scope key references per part, bound key reads, checksum keys Review of the key-area change found three defects that ordinary data can hit and three smaller ones. All six are fixed with a test that fails beforehand. Key references are file-local offsets, and the ordering state kept them across parts: a series continuing into another part looked like an overlapping reference (or a new series with a non-zero ordinal), and two different series in structurally identical parts could be mistaken for one because their offsets and lengths coincided. Reference ordering is now scoped to a part; continuation across a boundary is decided on the exact key bytes plus time/ordinal, and the loaded keys must be nondecreasing, a repeated key must continue its series. A key prefetch fetched the whole window and only then grew the reservation, and the per-key fallback charged after its read, so a small pool could read and assemble a page of long keys before reporting exhaustion. The peak (fetched entry + owned key + alignment/assembly) is now reserved before the read; on refusal the reader drops alignment first, then halves the window, and a returned key is cloned only after the block reserved for it. A test asserts that an unpayable prefetch issues no key read at all. The key area had no integrity check, so flipping one byte of a label - leaving framing and `(table_id, tsid)` intact - could silently change which series a block belongs to. Every stored key now carries its XXH64, excluded from the reference length and verified when fetched. The manifest discriminator now names the layout (`exact-sparse-key-area` for samples; the series index keeps `exact-sparse-key`), so an artifact with inline keys is refused at open instead of being marked reusable by maintenance and failing later at a part read. No version number changes anywhere. A key copy is also counted correctly when a series spans a metadata rollover: the "first copy of this part" decision is taken after the rollover, not before. Verified: `series_index::samples` 51 passed, `series_scan::prepared` 27 passed, `series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): charge the key peak on every path and chain the order check Four follow-ups from the confirmation review of the key-area change. The per-key fallback reserved only the retained and stored bytes and then fetched through the aligned payload cache, whose aligned/assembly peak it never charged, so a leftover key could read a whole aligned block the pool had not paid for. It now reserves the same peak as the page prefetch: aligned budget first, and on refusal an explicitly cache-less reader with the unaligned peak, before any read. The key carried across a part boundary was a deep copy taken while the per-part reservation was being released, so it lived on uncharged until the next part. It is now held by its own pool consumer, charged before the copy and released when it is replaced or when the boundary falls behind the scan. The content-order check was chained over the last *traversed* block, which an unloaded block overwrote, so a block pruned by selection could hide reversed or restarted keys - and could drop the boundary key entirely when a part ended on a pruned block. It is now chained over the last *loaded* key, with adjacency taken from "the last traversed block is the last loaded one", so a gap still requires increasing order and a strictly greater ordinal but not exactly +1. Finally, the key-area read counter in the tests only saw reads fully inside the key area, missing aligned reads that start before it; it now counts reads that intersect, and the cached-read test is documented as a cache-behaviour guard rather than a failing-before proof for the budget work. Verified: `series_index::samples` 55 passed, `series_scan::prepared` 27 passed, `series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean. Failing-before evidence: a 16384-byte key-area read issued with only 15080 reserved; a boundary key with 0 of its 1195 bytes charged; reversed keys hidden behind a skipped block; and a pruned block hiding the boundary key. Signed-off-by: discord9 <discord9@163.com> * docs(mito2): qualify the ordinal rule to one exact key The samples format said a loaded key's ordinal must still grow, which reads as if ordinals had to increase across different series too. They do not: only an equal exact key continues, an adjacent block of that key requires +1, and a different key starts at zero. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>
32 KiB
Mito columnar samples format
This document specifies the version 1 on-disk layout, encodings, ordering rules,
and validation limits for samples artifacts. The version number stays 1: the
exact-identity change is enforced by the mandatory identity_encoding manifest
discriminator, not by a version bump.
Every block names the exact series identity (the canonical sparse primary key), which
the part stores once per series in its key area, followed by that copy's
checksum.
An artifact written before exact identity has the same version number and recorded
only (table_id, tsid), a hash; it can be structurally valid and perfectly ordered
while two distinct series were merged. Nothing in such an artifact proves
otherwise, so a manifest without the identity_encoding discriminator is
rejected, and its artifact rebuilt from SSTs; there is no dual reader. The same
applies to an artifact of the earlier layout of this format, whose keys sat inside
the block descriptors: its identity_encoding value is the older
exact-sparse-key, so it is refused when the manifest is opened rather than when a
part is first read. It shares artifact conventions with
Mito columnar series format.
Samples store nullable Float64 fields, timestamps, sequences, and operation types in single-series blocks bounded by rows and decoded bytes.
Artifact manifest
The catalog identifier is columnar-samples. Its file and manifest version is 1,
independent of the series format's version and both formats' codec identifiers.
Existing parquet and columnar-series identifiers keep their meaning.
An artifact is derived either from one immutable source SST (legacy) or from a bounded group of snapshot SSTs contributing to one UTC day. It owns these objects:
<region_id>/samples/<generation>/manifest.json
<region_id>/samples/<generation>/part-0.samples
<region_id>/samples/<generation>/part-1.samples
...
Daily catalog entries add source_file_ids (the complete, sorted source SST
set) and window (signed UTC day number since the Unix epoch). Missing fields
retain legacy whole-SST semantics. The catalog key and manifest owner ID for a
daily artifact are its unique generation ID, not an individual source SST.
Each source group is merged once and rows are routed to nonempty days; sequence numbers, deletion markers,
and duplicate versions are preserved. Blocks and parts retain their existing
size bounds. An SST crossing midnight contributes to multiple artifacts.
Publication uses the existing atomic samples catalog, which also commits series
and range indexes. Readers pin each generation once. A grouped artifact is
ineligible if any source SST is absent from the query's data snapshot; exact
sequence-range or SST-sequence-pruned scans fall back to the existing path.
Maintenance retains complete visible groups and groups only uncovered sources,
with at most 64 sources per merge. One anchor entry per group records source_windows, the
actual populated days for every source in its group. Other daily entries share
the same ordered source set without repeating the coverage map. Coverage checks use this
record rather than enumerating the timestamp span. Earlier daily entries without
this record require rebuilding. Publication commits all days together; retired
objects remain pinned until readers release them. Source Parquet SSTs are unchanged.
The source file ID and generation use existing FileId serialization. The
generation is unique for each build. A manifest contains:
| Field | Meaning |
|---|---|
format |
String columnar-samples |
version |
Integer 1 |
identity_encoding |
String exact-sparse-key-area; required, never defaulted. It names the identity and the layout that stores it, so exact-sparse-key (keys inline in the block descriptors) is different and not readable |
region_id |
Physical region ID, u64 |
source_file_id |
Artifact owner ID matching SamplesEntry.file_id; the SST ID for legacy artifacts, or the generation ID for daily artifacts |
generation |
FileId matching the catalog artifact generation |
timestamp |
Object with column_id, name, and unit |
fields |
Ordered objects with column_id, name, type, and nullable |
files |
Ordered part descriptors, defined below |
The timestamp's ID is u32, its name is the region's time-index name, and its unit
is one of second, millisecond, microsecond, or nanosecond. Values are
signed i64 counts since the Unix epoch in that unit. There is no dataset-relative
origin, unit conversion, or timezone in the file. The timestamp is non-null.
The field list records the build-time region field schema in metadata order.
Require 1..=1,024 fields, each with type = "float64" and nullable = true.
A non-nullable source Float64 field is represented by this same nullable logical
type with no null values. Sparse logical-table fields may be entirely null.
Timestamp and field IDs must be distinct. All output names must be unique,
including the internal names below. Tags are not sample fields and are not stored here.
The stored logical Arrow schema is __table_id: UInt32, __tsid: UInt64, the
timestamp column with its recorded unit, __sequence: UInt64, __op_type: UInt8,
then nullable Float64 fields in manifest order, and finally
__primary_key: Binary naming the exact key of the block that produced each row.
The five internal columns and __primary_key are non-null; value fields stay
nullable, as the manifest declares them. __table_id/__tsid are the coarse,
hash-based prefix of that key. Tag columns are not stored; the exact key is the
identity. A reader returns that key once per output batch — each block belongs to
exactly one series — instead of materializing the column per row, and rejects a
request that projects it.
Each files entry contains part_id: u32, file_size: u64, num_rows: u64,
num_blocks: u64, first, last, min_timestamp: i64, and max_timestamp: i64.
first and last are inclusive { table_id: u32, tsid: u64 } bounds. Time bounds
are the minimum and maximum over all blocks in the part, not just its first
and last rows. Every part is nonempty. Part IDs start at zero, increase by one,
and derive part-<part_id>.samples names with decimal IDs and no leading zeros.
No object path is stored in a part descriptor.
Parts follow input order. Adjacent identifier ranges satisfy
previous.last <= next.first; equality is allowed because a series can continue
in another part. The shared boundary series can also have equal timestamps on
both sides. These ranges are not disjoint, unlike the series-index format.
Readers consider every matching part rather than choosing a unique owner.
ArtifactMetadata.objects records the manifest and every part as { path, file_size };
the artifact's file_size is their checked sum. An empty artifact has an empty
file list and no part files. The manifest is at most 64 MiB.
Region, source SST, and generation must match the catalog entry. The artifact object list must contain exactly the derived manifest and part paths, without duplicates, extra objects, or inconsistent size totals. Part order comes from the manifest.
Part file
┌──────────────────────────────────────────────┐
│ Header 16 bytes │
├──────────────────────────────────────────────┤
│ Independently encoded/compressed chunks │
│ Block 0: time, sequence, operation, fields │
│ Block 1: ... │
├──────────────────────────────────────────────┤ ← payload_end
│ Key area: one exact series key per series, │
│ back to back, not page bounded │
├──────────────────────────────────────────────┤ ← key_end
│ Leaf pages (one is the root at depth 1) │
│ Optional internal root (depth 2) │
├──────────────────────────────────────────────┤
│ Trailer 120 bytes │
└──────────────────────────────────────────────┘
All fixed-width numbers are little-endian, without alignment padding. Byte ranges
are half-open; offsets are absolute from the start of the part. Parse each field
explicitly. Rust struct layout, enum order, and usize are not wire contracts.
Reserved fields must be zero, and decoded structures must have no trailing bytes.
Header
| Field | Bytes | Value |
|---|---|---|
| Magic | 8 | ASCII GTSAMPLE |
| File version | 4 | 1 |
| Reserved | 4 | Zero |
Column payloads and row semantics
Each block contains exactly one series and names it by that series' exact canonical
sparse primary key, which is stored once per series in the part's key area
and referenced by the block descriptor. The key is the SST's own encoding of
(table_id, tsid, label column-id/value pairs sorted by name, non-null only), and
(table_id, tsid) is only its order-preserving prefix. Two distinct series may
share a prefix. Timestamp, sequence, operation type, and each present value field
are independent chunks; their references are in the metadata, not inline headers.
Emit chunks in that order, with value fields following manifest order.
Rows preserve delete markers, duplicate timestamps, and equal-key ties, including conflicting values at the same sequence. The format does not encode a deduplicated or snapshot-filtered view.
Block keys are nondecreasing in byte-lexicographic order, timestamps are
nondecreasing within one exact key, and sequences are nonincreasing within equal
timestamps, including across parts. A change of exact key restarts timestamps and
ordinals, even when the coarse (table_id, tsid) prefix is unchanged. Equal
ordering keys retain input order; operation type is not a sorting key. The key area
follows the same order, one copy per series. Block keys never decrease, so a key that
is read must not sort before the key read before it; a repeated key must continue its
timestamps and ordinals rather than restart them. Because key bytes are read lazily,
the descriptor-only checks use the (table_id, tsid) prefix and the order, disjointness,
and reuse of key references inside one part, and only the keys a request actually loads
are checked for byte order and continuation.
Blocks split on an identity change or before the next row would exceed a packing
limit. There is no hourly split and no padding of short blocks. The default row
limit is 8,192, also the version-1 reader maximum; a writer may choose a smaller
positive limit without changing the version. With F manifest fields and N
rows, the decoded accounting size is:
N * (8 timestamp + 8 sequence + 1 operation + 8 * F values)
+ F * ceil(N / 8) validity bytes
Require this to be at most 262,144 bytes. Count all manifest fields, including omitted/all-null fields, and reserve validity bytes even for all-valid fields. This is a bound on decoded column buffers, not total allocator usage or RSS. The effective row cap is the largest integer satisfying both limits: 8,192 for one field and 31 for 1,024 fields. Encoded buffers and metadata have separate bounds below. Increasing the reader maximum requires an explicit compatibility decision; it is not merely a writer option in this version.
Payload reference
Every stored column and metadata page uses the same 32-byte reference layout as the series format:
| Field | Bytes | Meaning |
|---|---|---|
| Offset | 8 | Absolute location of stored bytes |
| Stored length | 4 | Bytes to fetch |
| Decoded length | 4 | Bytes after decompression, before column/page decoding |
| Checksum | 8 | XXH64 of stored bytes, seed zero |
| Encoding ID | 2 | Sample column or metadata encoding, according to context |
| Compression ID | 2 | Compression of the encoded bytes |
| Reserved | 4 | Zero |
Lengths are nonzero. Check offset + stored_length for u64 overflow before
checking the enclosing area. u32 payload lengths do not limit the part or its
offsets to 4 GiB. Verify the checksum before decompression; require exactly the
declared decoded length, with bounded output even if a codec frame claims more.
Checksums detect accidental corruption, not malicious modification.
Key area
The exact series keys occupy their own region between the column payloads and the
metadata pages: [payload_end, key_end). Keys are written back to back with no
padding and no alignment, in block order, and every block of one series shares one
copy of that series' key. The area therefore holds exactly one key per distinct
series of the part, ordered by exact key. A descriptor's key reference is unique per
series, ordered, and disjoint from every other series' copy, which is why an equal
reference means "the same exact series" without reading the bytes.
A key's length is bounded by the part, not by a page or a chunk. Version 1 puts no
page inside the key area, so one key may be far longer than the 65,536-byte page
limit; the u32 length in a descriptor is validated against key_end, so a
corrupt length can never address bytes outside the part.
Each copy is [key bytes][checksum], where the checksum is the u64 XXH64 of those
key bytes with seed zero, stored little-endian. A reference's length counts the key
bytes only, so the checksum is always the eight bytes after offset + length and is
never part of what the reference names. There is no per-key compression and no other
framing: the key bytes between them are stored exactly as the SST encodes them, and
their own primary-key framing (a fixed reserved prefix plus dense or sparse label
encoding) is what decodes them. The checksum is what protects them. A key previously
sat inside a metadata page, whose stored bytes the page reference already checksummed;
without a per-key checksum, flipping one label byte would keep the framing and the
(table_id, tsid) prefix valid and silently attribute the block's rows to another
series.
Reading stays lazy. A reader reads only the byte ranges of the keys of the blocks a
request selects, at most once per series, and validates a fetched key's checksum, its
framing, and its (table_id, tsid) prefix against the descriptor that referenced it. A
reference is rejected as soon as its descriptor is decoded — before any row is emitted
— when it starts before payload_end, when offset + length + 8 exceeds key_end,
when the length is zero, or when two consecutive descriptors of one part that do not
name one series overlap.
A key reference is an offset into one part's key area, so the references of two parts are unrelated numbers and are never compared. Whether one exact series continues in another part is decided on the exact key bytes: the last key a request loaded from one part is compared with the first it loads from the next, and an equal key must continue timestamps and ordinals, while a greater key starts a new series at ordinal zero.
Because keys are only read for blocks a request selects, this content check covers the keys actually loaded: unvisited ranges are validated structurally, not by content. It is chained, though, so a selection that skips a block, a page, or a whole part cannot hide the two keys around the gap: the check compares each loaded key with the previous loaded key, not with the previous traversed block. Loaded keys must be nondecreasing. Within one exact key the ordinal must strictly increase, and only immediately adjacent blocks of that same key require it to be exactly one greater; a different key starts a new series at ordinal zero, so no ordering between the two series' ordinals is required.
Metadata pages
A leaf describes blocks. An internal page references leaves. The root is either the only leaf (depth 1) or an internal page (depth 2). Other depths are unsupported in version 1; roll parts before a third level would be needed.
As in the series format, the entire page, including its kind and count, is encoded
and independently compressed. The reference in the parent or trailer supplies
its storage location, lengths, checksum, and codecs. There is no separate stored
page header. Decoded pages begin with kind: u32 and entry_count: u64:
Leaf (kind 0): Internal root (kind 1):
kind u32 kind u32
entry_count u64 entry_count u64
block descriptor 0 page reference 0 72 bytes
... ...
block descriptor N-1 page reference N-1 72 bytes
Pages are nonempty and limited to 65,536 decoded bytes, including the 12-byte
prefix. Descriptors never straddle pages, and every descriptor is fixed-width because
the key it names is not stored in it. The writer requires the 12-byte leaf prefix and
one descriptor to fit in a configured page, which for the 1,024-field maximum is
12 + 56 + 44 * 1,027 = 45,256 decoded bytes. The page limit is a bound on
descriptors only: it no longer bounds the length of a series key.
Block descriptor
| Field | Bytes | Meaning |
|---|---|---|
| Table ID | 4 | Coarse prefix of the constant key in this block |
| TSID | 8 | Coarse prefix of the constant key in this block |
| Key offset | 8 | Absolute offset of the block's exact key in the part's key area |
| Key length | 4 | Nonzero; the key's (table_id, tsid) prefix must equal the two fields above |
| Minimum timestamp | 8 | Signed i64, first row's timestamp |
| Maximum timestamp | 8 | Signed i64, last row's timestamp |
| Block ordinal | 8 | Per-identifier ordinal across the artifact |
| Row count | 4 | Nonzero, subject to decoded-block limits |
| Column count | 4 | Number of following column descriptors |
| Column descriptors | 44 each | Defined below |
The fixed prefix is 56 bytes and does not depend on the key: 4 table ID + 8 TSID + 8 key offset + 4 key length + 8 minimum timestamp + 8 maximum timestamp + 8 ordinal + 4 row count + 4 column count. Ordinals start at zero for each exact key and increase
by one across block and part boundaries while that key continues. Blocks sort by exact
key. Adjacent blocks of the same exact key share one key reference and satisfy
previous.max_timestamp <= next.min_timestamp; equality is permitted. Blocks of
different keys have no timestamp relation at all, and a new series' key copy starts at
or after the end of the previous copy.
Each column descriptor is:
| Field | Bytes | Meaning |
|---|---|---|
| Role | 4 | 1 timestamp, 2 sequence, 3 operation type, 4 value field |
| Column ID | 4 | Timestamp/field physical ID; zero for sequence/operation |
| Payload reference | 32 | Independently stored chunk |
| Null count | 4 | Number of null rows |
Role and column ID together identify a chunk. The role namespace avoids reserving
physical field IDs for internal columns. Time, sequence, and operation descriptors
are mandatory, appear first in that order, and have zero null counts. Value
descriptors follow manifest order with no repeated or unknown IDs. A value with
all rows null is omitted and synthesized as null by the reader; present value
chunks have null_count < row_count. No value statistics are stored.
A block with P present fields has C = 3 + P chunks and a descriptor of
56 + 44 * C bytes, whatever its key's length. For one present field that is
56 + 44 * 4 = 232 bytes; at the 1,024-field maximum it is 56 + 44 * 1,027 = 45,244
bytes, and the 12-byte leaf prefix makes that 45,256. Key length is therefore not
part of any page budget: a part whose exact key is far longer than a page produces the
same descriptors, and the earlier base + key_length ceiling — 65,344 bytes with no
value field, 65,300 with one, and 20,288 at the 1,024-field maximum — no longer
exists. All chunks are physically ordered and disjoint in the column area, across
blocks as well as within a block, and the key area follows the column area. There is no
inline block header or block-wide compression.
Page reference and time bounds
Samples extend the series format's reference with two timestamp bounds:
| Field | Bytes | Meaning |
|---|---|---|
| First table ID; first TSID | 4 + 8 | Inclusive first identifier |
| Last table ID; last TSID | 4 + 8 | Inclusive last identifier |
| Minimum timestamp | 8 | Minimum over all descendant blocks |
| Maximum timestamp | 8 | Maximum over all descendant blocks |
| Payload reference | 32 | Location and codecs of the child page |
The reference is 72 bytes. Compare identifiers as integer pairs, not as serialized
little-endian bytes. Time bounds use the manifest unit. The time range is an
envelope over all identifiers in the page; it need not describe the first or last
identifier alone. This can admit extra reads, but cannot exclude matching rows.
Every part/page identifier interval requires first <= last, and all part/page/
block time intervals require min_timestamp <= max_timestamp, even when pruning
would skip their payloads.
Child pages follow block order. Their identifier bounds may touch at a shared series, just like part bounds. Do not require strict disjointness or use an exclusive next-child lower bound. Validate each loaded page's derived identifier and time bounds against its reference. At depth 2 all children must be leaves; reject an internal child. Child byte ranges are ordered, disjoint, within the metadata area, and precede the parent. The root ends immediately before the trailer.
An internal root has 12 + 72 * N decoded bytes, so it holds at most
floor((65536 - 12) / 72) = 910 leaf references. The writer must account for the
next descriptor's leaf placement before writing its block. If the descriptor
would require a 911th leaf, finish this part and start the block in a new part.
Navigation and equal-timestamp example
For a requested qualified identity and optional half-open interval [start, end),
select every part/page whose identifier interval contains the identity and whose
time envelope overlaps the request:
(start is absent || max_timestamp >= start)
&& (end is absent || min_timestamp < end)
For multiple selected identities, intersect the ordered ID set with the inclusive identifier range. A table restriction also intersects this selection. Visit each selected page once per request. Apply the same overlap test to leaf blocks, then filter decoded timestamps exactly. Metadata bounds are conservative candidates. No predecessor-only or unique-child lookup is used: a shared identity or timestamp can require several adjacent children, and even several parts.
Part 0, depth 2:
Root R
├─ Leaf A: IDs [(7,42),(7,42)], times [0,100]
│ block ordinal 0: (7,42), times [0,100]
└─ Leaf B: IDs [(7,42),(7,43)], times [0,200]
block ordinal 1: (7,42), times [100,200]
block ordinal 0: (7,43), times [0,50]
Part 1:
Leaf/root C: IDs [(7,43),(7,43)], times [50,80]
block ordinal 1: (7,43), times [50,80]
For (7,42) and [100,101), visit both A and B, keeping all rows at timestamp
100 from blocks 0 and 1. Do not decode (7,43) payloads. For (7,43) and
[50,51), visit B and part 1's C: both may contain rows at 50. A [51,60) request
for (7,43) can skip B's (7,43) block even though B's time envelope overlaps.
For (7,42) and [90,95), A is still visited although its block starts before
the requested interval. Global time envelopes must not be computed from just
the first and last blocks: B's range is [0,200], not [100,50].
The series-index format has no sample timestamps or time bounds.
Trailer
| Field | Bytes | Meaning |
|---|---|---|
| Root page reference | 72 | Inclusive ID/time bounds and payload reference |
| Index depth | 4 | 1 or 2 |
| File version | 4 | 1 |
| File length | 8 | Includes header and trailer |
payload_end |
8 | End of column area, start of key area |
key_end |
8 | End of key area, start of metadata area |
| Magic | 8 | ASCII GTSAMPLE |
| Trailer checksum | 8 | XXH64 of the preceding 112 trailer bytes (the 104-byte body and the magic), seed zero |
The trailer is 120 bytes. Locate it using the recorded object size without a stat
request. Validate the checksum, magic, version, length, depth, area boundaries,
root bounds against the manifest part descriptor, and root end position before
traversal. Column references must lie in [16, payload_end); key references must lie
in [payload_end, key_end); page references lie in [key_end, file_length - 120).
The header and trailer versions must agree.
Encodings and compression
Sample column encoding IDs below are distinct from series column encoding IDs, even where their numeric values coincide. Metadata and compression have separate namespaces as well. Never reuse an assigned ID with a different meaning.
| Namespace | ID | Meaning |
|---|---|---|
| Column encoding | 1 | Timestamp constant step |
| Column encoding | 2 | Timestamp delta-of-delta |
| Column encoding | 3 | Constant sequence |
| Column encoding | 4 | Sequence delta |
| Column encoding | 5 | Nullable Float64 XOR |
| Column encoding | 6 | Constant operation type |
| Column encoding | 7 | Raw operation types |
| Metadata encoding | 1 | Leaf/internal page layout above |
| Compression | 0 | Raw encoded bytes |
| Compression | 1 | Zstd frame |
Encode first, then compress each chunk/page independently. The default writer tries Zstd level 3 and chooses it only if strictly smaller than the raw encoded bytes. Compression 0 requires equal stored and decoded lengths. Compression level is a writer choice; decoding information is in the frame.
Unsigned varints use base-128/LEB128: seven data bits per byte and a high
continuation bit. Writers emit the shortest representation. Readers reject
truncation, more than ten bytes, a tenth byte greater than 1, and trailing bytes
after the expected values. ZigZag maps signed i64 x to
((x as u64) << 1) ^ ((x >> 63) as u64) and reverses that mapping on decode.
All wrapping arithmetic below is modulo 2^64, not signed overflowing arithmetic.
Timestamp encodings 1 and 2
Encoding 1 contains first: i64 and step: i64, exactly 16 bytes. Use it when
every adjacent difference is the same nonnegative, representable i64. A singleton
uses step zero. Reconstruct by checked addition; reject overflow. Zero step is
valid for repeated timestamps.
Encoding 2 contains the first i64, followed for two or more rows by the first delta as a ZigZag varint, then ZigZag varints of differences between deltas. Compute deltas and delta differences with wrapping arithmetic; reconstruct in the same way. This permits differences across the full signed timestamp range. After either decoding, require nondecreasing signed timestamps and exact agreement of first/last values with the descriptor. Consume exactly the recorded row count. Use encoding 1 when applicable, otherwise encoding 2.
Sequence encodings 3 and 4
Encoding 3 contains one u64, repeated for all rows. Encoding 4 contains the first u64 followed by each wrapping u64 difference from its predecessor, reinterpreted as i64 and written as a ZigZag varint. Decode with wrapping addition. Use 3 if all sequences are equal, otherwise 4. Preserve all u64 bit patterns. Sequence need not be monotonic across different timestamps; equal-timestamp runs retain the input's nonincreasing order.
Nullable Float64 encoding 5
validity_kind <u8> # 0 = all valid, 1 = bitmap
validity <ceil(row_count/8) bytes> # only for kind 1
first_non_null_bits <u64>
xor_deltas <one uvarint per remaining non-null value>
Bitmap bit i % 8 in byte i / 8 is one for a present value; unused high bits of
the final byte are zero. Kind 0 has no bitmap and requires zero null count. Kind
1 requires 0 < null_count < row_count and an exactly matching count of unset
in-range bits. All-null columns have no descriptor or payload, so every stored
Float64 chunk contains at least one non-null value.
Process non-null values in row order. Write the first IEEE-754 bit pattern as
u64, then varints of XOR with the previous non-null bit pattern. Nulls do not
reset that previous value. Decode exactly row_count - null_count values and
scatter according to validity. Preserve all bits, including signed zero, NaN
payloads, infinities, and subnormals. Null and NaN are distinct. The unspecified
value-buffer bits behind nulls are not part of the logical preservation contract.
Operation encodings 6 and 7
Operation values are u8: 0 delete, 1 put, matching Mito's current OpType values.
Unknown operation values fail decoding. Encoding 6 stores one byte repeated for
the whole block; encoding 7 stores exactly one byte per row. Use 6 when all values
are equal, otherwise 7. Neither encoding filters delete rows or discards their
field values.
Limits and evolution
An individual chunk has at most 1 MiB of encoded bytes after decompression and
1 MiB plus 64 KiB of stored bytes. Pages have at most 65,536 decoded and 131,072
stored bytes. Series keys are stored in the key area rather than in a page or a
chunk, so a key's length is bounded by the part and by its u32 reference length,
not by PAGE_LIMIT; a key longer than a page is a supported part of this version. Bound sizes before allocating, fetching, or decompressing. Validate
counts against enclosing bytes before allocating descriptor vectors; row counts
must also satisfy the decoded-block formula. Null output materialization remains
subject to the same block bound even when chunks are omitted.
Unknown file versions, depths, reserved fields, roles, or metadata codecs fail open/read explicitly. Column role must agree with its encoding: time 1/2, sequence 3/4, Float64 5, operation 6/7. An unknown column encoding or compression can be skipped if its chunk is not needed, but attempting to decode it returns an unsupported-codec error. Known encodings assigned to a different role are invalid metadata, even for an unprojected column. Unknown fields or malformed references must not be interpreted as omitted all-null columns.
New column encodings or compression methods receive new IDs. Structural changes, changed field meanings, or incompatible limits require a new version. Deeper metadata trees require an explicit reader-support and rollout decision; an unchanged layout alone does not make them readable by version-1 implementations.
The local cache root contains a layout-version marker. Startup clears the entire series-index cache when this marker is absent or incompatible, then rebuilds artifacts through existing maintenance. Source SST identity remains in metadata.
Local cache rollout
Daily consolidation is the default for the opt-in columnar samples backend, including manual reconciliation. Legacy-format defaults are unchanged. A day may contain several generations with disjoint source sets (at most 64 SSTs per merge), and each generation may contain multiple bounded part files. Source Parquet files are never rewritten.
This branch has no deployments, so the local series-index cache token is pinned to
1: a cache written by any other build is discarded on startup and rebuilt from
the SSTs before regions open, and ordinary SST reads remain available meanwhile.
That is the only cache migration this layout has. An artifact written by an earlier
build of this branch — keys inside the block descriptors, a key area without stored
checksums, a 112-byte trailer without key_end — carries a different
identity_encoding value, exact-sparse-key instead of exact-sparse-key-area, so
it is refused when its manifest is opened. Maintenance treats that refusal as "not
reusable" and rebuilds the artifact, so an old artifact is never marked reusable and
then fails on the first part read. A part reached without that check would also be
rejected: the reader locates a trailer from the recorded object size, reads the last
120 bytes of such a part, and refuses the shifted frame instead of decoding it.
Allow for rebuild time and I/O in either direction; do not reuse daily catalogs with
an older reader outside this startup check. Existing per-SST catalog entries remain
readable by the new reader.
A prepared scan uses a grouped artifact only when all of its sources belong to that scan's assignment or were excluded solely by its time filter. Partial coverage or a stale source set selects the ordinary SST path until reconciliation supplies complete valid coverage.