* refactor(mito2): drop the unreachable per-SST samples builder Production only ever constructs the columnar samples format: SeriesFormats::from_config is reached from with_format with experimental_series_index_format, where Legacy yields no samples backend and Columnar yields ColumnarSamplesFormat. The other SamplesFormat impl is cfg(test). So the per-SST branch of the maintenance task could never run, and build_samples had no caller but its own test. The daily columnar body is now unconditional, and the former branch position holds an explicit UnsupportedOperation error so an unknown backend fails loudly instead of building nothing. The two fake-format lifecycle tests were duplicates of real_samples_reconcile_reopen_and_retire, which drives the same build/reuse/reopen/ catalog-failure/purge scenario against the real format; the abort test moved to build_sample_windows, and a new test pins the refusal. The compatibility fixtures are untouched and still pass. Signed-off-by: discord9 <discord9@163.com> * test(mito2): cover retirement after a successful samples rebuild format_skeleton_build_reopen_and_retire was the only test of the retirement that follows a successful rebuild; real_samples_reconcile_reopen_and_retire covered the failed-publish path only. Removing the samples loop's mark_deleted() from publish_index_version left every other series_index and series_scan test green, which is what made the gap invisible. The real-format test now also rebuilds a day by deleting one of its objects, checks that the superseded generation is no longer in the published snapshot, that nothing is purged while the old snapshot and its handle are alive, and that its objects are purged once they are dropped. With mark_deleted() commented out, all four cases fail on that last assertion. Signed-off-by: discord9 <discord9@163.com> * test(mito2): pin the purge to the last reader handle, not the snapshot Dropping the published snapshot was allowed to release a superseded generation as long as one of its handles was still held elsewhere, which no test covered: the retirement assertions only checked that nothing was purged while the snapshot lived, then dropped every handle at once. The test now holds a reader handle on every generation the superseded snapshot published before releasing it, so that release must purge nothing, and asserts that between the two drops. A rebuild replaces several generations of a source group at once, so holding only the one the test tracks would let its siblings be released legitimately and the assertion would fail for a reason unrelated to the lifecycle under test. A mutation that purges on snapshot drop -- ignoring live reader handles -- fails this new assertion in all four cases and none of the later ones. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>
32 KiB
Mito columnar samples format
This document specifies the version 1 on-disk layout, encodings, ordering rules,
and validation limits for samples artifacts. The version number stays 1: the
exact-identity change is enforced by the mandatory identity_encoding manifest
discriminator, not by a version bump.
Every block names the exact series identity (the canonical sparse primary key), which
the part stores once per series in its key area, followed by that copy's
checksum.
An artifact written before exact identity has the same version number and recorded
only (table_id, tsid), a hash; it can be structurally valid and perfectly ordered
while two distinct series were merged. Nothing in such an artifact proves
otherwise, so a manifest without the identity_encoding discriminator is
rejected, and its artifact rebuilt from SSTs; there is no dual reader. The same
applies to an artifact of the earlier layout of this format, whose keys sat inside
the block descriptors: its identity_encoding value is the older
exact-sparse-key, so it is refused when the manifest is opened rather than when a
part is first read. It shares artifact conventions with
Mito columnar series format.
Samples store nullable Float64 fields, timestamps, sequences, and operation types in single-series blocks bounded by rows and decoded bytes.
Artifact manifest
The catalog identifier is columnar-samples. Its file and manifest version is 1,
independent of the series format's version and both formats' codec identifiers.
Existing parquet and columnar-series identifiers keep their meaning.
An artifact is derived either from one immutable source SST (legacy) or from a bounded group of snapshot SSTs contributing to one UTC day. It owns these objects:
<region_id>/samples/<generation>/manifest.json
<region_id>/samples/<generation>/part-0.samples
<region_id>/samples/<generation>/part-1.samples
...
Daily catalog entries add source_file_ids (the complete, sorted source SST
set) and window (signed UTC day number since the Unix epoch). Missing fields
retain legacy whole-SST semantics. The catalog key and manifest owner ID for a
daily artifact are its unique generation ID, not an individual source SST.
Each source group is merged once and rows are routed to nonempty days; sequence numbers, deletion markers,
and duplicate versions are preserved. Blocks and parts retain their existing
size bounds. An SST crossing midnight contributes to multiple artifacts.
Publication uses the existing atomic samples catalog, which also commits series
and range indexes. Readers pin each generation once. A grouped artifact is
ineligible if any source SST is absent from the query's data snapshot; exact
sequence-range or SST-sequence-pruned scans fall back to the existing path.
Maintenance retains complete visible groups and groups only uncovered sources,
with at most 64 sources per merge. One anchor entry per group records source_windows, the
actual populated days for every source in its group. Other daily entries share
the same ordered source set without repeating the coverage map. Coverage checks use this
record rather than enumerating the timestamp span. Earlier daily entries without
this record require rebuilding. Publication commits all days together; retired
objects remain pinned until readers release them. Source Parquet SSTs are unchanged.
The source file ID and generation use existing FileId serialization. The
generation is unique for each build. A manifest contains:
| Field | Meaning |
|---|---|
format |
String columnar-samples |
version |
Integer 1 |
identity_encoding |
String exact-sparse-key-area; required, never defaulted. It names the identity and the layout that stores it, so exact-sparse-key (keys inline in the block descriptors) is different and not readable |
region_id |
Physical region ID, u64 |
source_file_id |
Artifact owner ID matching SamplesEntry.file_id; the SST ID for legacy artifacts, or the generation ID for daily artifacts |
generation |
FileId matching the catalog artifact generation |
timestamp |
Object with column_id, name, and unit |
fields |
Ordered objects with column_id, name, type, and nullable |
files |
Ordered part descriptors, defined below |
The timestamp's ID is u32, its name is the region's time-index name, and its unit
is one of second, millisecond, microsecond, or nanosecond. Values are
signed i64 counts since the Unix epoch in that unit. There is no dataset-relative
origin, unit conversion, or timezone in the file. The timestamp is non-null.
The field list records the build-time region field schema in metadata order.
Require 1..=1,024 fields, each with type = "float64" and nullable = true.
A non-nullable source Float64 field is represented by this same nullable logical
type with no null values. Sparse logical-table fields may be entirely null.
Timestamp and field IDs must be distinct. All output names must be unique,
including the internal names below. Tags are not sample fields and are not stored here.
The stored logical Arrow schema is __table_id: UInt32, __tsid: UInt64, the
timestamp column with its recorded unit, __sequence: UInt64, __op_type: UInt8,
then nullable Float64 fields in manifest order, and finally
__primary_key: Binary naming the exact key of the block that produced each row.
The five internal columns and __primary_key are non-null; value fields stay
nullable, as the manifest declares them. __table_id/__tsid are the coarse,
hash-based prefix of that key. Tag columns are not stored; the exact key is the
identity. A reader returns that key once per output batch — each block belongs to
exactly one series — instead of materializing the column per row, and rejects a
request that projects it.
Each files entry contains part_id: u32, file_size: u64, num_rows: u64,
num_blocks: u64, first, last, min_timestamp: i64, and max_timestamp: i64.
first and last are inclusive { table_id: u32, tsid: u64 } bounds. Time bounds
are the minimum and maximum over all blocks in the part, not just its first
and last rows. Every part is nonempty. Part IDs start at zero, increase by one,
and derive part-<part_id>.samples names with decimal IDs and no leading zeros.
No object path is stored in a part descriptor.
Parts follow input order. Adjacent identifier ranges satisfy
previous.last <= next.first; equality is allowed because a series can continue
in another part. The shared boundary series can also have equal timestamps on
both sides. These ranges are not disjoint, unlike the series-index format.
Readers consider every matching part rather than choosing a unique owner.
ArtifactMetadata.objects records the manifest and every part as { path, file_size };
the artifact's file_size is their checked sum. An empty artifact has an empty
file list and no part files. The manifest is at most 64 MiB.
Region, source SST, and generation must match the catalog entry. The artifact object list must contain exactly the derived manifest and part paths, without duplicates, extra objects, or inconsistent size totals. Part order comes from the manifest.
Part file
┌──────────────────────────────────────────────┐
│ Header 16 bytes │
├──────────────────────────────────────────────┤
│ Independently encoded/compressed chunks │
│ Block 0: time, sequence, operation, fields │
│ Block 1: ... │
├──────────────────────────────────────────────┤ ← payload_end
│ Key area: one exact series key per series, │
│ back to back, not page bounded │
├──────────────────────────────────────────────┤ ← key_end
│ Leaf pages (one is the root at depth 1) │
│ Optional internal root (depth 2) │
├──────────────────────────────────────────────┤
│ Trailer 120 bytes │
└──────────────────────────────────────────────┘
All fixed-width numbers are little-endian, without alignment padding. Byte ranges
are half-open; offsets are absolute from the start of the part. Parse each field
explicitly. Rust struct layout, enum order, and usize are not wire contracts.
Reserved fields must be zero, and decoded structures must have no trailing bytes.
Header
| Field | Bytes | Value |
|---|---|---|
| Magic | 8 | ASCII GTSAMPLE |
| File version | 4 | 1 |
| Reserved | 4 | Zero |
Column payloads and row semantics
Each block contains exactly one series and names it by that series' exact canonical
sparse primary key, which is stored once per series in the part's key area
and referenced by the block descriptor. The key is the SST's own encoding of
(table_id, tsid, label column-id/value pairs sorted by name, non-null only), and
(table_id, tsid) is only its order-preserving prefix. Two distinct series may
share a prefix. Timestamp, sequence, operation type, and each present value field
are independent chunks; their references are in the metadata, not inline headers.
Emit chunks in that order, with value fields following manifest order.
Rows preserve delete markers, duplicate timestamps, and equal-key ties, including conflicting values at the same sequence. The format does not encode a deduplicated or snapshot-filtered view.
Block keys are nondecreasing in byte-lexicographic order, timestamps are
nondecreasing within one exact key, and sequences are nonincreasing within equal
timestamps, including across parts. A change of exact key restarts timestamps and
ordinals, even when the coarse (table_id, tsid) prefix is unchanged. Equal
ordering keys retain input order; operation type is not a sorting key. The key area
follows the same order, one copy per series. Block keys never decrease, so a key that
is read must not sort before the key read before it; a repeated key must continue its
timestamps and ordinals rather than restart them. Because key bytes are read lazily,
the descriptor-only checks use the (table_id, tsid) prefix and the order, disjointness,
and reuse of key references inside one part, and only the keys a request actually loads
are checked for byte order and continuation.
Blocks split on an identity change or before the next row would exceed a packing
limit. There is no hourly split and no padding of short blocks. The default row
limit is 8,192, also the version-1 reader maximum; a writer may choose a smaller
positive limit without changing the version. With F manifest fields and N
rows, the decoded accounting size is:
N * (8 timestamp + 8 sequence + 1 operation + 8 * F values)
+ F * ceil(N / 8) validity bytes
Require this to be at most 262,144 bytes. Count all manifest fields, including omitted/all-null fields, and reserve validity bytes even for all-valid fields. This is a bound on decoded column buffers, not total allocator usage or RSS. The effective row cap is the largest integer satisfying both limits: 8,192 for one field and 31 for 1,024 fields. Encoded buffers and metadata have separate bounds below. Increasing the reader maximum requires an explicit compatibility decision; it is not merely a writer option in this version.
Payload reference
Every stored column and metadata page uses the same 32-byte reference layout as the series format:
| Field | Bytes | Meaning |
|---|---|---|
| Offset | 8 | Absolute location of stored bytes |
| Stored length | 4 | Bytes to fetch |
| Decoded length | 4 | Bytes after decompression, before column/page decoding |
| Checksum | 8 | XXH64 of stored bytes, seed zero |
| Encoding ID | 2 | Sample column or metadata encoding, according to context |
| Compression ID | 2 | Compression of the encoded bytes |
| Reserved | 4 | Zero |
Lengths are nonzero. Check offset + stored_length for u64 overflow before
checking the enclosing area. u32 payload lengths do not limit the part or its
offsets to 4 GiB. Verify the checksum before decompression; require exactly the
declared decoded length, with bounded output even if a codec frame claims more.
Checksums detect accidental corruption, not malicious modification.
Key area
The exact series keys occupy their own region between the column payloads and the
metadata pages: [payload_end, key_end). Keys are written back to back with no
padding and no alignment, in block order, and every block of one series shares one
copy of that series' key. The area therefore holds exactly one key per distinct
series of the part, ordered by exact key. A descriptor's key reference is unique per
series, ordered, and disjoint from every other series' copy, which is why an equal
reference means "the same exact series" without reading the bytes.
A key's length is bounded by the part, not by a page or a chunk. Version 1 puts no
page inside the key area, so one key may be far longer than the 65,536-byte page
limit; the u32 length in a descriptor is validated against key_end, so a
corrupt length can never address bytes outside the part.
Each copy is [key bytes][checksum], where the checksum is the u64 XXH64 of those
key bytes with seed zero, stored little-endian. A reference's length counts the key
bytes only, so the checksum is always the eight bytes after offset + length and is
never part of what the reference names. There is no per-key compression and no other
framing: the key bytes between them are stored exactly as the SST encodes them, and
their own primary-key framing (a fixed reserved prefix plus dense or sparse label
encoding) is what decodes them. The checksum is what protects them. A key previously
sat inside a metadata page, whose stored bytes the page reference already checksummed;
without a per-key checksum, flipping one label byte would keep the framing and the
(table_id, tsid) prefix valid and silently attribute the block's rows to another
series.
Reading stays lazy. A reader reads only the byte ranges of the keys of the blocks a
request selects, at most once per series, and validates a fetched key's checksum, its
framing, and its (table_id, tsid) prefix against the descriptor that referenced it. A
reference is rejected as soon as its descriptor is decoded — before any row is emitted
— when it starts before payload_end, when offset + length + 8 exceeds key_end,
when the length is zero, or when two consecutive descriptors of one part that do not
name one series overlap.
A key reference is an offset into one part's key area, so the references of two parts are unrelated numbers and are never compared. Whether one exact series continues in another part is decided on the exact key bytes: the last key a request loaded from one part is compared with the first it loads from the next, and an equal key must continue timestamps and ordinals, while a greater key starts a new series at ordinal zero.
Because keys are only read for blocks a request selects, this content check covers the keys actually loaded: unvisited ranges are validated structurally, not by content. It is chained, though, so a selection that skips a block, a page, or a whole part cannot hide the two keys around the gap: the check compares each loaded key with the previous loaded key, not with the previous traversed block. Loaded keys must be nondecreasing. Within one exact key the ordinal must strictly increase, and only immediately adjacent blocks of that same key require it to be exactly one greater; a different key starts a new series at ordinal zero, so no ordering between the two series' ordinals is required.
Metadata pages
A leaf describes blocks. An internal page references leaves. The root is either the only leaf (depth 1) or an internal page (depth 2). Other depths are unsupported in version 1; roll parts before a third level would be needed.
As in the series format, the entire page, including its kind and count, is encoded
and independently compressed. The reference in the parent or trailer supplies
its storage location, lengths, checksum, and codecs. There is no separate stored
page header. Decoded pages begin with kind: u32 and entry_count: u64:
Leaf (kind 0): Internal root (kind 1):
kind u32 kind u32
entry_count u64 entry_count u64
block descriptor 0 page reference 0 72 bytes
... ...
block descriptor N-1 page reference N-1 72 bytes
Pages are nonempty and limited to 65,536 decoded bytes, including the 12-byte
prefix. Descriptors never straddle pages, and every descriptor is fixed-width because
the key it names is not stored in it. The writer requires the 12-byte leaf prefix and
one descriptor to fit in a configured page, which for the 1,024-field maximum is
12 + 56 + 44 * 1,027 = 45,256 decoded bytes. The page limit is a bound on
descriptors only: it no longer bounds the length of a series key.
Block descriptor
| Field | Bytes | Meaning |
|---|---|---|
| Table ID | 4 | Coarse prefix of the constant key in this block |
| TSID | 8 | Coarse prefix of the constant key in this block |
| Key offset | 8 | Absolute offset of the block's exact key in the part's key area |
| Key length | 4 | Nonzero; the key's (table_id, tsid) prefix must equal the two fields above |
| Minimum timestamp | 8 | Signed i64, first row's timestamp |
| Maximum timestamp | 8 | Signed i64, last row's timestamp |
| Block ordinal | 8 | Per-identifier ordinal across the artifact |
| Row count | 4 | Nonzero, subject to decoded-block limits |
| Column count | 4 | Number of following column descriptors |
| Column descriptors | 44 each | Defined below |
The fixed prefix is 56 bytes and does not depend on the key: 4 table ID + 8 TSID + 8 key offset + 4 key length + 8 minimum timestamp + 8 maximum timestamp + 8 ordinal + 4 row count + 4 column count. Ordinals start at zero for each exact key and increase
by one across block and part boundaries while that key continues. Blocks sort by exact
key. Adjacent blocks of the same exact key share one key reference and satisfy
previous.max_timestamp <= next.min_timestamp; equality is permitted. Blocks of
different keys have no timestamp relation at all, and a new series' key copy starts at
or after the end of the previous copy.
Each column descriptor is:
| Field | Bytes | Meaning |
|---|---|---|
| Role | 4 | 1 timestamp, 2 sequence, 3 operation type, 4 value field |
| Column ID | 4 | Timestamp/field physical ID; zero for sequence/operation |
| Payload reference | 32 | Independently stored chunk |
| Null count | 4 | Number of null rows |
Role and column ID together identify a chunk. The role namespace avoids reserving
physical field IDs for internal columns. Time, sequence, and operation descriptors
are mandatory, appear first in that order, and have zero null counts. Value
descriptors follow manifest order with no repeated or unknown IDs. A value with
all rows null is omitted and synthesized as null by the reader; present value
chunks have null_count < row_count. No value statistics are stored.
A block with P present fields has C = 3 + P chunks and a descriptor of
56 + 44 * C bytes, whatever its key's length. For one present field that is
56 + 44 * 4 = 232 bytes; at the 1,024-field maximum it is 56 + 44 * 1,027 = 45,244
bytes, and the 12-byte leaf prefix makes that 45,256. Key length is therefore not
part of any page budget: a part whose exact key is far longer than a page produces the
same descriptors, and the earlier base + key_length ceiling — 65,344 bytes with no
value field, 65,300 with one, and 20,288 at the 1,024-field maximum — no longer
exists. All chunks are physically ordered and disjoint in the column area, across
blocks as well as within a block, and the key area follows the column area. There is no
inline block header or block-wide compression.
Page reference and time bounds
Samples extend the series format's reference with two timestamp bounds:
| Field | Bytes | Meaning |
|---|---|---|
| First table ID; first TSID | 4 + 8 | Inclusive first identifier |
| Last table ID; last TSID | 4 + 8 | Inclusive last identifier |
| Minimum timestamp | 8 | Minimum over all descendant blocks |
| Maximum timestamp | 8 | Maximum over all descendant blocks |
| Payload reference | 32 | Location and codecs of the child page |
The reference is 72 bytes. Compare identifiers as integer pairs, not as serialized
little-endian bytes. Time bounds use the manifest unit. The time range is an
envelope over all identifiers in the page; it need not describe the first or last
identifier alone. This can admit extra reads, but cannot exclude matching rows.
Every part/page identifier interval requires first <= last, and all part/page/
block time intervals require min_timestamp <= max_timestamp, even when pruning
would skip their payloads.
Child pages follow block order. Their identifier bounds may touch at a shared series, just like part bounds. Do not require strict disjointness or use an exclusive next-child lower bound. Validate each loaded page's derived identifier and time bounds against its reference. At depth 2 all children must be leaves; reject an internal child. Child byte ranges are ordered, disjoint, within the metadata area, and precede the parent. The root ends immediately before the trailer.
An internal root has 12 + 72 * N decoded bytes, so it holds at most
floor((65536 - 12) / 72) = 910 leaf references. The writer must account for the
next descriptor's leaf placement before writing its block. If the descriptor
would require a 911th leaf, finish this part and start the block in a new part.
Navigation and equal-timestamp example
For a requested qualified identity and optional half-open interval [start, end),
select every part/page whose identifier interval contains the identity and whose
time envelope overlaps the request:
(start is absent || max_timestamp >= start)
&& (end is absent || min_timestamp < end)
For multiple selected identities, intersect the ordered ID set with the inclusive identifier range. A table restriction also intersects this selection. Visit each selected page once per request. Apply the same overlap test to leaf blocks, then filter decoded timestamps exactly. Metadata bounds are conservative candidates. No predecessor-only or unique-child lookup is used: a shared identity or timestamp can require several adjacent children, and even several parts.
Part 0, depth 2:
Root R
├─ Leaf A: IDs [(7,42),(7,42)], times [0,100]
│ block ordinal 0: (7,42), times [0,100]
└─ Leaf B: IDs [(7,42),(7,43)], times [0,200]
block ordinal 1: (7,42), times [100,200]
block ordinal 0: (7,43), times [0,50]
Part 1:
Leaf/root C: IDs [(7,43),(7,43)], times [50,80]
block ordinal 1: (7,43), times [50,80]
For (7,42) and [100,101), visit both A and B, keeping all rows at timestamp
100 from blocks 0 and 1. Do not decode (7,43) payloads. For (7,43) and
[50,51), visit B and part 1's C: both may contain rows at 50. A [51,60) request
for (7,43) can skip B's (7,43) block even though B's time envelope overlaps.
For (7,42) and [90,95), A is still visited although its block starts before
the requested interval. Global time envelopes must not be computed from just
the first and last blocks: B's range is [0,200], not [100,50].
The series-index format has no sample timestamps or time bounds.
Trailer
| Field | Bytes | Meaning |
|---|---|---|
| Root page reference | 72 | Inclusive ID/time bounds and payload reference |
| Index depth | 4 | 1 or 2 |
| File version | 4 | 1 |
| File length | 8 | Includes header and trailer |
payload_end |
8 | End of column area, start of key area |
key_end |
8 | End of key area, start of metadata area |
| Magic | 8 | ASCII GTSAMPLE |
| Trailer checksum | 8 | XXH64 of the preceding 112 trailer bytes (the 104-byte body and the magic), seed zero |
The trailer is 120 bytes. Locate it using the recorded object size without a stat
request. Validate the checksum, magic, version, length, depth, area boundaries,
root bounds against the manifest part descriptor, and root end position before
traversal. Column references must lie in [16, payload_end); key references must lie
in [payload_end, key_end); page references lie in [key_end, file_length - 120).
The header and trailer versions must agree.
Encodings and compression
Sample column encoding IDs below are distinct from series column encoding IDs, even where their numeric values coincide. Metadata and compression have separate namespaces as well. Never reuse an assigned ID with a different meaning.
| Namespace | ID | Meaning |
|---|---|---|
| Column encoding | 1 | Timestamp constant step |
| Column encoding | 2 | Timestamp delta-of-delta |
| Column encoding | 3 | Constant sequence |
| Column encoding | 4 | Sequence delta |
| Column encoding | 5 | Nullable Float64 XOR |
| Column encoding | 6 | Constant operation type |
| Column encoding | 7 | Raw operation types |
| Metadata encoding | 1 | Leaf/internal page layout above |
| Compression | 0 | Raw encoded bytes |
| Compression | 1 | Zstd frame |
Encode first, then compress each chunk/page independently. The default writer tries Zstd level 3 and chooses it only if strictly smaller than the raw encoded bytes. Compression 0 requires equal stored and decoded lengths. Compression level is a writer choice; decoding information is in the frame.
Unsigned varints use base-128/LEB128: seven data bits per byte and a high
continuation bit. Writers emit the shortest representation. Readers reject
truncation, more than ten bytes, a tenth byte greater than 1, and trailing bytes
after the expected values. ZigZag maps signed i64 x to
((x as u64) << 1) ^ ((x >> 63) as u64) and reverses that mapping on decode.
All wrapping arithmetic below is modulo 2^64, not signed overflowing arithmetic.
Timestamp encodings 1 and 2
Encoding 1 contains first: i64 and step: i64, exactly 16 bytes. Use it when
every adjacent difference is the same nonnegative, representable i64. A singleton
uses step zero. Reconstruct by checked addition; reject overflow. Zero step is
valid for repeated timestamps.
Encoding 2 contains the first i64, followed for two or more rows by the first delta as a ZigZag varint, then ZigZag varints of differences between deltas. Compute deltas and delta differences with wrapping arithmetic; reconstruct in the same way. This permits differences across the full signed timestamp range. After either decoding, require nondecreasing signed timestamps and exact agreement of first/last values with the descriptor. Consume exactly the recorded row count. Use encoding 1 when applicable, otherwise encoding 2.
Sequence encodings 3 and 4
Encoding 3 contains one u64, repeated for all rows. Encoding 4 contains the first u64 followed by each wrapping u64 difference from its predecessor, reinterpreted as i64 and written as a ZigZag varint. Decode with wrapping addition. Use 3 if all sequences are equal, otherwise 4. Preserve all u64 bit patterns. Sequence need not be monotonic across different timestamps; equal-timestamp runs retain the input's nonincreasing order.
Nullable Float64 encoding 5
validity_kind <u8> # 0 = all valid, 1 = bitmap
validity <ceil(row_count/8) bytes> # only for kind 1
first_non_null_bits <u64>
xor_deltas <one uvarint per remaining non-null value>
Bitmap bit i % 8 in byte i / 8 is one for a present value; unused high bits of
the final byte are zero. Kind 0 has no bitmap and requires zero null count. Kind
1 requires 0 < null_count < row_count and an exactly matching count of unset
in-range bits. All-null columns have no descriptor or payload, so every stored
Float64 chunk contains at least one non-null value.
Process non-null values in row order. Write the first IEEE-754 bit pattern as
u64, then varints of XOR with the previous non-null bit pattern. Nulls do not
reset that previous value. Decode exactly row_count - null_count values and
scatter according to validity. Preserve all bits, including signed zero, NaN
payloads, infinities, and subnormals. Null and NaN are distinct. The unspecified
value-buffer bits behind nulls are not part of the logical preservation contract.
Operation encodings 6 and 7
Operation values are u8: 0 delete, 1 put, matching Mito's current OpType values.
Unknown operation values fail decoding. Encoding 6 stores one byte repeated for
the whole block; encoding 7 stores exactly one byte per row. Use 6 when all values
are equal, otherwise 7. Neither encoding filters delete rows or discards their
field values.
Limits and evolution
An individual chunk has at most 1 MiB of encoded bytes after decompression and
1 MiB plus 64 KiB of stored bytes. Pages have at most 65,536 decoded and 131,072
stored bytes. Series keys are stored in the key area rather than in a page or a
chunk, so a key's length is bounded by the part and by its u32 reference length,
not by PAGE_LIMIT; a key longer than a page is a supported part of this version. Bound sizes before allocating, fetching, or decompressing. Validate
counts against enclosing bytes before allocating descriptor vectors; row counts
must also satisfy the decoded-block formula. Null output materialization remains
subject to the same block bound even when chunks are omitted.
Unknown file versions, depths, reserved fields, roles, or metadata codecs fail open/read explicitly. Column role must agree with its encoding: time 1/2, sequence 3/4, Float64 5, operation 6/7. An unknown column encoding or compression can be skipped if its chunk is not needed, but attempting to decode it returns an unsupported-codec error. Known encodings assigned to a different role are invalid metadata, even for an unprojected column. Unknown fields or malformed references must not be interpreted as omitted all-null columns.
New column encodings or compression methods receive new IDs. Structural changes, changed field meanings, or incompatible limits require a new version. Deeper metadata trees require an explicit reader-support and rollout decision; an unchanged layout alone does not make them readable by version-1 implementations.
The local cache root contains a layout-version marker. Startup clears the entire series-index cache when this marker is absent or incompatible, then rebuilds artifacts through existing maintenance. Source SST identity remains in metadata.
Local cache rollout
Daily consolidation is the only layout the opt-in columnar samples backend writes, including manual reconciliation. Legacy-format defaults are unchanged. A day may contain several generations with disjoint source sets (at most 64 SSTs per merge), and each generation may contain multiple bounded part files. Source Parquet files are never rewritten.
This branch has no deployments, so the local series-index cache token is pinned to
1: a cache written by any other build is discarded on startup and rebuilt from
the SSTs before regions open, and ordinary SST reads remain available meanwhile.
That is the only cache migration this layout has. An artifact written by an earlier
build of this branch — keys inside the block descriptors, a key area without stored
checksums, a 112-byte trailer without key_end — carries a different
identity_encoding value, exact-sparse-key instead of exact-sparse-key-area, so
it is refused when its manifest is opened. Maintenance treats that refusal as "not
reusable" and rebuilds the artifact, so an old artifact is never marked reusable and
then fails on the first part read. A part reached without that check would also be
rejected: the reader locates a trailer from the recorded object size, reads the last
120 bytes of such a part, and refuses the shifted frame instead of decoding it.
Allow for rebuild time and I/O in either direction; do not reuse daily catalogs with
an older reader outside this startup check. Existing per-SST catalog entries remain
readable by the new reader.
A prepared scan uses a grouped artifact only when all of its sources belong to that scan's assignment or were excluded solely by its time filter. Partial coverage or a stale source set selects the ordinary SST path until reconciliation supplies complete valid coverage.