greptimedb/docs/formats/mito-samples.md

Mito columnar samples format

This document specifies the version 1 on-disk layout, encodings, ordering rules, and validation limits for samples artifacts. It shares artifact conventions with Mito columnar series format.

Samples store nullable Float64 fields, timestamps, sequences, and operation types in single-series blocks bounded by rows and decoded bytes.

Artifact manifest

The catalog identifier is columnar-samples. Its file and manifest version is 1, independent of the series format's version and both formats' codec identifiers. Existing parquet and columnar-series identifiers keep their meaning.

One artifact is derived from one immutable source SST. It owns these objects:

<region_id>/samples/<generation>/manifest.json
<region_id>/samples/<generation>/part-0.samples
<region_id>/samples/<generation>/part-1.samples
...

The source file ID and generation use existing FileId serialization. The generation is unique for each build. A manifest contains:

Field	Meaning
format	String columnar-samples
version	Integer 1
region_id	Physical region ID, u64
source_file_id	SST FileId matching SamplesEntry.file_id
generation	FileId matching the catalog artifact generation
timestamp	Object with column_id, name, and unit
fields	Ordered objects with column_id, name, type, and nullable
files	Ordered part descriptors, defined below

The timestamp's ID is u32, its name is the region's time-index name, and its unit is one of second, millisecond, microsecond, or nanosecond. Values are signed i64 counts since the Unix epoch in that unit. There is no dataset-relative origin, unit conversion, or timezone in the file. The timestamp is non-null.

The field list records the build-time region field schema in metadata order. Require 1..=1,024 fields, each with type = "float64" and nullable = true. A non-nullable source Float64 field is represented by this same nullable logical type with no null values. Sparse logical-table fields may be entirely null. Timestamp and field IDs must be distinct. All output names must be unique, including the internal names below. Tags are not sample fields and are not stored here.

The stored logical Arrow schema is __table_id: UInt32, __tsid: UInt64, the timestamp column with its recorded unit, __sequence: UInt64, __op_type: UInt8, then nullable Float64 fields in manifest order. The first five columns are non-null. No __primary_key or tag columns are stored.

Each files entry contains part_id: u32, file_size: u64, num_rows: u64, num_blocks: u64, first, last, min_timestamp: i64, and max_timestamp: i64. first and last are inclusive { table_id: u32, tsid: u64 } bounds. Time bounds are the minimum and maximum over all blocks in the part, not just its first and last rows. Every part is nonempty. Part IDs start at zero, increase by one, and derive part-<part_id>.samples names with decimal IDs and no leading zeros. No object path is stored in a part descriptor.

Parts follow input order. Adjacent identifier ranges satisfy previous.last <= next.first; equality is allowed because a series can continue in another part. The shared boundary series can also have equal timestamps on both sides. These ranges are not disjoint, unlike the series-index format. Readers consider every matching part rather than choosing a unique owner.

ArtifactMetadata.objects records the manifest and every part as { path, file_size }; the artifact's file_size is their checked sum. An empty artifact has an empty file list and no part files. The manifest is at most 64 MiB.

Region, source SST, and generation must match the catalog entry. The artifact object list must contain exactly the derived manifest and part paths, without duplicates, extra objects, or inconsistent size totals. Part order comes from the manifest.

Part file

┌──────────────────────────────────────────────┐
│ Header                              16 bytes │
├──────────────────────────────────────────────┤
│ Independently encoded/compressed chunks      │
│   Block 0: time, sequence, operation, fields  │
│   Block 1: ...                               │
├──────────────────────────────────────────────┤ ← payload_end
│ Leaf pages (one is the root at depth 1)       │
│ Optional internal root (depth 2)              │
├──────────────────────────────────────────────┤
│ Trailer                            112 bytes │
└──────────────────────────────────────────────┘

All fixed-width numbers are little-endian, without alignment padding. Byte ranges are half-open; offsets are absolute from the start of the part. Parse each field explicitly. Rust struct layout, enum order, and usize are not wire contracts. Reserved fields must be zero, and decoded structures must have no trailing bytes.

Header

Field	Bytes	Value
Magic	8	ASCII GTSAMPLE
File version	4	1
Reserved	4	Zero

Column payloads and row semantics

Each block contains exactly one (table_id, tsid). Identity is stored once in its descriptor. Timestamp, sequence, operation type, and each present value field are independent chunks; their references are in the metadata, not inline headers. Emit chunks in that order, with value fields following manifest order.

Rows preserve delete markers, duplicate timestamps, and equal-key ties, including conflicting values at the same sequence. The format does not encode a deduplicated or snapshot-filtered view.

Identifiers are nondecreasing, timestamps are nondecreasing within an identifier, and sequences are nonincreasing within equal timestamps, including across parts. Equal ordering keys retain input order; operation type is not a sorting key.

(table_id, tsid) is the complete identity under the uniqueness assumption. Equal TSIDs in different tables are distinct. Exact sparse primary keys, tags, and collision discriminators are not stored.

Blocks split on an identity change or before the next row would exceed a packing limit. There is no hourly split and no padding of short blocks. The default row limit is 8,192, also the version-1 reader maximum; a writer may choose a smaller positive limit without changing the version. With F manifest fields and N rows, the decoded accounting size is:

N * (8 timestamp + 8 sequence + 1 operation + 8 * F values)
    + F * ceil(N / 8) validity bytes

Require this to be at most 262,144 bytes. Count all manifest fields, including omitted/all-null fields, and reserve validity bytes even for all-valid fields. This is a bound on decoded column buffers, not total allocator usage or RSS. The effective row cap is the largest integer satisfying both limits: 8,192 for one field and 31 for 1,024 fields. Encoded buffers and metadata have separate bounds below. Increasing the reader maximum requires an explicit compatibility decision; it is not merely a writer option in this version.

Payload reference

Every stored column and metadata page uses the same 32-byte reference layout as the series format:

Field	Bytes	Meaning
Offset	8	Absolute location of stored bytes
Stored length	4	Bytes to fetch
Decoded length	4	Bytes after decompression, before column/page decoding
Checksum	8	XXH64 of stored bytes, seed zero
Encoding ID	2	Sample column or metadata encoding, according to context
Compression ID	2	Compression of the encoded bytes
Reserved	4	Zero

Lengths are nonzero. Check offset + stored_length for u64 overflow before checking the enclosing area. u32 payload lengths do not limit the part or its offsets to 4 GiB. Verify the checksum before decompression; require exactly the declared decoded length, with bounded output even if a codec frame claims more. Checksums detect accidental corruption, not malicious modification.

Metadata pages

A leaf describes blocks. An internal page references leaves. The root is either the only leaf (depth 1) or an internal page (depth 2). Other depths are unsupported in version 1; roll parts before a third level would be needed.

As in the series format, the entire page, including its kind and count, is encoded and independently compressed. The reference in the parent or trailer supplies its storage location, lengths, checksum, and codecs. There is no separate stored page header. Decoded pages begin with kind: u32 and entry_count: u64:

Leaf (kind 0):                     Internal root (kind 1):
  kind                 u32          kind                 u32
  entry_count          u64          entry_count          u64
  block descriptor 0                page reference 0     72 bytes
  ...                               ...
  block descriptor N-1              page reference N-1   72 bytes

Pages are nonempty and limited to 65,536 decoded bytes, including the 12-byte prefix. Descriptors never straddle pages. Unlike series label descriptors, sample descriptors have no variable-size strings, and even the widest supported descriptor fits in one page. No oversized-leaf exception is needed.

Block descriptor

Field	Bytes	Meaning
Table ID	4	Constant for this block
TSID	8	Constant for this block
Minimum timestamp	8	Signed i64, first row's timestamp
Maximum timestamp	8	Signed i64, last row's timestamp
Block ordinal	8	Per-identifier ordinal across the artifact
Row count	4	Nonzero, subject to decoded-block limits
Column count	4	Number of following column descriptors
Column descriptors	44 each	Defined below

The fixed prefix is 44 bytes. Ordinals start at zero for each identity and increase by one across block and part boundaries. Blocks sort lexicographically by (table_id, tsid, min_timestamp, ordinal). Adjacent blocks of the same series satisfy previous.max_timestamp <= next.min_timestamp; equality is permitted.

Each column descriptor is:

Field	Bytes	Meaning
Role	4	1 timestamp, 2 sequence, 3 operation type, 4 value field
Column ID	4	Timestamp/field physical ID; zero for sequence/operation
Payload reference	32	Independently stored chunk
Null count	4	Number of null rows

Role and column ID together identify a chunk. The role namespace avoids reserving physical field IDs for internal columns. Time, sequence, and operation descriptors are mandatory, appear first in that order, and have zero null counts. Value descriptors follow manifest order with no repeated or unknown IDs. A value with all rows null is omitted and synthesized as null by the reader; present value chunks have null_count < row_count. No value statistics are stored.

A block with P present fields has C = 3 + P chunks and a descriptor of 44 + 44 * C bytes. This is 220 bytes for one present field and 45,232 bytes for 1,024 present fields. The leaf prefix makes the latter 45,244 bytes. All chunks are physically ordered and disjoint in the column area, across blocks as well as within a block. There is no inline block header or block-wide compression.

Page reference and time bounds

Samples extend the series format's reference with two timestamp bounds:

Field	Bytes	Meaning
First table ID; first TSID	4 + 8	Inclusive first identifier
Last table ID; last TSID	4 + 8	Inclusive last identifier
Minimum timestamp	8	Minimum over all descendant blocks
Maximum timestamp	8	Maximum over all descendant blocks
Payload reference	32	Location and codecs of the child page

The reference is 72 bytes. Compare identifiers as integer pairs, not as serialized little-endian bytes. Time bounds use the manifest unit. The time range is an envelope over all identifiers in the page; it need not describe the first or last identifier alone. This can admit extra reads, but cannot exclude matching rows. Every part/page identifier interval requires first <= last, and all part/page/ block time intervals require min_timestamp <= max_timestamp, even when pruning would skip their payloads.

Child pages follow block order. Their identifier bounds may touch at a shared series, just like part bounds. Do not require strict disjointness or use an exclusive next-child lower bound. Validate each loaded page's derived identifier and time bounds against its reference. At depth 2 all children must be leaves; reject an internal child. Child byte ranges are ordered, disjoint, within the metadata area, and precede the parent. The root ends immediately before the trailer.

An internal root has 12 + 72 * N decoded bytes, so it holds at most floor((65536 - 12) / 72) = 910 leaf references. The writer must account for the next descriptor's leaf placement before writing its block. If the descriptor would require a 911th leaf, finish this part and start the block in a new part.

Navigation and equal-timestamp example

For a requested qualified identity and optional half-open interval [start, end), select every part/page whose identifier interval contains the identity and whose time envelope overlaps the request:

(start is absent || max_timestamp >= start)
    && (end is absent || min_timestamp < end)

For multiple selected identities, intersect the ordered ID set with the inclusive identifier range. A table restriction also intersects this selection. Visit each selected page once per request. Apply the same overlap test to leaf blocks, then filter decoded timestamps exactly. Metadata bounds are conservative candidates. No predecessor-only or unique-child lookup is used: a shared identity or timestamp can require several adjacent children, and even several parts.

Part 0, depth 2:
  Root R
    ├─ Leaf A: IDs [(7,42),(7,42)], times [0,100]
    │    block ordinal 0: (7,42), times [0,100]
    └─ Leaf B: IDs [(7,42),(7,43)], times [0,200]
         block ordinal 1: (7,42), times [100,200]
         block ordinal 0: (7,43), times [0,50]

Part 1:
  Leaf/root C: IDs [(7,43),(7,43)], times [50,80]
    block ordinal 1: (7,43), times [50,80]

For (7,42) and [100,101), visit both A and B, keeping all rows at timestamp 100 from blocks 0 and 1. Do not decode (7,43) payloads. For (7,43) and [50,51), visit B and part 1's C: both may contain rows at 50. A [51,60) request for (7,43) can skip B's (7,43) block even though B's time envelope overlaps. For (7,42) and [90,95), A is still visited although its block starts before the requested interval. Global time envelopes must not be computed from just the first and last blocks: B's range is [0,200], not [100,50].

The series-index format has no sample timestamps or time bounds.

Trailer

Field	Bytes	Meaning
Root page reference	72	Inclusive ID/time bounds and payload reference
Index depth	4	1 or 2
File version	4	1
File length	8	Includes header and trailer
payload_end	8	End of column area, start of metadata area
Magic	8	ASCII GTSAMPLE
Trailer checksum	8	XXH64 of the preceding 104 trailer bytes, seed zero

The trailer is 112 bytes. Locate it using the recorded object size without a stat request. Validate the checksum, magic, version, length, depth, area boundaries, root bounds against the manifest part descriptor, and root end position before traversal. Column references must lie in [16, payload_end); page references lie in [payload_end, file_length - 112). The header and trailer versions must agree.

Encodings and compression

Sample column encoding IDs below are distinct from series column encoding IDs, even where their numeric values coincide. Metadata and compression have separate namespaces as well. Never reuse an assigned ID with a different meaning.

Namespace	ID	Meaning
Column encoding	1	Timestamp constant step
Column encoding	2	Timestamp delta-of-delta
Column encoding	3	Constant sequence
Column encoding	4	Sequence delta
Column encoding	5	Nullable Float64 XOR
Column encoding	6	Constant operation type
Column encoding	7	Raw operation types
Metadata encoding	1	Leaf/internal page layout above
Compression	0	Raw encoded bytes
Compression	1	Zstd frame

Encode first, then compress each chunk/page independently. The default writer tries Zstd level 3 and chooses it only if strictly smaller than the raw encoded bytes. Compression 0 requires equal stored and decoded lengths. Compression level is a writer choice; decoding information is in the frame.

Unsigned varints use base-128/LEB128: seven data bits per byte and a high continuation bit. Writers emit the shortest representation. Readers reject truncation, more than ten bytes, a tenth byte greater than 1, and trailing bytes after the expected values. ZigZag maps signed i64 x to ((x as u64) << 1) ^ ((x >> 63) as u64) and reverses that mapping on decode. All wrapping arithmetic below is modulo 2^64, not signed overflowing arithmetic.

Timestamp encodings 1 and 2

Encoding 1 contains first: i64 and step: i64, exactly 16 bytes. Use it when every adjacent difference is the same nonnegative, representable i64. A singleton uses step zero. Reconstruct by checked addition; reject overflow. Zero step is valid for repeated timestamps.

Encoding 2 contains the first i64, followed for two or more rows by the first delta as a ZigZag varint, then ZigZag varints of differences between deltas. Compute deltas and delta differences with wrapping arithmetic; reconstruct in the same way. This permits differences across the full signed timestamp range. After either decoding, require nondecreasing signed timestamps and exact agreement of first/last values with the descriptor. Consume exactly the recorded row count. Use encoding 1 when applicable, otherwise encoding 2.

Sequence encodings 3 and 4

Encoding 3 contains one u64, repeated for all rows. Encoding 4 contains the first u64 followed by each wrapping u64 difference from its predecessor, reinterpreted as i64 and written as a ZigZag varint. Decode with wrapping addition. Use 3 if all sequences are equal, otherwise 4. Preserve all u64 bit patterns. Sequence need not be monotonic across different timestamps; equal-timestamp runs retain the input's nonincreasing order.

Nullable Float64 encoding 5

validity_kind <u8>                 # 0 = all valid, 1 = bitmap
validity <ceil(row_count/8) bytes>  # only for kind 1
first_non_null_bits <u64>
xor_deltas <one uvarint per remaining non-null value>

Bitmap bit i % 8 in byte i / 8 is one for a present value; unused high bits of the final byte are zero. Kind 0 has no bitmap and requires zero null count. Kind 1 requires 0 < null_count < row_count and an exactly matching count of unset in-range bits. All-null columns have no descriptor or payload, so every stored Float64 chunk contains at least one non-null value.

Process non-null values in row order. Write the first IEEE-754 bit pattern as u64, then varints of XOR with the previous non-null bit pattern. Nulls do not reset that previous value. Decode exactly row_count - null_count values and scatter according to validity. Preserve all bits, including signed zero, NaN payloads, infinities, and subnormals. Null and NaN are distinct. The unspecified value-buffer bits behind nulls are not part of the logical preservation contract.

Operation encodings 6 and 7

Operation values are u8: 0 delete, 1 put, matching Mito's current OpType values. Unknown operation values fail decoding. Encoding 6 stores one byte repeated for the whole block; encoding 7 stores exactly one byte per row. Use 6 when all values are equal, otherwise 7. Neither encoding filters delete rows or discards their field values.

Limits and evolution

An individual chunk has at most 1 MiB of encoded bytes after decompression and 1 MiB plus 64 KiB of stored bytes. Pages have at most 65,536 decoded and 131,072 stored bytes. Bound sizes before allocating, fetching, or decompressing. Validate counts against enclosing bytes before allocating descriptor vectors; row counts must also satisfy the decoded-block formula. Null output materialization remains subject to the same block bound even when chunks are omitted.

Unknown file versions, depths, reserved fields, roles, or metadata codecs fail open/read explicitly. Column role must agree with its encoding: time 1/2, sequence 3/4, Float64 5, operation 6/7. An unknown column encoding or compression can be skipped if its chunk is not needed, but attempting to decode it returns an unsupported-codec error. Known encodings assigned to a different role are invalid metadata, even for an unprojected column. Unknown fields or malformed references must not be interpreted as omitted all-null columns.

New column encodings or compression methods receive new IDs. Structural changes, changed field meanings, or incompatible limits require a new version. Deeper metadata trees require an explicit reader-support and rollout decision; an unchanged layout alone does not make them readable by version-1 implementations.

The local cache root contains a layout-version marker. Startup clears the entire series-index cache when this marker is absent or incompatible, then rebuilds artifacts through existing maintenance. Source SST identity remains in metadata.