feat(log-store): add the object store WAL object format (#9200)

* feat(log-store): add the object store WAL object format

Add the byte format of a single object store WAL object: a header with
the GTWALOBJ magic, format version 1, the object sequence and the writer
instance id; one segment per region ordered by region id with entries
ordered by entry id; a footer that records each segment's region id,
entry id range, entry count, byte range and CRC32; and a fixed trailer
with the GTWALTRL magic, the footer location, the footer CRC32 and the
whole-object CRC32.

The module encodes objects deterministically and decodes the header,
trailer, footer and segments separately, with structural checks on
footer ranges and segment tiling. It has no callers yet; the store that
writes and reads objects follows in later changes.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): pin the object store WAL format with a byte fixture

Add a fixed version 1 object with two regions and five entries as a hex
literal. The test decodes it and checks the exact header, trailer,
footer entries and records, and checks that encoding the same records,
in either input order, reproduces the fixture byte for byte.

Round-trip tests alone pass when a refactor changes field order,
endianness or checksum coverage in both the encoder and the decoder.
The fixture bytes were derived from the documented layout rather than
from the encoder, so such a change now fails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): reject an empty footer when decoding a WAL object

The encoder never writes an object without records, but decode_footer
accepted a footer that declares zero segments, and
verify_segment_ranges accepts an empty footer too. Only the test-only
decode_object rejected it, so a checksum-valid empty object would pass
the header, trailer and footer checks that recovery runs.

Reject a zero entry count in decode_footer and drop the now unreachable
check in decode_object. Add a test that builds a checksum-valid object
with an empty footer and checks that decode_footer and decode_object
reject it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): use pub(crate) for the WAL object format API

Other log-store modules use pub(crate) for items shared across module
boundaries. Switch the format module from pub(super) to pub(crate) to
follow that convention. No behavior change.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

Cargo.lock

@@ -8047,6 +8047,7 @@ dependencies = [
  "common-test-util",
  "common-time",
  "common-wal",
+ "crc32fast",
  "dashmap",
  "delta-encoding",
  "derive_builder 0.20.2",

Cargo.toml

@@ -126,6 +126,7 @@ chrono-tz = { version = "0.10", features = ["case-insensitive"] }
 clap = { version = "4.4", features = ["derive"] }
 config = "0.13.0"
 const_format = "0.2"
+crc32fast = "1"
 criterion = "0.7"
 crossbeam-utils = "0.8"
 dashmap = "6.1"

src/log-store/Cargo.toml

@@ -25,6 +25,7 @@ common-runtime.workspace = true
 common-telemetry.workspace = true
 common-time.workspace = true
 common-wal.workspace = true
+crc32fast.workspace = true
 dashmap.workspace = true
 delta-encoding = "0.4"
 derive_builder.workspace = true

src/log-store/src/error.rs

@@ -308,6 +308,13 @@ pub enum Error {
         #[snafu(implicit)]
         location: Location,
     },
+
+    #[snafu(display("Corrupted WAL object, {}", reason))]
+    CorruptedWalObject {
+        reason: String,
+        #[snafu(implicit)]
+        location: Location,
+    },
 }
 
 pub type Result<T> = std::result::Result<T, Error>;
@@ -353,6 +360,8 @@ impl ErrorExt for Error {
             | WaitDumpIndex { .. }
             | MetaLengthExceededLimit { .. } => StatusCode::Internal,
 
+            CorruptedWalObject { .. } => StatusCode::Unexpected,
+
             // Object store related errors
             CreateWriter { .. } | WriteIndex { .. } | ReadIndex { .. } | Io { .. } => {
                 StatusCode::StorageUnavailable

src/log-store/src/lib.rs

@@ -16,5 +16,6 @@ pub mod error;
 pub mod kafka;
 pub mod metrics;
 pub mod noop;
+pub mod object_store_wal;
 pub mod raft_engine;
 pub mod test_util;

src/log-store/src/object_store_wal.rs

@@ -0,0 +1,35 @@
+// Copyright 2023 Greptime Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Durable primitives of a WAL that stores entries as immutable object store
+//! objects.
+//!
+//! Entries of many regions are batched into a single object, written once and
+//! never mutated afterwards. An object is laid out as
+//!
+//! ```text
+//! header | segment (region 1) | ... | segment (region N) | footer | trailer
+//! ```
+//!
+//! The header carries the magic `GTWALOBJ`, the format version, the object
+//! sequence and the instance that wrote the object. Each segment holds the
+//! entries of exactly one region, ordered by entry id, and segments are ordered
+//! by region id. The footer indexes every segment with its region id, entry id
+//! range, byte range and CRC32. The fixed-size trailer points at the footer and
+//! carries the CRC32 of the footer and of the whole object, so a reader locates
+//! the footer by reading the fixed-length trailer at the end of the object.
+
+// The format has no callers until the store that writes and reads objects lands.
+#[allow(dead_code)]
+mod format;