Hengfei YangandClaude Fable 5 8ed1f3b799 feat(ingester): write RecordBatch to WAL in Arrow IPC format instead of JSON (#13171) ...

## Summary

Resolves #13172

The ingest write path already converts JSON data to an Arrow RecordBatch
for the memtable. This PR reuses that RecordBatch and serializes it as
an Arrow IPC stream for the WAL entry, instead of serializing the parsed
data back to JSON. WAL replay decodes the RecordBatch directly (the IPC
stream carries the schema), skipping JSON parsing and schema inference
entirely.

Write path: any protocol -> JSON -> format -> RecordBatch -> memtable +
WAL (Arrow IPC).

## Benchmarks

Micro-benchmark, per 3846-record batch (7.3MB pretty-printed JSON, k8s
logs), Apple M-series:

| | JSON (before) | Arrow IPC (after) | change |
|---|---|---|---|
| WAL entry serialize | 5.25 ms | 0.42 ms | 12x faster |
| WAL entry size | 7.35 MB | 2.87 MB | 61% smaller |
| WAL entry decode (replay) | 17.1 ms | 0.46 ms | 37x faster |

End-to-end, 10GB HTTP `_json` ingest (1200 x 8.4MB requests, 8
concurrent, 4 alternating rounds, single node):

- throughput: ~734 MB/s -> ~788 MB/s (**+8-11%**, one round reversed
under background-load noise)
- server CPU for the same work: **-11~14%** (consistent in every round)
- record count verified exact after every run (4,619,046)

End-to-end crash recovery (ingest 1GB / 461k records, `kill -9`,
restart):

- WAL replay: 2.95s -> 0.61s (**~5x faster**), recovered count exact

## Compatibility

- New entries start with a `0xFFFF` marker. The legacy format starts
with the stream-name length (u16), which can never be 0xFFFF (names are
bounded far below 64KB by the HTTP request-head limit), so
`Entry::from_bytes` detects the format per entry.
- **Upgrade:** legacy JSON WAL files replay through the unchanged legacy
path (covered by tests).
- **Downgrade caveat — draining the WAL before rollback is mandatory:**
an old binary cannot decode Arrow-format WAL entries. Entries smaller
than 64KiB fail with `ReadDataError` and are skipped per entry; entries
**larger than 64KiB** fail with `FromUtf8Error` (the 0xFFFF marker is
read as a 65,535-byte stream name containing non-UTF8 IPC bytes), which
the old replay loop does not skip — it aborts `replay_wal_files` and
leaves all remaining WAL files unreplayed. Always shut down gracefully
(drain WAL) before downgrading.
- Rotation checks use the actual serialized WAL bytes (Arrow IPC size),
matching the units of `wal::Writer::size()`; JSON size estimates remain
only for stats accounting.
- `data_size` left as 0 by a caller is normalized from the data at
preprocess time, and memtable emptiness is now determined by the batch
row count (the legacy `into_bytes()` used to backfill `data_size`, and 0
means "skip" downstream).
- `data_size` (original JSON size, used for stats/billing) is now the
size estimated at ingest time rather than re-serialized JSON length; the
two differ only marginally.

## Tests

- Arrow entry roundtrip (fields + batch equality), empty-batch and
unicode-field roundtrips
- Legacy JSON roundtrips still pass, with an added assertion that legacy
bytes are not mis-detected as Arrow
- `wal-reader` debug tool handles both formats

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

2026-07-14 07:56:34 +00:00

..

wal-reader.rs

feat(ingester): write RecordBatch to WAL in Arrow IPC format instead of JSON (#13171)

2026-07-14 07:56:34 +00:00