Files
greptimedb/src
jeremyhi 29d5854e6a test(log-store): drive recovery, read, delete and stall failures through the fault matrix (#9488)
* test(log-store): drive recovery, read, delete and stall failures through the fault matrix

Add the second half of the object store WAL fault matrix: the failures of
the object store access and of the store state that the conditional-create
rows did not drive. Each case asserts the failed waiters and their error,
whether the store is poisoned, the sequence, both indexed gauges at the
failure, the exact movement of every counter and what a read sees after.

Recovery: a listing, a trailer fetch and a footer fetch that fail, the
catalog rejecting an object after every footer was read, and an object
gone by the time it is fetched, below the boundary of the tip (ignored)
and at it (construction fails); each leaves the gauges and the recovery
metrics untouched. Reads: a segment fetch that fails, without a hole or a
read timer sample, and a later read of the region that succeeds.
Collection: a delete that fails and is attempted again at the next
collection, and a stop that waits for a delete in flight, with the catalog
agreeing with the listing in both outcomes. Store state: per-region entry
id exhaustion refusing the append without touching the batch it sealed,
and the durable stop paths with several creates in flight that succeed,
fail transiently or conflict after stop began. Stalled appends released by
admission, by a stop, by a terminal error and by the teardown of the actor,
the first two timed as a pair. The recovery and read timers observed around
parked fetches, twice each, with holds that cannot overlap.

The cases use the existing doubles, extended with a failing listing and a
failing range read at an offset on RecordingIo and parked range reads on
ParkedIo. Case::row is now optional, and the assertion that every row of
the failure matrix has a case is unchanged.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): lose a wide object at its header and footer reads in the fault matrix

The vanished-object case now runs three ways: a short object read whole,
and a wide object deleted before its header and tail window reads or
before its separate footer read, each below and at the boundary of the
tip. The entry id exhaustion case reads the other region of the sealed
batch back, the teardown case asserts the indexed gauges before and after
the actor is gone, and two comments that restated their names are gone.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): prove the stop-path ordering and share the wide-footer fixture in the fault matrix

The case of creates that succeed after stop began keeps object 1 parked
while objects 4 to 2 complete, round-trips a command through the actor and
asserts that no append is acknowledged and nothing is indexed before
object 1 is released. The stop-with-delete case replaces its yields with
the same command barrier, queued behind the stop command. The wide-footer
fixture moves to the tests module and is shared with the recovery test
that fetches a footer longer than the tail window. The gauge helper is
named for its side effect and the vanished-object stage enum for what it
names.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): keep the four-create scenario doc on its fault matrix case

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-12 07:17:44 +00:00
..