mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-15 16:32:34 +00:00
jeremyhi
a08e5bcceb
* fix(log-store): keep the enqueued backlog on permanent object store errors On object storage the WAL's object store sits behind OpenDAL's retry layer, which marks every error it returns as persistent. The enqueued mode classified create failures by that status, so a 403 or a missing bucket was repeated every 100 ms on every create in flight, logged nothing, and made appends hang once the sealed batches were full. Object store errors are now classified by kind: denied access, an invalid configuration (how OpenDAL reports a missing S3 bucket) and a write to a missing location are non-retriable. In the enqueued mode such an error makes the store failing instead of poisoning it: the acknowledged backlog is kept under its sequences and bytes, its lowest pending batch is attempted alone with a delay that doubles from 100 ms to 30 s, appends are rejected at once with a retryable error that carries the last object store error, and the first create that succeeds ends the state. Stop uploads the backlog at once and reports a failure as before. The state is logged on entry, at most once a minute while it lasts and on exit, and exported as the `failing` gauge and the `rejected_appends_total` counter. Conflicts, encoding and catalog errors still poison the store; the durable mode is unchanged. An append held back because the sealed batches are full is now counted and timed by the stall metrics in both modes. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how the enqueued object store WAL handles refused creates Describe the enqueued failure contract as the store now implements it: non-retriable object store errors are told apart by kind because OpenDAL's retry layer under the store marks every error persistent, a failing store keeps its backlog and attempts it with backoff while it rejects appends, and stop uploads the backlog at once. Add the failure matrix row, the failing gauge and the rejected appends counter, the sealed-batch admission limit and the stall metrics that now count it, and the fault matrix case counts. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): time the failing attempts where they start and fix two statements The test of the attempts of a failing store now records when each refused write starts and checks the gaps between them, instead of sampling a counter after a sleep of the test task. The delay comment and the RFC describe 30 seconds as the longest delay between two attempts, not as a bound on when appends resume, and the RFC states that the next append is held once seven sealed batches wait. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): drop the wait_until helper that main now defines Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): name errors that need an operator and trim the RFC contract Rename is_non_retriable to requires_intervention: such a create is still repeated, only slowly, until someone fixes the credentials, configuration or bucket. Name the backoff test after what it asserts, cut the test of the lowest pending batch to two failed rounds since the cadence is covered by the backoff test, and shorten the RFC failure contract to the contract and its rationale, leaving error mappings, request counts and logging cadence to the code comments. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): tie the slow repeat to errors requiring intervention Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com>