mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-12 23:12:28 +00:00
discord9
198d92407c
* fix(operator): bound FlowMirrorTask pending rows to prevent frontend OOM When flownode is slow or unreachable, FlowMirrorTask::detach spawns unbounded tasks that each pin a full insert batch for ~10s (gRPC timeout). With continuous writes, in-flight memory grows unboundedly (throughput x 10s x peers), causing frontend-write OOMKill and a million-per-hour ERROR storm. Add a MAX_MIRROR_PENDING_ROWS cap (1M rows) on Inserter: detach fetch_adds the batch's row count before spawning; if the cap is exceeded the reservation is rolled back, the batch is dropped, a DIST_MIRROR_DROPPED_ROW_COUNT metric is bumped, and a warn is logged instead of spawning. On normal completion the reservation is released in both success and error paths. This keeps core writes alive while degrading flow mirror writes to best-effort, which is the intended trade-off: protect the primary insert path at the cost of possible flow aggregation gaps. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address review - balanced reservation, gauge race, log rate limit P1: Compute pending_rows() from actual per-peer payloads in self.requests instead of the pre-fanout num_rows field. This fixes under-reservation for same-table multi-region batches and over-release in multi-flownode fanout. Each peer task now releases exactly its own share so reservations and releases balance. P2: Replace load+set gauge publication with add/sub symmetry. reserve_mirror_pending_rows does fetch_add + gauge.add, release does fetch_update + gauge.sub, eliminating the stale-value race. P3: Rate-limit the dropped-batch warning to at most once per 10s via MirrorDropLog (atomic compare_exchange), accumulating batch/row counts between reports. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix race in dual-peer test - wait for gauge release before unlocking The dual-peer test previously only waited for the pending budget to return to its start value before releasing MIRROR_METRIC_TEST_LOCK. But release_mirror_pending_rows moves the budget before the gauge, so a worker's gauge.sub could still be in flight when the next test acquires the lock and records its baseline, polluting the result. Now waits for both pending == start AND gauge == gauge_before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(operator): preserve instant writes and limit mirror failure logs Reject saturated requests containing instant-TTL rows with the existing retryable write-admission error before mirror or datanode dispatch. Keep normal source-table inserts independent of best-effort mirror drops. Reuse a smaller process-wide event limiter for both dropped batches and failed peer tasks, reporting at most once per 10 seconds with failure count, peer context, and error chain. Cover actual mirror construction, mixed and instant admission, fast/slow peer shares, failed-task release, and log concurrency. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(operator): reject oversized instant mirror batches permanently Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>