mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-06 00:09:22 +00:00
Hengfei YangandClaude Opus 4.7
bf3f1fbcd8
## Summary - Reverts the three behavioral changes from #8744 in `src/infra/src/cluster/mod.rs` (the file has since moved from `src/common/infra/cluster/mod.rs`). - Keeps the `heatbeat` → `heartbeat` comment typo fix. ## Why The original `continue` on a missing `NODES_HEALTH_CHECK` entry was correct self-healing behavior. The replacement in #8744 introduced three problems: 1. **Cluster-wide failure from one stale entry.** `check_nodes_status` iterates all online nodes, so `return Err(...)` on a missing uuid aborts the entire loop. Every node not yet checked in that tick is silently skipped. 2. **The proactive init on node add is redundant.** The success path in `check_nodes_status` already runs `w.entry(node.uuid.clone()).or_insert(0)`, so a node with a missing entry was never actually unreachable by the health checker. 3. **There's a race window.** `NODES.write` and `NODES_HEALTH_CHECK.write` in `Event::Put` are two separate lock acquisitions. A concurrent `check_nodes_status` reading `NODES` can observe the node present there but not yet in `NODES_HEALTH_CHECK`. Pre-#8744 that just `continue`d; post-#8744 it aborted the whole tick. PR-Agent's own reviewer flagged (1) and (3); those concerns weren't addressed before merge. ## Test plan - [ ] `cargo check -p infra` passes (confirmed locally) - [ ] Manual / staging: bring up a multi-node cluster, bounce one node, confirm health checks for the remaining nodes continue through the tick where the bounced node becomes briefly inconsistent. - [ ] Confirm the failure-threshold eviction path (`times >= failed_times`) still works for a node that stops responding. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>