mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-10-29 15:06:59 +00:00

Files

T

jeremyhi 6f5c064f54 fix(meta): fence datanode processes replaced by a newer process (#9878 )

* fix(meta): fence datanode processes replaced by a newer process

Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.

The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.

Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): keep the leader cache load and pusher updates coherent

A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.

A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.

The epoch check moves into the heartbeat service.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): keep the leader cache coherent with writes while it is disabled

A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(meta): pin the epoch 0 exemption and pusher order, and update docs

The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): register a datanode pusher only after its first request passes

A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.

Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): check a datanode epoch again before replying

A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): handle a datanode's requests one at a time

A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.

The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

2026-10-26 09:02:40 +00:00

6.3 KiB

Raw Blame History

meta-srv — Agent & Contributor Guide

Navigation aid for src/meta-srv. Keep it short and point to code. Paths are relative to the repo root.

Repo-wide rules that apply here: .agents/architecture-invariants.md.

What this crate does

Metasrv is the metadata and coordination service for distributed mode: it persists cluster/table/region metadata, runs leader election, drives the heartbeat loop that tracks cluster topology, and orchestrates distributed procedures (DDL, region migration, repartition). Data models, the KV backend abstraction, election, key encoding, and the DDL manager live in common-meta; meta-srv implements the server, state machine, and control logic on top.

Module map

Module	Path	Purpose
`bootstrap`	`src/meta-srv/src/bootstrap.rs`	Wires up KV backend, election, gRPC services, procedure manager
`metasrv`	`src/meta-srv/src/metasrv.rs`, `src/meta-srv/src/metasrv/builder.rs`	`Metasrv` struct, `MetasrvOptions`, `MetasrvBuilder`, startup
`state`	`src/meta-srv/src/state.rs`	Leader/Follower state machine and transitions
`handler`	`src/meta-srv/src/handler/`	Heartbeat handler chain (`HeartbeatHandler`): leader check, region lease, stats, mailbox
`service`	`src/meta-srv/src/service/`	gRPC services (`heartbeat`, `procedure`, `cluster`, `store`) and HTTP admin
`procedure`	`src/meta-srv/src/procedure/`	Distributed procedures: `region_migration/`, `repartition.rs`, `wal_prune/`
`region`	`src/meta-srv/src/region/`	Region supervisor, lease keeper, failover triggers
`discovery`	`src/meta-srv/src/discovery/`	Node discovery and lease-backed node info
`pubsub`	`src/meta-srv/src/pubsub/`	Heartbeat topic publish/subscribe support
`gc`	`src/meta-srv/src/gc/`	Metadata-driven garbage-collection procedures and scheduling
`selector`	`src/meta-srv/src/selector/`	Region placement strategies (round-robin / load-based / lease-based)
`peer`	`src/meta-srv/src/peer.rs`	Peer allocation through selectors
`cluster`	`src/meta-srv/src/cluster.rs`	`MetaPeerClient`: internal RPC with leader fallback
`cache_invalidator`	`src/meta-srv/src/cache_invalidator.rs`	Pushes cache-invalidation to frontends/datanodes
`key`	`src/meta-srv/src/key/`	Metasrv-side KV key encoding

Core flows

Heartbeat: service/heartbeat.rs receives datanode/frontend heartbeats and runs the handler chain in handler/; responses carry mailbox messages (DDL results, cache invalidation). Node information is refreshed in memory on each heartbeat. Datanode statistics collection and flownode remapping share handler/utils.rs for per-node address persistence. Datanode address updates retain the statistics handler's per-node ordering. Address epochs are confirmed only after a successful save; cache invalidation remains best effort.
Datanode process fence: before the handler chain, the leader's heartbeat session checks a datanode request's non-zero node_epoch against the highest one recorded under __datanode_epoch, read and raised through the leader cache. A lower epoch gets a reply without running the chain; requests that carry an epoch take a per-node lock through the check and the chain; epoch 0 (handshakes, older datanodes) is not checked. A datanode stream registers its pusher on its first request that passes, not on the handshake; pushers carry the epoch, and a lower one cannot replace a higher one.
DDL: service/procedure.rs (ddl) is leader-only and hands off to common-meta's DdlManager, executed via common-procedure.
Region migration: procedure/region_migration/manager.rs plus per-step files; triggered by failover or an explicit command.
Leader election: via common-meta's election; transitions in state.rs.

Public surface

Clients use the meta-client crate (MetaClient). On the server side the entry points are the gRPC services under src/meta-srv/src/service/ and the HTTP admin API under src/meta-srv/src/service/admin/.

When you change X, also touch Y

Heartbeat interval / leases: timing constants live in common-meta (distributed_time_constants); changing them affects lease and failover timing in handler/region_lease_handler.rs and region/supervisor.rs.
Procedures: state must stay persisted in the KV backend and execute() must be idempotent for crash recovery.
Cache invalidation: new metadata types need matching invalidation in cache_invalidator.rs and the heartbeat publish handler.
Affinity: common-meta/affinity.rs defines requirements, matching, batch resolution, and the query allocation-context provider. The selector factory optionally supplies a resolver shared by DDL allocation and the region supervisor; without one, allocation skips source lookup. DDL configurators can install a query provider before procedure loaders are registered. Repartition and failover resolve source requirements before selection. Each selector allows one unconstrained retry on candidate shortage when its context permits it. Failover always permits this retry and continues without affinity when source resolution fails or returns no requirements; DDL/repartition retain their own policy. Candidate discovery errors still propagate. Procedure state is unchanged.
Selector stats fields: keep selector/ implementations in sync with the stats they consume.

Testing

cargo nextest run -p meta-srv
# mock election / KV backend:
cargo nextest run -p meta-srv --features mock

Helpers: src/meta-srv/src/mocks.rs, src/meta-srv/src/test_util.rs, src/meta-srv/src/procedure/test_util.rs.

Gotchas

Most mutating operations are leader-only — guard them (see the leader check in handler/ and service/procedure.rs). Non-leaders return "not leader".
In-memory state is not durable; anything needed after a leader change must be persisted to the KV backend. The leader uses a cached KV backend that resets on leader change.
Region lease renewal timing and the supervisor's check interval must be coordinated, or a region can appear active on two datanodes.

Maintenance contract

Update this file when you add a procedure, change the heartbeat handler chain, change leader-only boundaries, or alter the metasrv ↔ common-meta split.

6.3 KiB Raw Blame History