feat: track per-node scheduling state and fence region placement
Add a cluster-wide, persistent "can this node receive new regions" flag and
make every metasrv placement path respect it. This is the fence half of
`ADMIN drain_node`; the drain procedure, gRPC entry and region evacuation land
separately.
State layout
- New namespace `__node_scheduling/{datanode,flownode}/{node_id}` holding
`{"schedulable": false}`. A missing key means schedulable, so undrain is a
delete and no stale `schedulable: true` records are left behind.
- `NodeSchedulingKey`/`NodeSchedulingValue` mirror the existing KV conventions
(`NodeInfoKey`'s `{role, node_id}` shape, `LeaseValue`'s `*Value` suffix).
The role is part of the prefix, so extending to flownodes needs no key change.
- `NodeSchedulingManager` writes with `put_if_not_exists`/`compare_and_put`, so
concurrent marks change the state exactly once, and reads a single prefix
range per placement decision (DDL/failover only, not per query).
Placement
- `NodeInfo` gains `#[serde(skip)] schedulable: Option<bool>` plus
`is_schedulable()`; `None` (unresolved) counts as schedulable, and the field
never reaches heartbeat bytes, so the wire/in-memory format is unchanged.
- `SchedulingAwarePeerDiscovery` stamps the flag on `active_datanodes`. It is
additive (never hides nodes) because failover still needs the full active set
to compute its exclusion list.
- All three built-in selectors now fetch candidates through
`selector::common::active_schedulable_datanodes`; `RegionSupervisor` adds
unschedulable datanodes to `exclude_peer_ids` for both selector variants.
Enforcing the single fence
- `MetaPeerClient` no longer implements `PeerDiscovery`; its discovery methods
are inherent, and `DatabaseOperator` (frontends only) takes the concrete
client. `SchedulingAwarePeerDiscovery`, built once in `MetasrvBuilder`, is
therefore the only way to obtain a `PeerDiscoveryRef`, so bypassing the fence
is a compile error rather than a silent misplacement.
Compatibility
- New key namespace only; `NodeInfo` serialization is byte-identical, so old
metasrv nodes and persisted heartbeats keep working. The flag itself is stored
in the durable KV, so a follower metasrv sees marks written by the leader.
Verification
- `cargo nextest run -p meta-srv node_scheduling` (13), `selector` (12),
`supervisor` (14), `-p common-meta cluster` (9), `-p meta-client` (36),
`-p standalone` (compiles, no tests).
- `cargo clippy -p meta-srv -p common-meta -p meta-client -p standalone
--all-targets -- -D warnings`, `cargo fmt --all -- --check`.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(config): support overriding region engine options from env
`region_engine` is a `Vec` of externally tagged enums, so plain dotted
environment keys cannot address it: `config` walks the path, replaces the
default sequence with a table and loading fails with `invalid type: map,
expected a sequence`.
Address the entries by tag name instead
(`<PREFIX>__REGION_ENGINE__<TAG>__<FIELD>`) and merge the overrides onto the
default entries in a dedicated source, so the engines and fields that are not
mentioned keep their configured value. An entry the default sequence does not
have is appended; a numeric index, an override of the whole sequence, a field
below a non-table default and two variables that set a path and a field below
it are reported with a dedicated error instead of a serde type error.
A configuration file that configures the sequence as a whole still replaces
the merged value, which the example configs document.
Closes#8620
Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
* docs(config): clarify that the metric engine is always enabled
`region_engine` defaults to [mito, file], but the datanode builds and
registers the metric engine unconditionally. Reword the example configs so
`mito` and `file` read as the default configuration entries and naming
`metric` only supplies its options, then regenerate config.md.
Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
* docs(config): drop the env-var explanation from the config reference
Environment variables are documented in the GreptimeTeam/docs repository, so the example configs and the generated config reference only describe the config file. Keep the note that the metric engine is always registered.
Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
* test(config): cover tagged sequence environment collection
Add unit tests for TaggedSeqEnvSource::collect: merging by tag, appending unknown tags, empty and null defaults, flat key removal, the error paths and the take_entries and is_index_like helpers. The variables are injected through Environment::source, so the tests do not touch the process environment.
Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
---------
Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
Co-authored-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
* fix(flow): retain tracked write completion across transient failures
Use receiver-issued single-write capabilities at the frontend and datanode. Rejoin exact admitted writes across unknown transport outcomes, then close and drain before allowing subsequent rounds. Keep ordinary clients and reads on their existing paths.
Separate schema map equality from encoded row identity so fresh protobuf decodes replay consistently even with NaN values. Cover receiver admission, owner cancellation, all-child drain, and retired capabilities.
Local feature-worktree checkpoint only. Main verified core enterprise check and 484 tests, plus DN enterprise check and all four tracked owner tests. Corrected seven-case timer acceptance remains to be rerun; this checkpoint does not authorize publishing or integration.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): preserve upstream dependencies after rebase
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): fail fast on definitive tracked-write open rejection
A tracked write treats every application-level error from the frontend as
an ambiguous transport failure and retries the same pinned frontend
forever. During a rolling upgrade an older frontend does not recognize the
tracked-write command extension and processes Open as an ordinary empty
SQL query, rejecting it with InvalidSql (InvalidArguments); a non-internal
endpoint rejects the extension outright. Both are definitive rejections
that no same-frontend retry can ever resolve, so the flow stalls.
Classify the Open-phase errors by status code: a definitive rejection
(Unsupported / InvalidArguments / PermissionDenied) fails fast with a
terminal pre-write error, while genuinely ambiguous transport failures
(Unavailable, timeout, connection reset, ...) keep the same-capability
retry. Write/Close rejections are deliberately NOT failed fast: once a
write may have been admitted, a same-code rejection (e.g. WriterLost, or
UnknownCapability after a VIP moved Open->A/Write->B) leaves the outcome
ambiguous, and failing fast would let the caller replay and duplicate
rows.
Add a regression test that a legacy empty-SQL Open rejection fails fast
with a single attempt and surfaces as Error::External carrying a
TerminalError.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): preflight tracked writes without submitting data
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): retain tracked command extensions while decoding
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(operator): route shared requests through NodeManager
Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): preserve retry policy through error hints
Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): restore sparse partition tags when rebuilding writes
Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): stop detached write retries when callers cancel
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(frontend): configure retries across distributed write paths
Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage.
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(pipeline): coalesce concurrent pipeline cache misses
The pipeline cache reads with a plain `moka::sync::Cache::get` and falls
through to a distributed query on a miss, so when the 10s TTL expires every
in-flight write request on a frontend issues its own scan of the single-region
`greptime_private.pipelines` table. Concurrent scans per expiry scale with
write QPS, and every frontend's burst lands on the same datanode. A user
running high-throughput ingestion through a pipeline saw that datanode
overloaded.
Switch to `moka::future::Cache::try_get_with` so concurrent misses on the same
key share one loader. This requires a single-key lookup, so cache entries are
now keyed by the requested schema rather than the schema the pipeline is stored
under; resolving a request to a stored schema stays in the loader, which is the
authoritative path and already handles the empty-schema and multi-schema cases.
A lookup for a schema not yet cached costs one extra read, now protected from
amplification by the coalescing it enables.
`remove_cache` previously only walked the compiled-pipeline cache, so an entry
populated by `get_pipeline_str` alone (the pipeline read API) survived deletion
until it expired. It now walks all three caches.
Also make the TTL configurable as `pipeline.cache_ttl`, default unchanged at
10s. The TTL is what propagates a pipeline change to other frontends, so
raising it trades staleness for fewer reads.
Refs #9021
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(pipeline): restore cross-schema semantics broken by the new cache key
Keying cache entries by the requested schema dropped two behaviours that the
previous stored-schema key provided for free.
Creating a new version only wrote the creating request's schema, so another
schema on the same frontend kept serving its cached `latest` — an older
version — until the entry expired. Since the whole point of making the TTL
configurable is to let operators raise it, that window is not bounded by
anything useful. Creation now invalidates every schema's `latest` alias for
that name before priming the cache, leaving the version-pinned keys alone.
The failover cache lost its reach across schemas the same way: a global
pipeline (stored under the empty schema) loaded by schema A was cached under
`A`, so schema B using it for the first time while the pipeline table was down
missed and failed ingestion. The failover cache has no loader and so is not
subject to the single-key model of `try_get_with`; it keeps the stored-schema
key and the empty-schema-first resolution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(pipeline): drop cache priming on create and fold the sweep helpers
Priming the cache on create saved one read on a low-frequency operation and
cost a concept: entries were written under the creating request's schema while
`PipelineContent.schema` said empty, so the two schemas in play disagreed.
Invalidating the `latest` aliases is required regardless — that is what makes
a new version visible to other schemas — so dropping the priming loses only
the saved read, which coalescing now protects anyway. `insert_and_compile` no
longer needs the caller's schema.
`remove_cache` and the create-time invalidation collapse into one
`invalidate(name, version)`; `None` sweeps only the `latest` aliases, which is
exactly what creation wants. That leaves `invalidate_by_suffixes` and
`cache_keys` with a single caller each, so both are inlined.
Drop the `PipelineOptions` humantime test: `load_config_test` loads both
example TOMLs, which now carry `cache_ttl = "10s"`, and would fail the same
way if the serde attribute were lost. The `toml` dev-dependency goes with it.
The two invalidation tests are now checked to be orthogonal: removing the
version suffix fails only the delete test, and sweeping just the compiled
cache fails both.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(pipeline): keep failover populated across a create
The `latest` sweep on create clears the failover cache along with the loaded
ones, and after dropping the priming there was nothing writing it back. An
outage between the create and the first read-back left neither `latest` nor the
explicit version with anything to fall back on, failing ingestion — worse than
before, since the previous version's failover entry was swept too.
Creation now goes through `PipelineCache::on_pipeline_created`, which pairs the
sweep with a failover write of the new empty-schema definition. The two must
happen together, so they live behind one method rather than at the call site.
Also commit the Cargo.lock entry for the dropped `toml` dev-dependency, and
trim the comments added over the last few commits down to what the code does
not already say.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: report region read load in heartbeat
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: expose region query stats in information schema
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update sqlness result
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record region query stats on stream drop
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: keep region query cpu stats in nanoseconds
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: global switch for creating table automatically
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: make auto_create_table as comment by default
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: respect gloabl switch for metric engine
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: support reporting env vars in heartbeat messages to metasrv
Add `heartbeat_env_vars` config option for datanode and frontend. When
configured, the specified environment variable values are read at startup
and sent to metasrv in every heartbeat via the `extensions` map. Metasrv
extracts and stores them in `NodeInfo` for use in routing decisions
(e.g. AZ-aware region placement).
- Add `EnvVars` helper in `common/meta/src/datanode.rs` following the
existing `GcStat` extension pattern with `into_extensions`/`from_extensions`
- Add `env_vars: HashMap<String, String>` field to `NodeInfo` in
`common/meta/src/cluster.rs` with `#[serde(default)]` for backward compat
- Add `heartbeat_env_vars: Vec<String>` config field to `DatanodeOptions`,
`FrontendOptions`, and `StandaloneOptions`
- Inject env vars into heartbeat `extensions` in both datanode and frontend
heartbeat tasks (`datanode/src/heartbeat.rs`, `frontend/src/heartbeat.rs`)
- Extract env vars from `req.extensions` in all three metasrv
`CollectXxxClusterInfoHandler`s
- Update `NodeInfo` construction sites in `meta-client`,
`discovery/lease.rs`, and `standalone/information_extension.rs`
- Update expected TOML output in `tests-integration/tests/http.rs`
- Add unit tests for `EnvVars` round-trip and `NodeInfo` backward compat
Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: address heartbeat env review feedback
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: log error on deserialization failure
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: send heartbeat env vars once
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: resend heartbeat env vars after reconnect
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* revert: keep env vars in every heartbeat
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add repartition procedure factory support to DdlManager
- Introduce RepartitionProcedureFactory trait for creating and registering
repartition procedures
- Implement DefaultRepartitionProcedureFactory for metasrv with full support
- Implement StandaloneRepartitionProcedureFactory for standalone (unsupported)
- Add procedure loader registration for RepartitionProcedure and
RepartitionGroupProcedure
- Add helper methods to TableMetadataAllocator for allocator access
- Add error types for repartition procedure operations
- Update DdlManager to accept and use RepartitionProcedureFactoryRef
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: integrate repartition procedure into DdlManager
- Add submit_repartition_task() to handle repartition from alter table
- Route Repartition operations in submit_alter_table_task() to repartition factory
- Refactor: rename submit_procedure() to execute_procedure_and_wait()
- Make all DDL operations wait for completion by default
- Add submit_procedure() for fire-and-forget submissions
- Add CreateRepartitionProcedure error type
- Add placeholder Repartition handling in grpc-expr (unsupported)
- Update greptime-proto dependency
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: implement ALTER TABLE REPARTITION procedure submission
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(repartition): handle central region in apply staging manifest
- Introduce ApplyStagingManifestInstructions struct to organize instructions
- Add special handling for central region when applying staging manifests
- Transition state from UpdateMetadata to RepartitionEnd after applying staging manifests
- Remove next_state() method in RepartitionStart and inline state transitions
- Improve logging and expression serialization in DDL statement executor
- Move repartition tests from standalone to distributed test suite
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: apply suggestions from CR
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update proto
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>