Files
greptimedb/Cargo.toml
T
jeremyhi c6962802a9 feat(meta-srv): bind candidate region leases to the migration's registration (#10440)
* feat(meta-srv): bind candidate region leases to the migration's registration

A region migration now gives its candidate region registration a random non-zero
candidate lease id. Leases granted because of that registration carry it, but only
while the table route exists, and the upgrade instruction carries the same id. The
datanode passes the id of each lease to the engine, and Mito records the lease of
the latest heartbeat per region outside the region instance, so it survives a
reopen during catchup.

The migration also drops the candidate registration before it restores the route
on rollback, and before the procedure leaves the migration tracker when it ends
without a rollback.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): record a lease only while the region stays open

Record a granted lease under the same lock that removes it after the region closes, so a lease recorded during a close does not outlive the region. The metric engine records the data region's lease when the metadata region ignores it. Remove the roles-only extraction of operating regions, and cover a present candidate lease id in the upgrade JSON.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* build: point greptime-proto at the candidate lease id change

Use the greptime-proto commit that adds GrantedRegion.candidate_lease_id (greptime-proto#357), which also brings in the RawSeriesSource Substrait payload message. Rely on the declaration order of RegionMigrationProcedure to drop the candidate registration before the tracker entry, and document it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): drop lease records when workers stop

Remove the lease records of the regions a stopping worker closes. Check under the removal lock deterministically in tests, cover a lease that arrives after a region is removed, assert the candidate lease id on the migration's real registration and its retry, and match ids per region in a batched upgrade.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): drop a redundant helper comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(store-api): pin the candidate lease id wire bytes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: document candidate registration and lease record lifecycles

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep candidate lease ids through customized lease renewers

The heartbeat handler now sets the candidate lease id of each lease a
customized renewer returns from the renewed region with the same id.

The metric engine records a lease for the data region only when the
metadata region ignored it, since the data region records its own
leases. The countdown reset command carries a named struct instead of
a tuple.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(metric-engine): cover a lease the data region ignores

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: tighten lease id lookups and lease test assertions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: align lease naming and share the txn-observing kv backend

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: tidy lease naming, docs and shared test helpers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop field comments that restate their names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop method comments that restate their names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: say which heartbeat a lease's send time refers to

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop the reset deadline field comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: state when an operating region's role and id reach a lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: pin greptime-proto to the merged candidate lease id change

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: add the candidate lease id argument to the direct demotion test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: order new keeper helpers after public items and test equal send times

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-16 18:37:17 +00:00

427 lines
16 KiB
TOML

[workspace]
members = [
"src/api",
"src/auth",
"src/cache",
"src/catalog",
"src/cli",
"src/client",
"src/cmd",
"src/common/base",
"src/common/batcher",
"src/common/cache",
"src/common/catalog",
"src/common/config",
"src/common/datasource",
"src/common/decimal",
"src/common/error",
"src/common/event-recorder",
"src/common/frontend",
"src/common/function",
"src/common/greptimedb-telemetry",
"src/common/grpc",
"src/common/grpc-expr",
"src/common/macro",
"src/common/mem-prof",
"src/common/memory-manager",
"src/common/meta",
"src/common/options",
"src/common/plugins",
"src/common/pprof",
"src/common/procedure",
"src/common/procedure-test",
"src/common/query",
"src/common/recordbatch",
"src/common/runtime",
"src/common/session",
"src/common/sql",
"src/common/stat",
"src/common/substrait",
"src/common/telemetry",
"src/common/test-util",
"src/common/time",
"src/common/version",
"src/common/wal",
"src/common/workload",
"src/datanode",
"src/datatypes",
"src/file-engine",
"src/flow",
"src/frontend",
"src/index",
"src/log-query",
"src/log-store",
"src/meta-client",
"src/meta-srv",
"src/metric-engine",
"src/mito-codec",
"src/mito2",
"src/object-store",
"src/operator",
"src/partition",
"src/pipeline",
"src/plugins",
"src/promql",
"src/puffin",
"src/query",
"src/relational-engine",
"src/standalone",
"src/servers",
"src/session",
"src/sql",
"src/store-api",
"src/table",
"tests-fuzz",
"tests-integration",
"tests/runner",
]
resolver = "2"
[workspace.package]
version = "1.4.0-alpha.0"
edition = "2024"
license = "Apache-2.0"
[workspace.lints]
clippy.print_stdout = "warn"
clippy.print_stderr = "warn"
clippy.dbg_macro = "warn"
clippy.implicit_clone = "warn"
clippy.result_large_err = "allow"
clippy.large_enum_variant = "allow"
clippy.doc_overindented_list_items = "allow"
clippy.uninlined_format_args = "allow"
rust.unknown_lints = "deny"
rust.unexpected_cfgs = { level = "warn", check-cfg = ['cfg(tokio_unstable)'] }
[workspace.dependencies]
# DO_NOT_REMOVE_THIS: BEGIN_OF_EXTERNAL_DEPENDENCIES
# We turn off default-features for some dependencies here so the workspaces which inherit them can
# selectively turn them on if needed, since we can override default-features = true (from false)
# for the inherited dependency but cannot do the reverse (override from true to false).
#
# See for more detaiils: https://github.com/rust-lang/cargo/issues/11329
ahash = { version = "0.8", features = ["compile-time-rng"] }
aquamarine = "0.6"
arrow = { version = "=59.2.0", features = ["prettyprint"] }
arrow-array = { version = "=59.2.0", default-features = false, features = ["chrono-tz"] }
arrow-buffer = "=59.2.0"
arrow-flight = "=59.2.0"
arrow-ipc = { version = "=59.2.0", default-features = false, features = ["lz4", "zstd"] }
arrow-schema = { version = "=59.2.0", features = ["serde"] }
arrow-select = "=59.2.0"
async-stream = "0.3"
async-trait = "0.1"
# Remember to update axum-extra, axum-macros when updating axum
axum = "0.8"
axum-extra = "0.10"
axum-macros = "0.5"
backon = "1"
base64 = "0.22"
bigdecimal = "0.4.2"
bitflags = "2.4.1"
bytemuck = "1.12"
bytes = { version = "1.11", features = ["serde"] }
cap-std = "4.0"
chrono = { version = "0.4", features = ["serde"] }
chrono-tz = { version = "0.10", features = ["case-insensitive"] }
clap = { version = "4.4", features = ["derive"] }
config = "0.13.0"
const_format = "0.2"
crc32fast = "1"
criterion = "0.7"
crossbeam-utils = "0.8"
dashmap = "6.1"
datafusion = "=55.1.0"
datafusion-common = "=55.1.0"
datafusion-datasource = "=55.1.0"
datafusion-execution = "=55.1.0"
datafusion-expr = "=55.1.0"
datafusion-expr-common = "=55.1.0"
datafusion-functions = "=55.1.0"
datafusion-functions-aggregate-common = "=55.1.0"
datafusion-functions-window-common = "=55.1.0"
datafusion-optimizer = "=55.1.0"
datafusion-orc = "0.10.0"
datafusion-pg-catalog = "0.19.0"
datafusion-physical-expr = "=55.1.0"
datafusion-physical-plan = "=55.1.0"
datafusion-proto = "=55.1.0"
datafusion-sql = "=55.1.0"
datafusion-substrait = "=55.1.0"
datafusion_object_store = { package = "object_store", version = "0.13.2" }
deadpool = "0.12"
deadpool-postgres = "0.14"
derive_builder = "0.20"
derive_more = { version = "2.1", features = ["full"] }
dotenv = "0.15"
either = "1.15"
etcd-client = { version = "0.17", features = [
"tls",
"tls-roots",
] }
fs2 = "0.4"
fst = "0.4.7"
futures = "0.3"
futures-util = "0.3"
greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "c46f7ee9c82ff90f5762ae192fcf8bb2b0d6046f" }
hdfs-native = "0.14.6"
hex = "0.4"
hostname = "0.4.0"
http = "1"
humantime = "2.1"
humantime-serde = "1.1"
hyper = "1.1"
hyper-util = "0.1"
icu_properties = "2.0.1"
indicatif = "0.17"
intern-arc = "0.6.2"
itertools = "0.14"
jsonb = { git = "https://github.com/GreptimeTeam/jsonb", rev = "5a6472858d860ac034075c17253f41617bdfa956", default-features = false, features = [
"databend",
] }
lazy_static = "1.4"
local-ip-address = "0.6"
loki-proto = { git = "https://github.com/GreptimeTeam/loki-proto.git", rev = "f69c8924c4babe516373e26a4118be82d976629c" }
meter-core = { git = "https://github.com/GreptimeTeam/greptime-meter.git", rev = "f743c3a5fe8c1f57363eac5866c356b7d91f9b39" }
mockall = "0.13"
moka = "0.12"
nalgebra = "0.33"
nix = { version = "0.30.1", default-features = false, features = ["event", "fs", "process"] }
notify = "8.0"
num_cpus = "1.16"
# Keep object_store 0.13 compatibility while using OpenDAL 0.59.
# TODO(#9240): Switch back to the official bridge after the DataFusion/Arrow/Parquet
# upgrade aligns object_store versions. Keep the bridge and OpenDAL releases compatible.
# https://github.com/GreptimeTeam/greptimedb/issues/9240
object_store_opendal = { git = "https://github.com/GreptimeTeam/opendal.git", rev = "c03e56802b47de3ccfc8fe028e59a459b278a0d2" }
once_cell = "1.18"
opentelemetry-proto = { version = "0.32", features = [
"gen-tonic",
"metrics",
"trace",
"with-serde",
"logs",
] }
ordered-float = { version = "4.3", features = ["serde"] }
otel-arrow-rust = { git = "https://github.com/GreptimeTeam/otel-arrow", rev = "5da284414e9b14f678344b51e5292229e4b5f8d2", features = [
"server",
] }
parking_lot = "0.12"
parquet = { version = "=59.2.0", default-features = false, features = ["arrow", "async", "object_store"] }
parquet-variant = "=59.2.0"
parquet-variant-compute = "=59.2.0"
parquet-variant-json = "=59.2.0"
paste = "1.0"
percent-encoding = "2.3"
pin-project = "1.0"
pretty_assertions = "1.4.0"
prometheus = { version = "0.14", features = ["process"] }
promql-parser = { version = "0.10.0", features = ["ser"] }
prost = { version = "0.14", features = ["no-recursion-limit"] }
prost-types = "0.14"
protoc-bin-vendored = "3.2"
protoc-rust = "2.28"
raft-engine = { version = "0.4.1", default-features = false }
rand = "0.9"
ratelimit = "0.10"
regex = "1.12"
regex-automata = "0.4"
reqwest = { version = "0.13", default-features = false, features = [
"form",
"json",
"query",
"rustls",
"stream",
"multipart",
] }
rskafka = { git = "https://github.com/influxdata/rskafka.git", rev = "80f42eb15eb9419579a376bdf4411893a7da2728", features = [
"transport-tls",
] }
rstest = "0.25"
rstest_reuse = "0.7"
rust_decimal = "1.33"
rustls = { version = "0.23.46", default-features = false }
ryu = "1.0"
sea-query = "0.32"
serde = { version = "1.0", features = ["derive"] }
serde_json = { version = "1.0", features = ["float_roundtrip"] }
serde_with = "3"
serde_yaml_ng = "0.10"
sha2 = "0.10"
simd-json = "0.19"
similar-asserts = "1.6.0"
smallvec = { version = "1", features = ["serde"] }
snafu = "0.8"
sqlparser = { version = "0.62.0", default-features = false, features = ["std", "visitor", "serde"] }
sqlx = { version = "0.9", default-features = false, features = [
"any",
"macros",
"json",
"runtime-tokio",
"tls-rustls-aws-lc-rs",
"rust_decimal",
] }
stringprep = "0.1"
strum = { version = "0.27", features = ["derive"] }
sysinfo = "0.33"
temp-env = "0.3"
tempfile = "3"
tokio = { version = "1.47", features = ["full"] }
tokio-postgres = "0.7.18"
tokio-rustls = { version = "0.26.2", default-features = false }
tokio-stream = "0.1"
tokio-util = { version = "0.7", features = ["io-util", "compat"] }
toml = "0.8.8"
url = "2.3"
# Pin below 0.14.5 to avoid the `connection_timeout_future` "async fn resumed after
# completion" panic (greptimedb#8265 / grpc-rust#2522). Remove once tonic ships a fix.
tonic = { version = "0.14, <0.14.5", features = ["tls-aws-lc", "gzip", "zstd"] }
tonic-prost = "0.14"
tower = "0.5"
tracing = "0.1"
tracing-appender = "0.2"
tracing-opentelemetry = "0.33.0"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json", "fmt"] }
typetag = "0.2"
uddsketch-rs = "0.1.0"
uuid = { version = "1.17", features = ["serde", "v4", "v7", "fast-rng"] }
vrl = "0.33"
vu128 = "1.1"
winner_tree = "0.1.1"
zstd = "0.13"
# DO_NOT_REMOVE_THIS: END_OF_EXTERNAL_DEPENDENCIES
## workspaces members
api = { path = "src/api" }
auth = { path = "src/auth" }
cache = { path = "src/cache" }
catalog = { path = "src/catalog" }
catio = { git = "https://github.com/GreptimeTeam/catio.git", rev = "c75be64e7a58f620f6340c24b5b8d76f131af6aa" }
cli = { path = "src/cli" }
client = { path = "src/client" }
cmd = { path = "src/cmd", default-features = false }
common-base = { path = "src/common/base" }
common-batcher = { path = "src/common/batcher" }
common-cache = { path = "src/common/cache" }
common-catalog = { path = "src/common/catalog" }
common-config = { path = "src/common/config" }
common-datasource = { path = "src/common/datasource" }
common-decimal = { path = "src/common/decimal" }
common-error = { path = "src/common/error" }
common-event-recorder = { path = "src/common/event-recorder" }
common-frontend = { path = "src/common/frontend" }
common-function = { path = "src/common/function" }
common-greptimedb-telemetry = { path = "src/common/greptimedb-telemetry" }
common-grpc = { path = "src/common/grpc" }
common-grpc-expr = { path = "src/common/grpc-expr" }
common-macro = { path = "src/common/macro" }
common-mem-prof = { path = "src/common/mem-prof" }
common-memory-manager = { path = "src/common/memory-manager" }
common-meta = { path = "src/common/meta" }
common-options = { path = "src/common/options" }
common-plugins = { path = "src/common/plugins" }
common-pprof = { path = "src/common/pprof" }
common-procedure = { path = "src/common/procedure" }
common-procedure-test = { path = "src/common/procedure-test" }
common-query = { path = "src/common/query" }
common-recordbatch = { path = "src/common/recordbatch" }
common-runtime = { path = "src/common/runtime" }
common-session = { path = "src/common/session" }
common-sql = { path = "src/common/sql" }
common-stat = { path = "src/common/stat" }
common-telemetry = { path = "src/common/telemetry" }
common-test-util = { path = "src/common/test-util" }
common-time = { path = "src/common/time" }
common-version = { path = "src/common/version" }
common-wal = { path = "src/common/wal" }
common-workload = { path = "src/common/workload" }
datanode = { path = "src/datanode" }
datatypes = { path = "src/datatypes" }
file-engine = { path = "src/file-engine" }
flow = { path = "src/flow" }
frontend = { path = "src/frontend", default-features = false }
index = { path = "src/index" }
log-query = { path = "src/log-query" }
log-store = { path = "src/log-store" }
meta-client = { path = "src/meta-client" }
meta-srv = { path = "src/meta-srv" }
metric-engine = { path = "src/metric-engine" }
mito-codec = { path = "src/mito-codec" }
mito2 = { path = "src/mito2" }
object-store = { path = "src/object-store" }
operator = { path = "src/operator" }
partition = { path = "src/partition" }
pipeline = { path = "src/pipeline" }
plugins = { path = "src/plugins" }
promql = { path = "src/promql" }
puffin = { path = "src/puffin" }
query = { path = "src/query" }
relational-engine = { path = "src/relational-engine" }
servers = { path = "src/servers" }
session = { path = "src/session" }
sql = { path = "src/sql" }
standalone = { path = "src/standalone" }
store-api = { path = "src/store-api" }
substrait = { path = "src/common/substrait" }
table = { path = "src/table" }
[workspace.dependencies.meter-macros]
git = "https://github.com/GreptimeTeam/greptime-meter.git"
rev = "f743c3a5fe8c1f57363eac5866c356b7d91f9b39"
[patch.crates-io]
substrait = { git = "https://github.com/GreptimeTeam/substrait-rs.git", rev = "91ec978b0649417ad3da8390e7a515baec723b1b" }
datafusion = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-datasource = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-execution = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-functions = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-functions-aggregate-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-functions-window-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-optimizer = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-physical-expr = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-physical-expr-common = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-physical-plan = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-proto = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-sql = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-substrait = { git = "https://github.com/GreptimeTeam/datafusion.git", rev = "638a77fb504e86f85ef6053c552bbee1ed814fb9" }
datafusion-pg-catalog = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "3ae0a6cb30b1c5131687f4637247ff4f84ac11e4" }
arrow-pg = { git = "https://github.com/datafusion-contrib/datafusion-postgres.git", rev = "3ae0a6cb30b1c5131687f4637247ff4f84ac11e4" }
sqlparser = { git = "https://github.com/GreptimeTeam/sqlparser-rs.git", rev = "9e9019bb1c7040ed956f654e39378dd42ab17884" }
# pprof 0.15.0 initializes framehop inside SIGPROF; pin the unreleased fix
# that initializes it before sampling. Remove once tikv/pprof-rs#282 is released.
pprof = { git = "https://github.com/tikv/pprof-rs", rev = "01cff82dbe6fe110a707bf2b38d8ebb1d14a18f8" }
# Temporary: use the GreptimeTeam fork of tikv-jemalloc-sys embedding
# jemalloc 5.3.1 + backport of 54f22c83 ("Initialize TSD tcache before
# enabling it"), fixing the TSD/tcache enable-before-init reentrancy
# window. Remove once tikv/jemallocator ships a jemalloc snapshot that
# includes the fix. See GreptimeTeam/jemalloc#1.
tikv-jemalloc-sys = { git = "https://github.com/GreptimeTeam/jemallocator", rev = "e254a7eaf13a9ede04953ddefd173b5303018530" }
[profile.release]
debug = 1
[profile.nightly]
inherits = "release"
strip = "debuginfo"
lto = "thin"
debug = false
incremental = false
[profile.ci]
inherits = "dev"
strip = true
[profile.dev.package.sqlness-runner]
debug = false
strip = true
[profile.dev.package.tests-fuzz]
debug = false
strip = true