* fix(mito2): cancel flushes waiting for the WAL when the engine stops
A flush waits for the WAL to be durable through the entry id it records
before it writes its manifest edit. With the object store WAL in the
`enqueued` mode that wait lasts until an upload succeeds, while the log
store is stopped only after the engines. A graceful shutdown during an
outage of the WAL's object store therefore never finished: closing a
region does not cancel its running flush, stopping the worker group
waits for running flushes and runs the queued ones, and a region holding
writes that skipped the WAL is flushed on close.
The worker group now owns a cancellation token, raised before its
schedulers stop and kept raised. `RegionEngine::begin_stop`, which the
region server calls before it closes regions, raises it earlier, so that
a close-time flush cannot wait either. A flush polls its durability
barrier first and gives up at the barrier once the token is raised: it
writes no manifest edit, deletes its SSTs and fails as cancelled. A
flush whose entries are already durable commits as before. The region
server reports a failed close, which loses the region's writes that
skipped the WAL, in its stop result.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): wait for the outage on the store under test
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): give a healthy WAL a grace period before a stopping engine cancels flushes
A flush of a stopping engine no longer gives up its wait for the WAL as
soon as the stop begins. It gives up at once while the log store reports
that it fails to make entries durable, through the new
`LogStore::is_failing`, and otherwise 5 s after the engine began to stop,
so a healthy object store WAL uploads its last entries first and a
close-time flush keeps the writes that skipped the WAL.
The object store WAL reports that it fails while the last create of its
backlog failed, which covers a refused and an unreachable object store,
and once it is poisoned or stopped.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): drain the WAL for a stopping flush and cancel its failed waits
A flush of a stopping engine now asks the log store to drain before it
waits out the grace period, through a new `LogStore::begin_drain` that
does nothing by default. The object store WAL then seals its open batch
at once and, until it stops, as soon as a durability wait needs a region
it holds, so a healthy store makes the entries durable after about one
upload whatever its `flush_interval`, which has no upper bound.
A barrier wait that fails while the engine stops, for instance on a
poisoned or stopped log store, now fails the flush as cancelled like a
wait that gives up.
The healthy-store test no longer depends on the seal timer: its store
never seals on its own, and the close must finish well within the grace
period.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): drain the WAL when the engine begins to stop
The drain started only when a stopping flush reached its durability
barrier, so a close behind slow flushes that reached it after the grace
period had no time left to upload its entries. The worker group now
asks the log store to drain as soon as it begins to stop, and the
object store WAL seals its open batch then and every later admission
right away, so its entries become durable while the flushes still write
their SSTs. `LogStore::begin_drain` becomes synchronous for that.
The healthy-store test now writes after the stop began and closes the
region only after the grace period.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): refuse appends once the object store WAL drains
A WAL write admitted late in a stop could still be uploading when the
close-time flush of its region reached an expired grace period, so the
writes that skipped the WAL in that region were lost although the store
was healthy. A draining object store WAL now refuses every later append,
also one held back, with the stopped error, so the entries a stopping
engine waits for are fixed when the stop begins and become durable while
the regions still close; writes that skip the WAL are still taken.
Tests cover a close-time flush whose wait completes within the grace
period once a parked create is released, a close that reaches its wait
after the grace period, and a drain refusing a held-back append.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(datanode): word a failed region close neutrally in the shutdown log
A close can fail for reasons other than its flush, such as a stopped
worker, and for regions without writes that skipped the WAL, so the
error line no longer states that such writes were lost: it says they
may be.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(datanode): stop the object store WAL before the region server
The datanode stopped the object store WAL only after the region server,
whose stop waits for flushes that wait for the WAL to become durable, so a
shutdown never finished while the WAL's uploads kept failing.
The WAL now stops first: its stop uploads the backlog or drops it and
answers every wait for its entries, so a running, queued or close-time
flush either commits or fails with the WAL-stopped error without writing
a manifest edit. This replaces the stop signal and grace period the
engine used to give up those waits, and the log store and engine hooks
they needed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(datanode): check rows after a healthy shutdown and wait for the WAL stop
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: keep only the shutdown tests that prove the new stop order
The engine-level tests pinned the log store's stop contract, which the
log store's own tests already cover, and the stalled-create test
asserted a limitation rather than a guarantee. Drop them with the
testing hooks only they used; the datanode and region server tests
cover the change end to end.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(log-store): keep the object store WAL reclaim boundary independent of the region count
The WAL actor recomputed the reclaim boundary after every watermark
update, release and indexed object by scanning the footers of the objects
from the previous boundary, checking each segment against the obsolete
watermarks. Each scan started at the first segment of the boundary object,
so when one object holds segments of many regions whose watermarks advance
one region at a time, every update inspected the segments already known to
be obsolete again, about R^2/2 footer entries in total for R regions.
The catalog now keeps the obsolete watermarks itself and counts the live
segments of every object. A watermark move visits only the segments of its
region that it newly reaches, through the per-region index, and the
boundary is the first object at or above the floor that still holds a live
segment. The boundary is unchanged: segments of release records never hold
it, a region without a watermark does, a watermark of EntryId::MAX releases
every segment of its region, and it never moves below the floor. The store
records each watermark in the catalog together with the one reads use.
The per-region index is now keyed by the largest entry id of each segment,
which is also object sequence order because the entry ranges of a region
increase strictly with the sequence. Indexing an object already enforced
that; it now checks the neighbours by entry id instead of by sequence,
which accepts and rejects the same objects.
A property test checks the boundary against a scan of the footers over
random appends, watermark moves, releases, drops, collections and
recoveries, and ignored measurement tests time watermark updates and tail
reads through the store.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(log-store): seek the object store WAL objects of a region by entry id
A read found the objects of its region by filtering every object of the
region that the catalog retains, so a read of the last few objects cost
time in proportion to the whole retained history of the region, under the
catalog read lock.
The per-region index is keyed by the largest entry id of each segment, and
the entry ranges of a region increase strictly with the object sequence,
which the catalog enforces when it indexes an object, whatever entry ids
the objects carry. So the lookup seeks to the first segment that ends at
or after the start of the range and stops at the first that starts after
its end, and the result is still in sequence order.
The property test of the catalog now also checks range lookups against a
scan of the footers.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(promql): honor @ modifier on subqueries
`prom_subquery_expr_to_plan` never read the `at` field of a subquery, so
`sum_over_time(m[20s:10s] @ 60)` was evaluated like `m[20s:10s]` and
followed the outer evaluation grid.
Handle `@` the way Prometheus does: fold the anchor into the subquery
offset (`offset + (eval_start - anchor)`, as `setOffsetForAtModifier`
does), evaluate the subquery once at the start of the evaluation, and
replay the folded window over the outer grid, reusing the anchored
range selector machinery. A call whose range argument is an anchored
subquery is promoted as step-invariant, like one over an anchored range
selector. `@ start()`/`@ end()` and `offset` combine as for selectors,
and an unrepresentable anchor is rejected with the existing error.
Closes#9411
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* test(promql): add sqlness cases for @ on subqueries
Cover the #9411 reproducer end to end: `@ <ts>`, `@ start()`/`@ end()`,
`@` combined with `offset` in either order, an instant query away from
the anchor, `*_over_time` functions and `rate` over an anchored
subquery, a computed inner expression, and an aggregation above the
anchored call. Two series per table so a dropped or merged series
shows up as a wrong row.
Refs #9411
Signed-off-by: Aarav <aaravsjadav@gmail.com>
---------
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* perf(log-store): reuse the release records the object store WAL recovery already read
Recovery reads every object it lists to verify its header and footer: an
object of at most 64 KiB whole, a larger one as its header and the last
64 KiB. It then read the segment of release records of every indexed
object that holds one again with a range read, although the segment, the
last one before the footer, is almost always among the bytes already read.
A prefix of many dropped regions, each released in an object of its own,
paid two reads per such object.
The footer read now keeps a copy of that segment when the bytes it read
hold it, and recovery decodes the copy, or reads the segment as before when
the copy is missing, only for the objects on the selected chain. The
segment is copied so that the read window is not held until the releases
are decoded. Chain selection, the merge of the largest released entry id
per region, and the failure of recovery on a release segment that cannot
be fetched or decoded, whatever on_corrupted_segment says, are unchanged.
Tests cover the reads of a recovery whose release segments are all in the
read bytes, a release segment that starts before the tail window, and that
a damaged release segment fails recovery on the chain and is never decoded
off it. An ignored measurement test counts the reads and times recovery
over a prefix of dropped regions.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(log-store): bound the release segments object store WAL recovery keeps
Recovery keeps the segment of release records of every object it fetches
before it selects the chain, so an object off the chain could hold up to
the tail window until recovery ends. Keep a segment only up to 4 KiB, far
above the records of the few regions a release seals at once, and read a
larger one again only when its object is on the chain, as before.
Tests cover a release segment inside the tail window of an object read as
its header and its tail, which exercises the offset into the window, and a
segment above the kept size in an object read whole.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: add crash, restart, multi-region and drop cells to the S3 WAL benchmark
Extend the object store WAL S3 benchmark beyond one load per configuration:
- Inputs for a full image reference, the configurations to run (R0 adds Raft
Engine with the standalone default sync_write = false), extra environment
for every server container, a restart after each load, and a JSON list of
named cell groups that override those inputs.
- Restart cells SIGKILL the loaded server, restart it on the same prefix and
data home, and time /health, the first correct count and the log store
recovery; min_objects fails a cell that left too few objects indexed.
- Crash groups write striped 1000-row batches from several writers, kill the
server at a random point, restart it and check every acknowledged batch
against the table (durable: present exactly once; enqueued: no duplicate,
no partial batch, losses counted). A pinned variant keeps one row of a
second table unflushed and samples the indexed objects while it runs.
- Idle windows scrape the server before and after a quiet period.
- A load group may split the loaded table into equal event_time ranges, so
every batch writes to every region, and checks that each region took rows.
- Drop groups write a table, drop it, kill and restart the server, then
write and flush another table and wait for the dropped table's objects to
be collected.
- The summary adds WAL traffic, objects per second, mean object size,
entries and appends per object, S3 requests per operation, server and host
CPU, the loader's own request latency, rows per region, crash cycles, and
each restart's replay per region from the server log.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: report the object sequences a replay spans instead of objects replayed
The width of a region's replayed entry range counts every object sequence in it, including objects that hold no entry of the region and are skipped, so the summary labels it as a span.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail a crash cycle that loses a batch an earlier restart recovered
The oracle accepted every missing acknowledged batch in the enqueued mode,
so a batch that one restart recovered and a later restart no longer found
passed. Every cycle keeps what its restart found, and a cycle now fails in
either mode unless every batch an earlier restart recovered is still
present, so only the current cycle's backlog that was not durable at its
kill may be missing.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: keep the pinned WAL pinned and label OpenDAL operations as such
The pin table inherited the engine's 10-minute auto flush, so a longer
pinned cycle could flush its row and release the WAL before the kill. It now
sets its own auto_flush_interval of one day, longer than any crash group can
run, and a pinned cycle fails unless its indexed objects never fall while
it runs.
The summary's request counts come from opendal_operation_duration_seconds,
which counts OpenDAL calls rather than HTTP requests (a paginated list or a
multipart write is one operation), so the fields and columns now say S3
operations.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check a pinned cycle's retention by the collected-object counter
Net growth of the indexed objects can hide collection, and short cycles take no periodic sample, so a pinned cycle now fails unless deleted_objects_total is the same in the scrape before its writers start and in the one before the kill, both of the same server process. The operation columns name OpenDAL.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: let a pinned cell collect only the objects below its pin
The first write after a start lets the store collect the empty start object asynchronously, so a deletion after a cycle's baseline does not mean the pin let go. A pinned cell now scrapes the server before it writes the pin row and fails only once its servers together have collected more objects than were indexed then, counting the first server from that scrape and every restarted one from its start.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: test that a pinned cell's allowance spans its server restarts
Two servers that each collect fewer objects than lie below the pin but more together now fail the cell in the second cycle.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(otlp): chunk trace spans across resource groups
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): plan spans of a mixed binary/JSON chunk individually
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(otlp): describe cross-group trace chunking
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add column encoding option and set it for trace span ids
COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'.
Either disables the dictionary for the column; delta_binary_packed is
limited to integer and timestamp columns.
OTLP trace v1/v2 tables now create span_id and parent_span_id with plain
encoding and 64KiB pages, and timestamp_end with delta_binary_packed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add compatibility cases for column encoding options
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: move column encoding downgrade checks into downgrade_compatibility
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): keep processing bulk requests after one fails to encode
When `push_bulk` fails to build the WAL entry of a bulk request, it answers
that request with the error. `process_bulk_requests` then returned from the
loop, so the remaining requests of the drained batch were dropped without an
answer and their callers only saw a receive error. Skip the failed request
and continue with the rest of the batch instead.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): keep bulk install barriers independent per target
The bulk install test barrier was a single process-wide slot, so tests
that arm it while running in the same process replaced each other's
barrier. Keep one barrier per armed target instead.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
refactor(mito): avoid pinning region version when rebuilding indexes
handle_rebuild_index kept the whole region version alive across the
manifest read and schedule_build awaits, which pins memtables and the
SST version even though index build tasks only need the region metadata,
index options and the candidate SST file handles.
Extract those inputs into a small IndexBuildSnapshot and drop the version
before awaiting.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: cover pre-releases and upgrade notes in the release-note skill
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: add style and length rules to the release-note skill
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: drop machine-specific notes from the release-note skill
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: base the first pre-release on the previous release branch point
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: subtract the whole previous line when building a minor release note
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(servers): accept application/x-json-stream for log ingestion
Support fluentbit's json_stream output format on the HTTP log ingestion
endpoints by accepting `application/x-json-stream` as a payload content
type.
The payload is parsed as concatenated JSON -- multiple JSON values placed
back-to-back without a separator, e.g. `{"a":1}{"b":2}` -- reusing the
streaming value iterator already used for `application/json`, so the
behavior for concatenated or mixed payloads is identical.
Aimed at #5404 (msgpack support is left as a possible follow-up).
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
* chore(servers): remove redundant comments per review feedback
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
---------
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
* fix(log-store): refuse to open object store WAL regions whose history is missing
A region persists only its WAL prefix, not the object store that holds it.
After `[wal].storage_provider`, or the bucket or root of the provider it
names, changes, or after the objects under the prefix are deleted, every
region reopened against a prefix without its history, replayed nothing and
silently lost its acknowledged writes that were not flushed.
Add `LogStore::check_region_history` and `LogStore::start_region_history`,
whose defaults accept and do nothing, so Raft Engine and Kafka are
unchanged. The object store WAL reports the history of a region as missing
when recovery found no object under the prefix, or when the region's
watermark names an object above the newest object holding entries that
recovery indexed, unless the catalog holds the region's entries through the
watermark or the region's history starts in the store. Both rest on
invariants of the prefix: an opened prefix is never empty, and a persisted
watermark names an object at or below the newest object holding entries,
which collection never deletes. The segment of release records counts as
entries, since an object holding only release records can be the one
collection keeps; start objects hold no segment and do not move the bound.
When a store that found its prefix empty refuses a region, the actor gives
up its start object, so the next start refuses the region again. It agrees
only while nothing was handed an id, no batch, append, release or create
waits, no failed release is kept, no other object was listed, the catalog
holds only the start object and no region's history started in the store,
deciding under the lock region registration takes. It then records a
terminal error, deletes the object as one of its tracked deletes, which
`stop` waits for, and answers every refusal once that delete settled.
Mito runs the check before it replays an existing region on open, on
catch-up and when a create finds the region present, with the larger of
the flushed and truncated entry ids, and records the history of a region
it creates; regions that skip the WAL and noop regions are not checked. A
missing history fails the open with an error naming the region, the
prefix, the configured storage and how to move the WAL by copying its
objects. The new `[wal].accept_missing_history` option, off by default,
opens such a region without its unflushed writes instead, with a warning
and the counter
`greptime_logstore_object_store_wal_missing_history_accepted_total`.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe the object store WAL history check
Describe the open-time history check, the prefix invariants it relies on,
how release records bound the history, the start object withdrawal of a
refused open and the window that remains, the accept_missing_history
option and how to move a WAL by copying its objects, and update the
metrics, Mito integration, evidence, compatibility and cluster alignment
sections.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): unindex the start object a refused open withdrew
A withdrawal that deleted the start object left it in the catalog and in the indexed object and byte gauges for the rest of the process. Remove it from both once the delete succeeded, and keep it when the delete failed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): separate the withdrawal windows of start objects and entries
An append or a release admitted before a refusal leaves an object holding entries, which may have collected the start object, so the next start also passes watermarks at or below that object.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): keep a successor's chain complete across a late withdrawal
A start object withdrawn after another instance recovered it was linked by that instance's start object, which carried reclaim boundary 0, so once the delete landed the next open found no complete chain. A start object now reclaims every object below it when the recovered chain holds no segment, so such a delete stays below its boundary.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): record a created region's WAL history before its manifest
A create recorded the region's WAL history after it persisted the manifest, so a withdrawal in between failed the create but left a manifest that the next start refused as a region without its history. Record the history first, so such a create fails before it writes anything.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe the start boundary of a chain without segments and the create order
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
The merge_group coverage job did not set RUST_MIN_STACK, so test threads
ran with the default 2MiB stack while the regular test job (and the
production runtime in debug) use 8MiB. The deeper PromQL planning chain
from #9494 overflows 2MiB under llvm-cov instrumentation, aborting
tests::promql_test::anon_promql_ratio_repro::case_1_test_with_standalone
with SIGABRT (12/12 failures across three merge-queue candidates).
Align the coverage job with the existing 8MiB test policy.
Signed-off-by: discord9 <discord9@163.com>
* fix(flow): retain tracked write completion across transient failures
Use receiver-issued single-write capabilities at the frontend and datanode. Rejoin exact admitted writes across unknown transport outcomes, then close and drain before allowing subsequent rounds. Keep ordinary clients and reads on their existing paths.
Separate schema map equality from encoded row identity so fresh protobuf decodes replay consistently even with NaN values. Cover receiver admission, owner cancellation, all-child drain, and retired capabilities.
Local feature-worktree checkpoint only. Main verified core enterprise check and 484 tests, plus DN enterprise check and all four tracked owner tests. Corrected seven-case timer acceptance remains to be rerun; this checkpoint does not authorize publishing or integration.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): preserve upstream dependencies after rebase
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): fail fast on definitive tracked-write open rejection
A tracked write treats every application-level error from the frontend as
an ambiguous transport failure and retries the same pinned frontend
forever. During a rolling upgrade an older frontend does not recognize the
tracked-write command extension and processes Open as an ordinary empty
SQL query, rejecting it with InvalidSql (InvalidArguments); a non-internal
endpoint rejects the extension outright. Both are definitive rejections
that no same-frontend retry can ever resolve, so the flow stalls.
Classify the Open-phase errors by status code: a definitive rejection
(Unsupported / InvalidArguments / PermissionDenied) fails fast with a
terminal pre-write error, while genuinely ambiguous transport failures
(Unavailable, timeout, connection reset, ...) keep the same-capability
retry. Write/Close rejections are deliberately NOT failed fast: once a
write may have been admitted, a same-code rejection (e.g. WriterLost, or
UnknownCapability after a VIP moved Open->A/Write->B) leaves the outcome
ambiguous, and failing fast would let the caller replay and duplicate
rows.
Add a regression test that a legacy empty-SQL Open rejection fails fast
with a single attempt and surfaces as Error::External carrying a
TerminalError.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): preflight tracked writes without submitting data
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): retain tracked command extensions while decoding
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
rustls 0.23.46 fixesrustls/rustls#3301: on TLS 1.2 with AES-GCM, reaching the 2^24 record limit dropped a fragment while reporting the write as complete, then kept sending records after close_notify. This stalled Kafka WAL produce requests until the request timeout.
fix(pprof): pin upstream framehop initialization fix
Pin pprof workspace-wide to 01cff82d containing tikv/pprof-rs#282. Initialize framehop before sampling instead of allocating and discovering shared libraries in the first SIGPROF handler. Regenerate the lockfile without unrelated dependency changes.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: add repeatable metric snapshot S3 benchmark
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: use only existing CI S3 permissions in the metric snapshot benchmark
The CI S3 user is not allowed GetBucketLocation or ListBucketMultipartUploads,
so the bucket probe and the prefix cleanup failed with AccessDenied. Use only
list, put, get and delete under the run prefix, as the object store WAL
benchmark does, and probe S3 before allocating the EC2 runner.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: scope metric snapshot inventory to the current round
The full round's S3 inventory listed the whole run prefix, so it also counted
the smoke snapshot. List only the round's snapshot prefix while cleanup still
covers the run, and assert that the teardown's S3 cleanup step runs always()
with the validated run prefix.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs: record the AWS validation run of the metric snapshot benchmark
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(query): keep state column names unique when splitting aggregates
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): probe allocated names when renaming colliding state columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(jaeger): return the most recent traces when a search hits its limit
Trace search applied LIMIT to an unordered DISTINCT trace_id, so a search
matching more traces than the limit returned an arbitrary subset that
changed between identical requests. Order traces by their latest matching
span before applying the limit.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(jaeger): name the trace dataframe builder with a verb and document execution
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
RegionScanExec reported null_count = 0 and distinct_count = total_rows for
every column of an append-only table. DataFusion answers count(col) and
count(DISTINCT col) from these statistics, so both returned the row count.
Keep the exact row count and an exact zero null count for non-nullable
columns, so count(*) is still answered from statistics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
On append-mode tables a region reports an exact row count. When it is
zero, DataFusion's AggregateStatistics replaces the region-side
__min_state/__max_state aggregate with a non-null struct literal, so the
remote stream advertises the state column as non-nullable. The MergeScan
schema check required equal nullability and failed the query.
Accept a remote field that is non-nullable where the expected field is
nullable, matching DataFusion's own physical-plan schema invariant.
Batches are already rebuilt with the expected schema. Widening is still
rejected, and nested field nullability is still compared exactly as part
of the data type.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(log-store): persist object store WAL region releases
A region's release lived only in memory: a restart after DROP TABLE and
before the next object never re-established the dropped region's
watermark, so its segments counted as live and pinned garbage collection
of the whole prefix.
A release now writes a release record (region id and the entry id the
region is released through) into an object sealed at once, and returns
once that object is durable and indexed, in both acknowledgement modes.
Recovery seeds the obsolete watermarks from the release records of the
indexed objects before it computes any reclaim boundary. obsolete_all
releases in full; the new LogStore::release_region, a no-op by default,
releases a dropped region through the largest entry id the store handed
out for it, and Mito calls it after obsolete when it drops a region.
Objects with release records use format version 2, the version 1 layout
with the records at the end of the footer; every other object is still
written as version 1. A counter of release records made durable is added.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe persisted object store WAL region releases
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): carry a failed region release into the next object
A release whose batch failed left its record only in an orphan object,
so a drop that failed there could still leave the region unreleased
after a restart. The store now keeps the release records of failed
batches and seals them with its next batch, on a timer tick when no
entry is admitted, so the release becomes durable once a create
succeeds again.
The version 1 and version 2 object fixtures are frozen under
tests/compatibility/wal/object_store and loaded by the format tests,
and the RFC no longer says that state is persisted only in headers.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): complete a region drop whose WAL release fails
A drop request is not sent again once the region was removed, so a drop
whose WAL release failed used to leave the region half dropped. The
drop now completes, cleaning up the region's runtime state and
scheduling its files for deletion, and reports the release failure.
In the durable mode the object store WAL now creates one more object at
stop with the release records of failed and dropped batches, so a
graceful stop does not lose a release that failed. The release record
counter is described as counting the records of indexed objects.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(log-store): describe the stop-time release create of the WAL actor
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): state the accepted window of unreleased dropped regions
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(log-store): describe release waiters in the WAL actor's outcome table
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): fail region releases while the object store WAL is failing
A failing store keeps its backlog and rejects appends until a create
succeeds, so a release would otherwise wait for an upload that is not
coming. A release that arrives in that state now fails at once with the
retryable error, and one whose batch is still pending when the store
enters it fails as well. Neither poisons the store: the record of the
first waits for the next sealed batch, like that of a batch that failed,
and the record of the second stays in the backlog, so both become durable
once a create succeeds.
Rejected releases count in rejected_appends_total.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe region releases while the object store WAL is failing
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): place the drop crash window in the drop procedure
The crash window between the deletion of a table's metadata and a
durable release record is a gap of the drop procedure, not of the WAL.
Describe it as far as the code shows, name the remedy that belongs in
the procedure, cleaning up offline a region a resumed region drop finds
not open, and list it under future work.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): keep rejected release records coalesced while failing
A release rejected while the store is failing kept its record for the
next sealed batch, but every timer tick then sealed that record alone
behind the refused backlog, so a release repeated across ticks grew the
backlog without bound. A failing store now keeps such records coalesced
and seals them on the first tick after a create succeeds.
The fault-matrix case of a release while the backlog is refused now runs
with a 10 ms flush interval, repeats the release, and checks that the
backlog does not grow and that recovery reads the record back.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): keep rejected release records unsealed while failing
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): cite the drop procedure issue for the crash window
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): let a stop seal rejected release records of a failing store
Keeping the records of rejected releases out of sealed batches while the
store is failing also kept them out of the seal a stop requests, so a
stop whose last attempt of the backlog succeeded dropped them and the
released region pinned collection after the restart. A stop now seals
them with the backlog.
A fault-matrix case rejects a release while the store is failing, stops,
lets both creates succeed, and checks the record after a reopen.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe the stop of a failing store with rejected releases
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): keep release records readable by older object store WAL readers
Release records were written as a version 2 footer, which a build without
it refuses to read, so rolling a node back after a drop failed to open
the whole WAL prefix. They are now the single entry of a segment of the
reserved region u64::MAX at the first position of a version 1 object:
an older reader indexes that segment as a region it never opens, and
recovery reads the segment back with one range read per object that
holds one. The segment never holds the reclaim boundary and names no
namespace.
A durable stop whose last create of release records fails now reports
the error instead of returning Ok and losing the records silently.
The tests are trimmed: the release unit tests and fault-matrix cases
that repeated each other are merged, and the Mito drop tests share one
parameterized case.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe release records as a segment of a reserved region
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): report release records a poisoning stop drops
A durable stop whose last create of release records met a conflict
poisoned the store and returned Ok, losing the records silently. Stop
now reports the error whenever release records are dropped after it
began, by a rollback or by poisoning. The test of a failed release
covers that conflict, and seals its record on a tick after a one-shot
create failure, so the next tick cannot fail too.
The documentation now says that recovery reads the segments of release
records, and that one that does not decode fails recovery.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe how recovery reads segments of release records
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): say that stop reports release records a conflict drops
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): restore the transiently failing release create in enqueued mode
The fault matrix again covers a release whose create fails transiently in
the enqueued mode: the release waits for the repeat under the same
sequence and returns once it succeeds. The encode_batch doc now says the
release segment is the last segment of the object.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): place the release segment and count the restored case
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): do not fail a region drop on a failed WAL release
Returning the release error from the drop left a metric physical region
half dropped: the metric engine stops after the data region and never
drops the metadata region or forgets the physical region, and a retried
drop finds the data region gone and counts it as done. The release error
gave the caller nothing to retry, since the region is already removed
and the log store keeps the record for its next object. The drop now
logs the error and succeeds.
A metric engine test drops a physical region while the WAL releases of
both mito regions fail, and checks that both regions and the physical
region state are gone.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): a failed release no longer fails the drop
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): do not fail a region drop on a failed WAL obsolete
A store that the release of another region poisoned fails obsolete as
well, which left the metric engine with its metadata region undropped.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(metric-engine): drop a physical region whose WAL obsolete or release fails
Cover a failed data release, two failed releases and a conflicting
release that poisons the store, after the regions wrote to the WAL, and
check that a store opened later keeps no segment of either region.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): a failed obsolete does not fail the drop either
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): fail the drop on an obsolete error of other WAL providers
Only the object store WAL is shared by the regions of a store that the
release of one can poison; Raft Engine and Kafka keep failing the drop
when obsolete fails.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): a stopped or poisoned store keeps no failed release
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
The flow and nested-MergeScan ingress tests hardcoded "follower" among
the read-preference values that must be rejected, coupling them to the
current leader-only ReadPreference enum. Assert only genuinely invalid
values there, matching the wire-values test which already limits itself
to ["", "unknown"]. This keeps the ingress tests valid for builds that
extend ReadPreference with more variants.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat(mito): decouple compaction unit execution and publication
Execute shared-input dependency groups as independent local compaction units. Publish and acknowledge each unit separately, retain successful sibling results, and release each unit's input leases after worker apply.
Preserve picker priority, panic notifications, snapshot-free execution context, and indexed input leases. Include regression coverage for independent visibility, purge, cancellation, partial failure, and restart recovery.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): use crate visibility for compaction unit items
Replace the newly introduced pub(super) declarations with pub(crate), and make CompactingFiles::try_reserve private since it is only used within its module, per docs/style-guide.md.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): simplify compaction unit state and test fixtures
Keep local input leases in the task and derive running units from tracked and pending work. Replace the single-implementation task trait with an inherent async method and align exposed state type visibility.
Use snapshot-free region fixtures and named test scenarios, and cover independent input purging across three units with bounded parallelism.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): align compaction task and test module names
Rename CompactionTaskImpl to CompactionTask now that the CompactionTask
trait has been removed and the Impl suffix no longer marks a trait
implementation. Rename the state test module from lease_tests to tests
to follow the test module naming convention.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): align compaction task debug label
The Debug impl reported "TwcsCompactionTask" while the type is now
CompactionTask. Report the type name for consistency.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): label compaction elapsed metric per published unit
Record greptime_mito_compaction_total_elapsed once per published unit and
label it by type: a local dependency unit or a remote job. Update the
metric docs and group the Grafana compaction OPS/latency panels by type.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito): clarify compacting file lease sharing
`CompactingFiles::for_inputs` clones the shared per-file leases instead of
transferring them. Document that the original collection keeps its leases,
so an input is released only after every owner drops its lease.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito): document local compaction unit map key
Document that `LocalCompaction::units` is keyed by each unit's execution
plan ID, which matches `CompactionExecution::plan_id()` and the `plan_id`
carried by unit notifications.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): rename submit_prepared_compaction
The function submits remote jobs but only prepares local unit tasks; the
caller dispatches them after installing the returned phase. Rename it to
`prepare_or_submit_compaction` and document that ordering.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(hdfs): clean up temporary file when an atomic write fails
Signed-off-by: fengyue-xve <1759519137@qq.com>
* style: apply rustfmt to hdfs layer test
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(hdfs): report earlier write failures and log cleanup failures
- Later `write`/`close` calls on a writer whose failed write was
cleaned up now mention the original failure instead of the bare
"unavailable after close or abort" message.
- `abort_and_delete` logs abort and deletion failures with the
temporary path instead of discarding them; the original write error
is still returned to the caller.
- The temporary-file cleanup test now lists the directory holding the
temporary file, so it fails if the cleanup is reverted.
Signed-off-by: 风月 <1759519137@qq.com>
Signed-off-by: fengyue-xve <1759519137@qq.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
Signed-off-by: 风月 <1759519137@qq.com>
* fix: align fulltext bloom tokens with matches_term and use them for LIKE
The bloom fulltext index stored tokens that did not follow
matches_term's word boundaries (`_`, ASCII next to Han, Han substrings),
so pruning dropped rows matches_term accepts.
Tokens and query probes now share the character classes of
matches_term, Han runs are indexed as bigrams, and LIKE patterns use the
same probes. The bloom blob is written under a new key; blobs from older
versions are no longer probed. Tantivy no longer serves matches_term.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: drop unused lazy_static from index and avoid typo-flagged test tokens
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: link like_probes to tokenize instead of the downstream ScriptTokenizer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: name probe extraction with verbs and document CharClass
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: rebuild legacy fulltext bloom indexes and probe lower(col)
The manifest records the fulltext bloom version per column, so ADMIN
BUILD_INDEX now rebuilds SSTs whose bloom index predates the current
tokens, and ssts_index_meta shows the version.
Case-insensitive bloom indexes also store the tokens of the whole
lowercased text for rows where it can differ (U+0130, U+212A, sigma),
so matches_term(lower(col), ..) and lower(col) LIKE .. use the index.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover legacy bloom index rewrite by compaction in the compat case
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: log outdated fulltext bloom indexes in the consistency check
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: explain the unrecorded fulltext bloom version
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): centralize PromQL function registration in promql::functions
Move the PromQL scalar UDF and UDAF registrations out of the query plan
decoder into a single `promql::functions::register_promql_functions`
entry point, so plan decoding no longer has to enumerate every PromQL
function implementation. The registered function set, aliases (e.g.
`prom_holt_winters`) and registration order are unchanged.
Add a registration test pinning the full set of `prom_*` scalar
functions and the `prom_holt_winters` alias, plus a decode regression
test covering the centralized registration path.
Fixes#9493
Signed-off-by: fengyue-xve <1759519137@qq.com>
* refactor(promql): register Ieee UDFs centrally and address review
Rebase onto main picked up #9479, which registered the Ieee comparison
and sqrt UDFs from the decoder directly. Register them from
`register_promql_functions` instead so the single entry point stays
complete, and add the seven `prom_eq`..`prom_sqrt` names to the
registration test.
Also per review: drop the doc paragraph that explains why the function
was moved, drop the redundant `test_serializer_decode_centralized_promql_registration`
test and the registration comment in the decoder, move `use std::sync::Arc`
up with the imports, and simplify the registration test assertions.
Signed-off-by: 风月 <1759519137@qq.com>
Signed-off-by: fengyue-xve <1759519137@qq.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
Signed-off-by: 风月 <1759519137@qq.com>
* fix(log-store): keep the enqueued backlog on permanent object store errors
On object storage the WAL's object store sits behind OpenDAL's retry
layer, which marks every error it returns as persistent. The enqueued
mode classified create failures by that status, so a 403 or a missing
bucket was repeated every 100 ms on every create in flight, logged
nothing, and made appends hang once the sealed batches were full.
Object store errors are now classified by kind: denied access, an
invalid configuration (how OpenDAL reports a missing S3 bucket) and a
write to a missing location are non-retriable. In the enqueued mode such
an error makes the store failing instead of poisoning it: the
acknowledged backlog is kept under its sequences and bytes, its lowest
pending batch is attempted alone with a delay that doubles from 100 ms
to 30 s, appends are rejected at once with a retryable error that
carries the last object store error, and the first create that succeeds
ends the state. Stop uploads the backlog at once and reports a failure
as before. The state is logged on entry, at most once a minute while it
lasts and on exit, and exported as the `failing` gauge and the
`rejected_appends_total` counter. Conflicts, encoding and catalog errors
still poison the store; the durable mode is unchanged.
An append held back because the sealed batches are full is now counted
and timed by the stall metrics in both modes.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(rfc): describe how the enqueued object store WAL handles refused creates
Describe the enqueued failure contract as the store now implements it:
non-retriable object store errors are told apart by kind because
OpenDAL's retry layer under the store marks every error persistent, a
failing store keeps its backlog and attempts it with backoff while it
rejects appends, and stop uploads the backlog at once. Add the failure
matrix row, the failing gauge and the rejected appends counter, the
sealed-batch admission limit and the stall metrics that now count it,
and the fault matrix case counts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): time the failing attempts where they start and fix two statements
The test of the attempts of a failing store now records when each
refused write starts and checks the gaps between them, instead of
sampling a counter after a sleep of the test task. The delay comment and
the RFC describe 30 seconds as the longest delay between two attempts,
not as a bound on when appends resume, and the RFC states that the next
append is held once seven sealed batches wait.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): drop the wait_until helper that main now defines
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(log-store): name errors that need an operator and trim the RFC contract
Rename is_non_retriable to requires_intervention: such a create is still
repeated, only slowly, until someone fixes the credentials, configuration
or bucket. Name the backoff test after what it asserts, cut the test of
the lowest pending batch to two failed rounds since the cadence is
covered by the backoff test, and shorten the RFC failure contract to
the contract and its rationale, leaving error mappings, request counts
and logging cadence to the code comments.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(log-store): tie the slow repeat to errors requiring intervention
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
The manage-region-follower chain submitted its procedures with an empty
executor context: `ProcedureServiceHandler::manage_region_follower`
accepted no query context, the operator built an
`ExecutorContext::default()`, and `MetaClient` dropped the context before
delegating to `RegionFollowerClient`. The submitting user (actor) and the
procedure event metadata therefore never reached the submitted
procedures.
- `ProcedureServiceHandler::manage_region_follower` now takes a
`QueryContextRef`, consistent with `migrate_region`, `gc_regions` and
`gc_table`.
- `ProcedureServiceOperator` derives the `ExecutorContext` (actor and
`TriggerReason::Manual` event input) from the query context.
- The `RegionFollowerClient` trait methods now accept `&ExecutorContext`
and `MetaClient` forwards it, so client implementations can transport
the actor and event context on the wire.
- `procedure_event_context`/`procedure_actor` are now public so client
implementations can reuse the trusted protocol derivation.
* refactor(operator): route shared requests through NodeManager
Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): preserve retry policy through error hints
Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): restore sparse partition tags when rebuilding writes
Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): stop detached write retries when callers cancel
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(frontend): configure retries across distributed write paths
Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage.
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: support compaction options in HTTP hints
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: share TWCS compaction option prefix
Use the store-api constant for both HTTP hint inference and Mito region option validation so the two paths cannot drift.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
fys
jeremyhi
DeviousCardi
dennis zhuang
Weny Xu
Lei, HUANG
localhost
discord9
YingNing
shuiyisong
Yingwen
Ning Sun
fengyue-xve
sun
NightG1owZandNightG1owZ