* fix(promql): default a stepless subquery to a one-minute step
`x[5m:]` evaluated the inner expression at the query step, which is 1s
for instant queries. Prometheus uses the evaluation interval, one minute
by default.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover the default subquery step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover the default subquery step in a range query
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 845fc0af9c)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): return range-vector results for instant queries
An instant query of a range selector or subquery (e.g. `up[5m]`) returned
the internal range arrays, which panicked when the frontend decoded them as
dictionaries. Plan such queries as the selected samples with their own
timestamps, and reject range-vector expressions in range queries like
Prometheus does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): keep encode_substrait on a valid range query
A range query of a range-vector expression now fails like in Prometheus.
Wrap the subquery in max_over_time so the case still encodes both
RangeManipulate nodes through substrait.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): plan wrapped range-vector roots and reject a zero subquery step
TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the
range-vector path did not unwrap. A zero subquery step divided by zero.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover wrapped range-vector roots and a zero subquery step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): default subquery step and reject range vectors in range queries
A root subquery without a step uses the one-minute default instead of
the query step. The HTTP range query endpoint rejects range-vector
expressions even when start equals end.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover root subquery default step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): align root subqueries with subquery_child_window
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): return millisecond timestamps for range-vector results
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): keep the time index timezone and name its table in errors
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit b913be2af7)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): unify unknown flow option rejection across OSS and enterprise builds
Remove the enterprise-gated passthrough that let unknown flow options
through SQL-layer validation. Both OSS and enterprise binaries now reject
unknown flow options with a single 1004(InvalidArguments) error at the
SQL layer, producing identical sqlness output.
The metasrv-side validate_flow_options remains as a defensive check.
Signed-off-by: discord9 <discord9@163.com>
* fix(flow): use DISTINCT output columns as sink key for streaming flows
When a streaming flow's source table has no primary key, sink key
inference produced an empty key. For SELECT DISTINCT v, the DISTINCT
establishes v as the output key, but the inference only looked at
source primary keys. Combined with the zero-valued placeholder
timestamp, all output rows shared the same storage identity and were
deduplicated into a single row.
Use all non-time-index output columns as the sink key when the plan
is a DISTINCT and no source primary key was inherited.
Signed-off-by: discord9 <discord9@163.com>
* fix(flow): preserve extension options and complete DISTINCT sink identity
Keep enterprise CREATE FLOW options available to the existing handler and report unknown ordinary options as InvalidArguments with the OSS SQL error text. Infer DISTINCT sink keys from every selected non-time-index output column, including when source key columns survive. Wait for asynchronous mirrors in show_create_flow and exercise 1000 distinct sink rows in the SQL regression.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): align meta unknown option status assertions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): use valid aliases in DISTINCT sink regression SQL
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): record complete DISTINCT sink SQL results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 178a0021b8)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): accept evaluation times before the Unix epoch
Prometheus accepts negative timestamps in query parameters; GreptimeDB
rejected them while parsing and would panic converting them in the planner.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: update flow_tql error for a negative TQL end time
A negative end now parses and fails the end >= start check instead of
the timestamp conversion.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): reject evaluation times outside the millisecond range
The signed millisecond conversion cast u128 to i64 unchecked, so a far-past
time wrapped into the future. Reuse the checked conversion from
label_values and return an error instead.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: drop an unused import
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): round query times to milliseconds like Prometheus
Numeric times round the fraction of a second to milliseconds, and
pre-epoch times floor sub-millisecond parts, matching Prometheus'
parseTime and timestamp.FromTime.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): round the float fraction before converting to nanoseconds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: skip the overflow case where SystemTime cannot hold it
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit a8e293fd79)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(promql): zero-copy timestamp payload for millisecond zero-offset ranges
RangeManipulate rebuilt the timestamp range payload per sample through
wide-precision i128 conversion even for millisecond input with zero
offset, where the input is already millisecond and nothing changes.
That turned a previously zero-copy path into a per-sample loop plus a
Vec allocation, regressing range-function queries.
Reuse the input timestamp array directly when the time index is already
millisecond and the offset is zero. Other units and non-zero offsets
keep the wide-precision conversion (required so ns/us data near the
native type limits stays correct under large offsets).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): vectorize timestamp payload with plain i64
RangeManipulate rebuilt the timestamp range payload per sample through
wide-precision i128 conversion, turning a hot path into a scalar loop
with a Vec allocation. Realistic timestamps and offsets stay far inside
i64, so the wide path is unnecessary.
- millisecond + zero offset reuses the input array with zero copy;
- other units / offsets shift in one vectorized i64 pass (no i128, no
per-sample Option, no overflow branch).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): check timestamp payload overflow via arrow try_unary
Replace the silent wrapping shift with try_unary + checked_mul/checked_add.
checked arithmetic lowers to a per-lane overflow flag so the single pass
still vectorizes over the dense i64 buffer; any overflow short-circuits to
an error instead of silently wrapping to a wrong timestamp. No i128, no
per-sample Option.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): vectorize cursor scans with i64 fast path
RangeManipulate::calculate_range and InstantManipulate::manipulate
compared every sample through i128 wide-precision arithmetic. For
realistic timestamps and offsets the shifted values stay far inside
i64, so the wide path was pure overhead.
Split both scans into an i64 fast path (checked arithmetic precomputed
once per batch) and an i128 exact fallback. Any input that would
overflow i64 falls back to the i128 scan, keeping window membership
semantics identical.
Verified: 47/47 range_manipulate + instant_manipulate tests, 275/275
promql crate tests, 1612 randomized cases byte-identical to i128 scan.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* refactor(promql): generalize cursor scans and share endpoint precheck
Address review comments on PR #9417:
- Generalize scan_ranges/scan_instants over Nanoseconds (i64/i128),
eliminating the duplicated scan bodies.
- Extract shared narrow_scan precheck used by both RangeManipulate and
InstantManipulate, with identical checked endpoint logic.
- Trim doc comments to one line; fix factual errors about fallback
trigger (2262 for ms/s, not native type limits).
Semantics unchanged: same cursor algorithm, same overflow fallback,
same i64-vs-i128 selection, same stale/null/tie handling.
Verified: 47/47 range_manipulate + instant_manipulate tests, clippy clean.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): vectorize payload with try_unary and add scan carrier bench
Replace the per-sample i128 payload loop with a single arrow try_unary
pass: i128 intermediate holds any native timestamp shift, only the final
millisecond result must fit i64. This is semantically equivalent to the
original i128 implementation (no silent wrap, no per-sample branching).
Add cursor_scan_carriers A/B benchmark: i64 fast path vs i128 exact
fallback on four workloads. Add five deterministic differential tests
pinning scan_ranges::<i64> and scan_ranges::<i128> to be pointwise
equivalent on boundary shapes (sub-ms samples, fallback triggers,
nanosecond extremes, seeded precision matrix).
Bench (N=4096, 15s cadence):
- payload wall_time 20x15s: 74.94us -> 48.98us (-35%)
- payload wall_time 240x15s: 102.44us
- scan i64 fast path: 10.97us vs i128 19.84us (1.81x)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): restore zero-copy payload for ms+offset=0 and fix comment
Restore the zero-copy fast path for millisecond timestamps with zero
offset, guarded by the input column's exact data type (Timestamp
(Millisecond, None)) rather than self.time_unit. The old time_unit guard
would accept a tz-carrying millisecond column and produce a dictionary
value type disagreeing with the output schema; the datatype guard is the
correct check.
Also fix the comment: arrow's try_unary is a fallible kernel that LLVM
cannot auto-vectorize; the win over the hand-written loop comes from
eliminating the per-sample Option/Vec and branches, not SIMD.
Add a regression test with a timezone-carrying millisecond index to
prove it falls back to the try_unary path.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): replace try_unary with unary + Cell overflow flag
Arrow's try_unary is a fallible kernel that LLVM cannot auto-vectorize.
Replace it with the infallible unary kernel: the closure is a plain
i64 -> i64 mapping that sets a Cell<bool> on overflow instead of
returning Err per sample. The flag is checked once after the loop.
This removes the per-lane Result construction and lets LLVM vectorize
the shift+truncate sequence. Bench (N=4096, 15s cadence):
- window4x15s: 42.12us -> 28.77us (-32%)
- window20x15s: 48.98us -> 27.68us (-43%)
- window240x15s: 102.44us -> 27.61us (-73%)
- regular_sparse: 90.04us -> 29.20us (-68%)
All cases now run at ~27-29us regardless of window size; the zero-copy
path dominates for ms+offset=0.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): dispatch payload conversion on null_count; drop wall-time bench
Null-free batches (dominant PromQL case) take arrow's vectorizable unary
kernel with a Cell<bool> overflow flag; nullable batches take try_unary so
a NULL slot's arbitrary underlying value cannot raise a false batch-wide
overflow error. Add a regression test exercising both paths.
Remove bench_range_manipulate_wall_time (all-ms zero-offset cases only
exercised the zero-copy path, measuring nothing about the conversion) and
its orphaned helpers; restore the cursor-scan bench helpers it shared.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(promql): keep nullable timestamp cursor scans on i128
NULL slots may contain unordered arbitrary values, so the endpoint-only
narrow-scan proof cannot bound their interior timestamp arithmetic. Both
range and instant scans now attempt i64 only for null-free timestamp arrays
and otherwise retain the existing exact i128 path.
Add focused interior-NULL regressions through both active entrypoints,
checking concrete selections and output against the existing wide scan.
The null-free fast path and timestamp payload conversion are unchanged.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit b3c756a7dd)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): keep metric name for comparison operators
`collect_metric_names` dropped the response `__name__` for every binary
operator except `and` / `or` / `unless`, so comparisons such as
`node_load1 > 5` came back without the metric name. Grafana rendered them
as `{instance="..."}` and alert templates reading `$labels.__name__` got
nothing.
Follow Prometheus' `shouldDropMetricName` / `resultMetric`: arithmetic
operators drop the name, comparisons keep the left-hand name, except when
the comparison returns a bool or is a one-to-one `on(...)` match. A
`group_left` / `group_right` modifier does not reduce the labels, so the
name is kept there.
Both the HTTP PromQL API and the gRPC gateway share
`retrieve_metric_name_and_result_type`, so one fix covers both.
Signed-off-by: Aumnertic <hznuzc@gmail.com>
* docs: document binary_keeps_metric_name
Signed-off-by: Aumnertic <hznuzc@gmail.com>
* fix(promql): handle on(__name__), scalar lhs and group_right
Signed-off-by: Aumnertic <hznuzc@gmail.com>
* chore(promql): correct result-side docs and drop unreachable String arm
Signed-off-by: Aumnertic <hznuzc@gmail.com>
* fix(promql): drop metric name when explicitly ignored
Handle ignoring(__name__) for one-to-one comparisons and cover
comparison, group_left, group_right and unless behavior.
Signed-off-by: Aumnertic <hznuzc@gmail.com>
---------
Signed-off-by: Aumnertic <hznuzc@gmail.com>
(cherry picked from commit c33a6cfa56)
Signed-off-by: WenyXu <wenymedia@gmail.com>
fix(promql): irate/idelta return no point for a zero-ms sample gap
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
(cherry picked from commit 0dde3a7b70)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): keep vector labels when a computed scalar is on the right
`metric + time()` or `metric * scalar(other)` planned the join with the
scalar side's context, which has no tags, so every series collapsed into
one unlabeled series. Use the vector side's context when the right operand
is scalar-typed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): update results for labels kept with a computed scalar on the right
v * pi() and v + time() now keep the vector's labels and time index name,
as in Prometheus.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): qualify columns of an unnamed operand in vector joins
A comparison filter's projection has no table name, so after the join its
time index resolved unqualified and was ambiguous when the other operand had
the same column. Also keep the empty-metric branch unchanged for scalar
left operands.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover comparison filters joined with a scalar
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 6da5352c53)
Signed-off-by: WenyXu <wenymedia@gmail.com>
Update opensrv-mysql from f3135e5 to 1f3d88c, which replies a proper EOF packet to COM_FIELD_LIST instead of a malformed OK packet that MariaDB 12.3 misparses and hangs on USE information_schema.
Fixes#9158
Signed-off-by: tian1220A <1573612521@qq.com>
(cherry picked from commit 21536a092b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): bound FlowMirrorTask pending rows to prevent frontend OOM
When flownode is slow or unreachable, FlowMirrorTask::detach spawns
unbounded tasks that each pin a full insert batch for ~10s (gRPC
timeout). With continuous writes, in-flight memory grows unboundedly
(throughput x 10s x peers), causing frontend-write OOMKill and a
million-per-hour ERROR storm.
Add a MAX_MIRROR_PENDING_ROWS cap (1M rows) on Inserter: detach
fetch_adds the batch's row count before spawning; if the cap is
exceeded the reservation is rolled back, the batch is dropped, a
DIST_MIRROR_DROPPED_ROW_COUNT metric is bumped, and a warn is logged
instead of spawning. On normal completion the reservation is released
in both success and error paths.
This keeps core writes alive while degrading flow mirror writes to
best-effort, which is the intended trade-off: protect the primary
insert path at the cost of possible flow aggregation gaps.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix: address review - balanced reservation, gauge race, log rate limit
P1: Compute pending_rows() from actual per-peer payloads in
self.requests instead of the pre-fanout num_rows field. This fixes
under-reservation for same-table multi-region batches and over-release
in multi-flownode fanout. Each peer task now releases exactly its own
share so reservations and releases balance.
P2: Replace load+set gauge publication with add/sub symmetry.
reserve_mirror_pending_rows does fetch_add + gauge.add, release does
fetch_update + gauge.sub, eliminating the stale-value race.
P3: Rate-limit the dropped-batch warning to at most once per 10s via
MirrorDropLog (atomic compare_exchange), accumulating batch/row counts
between reports.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: fix race in dual-peer test - wait for gauge release before unlocking
The dual-peer test previously only waited for the pending budget to
return to its start value before releasing MIRROR_METRIC_TEST_LOCK.
But release_mirror_pending_rows moves the budget before the gauge, so
a worker's gauge.sub could still be in flight when the next test
acquires the lock and records its baseline, polluting the result.
Now waits for both pending == start AND gauge == gauge_before.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(operator): preserve instant writes and limit mirror failure logs
Reject saturated requests containing instant-TTL rows with the existing retryable write-admission error before mirror or datanode dispatch. Keep normal source-table inserts independent of best-effort mirror drops.
Reuse a smaller process-wide event limiter for both dropped batches and failed peer tasks, reporting at most once per 10 seconds with failure count, peer context, and error chain. Cover actual mirror construction, mixed and instant admission, fast/slow peer shares, failed-task release, and log concurrency.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(operator): reject oversized instant mirror batches permanently
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 198d92407c)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* docs(config): align example configs and config.md with actual defaults
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(cmd): compare example defaults inside optional sections
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit e2fd8c5695)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(otlp): coerce empty and "-" trace attribute strings to null
Instrumentations fill numeric attributes from absent headers with an empty
string (PSR-7 getHeaderLine returns ""). Coercing that into an existing
numeric column failed and rejected the whole span.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: trim redundant row from empty-string rewrite test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit d640ab8bc5)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: accept a value for the CLI --auto-create-schema flag
clap derives ArgAction::SetTrue for bool fields, so --auto-create-schema
took no value and could only set true, which is already the default.
Users could not turn automatic schema creation off from the CLI, unlike
metasrv's auto_create_schema = false.
Set action = clap::ArgAction::Set so the flag accepts an explicit
true/false value while keeping true as the default.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* chore: enhance CLI argument for auto_create_schema option
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
(cherry picked from commit 329edc88ec)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: coalesce equal histogram quantile bucket boundaries
Merge numerically equivalent bucket boundaries before monotonicity repair, preserving the allocation-free path for unique boundaries. Share coalescing with histogram_fraction and cover duplicate boundaries with unit and SQL regressions.
Fixes#9443
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: keep duplicate infinity buckets in one histogram group
Confirm classic histogram completion at a series/timestamp boundary or EOF instead of the first positive infinity. Retain lookahead across batches and reject incomplete optimistic groups so equivalent infinity labels reach numeric coalescing together.
Add stream and SQL regressions for duplicate infinity labels, including established bucket counts and EOF. Correct older fixtures to use distinct keys for distinct histogram groups.
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: avoid buffering histogram groups with invalid trailing bounds
Discover candidate bucket counts at the first group boundary and let optimistic validation fall back to safe mode for malformed bounds. Document discovery and cover trailing invalid and null labels with a focused streaming regression.
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 97913b4b3a)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): truncate parquet column index min/max for SST writes
The SST writer and the bulk part encoder disabled column index truncation
together with statistics truncation in #6977. Mito never reads the column
index, but every page of a large string or binary column still stored its
full min and max values, uncompressed. Truncate the column index to the
parquet default of 64 bytes and keep chunk statistics untruncated, since
primary key pruning decodes them.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(mito2): note that the column index must not be used to decode primary keys
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit a314ac4281)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): align timestamp() and label_join with Prometheus semantics
- timestamp() over a selector reports the selected sample's timestamp, without adding the offset.
- timestamp() over any other expression reports the evaluation time instead of the input value.
- label_join that overwrites an existing label rejects duplicate label sets at runtime.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): align label_replace and label_join edge cases with Prometheus
- label_replace leaves non-matching series unchanged instead of copying the source value into the destination label.
- label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join.
- label_join with no source labels removes the destination label, and an empty source label name is rejected.
- Empty label values produced by these functions are NULL, the same as an absent label.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): join absent labels as empty strings in label_join
concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): skip rows without a sample in timestamp()
timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 78a7b93292)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): resolve dotted column names as unqualified columns
col(), From<&str>/From<String> for Column and string join keys go through
Column::from_qualified_name, which splits `service.name` into relation
`service` and column `name` and lowercases unquoted identifiers. Build
PromQL column references with Column::from_name / ident() instead.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): resolve remote read matcher labels as unqualified columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover same-name columns differing in case and without()
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 218000e21b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): apply offset to subquery evaluation window
`prom_subquery_expr_to_plan` destructured `SubqueryExpr` without reading
`offset`, so `<subquery>[range:step] offset <d>` planned exactly the same
window as the un-offset form and silently returned data for the wrong time
range. The plain vector/matrix-selector paths already threaded the offset
through `selector_to_series_normalize_plan` and `RangeManipulate`.
Shift the inner evaluation window back by the offset and pass the offset to
the subquery's `RangeManipulate`, which maps the inner samples forward onto
the evaluation timeline before bucketing them into ranges. This matches
Prometheus, whose `evaluator.subqueryTimeRange` evaluates the inner
expression over `(start - offset - range, end - offset]` and whose
`evalSubquery` then hands the samples to the outer range-vector function as
a `MatrixSelector` that still carries the subquery offset. An offset on the
inner selector composes additively, as `subqueryTimes` documents.
`RangeManipulate`'s protobuf message has no offset field and recovers it on
decode from an immediately underlying `SeriesNormalize`. Since
`RangeManipulate` is commutative in `dist_plan` and can be pushed below a
`MergeScan`, insert that carrier node so the offset survives distributed
planning instead of decoding as zero.
Known divergence, unchanged by this commit: Prometheus anchors subquery step
points on absolute epoch multiples of the step, while GreptimeDB anchors them
on the evaluation start. The two agree whenever the offset is a multiple of
the subquery step; the added sqlness cases stay within that range.
Closes#9330
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* docs(promql): correct the subquery-offset rationale and pin the histogram shift
Follow-up to e132c267, which stated two things inaccurately and left one
path untested. No behaviour change beyond a conditional clone.
The `SeriesNormalize` carrier was described as existing "so the offset
survives distributed planning", implying standalone is unaffected. That is
wrong. `local_offset` is read on the substrait decode path
(`range_manipulate.rs`, `instant_manipulate.rs`), and GreptimeDB routes
PromQL plans through `MergeScan`/substrait in standalone too -- the
standalone results `promql/encode_substrait.result` and `precisions.result`
both show `MergeScan [is_placeholder=false, remote_input=[...]]`. Deleting
the carrier therefore empties subquery-with-offset results in standalone as
well, while `count_over_time_subquery_with_offset` keeps passing because it
asserts the pre-serialization plan. Say so at the node, so the next reader
does not remove it believing standalone is safe.
The carrier is also not semantically inert: `SeriesNormalizeStream::normalize`
shifts native histogram `start_timestamp` payloads by `offset`. On the
subquery path that runs on a computed inner result, on top of any offset the
inner selector already applied, and nothing pinned it. It is consistent --
`RangeManipulate` moves the millisecond time index forward by the same
`offset_ms` -- so the distance from a histogram's start timestamp to the
sample carrying it, which reset/rate detection reads, is invariant. Add
`subquery_offset_shifts_histogram_start_and_time_index_together`, which
executes the subquery's node stack over native histogram samples for a
zero and a non-zero inner offset. It cannot be a sqlness case: native
histograms are a struct column with no SQL type or literal
(`sql_data_type_to_concrete_data_type` rejects structs) and the sqlness
runner speaks only MySQL/Postgres, so such rows only arrive over
gRPC/remote-write v2.
The divergence note blamed sub-step offsets. Measured, that condition is too
narrow: an unaligned evaluation timestamp alone diverges, with no offset at
all -- `sum_over_time(fine[20s:10s])` at t=57 samples 47s and 57s here
against Prometheus's 40s and 50s. The real condition is that
`(start - offset)` is not a multiple of the subquery step, and it is
pre-existing: the committed `tql eval (359, 359, '1s')
sum_over_time(metric_total[60s:10s])` case is already an instance of it
(`359 - 60 + 10 = 309`). Reword the comment and the sqlness header
accordingly. A sub-step offset stays accepted: `foo[20s:10s] offset 5s` is
valid PromQL, erroring on it would be a regression and would not close the
gap anyway. This repo has no PromQL-compatibility documentation page, so
there is nowhere else to record it.
Also stop cloning the series key columns when there is no offset, and stop
the `offset 0s` sqlness comment implying its parser error is evidence about
this fix -- the same message comes from a `u32` overflow in
`promql-parser`'s shared duration check (`offset 9999999999d`).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* refactor(promql): trim subquery-offset comments and tests per review
Shorten the planner and sqlness comments to the semantic contract, build
`SeriesDivide` with a clone instead of an `Option`, and drop the
normalize.rs histogram test and the `offset 0s` parser case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* fix(promql): record the subquery offset as the range fold offset
The subquery's `RangeManipulate` now shifts the payload timestamps by the
offset, so `predict_linear` over an offset subquery must recover the
evaluation time with that offset instead of 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>
---------
Signed-off-by: Aarav <aaravsjadav@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit d6474b961c)
Signed-off-by: WenyXu <wenymedia@gmail.com>
fix: bound decompressed request size and close memory-admission gaps for compressed requests (#9264)
* fix: bound decompressed request size and close memory-admission gaps for compressed requests
The request-memory accounting only bounds and charges the encoded bytes on
the wire, but a compressed body can expand far beyond that during
decompression, so tiny requests could allocate disproportionate frontend
memory before any protobuf validation or quota charge.
- Handler-level decompression (Prometheus remote read/write v1+v2, Loki)
now enforces a hard 512 MiB decoded-size cap, checked before any output
buffer is allocated, and charges the decoded bytes to the shared
ServerMemoryLimiter, holding the permits for the lifetime of the
decompressed buffer.
- gRPC requests with transport compression reserve the configured
max_recv_message_size before tonic decompresses, so the decoding phase
is admitted against max_in_flight_write_bytes; the later per-message
charge is skipped to avoid double accounting.
- The HTTP memory-limit middleware keeps its upfront Content-Length charge
but now also accounts the bytes actually streamed beyond it, so chunked
requests and understated Content-Length headers no longer bypass the
aggregate quota.
- Routes that decompress request bodies via RequestDecompressionLayer
(InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards)
now charge the decompressed bytes as handlers consume them: the global
middleware marks Content-Encoding requests, and a route-local
accounting layer inside the decompression layer charges the decoded
stream. Plain requests are skipped to avoid double-counting.
* fix: address review comments on memory admission guard lifetimes and 429 mapping
- Retain the gRPC pre-decode reservation for the whole request: the
extensions holding the guard were dropped when the request was consumed
(into_inner), releasing the reservation while per-message charges
stayed skipped. Both the unary and streaming handlers now clone and
hold the reservation marker for the duration of request handling.
- Retain the HTTP body permits across the handler: the AccountedBody
wrapper and the request extensions are dropped once the extractors
finish collecting the body, before the handler is done with the decoded
data. Both accounting middlewares now keep their own accounting handle
alive across next.run(req).await.
- Return a ChargedBuffer from the Loki snappy decompressor so the
reservation outlives the decompressed bytes through the caller's
protobuf decoding, matching the Prometheus path.
- Map mid-stream quota exhaustion to 429 instead of the generic body
error (400): the accounting flags exhaustion and the middlewares
rewrite the extractor rejection, so clients can distinguish
backpressure from malformed input.
* fix: resolve the in-flight body acquisition before trying a new one
A parked acquisition in AccountedBody::charge always belongs to the
currently buffered frame, so it must be resolved before any new
acquisition is tried. Letting the fast-path try_acquire succeed while a
waiter is parked left the waiter alive, and its late completion would
then be credited with a later frame's byte count, under-reserving memory
relative to what was marked charged.
Adds a regression test that reproduces the misattribution: with the
buggy ordering the test delivers a body the quota cannot cover; with the
fix the over-quota frame waits for its own acquisition and times out.
---------
(cherry picked from commit 3131cdbcb6)
Signed-off-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
* fix(meta-srv): use NoTls for disabled and Unix socket Postgres KV backends
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* fix(meta-srv): treat Postgres config as unix socket only when every host is a socket
tokio-postgres dials hostaddr over TCP even when host is a socket path,
and a mixed host list with Require/VerifyFull TLS would otherwise end up
sending plaintext over TCP. is_unix_socket_url now parses via
tokio_postgres::Config and requires no hostaddr and all-Unix hosts.
Adds regression cases for the libpq keyword form, the user@ percent
encoded socket URL, and the mixed host / hostaddr negatives.
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* chore: remove dead code left behind by removed features
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: remove unused RouteInfoCorrupted error
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: allow customized time index unit for metric engine table
* test: provide query tests
* refactor: revert unnecessary change
* refactor: share timestamp unit conversions in api helper
Address review feedback on the time index unit changeset:
- Add shared timestamp_unit/timestamp_datatype helpers to api::helper
(the only crate that sees both proto ColumnDataType and TimeUnit due
to layering; common-time and datatypes have no greptime-proto dep).
This removes the ColumnDataType -> TimeUnit match duplicated between
operator's insert path and the OTLP logs path.
- Collapse the two TimeUnit <-> ValueData matches in
convert_timestamp_value_data by reusing api::helper::to_grpc_value
for the construction side.
- Note that convert_rows_time_unit rewrites the schema before the
values, so an overflow mid-batch leaves the request half-converted;
harmless because the error aborts the whole insert request.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: align time units per destination table and floor remote-read timestamps
Address review feedback on PR #9236:
- Align each metric insert request to the unit of the table it actually
targets: an existing logical table keeps its own unit (it may be bound
to a different physical table than the one selected by the request),
and only new tables use the selected physical table's unit. The
previous blanket conversion rewrote valid millisecond samples to the
selected physical table's unit and the engine rejected them.
Regression test: writing an existing millisecond logical table and a
new table in one request that selects a microsecond physical table.
- Remote read now floors narrowing timestamp conversions towards
negative infinity (div_euclid), consistent with
Timestamp::convert_to on the ingestion path; arrow's cast truncates
towards zero and returned -1ms for a stored -1001us. Widening
(second -> millisecond) keeps the exact arrow cast. Regression test:
a negative, non-aligned timestamp round-trips as -2ms.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf: fold time unit alignment into existing table lookups
Address review feedback on PR #9236:
- The per-destination unit alignment no longer runs its own pass of
table lookups: create_or_alter_tables_on_demand gains an
align_time_index_unit parameter (metric engine path only) and
converts each request inside the lookups it already performs —
existing tables to their own unit, new tables to the selected
physical table's. Default ingest paths now issue zero additional
catalog lookups compared to main; the separate alignment pass remains
only in the opt-in logical batcher pre-gate, next to the eligibility
check that already looks up the same tables.
- convert_rows_time_unit indexes the time index position directly
(validate_column_count_match guarantees row widths) instead of
Optional get_mut; the gate-side alignment validates widths itself.
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf: resolve the batcher time index guard once per write target
All batches of one remote write request share the same write target
(catalog, schema, physical table), so the batcher time index guard now
resolves each distinct target once instead of once per batch.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: support non-millisecond time index units in the logical batcher
Make the logical table batcher's bulk encode path unit-aware so physical
metric tables with a non-millisecond time index (e.g. TIMESTAMP(6)) can
use logical batching instead of falling back to the ordinary insert path.
- rows_to_aligned_record_batch builds the time index column in the
TARGET schema's unit, converting any timestamp encoding via
Timestamp::convert_to (flooring on narrowing, consistent with the
ordinary insert path).
- New tables created by the batcher use the selected physical table's
time index unit (resolved once per submit; a missing physical table
keeps the millisecond auto-create default).
- columns_taxonomy and the can_batch_metric_rows schema whitelist accept
any timestamp unit; the prometheus remote write v1/v2 batcher gates
and the OTLP pre-gate alignment are removed together with
Inserter::align_metric_row_inserts_time_unit, as the batcher now
converts internally.
Closes#9342
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
* fix: address review issue
* refactor: drop the OTLP pre-gate unit alignment made redundant by the bulk path
The main merge of #9236 (squash) resurrected the OTLP pre-gate
alignment and Inserter::align_metric_row_inserts_time_unit, which this
branch had removed. Drop them again:
- The pre-gate existed because the #9236-era bulk eligibility gate only
accepted millisecond schemas, so nanosecond-encoded OTLP requests had
to be converted before the check. This branch makes the bulk path
unit-aware (the gate accepts all time index units and batch alignment
converts each request to its destination's unit), so the pre-gate is
redundant and only added N+1 catalog lookups per batched request —
the very lookup-count overhead raised in the #9236 review.
- The per-destination unit semantics it implemented remain enforced in
the two paths that need them: the ordinary insert path
(create_or_alter_tables_on_demand converts inside its existing table
lookups) and the batched path (batch alignment resolves each
destination schema and converts to it).
test_otlp_logical_batcher_alignment (the test the pre-gate originally
fixed) and the mixed-physical-table regression both pass without it.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: cover OTLP batcher cross-physical fallback and nanosecond physical
Extend the logical batcher integration coverage for the cases
previously guarded by the removed OTLP pre-gate alignment:
- test_otlp_logical_batcher_fallback_for_cross_physical_destination:
with the batcher enabled, an OTLP request targeting an existing
logical table bound to another physical table must NOT enter the
batcher (the bulk eligibility check rejects the destination binding)
and the ordinary insert path must convert it to the destination's
unit (60s -> 60_000_000us).
- test_otlp_logical_batcher_non_millisecond_physical_table now covers
both microsecond and nanosecond physical tables (parameterized),
asserting batcher submissions and unit-precise stored values.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: validate per-batch physical bindings and avoid intermediate timestamp buffers
Address review feedback on PR #9346:
- accepts_bulk_destinations dedupes on (schema, table, selected physical)
instead of (schema, table): one request can select different physical
tables per batch (per-series x_greptime_physical_table labels), and the
old key let a second selection skip validation and flush rows through
the wrong physical's regions. Missing tables additionally reject
conflicting physical selections within the same request. Regression
test covers an existing destination, a missing destination, and a
consistent selection (which must still batch).
- The timestamp column builder appends each value directly into the
target-unit Arrow builder; values already in the target unit (the
unchanged millisecond fast path) are appended without conversion, so
the default millisecond physical pays no Timestamp construction or
intermediate Vec allocation.
- The non-millisecond batching tests assert the submit_build_and_align
counter, which increments on every batcher submission in both
acknowledgement modes, so a silent fallback to ordinary insertion
fails the tests instead of passing on stored values alone.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat(query): implement PromQL @ modifier on vector and matrix selectors
The planner previously dropped the `at` field of selectors (`at: _`),
so `some_metric @ 300` silently returned step-following values instead
of the samples anchored at the fixed timestamp.
Anchoring follows Prometheus's `setOffsetForAtModifier` + `refetch`
semantics: resolve the anchor (`@ <ts>`, `@ start()`, `@ end()`), apply
`offset` to the anchor, rewrite the selector offset to
`eval_start - anchor`, scan only the anchored window, then report the
same window at every evaluation step via a grid-wide-replay
`InstantManipulate`. `@ start()` / `@ end()` resolve against the
statement's evaluation range (`stmt_start` / `stmt_end`), which subquery
planning does not rewrite.
Timestamps before the Unix epoch are accepted; unrepresentable ones are
rejected with `AtModifierTimestampOutOfRange` instead of wrapping.
Selectors without `@` are planned exactly as before.
Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-2
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(promql): center predict_linear on the evaluation timestamp
predict_linear_impl used the window's last sample time as the evaluation
timestamp, and the UDF took only (ts_range, value_range, t) with no channel
for the current instant. After a range selector is folded for @ (or with an
offset) the same window is replayed at every step, so the prediction stayed
constant at the anchor's answer; even a plain window ended before the step
produced a stale value.
Give prom_predict_linear a 4th argument carrying the step's evaluation
instant (ms timestamp), derived in the planner from the row's time index
plus the offset the window was folded with (at_offset for @, offset_ms
otherwise, recorded on PromPlannerContext). The regression is centered on
that instant, matching Prometheus' use of enh.Ts. The mixed
float/native-histogram path forwards the same argument.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(query): tighten @ modifier planner and predict_linear
- range_fold_offset is always a concrete millisecond offset, not an
Option; the single-step replay helper no longer wraps an infallible
plan in Result.
- predict_linear's eval timestamp is always cast to Timestamp(ms) at the
call site, so the UDF drops its dead Int64 branch and the extra
func_name argument.
- Drop two redundant comparison cases from the at_modifier sqlness test
and fix the lookback window comment to the half-open (0s, 300s].
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): accept parse-time rejection of @ on Windows
`@ 1e16` is 10^19 milliseconds, beyond i64::MAX. A Unix SystemTime
holds it and the planner rejects the anchor it cannot represent, but a
Windows SystemTime tops out near 1.8e12 seconds, so the parser's
checked_add fails first and the same literal is rejected while parsing.
Accept either rejection path so the test passes on both platforms.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): avoid replaying label_join over rewritten series keys
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): narrow anchored range call promotion
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): address @ modifier review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): satisfy super import format check
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(query): address at modifier review follow-ups
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(log-store): add the enqueued acknowledgement mode to the object store WAL
Add `ack_mode` (`durable` by default, or `enqueued`) and the backlog
thresholds `max_unpersisted_bytes` and `max_unpersisted_age` to the
object store WAL config, validated by the datanode and the store.
In the `enqueued` mode `append_batch` returns on admission with the
entry ids assigned and the object is created in the background. At a
backlog threshold the next append is held back until an upload
completes. A transient create failure is repeated under the same
sequence with the same bytes; any conflicting object poisons the
store. `stop` uploads the backlog, or returns the error that dropped
it once stop began. `obsolete` clamps the watermark to the durable
entry id, and an id the store handed out needs no sequence floor.
Add `LogStore::wait_durable` with a default that returns at once. The
object store WAL answers it once the region is durable and indexed
through the entry id, and fails it after a backlog was lost.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): poison an enqueued store on permanent create failures
Repeat a failed create in the enqueued mode only when the storage error
is retryable; any other storage error poisons the store. A conflicting
object poisons an enqueued store without reading its epoch, so a failed
header read cannot turn the conflict into a retry.
A durability wait for an id above the highest id the store handed out
now waits for the handed-out ids of the region below it instead of
returning at once.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): poison on permanent create failures after stop begins
A create that fails with a storage error that is not retryable poisons
an enqueued store even after stop began; only a transient failure drops
the backlog without poisoning. Durability waiters whose callers stopped
waiting are pruned before a new waiter is queued. The backlog age test
no longer depends on a follow-up append finishing within the threshold.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): answer a durability wait once no earlier entry is pending
A durability wait now returns once the region holds no entry at or
below the target that is handed out but not durable, instead of waiting
for the largest id handed out to the region. A later object of the
region that is still being created no longer holds back a wait whose
target it does not cover.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): order the pending durability wait check after the actor
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(store-api): state that the default wait_durable keeps each store's guarantee
The default `LogStore::wait_durable` returns at once, which keeps each
log store's own acknowledgement guarantee; Raft Engine with
`sync_write = false` acknowledges before its periodic sync, so the
documentation no longer claims that every entry id a caller holds is
durable. The object store WAL configuration test now also serializes
the new options and reads them back.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(log-store): limit the acknowledgement guarantees to the durable mode
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(log-store): repeat enqueued creates that a retry layer marks persistent
An object store wrapped in the OpenDAL retry layer reports a temporary
error that outlasted its retries as persistent rather than temporary.
The enqueued mode now repeats a create after any storage error that is
not permanent, so a transient outage behind the retry layer no longer
poisons the store and drops the acknowledged backlog; after stop began
such a failure still drops the backlog without poisoning.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): cover a persistent create failure after stop begins
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
dennis zhuang
Weny Xu
discord9
Dragon Roar
Aum
Niladri Adhikary
fengyue-xve
zhaiAohan
Ning Sun
Aether
Yingwen
DeviousCardi
LFCandshuiyisong
fys
Mohd Quamar Tyagi
jeremyhi