* fix(promql): correct counter reset accumulation in rate windows
`prom_rate` and `prom_increase` reused the previous window's counter-reset
correction when the next window slid forward by exactly one sample, adding
the entering reset and subtracting the leaving one. Running a sum through
addition and subtraction does not restore the earlier terms in f64: a large
reset absorbs the smaller ones that must survive it, and an expired infinity
leaves a NaN that no later window can clear. `prom_delta` shares the code but
is not a counter function, so it never took that path.
Index the reset positions of the value array once instead, and reduce each
window over the resets it contains, in sample order. The result is
bit-identical to scanning the window directly, so windows keep the direct
reduction when they request fewer sample pairs than the input has.
Also sweep the query step in the rate benchmarks: the cost of the reset
correction depends on how much the windows overlap, which no existing case
varied.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* test(promql): cover counter reset precision over adjacent rate windows
The unit tests build the range windows directly, so they do not show that a
plain PromQL range query produces the window layout that lost the correction.
This case does: with a query step equal to the sample interval, `increase`
over the second window returns 1.333 before the fix and 2.667 after it.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* perf(promql): advance the counter reset bounds instead of searching
Locating a window's resets with two binary searches costs more than the
reduction it replaces once a series resets often enough for the searches to
get deep: on a 20k-sample counter resetting every 37 samples, stepping the
windows by one sample was 2.7x slower than the previous code, against 1.2x
for a counter that never resets.
Windows normally advance, so walk the bounds forward from the previous
window and only search when they move back. The cost then no longer depends
on the reset density.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* perf(promql): cut the per-window cost of the counter reset index
Two costs the index added showed up on a one-sample query step, where the
removed fast path used to answer each window with two comparisons.
Cache the two reset positions that bound the active slice. A window that only
advanced and reached neither of them covers the same resets as the previous
one, so the common case is four integer comparisons and no lookup at all.
Stop summing the requested sample pairs once they exceed one pass over the
values. The sum only decides which side of that comparison the input falls on,
and a query with a short lookback and a long step settles it after a few
windows instead of after every key.
Together these take the one-sample step from 25-32% slower than the previous
code down to 6-11%, measured as before / after / before to bound drift. No
other step value regresses, and a ten-sample step stays about 88% faster.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* fix(promql): accumulate counter resets into the running result
`prom_rate` and `prom_increase` summed a window's counter-reset corrections
on their own and added that sum to `last - first`. Prometheus folds each
reset into the running result instead, and so did this code before #7880.
The two are not interchangeable in f64: over samples `[1e16, 1, 0, 1]` the
isolated sum rounds `1e16 + 1.0` back to `1e16`, which then cancels against
the first sample and reports no increase at all, where folding the resets in
one at a time keeps the 1.0.
Restore the original order. The reset index accumulates into the result the
same way, so it still matches a direct scan of the window bit for bit, but a
window's contribution can no longer be cached as a standalone value and is
re-added from its own difference each time. The bounds are still cached, so
a window that did not cross a reset skips the lookup, and one that holds no
resets returns without touching the index at all.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* test(promql): note which reset boundaries the stride of one walks
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
---------
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* perf(servers): drop the output sort for Prometheus range queries
Matrix responses sort samples by timestamp and series by labels while
serializing, so the sort the PromQL planner puts at the root of a range
query plan never reaches the client.
Mark the query as not requiring output ordering in `do_range_query`, and
let the frontend strip the root sort (through projections, and only when
it carries no fetch) before execution. Instant queries are unchanged, so
`sort()` and `sort_desc()` keep their observable order.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* test(servers): pin the matrix response against input row order
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* test(servers): drop redundant response assertions in the ordering test
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* perf(servers): drop the output sort for gRPC gateway range queries
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
---------
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
Building a matrix response allocated one `String` per sample while the
record batches were scanned, then dropped it after the JSON body was
written. Keep the `f64` in `PromSampleValue::Number` instead and format
it with ryu while serializing, so no per-sample string is allocated.
`PromSampleValue::Text` keeps values parsed from a JSON body, so
deserializing and re-serializing a response is unchanged. Vector and
scalar results still expose `String`, since they hold a single sample.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
The scheduled (nightly) release version was created by appending
'-nightly-YYYYMMDD' to NEXT_RELEASE_VERSION as-is. When the Cargo.toml
version carries a pre-release extension (e.g. v1.3.0-alpha.1), this
produced invalid tags like 'v1.3.0-alpha.1-nightly-20260907', stacking
'nightly' on top of the 'alpha.1' pre-release.
Strip the pre-release extension first so 'nightly' itself becomes the
only pre-release extension: 'v1.3.0-alpha.1' -> 'v1.3.0-nightly-20260908'.
Stable versions are unaffected; nightly-build ('nightly-YYYYMMDD-sha'),
dev-build, tag push, and manual dispatch paths are unchanged.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows
Split the single TWCS trigger_file_num into per-window-state thresholds:
the active window keeps the existing trigger (default 4, legacy
compaction.twcs.trigger_file_num stays a compatible alias), while
inactive windows use a new trigger (default 2). Inactive windows
additionally fall back from balanced L0-only/L1-only candidates to a
progress-making unbalanced mixed candidate so historical windows can
converge; the active window retains the row/byte balance guards.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): bound inactive TWCS window convergence by rewrite budget
Inactive windows that cannot compact within one level previously either
stayed stuck (a threshold-qualified but unbalanced level returned no
candidate without trying any fallback) or fell back to a mixed merge
with no balance checks at all, which could rewrite a huge compacted file
to absorb tiny fresh files.
Inactive windows now converge progressively: threshold-qualified
balanced picks, sub-threshold balanced single-level picks, a mixed merge
whose total rewrite must fit in the output file budget, and finally an
L0-only merge without balance checks. Windows that qualify for none of
these are left uncompacted, bounding write amplification.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover TWCS window trigger options in alter_table_options sqlness case
Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and
compaction.twcs.inactive_window.trigger_file_num end to end, including
that setting the canonical active key removes the legacy
compaction.twcs.trigger_file_num alias from the table options.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): derive TWCS active window from the max-sequence file
The active window was determined by the max event-time window among
level-0 files. Between an L0 compaction removing its inputs and the next
flush landing, level 0 is empty, so the active window transiently became
None and every window fell back to the inactive rules - triggering
full-window convergence merges during ongoing ingestion whose outputs
are then superseded by new data.
Flush and compaction outputs both inherit the max input sequence, so the
file with the highest sequence across all levels always tracks the most
recent write. Use its window as the active window, falling back to the
previous L0-based rule when no file carries a sequence (legacy files).
The new helper deliberately computes window keys with the
assign_to_windows convention (truncate to seconds, then align up),
because the result is compared against window keys produced there; the
older ceil-based helper is kept unchanged for the legacy fallback path.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): add active-window L1 compaction safety trigger
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(compat): cover TWCS active window options
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): align TWCS window trigger validation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): resolve database TWCS trigger aliases
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): prioritize newer compaction windows
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): repick serial compaction outputs
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): configure inactive-window L1 trigger
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): prioritize TWCS compaction candidates
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): preserve TWCS trigger downgrade compatibility
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(sql): distinguish invalid database option values
Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(meta): explain TWCS legacy key compatibility
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify active window trigger field
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(options): normalize TWCS trigger aliases
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): ignore ineligible files for active window
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): mark explicit TWCS options as overrides
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): normalize zero compaction output size
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(options): validate database TWCS trigger values
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(store-api): collapse TWCS alias conflict condition
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: record time unit per series index file
The series index stores __series_min_ts/__series_max_ts as raw i64 in
the time index unit at write time, but the searcher built its range
predicates from the region's current unit, so files written before a
time index unit widening would be compared in the wrong unit.
Record the unit in the min/max ts fields' Arrow metadata when writing
and build the per-file time predicates from it when searching, so each
file is interpreted in the unit it was written with. The writer now
also requires a timestamp time index.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on series index time units
- split schema validation (validate_index_schema) from unit extraction
(index_time_unit), distinguishing missing vs unsupported unit metadata
in the errors instead of one misleading 'missing a valid metadata'
- encode the recorded unit with an explicit exhaustive match rather than
Debug formatting, so the on-disk encoding is reviewed next to its parser
- extract time_index_unit to drop the unwrap in series_index_schema and
the duplicated timestamp-time-index ensure in validate_metadata
- test one searcher reading files with different recorded units, and the
rejection of missing, unknown and mismatched units
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: reuse TimeUnit's Display form for the recorded unit string
common_time::timestamp::TimeUnit already implements Display with the
exact strings the series index records ("Second"/"Millisecond"/
"Microsecond"/"Nanosecond"), so drop the local time_unit_as_str
mapping and use it; the parse side stays local since no FromStr
counterpart exists anywhere yet.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: parse TimeUnit from its Display form in common-time
Add FromStr for common_time::timestamp::TimeUnit, accepting exactly the
Display form ("Second"/"Millisecond"/"Microsecond"/"Nanosecond") and
failing with a new UnsupportedTimeUnit error (InvalidArguments). The
series index now records and parses the unit with the common codec,
dropping its local parse_time_unit.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: parse TimeUnit case-insensitively
Lowercase the input before matching so "millisecond" and "MILLISECOND"
parse like "Millisecond"; the error still reports the original string.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: store series index min/max timestamps as native Timestamp columns
Replace the Int64 min/max columns plus 'time_unit' field metadata with
native Timestamp(unit) columns, so the unit rides on the datatype and
each file is interpreted in the unit it was written with naturally.
- series_index_schema types the columns from the time index unit; the
writer reinterprets the raw i64 series bounds in that type (arrow's
Int64->Timestamp cast reinterprets, it does not rescale)
- the searcher reads the unit from each file's column datatype, builds
Timestamp-typed predicates via datatypes' timestamp_to_scalar_value,
and reinterprets parquet INT64 statistics in the column type so
row-group pruning compares like-typed values
- index files whose min/max columns are not Timestamp (written before
this change) are rejected
- the pruning test now asserts time-range predicates prune row groups,
not just tag predicates
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: drop the TimeUnit string codec from common-time
With the unit carried by the Timestamp datatype, the FromStr impl and
UnsupportedTimeUnit error added for the field-metadata approach have no
consumer; remove them.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: fold index file validation into a single pass
The series index format is unreleased and unwired, so no compatibility
classes are needed: validate_index_schema checks all columns and
returns the min/max columns' unit directly, replacing the separate
index_time_unit extraction.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: carry timestamps through SeriesIndexRow
SeriesIndexRow and the aggregation path now hold common_time::Timestamp
instead of raw i64s: timestamp_values interprets the input column in the
writer's unit (rejecting a timestamp array whose unit differs, instead
of silently reinterpreting it), and rows_to_batch builds the native
Timestamp columns directly from the rows' units without an Int64 round
trip through arrow cast.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: require a timestamp time index column in series index input
The writer already refuses non-timestamp time indexes on the metadata
side, and its input batches always carry the region's ts column as a
timestamp array, so accepting plain Int64 columns only left a silent
unit-interpretation hole; reject them instead.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: rescale series index input timestamps into the file unit
The input array's unit is self-describing, so converting with
Timestamp::convert_to cannot mislabel values; a mismatch no longer
needs to be an error. Only a value that overflows the file's unit
fails the write. This also makes the writer ready to aggregate
old-unit batches after a time index widening.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: drop redundant unit checks in series index writer
The alter path flushes memtables before widening the region's time
index unit, so a writer never receives batches in the region's
previous unit. Reject a unit mismatch at the input boundary instead
of rescaling per value, and build the index batch in the writer's
recorded unit instead of re-deriving it from the schema and
re-checking every row.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(backport): label backport PRs with their version name
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(backport): restore multi-line PR body string
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(prometheus): align batch flush deadline with creation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(servers): sync pending worker submission with explicit ack
Replace the mpsc capacity() polling in the pending rows batcher deadline
test with a test-only WorkerCommand::Ack round trip. The FIFO channel
guarantees the worker has dequeued and processed the submission (and
anchored the flush deadline) before the test advances virtual time,
removing reliance on an implementation detail that can be flaky under
scheduling variance.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(servers): await pending worker flush results with bounded timeout
Replace the try_recv() yield-polling loop in the pending rows batcher
deadline test with a direct await bounded by tokio::time::timeout. Under
paused time the timeout auto-advances the clock and fires
deterministically, so the test fails reliably instead of intermittently
missing the result on slower CI or under contention.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): add series index catalog and lifecycle components
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat(mito2): restore series index catalogs on region open
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): simplify series index foundation and maintenance
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): separate series index purge task and simplify tests
Signed-off-by: evenyag <realevenyag@gmail.com>
* docs: defer experimental series index configuration examples
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat(mito): make series index maintenance interval configurable
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): correct series index cleanup on close and drop
Signed-off-by: evenyag <realevenyag@gmail.com>
* test(mito2): revert drop test changes
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update config API expectation for series index settings
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: use tokio unbounded channel for series index purger
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): simplify review test scope and clarify index config
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): reject non-flat batches in flat merge instead of panicking
- SortColumns::new becomes fallible try_new: batches missing the
flat-format internal columns (time index, __primary_key, __sequence)
at the fixed trailing positions now yield InvalidRecordBatch instead
of a downcast panic, completing the generic-schema gate that only
covered BatchBuilder output assembly. Document the flat-format input
contract on FlatMergeIterator/FlatMergeReader.
- Clarify why BatchBuilder's schema gate uses >= 3 columns when a real
flat-format schema always has at least 4.
- Add schema-structure tests: empty primary keys (tables without tags),
dictionary-encoded string tag columns with per-source dictionaries,
and graceful rejection of batches without internal columns.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* test(mito2): scan tables with various schemas through flat merge
Add an engine-level test that writes, flushes and scans regions without
tags (empty primary key) and with multiple string tags (dictionary-encoded
in the flat input schema), so the flat merge reader merges an SST with
the memtable on real schemas instead of hand-built batches.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor(mito2): use winner_tree dependency
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Update comments for FlatMergeIterator struct
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(cli): sanitize store_addrs in kvbackend build log
Close#7525. The CLI's kvbackend construction log was printing raw
store_addrs which could contain sensitive connection strings (e.g.
PostgreSQL DSNs with passwords).
Changes:
- Add sanitize_store_addrs() helper that reuses
common_meta::kv_backend::util::sanitize_connection_string(),
consistent with MetasrvOptions and StartCommand patterns.
- Replace raw store_addrs in the info! log with sanitized version.
- Add unit tests covering MySQL URLs, PostgreSQL DSNs, etcd addresses,
and empty store_addrs cases.
Signed-off-by: qiang_liu
Signed-off-by: qiang_liu <qiang_liu@trendmicro.com>
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>
* fix(cli): drop redundant sanitize tests per review
sanitize_connection_string in common_meta already covers MySQL URLs,
PostgreSQL DSNs and credential-free etcd addresses with its own tests.
The added tests only exercised a trivial map+collect wrapper, so remove
them per review nit.
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>
---------
Signed-off-by: qiang_liu
Signed-off-by: qiang_liu <qiang_liu@trendmicro.com>
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* feat(mito2): adapt bulk memtable encode bytes threshold to write buffer size
The default encode_bytes_threshold is now max(64MB,
min(global_write_buffer_size / 32, 512MB)) instead of a fixed 64MB, so
it scales with the memtable budget. GREPTIME_BULK_ENCODE_BYTES_THRESHOLD
and the per-region option still override the default.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): clarify binary units in bulk threshold test
The threshold test uses powers of 1024, so label its values as MiB and GiB instead of decimal MB and GB.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): resolve bulk encode threshold in memtable provider
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf: add scanbench partition diagnostics
Port benchmark source and documentation from c5dd5e6356a8903f1dcf493eb46dc318612dc953. Exclude the Mito metrics changes.
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: support scanbench query suites
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit def9c22069b0d29c611695a68ad1b31110c845cb)
Signed-off-by: evenyag <realevenyag@gmail.com>
* docs: align scanbench port with existing engine metrics
Remove documentation and fixture references to unported Mito metrics. Enable dev-tools in the build example and remove the obsolete force-flat-format option.
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: re-scan stream-backed tables in recursive CTEs
A recursive CTE re-executes its recursive term on every iteration, but
DfTableProviderAdapter hands StreamScanAdapter a single-use stream built at
planning time. The second iteration failed with "Stream already exhausted"
for every table served through DataSource::get_stream — information_schema,
pg_catalog, the computed entity-graph tables and numbers.
Keep that stream for the first execution and open a new one over the same
scan request for later executions.
Closes#9037
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: drop redundant binding in stream factory
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* Implement `/query-regression` command handling and admission workflow
- Add `query-regression-slash.py` script for processing `/query-regression` commands in PR comments, validating case arguments, and checking permissions.
- Update `checks.yml` to include tests for the new slash command functionality.
- Modify `query-regression-comment.yml` to trigger on the new `Query Regression Command` workflow.
- Create `query-regression-slash.yml` to handle the dispatched command, validate allowlist and permissions, and initiate the regression workflow.
- Enhance `query-regression.yml` to support additional inputs for PR admission and SHA verification.
- Introduce `slash-command-dispatch.yml` to parse and dispatch commands from PR comments.
- Document the new command admission process in `AGENTS.md` and `README.md`.
- Add unit tests in `test_query_regression_slash.py` to cover command parsing and admission logic.
* refactor: enhance query-regression command handling with comment validation and identity checks
* feat: implement admission identity handling for query regression workflows
* refactor: update PR admission logic in query regression workflow
* refactor: update token usage in slash command dispatch and README for clarity
* test: add cases for handling re-run failed jobs and stale runner artifacts
* refactor: improve repository metadata handling in query regression scripts
* chore: enable overwrite for artifact uploads to handle re-run failed jobs
* chore: enable overwrite for query regression admission uploads
* feat: enhance query-regression admission with HMAC signing and verification
- Introduced HMAC signing for admission markers in query-regression workflows to ensure integrity and authenticity.
- Updated `query-regression-comment.test.cjs` to include tests for signing and verifying admission markers.
- Modified `query-regression-slash.py` to handle admission marker signing and verification, including checks for dispatch sender and head SHA consistency.
- Enhanced workflows to securely manage admission markers and HMAC secrets, ensuring they are not exposed to untrusted contexts.
- Improved documentation to clarify the admission process and the role of HMAC in securing the workflow.
* test: add case to find newly posted marker among newer comments
* test: add case to verify multiline output handling in write_outputs function
* perf(mito2): optimize flat merge heap and primary-key interleave
Replace the per-row BinaryHeap pop/push cycle in FlatMerge with an
in-place root mutation plus a single sift-down repair on a custom
RootHeap, keeping the cold heap and direct-batch fast path unchanged.
Fallible or awaiting batch transitions move the hot node out of the
heap first, preserving error and cancellation semantics.
Exploit the globally sorted merge output to build the internal
Dictionary<UInt32, Binary> primary-key column with a one-pass ordered
gather: append a Binary value only when the PK changes and reuse the
current key for adjacent equal PKs, bypassing Arrow dictionary masks,
hash interning and key remapping. Non-PK columns still use Arrow
interleave.
Also cache the current primary-key byte range in RowCursor to avoid
repeated dictionary range decoding during comparisons, and add a
setup-free Criterion benchmark with exact output-row assertions.
32-way/1 row-per-series/40-tag improves 955.79ms -> 562.36ms (-41.2%);
0-tag -39.5%, 64 rows/series -79.8%, 8-way -56.1%, single-iterator
control +0.3%.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): add rows-per-series sweep to flat merge bench
Add 32-way/40-tag shapes for 1, 10, 100, 1000 and 10000 rows per
series, and allow FLAT_MERGE_BENCH_SHAPE to match shape name prefixes
so the whole sweep can run in one invocation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): add oracle-based correctness tests for RootHeap
Drive RootHeap and a std BinaryHeap oracle with the same seeded op
sequence (push / pop / mutate-root + repair) and assert peek, len,
best_child and the full drain order after every operation. A second
run with a tiny value range makes duplicates dominate, covering the
equal-key branches of sift_up/sift_down and best_child.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): replace hot heap with a tournament tree in flat merge
Replace the hot RootHeap with a fixed-capacity tournament (winner) tree
over per-node slots: every internal node caches the champion of its
subtree, so advancing the winner only replays the ~log2(k) nodes on its
leaf-to-root path with one compare per level, instead of the heap's
two-compares-per-level sift that also re-compares the same node pairs
on every row.
Two fast paths keep dense shapes at O(1) per row:
- champion retention: after mutating the winner in place, skip the
replay entirely when it still beats the runner-up (its path caches
are unchanged by construction);
- a second-best slot cache, invalidated on any structural change, so
the retention check costs a single compare without walking the tree.
The cold heap, hot/cold overlap window, direct-batch fast path and the
remove-before-fallible-fetch batch transition semantics are unchanged.
Vs the RootHeap version: 1rps/32way/40tag -19.7%, 0tag -34.4%,
8way -15.9%, 64rps -30.5%, sweep 10/100/1000/10000rps -29~32%;
vs the original BinaryHeap baseline the main shape is -52.8%.
The single-iterator control is +8% (+50ns one-time construction
allocation, no merge work).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): support generic schemas in flat merge
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): satisfy clippy in flat merge benchmark
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): cache flat merge primary key index
Compute the internal primary-key column index once when constructing BatchBuilder and reuse it for every output batch. Preserve the column-name gate for generic schemas.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): benchmark high-fan-in flat merges
Add sparse 64, 128, 256, and 512-way merge shapes while keeping the total input fixed at 3.2 million rows. Compared with the merge-base heap implementation, median time improves by 56.0%, 60.8%, 55.6%, and 56.1%, respectively.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: match system schema names case-insensitively
Database names that arrive over a protocol (the MySQL handshake and
COM_INIT_DB, the Postgres startup parameter, the HTTP `db` parameter, the
gRPC dbname header) never reach the SQL parser, which is what lowercases
unquoted identifiers. Since #8062 stopped lowercasing them wholesale,
connecting to `INFORMATION_SCHEMA` in any spelling but the canonical one
fails with "Unknown database" -- including the `USE <db>` that a MySQL
client turns into COM_INIT_DB.
Fold only system schema names to their canonical spelling, so user schema
names keep the case they were created with. `is_reserved_schema_name` uses
the same match, otherwise a quoted `CREATE DATABASE "INFORMATION_SCHEMA"`
creates a schema shadowed by the system one.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: hoist system schema names into a const
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: add backport workflow to create backport PRs from backport labels
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci: document backport labels in PR template and AGENTS.md
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(pipeline): coalesce concurrent pipeline cache misses
The pipeline cache reads with a plain `moka::sync::Cache::get` and falls
through to a distributed query on a miss, so when the 10s TTL expires every
in-flight write request on a frontend issues its own scan of the single-region
`greptime_private.pipelines` table. Concurrent scans per expiry scale with
write QPS, and every frontend's burst lands on the same datanode. A user
running high-throughput ingestion through a pipeline saw that datanode
overloaded.
Switch to `moka::future::Cache::try_get_with` so concurrent misses on the same
key share one loader. This requires a single-key lookup, so cache entries are
now keyed by the requested schema rather than the schema the pipeline is stored
under; resolving a request to a stored schema stays in the loader, which is the
authoritative path and already handles the empty-schema and multi-schema cases.
A lookup for a schema not yet cached costs one extra read, now protected from
amplification by the coalescing it enables.
`remove_cache` previously only walked the compiled-pipeline cache, so an entry
populated by `get_pipeline_str` alone (the pipeline read API) survived deletion
until it expired. It now walks all three caches.
Also make the TTL configurable as `pipeline.cache_ttl`, default unchanged at
10s. The TTL is what propagates a pipeline change to other frontends, so
raising it trades staleness for fewer reads.
Refs #9021
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(pipeline): restore cross-schema semantics broken by the new cache key
Keying cache entries by the requested schema dropped two behaviours that the
previous stored-schema key provided for free.
Creating a new version only wrote the creating request's schema, so another
schema on the same frontend kept serving its cached `latest` — an older
version — until the entry expired. Since the whole point of making the TTL
configurable is to let operators raise it, that window is not bounded by
anything useful. Creation now invalidates every schema's `latest` alias for
that name before priming the cache, leaving the version-pinned keys alone.
The failover cache lost its reach across schemas the same way: a global
pipeline (stored under the empty schema) loaded by schema A was cached under
`A`, so schema B using it for the first time while the pipeline table was down
missed and failed ingestion. The failover cache has no loader and so is not
subject to the single-key model of `try_get_with`; it keeps the stored-schema
key and the empty-schema-first resolution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(pipeline): drop cache priming on create and fold the sweep helpers
Priming the cache on create saved one read on a low-frequency operation and
cost a concept: entries were written under the creating request's schema while
`PipelineContent.schema` said empty, so the two schemas in play disagreed.
Invalidating the `latest` aliases is required regardless — that is what makes
a new version visible to other schemas — so dropping the priming loses only
the saved read, which coalescing now protects anyway. `insert_and_compile` no
longer needs the caller's schema.
`remove_cache` and the create-time invalidation collapse into one
`invalidate(name, version)`; `None` sweeps only the `latest` aliases, which is
exactly what creation wants. That leaves `invalidate_by_suffixes` and
`cache_keys` with a single caller each, so both are inlined.
Drop the `PipelineOptions` humantime test: `load_config_test` loads both
example TOMLs, which now carry `cache_ttl = "10s"`, and would fail the same
way if the serde attribute were lost. The `toml` dev-dependency goes with it.
The two invalidation tests are now checked to be orthogonal: removing the
version suffix fails only the delete test, and sweeping just the compiled
cache fails both.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(pipeline): keep failover populated across a create
The `latest` sweep on create clears the failover cache along with the loaded
ones, and after dropping the priming there was nothing writing it back. An
outage between the create and the first read-back left neither `latest` nor the
explicit version with anything to fall back on, failing ingestion — worse than
before, since the previous version's failover entry was swept too.
Creation now goes through `PipelineCache::on_pipeline_created`, which pairs the
sweep with a failover write of the new empty-schema definition. The two must
happen together, so they live behind one method rather than at the call site.
Also commit the Cargo.lock entry for the dropped `toml` dev-dependency, and
trim the comments added over the last few commits down to what the code does
not already say.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
WenyXu
fys
dennis zhuang
discord9
Lei, HUANG
Ning Sun
LFCandgreptimedb-ci
shuiyisong
Mohd Quamar Tyagi
Yingwen
Qiang-Liuanddennis zhuang
sun
Dhruv Vaishnav
XuanwoandWenyXu
localhost
liyang