* fix(mysql): strip leading comments before the federated statement filter
JDBC clients prefix every statement with a comment. DataGrip sends
`/* ApplicationName=DataGrip <version> */` in front of each one, and every
pattern in the federated filter is anchored with `^`, so the prefix makes all
of them miss.
Two failures follow. `SET TRANSACTION READ WRITE` reaches the SQL parser and is
rejected. Worse, the DBeaver-specific entries such as
`^(/\* ApplicationName=(.*)SELECT @@(.*))` do match DataGrip's prefix, so
`SELECT @@GLOBAL.event_scheduler` and `SHOW VARIABLES LIKE ...` are absorbed
into a zero-column output, which the MySQL writer sends as an OK packet. The
JDBC driver reports that the statement returned no cursor.
Strip leading whitespace and comments before matching, and drop the
ApplicationName-specific entries the stripping makes redundant. The three that
had no unprefixed counterpart (`SHOW PLUGINS`, `SHOW ENGINES`, `SHOW @@...`)
keep their behaviour as plain patterns.
Also covers gaps found while probing the same path:
- `BEGIN` is absorbed like `START TRANSACTION`/`COMMIT`/`ROLLBACK`.
- `SHOW [GLOBAL|SESSION|LOCAL] VARIABLES|STATUS` parses; the scope is ignored
because GreptimeDB keeps no global/session split.
- `USER()`, `CURRENT_USER()`, `SYSTEM_USER()` and `SCHEMA()` are registered.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mysql): keep executable comments, reject multi-statement absorption
Review follow-up on the comment stripping, plus the remaining MySQL
compatibility gaps from #9155.
`/*!...*/` is an executable comment: mysqldump emits its initialization as
`/*!40101 SET NAMES ... */`, and the patterns match those verbatim. Stripping
it left an empty statement, so nothing matched and the original SQL reached the
parser, which rejects `SetNames` and `MultipleAssignments`. Leave executable
comments in place.
Absorbing a request also has to stop at a statement boundary, because every
pattern ends in `(.*)`. `BEGIN; INSERT INTO t VALUES (1)` used to fail on the
unsupported `BEGIN`; once `BEGIN` became absorbable the whole request would
report success and write nothing. A request is now scanned for a second
statement and handed to the query engine if it has one. The scan skips plain
comments and string literals so a `;` inside either is not a boundary, and
treats a `/*!...*/` that is not the request itself as a statement, since it
carries SQL.
`check()` now dispatches on the leading statement keyword, so INSERT, UPDATE,
CREATE and ordinary SELECTs run no regex at all — this replaces the narrower
hand-rolled INSERT shortcut. The statement scan runs only for a request the
patterns already matched, which is always a short one.
New compatibility surface:
- `SELECT CURRENT_USER|SESSION_USER|SYSTEM_USER|USER` without parentheses, and
`SELECT @var`, are answered here rather than in the parser. Both are anchored
to the whole statement, so `SELECT user FROM t` still reads the column.
- `information_schema.plugins`, `user_privileges` and `processlist` are
registered as empty tables, like the other MySQL-shape tables around them.
Sessions are reported through `information_schema.process_list` and
`SHOW PROCESSLIST`; `processlist` carries the column shape only.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(mito2): reconcile series indexes in background
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): clean up series indexes published during region drop
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(config): align series index examples with upstream enable flag
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): run series index tasks on compaction runtime
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): deliver complete WAL entries without waiting for next input
Entry completeness is self-contained via Entry::is_complete(); no
lookahead to the next entry is needed. Decode and yield complete entries
immediately in both the log-store reader and the entry distributor.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): simplify live WAL entry fixtures
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix: use join_dir for series index config path
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: configure series indexes with an enable flag
Signed-off-by: evenyag <realevenyag@gmail.com>
* docs: omit experimental series index from example configs
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: preserve legacy cache cleanup path behavior
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: remove trivial path joining tests
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: isolate worker group WAL directories on Windows
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* perf(servers): group Prometheus response rows by label runs
Query output tends to be clustered by series, but a matrix response read
the same label values out of the tag columns once per row, and allocated
a key vector per row to look the series up.
Use `arrow::compute::partition` to find the runs of rows that share their
labels and build the series key once per run. The key buffer is hoisted
out of the row loop and handed to the map only when the series is new,
through the raw entry API so the key is hashed once either way.
Partitioning does not pay off when rows are not clustered, so a few
adjacent row pairs are probed first to pick between the run path and the
row-by-row path. Both paths produce the same series.
Drop the per-row "same labels as the previous row" check from #8815. Runs
cover the clustered case it was written for, and it now costs more than
it saves: 10% on a result with one row per series, 1-3% on clustered ones.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* refactor(servers): find label runs with cmp::distinct
`arrow::compute::partition` computes the same ranges on the same kernel,
but its contract takes lexicographically sorted columns, and query output
is not sorted: range queries run without the plan's output sort since
#9090, `sort`/`topk` order by value, and the tag column order in the
schema does not have to match any sort key. An implementation that
exploited the precondition would merge `a, b, a` into one run and
attribute one series' samples to another, without failing.
`cmp::distinct` is element-wise, so it holds for any row order, and its
null handling is the one a series key needs: a null label and an empty
one are distinct, two nulls are not. Building the ranges from the
boundary bitmask also folds away the `tag_columns.is_empty()` case, since
no columns means no boundaries means a single run.
Same kernel, so the benchmark does not move: -1.4% to +1.4% across shapes
with no consistent sign, against +-3% run-to-run drift.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
---------
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
`PlanRewriter::should_expand` encoded every node it ascended past to
Substrait. Each call clones the plan, runs an analyzer over it and builds
a fresh `SessionState` with all default features, so a deep PromQL plan
pays that cost once per level.
Encode the root once in `PlanRewriter::new` and skip the per-node call
when it succeeds. `to_substrait_plan` recurses from the root to the
leaves, and `should_expand` only receives sub-trees that `f_down` pushed
on the stack untouched, so a root that encodes proves the whole descent
encodes. A root that does not encode falls back to the per-node check,
which is what locates the node that has to stay on the frontend.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): treat NULL field values as absent samples in range functions
Range functions read the value column through `Float64Array::values()`,
which returns the raw buffer and ignores the null bitmap. A NULL field
value means the series has no sample at that timestamp, so the padding
under a null slot (0.0 in practice) was counted as a real sample.
`rate`, `increase`, `delta`, `changes`, `resets`, `idelta`, `irate`,
`quantile_over_time` and `avg_over_time` now work on samples instead of
slots. `stddev_over_time` and `stdvar_over_time` used to panic on a NULL
slot, and tokio swallowed the panic so the query returned success with
that series missing.
Null handling is gated on `null_count() == 0` over the whole backing
array, checked once per batch, so tables without NULLs keep the existing
code path. `deriv` and `predict_linear` already had this guard through
`linear_regression_slices` and are untouched.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep other fields when one has no sample in the window
Review follow-up. Two things surfaced once range functions started
returning NULL for a window without samples.
The filter after a function call required every field column to be
non-NULL, so on a multi-field table one field with no samples in a
window would drop the other fields' results with it. It now keeps a row
when any field has a sample, which is the shape a selector already
emits. On a single field column the two predicates are identical.
`quantile_over_time` returned NaN rather than NULL for a window without
samples, so the row survived that filter. Prometheus returns an empty
vector there, so the emptiness check now sits in the range UDF; the
shared quantile kernel still yields NaN for an empty slice, matching
upstream's `quantile()` helper that `quantile_aggr` depends on.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(promql): correct two comments on the null handling
The planner one did not say why `preserve_any_value` is hardcoded at
that call site, which is the question a reader arrives with. The
`quantile_over_time` one described the empty-window behaviour while
sitting on the `has_nulls` line, and that behaviour had moved into
`window_quantile`.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): clamp extrapolation before snapping a counter to zero
Prometheus clamps `durationToStart` to half an average interval once the
first sample is past the extrapolation threshold, and only then lets the
counter zero-snap shorten it further, so the snap can never lengthen the
leading extrapolation. Running the snap first let it rescue a duration
the clamp should have cut, and `rate` and `increase` over-extrapolated
to the left. For samples 1@0s and 2@1s in a 4s window at 1s, upstream
gives 0.375 and this returned 0.5.
`extrapolation_matches_prometheus_on_seeded_windows` carries a
line-by-line port of upstream `extrapolatedRate` as an oracle and diffs
it against the UDF over seeded windows, so the order stays pinned.
`factor` also picked up upstream's guard against a zero sampled
interval, which previously divided by zero.
Two sqlness results move, both verified against the upstream algorithm.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): fold range_presence_null into null_samples
#9104 landed its own NULL-sample case whose data is the same series this
one already used: one host with interior NULLs, one host with nothing
but NULLs. Keeping both means two files asserting the same semantics on
the same rows.
The merged case keeps every query from both, so the presence functions
still cover the trailing-NULL window, the count of two, the empty
left-open window at t=7, and the all-NULL windows.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): look up row group stats by parquet leaf index for nested columns
On flat-format tables a logical column can expand to multiple parquet
leaf columns (e.g. a JSON2 struct stores the remainder and one leaf per
promoted path). ParquetFlat used the logical column index in the SST
schema directly as the leaf index when reading row group statistics, so
min/max/null stats of every column after a nested column were read from
wrong leaves.
When the misplaced leaf held order-compatible statistics (e.g. a small
Int64 JSON path vs. the timestamp window predicate), min-max pruning
dropped whole row groups by mistake. SWCS compaction reads inputs with a
time window predicate, so it silently lost all rows of such files; plain
queries with time-range predicates were affected as well.
Map each column to its first parquet leaf column and report NoStats for
columns with multiple leaves, which makes pruning conservative for them.
Add a unit test and a sqlness regression case that reproduces the data
loss on the unfixed binary.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): skip nested root stats and correct SWCS regression baseline
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): resolve statistics leaves for primary-key SST readers
Resolve scalar roots against the actual Parquet schema in shared statistics helpers, covering both flat and primary-key readers. Remove flat-side translation to avoid mapping twice and align encoded primary-key statistics as well. Cover dense flat, legacy dense and sparse layouts with statistics and time-pruning regressions.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): distinguish known null counts from unknown statistics
Assert validity before reading timestamp null counts and add a nullable scalar after the nested root with a known nonzero count. Exercise the assertions for flat and primary-key SST layouts.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): check JSON2 time pruning before SWCS compaction
Query the first time window immediately after FLUSH to cover predicate reads on flush-written SSTs independently of compaction outputs. Regenerate the sqlness expectation and retain the post-compaction checks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): validate all business columns after JSON2 SWCS
Expand the final regression query to all eight business columns so the generated expectation verifies complete rows, including tags and scalar fields, after repeated compaction.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): preserve mixed JSON2 types during compaction
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito): assert all restored JSON2 rows in unordered merge test
The regression test for aligning JSON2 layouts across unordered bulk
parts only round-tripped the first part's `a` values. A merge that
dropped or corrupted the second source's `b` values would still pass.
Assert the merged batch has four rows and that the restored values of
the second part (`{"b": 3}`, `{"b": 4}`) survive the merge, so the
test covers both opaque sources as its comment claims.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(pprof): switch CPU profiler to framehop unwinder
Replace the default libgcc-based unwinder in pprof-rs with the
framehop unwinder, which is designed to be async-signal-safe:
- framehop performs no heap allocation during unwinding
(MustNotAllocateDuringUnwind)
- It handles prologue/epilogue interruption correctly
- It falls back to frame-pointer unwinding when CFI is unavailable
- It does not depend on libgcc's unwind implementation, which is
documented as not signal-safe (see tikv/pprof-rs#36)
Bump pprof from 0.14 to 0.15 in all three consumers (common-pprof,
cmd, servers) to unify on a single version. pprof 0.15 also replaces
parking_lot with spin-rs to avoid a potential profiler deadlock (#268).
This addresses the libgcc_s.so.1 #GP crash observed in production
when CPU profiling is active, by eliminating the libgcc unwinder
from the signal handler path entirely.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(pprof): gate framehop-unwinder to supported targets
framehop-unwinder is only available on x86_64/aarch64 Linux/macOS.
Enabling it unconditionally for all Unix targets breaks the build on
riscv64 and other platforms: pprof disables its backtrace-rs fallback
when framehop-unwinder is set, but the framehop module is not compiled
on unsupported targets, leaving no TraceImpl implementation.
Split the pprof dependency: the base target.'cfg(unix)' block carries
the common features (flamegraph, prost-codec, protobuf), and a separate
target block adds framehop-unwinder only on supported targets.
Cargo unions features from both blocks on matching targets, so x86_64
and aarch64 Linux/macOS get the full feature set while other Unix
targets fall back to the default backtrace-rs implementation.
Addresses review comment discussion_r3987313632.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): add series index planning and builders
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): track window coverage and separate index builds
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): normalize index inputs and bound SST window expansion
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): simplify series index source summaries
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): assign series index deduplication TODO
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): initialize skip_wal in series index tests
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(ci): check Windows test targets before merge
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): use standard Windows runner for checks
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): omit dashboard assets from Windows checks
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix
Upgrade tikv-jemallocator / tikv-jemalloc-ctl / tikv-jemalloc-sys from
0.6 to 0.7, which embeds jemalloc 5.3.1 (includes a056c20d 'Handle
tcache init failures gracefully').
On top of that, patch tikv-jemalloc-sys to the GreptimeTeam fork that
adds the remaining upstream fix 54f22c83 'Initialize TSD tcache before
enabling it' (GreptimeTeam/jemalloc#1, GreptimeTeam/jemallocator#1).
Without the ordering fix, a reentrant allocation during TSD bootstrap
(e.g. heap-profiling prof_tdata init / sampled backtrace when prof:true
is active) can observe an enabled-but-uninitialized tcache, corrupting
per-thread tcache metadata and crashing the process in arena_stats_merge,
calloc, or the libgcc unwinder.
The patch is pinned by rev and should be removed once tikv/jemallocator
ships a jemalloc snapshot that includes 54f22c83.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump tikv-jemalloc-sys patch rev to merged release-5.3.1
GreptimeTeam/jemalloc#1 has been merged; point the patch at the
jemallocator commit referencing the merge commit on release-5.3.1.
Jemalloc source content is unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main
GreptimeTeam/jemallocator#1 has been merged; reference the merge
commit e1846d8c on main instead of the PR head branch. Jemalloc
source content is unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(servers): bump tikv-jemallocator dev-dependency to 0.7
main added a target.'cfg(not(windows))'.dev-dependencies entry on
tikv-jemallocator 0.6 for servers after this branch diverged. On the
merge ref it pulled tikv-jemalloc-sys 0.6 from crates.io, which
conflicts with the patched 0.7 (links = "jemalloc" may only appear
once in the dependency graph), failing version selection in CI.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(promql): correct counter reset accumulation in rate windows
`prom_rate` and `prom_increase` reused the previous window's counter-reset
correction when the next window slid forward by exactly one sample, adding
the entering reset and subtracting the leaving one. Running a sum through
addition and subtraction does not restore the earlier terms in f64: a large
reset absorbs the smaller ones that must survive it, and an expired infinity
leaves a NaN that no later window can clear. `prom_delta` shares the code but
is not a counter function, so it never took that path.
Index the reset positions of the value array once instead, and reduce each
window over the resets it contains, in sample order. The result is
bit-identical to scanning the window directly, so windows keep the direct
reduction when they request fewer sample pairs than the input has.
Also sweep the query step in the rate benchmarks: the cost of the reset
correction depends on how much the windows overlap, which no existing case
varied.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* test(promql): cover counter reset precision over adjacent rate windows
The unit tests build the range windows directly, so they do not show that a
plain PromQL range query produces the window layout that lost the correction.
This case does: with a query step equal to the sample interval, `increase`
over the second window returns 1.333 before the fix and 2.667 after it.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* perf(promql): advance the counter reset bounds instead of searching
Locating a window's resets with two binary searches costs more than the
reduction it replaces once a series resets often enough for the searches to
get deep: on a 20k-sample counter resetting every 37 samples, stepping the
windows by one sample was 2.7x slower than the previous code, against 1.2x
for a counter that never resets.
Windows normally advance, so walk the bounds forward from the previous
window and only search when they move back. The cost then no longer depends
on the reset density.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* perf(promql): cut the per-window cost of the counter reset index
Two costs the index added showed up on a one-sample query step, where the
removed fast path used to answer each window with two comparisons.
Cache the two reset positions that bound the active slice. A window that only
advanced and reached neither of them covers the same resets as the previous
one, so the common case is four integer comparisons and no lookup at all.
Stop summing the requested sample pairs once they exceed one pass over the
values. The sum only decides which side of that comparison the input falls on,
and a query with a short lookback and a long step settles it after a few
windows instead of after every key.
Together these take the one-sample step from 25-32% slower than the previous
code down to 6-11%, measured as before / after / before to bound drift. No
other step value regresses, and a ten-sample step stays about 88% faster.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* fix(promql): accumulate counter resets into the running result
`prom_rate` and `prom_increase` summed a window's counter-reset corrections
on their own and added that sum to `last - first`. Prometheus folds each
reset into the running result instead, and so did this code before #7880.
The two are not interchangeable in f64: over samples `[1e16, 1, 0, 1]` the
isolated sum rounds `1e16 + 1.0` back to `1e16`, which then cancels against
the first sample and reports no increase at all, where folding the resets in
one at a time keeps the 1.0.
Restore the original order. The reset index accumulates into the result the
same way, so it still matches a direct scan of the window bit for bit, but a
window's contribution can no longer be cached as a standalone value and is
re-added from its own difference each time. The bounds are still cached, so
a window that did not cross a reset skips the lookup, and one that holds no
resets returns without touching the index at all.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* test(promql): note which reset boundaries the stride of one walks
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
---------
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* perf(servers): drop the output sort for Prometheus range queries
Matrix responses sort samples by timestamp and series by labels while
serializing, so the sort the PromQL planner puts at the root of a range
query plan never reaches the client.
Mark the query as not requiring output ordering in `do_range_query`, and
let the frontend strip the root sort (through projections, and only when
it carries no fetch) before execution. Instant queries are unchanged, so
`sort()` and `sort_desc()` keep their observable order.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* test(servers): pin the matrix response against input row order
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* test(servers): drop redundant response assertions in the ordering test
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* perf(servers): drop the output sort for gRPC gateway range queries
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
---------
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
Building a matrix response allocated one `String` per sample while the
record batches were scanned, then dropped it after the JSON body was
written. Keep the `f64` in `PromSampleValue::Number` instead and format
it with ryu while serializing, so no per-sample string is allocated.
`PromSampleValue::Text` keeps values parsed from a JSON body, so
deserializing and re-serializing a response is unchanged. Vector and
scalar results still expose `String`, since they hold a single sample.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows
Split the single TWCS trigger_file_num into per-window-state thresholds:
the active window keeps the existing trigger (default 4, legacy
compaction.twcs.trigger_file_num stays a compatible alias), while
inactive windows use a new trigger (default 2). Inactive windows
additionally fall back from balanced L0-only/L1-only candidates to a
progress-making unbalanced mixed candidate so historical windows can
converge; the active window retains the row/byte balance guards.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): bound inactive TWCS window convergence by rewrite budget
Inactive windows that cannot compact within one level previously either
stayed stuck (a threshold-qualified but unbalanced level returned no
candidate without trying any fallback) or fell back to a mixed merge
with no balance checks at all, which could rewrite a huge compacted file
to absorb tiny fresh files.
Inactive windows now converge progressively: threshold-qualified
balanced picks, sub-threshold balanced single-level picks, a mixed merge
whose total rewrite must fit in the output file budget, and finally an
L0-only merge without balance checks. Windows that qualify for none of
these are left uncompacted, bounding write amplification.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover TWCS window trigger options in alter_table_options sqlness case
Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and
compaction.twcs.inactive_window.trigger_file_num end to end, including
that setting the canonical active key removes the legacy
compaction.twcs.trigger_file_num alias from the table options.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): derive TWCS active window from the max-sequence file
The active window was determined by the max event-time window among
level-0 files. Between an L0 compaction removing its inputs and the next
flush landing, level 0 is empty, so the active window transiently became
None and every window fell back to the inactive rules - triggering
full-window convergence merges during ongoing ingestion whose outputs
are then superseded by new data.
Flush and compaction outputs both inherit the max input sequence, so the
file with the highest sequence across all levels always tracks the most
recent write. Use its window as the active window, falling back to the
previous L0-based rule when no file carries a sequence (legacy files).
The new helper deliberately computes window keys with the
assign_to_windows convention (truncate to seconds, then align up),
because the result is compared against window keys produced there; the
older ceil-based helper is kept unchanged for the legacy fallback path.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): add active-window L1 compaction safety trigger
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(compat): cover TWCS active window options
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): align TWCS window trigger validation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): resolve database TWCS trigger aliases
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): prioritize newer compaction windows
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): repick serial compaction outputs
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): configure inactive-window L1 trigger
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): prioritize TWCS compaction candidates
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): preserve TWCS trigger downgrade compatibility
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(sql): distinguish invalid database option values
Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(meta): explain TWCS legacy key compatibility
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify active window trigger field
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(options): normalize TWCS trigger aliases
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): ignore ineligible files for active window
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): mark explicit TWCS options as overrides
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): normalize zero compaction output size
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(options): validate database TWCS trigger values
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(store-api): collapse TWCS alias conflict condition
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: record time unit per series index file
The series index stores __series_min_ts/__series_max_ts as raw i64 in
the time index unit at write time, but the searcher built its range
predicates from the region's current unit, so files written before a
time index unit widening would be compared in the wrong unit.
Record the unit in the min/max ts fields' Arrow metadata when writing
and build the per-file time predicates from it when searching, so each
file is interpreted in the unit it was written with. The writer now
also requires a timestamp time index.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on series index time units
- split schema validation (validate_index_schema) from unit extraction
(index_time_unit), distinguishing missing vs unsupported unit metadata
in the errors instead of one misleading 'missing a valid metadata'
- encode the recorded unit with an explicit exhaustive match rather than
Debug formatting, so the on-disk encoding is reviewed next to its parser
- extract time_index_unit to drop the unwrap in series_index_schema and
the duplicated timestamp-time-index ensure in validate_metadata
- test one searcher reading files with different recorded units, and the
rejection of missing, unknown and mismatched units
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: reuse TimeUnit's Display form for the recorded unit string
common_time::timestamp::TimeUnit already implements Display with the
exact strings the series index records ("Second"/"Millisecond"/
"Microsecond"/"Nanosecond"), so drop the local time_unit_as_str
mapping and use it; the parse side stays local since no FromStr
counterpart exists anywhere yet.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: parse TimeUnit from its Display form in common-time
Add FromStr for common_time::timestamp::TimeUnit, accepting exactly the
Display form ("Second"/"Millisecond"/"Microsecond"/"Nanosecond") and
failing with a new UnsupportedTimeUnit error (InvalidArguments). The
series index now records and parses the unit with the common codec,
dropping its local parse_time_unit.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: parse TimeUnit case-insensitively
Lowercase the input before matching so "millisecond" and "MILLISECOND"
parse like "Millisecond"; the error still reports the original string.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: store series index min/max timestamps as native Timestamp columns
Replace the Int64 min/max columns plus 'time_unit' field metadata with
native Timestamp(unit) columns, so the unit rides on the datatype and
each file is interpreted in the unit it was written with naturally.
- series_index_schema types the columns from the time index unit; the
writer reinterprets the raw i64 series bounds in that type (arrow's
Int64->Timestamp cast reinterprets, it does not rescale)
- the searcher reads the unit from each file's column datatype, builds
Timestamp-typed predicates via datatypes' timestamp_to_scalar_value,
and reinterprets parquet INT64 statistics in the column type so
row-group pruning compares like-typed values
- index files whose min/max columns are not Timestamp (written before
this change) are rejected
- the pruning test now asserts time-range predicates prune row groups,
not just tag predicates
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: drop the TimeUnit string codec from common-time
With the unit carried by the Timestamp datatype, the FromStr impl and
UnsupportedTimeUnit error added for the field-metadata approach have no
consumer; remove them.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: fold index file validation into a single pass
The series index format is unreleased and unwired, so no compatibility
classes are needed: validate_index_schema checks all columns and
returns the min/max columns' unit directly, replacing the separate
index_time_unit extraction.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: carry timestamps through SeriesIndexRow
SeriesIndexRow and the aggregation path now hold common_time::Timestamp
instead of raw i64s: timestamp_values interprets the input column in the
writer's unit (rejecting a timestamp array whose unit differs, instead
of silently reinterpreting it), and rows_to_batch builds the native
Timestamp columns directly from the rows' units without an Int64 round
trip through arrow cast.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: require a timestamp time index column in series index input
The writer already refuses non-timestamp time indexes on the metadata
side, and its input batches always carry the region's ts column as a
timestamp array, so accepting plain Int64 columns only left a silent
unit-interpretation hole; reject them instead.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: rescale series index input timestamps into the file unit
The input array's unit is self-describing, so converting with
Timestamp::convert_to cannot mislabel values; a mismatch no longer
needs to be an error. Only a value that overflows the file's unit
fails the write. This also makes the writer ready to aggregate
old-unit batches after a time index widening.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: drop redundant unit checks in series index writer
The alter path flushes memtables before widening the region's time
index unit, so a writer never receives batches in the region's
previous unit. Reject a unit mismatch at the input boundary instead
of rescaling per value, and build the index batch in the writer's
recorded unit instead of re-deriving it from the schema and
re-checking every row.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(prometheus): align batch flush deadline with creation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(servers): sync pending worker submission with explicit ack
Replace the mpsc capacity() polling in the pending rows batcher deadline
test with a test-only WorkerCommand::Ack round trip. The FIFO channel
guarantees the worker has dequeued and processed the submission (and
anchored the flush deadline) before the test advances virtual time,
removing reliance on an implementation detail that can be flaky under
scheduling variance.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(servers): await pending worker flush results with bounded timeout
Replace the try_recv() yield-polling loop in the pending rows batcher
deadline test with a direct await bounded by tokio::time::timeout. Under
paused time the timeout auto-advances the clock and fires
deterministically, so the test fails reliably instead of intermittently
missing the result on slower CI or under contention.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): add series index catalog and lifecycle components
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat(mito2): restore series index catalogs on region open
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): simplify series index foundation and maintenance
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): separate series index purge task and simplify tests
Signed-off-by: evenyag <realevenyag@gmail.com>
* docs: defer experimental series index configuration examples
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat(mito): make series index maintenance interval configurable
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): correct series index cleanup on close and drop
Signed-off-by: evenyag <realevenyag@gmail.com>
* test(mito2): revert drop test changes
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update config API expectation for series index settings
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: use tokio unbounded channel for series index purger
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): simplify review test scope and clarify index config
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): reject non-flat batches in flat merge instead of panicking
- SortColumns::new becomes fallible try_new: batches missing the
flat-format internal columns (time index, __primary_key, __sequence)
at the fixed trailing positions now yield InvalidRecordBatch instead
of a downcast panic, completing the generic-schema gate that only
covered BatchBuilder output assembly. Document the flat-format input
contract on FlatMergeIterator/FlatMergeReader.
- Clarify why BatchBuilder's schema gate uses >= 3 columns when a real
flat-format schema always has at least 4.
- Add schema-structure tests: empty primary keys (tables without tags),
dictionary-encoded string tag columns with per-source dictionaries,
and graceful rejection of batches without internal columns.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* test(mito2): scan tables with various schemas through flat merge
Add an engine-level test that writes, flushes and scans regions without
tags (empty primary key) and with multiple string tags (dictionary-encoded
in the flat input schema), so the flat merge reader merges an SST with
the memtable on real schemas instead of hand-built batches.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor(mito2): use winner_tree dependency
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Update comments for FlatMergeIterator struct
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(cli): sanitize store_addrs in kvbackend build log
Close#7525. The CLI's kvbackend construction log was printing raw
store_addrs which could contain sensitive connection strings (e.g.
PostgreSQL DSNs with passwords).
Changes:
- Add sanitize_store_addrs() helper that reuses
common_meta::kv_backend::util::sanitize_connection_string(),
consistent with MetasrvOptions and StartCommand patterns.
- Replace raw store_addrs in the info! log with sanitized version.
- Add unit tests covering MySQL URLs, PostgreSQL DSNs, etcd addresses,
and empty store_addrs cases.
Signed-off-by: qiang_liu
Signed-off-by: qiang_liu <qiang_liu@trendmicro.com>
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>
* fix(cli): drop redundant sanitize tests per review
sanitize_connection_string in common_meta already covers MySQL URLs,
PostgreSQL DSNs and credential-free etcd addresses with its own tests.
The added tests only exercised a trivial map+collect wrapper, so remove
them per review nit.
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>
---------
Signed-off-by: qiang_liu
Signed-off-by: qiang_liu <qiang_liu@trendmicro.com>
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* feat(mito2): adapt bulk memtable encode bytes threshold to write buffer size
The default encode_bytes_threshold is now max(64MB,
min(global_write_buffer_size / 32, 512MB)) instead of a fixed 64MB, so
it scales with the memtable budget. GREPTIME_BULK_ENCODE_BYTES_THRESHOLD
and the per-region option still override the default.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): clarify binary units in bulk threshold test
The threshold test uses powers of 1024, so label its values as MiB and GiB instead of decimal MB and GB.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): resolve bulk encode threshold in memtable provider
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf: add scanbench partition diagnostics
Port benchmark source and documentation from c5dd5e6356a8903f1dcf493eb46dc318612dc953. Exclude the Mito metrics changes.
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: support scanbench query suites
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit def9c22069b0d29c611695a68ad1b31110c845cb)
Signed-off-by: evenyag <realevenyag@gmail.com>
* docs: align scanbench port with existing engine metrics
Remove documentation and fixture references to unported Mito metrics. Enable dev-tools in the build example and remove the obsolete force-flat-format option.
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
dennis zhuang
Yingwen
Weny Xu
Lei, HUANG
Immanuel Tikhonov
discord9
jeremyhi
Dhruv Vaishnav
fys
Ning Sun
shuiyisong
Mohd Quamar Tyagi
Qiang-Liuanddennis zhuang
sun
XuanwoandWenyXu