mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-21 19:32:34 +00:00
2b05da4c06ac38f2e182b6479eeecfcb45836d8f
6082
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
2b05da4c06 |
fix: backport suggested independent fixes to release/5e02efa9 (#9795)
* fix(servers): resolve port from IPv6 bind addresses (#9142) Signed-off-by: immanuwell <pchpr.00@list.ru> (cherry picked from commit |
||
|
|
fd3faeb7ed |
fix: backport scan statistics and partitioning fixes to release/5e02efa9 (#9783)
* fix: preserve count correctness after repartition (#9154) * fix: preserve count correctness after repartition Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: restore full predicate guard for count statistics Signed-off-by: WenyXu <wenymedia@gmail.com> * test: update count plans for full predicate guard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve count statistics for safe partition scans Signed-off-by: WenyXu <wenymedia@gmail.com> * test: retain repartition home guard and cover staging flush Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit |
||
|
|
1e5fef5809 |
fix(mito): backport bulk memtable version ordering fix to release/5e02efa9 (#9789)
* fix(mito2): preserve last-non-null values across compaction (#9169) * fix(mito2): preserve last-non-null values during compaction Use a dedicated LastNonNullPicker to close TWCS seeds over overlapping SSTs, merge connected seed groups, and defer closures with busy inputs. Keep LastRow and strict-window selection unchanged. Index overlap candidates on the blocking compaction runtime and cover field promotion, transitive dependencies, expiration, and conservative key-range pruning with regression tests. Refs #9146 Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): guard last-non-null compaction against pending memtables Track conservative memtable sequence lower bounds and capture them from the same version as compaction inputs. Defer unsafe SST closures and strict-window plans until pending writes no longer intersect their sequence span. Cache minimum sequences during bulk conversion and expose a lightweight memtable getter to avoid rescanning rows or computing full statistics. Fixes #9157 Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): limit eligible last-non-null compaction outputs Enumerate all TWCS seeds for LastNonNull while retaining bounded window planning concurrency. Apply the output budget after closure expansion and busy/memtable safety checks, preserving execution priority and ordinary TWCS behavior. Cover rejected and absorbed seeds, priority ordering, and bounded and unlimited output counts with real-picker regression tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): include all candidates before last-non-null closure Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify last-non-null compaction closure Extract transitive closure expansion from output scheduling and clarify seed handling. Document the dedicated picker rationale and schema-evolution risks in PK overlap checks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover deferred non-seed compaction bridges Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): defer strict-window compaction with busy dependencies Keep busy SSTs when building the LastNonNull strict-window dependency closure, then defer the entire rewrite plan if any selected input is busy. Preserve LastRow filtering and expired-file cleanup.\n\nCover requested, transitive and unrelated busy windows, including retry after the dependency is released. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document compaction overlap index fields Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> (cherry picked from commit |
||
|
|
1506974bd9 |
fix: backport SST write buffer release to release/5e02efa9 (#9774)
* fix: release completed SST write buffers during compaction (#9243)
* fix: upgrade OpenDAL and honor SST write buffer size
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: track removal of the OpenDAL bridge fork
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): honor SST write buffers across upload paths
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit
|
||
|
|
508d1a1a33 |
fix: backport clean must-have fixes to release/5e02efa9 (#9772)
* fix(mito2): preserve effective sequences during compaction (#9147) * fix(mito2): preserve effective row sequences during compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): inherit compaction input sequence bounds Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> (cherry picked from commit |
||
|
|
ef3c9d7ccc |
fix(query): backport region scan hash partitioning fix to release/5e02efa9 (#9769)
fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned
* test(query): use current DataFusion plan downcasts
* test(query): match null rendering in join regression
* test(query): refresh union plan for unknown scan partitioning
* test(query): refresh shared plans after distribution rule removal
* test(query): cover semi joins with remote dynamic filtering
* test(query): strengthen partitioned merge scan regressions
* test(query): remove unused merge scan test helpers
* test(perf): measure merge scan hash repartition cost
* fix(query): adapt merge scan deserialization to two-arg constructor
Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion.
* test(query): adapt nested merge scan integration constructor
---------
(cherry picked from commit
|
||
|
|
8282f72c74 |
fix(mito): backport skip-WAL close flush to release/5e02efa9 (#9625)
Track skip-WAL writes across mutable and immutable memtables and flush them on graceful region close. Distinguish incoming requests from WAL replay, and preserve WAL-only recovery tests by bypassing Close.
Adapt bulk tracking to the release branch's existing region-level WAL policy without adding request-level bulk skip-WAL support. Keep release-only tests and omit changes to the absent SST primary-key module.
(cherry picked from commit
|
||
|
|
d878fb2a57 |
chore: backport rustls 0.23.46 to release/5e02efa9 (#9624)
Backport #9614 to fix TLS 1.2 record-limit handling. Raise the rustls minimum and update rustls-webpki and aws-lc dependencies. Preserve the release branch's rustc-hash dependency and raise cc to 1.2.41, the minimum required by aws-lc-sys 0.46.0.
(cherry picked from commit
|
||
|
|
12b7ebe7d5 |
feat: backport bulk skip-WAL support and integration coverage (#9627)
* test: backport request-level skip-WAL integration coverage Backport #9093 as the shared protocol and storage test harness prerequisite for bulk request WAL skipping. Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit |
||
|
|
bd01d2a138 |
fix(pprof): backport framehop unwinder to release/5e02efa9 (#9605)
* feat(pprof): switch CPU profiler to framehop unwinder Replace the default libgcc-based unwinder in pprof-rs with the framehop unwinder, which is designed to be async-signal-safe: - framehop performs no heap allocation during unwinding (MustNotAllocateDuringUnwind) - It handles prologue/epilogue interruption correctly - It falls back to frame-pointer unwinding when CFI is unavailable - It does not depend on libgcc's unwind implementation, which is documented as not signal-safe (see tikv/pprof-rs#36) Bump pprof from 0.14 to 0.15 in all three consumers (common-pprof, cmd, servers) to unify on a single version. pprof 0.15 also replaces parking_lot with spin-rs to avoid a potential profiler deadlock (#268). This addresses the libgcc_s.so.1 #GP crash observed in production when CPU profiling is active, by eliminating the libgcc unwinder from the signal handler path entirely. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> (cherry picked from commit |
||
|
|
5ae1145380 |
test: backport renewed TLS fixtures to release/5e02efa9 (#9623)
Refresh the expired integration TLS certificate/key fixtures from #9196. Preserve the compatibility window already updated by #9609.
(cherry picked from commit
|
||
|
|
f4c2b2161a |
ci: backport MinIO image fix to release/5e02efa9 (#9622)
Use Silo server and mc images to avoid unauthorized pulls of the chart's default quay.io/minio images during distributed fuzz setup.
(cherry picked from commit
|
||
|
|
9033bb4e7b |
fix: backport jemalloc TSD fixes to release/5e02efa9 (#9606)
chore: bump tikv-jemalloc-sys patch to jemalloc dev (ff80bf2d) (#9119)
* chore: bump tikv-jemalloc-sys patch to jemalloc dev (ff80bf2d)
Update the [patch.crates-io] rev for tikv-jemalloc-sys from
e1846d8c (5.3.1 + 54f22c83 backport) to ff444d4 (upstream dev HEAD,
161 commits ahead of 5.3.1).
The dev branch includes additional TSD/tcache fixes beyond the
original backport:
- fb5499aa9c: Handle jemalloc calls after TSD teardown
- 61dc1da395: Fix possible tcache corruption on fiber migration
- 1e92317014: Fix thread-exit TSD cleanup
See GreptimeTeam/jemallocator branch bump-jemalloc-dev and
tikv/jemallocator#182.
* chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main
GreptimeTeam/jemallocator#2 has been merged; reference the merge
commit e254a7ea on main instead of the PR head branch. Jemalloc
source content is unchanged (still upstream dev ff80bf2d).
---------
(cherry picked from commit
|
||
|
|
d37f5dffa0 |
chore(ci): backport v1.2.1 compatibility window to release/5e02efa9 (#9609)
chore(ci): update compatibility test window to v1.2.1
(cherry picked from commit
|
||
|
|
33c0e7b0ec |
fix(operator): backport catalog-based write retries to release/5e02efa9 (#9584)
fix(operator): reuse catalog table resolution for write retries (#9574)
* fix(operator): inherit logical table partitions when retrying writes
* fix(operator): resolve retried writes through catalog manager
---------
(cherry picked from commit
|
||
|
|
372f33540a |
feat(frontend): backport write retries to release/5e02efa9 (#9569)
* feat(client): support region request reuse and route refresh hints (#9540) * feat(client): support shared region requests without deep cloning Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate route refresh retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * style: preserve local error variant imports in retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): distinguish staging rejection from write backpressure Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): preserve legacy retry hints in instruction replies Signed-off-by: WenyXu <wenymedia@gmail.com> * test(client): inline retry hint compatibility cases Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): align batch-open retry hint assertion Signed-off-by: WenyXu <wenymedia@gmail.com> * style: use imported types in region request tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit |
||
|
|
c66a4aeb37 |
fix(mito2): hold the bulk compact permit for the whole blocking task (#9252)
Keep the semaphore permit alive throughout queued and running bulk compaction in `src/mito2/src/memtable/bulk.rs`.
(cherry picked from commit
|
||
|
|
1cb529510c |
fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)
Backport #9304 across `access_layer.rs`, `cache.rs`, `cache/file_cache.rs`, `read/pruner.rs`, `region/opener.rs`, `sst/index.rs`, and `sst/parquet/reader.rs` under `src/mito2/src`.
(cherry picked from commit
|
||
|
|
ab0b1f5cce |
fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix (#9103)
* fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix Upgrade tikv-jemallocator / tikv-jemalloc-ctl / tikv-jemalloc-sys from 0.6 to 0.7, which embeds jemalloc 5.3.1 (includes a056c20d 'Handle tcache init failures gracefully'). On top of that, patch tikv-jemalloc-sys to the GreptimeTeam fork that adds the remaining upstream fix 54f22c83 'Initialize TSD tcache before enabling it' (GreptimeTeam/jemalloc#1, GreptimeTeam/jemallocator#1). Without the ordering fix, a reentrant allocation during TSD bootstrap (e.g. heap-profiling prof_tdata init / sampled backtrace when prof:true is active) can observe an enabled-but-uninitialized tcache, corrupting per-thread tcache metadata and crashing the process in arena_stats_merge, calloc, or the libgcc unwinder. The patch is pinned by rev and should be removed once tikv/jemallocator ships a jemalloc snapshot that includes 54f22c83. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump tikv-jemalloc-sys patch rev to merged release-5.3.1 GreptimeTeam/jemalloc#1 has been merged; point the patch at the jemallocator commit referencing the merge commit on release-5.3.1. Jemalloc source content is unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main GreptimeTeam/jemallocator#1 has been merged; reference the merge commit e1846d8c on main instead of the PR head branch. Jemalloc source content is unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(servers): bump tikv-jemallocator dev-dependency to 0.7 main added a target.'cfg(not(windows))'.dev-dependencies entry on tikv-jemallocator 0.6 for servers after this branch diverged. On the merge ref it pulled tikv-jemalloc-sys 0.6 from crates.io, which conflicts with the patched 0.7 (links = "jemalloc" may only appear once in the dependency graph), failing version selection in CI. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
5d5f0d6d70 |
fix(query): prevent incomplete aggregate dynamic filtering (#9102)
* fix(query): prevent incomplete aggregate dynamic filtering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): cover mixed expression and column maxima Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): pin upstream aggregate regression backport Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): assert aggregate filter behavior on datanode scans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): pin merged aggregate dynamic filter fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
fa128adb8e |
feat: support request-level insert WAL skipping (#9088)
* refactor: add skip_wal fields to internal write requests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(deps): update greptime-proto for insert skip_wal Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): support request-level WAL skipping Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): handle request-level WAL policies Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate insert WAL policy through query context Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: support session-level insert WAL policy via SET Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(metric-engine): require uniform WAL policy in batch puts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(metric-engine): simplify WAL policy coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(servers): simplify gRPC hint extraction Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten Mito and Metric WAL scenario orchestration Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate WAL and memtable-only mutations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: carry skip-WAL policy in table insert requests Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten skip-WAL policy cases Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify WAL notifier naming Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin merged skip-WAL proto revision Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
536f42e4a2 |
feat(json2): support ALTER syntax for JSON column settings (#9094)
feat(json2): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> This is the commit message #3: |
||
|
|
0f625a7e92 |
fix(promql): correct counter reset accumulation in rate windows (#9089)
* fix(promql): correct counter reset accumulation in rate windows `prom_rate` and `prom_increase` reused the previous window's counter-reset correction when the next window slid forward by exactly one sample, adding the entering reset and subtracting the leaving one. Running a sum through addition and subtraction does not restore the earlier terms in f64: a large reset absorbs the smaller ones that must survive it, and an expired infinity leaves a NaN that no later window can clear. `prom_delta` shares the code but is not a counter function, so it never took that path. Index the reset positions of the value array once instead, and reduce each window over the resets it contains, in sample order. The result is bit-identical to scanning the window directly, so windows keep the direct reduction when they request fewer sample pairs than the input has. Also sweep the query step in the rate benchmarks: the cost of the reset correction depends on how much the windows overlap, which no existing case varied. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * test(promql): cover counter reset precision over adjacent rate windows The unit tests build the range windows directly, so they do not show that a plain PromQL range query produces the window layout that lost the correction. This case does: with a query step equal to the sample interval, `increase` over the second window returns 1.333 before the fix and 2.667 after it. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * perf(promql): advance the counter reset bounds instead of searching Locating a window's resets with two binary searches costs more than the reduction it replaces once a series resets often enough for the searches to get deep: on a 20k-sample counter resetting every 37 samples, stepping the windows by one sample was 2.7x slower than the previous code, against 1.2x for a counter that never resets. Windows normally advance, so walk the bounds forward from the previous window and only search when they move back. The cost then no longer depends on the reset density. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * perf(promql): cut the per-window cost of the counter reset index Two costs the index added showed up on a one-sample query step, where the removed fast path used to answer each window with two comparisons. Cache the two reset positions that bound the active slice. A window that only advanced and reached neither of them covers the same resets as the previous one, so the common case is four integer comparisons and no lookup at all. Stop summing the requested sample pairs once they exceed one pass over the values. The sum only decides which side of that comparison the input falls on, and a query with a short lookback and a long step settles it after a few windows instead of after every key. Together these take the one-sample step from 25-32% slower than the previous code down to 6-11%, measured as before / after / before to bound drift. No other step value regresses, and a ten-sample step stays about 88% faster. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * fix(promql): accumulate counter resets into the running result `prom_rate` and `prom_increase` summed a window's counter-reset corrections on their own and added that sum to `last - first`. Prometheus folds each reset into the running result instead, and so did this code before #7880. The two are not interchangeable in f64: over samples `[1e16, 1, 0, 1]` the isolated sum rounds `1e16 + 1.0` back to `1e16`, which then cancels against the first sample and reports no increase at all, where folding the resets in one at a time keeps the 1.0. Restore the original order. The reset index accumulates into the result the same way, so it still matches a direct scan of the window bit for bit, but a window's contribution can no longer be cached as a standalone value and is re-added from its own difference each time. The bounds are still cached, so a window that did not cross a reset skips the lookup, and one that holds no resets returns without touching the index at all. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * test(promql): note which reset boundaries the stride of one walks Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> --------- Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> |
||
|
|
adda50e03f |
feat: add repartition partition count hint (#9080)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
2d78220a4f |
fix(flow): drain frontend probe response before selecting peer (#9082)
* fix(flow): drain frontend probe response before selecting peer Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(flow): reuse record batch collector for frontend probe Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
2933f95672 |
perf(servers): drop the output sort for Prometheus range queries (#9090)
* perf(servers): drop the output sort for Prometheus range queries Matrix responses sort samples by timestamp and series by labels while serializing, so the sort the PromQL planner puts at the root of a range query plan never reaches the client. Mark the query as not requiring output ordering in `do_range_query`, and let the frontend strip the root sort (through projections, and only when it carries no fetch) before execution. Instant queries are unchanged, so `sort()` and `sort_desc()` keep their observable order. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * test(servers): pin the matrix response against input row order Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * test(servers): drop redundant response assertions in the ordering test Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * perf(servers): drop the output sort for gRPC gateway range queries Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> --------- Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> |
||
|
|
60899d19ee |
perf(servers): defer Prometheus sample value formatting to serialization (#9091)
Building a matrix response allocated one `String` per sample while the record batches were scanned, then dropped it after the JSON body was written. Keep the `f64` in `PromSampleValue::Number` instead and format it with ryu while serializing, so no per-sample string is allocated. `PromSampleValue::Text` keeps values parsed from a JSON body, so deserializing and re-serializing a response is unchanged. Vector and scalar results still expose `String`, since they hold a single sample. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> |
||
|
|
38577c8854 |
refactor: support plugin-backed admin functions (#9083)
* feat: add bulk load SQL extension points Add `CREATE BULK LOAD` parsing and `BulkLoadHandler` dispatch. Expose private system table factories through `src/catalog`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: support plugin-backed admin functions Replace the custom bulk-load SQL hooks with access to process `Plugins` through `FunctionState`. Preserve typed external errors from ADMIN functions in `src/operator/src/error.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
a7ce7486e1 |
fix(ci): strip pre-release extension when creating nightly release version (#9081)
The scheduled (nightly) release version was created by appending
'-nightly-YYYYMMDD' to NEXT_RELEASE_VERSION as-is. When the Cargo.toml
version carries a pre-release extension (e.g. v1.3.0-alpha.1), this
produced invalid tags like 'v1.3.0-alpha.1-nightly-20260907', stacking
'nightly' on top of the 'alpha.1' pre-release.
Strip the pre-release extension first so 'nightly' itself becomes the
only pre-release extension: 'v1.3.0-alpha.1' -> 'v1.3.0-nightly-20260908'.
Stable versions are unaffected; nightly-build ('nightly-YYYYMMDD-sha'),
dev-build, tag push, and manual dispatch paths are unchanged.
Signed-off-by: Ning Sun <sunning@greptime.com>
|
||
|
|
6fa1023b7f |
feat(mito2): introduce TWCS active window compaction (#9011)
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows Split the single TWCS trigger_file_num into per-window-state thresholds: the active window keeps the existing trigger (default 4, legacy compaction.twcs.trigger_file_num stays a compatible alias), while inactive windows use a new trigger (default 2). Inactive windows additionally fall back from balanced L0-only/L1-only candidates to a progress-making unbalanced mixed candidate so historical windows can converge; the active window retains the row/byte balance guards. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): bound inactive TWCS window convergence by rewrite budget Inactive windows that cannot compact within one level previously either stayed stuck (a threshold-qualified but unbalanced level returned no candidate without trying any fallback) or fell back to a mixed merge with no balance checks at all, which could rewrite a huge compacted file to absorb tiny fresh files. Inactive windows now converge progressively: threshold-qualified balanced picks, sub-threshold balanced single-level picks, a mixed merge whose total rewrite must fit in the output file budget, and finally an L0-only merge without balance checks. Windows that qualify for none of these are left uncompacted, bounding write amplification. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover TWCS window trigger options in alter_table_options sqlness case Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and compaction.twcs.inactive_window.trigger_file_num end to end, including that setting the canonical active key removes the legacy compaction.twcs.trigger_file_num alias from the table options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): derive TWCS active window from the max-sequence file The active window was determined by the max event-time window among level-0 files. Between an L0 compaction removing its inputs and the next flush landing, level 0 is empty, so the active window transiently became None and every window fell back to the inactive rules - triggering full-window convergence merges during ongoing ingestion whose outputs are then superseded by new data. Flush and compaction outputs both inherit the max input sequence, so the file with the highest sequence across all levels always tracks the most recent write. Use its window as the active window, falling back to the previous L0-based rule when no file carries a sequence (legacy files). The new helper deliberately computes window keys with the assign_to_windows convention (truncate to seconds, then align up), because the result is compared against window keys produced there; the older ceil-based helper is kept unchanged for the legacy fallback path. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): add active-window L1 compaction safety trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(compat): cover TWCS active window options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): align TWCS window trigger validation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve database TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize newer compaction windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): repick serial compaction outputs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): configure inactive-window L1 trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): prioritize TWCS compaction candidates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): preserve TWCS trigger downgrade compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(sql): distinguish invalid database option values Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(meta): explain TWCS legacy key compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify active window trigger field Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): normalize TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): ignore ineligible files for active window Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark explicit TWCS options as overrides Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): normalize zero compaction output size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): validate database TWCS trigger values Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(store-api): collapse TWCS alias conflict condition Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
7cf84892d2 |
perf(table): filter decoded rows with dynamic predicates (#9004)
* perf(table): filter decoded rows with dynamic predicates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): preserve unknown dynamic filter rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(table): use null guards for dynamic filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): preserve null inputs during dynamic pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover frontend join dynamic filter transfer Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): reset dynamic filters and scanner pruning state Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
da5cb1a190 |
perf(promql): push down last row for instant queries (#9034)
* perf(promql): push down last row for instant queries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: guard instant last row correctness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update instant query explain results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: apply last row after source deduplication Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: scope post-merge last row selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover instant PromQL last row Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): sort generated SST rows before writing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover instant last row selection in sqlness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): avoid last row hints for lossy timestamp casts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): preserve stale marker semantics across flushes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): decode dictionary labels in stale regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): avoid reserved column name in stale fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): exercise LastRow hints and filtered results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: keep after-merge mode in LastRow selector Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: reject instant LastRow across residual filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: expect after-merge selector in instant vector guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: explain instant LastRow filter eligibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restrict instant LastRow to safe selector nodes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: show LastRow merge mode directly in diagnostics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh LastRow display in explain expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
c65a4e545e |
feat: make the series index resilient to time index unit widening (#8996)
* feat: record time unit per series index file The series index stores __series_min_ts/__series_max_ts as raw i64 in the time index unit at write time, but the searcher built its range predicates from the region's current unit, so files written before a time index unit widening would be compared in the wrong unit. Record the unit in the min/max ts fields' Arrow metadata when writing and build the per-file time predicates from it when searching, so each file is interpreted in the unit it was written with. The writer now also requires a timestamp time index. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on series index time units - split schema validation (validate_index_schema) from unit extraction (index_time_unit), distinguishing missing vs unsupported unit metadata in the errors instead of one misleading 'missing a valid metadata' - encode the recorded unit with an explicit exhaustive match rather than Debug formatting, so the on-disk encoding is reviewed next to its parser - extract time_index_unit to drop the unwrap in series_index_schema and the duplicated timestamp-time-index ensure in validate_metadata - test one searcher reading files with different recorded units, and the rejection of missing, unknown and mismatched units Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: reuse TimeUnit's Display form for the recorded unit string common_time::timestamp::TimeUnit already implements Display with the exact strings the series index records ("Second"/"Millisecond"/ "Microsecond"/"Nanosecond"), so drop the local time_unit_as_str mapping and use it; the parse side stays local since no FromStr counterpart exists anywhere yet. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: parse TimeUnit from its Display form in common-time Add FromStr for common_time::timestamp::TimeUnit, accepting exactly the Display form ("Second"/"Millisecond"/"Microsecond"/"Nanosecond") and failing with a new UnsupportedTimeUnit error (InvalidArguments). The series index now records and parses the unit with the common codec, dropping its local parse_time_unit. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: parse TimeUnit case-insensitively Lowercase the input before matching so "millisecond" and "MILLISECOND" parse like "Millisecond"; the error still reports the original string. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: store series index min/max timestamps as native Timestamp columns Replace the Int64 min/max columns plus 'time_unit' field metadata with native Timestamp(unit) columns, so the unit rides on the datatype and each file is interpreted in the unit it was written with naturally. - series_index_schema types the columns from the time index unit; the writer reinterprets the raw i64 series bounds in that type (arrow's Int64->Timestamp cast reinterprets, it does not rescale) - the searcher reads the unit from each file's column datatype, builds Timestamp-typed predicates via datatypes' timestamp_to_scalar_value, and reinterprets parquet INT64 statistics in the column type so row-group pruning compares like-typed values - index files whose min/max columns are not Timestamp (written before this change) are rejected - the pruning test now asserts time-range predicates prune row groups, not just tag predicates Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: drop the TimeUnit string codec from common-time With the unit carried by the Timestamp datatype, the FromStr impl and UnsupportedTimeUnit error added for the field-metadata approach have no consumer; remove them. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: fold index file validation into a single pass The series index format is unreleased and unwired, so no compatibility classes are needed: validate_index_schema checks all columns and returns the min/max columns' unit directly, replacing the separate index_time_unit extraction. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: carry timestamps through SeriesIndexRow SeriesIndexRow and the aggregation path now hold common_time::Timestamp instead of raw i64s: timestamp_values interprets the input column in the writer's unit (rejecting a timestamp array whose unit differs, instead of silently reinterpreting it), and rows_to_batch builds the native Timestamp columns directly from the rows' units without an Int64 round trip through arrow cast. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: require a timestamp time index column in series index input The writer already refuses non-timestamp time indexes on the metadata side, and its input batches always carry the region's ts column as a timestamp array, so accepting plain Int64 columns only left a silent unit-interpretation hole; reject them instead. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: rescale series index input timestamps into the file unit The input array's unit is self-describing, so converting with Timestamp::convert_to cannot mislabel values; a mismatch no longer needs to be an error. Only a value that overflows the file's unit fails the write. This also makes the writer ready to aggregate old-unit batches after a time index widening. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: drop redundant unit checks in series index writer The alter path flushes memtables before widening the region's time index unit, so a writer never receives batches in the region's previous unit. Reject a unit mismatch at the input boundary instead of rescaling per value, and build the index batch in the writer's recorded unit instead of re-deriving it from the schema and re-checking every row. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
0d5d97e7cb |
chore(ci): update compatibility versions (#9075)
Signed-off-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com> Co-authored-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com> |
||
|
|
27ea5dfb91 |
feat(auth): support per-user MySQL authentication methods (#9078)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
6b9165f633 |
fix(servers): return text[] for SELECT array[null] in PostgreSQL protocol (#9058)
fix(servers): return text array for null array literals Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> |
||
|
|
5706afdfeb |
ci(backport): label backport PRs with their version name (#9073)
* ci(backport): label backport PRs with their version name Signed-off-by: Ning Sun <sunning@greptime.com> * ci(backport): restore multi-line PR body string Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
4fd35462d6 |
fix(prometheus): align batch flush deadline with creation (#8802)
* fix(prometheus): align batch flush deadline with creation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(servers): sync pending worker submission with explicit ack Replace the mpsc capacity() polling in the pending rows batcher deadline test with a test-only WorkerCommand::Ack round trip. The FIFO channel guarantees the worker has dequeued and processed the submission (and anchored the flush deadline) before the test advances virtual time, removing reliance on an implementation detail that can be flaky under scheduling variance. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(servers): await pending worker flush results with bounded timeout Replace the try_recv() yield-polling loop in the pending rows batcher deadline test with a direct await bounded by tokio::time::timeout. Under paused time the timeout auto-advances the clock and fires deterministically, so the test fails reliably instead of intermittently missing the result on slower CI or under contention. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
b2a2ad64fb |
fix(ci): avoid cross-references in PR limit comments (#9065)
* fix(ci): use a search link in PR limit warnings Signed-off-by: evenyag <realevenyag@gmail.com> * fix(ci): list PR titles using redirect links Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
6fb5d2ebad |
feat(mito2): add series index catalog and lifecycle foundation (#9053)
* refactor(mito2): add series index catalog and lifecycle components Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito2): restore series index catalogs on region open Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify series index foundation and maintenance Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): separate series index purge task and simplify tests Signed-off-by: evenyag <realevenyag@gmail.com> * docs: defer experimental series index configuration examples Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): make series index maintenance interval configurable Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): correct series index cleanup on close and drop Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): revert drop test changes Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config API expectation for series index settings Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use tokio unbounded channel for series index purger Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): simplify review test scope and clarify index config Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
1494a5c6ee |
fix: prevent catalog testing feature from leaking into production builds (#9063)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
b46de8c828 |
feat(telemetry): add log directory size retention (#8997)
* feat(telemetry): add log directory size retention Signed-off-by: WenyXu <wenymedia@gmail.com> * test: update config API logging fixture Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(telemetry): recover log retention state Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(telemetry): handle log retention cleanup errors Signed-off-by: WenyXu <wenymedia@gmail.com> * test(telemetry): cover log count retention on rotation Signed-off-by: WenyXu <wenymedia@gmail.com> * test(telemetry): cover log directory retention Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(telemetry): avoid log filename allocation Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
be482553b0 |
fix(mito2): harden flat merge and use winner_tree (#9064)
* fix(mito2): reject non-flat batches in flat merge instead of panicking - SortColumns::new becomes fallible try_new: batches missing the flat-format internal columns (time index, __primary_key, __sequence) at the fixed trailing positions now yield InvalidRecordBatch instead of a downcast panic, completing the generic-schema gate that only covered BatchBuilder output assembly. Document the flat-format input contract on FlatMergeIterator/FlatMergeReader. - Clarify why BatchBuilder's schema gate uses >= 3 columns when a real flat-format schema always has at least 4. - Add schema-structure tests: empty primary keys (tables without tags), dictionary-encoded string tag columns with per-source dictionaries, and graceful rejection of batches without internal columns. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test(mito2): scan tables with various schemas through flat merge Add an engine-level test that writes, flushes and scans regions without tags (empty primary key) and with multiple string tags (dictionary-encoded in the flat input schema), so the flat merge reader merges an SST with the memtable on real schemas instead of hand-built batches. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(mito2): use winner_tree dependency Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Update comments for FlatMergeIterator struct Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
|
42a58080ea |
ci(backport): fix issue creation on cherry-pick conflict (#9048)
ci(backport): properly create issue on cherry-pick conflict Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
ad5ccc98ec |
fix(cli): sanitize store_addrs in kvbackend build log (#8967)
* fix(cli): sanitize store_addrs in kvbackend build log Close #7525. The CLI's kvbackend construction log was printing raw store_addrs which could contain sensitive connection strings (e.g. PostgreSQL DSNs with passwords). Changes: - Add sanitize_store_addrs() helper that reuses common_meta::kv_backend::util::sanitize_connection_string(), consistent with MetasrvOptions and StartCommand patterns. - Replace raw store_addrs in the info! log with sanitized version. - Add unit tests covering MySQL URLs, PostgreSQL DSNs, etcd addresses, and empty store_addrs cases. Signed-off-by: qiang_liu Signed-off-by: qiang_liu <qiang_liu@trendmicro.com> Signed-off-by: LiuQhahah <liuqiang9596@gmail.com> * fix(cli): drop redundant sanitize tests per review sanitize_connection_string in common_meta already covers MySQL URLs, PostgreSQL DSNs and credential-free etcd addresses with its own tests. The added tests only exercised a trivial map+collect wrapper, so remove them per review nit. Signed-off-by: LiuQhahah <liuqiang9596@gmail.com> --------- Signed-off-by: qiang_liu Signed-off-by: qiang_liu <qiang_liu@trendmicro.com> Signed-off-by: LiuQhahah <liuqiang9596@gmail.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> |
||
|
|
307fe0a692 |
feat(mito2): adapt bulk memtable encode threshold to write buffer size (#9056)
* feat(mito2): adapt bulk memtable encode bytes threshold to write buffer size The default encode_bytes_threshold is now max(64MB, min(global_write_buffer_size / 32, 512MB)) instead of a fixed 64MB, so it scales with the memtable budget. GREPTIME_BULK_ENCODE_BYTES_THRESHOLD and the per-region option still override the default. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): clarify binary units in bulk threshold test The threshold test uses powers of 1024, so label its values as MiB and GiB instead of decimal MB and GB. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): resolve bulk encode threshold in memtable provider Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
9a65561226 |
feat(auth): support bearer token authentication over SQL protocols (#8899)
* chore: add mysql and pg auth method choose hook Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat(auth): support bearer token authentication over SQL protocols Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mysql): authenticate bearer tokens in the handshake catalog Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
88d43ee648 | feat: update dashboard to v0.13.15 (#9051) | ||
|
|
35f5485974 |
feat(meta): record logical-table reconciliation events (#8941)
* feat(meta): add logical table reconciliation events Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(meta): preserve logical reconciliation progress Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(meta): preserve logical region retry progress Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(meta): simplify logical reconciliation events Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): assert logical event values Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> |
||
|
|
9940de4917 |
feat: add scanbench query suites and structured results (#9050)
* perf: add scanbench partition diagnostics Port benchmark source and documentation from c5dd5e6356a8903f1dcf493eb46dc318612dc953. Exclude the Mito metrics changes. Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support scanbench query suites Signed-off-by: evenyag <realevenyag@gmail.com> (cherry picked from commit def9c22069b0d29c611695a68ad1b31110c845cb) Signed-off-by: evenyag <realevenyag@gmail.com> * docs: align scanbench port with existing engine metrics Remove documentation and fixture references to unported Mito metrics. Enable dev-tools in the build example and remove the obsolete force-flat-format option. Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |