* fix: bound metric snapshot memory and reduce export overhead (#9505)
* fix(mito): bound retained scan memory
Compact unused tag dictionary values before merge retention and avoid splitting
a single scan source into redundant batches. Include regression coverage.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(snapshot): bound writes and reduce logical export overhead
Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible
string dictionaries, bound conversion work, combine Parquet row groups and
filter single-table exports before routing. Include resource-boundary and
packed-restore regression coverage.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs: clarify metric snapshot acceptance criteria
Specify populated scale fixtures, controlled S3 request delay, native macOS
validation, paging exclusions and alternating comparison rounds.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito): split batches only for merged readers
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(snapshot): cover supported types in both layouts
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs: consolidate metric snapshot experiment conclusions
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito): honor series statistics when splitting L0 scans
Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics.
Cover short-series overlapping L0 files with an ordered latest-value regression.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(mito): explain memory risk of splitting short L0 series
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor: address snapshot export review feedback
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(cli): accept source errors in lazy snapshot contexts
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 18726c7602)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(prom): align remote read label matcher semantics (#9633)
* fix(prom): align remote read label matcher semantics
Signed-off-by: fys <fengys1996@gmail.com>
* chore: optimize comment of validate_matchers
* fix(prom): preserve existing binary label regex handling
Signed-off-by: fys <fengys1996@gmail.com>
* refactor: condition
* test(prom): simplify remote read matcher tests and clarify comments
* fix(prom): return binary remote read labels as UTF-8
* fix(prom): reject binary labels in remote read
* refactor(prom): align remote read null label handling with PromQL
Signed-off-by: fys <fengys1996@gmail.com>
* refactor(prom): simplify null handling in label matcher branches
Signed-off-by: fys <fengys1996@gmail.com>
* fix(prom): restrict remote read labels to string types
Signed-off-by: fys <fengys1996@gmail.com>
* test(prom): use string label in default value column test
Signed-off-by: fys <fengys1996@gmail.com>
* fix: code review
---------
Signed-off-by: fys <fengys1996@gmail.com>
(cherry picked from commit 4e8bc6df86)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): restore source splitting for cross-range series merges (#9818)
* fix: restore batched series merges across ranges
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): split range sources before last-row selection
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): avoid tiny splits for single-source cross-range merges
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 27c531336a)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): pick TWCS L1 merges by closed units so compaction terminates (#9741)
* refactor(mito2): pick TWCS L1 merges by closed units
TWCS could keep rewriting L1 files without new writes. Overlap-only picks
could split their inputs into one more file than they read, and L1 file
reduction then rewrote up to 16 files to remove that single file. The
outputs, split by key, created new overlaps and the cycle repeated.
Replace the three global pick phases with one pick per window, scheduled by
class across windows. Every pick now decreases a termination measure when the
window receives no writes.
- Active windows only take merges that remove files. L1 files are grouped by
conflicts and merged in key order, so size-split tails are merged together.
A reduction rewrites at most 4 output files' worth of bytes per removed file.
- Inactive windows absorb L0 files, then take closed merges: no other file
intrudes on the inputs' bounding box, so outputs cannot add overlaps. Each
merge resolves conflicts or removes files, decreasing (conflicts, files).
Files that do not take part in an overlap are not rewritten.
- Overlaps that only share a boundary key no longer count as conflicts, so a
size-split sorted run does not look overlapping.
- Closed merges may take 4x the input limit to resolve a whole window once.
- Cache primary key ranges per pick instead of resolving them per comparison.
- Log the pick branch of each compaction output.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): state the compactor assumptions behind TWCS termination
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): resolve active window overlaps on manual compaction
A window that stops receiving writes keeps being the active window, whose
picks only remove files, so its overlaps were never resolved. Manual
compactions now pick every window as inactive, so a closed merge can resolve
them. Automatic compactions are unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): defer overlap resolution in the window before the active one
Late data keeps landing in the window right before the active window. A
closed merge that resolves its overlaps early rewrites the whole window,
then the late data brings the overlaps back, and the balanced run check
keeps them there. Overlap-only merges now skip that window until it is
older. File-reducing merges and manual compactions are unchanged.
On the 12h/1GB out-of-order bench this reaches overlap depth 1.0 instead
of 2.08 at the same write amplification (3.01 vs 3.02).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): only check earlier-starting files when closing units
Closing units only checks files that start before a unit ends, and the
overlap checks compare key bounds without cloning them.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): keep settling windows from rewriting L1 files
The window before the active one still receives late data. Skipping only
its overlap-only merges was not enough: file-reducing merges of
conflicting files and mixed L0/L1 merges still rewrote its L1 files,
and a mixed merge could rewrite a nearly full output for each small late
file. A settling window now only merges L0 files among themselves and
conflict-free files; the window resolves its overlaps once it is older.
On the out-of-order bench, write amplification drops from 3.88 to 3.30
at 2GB/6h, 3.05 to 2.94 at 1GB/6h and 3.01 to 2.96 at 1GB/12h, with
overlap depth 1.0 in all three.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): resolve late overlaps once a window is quiet
Late data that lands in a window older than the settling one is merged
into a small file covering the whole key range. Resolving it rewrites
the window's dominant sorted run, which the balanced run check refuses,
so the overlap stayed. Removing the check instead rewrites the window
for every late flush.
A window is quiet once the region has written at least as many rows as
the window holds since its newest file. Overlap merges in a quiet window
may rewrite a dominant run, so each quiet period pays for at most one
such rewrite. Sequences are per row and kept by compaction, so this
needs no state.
On the bench with late data up to 2 hours, mean overlap depth drops
from 1.67 to 1.17 at 2GB/6h and from 2.33 to 1.83 at 1GB/6h, at write
amplification 3.70 to 4.25 and 3.26 to 3.74; all inactive windows of
2GB/6h realtime reach depth 1 (write amplification 2.80 to 3.30). The
remaining depth is in the final active window.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): check closed merges of quiet and settling windows terminate
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): split TWCS closures over the input limit
A file spanning the keys of a large sorted run joins every run file into one
closure. Once a closure held more files than one compaction accepts, closed
merges skipped it, so the window never merged those files again and late
files kept piling up.
Split such closures into single files, like active-window picks do, and
replace the closed check with a conflict bound: outputs only conflict with
files intruding on the inputs' box, so a merge is taken if `outputs · I`
does not exceed the conflicts its inputs had with other files. Intervals of
whole closures have no such conflicts and still need to be closed, so other
windows pick as before, and `(conflicts, files)` still strictly decreases.
Small run files and late files over a large run can be merged again. A late
file over a run that cannot shrink stays: resolving it would rewrite the
whole run.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): address TWCS picker review findings
- Manual compactions may rewrite a dominant sorted run to resolve overlaps.
The active window is never quiet, so manual compactions could not resolve
its late-data overlaps before.
- Single-key intervals predict one output: the writer never splits a series,
so large files of one series were never merged.
- The active window falls back to single files when no group of conflicting
files qualifies, so a large file no longer keeps its small peers unmerged.
- Compute the settling window distance with checked_sub, which overflowed for
windows at the timestamp limits.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): count inner keys as TWCS conflicts and check balance per group
- A single key strictly inside another file's key range now conflicts, so
a late file of one series inside a wide file is resolved like other
overlaps. Files sharing only a boundary key still do not conflict.
Intrusion covers the new pairs, which a randomized test checks.
- Check the dominant-run guard per group of conflicting files. Runs of
separate groups never overlap, so pairing one group's run with another
group's peer made an interval of two dominant runs look balanced.
- A window with unknown row counts, written before they were recorded, is
not quiet: its threshold would drop to a single row.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): index TWCS files by key to avoid quadratic grouping
Grouping a window's files compared every file with all files sharing its
time range, and closing groups and checking intrusions scanned the whole
window. A sorted run split by key shares one time range, so a window of N
such files took O(N²) checks without any conflict. The 1024-file fallback
only covers the sorted-run path, not these groups.
Index the files by lower primary key in a segment tree that keeps each
subtree's largest upper key and its time bounds. Grouping, closing, and the
conflict bound only check the files whose key and time ranges meet, so a
file of a key-sliced run only meets its neighbors. Conflicts among split
files are precomputed per file and counted per interval with a binary
search instead of pairwise.
On a run of 20,000 key-sliced files with 5 late files spanning all keys,
a closed merge pick takes ~62 ms instead of ~3.9 s and an active pick
~19 ms instead of ~3.5 s.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(object-store): retry transient S3 transport failures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): bound TWCS closed-merge conflict graph work
Budget the candidate files visited while building conflict units and split-unit adjacency. When the budget is exceeded, treat every file as a split unit and pick a bounded file reduction instead of materializing a quadratic graph.
The fallback candidate is admitted only through adds_no_conflicts, so outputs do not add conflict pairs and the (P, N) termination invariant still holds. Dense picks are logged with their own branch.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): charge TWCS index nodes to the conflict-graph budget
The conflict-graph work budget only counted files that reached a callback, so a traversal with poor pruning could still visit a quadratic number of internal nodes without triggering the dense fallback. Charge every ExtentIndex node, including pruned nodes, and stop the traversal as soon as the shared budget is exhausted.
Add a regression test where the root straddles the query time but neither leaf matches, so a callback-only budget would never fire.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): retry smaller TWCS dense compaction candidates
If a dense fallback candidate fails admission, try one single-output subinterval within it before giving up. Reuse the reduction guards and check both candidates against the full window's external conflicts, preserving bounded work and termination.\n\nCover the chain regression through both fallback unit constructions and reject one-output retries that introduce conflicts across time gaps.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): compact shared primary-key boundaries
Count shared primary-key endpoints as conflicts and apply the same overlap rule to bounding-box intrusions. Existing compaction writers keep series intact, so fresh output ranges remain disjoint.
Align simulated output splits with the writer and cover boundary-only cleanup in inactive and manual windows, plus stable real output files after a subsequent TWCS request.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): compact small L1 files in inactive windows
Allow bounded small-L1 reductions in settling and older inactive windows when normal picks cannot proceed. Preserve full-window conflict checks, large-file protection, and LastNonNull closure semantics.\n\nCover selection limits, termination, and real SST compaction with tombstones and reopen checks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): clarify TWCS unit grouping guarantees
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit bb3f07b482)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: reclaim HDFS temporary files left behind by a datanode crash (#9616)
* fix: reclaim HDFS temporary files left behind by a datanode crash
Signed-off-by: lczllx <2181719471@qq.com>
* fix(mito2): stop the temporary file sweep from starving GC jobs
Signed-off-by: lczllx <2181719471@qq.com>
* feat(mito2): add metrics for the temporary file sweep
Signed-off-by: lczllx <2181719471@qq.com>
* refactor(object-store): share one definition of the temporary file name
Signed-off-by: lczllx <2181719471@qq.com>
* fix(mito2): skip the temporary file sweep when no GC job is allowed
Signed-off-by: lczllx <2181719471@qq.com>
* refactor: gate the temporary file sweep behind the hdfs-object-store feature
Signed-off-by: lczllx <2181719471@qq.com>
* fix(mito2): sweep temporary files when a create opens an existing region
Signed-off-by: lczllx <2181719471@qq.com>
* docs(config): document the temporary file sweep in the GC settings
Signed-off-by: lczllx <2181719471@qq.com>
* refactor: align the temporary file sweep code with the surrounding style
Signed-off-by: lczllx <2181719471@qq.com>
---------
Signed-off-by: lczllx <2181719471@qq.com>
(cherry picked from commit 88cb2a986e)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): fence datanode processes replaced by a newer process (#9878)
* fix(meta): fence datanode processes replaced by a newer process
Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.
The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.
Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache load and pusher updates coherent
A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.
A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.
The epoch check moves into the heartbeat service.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache coherent with writes while it is disabled
A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(meta): pin the epoch 0 exemption and pusher order, and update docs
The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): register a datanode pusher only after its first request passes
A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.
Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): check a datanode epoch again before replying
A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): handle a datanode's requests one at a time
A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.
The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 6f5c064f54)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: acknowledge writes only within the region's leader lease (#9904)
* fix: acknowledge writes only within the region's leader lease
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: drop cached logical columns after a failed logical drop
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: keep the prepared region for the lease check after the WAL append
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: serialize a logical drop with fills of the logical column cache
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: check the drop lock and grant long leases deterministically
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: cover a bulk insert that loses its lease during the WAL append
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 91a59b82fb)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito): preserve SST lifetime across index replacement (#9950)
* fix(mito): preserve SST lifetime across index replacement
Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity.
Fixes#9944
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): list-based SST purge and single FilePurger::remove_file
The final purge of a retired SST no longer depends on FileMeta: index
files of every version are discovered by listing
({region_dir}/index/{file_id}. prefix), so even when the last live view
is a stale one, the purge still removes all newer index files.
- SstLifetime is slimmed down to { deleted, file_purger }; the purge now
runs in the last FileHandleInner::drop (Arc::strong_count == 1, which
is race-free here since only inners of the same file hold the lifetime).
- FilePurger is merged back into a single remove_file(file_meta,
is_delete) method; is_delete means the file was retired AND this is the
last live view.
- LocalFilePurger deletes the SST and all its index files via the new
delete_file_and_all_indexes helper (list before delete, so a listing
failure leaves the file untouched).
- Remove the dead index_outdated flag and its accessors, which had no
callers in the entire repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): elect final SST view release atomically
Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface.
Add a deterministic concurrent-drop regression covering retired and live files.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): restore Arc-owned SST cleanup and clarify reference release
Restore the lifetime design from 835227dcbc, using the final Arc drop and latest metadata instead of manual view counting and per-SST index listing.
Rename FilePurger::remove_file to release_file_ref while retaining purge_file for final cleanup. Preserve and adapt the concurrent view-release regression test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): track SST references by file lifetime
Register one reference guard per logical SST lifetime and release it on final drop, including normal region close. Remove per-view release_file_ref and index_outdated handling.
Protect all index generations while their SST is referenced. Preserve the existing GC snapshot format by expanding the recorded index generation upper bound for older version-aware workers.
Update coverage for independent references, normal close, cross-region ownership, concurrent final cleanup and both GC listing modes.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): release SST references directly on lifetime drop
Remove the redundant FileReferenceGuard and let SstLifetime register and unregister its file reference directly. Preserve unconditional reference release on normal close and retirement-only purge.
Expose the reference manager through FilePurger and adapt reference manager tests without changing GC or snapshot behavior.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito): verify SST cleanup with real storage and isolate GC policy cases
Replace the recording-only concurrent purge test with filesystem assertions using LocalFilePurger and shared fixtures. Keep intermediate-view retention and retired versus live file cleanup coverage without claiming exhaustive thread interleavings.
Separate GC reference-policy cases from reader-release ordering and reuse their setup. Production code remains unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(mito): format SST lifetime changes
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): reclaim obsolete indexes by reader version
Track exact index-version references per metadata view and restore version-specific Puffin liveness in both GC paths. Keep file-level Parquet protection and shared SST lifetime cleanup, without expanding snapshots into unreferenced index generations.
Include old readers of imported files even when the source region is not open locally. Add real-storage GC coverage for reader release and lingering while retaining Parquet and the current index.
Validated workspace checks with default and all features, all-target all-feature Clippy with warnings denied, and all 1853 mito2 tests.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit 17ed988f49)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): coordinate range cache invalidation and singleflight reads (#9935)
* fix(mito2): invalidate retired SST range results and coalesce loads
Coordinate decoded range loads by their full scan key, bound shared work, and prevent late fills after SST retirement. Invalidate range results for local purge and object-store GC while preserving old snapshot reads.
Fixes#9934
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): preserve SST lifetime across index replacement
Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity.
Fixes#9944
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): list-based SST purge and single FilePurger::remove_file
The final purge of a retired SST no longer depends on FileMeta: index
files of every version are discovered by listing
({region_dir}/index/{file_id}. prefix), so even when the last live view
is a stale one, the purge still removes all newer index files.
- SstLifetime is slimmed down to { deleted, file_purger }; the purge now
runs in the last FileHandleInner::drop (Arc::strong_count == 1, which
is race-free here since only inners of the same file hold the lifetime).
- FilePurger is merged back into a single remove_file(file_meta,
is_delete) method; is_delete means the file was retired AND this is the
last live view.
- LocalFilePurger deletes the SST and all its index files via the new
delete_file_and_all_indexes helper (list before delete, so a listing
failure leaves the file untouched).
- Remove the dead index_outdated flag and its accessors, which had no
callers in the entire repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): elect final SST view release atomically
Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface.
Add a deterministic concurrent-drop regression covering retired and live files.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): restore Arc-owned SST cleanup and clarify reference release
Restore the lifetime design from 835227dcbc, using the final Arc drop and latest metadata instead of manual view counting and per-SST index listing.
Rename FilePurger::remove_file to release_file_ref while retaining purge_file for final cleanup. Preserve and adapt the concurrent view-release regression test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): track SST references by file lifetime
Register one reference guard per logical SST lifetime and release it on final drop, including normal region close. Remove per-view release_file_ref and index_outdated handling.
Protect all index generations while their SST is referenced. Preserve the existing GC snapshot format by expanding the recorded index generation upper bound for older version-aware workers.
Update coverage for independent references, normal close, cross-region ownership, concurrent final cleanup and both GC listing modes.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): release SST references directly on lifetime drop
Remove the redundant FileReferenceGuard and let SstLifetime register and unregister its file reference directly. Preserve unconditional reference release on normal close and retirement-only purge.
Expose the reference manager through FilePurger and adapt reference manager tests without changing GC or snapshot behavior.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito): verify SST cleanup with real storage and isolate GC policy cases
Replace the recording-only concurrent purge test with filesystem assertions using LocalFilePurger and shared fixtures. Keep intermediate-view retention and retired versus live file cleanup coverage without claiming exhaustive thread interleavings.
Separate GC reference-policy cases from reader-release ordering and reuse their setup. Production code remains unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(mito): format SST lifetime changes
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): use crate-rooted range cache imports
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): account for range cache delivery lifetime
Keep completed range results charged while watch subscribers or replay streams retain them. Cache entries keep weak delivery references, allowing cache-only ownership to release the delivery reservation. Cache hits reuse or nonblockingly acquire the reservation, falling back when the budget is exhausted.
Resize input reservations to completed-result estimates and cover successive slow consumers, cache-only handoff, hit sharing, invalidation, admission rejection, and cancellation with regression tests.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit 44df6e5ded)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): exclude route followers from failover targets (#10001)
* fix(meta): exclude route followers from failover targets
Failover target selection did not exclude the region's route followers.
The selector could keep picking a follower, the submit rejected it as a
peer conflict, and the region never failed over.
Group the failed regions by their followers and exclude them per group.
A group without a candidate is skipped instead of failing every region of
the datanode.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: check failover continues after a group without candidates
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep failover going when follower routes can't be read
Fall back to selecting without follower exclusion when the routes can't
be read, log skipped groups with their followers at warn level, and test
a region without a route.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(meta): select failover targets per exclusion set in select_peers
generate_failover_tasks now only reads the route followers of the regions.
select_peers resolves the affinity and the unschedulable datanodes once,
then selects per group of regions that exclude the same datanodes.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(meta): drop a comment that repeats the code
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(meta): tighten failover target selection
Name the route followers after the parameter they fill, read the routes
with unwrap_or_else, and log a group without a candidate as an error, as
the caller did before.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): let the region-stat-aware selector balance the whole batch
The region-stat-aware selector now receives each region's excluded peers
along with the whole batch, so it can still balance load across the
regions while keeping a region off its own followers. A region without a
candidate is left out of its output instead of failing the others. The
selectors driven by SelectorOptions keep selecting per group of regions
that exclude the same datanodes.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: name the batch type of the stat-aware recorder
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(meta): treat a selector without candidates the same in both arms
Both selector arms now place what they can and treat NoEnoughAvailableNode
as nothing placed; the stat-aware selector's contract says so. Rename its
parameter to exclude_peer_ids_by_region, check for unplaced regions in
linear time, and describe what the code does in the comments.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: keep the stat-aware recorder's placements off excluded peers
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 62ebf71f90)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(log-store): make raft-engine periodic WAL sync actually fsync (#9715)
* fix(log-store): make raft-engine periodic WAL sync actually fsync
On raft-engine 0.4.2, Engine::sync() writes an empty LogBatch and
write() returns before fsync (tikv/raft-engine#395). The periodic
sync task therefore never durable-flushed acknowledged WAL data.
Work around by writing a minimal non-empty marker with sync=true so
the fsync path is reached until a crates.io release includes
tikv/raft-engine#400.
Fixes#9712
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* perf(log-store): reuse LogBatch across periodic WAL syncs
Keep the periodic sync task's LogBatch across ticks instead of
allocating a fresh one (4 KiB encode buffer plus item vec) every
period. A successful Engine::write drains the batch but retains its
buffers; after a failed write or a panicked blocking task the batch is
discarded and recreated on the next tick, since it may be left
partially encoded.
Keep the marker put rather than deleting an empty key: the put
overwrites a single key, whereas every delete would become a recovery
tombstone that raft-engine keeps in memory while replaying append
files.
Also document that the marker value must not decode as a NamespaceImpl
(the marker key lies inside the range scanned by list_namespaces), add
a test covering batch reuse and namespace listing, and apply rustfmt.
Refs #9712
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
---------
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
(cherry picked from commit 180e2716fc)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(catalog): propagate frontend process-list failures (#9999)
* fix(catalog): propagate frontend process-list failures
Return the existing invocation error when a selected frontend cannot list processes, so a successful response cannot omit that peer.
Signed-off-by: Byte-Naut <Byte-Naut@proton.me>
* test(frontend): cover remote process-list errors and aggregation
Exercise real metadata discovery and gRPC listeners for a missing management service and successful catalog-scoped aggregation.
Signed-off-by: Byte-Naut <Byte-Naut@proton.me>
---------
Signed-off-by: Byte-Naut <Byte-Naut@proton.me>
(cherry picked from commit 3d2b73762a)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(log-store): skip records with unparsable keys instead of aborting the read (#9749)
* fix(log-store): skip records with unparsable keys instead of aborting the read
Signed-off-by: lczllx <2181719471@qq.com>
* fix(log-store): keep undecodable RecordMeta keys fatal
Signed-off-by: lczllx <2181719471@qq.com>
---------
Signed-off-by: lczllx <2181719471@qq.com>
(cherry picked from commit 5f0e22171e)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): register failure detectors for regions switched before a rollback (#9989)
* fix(meta): register failure detectors for regions switched before a rollback
A region migration switches the routes of its tables one table at a
time. If it aborts after some tables were switched, those regions are
already served by the target, but the rollback deregistered the
target's failure detectors for every region and registered only the
source's. The target's next heartbeat registers them again, so if the
target stopped before sending one, nothing detected its failure and
the switched regions stayed unavailable until the metasrv leader
changed or the target came back.
The rollback now collects, from the routes it already reads, the
regions whose leader is the target and resets the target's failure
detectors for them at the end. It also skips a table whose route no
longer exists, so a table dropped meanwhile no longer fails the
rollback before the detectors are handled.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(meta): roll back a downgrading leader in the rollback detector test
The table that was not switched yet keeps a downgrading leader in a real
abort, so the test now builds it that way and checks that the rollback
clears it.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): monitor regions switched before a rollback only on the target
The rollback registered the source's failure detectors for every region
of the migration, including the regions already switched to the target.
Those regions are served by the target after the rollback, so their
source detectors watched the wrong peer: when one fired, the failover
found the leader had changed and only removed it. The rollback now
registers the source's detectors only for the regions that were not
switched, and keeps resetting the target's detectors for the others.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor(meta): rename reset_failure_detectors_for_candidate_regions to reset_failure_detectors
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 05536cd7f5)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: ignore stale leader grants after a region is downgraded (#9987)
* fix: ignore stale leader grants after a region is downgraded
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: compare stale leader grants with the longest lease seen
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: treat a lease too long to store as unknown
A lease longer than a u64 of nanoseconds saturated to a shorter one, so the
derived send time could be later than the real one. Now such a lease makes
the send time unknown, which counts as earlier than any gate.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: address review comments on stale leader grants
Scope the early send time claim to after a lease has been seen, drop the
unused role parameter of demote, clarify a test comment and check that a
fresh grant still makes a reopened region writable.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor: compare stale leader grants with the heartbeat send time
Pass the send time of the heartbeat behind a lease to set_region_role_with_lease,
and return whether the role was applied. Mito compares the send time with the
grant gate directly, so it no longer derives it from the longest lease seen.
The datanode finalizes leadership only when the grant was applied.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: cover stale grants through the datanode and repeated downgrades
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor: keep leader grant rules in one region method
Apply a role grant in MitoRegion::apply_role_grant, which renews the lease
and decides whether the role changes; the engine only looks up the region.
Rename the gate to downgrade_started_at and demote to begin_downgrade, keep
the field private, and copy it on reopen through
inherit_downgrade_state_from.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor: drop a needless borrow
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor: keep lease concepts in the engine and role ordering in the region
ManifestContext, which owns the role, records when the role was last
demoted and promotes a region only for a promotion issued after that. The
time starts when the context is created, so a reopened region ignores
grants for heartbeats sent before the reopen, without copying state. The
engine translates leases: it renews the deadline and promotes with the
heartbeat send time. MitoRegion keeps no grant state.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix: keep the data region of a metric region off a grant its metadata region ignores
The metric engine applies a grant to the metadata region first and stops
if it is ignored, so the data region, whose role the engine reports, never
becomes a leader that skips finalization. Also describe what promoting
returns.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs: describe when a lease grant is ignored and flatten the finalize check
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor: move lease types out of the region and the Leader deadline into ManifestContext
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* refactor: name the promotion fence after the downgrade it records
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 89a5df3141)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): preserve live index files across remote compactions (#10046)
fix(mito2): reuse executor-owned compaction index resources
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit c7ea80558f)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(datanode): pick a node epoch above the recorded one (#9984)
* fix(datanode): pick a node epoch above the recorded one
The metasrv fences a datanode whose node epoch is lower than the
highest one it has recorded for the node id. A datanode used its start
time as its epoch, so after a restart on a host whose clock is behind
the previous process's, or after the clock stepped back, the new
process picked a lower epoch and was fenced for good: no region lease,
no keep-alive.
Before it creates the heartbeat task, the datanode now reads the
recorded epoch through its metadata backend and uses one more than it
if its start time is not higher. The node info still reports the local
start time. A failed read fails the startup, like the other metadata
reads.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(datanode): tighten the node epoch tests
Use a lower record that leaves room below the start time, so the test
fails if the start time is ignored, and a nonzero node id, so it fails
if the node id is ignored. Clarify the doc comment of the epoch choice.
This assumes the previous process of a node id has stopped before a new
one starts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit c40e86c653)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(auth): preserve physical metric table targets (#10084)
* fix(auth): preserve physical metric table targets
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix: nit
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit c17a5e09f7)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: include peer context in distributed query errors (#10118)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 8deb3872b3)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(auth): redact inline user-provider credentials (#10115)
* fix(auth): redact inline user-provider credentials
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(auth): close user-provider redaction gaps
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit adf29740b2)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: register trigger procedure loaders through DdlManager (#10123)
Add a loader registration hook to TriggerDdlManager and invoke it
before procedure recovery. Test that registration reaches the
configured trigger DDL manager.
Signed-off-by: fys <fengys1996@gmail.com>
(cherry picked from commit b6cf3384dc)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(time): parse zero-hour timezone offsets correctly (#10091)
fix(time): preserve zero-hour timezone offset signs
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
(cherry picked from commit 25003cea16)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): invalidate all datanode tables on address change (#9986)
When a datanode restarts with a new address, metasrv invalidated cached
table routes only for the tables in that heartbeat's region stats. The
first heartbeat after a restart often reports few or no regions (e.g.
with init_regions_in_background), so frontends kept dialing the old
address until metadata_cache_ttl expired.
Read the previous address via prev_kv when saving it, and when it
changed (or a failed save of the same epoch is retried) invalidate
every table registered for the datanode in __dn_table.
Signed-off-by: Floris Smit <florissmit10@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
(cherry picked from commit 351f4207b4)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(function): require ADMIN statement privileges to execute admin functions (#10120)
* fix(function): require ADMIN statement privileges to execute admin functions
Read-only accounts could execute the admin UDFs (flush_table,
compact_table, flush_flow, gc_regions, migrate_region, reconcile_*,
build_index, ...) through ordinary SELECT statements: SELECT is a
read-only statement so it passes the statement-level privilege check,
and the functions were only guarded by the greptime-catalog scope
check inside the UDF. That let read-only users trigger WAL flushes,
manifest version bumps, compactions, flow-sink writes and, in
distributed mode, region GC/migration and SST deletion.
Wrap the privilege check into the admin_fn macro itself, next to the
existing greptime-catalog check: before executing, every #[admin_fn]
function now requires the same privileges as its ADMIN statement. The
check synthesizes the PermissionReq::SqlStatement request an
ADMIN <fn>(...) statement produces and evaluates it with the same
PermissionCheckerRef plugin the frontend installs, so query invocation
(SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path
are gated identically — in particular read-only accounts are denied
with PermissionDenied (7006), while write-privileged users keep both
invocation styles. With auth disabled (no permission checker), the
functions stay invocable by everyone, exactly like ADMIN statements.
The denial is wrapped as External(BoxedError) so the 7006 status code
survives both the query execution path and the admin statement
executor.
Regression tests: unit tests for the check (no-checker allows,
read-only denied 7006, read-write allowed), an end-to-end test over
the macro-generated flush_table UDF, and an integration test where a
read-only account is denied 7006 on all 13 functions from SELECT while
a read-write account keeps SELECT and ADMIN FLUSH_TABLE working.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(function): expose admin privilege check to downstream macros
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
(cherry picked from commit b21b1cdb69)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(catalog): use advertised peer address as process identity (#10153)
* fix(catalog): use advertised peer address as process identity
The process manager identified the local frontend by its public gRPC
address, while metasrv discovers peers by the address advertised in the
heartbeat (the internal gRPC address when configured). With separate
public and internal listeners, list_all_processes failed to exclude the
local node, queried its own management service as a remote frontend, and
appended the local registry again, so every local query was listed twice.
The same mismatch broke routing of KILL to other frontends.
Identify a frontend by the address it advertises to metasrv:
- add FrontendOptions::peer_addr, shared by the heartbeat and the process
manager, and FrontendOptions::create_process_manager to wire it once;
- use the advertised address as the process identity, for self-exclusion
in list_all_processes, and for local/remote routing in kill_process.
Process ids now carry the address other frontends use to reach the node.
When no internal gRPC listener is configured this is the public gRPC
address, so the previous behavior is unchanged. The internal listener is
the only one serving the frontend management service, so process ids stay
routable.
The new distributed regression tests exercise separate public and internal
addresses: the local query must be listed exactly once while remote
frontends are still listed, and KILL must route to a remote frontend.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: keep local peer address reserved in kill-routing test
The test reserved the local frontend address with a temporary listener that
was dropped immediately, then reserved the remote address separately. The
kernel could assign the same port to the remote frontend, so the remote kill
request would take the local branch and fail to find the remote process.
Bind the local address once and keep the listener alive for the whole test so
the local and remote frontend identities cannot collide.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit 557843dd68)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito): use stored recovery-set bytes for manifest size (#10142)
* fix(mito): track stored bytes for manifest recovery
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito): retain manifest sizes on checkpoint and sync
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(mito): cover logical manifest size accounting
Signed-off-by: WenyXu <wenymedia@gmail.com>
* docs(mito): clarify manifest size retention invariants
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit d7036df73c)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(grpc): propagate cancellation for read query execution (#10166)
* x
* fix(grpc): restrict cascading cancellation to read query variants
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 2cb473607d)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(datatypes): escape object keys when rendering jsonb as text (#10112)
* fix(datatypes): escape object keys when rendering jsonb as text
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* fix(datatypes): skip unicode normalization for escaped jsonb keys
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* refactor(datatypes): serialize jsonb to text directly with serde_json
Serialize the binary jsonb with serde_json::to_string(&RawJsonb) so that
both object keys and string values are escaped by serde_json. This drops
fix_unicode_point, its regex and the hand-written key escaping. Errors are
propagated instead of falling back to lossy raw bytes.
Characters such as U+2028 are now emitted as-is, which is still valid
JSON, so the affected sqlness result lines are regenerated.
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
* fix(datatypes): accept JSON text bytes in jsonb_to_string
Some write paths (COPY FROM csv/json/parquet, JSON bound as text via
MySQL) store JSON text instead of binary jsonb in JSON columns. Re-encode
such bytes as jsonb before serializing with serde_json, so they still
render (properly escaped) instead of failing with InvalidJsonb.
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
---------
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
(cherry picked from commit ec1ad1f15b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): share bulk memtable compaction dispatcher across workers (#10022)
Each region worker built its own MemtableBuilderProvider, and every
provider created its own CompactDispatcher. The limit on background
bulk memtable compactions was therefore num_workers times
max_background_compactions, while SST compaction uses one
engine-wide pool of max_background_compactions.
Create one dispatcher in WorkerGroup::start (and start_for_test),
pass it to every worker, and make MemtableBuilderProvider::new take
it instead of creating its own. SST and memtable compaction keep
separate budgets of the same size.
Signed-off-by: luckys <190197589+luckysxx@users.noreply.github.com>
Co-authored-by: luckys <190197589+luckysxx@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
(cherry picked from commit 9f7cedf58f)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito): decode cached manifest payloads (#10179)
* test(mito2): cover compressed manifest cache recovery
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): decode cached manifest object payloads
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): address manifest cache review feedback
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 4263f5fc01)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: prioritize region ALTER over queued edits (#10193)
* fix: prioritize region ALTER over queued edits
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: use valid edits in ALTER priority regression
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: make region edit dispatch outcomes explicit
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: reject queued alters and edits when dropping regions
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: cover queued ALTER handoff before region edits
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit ae422ef465)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta-srv): invalidate leader cache on transaction errors (#10219)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 4688fa4be8)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): avoid reporting reader filter metrics twice (#10222)
* perf(mito2): cache fixed-label reader metric counters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): report reader filter metrics only once
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 5d88d283e5)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(mito2): reuse static manifest checks across row groups (#10220)
* perf(mito2): reuse static manifest checks for cached readers
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito2): drop redundant pruner test and release entry lock early
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(mito2): cache static manifest keep verdicts directly
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit c844d44721)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: adapt v1.3.1 backports to release interfaces
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix!: enforce sparse primary key schema (#10020)
* fix: preserve required sorting for ordinary sparse scans
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix: enforce sparse primary key schema
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* test: fix
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit efed847ca6)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: test (#10149)
fix: fixture
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit 4633c37dbb)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: tolerate invalid UTF-8 in JSON log ingestion (#10245)
* fix: tolerate invalid UTF-8 in JSON ingestion
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix: tolerate invalid UTF-8 in NDJSON ingestion
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* chore: upgrade simd json
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* feat: add server default for ignore_errors
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit 0cb70d4f84)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: simplify backported JSON recovery cases
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(otlp): add v1 schema for auto-created OTLP log tables (#10207)
* feat(otlp): add v1 schema for auto-created OTLP log tables
New tables created by direct OTLP logs ingestion get a v1 built-in schema:
service_name tag from resource service.name ahead of scope_name in the
primary key, a skipping index on trace_id, and event_name and
observed_timestamp fields. The table is marked with
table_data_model=greptime_log_v1. Existing tables without the marker keep
the legacy schema.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(otlp): share resource service.name parsing between logs and traces
Log tables now derive service_name with the same rules as trace spans, so
both signals agree on the value used to correlate them. The integration test
writes twice to cover the existing v1 table path.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 2714687ab7)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): bound scatter time filter expression depth (#10272)
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 361f3595db)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(query): use current partition columns in prepared plans (#10244)
A prepared statement keeps its logical plan, and that plan holds the
table as it was at prepare time. The analyzer reads the partition
columns from that table while the regions to scan come from the current
route. After a table's partition columns change, the analyzer can
consider a query safe to push down, so each region runs the whole query
and the frontend concatenates their results. The same happens to the
source of a prepared INSERT ... SELECT, which then stores wrong values.
Before executing a kept plan, compare each scanned table's partition
columns with the current table found by id. Scans of a table whose
columns changed get the current table when the schema is unchanged.
Otherwise they keep their schema, because a scan's projection holds
indices into it, and report the current columns as absent from it, so
the query keeps merging on the frontend.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 297a343fd4)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(time): avoid i32 overflow when converting an out-of-range date (#10299)
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
(cherry picked from commit 402e4a97fc)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: update dashboard to v0.14.5 (#10270)
Co-authored-by: dennis zhuang <killme2008@gmail.com>
(cherry picked from commit 4c4c8a58fd)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): keep rows written during staging in the WAL (#10253)
* fix(mito2): keep rows written during staging in the WAL
A flush during staging committed its SSTs to the staging manifest, then
dropped the memtable and released the WAL. If staging did not end
successfully those rows were gone. A flush that committed after staging
had ended lost its rows even though staging succeeded.
Staging regions no longer flush. They are skipped by memory-pressure,
periodic and region-limit flushes, and their writes are rejected instead
of queued at twice the region limit or when the node's staging memtables
reach half of the engine's flush limit. A manual or repartition flush on
a staging region with data gets a retryable error; the migration
pre-flush and the WAL prune flush reply success without flushing. Closing
a staging region leaves staging first, so skip-WAL rows are flushed to
the normal manifest.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): keep accounting and closing staging regions during staged edits
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): track staged edits without relying on the partition info
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): synchronize staging close tests with worker admission
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): observe the manifest lock wait when closing staging regions
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): detect staged edits through the staging partition info
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(mito2): record that staging regions must not flush
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): do not restore staging after a close abandons the edit
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): reject region edits during a close-time flush
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* style(mito2): format the staging tests
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): reject staging edits once staging is abandoned
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): bind apply-staging edits to their partition expression
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs(mito2): note that staging regions pin their topic's prune point
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 8da533c35f)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(query): push column projections through MergeScan (#10221)
* perf(query): push column projections through MergeScan
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): retain empty projections and refresh sqlness plans
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): update TSID join plans after projection pushdown
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 16123d806b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: adapt staging and prepared-plan regressions to v1.3
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: regenerate MergeScan semi-join snapshot on v1.3
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: restrict JSON retries to invalid UTF-8 errors (#10302)
* fix: restrict JSON retries to invalid UTF-8 errors
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* test: isolate UTF-8 retry counter assertions
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit 09c0291575)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): prepare Ubuntu build runners and default EC2 disks to 500 GiB (#10156)
* ci: share Linux runner dependency setup for builds and benchmarks
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: configure EC2 build runner volumes with a 500 GiB default
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit c02e1f58af)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): increase nightly JSONBench runner root volume (#10146)
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit e80cf9a9ae)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(datatypes): render jsonb stored as JSON text without panics or wrong values (#10197)
* fix(datatypes): decode jsonb before rendering it as text
jsonb_to_string serialized the bytes through RawJsonb's Serialize impl,
which panics with an out-of-range slice index on JSON text such as
"hello world" or " null" (stored by COPY FROM and MySQL text
parameters) and on truncated jsonb. Its text fallback was also stricter than
the extended parser reads used before.
Decode with jsonb::from_slice, as reads did before #10112, then write the
value out with serde_json escaping for keys and strings.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(datatypes): keep streaming jsonb serialization in jsonb_to_string
Decoding into a jsonb::Value tree allocates per array, object and key on
every JSON cell returned over PostgreSQL and MySQL. Go back to serializing
the raw bytes with RawJsonb's Serialize impl, and bump jsonb to a rev whose
Serialize returns an error instead of panicking on out-of-range payloads
(GreptimeTeam/jsonb#2). Bytes that fail to serialize are parsed as JSON
text with the extended parser, as jsonb::from_slice does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(datatypes): pin text values that used to read back as false
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: bump jsonb to reject scalar headers with non-zero length bits
JSON text over ~2 MB could still pass the payload bounds check and read back as a wrong value (GreptimeTeam/jsonb#2).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(datatypes): report the text parse error when JSON bytes are neither jsonb nor text
Also pin jsonb_to_serde_json on JSON text scalars that used to decode as false.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: pin jsonb to the merged commit on main
GreptimeTeam/jsonb#2 reached main through GreptimeTeam/jsonb#3.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit ddc16bafe8)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* revert(mito2): exclude range cache singleflight changes from v1.3.1
Revert b33634417e (#9935) from the release candidate. Preserve the separate SST lifetime fix (#9950). The range cache admission experiment did not restore Q2 performance.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: redact credential table options in user-visible output (#10346)
information_schema.tables.create_options (and SHOW TABLE STATUS) printed
object store credentials of external tables verbatim, and SHOW CREATE TABLE
masked only access_key_id and secret_access_key. Mask all four credential
keys in both paths, and sort extra options in create_options by key.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 62dad907d0)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(mito2): prefilter single-column dynamic filters (#10340)
* perf(mito2): prefilter dynamic join membership
Read the narrow dynamic-membership column before the full projection and use a stable per-row-group mask to select rows. The live filter above the scan stays authoritative; bounds-only and unsupported shapes keep all rows.
Signed-off-by: discord9 <discord9@163.com>
* test: cover dynamic prefilter cache and generations
Prove the prefilter-result cache keeps the static mask but never reuses a dynamic mask across producer generations. Document why a superseded dynamic mask is discarded rather than assumed monotone.
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit c937e3997b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(servers): authorize HTTP database overrides (#10378)
* fix(servers): authorize HTTP database overrides
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): update the HTTP output benchmark handler call
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): authorize Prometheus database overrides
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): retain authorized label lookup scope
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 5d88de5290)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(config)!: redact credentials from diagnostic TOML (#10381)
* fix(config): redact credentials from diagnostic TOML
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(config): redact credentials at their owning fields
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(config): cover skipped SAS tokens and export logs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(cli): mask SAS tokens before overlapping account names
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(cli): redact SAS tokens before overlapping account fields
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(cli): mask containing credentials before shorter values
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit ca1b8c7c98)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: update dashboard to v0.14.6 (#10365)
(cherry picked from commit 2e1ce9808a)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(query): serialize remote dynamic membership (#10341)
* feat(query): serialize remote dynamic membership
Recover single-column UTF-8 build-key membership from InList and HashTableLookupExpr producers and send it to remote datanodes using the existing PhysicalInListNode. Non-monotone or unsupported subtrees widen to true; the receiver keeps the live filter authoritative.
Signed-off-by: discord9 <discord9@163.com>
* refactor(query): send lookup membership as a key list
Serialize a HashTableLookupExpr as the existing PhysicalInListNode and preserve the rest of the expression tree. Remove per-subtree monotone and true rewrites; only an unrepresentable lookup or an over-budget payload falls back at the whole-payload level.
Signed-off-by: discord9 <discord9@163.com>
* docs: tighten narrow membership contract
Clarify that the key-list transport is a positive pruning hint, not universal Boolean equivalence, and test that a NULL probe survives the IS NULL wrapper.
Signed-off-by: discord9 <discord9@163.com>
* fix(query): retain safe bounds on membership fallback
When a key-list predicate cannot be shipped, keep top-level conjuncts that contain no membership and re-encode their conjunction. Drop membership-bearing conjuncts whole; fall back to true only when no safe bounds remain.
Signed-off-by: discord9 <discord9@163.com>
* fix(query): skip unrepresentable dynamic filter snapshots
Do not synthesize a true predicate when a runtime hash-table lookup has no wire representation. Capture and fanout skip the affected snapshot, keep the query running, and report PushedDown::No so the parent filter is retained. Safe membership-free conjuncts still travel, and a producer-supplied true remains encodable.
Cover restrictive-to-true updates, unsupported initial registrations with later supported updates, final unsupported cleanup, and NULL build keys.
Signed-off-by: discord9 <discord9@163.com>
* docs(query): clarify 642-key payload budget comment
Signed-off-by: discord9 <discord9@163.com>
* refactor(query): tighten dynamic membership encoding
Preserve static IN conjuncts when only a runtime lookup is unrepresentable, apply the IN-list exclusion only on the over-budget path, and derive the membership count guard from the byte budget before materializing literals. Document that decoded dynamic filters are pruning-only, fix the NOT/CASE contract wording, deduplicate test helpers, and justify the schema-less InList proto encoding.
Signed-off-by: discord9 <discord9@163.com>
* fix(query): reset stale dynamic filters on unsupported updates
Publish an explicit true reset when a producer generation cannot be carried, instead of leaving an older narrower predicate that could prune rows after widening. Keep the unsupported expression unrewritten; the reset is a transport-built invalidation. Type over-budget build-key memberships as DynFilterPayloadTooLarge, accept scalar-subquery IN lists under the conjunct-retention contract, and clarify the no-initial-predicate wording.
Signed-off-by: discord9 <discord9@163.com>
* docs(query): scope no-true rule to the encoder
Clarify that the encoder never substitutes true for a producer node, while the long-lived fanout path publishes an explicit true reset to invalidate a stale region predicate. Point at the fanout caller instead of claiming producers always skip refused generations.
Signed-off-by: discord9 <discord9@163.com>
* docs(query): correct reset wording
Fix the inverted conjunct-retention sentence and describe initial capture versus ongoing fanout resets consistently in the error docs.
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit f88af153ec)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(auth): secure MySQL and PostgreSQL database selection (#10382)
* fix(mysql): authorize the default database at login
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(auth): validate MySQL handshake scope and SQL USE
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(auth): authorize PostgreSQL schema selection
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): defer PostgreSQL database lookup until authorized
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(servers): propagate catalog lookup errors in protocol tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(postgres): skip unusable search path candidates
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(auth): log database authorization and selection outcomes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(auth): preserve search path responses and schema boundaries
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(auth): clarify database selection diagnostics
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mysql): format handshake username in structured logs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(auth): secure database-less sessions and failed startup
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mysql): retain authorized default database selection
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(auth): separate database selection from request dispatch
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mysql): align schema validation with database-less login
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(auth): explain database-less session restrictions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 6e025eef61)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: bump candidate release version to v1.3.1
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: lczllx <2181719471@qq.com>
Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
Signed-off-by: Byte-Naut <Byte-Naut@proton.me>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Floris Smit <florissmit10@gmail.com>
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: luckys <190197589+luckysxx@users.noreply.github.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Dragon Roar <2181719471@qq.com>
Co-authored-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
Co-authored-by: B. Naut <Byte-Naut@proton.me>
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
Co-authored-by: Minh Vu <vuhoangminh97@gmail.com>
Co-authored-by: Floris Smit <florissmit10@users.noreply.github.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
Co-authored-by: luckysxx <2824602789@qq.com>
Co-authored-by: luckys <190197589+luckysxx@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: Arpit Jain <3242828+arpitjain099@users.noreply.github.com>
Co-authored-by: sun <sunchang_long@163.com>
Co-authored-by: discord9 <discord9@163.com>
* fix(hdfs): clean up temporary file when an atomic write fails (#9483)
* fix(hdfs): clean up temporary file when an atomic write fails
Signed-off-by: fengyue-xve <1759519137@qq.com>
* style: apply rustfmt to hdfs layer test
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(hdfs): report earlier write failures and log cleanup failures
- Later `write`/`close` calls on a writer whose failed write was
cleaned up now mention the original failure instead of the bare
"unavailable after close or abort" message.
- `abort_and_delete` logs abort and deletion failures with the
temporary path instead of discarding them; the original write error
is still returned to the caller.
- The temporary-file cleanup test now lists the directory holding the
temporary file, so it fails if the cleanup is reverted.
Signed-off-by: 风月 <1759519137@qq.com>
Signed-off-by: fengyue-xve <1759519137@qq.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
Signed-off-by: 风月 <1759519137@qq.com>
(cherry picked from commit a35661d69c)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(jaeger): return the most recent traces when a search hits its limit (#9595)
* fix(jaeger): return the most recent traces when a search hits its limit
Trace search applied LIMIT to an unordered DISTINCT trace_id, so a search
matching more traces than the limit returned an arbitrary subset that
changed between identical requests. Order traces by their latest matching
span before applying the limit.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(jaeger): name the trace dataframe builder with a verb and document execution
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit cf16c7313b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(pprof): initialize framehop before signal sampling (#9612)
fix(pprof): pin upstream framehop initialization fix
Pin pprof workspace-wide to 01cff82d containing tikv/pprof-rs#282. Initialize framehop before sampling instead of allocating and discovering shared libraries in the first SIGPROF handler. Regenerate the lockfile without unrelated dependency changes.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit 0dc81777e3)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: bump rustls to 0.23.46 (#9614)
rustls 0.23.46 fixesrustls/rustls#3301: on TLS 1.2 with AES-GCM, reaching the 2^24 record limit dropped a fragment while reporting the write as complete, then kept sending records after close_notify. This stalled Kafka WAL produce requests until the request timeout.
(cherry picked from commit 33879b07c8)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(prom): return empty remote read results for missing metrics (#9613)
* fix: return empty remote read results for missing metrics
Signed-off-by: fys <fengys1996@gmail.com>
* test(prom): cover empty reads for missing databases
Signed-off-by: fys <fengys1996@gmail.com>
* test(prom): simplify remote read regression cases
Signed-off-by: fys <fengys1996@gmail.com>
* fix(prom): log missing tables during remote read
Signed-off-by: fys <fengys1996@gmail.com>
---------
Signed-off-by: fys <fengys1996@gmail.com>
(cherry picked from commit cf0f6c308a)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): compare release validation against the previous stable version (#9636)
Ancestry-based tag selection skips stable tags that live only on older release branches.
Signed-off-by: paomian <xpaomian@gmail.com>
(cherry picked from commit 3de8610e2e)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): keep processing bulk requests after one fails to encode (#9638)
* fix(mito2): keep processing bulk requests after one fails to encode
When `push_bulk` fails to build the WAL entry of a bulk request, it answers
that request with the error. `process_bulk_requests` then returned from the
loop, so the remaining requests of the drained batch were dropped without an
answer and their callers only saw a receive error. Skip the failed request
and continue with the rest of the batch instead.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): keep bulk install barriers independent per target
The bulk install test barrier was a single process-wide slot, so tests
that arm it while running in the same process replaced each other's
barrier. Keep one barrier per armed target instead.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 88bdab0f66)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): honor @ modifier on subqueries (#9446)
* fix(promql): honor @ modifier on subqueries
`prom_subquery_expr_to_plan` never read the `at` field of a subquery, so
`sum_over_time(m[20s:10s] @ 60)` was evaluated like `m[20s:10s]` and
followed the outer evaluation grid.
Handle `@` the way Prometheus does: fold the anchor into the subquery
offset (`offset + (eval_start - anchor)`, as `setOffsetForAtModifier`
does), evaluate the subquery once at the start of the evaluation, and
replay the folded window over the outer grid, reusing the anchored
range selector machinery. A call whose range argument is an anchored
subquery is promoted as step-invariant, like one over an anchored range
selector. `@ start()`/`@ end()` and `offset` combine as for selectors,
and an unrepresentable anchor is rejected with the existing error.
Closes#9411
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* test(promql): add sqlness cases for @ on subqueries
Cover the #9411 reproducer end to end: `@ <ts>`, `@ start()`/`@ end()`,
`@` combined with `offset` in either order, an instant query away from
the anchor, `*_over_time` functions and `rate` over an anchored
subquery, a computed inner expression, and an aggregation above the
anchored call. Two series per table so a dropped or merged series
shows up as a wrong row.
Refs #9411
Signed-off-by: Aarav <aaravsjadav@gmail.com>
---------
Signed-off-by: Aarav <aaravsjadav@gmail.com>
(cherry picked from commit 095c288f16)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): validate bulk insert batches against the region schema (#9698)
* fix(mito2): validate bulk insert batches against the region schema
Bulk inserts were only checked for the time index column, unlike row
writes, which WriteRequest::check_schema checks against the region
schema. A batch with columns of another type, nulls in the time index or
in non-nullable columns, duplicate or unknown columns reached the region
worker: a time index that is not a timestamp or holds only nulls made
the worker fail, and other mismatches failed after the WAL append or
were stored as written, depending on the SST format.
Move the column checks of check_schema into a core shared by row writes
and bulk inserts, and check bulk insert batches at the same points as
rows: when the engine builds the worker request, and again in the worker
if the region schema changed since. check_schema keeps its behavior.
Bulk insert columns must have the column's Arrow type, except that tags
may use any string encoding, as the memtables already convert them.
Sparse batches must carry one encoded primary key column of binary
without nulls. A batch that misses columns gets them filled by the
worker even if it claims to match the current schema.
The metric engine checks logical bulk inserts against the logical region
before rewriting them into the physical region, as the rewrite copies
columns without checking them and would encode a duplicated tag into the
primary key twice.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): report an error when a write is interrupted by a panic
WriteNotify answers its waiter when it is dropped and reported success
unless an error had been set, also when it was dropped while unwinding
from a panic in the middle of the write. Report an error in that case.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): tolerate a poisoned series lock when dropping a series map
SeriesMap::drop reads every series to update the memtable metrics and
unwrapped the lock, so dropping the map while unwinding from a panic
that poisoned a series lock panicked again. Read through the poisoned
lock instead.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 791f429246)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): fence writes before a graceful downgrade (#9716)
* test(mito2): cover graceful role changes racing with writes
A graceful role change flips the region state in a background task while
the worker keeps handling writes. Add bulk and row write tests that check a
write handled after the role change request is either rejected or covered
by the returned last_entry_id.
Add tests for DowngradingLeader while an ALTER keeps the region in
Altering, alone and in the downgrade sequence the datanode runs
(DowngradingLeader, Flush, Follower) with a migration candidate on its own
local WAL. Add a test that the downgrade sequence covers a write in flight
when a lease renewal sets Downgrading.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): fence writes before a graceful downgrade
set_region_role_state_gracefully hands the role change to a background task
that waits for the manifest lock. The worker keeps handling writes until the
task flips the state, so a write could be acknowledged with an entry id above
the returned last_entry_id. Set Downgrading in the worker before spawning the
task for Follower and DowngradingLeader requests, as lease renewal already
does, so every write the worker handles afterwards is rejected.
DowngradingLeader also did nothing and returned success when the region was
in another leader state such as Altering. The region became writable again
when the operation finished, and the datanode then flushed and turned it
into a follower in one step. Return a retryable error instead, so metasrv
retries the downgrade.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): assert the retryable downgrade error during ALTER
Check that DowngradingLeader during an ALTER returns a retryable error, and
release the paused ALTER on that path. The local WAL test now retries the
downgrade after the ALTER, as metasrv does.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): write before the retried downgrade in the local WAL test
Rows written before an ALTER were already flushed, so the candidate check
did not depend on the downgrade's final flush. Write rows after the ALTER
and before the downgrade is retried, which only that flush can carry to the
candidate.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 660c4efe1f)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): validate persisted partition bounds before altering regions (#9487)
* fix(meta): validate persisted partition bounds before altering regions
A partition bound is persisted with the type of the column at the time
it was written. When an ALTER TABLE changes the type of a column
referenced by a partition expression (widening the time index unit is
the common case), each region rewrites the bound literal to the new
type while building its scan predicate. A millisecond bound like year
3000 fits milliseconds but overflows nanoseconds, so every scan of the
region fails once the table schema is updated -- and some regions may
already be altered when the failure surfaces.
Validate every region's persisted expression on prepare, before any
region is modified: parse the persisted JSON (mirrored types, since
common-meta cannot depend on the partition crate which depends on it),
replay the same value -> ScalarValue conversion and checked cast the
region performs, and reject the ALTER naming the offending region when
a bound cannot be represented with the new column type. Incompatible
changes are rejected atomically; compatible widening proceeds unchanged.
Adds meta-level tests (multi-region rejection naming the second region,
compatible widening completing) and an end-to-end sqlness case.
Fixes#9454
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix: access ColumnSchema fields directly and use plain format args
ColumnSchema exposes name and data_type as public fields, not methods,
and the telemetry macros only accept a format string after the error.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: access ColumnSchema data_type field in alter table tests
Same fix as the previous commit: data_type is a public field on
ColumnSchema, not a method.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop borrows now that data_type is read as a field
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop useless format! in nested expression test
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: cover the partition tail in change_col_type sqlness case
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop spurious blank line in change_col_type result
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(meta): compare partition bounds with the scan comparison coercion
The preflight validator cast the persisted bound directly to the altered
column type, which rejected ALTERs the scan itself accepts: the region
coerces both the column and the bound to the common comparison type
(ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field
column with an out-of-range bound still scans. Compare through
comparison_coercion(target_type, bound_type) instead and only require the
bound to be castable to the common type.
Also add a round-trip test in the partition crate that serializes real
PartitionExprs and runs them through the validator, so drift between the
Persisted* mirror and the partition types fails CI, and fix the error
message to match the comparison semantics.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: update the sqlness fixture for the new incompatibility message
The error message now reflects the comparison semantics: the persisted
bound can no longer be compared with the new column type.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency
common-meta gained a datafusion-expr dependency in ff9c3f4 but the lock
file was not refreshed, so cargo check --locked refused to run on CI.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(meta): reject every non-widening type change of a partitioned column
The region routes rows on write through `MultiDimPartitionRule::find_region`,
which compares the persisted bound against row values directly: `Value::cmp`
panics on mismatched variants and equality misroutes silently. After a column
partitioned at `v < 3000000000` is altered from BIGINT to INT, every INSERT
compares `Value::Int32` rows against the persisted `Value::Int64` bound and
panics in the frontend, even though the scan-side coercion this check mirrors
still works.
Reject any type change of a column referenced by a persisted partition
expression unless it is a timestamp unit widening, which keeps the bound and
the rows comparable because `Timestamp` comparisons convert units first; the
widening keeps the scan cast check unchanged.
Signed-off-by: fengyue-xve <1759519137@qq.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
(cherry picked from commit 1e29e8d4b6)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(partition): route NULLs like row inserts and accept signed-minimum bounds (#9697)
* fix(partition): route NULLs like row inserts and accept signed-minimum bounds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): keep equality two-valued for NULL rows and accept unsigned -0
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(partition): treat NULL masks as false in batch routing and keep comparisons bare
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(compat): update partition filter in analyze verbose expected output
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit da88ff8653)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): re-check a filled write as sent after a schema change (#9701)
* fix(mito2): re-check a filled write as sent after a schema change
The engine fills the columns a write leaves out with their defaults when it
accepts the write, appending them to the rows without adding them to
`name_to_index` and `has_null`. If the region schema changes before the
worker handles the write, the worker checks the request again and
`check_schema` looks up a filled column in `name_to_index`, which panics and
stops the region worker. If the filled column was dropped, the write fails
with an unknown column that the client never sent.
Remove the filled columns before checking, so the request is checked as sent
and filled again for the current schema. Nothing is filled yet when the
engine checks the request, so that check does no extra work.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito2): rename put_request helper to build_put_request
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 93c3b116da)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(mito2): gather last rows with one take per input batch (#9737)
* perf(mito2): gather last rows with one take per input batch
FlatLastTimestampSelector buffered a single-row slice per series and
concatenated up to 8192 of them. Concatenating slices with dictionary columns
merges the dictionaries once per slice, which made high-cardinality instant
PromQL queries using the LastRow scan path (#9034) slower. Select the rows of
series that end inside a batch with one take instead.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito2): cover duplicate last timestamps in the take path
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): group last rows by primary key bytes
Flat batches may assign different dictionary keys to the same primary key, for
example when binary keys are wrapped with identity keys. Merge adjacent ranges
with equal key bytes before selecting last rows.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 4645ab9ed7)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): apply SST internal column encodings to bulk-encoded parts (#9735)
* fix(mito2): apply SST internal column encodings to bulk-encoded parts
Bulk memtable parts are encoded to parquet and flushed as SSTs as-is, but the
bulk encoder did not apply the time index / __sequence DELTA_BINARY_PACKED
encoding (with dictionary disabled) and the uncompressed op_type setting that
the regular SST writer uses. Share these settings through one helper.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito2): drop the answered bulk encoder compression TODO
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 977e76d0a4)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: avoid exact scan row counts for SSTs with unknown counts (#9759)
fix: avoid exact scan row counts for legacy SSTs
Treat selected SSTs with unknown row counts as non-exact before checking region partition visibility. Preserve exact statistics for known counts and cover legacy metadata mixed with known-count files.
FixesGreptimeTeam/greptimedb#9718
Signed-off-by: Aumnertic <hznuzc@gmail.com>
(cherry picked from commit 6f53d57d15)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: avoid duplicate keys in update flow metadata txn (#9816)
CREATE OR REPLACE FLOW deleted and then put the same flow route,
flownode flow and table flow keys in a single txn when the replacement
kept the partitions, flownodes or source tables, which etcd rejects
with "duplicate key given in txn request".
Only delete the old keys that are not put again. Closes#9810.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit a1b2362e58)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): increase release integration test stack size (#9817)
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit e6fa1a0b21)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(datanode): release region map guard before awaiting role changes (#9824)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 0dc649e028)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): align nightly Windows and macOS artifact test stack sizes (#9860)
* fix(ci): increase Windows nightly test stack size
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(ci): align optional macOS artifact test stack size
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 8b1d68369b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): keep SSTs when a manifest save ends with RateLimited or PermissionDenied (#9898)
* fix(mito2): keep SSTs when a manifest save ends with RateLimited
The object store's retry layer retries temporary errors and returns the
last attempt's error. A manifest delta PUT can be stored by the server
while the client sees a timeout, reset or 5xx, and the retries that
follow can be throttled, so the save ends with RateLimited even though
the delta exists. may_have_persisted_manifest_update treated RateLimited
as proof that nothing was written, so flush and compaction deleted the
SSTs the stored delta references. A later open or follower sync replays
the delta and reads fail on the missing files.
Treat RateLimited like other ambiguous errors: the SSTs are kept and
become orphans when the save really failed.
Fixes#9897
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): keep SSTs when a manifest save ends with PermissionDenied
The retry layer returns the last attempt's error, so a 403 from a revoked
key, a changed bucket policy or clock skew can follow a delta PUT that
the server already stored. Like RateLimited, PermissionDenied does not
prove that nothing was written, so it no longer lets flush or compaction
delete the SSTs; they become orphans when the save really failed.
The compaction test that rejects a manifest write before persistence now
injects Unsupported, which is raised before any request is sent.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
(cherry picked from commit 1cb01d0d44)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: upload staging manifests in chunks (#9919)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit f2be11a69e)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf: stream and prune SST manifest scans (#9931)
* refactor: add reusable metadata pruning and projected SST batches
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(mito2): stream and prune manifest SST metadata
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(datanode): stream inspection queries through reusable providers
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(catalog): push manifest filters through inspection queries
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor: align manifest column constants with module conventions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf: reduce manifest inspection allocations
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor: narrow manifest inspection API visibility
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor: retain residual filters for manifest inspection
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit c96cdd0026)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: improve SQL analyze scan diagnostics (#9565)
* feat: improve SQL analyze scan diagnostics
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: redact dynamic filters in sqlness results
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: stabilize analyze snapshots for merge queue
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: remove obsolete scan_details assertion
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: refresh explain snapshots after partition filter changes
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 3e2751d71d)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(query): avoid lossy casts in partition pruning (#9600)
* fix(query): keep lossy casts out of partition pruning
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): align cast pruning expectations with conservative fallback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): record lossy cast partition pruning regression
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): retain safe pruning conjuncts beside unsupported casts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): refresh cast pruning plans after rebase
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit df8f6848cc)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(query): preserve IGNORE NULLS semantics in distributed queries (#9601)
* fix(query): skip LastRow pushdown for IGNORE NULLS
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): keep IGNORE NULLS aggregates above remote scans
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): record IGNORE NULLS results before and after flush
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): share IGNORE NULLS coverage across deployments
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): refresh last-value plans after rebase
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 0dab3aef21)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): share peer discovery with plugin hooks (#9977)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 9f0e58e2af)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: adapt release backport test dependencies
Copy the writer observation test utility from #9505 (18726c7602) for the #9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: bump version to 1.3.0
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(query): pin DataFusion cast constraint propagation fix
Update the pinned GreptimeTeam/datafusion revision from e3221b8 to
cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger
regression test).
CastExpr::propagate_constraints used to cast a filter's output interval
straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton
output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so
FilterExec advertised k as constant, the equivalence classes dropped the
ordering required by ORDER BY k, and rows returned in scan order.
The sqlness case keeps every #9600 assertion and the explicit ORDER BY,
and adds ascending and descending checks for the partitioned table, the
control table and a table whose matching rows are stored in neither
ascending nor descending k order.
Cargo.lock only rewrites the 35 datafusion source lines.
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit fe637b04ce)
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
Signed-off-by: 风月 <1759519137@qq.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: paomian <xpaomian@gmail.com>
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
Signed-off-by: Aumnertic <hznuzc@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: fengyue-xve <1759519137@qq.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: localhost <xpaomian@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
Co-authored-by: DeviousCardi <115358213+DeviousCardi@users.noreply.github.com>
Co-authored-by: Aum <hznuzc@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: discord9 <discord9@163.com>
* ci(query-regression): bump RUNNER_IMAGE_EPOCH to 7 (#9395)
ci(query-regression): bump RUNNER_IMAGE_EPOCH to 7 for image m-0xihbfzm5xpbxolybo6i
Signed-off-by: greptimedb-ci <greptimedb-ci@greptime.com>
Co-authored-by: greptimedb-ci <greptimedb-ci@greptime.com>
(cherry picked from commit 886e02f0dc)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): apply offset to subquery evaluation window (#9364)
* fix(promql): apply offset to subquery evaluation window
`prom_subquery_expr_to_plan` destructured `SubqueryExpr` without reading
`offset`, so `<subquery>[range:step] offset <d>` planned exactly the same
window as the un-offset form and silently returned data for the wrong time
range. The plain vector/matrix-selector paths already threaded the offset
through `selector_to_series_normalize_plan` and `RangeManipulate`.
Shift the inner evaluation window back by the offset and pass the offset to
the subquery's `RangeManipulate`, which maps the inner samples forward onto
the evaluation timeline before bucketing them into ranges. This matches
Prometheus, whose `evaluator.subqueryTimeRange` evaluates the inner
expression over `(start - offset - range, end - offset]` and whose
`evalSubquery` then hands the samples to the outer range-vector function as
a `MatrixSelector` that still carries the subquery offset. An offset on the
inner selector composes additively, as `subqueryTimes` documents.
`RangeManipulate`'s protobuf message has no offset field and recovers it on
decode from an immediately underlying `SeriesNormalize`. Since
`RangeManipulate` is commutative in `dist_plan` and can be pushed below a
`MergeScan`, insert that carrier node so the offset survives distributed
planning instead of decoding as zero.
Known divergence, unchanged by this commit: Prometheus anchors subquery step
points on absolute epoch multiples of the step, while GreptimeDB anchors them
on the evaluation start. The two agree whenever the offset is a multiple of
the subquery step; the added sqlness cases stay within that range.
Closes#9330
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* docs(promql): correct the subquery-offset rationale and pin the histogram shift
Follow-up to e132c267, which stated two things inaccurately and left one
path untested. No behaviour change beyond a conditional clone.
The `SeriesNormalize` carrier was described as existing "so the offset
survives distributed planning", implying standalone is unaffected. That is
wrong. `local_offset` is read on the substrait decode path
(`range_manipulate.rs`, `instant_manipulate.rs`), and GreptimeDB routes
PromQL plans through `MergeScan`/substrait in standalone too -- the
standalone results `promql/encode_substrait.result` and `precisions.result`
both show `MergeScan [is_placeholder=false, remote_input=[...]]`. Deleting
the carrier therefore empties subquery-with-offset results in standalone as
well, while `count_over_time_subquery_with_offset` keeps passing because it
asserts the pre-serialization plan. Say so at the node, so the next reader
does not remove it believing standalone is safe.
The carrier is also not semantically inert: `SeriesNormalizeStream::normalize`
shifts native histogram `start_timestamp` payloads by `offset`. On the
subquery path that runs on a computed inner result, on top of any offset the
inner selector already applied, and nothing pinned it. It is consistent --
`RangeManipulate` moves the millisecond time index forward by the same
`offset_ms` -- so the distance from a histogram's start timestamp to the
sample carrying it, which reset/rate detection reads, is invariant. Add
`subquery_offset_shifts_histogram_start_and_time_index_together`, which
executes the subquery's node stack over native histogram samples for a
zero and a non-zero inner offset. It cannot be a sqlness case: native
histograms are a struct column with no SQL type or literal
(`sql_data_type_to_concrete_data_type` rejects structs) and the sqlness
runner speaks only MySQL/Postgres, so such rows only arrive over
gRPC/remote-write v2.
The divergence note blamed sub-step offsets. Measured, that condition is too
narrow: an unaligned evaluation timestamp alone diverges, with no offset at
all -- `sum_over_time(fine[20s:10s])` at t=57 samples 47s and 57s here
against Prometheus's 40s and 50s. The real condition is that
`(start - offset)` is not a multiple of the subquery step, and it is
pre-existing: the committed `tql eval (359, 359, '1s')
sum_over_time(metric_total[60s:10s])` case is already an instance of it
(`359 - 60 + 10 = 309`). Reword the comment and the sqlness header
accordingly. A sub-step offset stays accepted: `foo[20s:10s] offset 5s` is
valid PromQL, erroring on it would be a regression and would not close the
gap anyway. This repo has no PromQL-compatibility documentation page, so
there is nowhere else to record it.
Also stop cloning the series key columns when there is no offset, and stop
the `offset 0s` sqlness comment implying its parser error is evidence about
this fix -- the same message comes from a `u32` overflow in
`promql-parser`'s shared duration check (`offset 9999999999d`).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* refactor(promql): trim subquery-offset comments and tests per review
Shorten the planner and sqlness comments to the semantic contract, build
`SeriesDivide` with a clone instead of an `Option`, and drop the
normalize.rs histogram test and the `offset 0s` parser case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* fix(promql): record the subquery offset as the range fold offset
The subquery's `RangeManipulate` now shifts the payload timestamps by the
offset, so `predict_linear` over an offset subquery must recover the
evaluation time with that offset instead of 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>
---------
Signed-off-by: Aarav <aaravsjadav@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit d6474b961c)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: retry failed nightly release on following weekdays (#9394)
Previously the nightly release was scheduled only on Mondays, so a
failed run left users without nightly builds for a whole week.
Now the schedule triggers every weekday at 00:00 UTC, but the release
only proceeds when the latest published nightly release is older than
NIGHTLY_RELEASE_MAX_AGE_DAYS (5) days. The check runs in
allocate-runners before any EC2 runner is allocated, and gates all
schedule-driven jobs (including the Slack notification, so a skipped
nightly is silent). Tag pushes and manual dispatches are unaffected.
Signed-off-by: Ning Sun <sunning@greptime.com>
(cherry picked from commit a7f99b3425)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(client): yield Flight batches and affected rows without waiting for next message (#8918)
* fix(client): avoid Flight metrics lookahead stalls
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(client): extract trailing Flight metrics task
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(client): synchronize trailing metrics and bound cancellation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit abf1396c28)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): resolve dotted column names as unqualified columns (#9391)
* fix(promql): resolve dotted column names as unqualified columns
col(), From<&str>/From<String> for Column and string join keys go through
Column::from_qualified_name, which splits `service.name` into relation
`service` and column `name` and lowercases unquoted identifiers. Build
PromQL column references with Column::from_name / ident() instead.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): resolve remote read matcher labels as unqualified columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover same-name columns differing in case and without()
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 218000e21b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): align timestamp(), label_join and label_replace with Prometheus semantics (#9385)
* fix(promql): align timestamp() and label_join with Prometheus semantics
- timestamp() over a selector reports the selected sample's timestamp, without adding the offset.
- timestamp() over any other expression reports the evaluation time instead of the input value.
- label_join that overwrites an existing label rejects duplicate label sets at runtime.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): align label_replace and label_join edge cases with Prometheus
- label_replace leaves non-matching series unchanged instead of copying the source value into the destination label.
- label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join.
- label_join with no source labels removes the destination label, and an empty source label name is rejected.
- Empty label values produced by these functions are NULL, the same as an absent label.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): join absent labels as empty strings in label_join
concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): skip rows without a sample in timestamp()
timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 78a7b93292)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): truncate parquet column index min/max for SST writes (#9420)
* fix(mito2): truncate parquet column index min/max for SST writes
The SST writer and the bulk part encoder disabled column index truncation
together with statistics truncation in #6977. Mito never reads the column
index, but every page of a large string or binary column still stored its
full min and max values, uncompressed. Truncate the column index to the
parquet default of 64 bytes and keep chunk statistics untruncated, since
primary key pruning decodes them.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(mito2): note that the column index must not be used to decode primary keys
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit a314ac4281)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: upgrade EC2 runner action to restore runner registration (#9430)
ci: update EC2 runner action for minimum runner version enforcement
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 0d502e1fc1)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(catalog): skip backend batch_get on full cache hits (#9431)
fix(catalog): return cached batch values without an empty backend call
Count backend batch_get invocations in regression tests and cover empty input.
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit acebd6b35b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: make query regression non-blocking for scheduled nightly releases (#9434)
* ci: make query regression non-blocking for scheduled nightly releases
Scheduled nightly releases no longer gate publishing on the query
regression release test. The test still runs (validation policy stays
'all' for automatic releases) and uploads its report, but any result -
failure, timeout, cancellation, or skip - no longer blocks image
publishing or the GitHub release for schedule-triggered runs.
Since reusable-workflow caller jobs cannot use continue-on-error, the
gate is relaxed at the consumers instead: the if conditions of
release-images-to-dockerhub and publish-github-release now also pass
when github.event_name == 'schedule' regardless of the query-regression
result. All other gates (runner allocation, artifact builds,
prepare-release-validation) are unchanged.
Tag-push and manual-dispatch releases remain fully blocking unless a
skip policy is explicitly chosen via the release_validation input.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci: preserve downstream nightly release jobs after regression failures
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit a9ce6e2d01)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: coalesce equal histogram quantile bucket boundaries (#9451)
* fix: coalesce equal histogram quantile bucket boundaries
Merge numerically equivalent bucket boundaries before monotonicity repair, preserving the allocation-free path for unique boundaries. Share coalescing with histogram_fraction and cover duplicate boundaries with unit and SQL regressions.
Fixes#9443
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: keep duplicate infinity buckets in one histogram group
Confirm classic histogram completion at a series/timestamp boundary or EOF instead of the first positive infinity. Retain lookahead across batches and reject incomplete optimistic groups so equivalent infinity labels reach numeric coalescing together.
Add stream and SQL regressions for duplicate infinity labels, including established bucket counts and EOF. Correct older fixtures to use distinct keys for distinct histogram groups.
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: avoid buffering histogram groups with invalid trailing bounds
Discover candidate bucket counts at the first group boundary and let optimistic validation fall back to safe mode for malformed bounds. Document discovery and cover trailing invalid and null labels with a focused streaming regression.
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 97913b4b3a)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(sqlness): wait for mirrored inserts before flushing streaming flows (#9424)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit b78dc4b33b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: accept a bool value for the CLI --auto-create-schema flag (#9450)
* fix: accept a value for the CLI --auto-create-schema flag
clap derives ArgAction::SetTrue for bool fields, so --auto-create-schema
took no value and could only set true, which is already the default.
Users could not turn automatic schema creation off from the CLI, unlike
metasrv's auto_create_schema = false.
Set action = clap::ArgAction::Set so the flag accepts an explicit
true/false value while keeping true as the default.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* chore: enhance CLI argument for auto_create_schema option
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Signed-off-by: fengyue-xve <1759519137@qq.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
(cherry picked from commit 329edc88ec)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(otlp): coerce empty and "-" trace attribute strings to null (#9461)
* fix(otlp): coerce empty and "-" trace attribute strings to null
Instrumentations fill numeric attributes from absent headers with an empty
string (PSR-7 getHeaderLine returns ""). Coercing that into an existing
numeric column failed and rejected the whole span.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: trim redundant row from empty-string rewrite test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit d640ab8bc5)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta-client): bound heartbeat handshake timeout (#9438)
* fix(meta-client): bound heartbeat handshake timeout
Signed-off-by: Aetherance <inkToAether@gmail.com>
* chore(meta-client): fix test license header
Signed-off-by: Aetherance <inkToAether@gmail.com>
* test(meta-client): prefix heartbeat tests with test
Signed-off-by: Aetherance <inkToAether@gmail.com>
---------
Signed-off-by: Aetherance <inkToAether@gmail.com>
(cherry picked from commit 9c69e381ae)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* docs(config): align example configs and config.md with actual defaults (#9460)
* docs(config): align example configs and config.md with actual defaults
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(cmd): compare example defaults inside optional sections
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit e2fd8c5695)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(partition): preserve region mapping during repartition (#9458)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 90f117b957)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): restore exact filtering for nullable label matchers (#9418)
* fix(promql): restore exact filtering for nullable label matchers
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: verify compound filter semantics and last-row selection
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: consolidate label filter regression coverage
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: separate filter evaluation and timestamp adaptation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: inline predicate and timestamp tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: keep direct IN predicates outside simple filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: coerce partition bounds before simple filtering
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover partition bound overflow after timestamp widening
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 526dd48bda)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update rust toolchain to 1.98.1 (#9429)
(cherry picked from commit 7f0f592b6b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: update dev-builder image tag (#9469)
Signed-off-by: greptimedb-ci <greptimedb-ci@greptime.com>
Co-authored-by: greptimedb-ci <greptimedb-ci@greptime.com>
(cherry picked from commit 5affb50bb5)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(metasrv): retry failed heartbeat address updates (#9437)
* fix(metasrv): retry failed heartbeat address updates
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(metasrv): move shared heartbeat helpers to utils
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 65b9a35692)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): bound FlowMirrorTask pending rows to prevent frontend OOM (#9421)
* fix(operator): bound FlowMirrorTask pending rows to prevent frontend OOM
When flownode is slow or unreachable, FlowMirrorTask::detach spawns
unbounded tasks that each pin a full insert batch for ~10s (gRPC
timeout). With continuous writes, in-flight memory grows unboundedly
(throughput x 10s x peers), causing frontend-write OOMKill and a
million-per-hour ERROR storm.
Add a MAX_MIRROR_PENDING_ROWS cap (1M rows) on Inserter: detach
fetch_adds the batch's row count before spawning; if the cap is
exceeded the reservation is rolled back, the batch is dropped, a
DIST_MIRROR_DROPPED_ROW_COUNT metric is bumped, and a warn is logged
instead of spawning. On normal completion the reservation is released
in both success and error paths.
This keeps core writes alive while degrading flow mirror writes to
best-effort, which is the intended trade-off: protect the primary
insert path at the cost of possible flow aggregation gaps.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix: address review - balanced reservation, gauge race, log rate limit
P1: Compute pending_rows() from actual per-peer payloads in
self.requests instead of the pre-fanout num_rows field. This fixes
under-reservation for same-table multi-region batches and over-release
in multi-flownode fanout. Each peer task now releases exactly its own
share so reservations and releases balance.
P2: Replace load+set gauge publication with add/sub symmetry.
reserve_mirror_pending_rows does fetch_add + gauge.add, release does
fetch_update + gauge.sub, eliminating the stale-value race.
P3: Rate-limit the dropped-batch warning to at most once per 10s via
MirrorDropLog (atomic compare_exchange), accumulating batch/row counts
between reports.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: fix race in dual-peer test - wait for gauge release before unlocking
The dual-peer test previously only waited for the pending budget to
return to its start value before releasing MIRROR_METRIC_TEST_LOCK.
But release_mirror_pending_rows moves the budget before the gauge, so
a worker's gauge.sub could still be in flight when the next test
acquires the lock and records its baseline, polluting the result.
Now waits for both pending == start AND gauge == gauge_before.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(operator): preserve instant writes and limit mirror failure logs
Reject saturated requests containing instant-TTL rows with the existing retryable write-admission error before mirror or datanode dispatch. Keep normal source-table inserts independent of best-effort mirror drops.
Reuse a smaller process-wide event limiter for both dropped batches and failed peer tasks, reporting at most once per 10 seconds with failure count, peer context, and error chain. Cover actual mirror construction, mixed and instant admission, fast/slow peer shares, failed-task release, and log concurrency.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(operator): reject oversized instant mirror batches permanently
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 198d92407c)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): align subqueries to the absolute evaluation grid (#9400)
* fix(promql): align subqueries to the absolute evaluation grid
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(promql): streamline subquery comments
Remove redundant arithmetic narration and restated test comments. Keep grid, fallback, validation and overflow contracts, and preserve all regression tests and assertions. Regenerate SQLness output only for comment echoes.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 0d2cb1aa7e)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* build(deps): update opensrv-mysql to fix MariaDB 12.3 USE information_schema hang (#9357)
Update opensrv-mysql from f3135e5 to 1f3d88c, which replies a proper EOF packet to COM_FIELD_LIST instead of a malformed OK packet that MariaDB 12.3 misparses and hangs on USE information_schema.
Fixes#9158
Signed-off-by: tian1220A <1573612521@qq.com>
(cherry picked from commit 21536a092b)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): keep vector labels when a computed scalar is on the right (#9478)
* fix(promql): keep vector labels when a computed scalar is on the right
`metric + time()` or `metric * scalar(other)` planned the join with the
scalar side's context, which has no tags, so every series collapsed into
one unlabeled series. Use the vector side's context when the right operand
is scalar-typed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): update results for labels kept with a computed scalar on the right
v * pi() and v + time() now keep the vector's labels and time index name,
as in Prometheus.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): qualify columns of an unnamed operand in vector joins
A comparison filter's projection has no table name, so after the join its
time index resolved unqualified and was ambiguous when the other operand had
the same column. Also keep the empty-metric branch unchanged for scalar
left operands.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover comparison filters joined with a scalar
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 6da5352c53)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(otlp): convert boolean metric attributes into label values (#9468)
Signed-off-by: fengyue-xve <1759519137@qq.com>
(cherry picked from commit 2ebb03fbf6)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(catalog): optimize single-table lookups in information_schema.partitions (#9473)
* perf(catalog): optimize single-table lookups in information_schema.partitions
Signed-off-by: 风月 <1759519137@qq.com>
* style: apply rustfmt formatting to partitions tests
Signed-off-by: 风月 <1759519137@qq.com>
---------
Signed-off-by: 风月 <1759519137@qq.com>
(cherry picked from commit 4ade13d9b6)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(promql): cover label_replace normalization with distinct series (#9485)
The duplicate-label-set rejection must not fire when a label rewrite keeps
every series distinct through another label. Normalizing mixed histogram
boundary representations (`10.0` -> `10`) while `instance` still differs is
the false positive reported in #9444. The check already accounts for it
since it became a per-timestamp row count instead of a planning-time
rejection, but no fixture covered the positive case. Add one, together
with the aggregation variant that produces a genuine duplicate and must
still fail.
Signed-off-by: fengyue-xve <1759519137@qq.com>
(cherry picked from commit 764ab8bc1a)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): return no point for zero-interval irate/idelta (#9455)
fix(promql): irate/idelta return no point for a zero-ms sample gap
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
(cherry picked from commit 0dde3a7b70)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(metasrv): stop parking the heartbeat response on the region supervisor queue (#9474)
* fix(meta-srv): stop parking the heartbeat response on the region supervisor queue
Signed-off-by: lczllx <2181719471@qq.com>
* test(meta-srv): cover the heartbeat response path at the handler level
Signed-off-by: lczllx <2181719471@qq.com>
---------
Signed-off-by: lczllx <2181719471@qq.com>
(cherry picked from commit d1151b0ed6)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): keep metric name for comparison operators (#9332)
* fix(promql): keep metric name for comparison operators
`collect_metric_names` dropped the response `__name__` for every binary
operator except `and` / `or` / `unless`, so comparisons such as
`node_load1 > 5` came back without the metric name. Grafana rendered them
as `{instance="..."}` and alert templates reading `$labels.__name__` got
nothing.
Follow Prometheus' `shouldDropMetricName` / `resultMetric`: arithmetic
operators drop the name, comparisons keep the left-hand name, except when
the comparison returns a bool or is a one-to-one `on(...)` match. A
`group_left` / `group_right` modifier does not reduce the labels, so the
name is kept there.
Both the HTTP PromQL API and the gRPC gateway share
`retrieve_metric_name_and_result_type`, so one fix covers both.
Signed-off-by: Aumnertic <hznuzc@gmail.com>
* docs: document binary_keeps_metric_name
Signed-off-by: Aumnertic <hznuzc@gmail.com>
* fix(promql): handle on(__name__), scalar lhs and group_right
Signed-off-by: Aumnertic <hznuzc@gmail.com>
* chore(promql): correct result-side docs and drop unreachable String arm
Signed-off-by: Aumnertic <hznuzc@gmail.com>
* fix(promql): drop metric name when explicitly ignored
Handle ignoring(__name__) for one-to-one comparisons and cover
comparison, group_left, group_right and unless behavior.
Signed-off-by: Aumnertic <hznuzc@gmail.com>
---------
Signed-off-by: Aumnertic <hznuzc@gmail.com>
(cherry picked from commit c33a6cfa56)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(promql): reduce timestamp payload allocations and use narrow cursor scans (#9417)
* perf(promql): zero-copy timestamp payload for millisecond zero-offset ranges
RangeManipulate rebuilt the timestamp range payload per sample through
wide-precision i128 conversion even for millisecond input with zero
offset, where the input is already millisecond and nothing changes.
That turned a previously zero-copy path into a per-sample loop plus a
Vec allocation, regressing range-function queries.
Reuse the input timestamp array directly when the time index is already
millisecond and the offset is zero. Other units and non-zero offsets
keep the wide-precision conversion (required so ns/us data near the
native type limits stays correct under large offsets).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): vectorize timestamp payload with plain i64
RangeManipulate rebuilt the timestamp range payload per sample through
wide-precision i128 conversion, turning a hot path into a scalar loop
with a Vec allocation. Realistic timestamps and offsets stay far inside
i64, so the wide path is unnecessary.
- millisecond + zero offset reuses the input array with zero copy;
- other units / offsets shift in one vectorized i64 pass (no i128, no
per-sample Option, no overflow branch).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): check timestamp payload overflow via arrow try_unary
Replace the silent wrapping shift with try_unary + checked_mul/checked_add.
checked arithmetic lowers to a per-lane overflow flag so the single pass
still vectorizes over the dense i64 buffer; any overflow short-circuits to
an error instead of silently wrapping to a wrong timestamp. No i128, no
per-sample Option.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): vectorize cursor scans with i64 fast path
RangeManipulate::calculate_range and InstantManipulate::manipulate
compared every sample through i128 wide-precision arithmetic. For
realistic timestamps and offsets the shifted values stay far inside
i64, so the wide path was pure overhead.
Split both scans into an i64 fast path (checked arithmetic precomputed
once per batch) and an i128 exact fallback. Any input that would
overflow i64 falls back to the i128 scan, keeping window membership
semantics identical.
Verified: 47/47 range_manipulate + instant_manipulate tests, 275/275
promql crate tests, 1612 randomized cases byte-identical to i128 scan.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* refactor(promql): generalize cursor scans and share endpoint precheck
Address review comments on PR #9417:
- Generalize scan_ranges/scan_instants over Nanoseconds (i64/i128),
eliminating the duplicated scan bodies.
- Extract shared narrow_scan precheck used by both RangeManipulate and
InstantManipulate, with identical checked endpoint logic.
- Trim doc comments to one line; fix factual errors about fallback
trigger (2262 for ms/s, not native type limits).
Semantics unchanged: same cursor algorithm, same overflow fallback,
same i64-vs-i128 selection, same stale/null/tie handling.
Verified: 47/47 range_manipulate + instant_manipulate tests, clippy clean.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): vectorize payload with try_unary and add scan carrier bench
Replace the per-sample i128 payload loop with a single arrow try_unary
pass: i128 intermediate holds any native timestamp shift, only the final
millisecond result must fit i64. This is semantically equivalent to the
original i128 implementation (no silent wrap, no per-sample branching).
Add cursor_scan_carriers A/B benchmark: i64 fast path vs i128 exact
fallback on four workloads. Add five deterministic differential tests
pinning scan_ranges::<i64> and scan_ranges::<i128> to be pointwise
equivalent on boundary shapes (sub-ms samples, fallback triggers,
nanosecond extremes, seeded precision matrix).
Bench (N=4096, 15s cadence):
- payload wall_time 20x15s: 74.94us -> 48.98us (-35%)
- payload wall_time 240x15s: 102.44us
- scan i64 fast path: 10.97us vs i128 19.84us (1.81x)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): restore zero-copy payload for ms+offset=0 and fix comment
Restore the zero-copy fast path for millisecond timestamps with zero
offset, guarded by the input column's exact data type (Timestamp
(Millisecond, None)) rather than self.time_unit. The old time_unit guard
would accept a tz-carrying millisecond column and produce a dictionary
value type disagreeing with the output schema; the datatype guard is the
correct check.
Also fix the comment: arrow's try_unary is a fallible kernel that LLVM
cannot auto-vectorize; the win over the hand-written loop comes from
eliminating the per-sample Option/Vec and branches, not SIMD.
Add a regression test with a timezone-carrying millisecond index to
prove it falls back to the try_unary path.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): replace try_unary with unary + Cell overflow flag
Arrow's try_unary is a fallible kernel that LLVM cannot auto-vectorize.
Replace it with the infallible unary kernel: the closure is a plain
i64 -> i64 mapping that sets a Cell<bool> on overflow instead of
returning Err per sample. The flag is checked once after the loop.
This removes the per-lane Result construction and lets LLVM vectorize
the shift+truncate sequence. Bench (N=4096, 15s cadence):
- window4x15s: 42.12us -> 28.77us (-32%)
- window20x15s: 48.98us -> 27.68us (-43%)
- window240x15s: 102.44us -> 27.61us (-73%)
- regular_sparse: 90.04us -> 29.20us (-68%)
All cases now run at ~27-29us regardless of window size; the zero-copy
path dominates for ms+offset=0.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): dispatch payload conversion on null_count; drop wall-time bench
Null-free batches (dominant PromQL case) take arrow's vectorizable unary
kernel with a Cell<bool> overflow flag; nullable batches take try_unary so
a NULL slot's arbitrary underlying value cannot raise a false batch-wide
overflow error. Add a regression test exercising both paths.
Remove bench_range_manipulate_wall_time (all-ms zero-offset cases only
exercised the zero-copy path, measuring nothing about the conversion) and
its orphaned helpers; restore the cursor-scan bench helpers it shared.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(promql): keep nullable timestamp cursor scans on i128
NULL slots may contain unordered arbitrary values, so the endpoint-only
narrow-scan proof cannot bound their interior timestamp arithmetic. Both
range and instant scans now attempt i64 only for null-free timestamp arrays
and otherwise retain the existing exact i128 path.
Add focused interior-NULL regressions through both active entrypoints,
checking concrete selections and output against the existing wide scan.
The null-free fast path and timestamp payload conversion are unchanged.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit b3c756a7dd)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(metasrv): stop holding the pusher registry lock across mailbox sends (#9445)
fix(meta-srv): stop holding the pusher registry lock across mailbox sends
Signed-off-by: lczllx <2181719471@qq.com>
(cherry picked from commit 7605f27608)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito): flush memtables containing skip-WAL writes on region close (#9440)
* fix(mito): flush unlogged writes when closing regions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito): align skip-WAL naming and metric recovery tests
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit a026b3b280)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: enhance benchmark workflows with version comparisons and diagnostics (#9383)
* ci: update Agent and Traces benchmark runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use unrestricted benchmark runtimes by default
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: retry benchmark image pulls before generating datasets
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: increase tracesbench defaults to 8 CPU and 32 GiB
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: configure tracesbench query timeout and update runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: update tracesbench runtime and configure import scan workers
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use shared parallel tracesbench import preparation
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: update tracesbench runtime to preserve Tempo trace requests
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: summarize tracesbench query results and lifecycle
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: show benchmark resource usage and update parallel loader runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): resolve shared summary module in existing test harness
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: select tracesbench engines and GreptimeDB image version
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: update benchmark runtime with parallel Agent preparation
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: select benchmark runtime registry by cloud provider
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: retry long-range benchmark image pulls
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: compare baseline and candidate GreptimeDB benchmarks
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: pin published dual-version benchmark runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: prepare agent benchmark indexes with updated runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: validate agent admission and render traces version comparison
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use runtime with traces index preparation
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use consolidated runtime for benchmark version reports
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: add license header to benchmark comparison tests
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): read legacy Tempo ingestion metrics in traces summaries
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: require tracesbench query plan artifacts
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use optimized tracesbench report runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: update tracesbench runtime to merged report validation
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 6e1a137411)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): accept evaluation times before the Unix epoch (#9480)
* fix(promql): accept evaluation times before the Unix epoch
Prometheus accepts negative timestamps in query parameters; GreptimeDB
rejected them while parsing and would panic converting them in the planner.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: update flow_tql error for a negative TQL end time
A negative end now parses and fails the end >= start check instead of
the timestamp conversion.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): reject evaluation times outside the millisecond range
The signed millisecond conversion cast u128 to i64 unchecked, so a far-past
time wrapped into the future. Reuse the checked conversion from
label_values and return an error instead.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: drop an unused import
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): round query times to milliseconds like Prometheus
Numeric times round the fraction of a second to milliseconds, and
pre-epoch times floor sub-millisecond parts, matching Prometheus'
parseTime and timestamp.FromTime.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): round the float fraction before converting to nanoseconds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: skip the overflow case where SystemTime cannot hold it
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit a8e293fd79)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: unify unknown flow option rejection across OSS and enterprise builds (#9319)
* fix(operator): unify unknown flow option rejection across OSS and enterprise builds
Remove the enterprise-gated passthrough that let unknown flow options
through SQL-layer validation. Both OSS and enterprise binaries now reject
unknown flow options with a single 1004(InvalidArguments) error at the
SQL layer, producing identical sqlness output.
The metasrv-side validate_flow_options remains as a defensive check.
Signed-off-by: discord9 <discord9@163.com>
* fix(flow): use DISTINCT output columns as sink key for streaming flows
When a streaming flow's source table has no primary key, sink key
inference produced an empty key. For SELECT DISTINCT v, the DISTINCT
establishes v as the output key, but the inference only looked at
source primary keys. Combined with the zero-valued placeholder
timestamp, all output rows shared the same storage identity and were
deduplicated into a single row.
Use all non-time-index output columns as the sink key when the plan
is a DISTINCT and no source primary key was inherited.
Signed-off-by: discord9 <discord9@163.com>
* fix(flow): preserve extension options and complete DISTINCT sink identity
Keep enterprise CREATE FLOW options available to the existing handler and report unknown ordinary options as InvalidArguments with the OSS SQL error text. Infer DISTINCT sink keys from every selected non-time-index output column, including when source key columns survive. Wait for asynchronous mirrors in show_create_flow and exercise 1000 distinct sink rows in the SQL regression.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): align meta unknown option status assertions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): use valid aliases in DISTINCT sink regression SQL
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): record complete DISTINCT sink SQL results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 178a0021b8)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): return range-vector results for instant queries (#9477)
* fix(promql): return range-vector results for instant queries
An instant query of a range selector or subquery (e.g. `up[5m]`) returned
the internal range arrays, which panicked when the frontend decoded them as
dictionaries. Plan such queries as the selected samples with their own
timestamps, and reject range-vector expressions in range queries like
Prometheus does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): keep encode_substrait on a valid range query
A range query of a range-vector expression now fails like in Prometheus.
Wrap the subquery in max_over_time so the case still encodes both
RangeManipulate nodes through substrait.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): plan wrapped range-vector roots and reject a zero subquery step
TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the
range-vector path did not unwrap. A zero subquery step divided by zero.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover wrapped range-vector roots and a zero subquery step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): default subquery step and reject range vectors in range queries
A root subquery without a step uses the one-minute default instead of
the query step. The HTTP range query endpoint rejects range-vector
expressions even when start equals end.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover root subquery default step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): align root subqueries with subquery_child_window
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): return millisecond timestamps for range-vector results
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): keep the time index timezone and name its table in errors
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit b913be2af7)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): use current DataFusion plan downcasts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): match null rendering in join regression
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): refresh union plan for unknown scan partitioning
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): refresh shared plans after distribution rule removal
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): cover semi joins with remote dynamic filtering
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): strengthen partitioned merge scan regressions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): remove unused merge scan test helpers
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(perf): measure merge scan hash repartition cost
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): adapt merge scan deserialization to two-arg constructor
Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): adapt nested merge scan integration constructor
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 32e8467cda)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(table): require complete primary keys for hash partitioning (#9507)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 28ec3ff5aa)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(deps): bump OpenDAL to 0.59.4 for OSS RRSA credentials (#9520)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit e6f3ca10a5)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): default a stepless subquery to a one-minute step (#9481)
* fix(promql): default a stepless subquery to a one-minute step
`x[5m:]` evaluated the inner expression at the query step, which is 1s
for instant queries. Prometheus uses the evaluation interval, one minute
by default.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover the default subquery step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover the default subquery step in a range query
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 845fc0af9c)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): follow IEEE 754 float semantics in comparisons and aggregations (#9479)
* fix(promql): follow IEEE 754 float semantics in operators and aggregations
- Comparisons: Arrow orders floats totally (NaN == NaN, NaN > +Inf); PromQL
comparisons with NaN are false except !=.
- min/max: skip NaN unless the whole group is NaN. DataFusion's grouped
min/max let NaN replace and be replaced by neighbours, so the result
depended on input order.
- topk/bottomk rank NaN last.
- round() rounds ties up and returns NaN for a zero step, like Prometheus.
- sqrt() of a negative number is NaN instead of an error.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): round() without a step rounds to integers
The planner passed a zero step for single-argument round(), which now
yields NaN like Prometheus; pass 1 instead. Found by the compatibility harness.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): apply the sqrt guard to builtin functions and register min/max for plan decoding
sqrt resolves through the session's builtin functions, not the UDF branch.
Distributed plans are decoded with the serializer's session, which did not
know prom_min/prom_max.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover NaN and rounding semantics
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep the nested count rewrite for PromQL min/max
count(min(x) by (l)) was rewritten by count_nest_aggr only for the builtin
min/max names. prom_min/prom_max return NULL for the same groups, so the
rewrite stays valid. Update plan snapshots for the IEEE comparison, sqrt,
topk and min/max changes.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): register prom_min/prom_max in the function registry
Distributed plans split an aggregation into state and merge steps only for
functions whose state function exists in FUNCTION_REGISTRY; prom_min and
prom_max lived in the promql crate, so min/max stopped being pushed down to
datanodes. Move them to common-function and register them before the state
functions are derived.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: update sqlness results for PromQL float semantics
Plans and column names now show prom_min/prom_max, the NaN guards of
comparisons and sqrt, and the NaN sort key of topk. round() results follow
Prometheus' formula. Distributed min/max keep the state/merge split.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep column names and NULLs in IEEE comparisons and sqrt
Rewriting comparisons as `a op b AND NOT (isnan(a) OR isnan(b))` changed
TQL output column names and turned NULL operands into true for `!=` and
false for the bool forms. Implement comparisons and sqrt as PromQL UDFs
that propagate NULL and report the column name of the expression they
replace; prom_min/prom_max report min/max column names the same way.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: restore column names in sqlness results for PromQL float semantics
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): share the prom_min/prom_max groups accumulator with tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep distinct names for prom_min/prom_max
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit f3a0841f4f)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(query): keep filter fetch when rewriting matches term (#9521)
* test: cover matches term limit dropped by constant rewrite
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): keep filter fetch when rewriting matches term
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 483d1e49a3)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: adapt release OptionMap iteration to Rust 1.98
Signed-off-by: WenyXu <wenymedia@gmail.com>
* build: reduce checkout-specific inputs for cache reuse (#9198)
* test(mito2): cover regex inverted index pruning
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* build: keep build inputs checkout-relative for cross-worktree cache reuse
Two small changes so compilations are byte-identical across worktrees
of the same commit and can share a build cache (mbx, sccache):
- src/common/version/build.rs: derive the workspace root relative to
CARGO_MANIFEST_DIR instead of the CARGO_WORKSPACE_DIR injected by
.cargo/config.toml (an absolute path). The env var remains as an
explicit override for the submodule case; the [env] injection in
.cargo/config.toml is removed so the relative derivation is used by
default.
- Replace three env!("CARGO_MANIFEST_DIR") uses in test/bench code
(servers prom_decode bench + decode test, common-meta test_util)
with a runtime std::env::var lookup, so the compiled artifact no
longer embeds the checkout path.
No behavior change: same files are read, same version info emitted.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* build: address cross-worktree cache review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit f50f42140a)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito): preserve version ordering in bulk merges and flushes (#9517)
* fix(mito): preserve delete barriers in flat last-non-null dedup
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito): respect sequence ranges in bulk merges and flushes
Signed-off-by: evenyag <realevenyag@gmail.com>
* perf(mito): avoid merge selection on the bulk write path
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 674ba72669)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: benchmark the object store WAL against S3 from an EC2 runner (#9536)
* ci: benchmark the object store WAL against S3 from an EC2 runner
Add a workflow_dispatch workflow that loads the o11ybench agent observability
dataset into a fresh GreptimeDB container per cell and compares three WAL
configurations that differ only in [wal]: Raft Engine with sync_write on the
instance store, and the object store WAL in durable and enqueued mode, with
the data files on S3 in the runner's region for all three. Rounds alternate
the configuration order. A summary reports rows per second, D/R and E/R as
the geometric mean over rounds, append acknowledgement p50 and p99, and the
object store WAL counters. A smoke mode runs the same chain on ubuntu-latest
against the MinIO fixture.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scrape WAL metrics on failed loads and keep logic out of the benchmark YAML
Scrape the object store WAL metrics before stopping a cell's server whether
the load succeeded or failed, so failed cells keep their stall, poison and
create failure counters. Build the summary from the plan so that cells which
never ran are listed and leave their ratios undefined. Move budget, region,
runtime image, manifest and credential scan logic from the workflow into the
helper scripts, and drive the cell lifecycle in the tests through fake docker
and curl binaries.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: drop redundant required flags from benchmark inputs
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail a benchmark cell whose metrics scrape fails and zero untouched WAL counters
A load that succeeds without its metrics scrape now fails the cell with exit
code 4 and records metrics_scraped in cell.json. The summary reports an
event counter that never registered as zero when the cell is an object store
WAL cell whose scrape succeeded and created objects, and as unknown otherwise.
Planned cells that never ran are read through the same path as recorded ones.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: register the S3 benchmark workflow for dispatch from its branch
A workflow_dispatch workflow can only be dispatched once GitHub has
registered it, which happens on the default branch or when one of its
triggers fires. A push trigger limited to this branch registers it; validate
runs only for workflow_dispatch, so a push run skips every job.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: delete the benchmark S3 prefix one object at a time
boto3 now sends a CRC checksum instead of Content-MD5 with DeleteObjects,
which the MinIO fixture of the smoke run rejects. DeleteObject needs neither
and the prefix of one run holds at most a few thousand objects.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: default the S3 benchmark to c6id.2xlarge
c7id is not offered in the runner region ap-southeast-1, where DescribeInstanceTypes
rejects it. c6id is the compute optimized x86 family with a local NVMe instance
store that the region offers, with the same 8 vCPU and 16 GiB at 2xlarge.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: drop the registration trigger of the S3 benchmark workflow
The workflow is registered now, so dispatches no longer need a push trigger
on its branch.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail an S3 benchmark run on EC2 without a local instance store
Without an instance store the Raft Engine WAL of the R cells silently stayed
on the root EBS volume, so the run measured a different baseline. Only a smoke
run may keep it on the root volume now.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: let the instance store check fail its workflow step
The step piped wal-disk into tee without pipefail, so a rejected host still
passed. wal-disk now writes its record itself and the step runs it alone.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit c1a0ca9595)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): ignore missing inverted indexes in old SSTs (#9545)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 287ced2de2)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: prevent stale leader metadata after concurrent writes (#9549)
fix: fence leader cache fills around store mutations
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 394562aa9e)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): stop fabricating histogram fold hash partitioning (#9509)
* fix(promql): stop fabricating histogram fold hash partitioning
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(promql): cover histogram fraction binary join missing rows
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(promql): normalize dictionary labels in histogram regression
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: cover histogram joins on logical metric tables
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: specify insert columns for logical metric fixture
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: use supported histogram child replacement API
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(promql): keep histogram fold fixtures truthful
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 2039f508fc)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(client): support region request reuse and route refresh hints (#9540)
* feat(client): support shared region requests without deep cloning
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: propagate route refresh retry hints
Signed-off-by: WenyXu <wenymedia@gmail.com>
* style: preserve local error variant imports in retry hints
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): distinguish staging rejection from write backpressure
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): preserve legacy retry hints in instruction replies
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(client): inline retry hint compatibility cases
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(datanode): align batch-open retry hint assertion
Signed-off-by: WenyXu <wenymedia@gmail.com>
* style: use imported types in region request tests
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 684c8ff291)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(operator): centralize write partitioning and dispatch (#9541)
* refactor(operator): centralize write partitioning and dispatch
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): preserve write dispatch behavior during refactor
Signed-off-by: WenyXu <wenymedia@gmail.com>
* style(operator): import write dispatch error paths
Signed-off-by: WenyXu <wenymedia@gmail.com>
* style(operator): align dispatch imports and test utilities
Signed-off-by: WenyXu <wenymedia@gmail.com>
* style(operator): use imported types in partition tests
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit a2d28e6060)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(operator): retry failed writes with refreshed partitions (#9542)
* feat(operator): retry failed writes with refreshed partitions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* style(operator): simplify retry test type references
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(operator): clarify write request rebuilding names
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(operator): encapsulate write retry attempts
Signed-off-by: WenyXu <wenymedia@gmail.com>
* style(operator): group inserter configuration methods
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(operator): share routed peers and separate retry decisions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(operator): avoid temporary write routing allocations
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit d86741d0ad)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(frontend): retry transient distributed write failures (#9543)
* refactor(operator): route shared requests through NodeManager
Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(meta): preserve retry policy through error hints
Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): restore sparse partition tags when rebuilding writes
Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): stop detached write retries when callers cancel
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(frontend): configure retries across distributed write paths
Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage.
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 863eebdecb)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(table): stop claiming tag order for logical metric scans (#9522)
* test: cover metric logical scan tag ordering
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(table): only claim projected primary key prefix ordering
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(table): keep physical key ordering for metric projections
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(table): compare unknown partition counts explicitly
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(table): model scoped internal TSID scan properties
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 75bc4ce46d)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(promql): stop claiming scalar calculate preserves order (#9523)
* test: cover scalar calculate order contract
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(promql): stop claiming scalar calculate preserves order
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(test): describe scalar ordering regression as pre-fix behavior
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 0e40281262)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align fulltext bloom tokens with matches_term and use them for LIKE (#9470)
* fix: align fulltext bloom tokens with matches_term and use them for LIKE
The bloom fulltext index stored tokens that did not follow
matches_term's word boundaries (`_`, ASCII next to Han, Han substrings),
so pruning dropped rows matches_term accepts.
Tokens and query probes now share the character classes of
matches_term, Han runs are indexed as bigrams, and LIKE patterns use the
same probes. The bloom blob is written under a new key; blobs from older
versions are no longer probed. Tantivy no longer serves matches_term.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: drop unused lazy_static from index and avoid typo-flagged test tokens
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: link like_probes to tokenize instead of the downstream ScriptTokenizer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: name probe extraction with verbs and document CharClass
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: rebuild legacy fulltext bloom indexes and probe lower(col)
The manifest records the fulltext bloom version per column, so ADMIN
BUILD_INDEX now rebuilds SSTs whose bloom index predates the current
tokens, and ssts_index_meta shows the version.
Case-insensitive bloom indexes also store the tokens of the whole
lowercased text for rows where it can differ (U+0130, U+212A, sigma),
so matches_term(lower(col), ..) and lower(col) LIKE .. use the index.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover legacy bloom index rewrite by compaction in the compat case
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: log outdated fulltext bloom indexes in the consistency check
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: explain the unrecorded fulltext bloom version
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 5c4d9eb4f8)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(query): guard deep expressions during query planning (#9564)
* fix(query): avoid speculative clones in constant normalization
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(query): reject deeply nested expressions before optimization
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 00aa65de7a)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): reuse catalog table resolution for write retries (#9574)
* fix(operator): inherit logical table partitions when retrying writes
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(operator): resolve retried writes through catalog manager
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 36423a95c6)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(query): accept narrowed nullability in MergeScan remote schema check (#9592)
On append-mode tables a region reports an exact row count. When it is
zero, DataFusion's AggregateStatistics replaces the region-side
__min_state/__max_state aggregate with a non-null struct literal, so the
remote stream advertises the state column as non-nullable. The MergeScan
schema check required equal nullability and failed the query.
Accept a remote field that is non-nullable where the expected field is
nullable, matching DataFusion's own physical-plan schema invariant.
Batches are already rebuilt with the expected schema. Widening is still
rejected, and nested field nullability is still compared exactly as part
of the data type.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit ef1298ce02)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(table): stop claiming exact null and distinct counts for append scans (#9593)
RegionScanExec reported null_count = 0 and distinct_count = total_rows for
every column of an append-only table. DataFusion answers count(col) and
count(DISTINCT col) from these statistics, so both returned the row count.
Keep the exact row count and an exact zero null count for non-nullable
columns, so count(*) is still answered from statistics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 5cfdc30a69)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: avoid pinning memtables during series-index reconciliation (#9588)
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit ce66d8d4ac)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(query): infer safe aggregate input bounds from HAVING (#9582)
* perf(query): infer safe aggregate input bounds from HAVING
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(query): refresh explain snapshots for aggregate bound inference
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(query): avoid reinferring bounds across MergeScan
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(query): validate aggregate bound column provenance
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(query): retain zero-match groups for nonpositive HAVING thresholds
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(query): assert inferred bounds in remote scan plans
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit af396e60a3)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(query): keep state column names unique when splitting aggregates (#9594)
* fix(query): keep state column names unique when splitting aggregates
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): probe allocated names when renaming colliding state columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit f7a3a7033c)
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: bump version to 1.3.0-beta.2
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: greptimedb-ci <greptimedb-ci@greptime.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: fengyue-xve <1759519137@qq.com>
Signed-off-by: Aetherance <inkToAether@gmail.com>
Signed-off-by: tian1220A <1573612521@qq.com>
Signed-off-by: 风月 <1759519137@qq.com>
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
Signed-off-by: lczllx <2181719471@qq.com>
Signed-off-by: Aumnertic <hznuzc@gmail.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
Co-authored-by: greptimedb-ci <greptimedb-ci@greptime.com>
Co-authored-by: DeviousCardi <115358213+DeviousCardi@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: fengyue-xve <1759519137@qq.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Aether <98370028+Aetherance@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zhaiAohan <1573612521@qq.com>
Co-authored-by: Niladri Adhikary <91966855+niladrix719@users.noreply.github.com>
Co-authored-by: Dragon Roar <2181719471@qq.com>
Co-authored-by: Aum <hznuzc@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
* ci: update cargo fuzz command to use nightly toolchain explicitly
* ci: honor RUSTUP_TOOLCHAIN pin in fuzz orchestration script
An explicit `+toolchain` argument overrides the RUSTUP_TOOLCHAIN env var
in rustup precedence, so the hard-coded `cargo +nightly` in
run-fuzz-targets.sh bypassed the pinned FUZZ_RUST_TOOLCHAIN
(nightly-2026-03-21) configured in the workflow.
- Invoke `cargo +"${RUSTUP_TOOLCHAIN:-nightly}" fuzz run` in the script
so CI uses the pinned toolchain and local runs fall back to the
floating nightly
- Pass RUSTUP_TOOLCHAIN through to all four fuzz-test action invocations,
covering the no-prebuilt-binaries path and making the reproduce command
in the summary print the exact pinned toolchain
- Add test_rustup_toolchain_env_is_honored covering the pinned-env
scenario for both the cargo invocation args and the summary text
Addresses #9298 (review).
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
The scheduled (nightly) release version was created by appending
'-nightly-YYYYMMDD' to NEXT_RELEASE_VERSION as-is. When the Cargo.toml
version carries a pre-release extension (e.g. v1.3.0-alpha.1), this
produced invalid tags like 'v1.3.0-alpha.1-nightly-20260907', stacking
'nightly' on top of the 'alpha.1' pre-release.
Strip the pre-release extension first so 'nightly' itself becomes the
only pre-release extension: 'v1.3.0-alpha.1' -> 'v1.3.0-nightly-20260908'.
Stable versions are unaffected; nightly-build ('nightly-YYYYMMDD-sha'),
dev-build, tag push, and manual dispatch paths are unchanged.
Signed-off-by: Ning Sun <sunning@greptime.com>
* Implement `/query-regression` command handling and admission workflow
- Add `query-regression-slash.py` script for processing `/query-regression` commands in PR comments, validating case arguments, and checking permissions.
- Update `checks.yml` to include tests for the new slash command functionality.
- Modify `query-regression-comment.yml` to trigger on the new `Query Regression Command` workflow.
- Create `query-regression-slash.yml` to handle the dispatched command, validate allowlist and permissions, and initiate the regression workflow.
- Enhance `query-regression.yml` to support additional inputs for PR admission and SHA verification.
- Introduce `slash-command-dispatch.yml` to parse and dispatch commands from PR comments.
- Document the new command admission process in `AGENTS.md` and `README.md`.
- Add unit tests in `test_query_regression_slash.py` to cover command parsing and admission logic.
* refactor: enhance query-regression command handling with comment validation and identity checks
* feat: implement admission identity handling for query regression workflows
* refactor: update PR admission logic in query regression workflow
* refactor: update token usage in slash command dispatch and README for clarity
* test: add cases for handling re-run failed jobs and stale runner artifacts
* refactor: improve repository metadata handling in query regression scripts
* chore: enable overwrite for artifact uploads to handle re-run failed jobs
* chore: enable overwrite for query regression admission uploads
* feat: enhance query-regression admission with HMAC signing and verification
- Introduced HMAC signing for admission markers in query-regression workflows to ensure integrity and authenticity.
- Updated `query-regression-comment.test.cjs` to include tests for signing and verifying admission markers.
- Modified `query-regression-slash.py` to handle admission marker signing and verification, including checks for dispatch sender and head SHA consistency.
- Enhanced workflows to securely manage admission markers and HMAC secrets, ensuring they are not exposed to untrusted contexts.
- Improved documentation to clarify the admission process and the role of HMAC in securing the workflow.
* test: add case to find newly posted marker among newer comments
* test: add case to verify multiline output handling in write_outputs function
* feat(runtime): add weighted workload scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics
Use the GreptimeTeam/catio fork (pinned c20eafc) which adds
ClassStats::total_admission_wait and ClassStats::admitted, recorded
at each QUEUED -> ADMITTED transition. This exposes the scheduler's
own admission delay (excluding Tokio queueing and poll execution),
enabling admission-wait based fairness gates.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to dynamic-config revision
Bump the catio scheduler fork to 9f4b028 which adds
Scheduler::set_weight and Scheduler::set_max_concurrent_polls for
runtime configuration.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(perf): runtime-adjustable workload scheduler parameters
Expose dynamic adjustment of the experimental workload scheduler at
runtime:
- common-runtime: set_workload_scheduler_weights and
set_workload_scheduler_max_concurrent_polls, which forward to the
catio scheduler's set_weight/set_max_concurrent_polls when the
scheduler is enabled and reject zero values.
- servers: /debug/workload_scheduler/weights and
/debug/workload_scheduler/max_concurrent_polls POST handlers, so
operators can rebalance query/write shares or admission concurrency
without restarting the datanode.
Both endpoints return 400 with a clear reason when the scheduler is
disabled or the requested value is invalid.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(perf): add GET /debug/workload_scheduler status endpoint
Returns the current weights (per class), max_concurrent_polls,
active_polls and per-class counters (queued, tasks, wakes, polls,
completed, cancelled, admitted, total_admission_wait) as JSON. When the
scheduler is disabled, returns enabled=false with the other fields
omitted, so operators can distinguish 'disabled' from an error.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to time-accounting revision
Bump the catio scheduler fork to 257ba56 which replaces
admission-count accounting with real execution-time accounting
(pass += exec_time / (weight * concurrency)), so CPU share follows the
configured weights regardless of poll length.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to lock-free sampling revision
Bump the catio scheduler fork to efdc0a4 which adds an optional
downsampled clock sampling mode (SchedulerBuilder::sample_every_polls,
default off) with a lock-free per-class atomic counter, so the
downsampled path costs one fetch_add per poll instead of a global
mutex.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin catio to scheduler PR head
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): add scheduler bypass control
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: advance catio scheduler fixes
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin merged catio scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: regenerate config docs for workload scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin catio scheduler test fix
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(http): satisfy scheduler lifecycle clippy
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: add distributed scheduler toggle coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat: finalize workload scheduler runtime controls
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin merged catio atomic weights
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: preserve unrelated lockfile resolution
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(runtime): downsample scheduler time accounting
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(runtime): verify cross-runtime scheduler progress
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): configure scheduler poll sampling
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(runtime): clarify scheduler activation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(runtime): explain scheduler use case
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
* feat(ci): add aliyun ecs ephemeral runner path for query regression
Signed-off-by: paomian <xpaomian@gmail.com>
* fix: improve condition for query-regression job execution in workflow
* feat: update Docker installation to use official repository and add GPG key handling
* Refactor query regression runner setup and configuration
- Removed deprecated PersistentVolumeClaim for build cache.
- Introduced a new bootstrap script for setting up the ECS runner host.
- Deleted obsolete Helm values files for runner configuration.
- Updated the Aliyun ECS runner provisioning script to reflect new cache paths.
- Modified GitHub workflows to use the new Aliyun ECS runner setup.
- Adjusted documentation to clarify the new runner lifecycle and provisioning process.
* fix: enhance runner service management during bootstrap process
* fix: update alibabacloud_tea_openapi dependency version in metadata
* feat: enhance ECS runner scripts with region_id and resource_group_id support
* fix: move containerd content store to data root for improved storage management
* feat: rename query-regression runner to ephemeral-github runner and update related scripts
* fix: update sentinel polling method to use serial console output for improved reliability
* fix: add environment variable checks for Alibaba Cloud access keys in ECS client
* fix: improve error handling in GitHub API requests for better diagnostics
* fix: improve cache disk detection logic for Aliyun ECS instances
* fix: enhance cache disk waiting logic with detailed output and error handling
* fix: update dependency version for alibabacloud_tea_openapi in teardown script
* fix: enhance cache disk waiting logic for better compatibility and clarity
* fix: enhance console output handling and add incremental logging during instance provisioning
* fix: add PATH environment variable for runner jobs in service and provision script
* fix: add machine telemetry sampling and logging during query regression jobs
* fix: update query regression documentation and provision script for cache disk handling
* fix: update SCCACHE_CACHE_SIZE validation to 10G for improved caching efficiency
* fix: remove outdated cache size checks and cleanup logic for fresh system disk runs
* fix: enhance instance deletion logic with region handling and console output export
* fix: add swap file setup and OOM handling for ECS runner to improve stability
* fix: update OOM handling and service restart logic for ECS runner to enhance stability
* fix: increase system disk size to 100 GiB for cold double nightly builds to prevent ENOSPC errors
* fix: increase system disk size to 150 GiB for ECS runner to prevent ENOSPC errors
* fix: add keep_instance option to preserve ECS instance for post-mortem debugging
* fix: disable unattended upgrades to prevent job cancellations during library updates
* fix: reduce system disk size to 40 GiB for ECS runner to prevent ENOSPC errors
* feat: Refactor Aliyun ECS runner provisioning and introduce nightly regression comparison
- Update `aliyun-ecs-runner-provision.py` to remove cache disk handling, simplifying the provisioning process.
- Introduce `query-regression-nightly-refs.py` to resolve and compare SHAs from successful nightly builds.
- Create `query-regression-nightly.yml` workflow to trigger nightly comparisons based on successful builds.
- Enhance `query-regression.yml` to include a `test-tooling` job for validating Python scripts before provisioning.
- Update tests for the new nightly reference selection logic and refactor existing tests to align with the new caching strategy.
- Modify documentation to reflect changes in caching and nightly comparison workflows.
* fix: enhance runner image tool verification with detailed checks
* fix: improve error handling in runner image tool verification
* fix: update tool versions in ECS image and workflow for consistency
* fix: correct typo in error message for unparseable ECS creation time
* fix: update README and workflow files for query regression tests and image hygiene
---------
Signed-off-by: paomian <xpaomian@gmail.com>
Posting to `/issues/{n}/comments` is authorized against the target object, and
that object is a pull request, so `issues: write` alone is refused with 403 and
the warning comment never lands.
Drafts are no longer counted and no longer warned about. `ready_for_review` is
added to the trigger types so that opening as a draft and flipping it to ready
still goes through the check.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(ci): identify team members by repository permission
`author_association` is computed from what the caller can see, so GITHUB_TOKEN
reports a private organization member as CONTRIBUTOR. Only 5 of GreptimeTeam's
members have public membership, so the open-pull-request check skipped almost
everyone it was written for.
Use the repository permission of the author instead, which is
viewer-independent. On error, apply the limit rather than skipping, so a token
that cannot read permissions cannot silently disable the check again.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(ci): do not log repository permission levels
Job logs are public. Resolving the author's permission is fine; printing the
level is not.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* chore(ci): warn when a member has too many open pull requests
Review capacity is the bottleneck. Add a `pull_request_target` workflow that
counts an org member's open pull requests (drafts included) on open/reopen and
posts a warning comment when the count exceeds the limit.
Advisory only for now: nothing is closed and no check fails.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(ci): address review on the pr-open-limit workflow
- Only match marker comments authored by the Actions bot; a marker pasted
by anyone else would otherwise be picked up and fail the edit with 403.
- Validate MAX_OPEN_PRS and fall back to 5 on a non-numeric variable.
- Drop pull-requests write permission; commenting goes through the issues API.
- continue-on-error so a script failure never marks the pull request red.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Move case/storage thresholds out of the main summary table into a
collapsible <details> block below it, so measured cases render one row
per query instead of an extra Query=N/A storage row that stretched the
table. Drop the backtick code span around the status cell (the escaping
in text() turned it into a visible backtick) and unify multi-threshold
separators on '; '.
- query-regression-comment.cjs: classifyThresholds shared helper,
syntheticThresholdDetail for case-level thresholds, kind marker on
synthetic rows, renderThresholdDetails emitting unescaped details tags
with text()-escaped entry content, plain-text status cell
- query-regression-comment.test.cjs: updated separator/kind assertions,
new test covering details-block rendering and plain-text status
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor: port query regression runner to Rust
Signed-off-by: discord9 <discord9@163.com>
* ci: remove optional OTLP report plotter
Signed-off-by: discord9 <discord9@163.com>
* refactor: split query regression runner into modules
Signed-off-by: discord9 <discord9@163.com>
* style: use crate-qualified imports in query regression runner
Signed-off-by: discord9 <discord9@163.com>
* refactor: simplify query regression runner internals
Signed-off-by: discord9 <discord9@163.com>
* feat: abstract inspect-footer storage access behind object store destination
Add an optional --destination <TOML> to inspect-footer (and
--base-destination/--candidate-destination to finalize-remote) so the
storage inspection reads DB data files through the opendal-backed
object_store abstraction instead of bare std::fs. Local paths keep
working unchanged via the --root shortcut (File backend); remote
backends (S3/GCS/...) are described by a DestinationConfig TOML
reusing the object-store crate's ObjectStoreConfig serde shape.
- inspect_footer: list via ObjectStore::list + ObjectMeta filtering
(parquet keys, non-zero size, metadata/ segment), read footers
async via ParquetObjectReader + ParquetMetaDataReader with known
file size (no extra HEAD); output JSON schema unchanged
- finalize-remote: --base-data-home/--candidate-data-home become
optional, mutually exclusive with the new --*-destination args
- cmd deps: add object_store_opendal + datafusion_object_store
- tests: fs-backend list+footer integration tests (metadata filtering,
destination TOML mode, root/destination exclusivity)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* style: drop needless borrow in inspect footer test
Fix clippy::needless_borrows_for_generic_args in the inspect-footer test
(fs::create_dir_all(table.join("metadata"))). Missed by the earlier
focused clippy run because it only covered --bin targets.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* ci: automate compatibility version window
Signed-off-by: discord9 <discord9@163.com>
* ci: address compat version window review feedback
Address all four review comments on the compat version window
automation:
- Keep the PR window to the sliding window only (latest patch of the two
newest stable minor lines). Exact =vX.Y.Z anchors from case.toml are no
longer unioned into from_versions; they are validated by the new
--check-anchors mode and exercised by nightly runs via --nightly-window.
- Add --published-only: the window is computed over stable git tags that
have a published, non-draft GitHub release carrying the sqlness compat
artifacts (greptime-linux-amd64 tar.gz and sha256sum), so failed releases
cannot land in the window.
- Run the updater Python tests plus the window/anchor consistency check in
PR and merge-group CI (new compat-updater-check job in integration.yml).
- Regenerate tests/compatibility/ci.toml to the current sliding window
[v1.0.2, v1.1.4].
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): use two pointers for sliding range boundaries
Replace the stale cursor heuristic in RangeManipulateStream::calculate_range
with monotonic left/right cursors. The old path rescanned each evaluation
window (O(E x samples-per-window)) and could lose valid samples after sparse
gaps or trailing empty windows. The two pointers keep strict monotonic
progress, reducing boundary generation to O(N + E) while preserving
(curr-range, curr] semantics, start/end shortening, and empty-window output.
Controlled release benchmarks (fixed CPU, ABBA):
- Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s,
~96% at 1h/15s.
- Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end;
shorter windows stayed within run-order noise.
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): specialize changes/resets with adaptive edge counting
The generic range_fn macro slices, downcasts, and rescans every overlapping
window for changes() and resets(). Replace the macro path for these two
functions with hand-written UDF wrappers backed by a shared private
edge-count kernel: direct raw-offset scans when requested edges are few,
otherwise one global u64 edge prefix so each window is answered by a prefix
difference.
Behavior is preserved bit-for-bit, including raw null-buffer values, NaN
semantics, signed zero, infinities, empty/singleton windows, independent
timestamp/value offsets, arbitrary window layouts, and exact DataFusion
error messages. The shared proc macro, planner, serializer, and other range
functions are untouched.
Controlled release benchmarks (fixed CPU, ABBA):
- Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time.
- Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster.
- Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client
and 12.0-20.9% server latency improvement; controls stayed within drift.
Signed-off-by: discord9 <discord9@163.com>
* ci(query-regression): include PromQL range boundary case in defaults
An audit of historical query-regression runs found zero range-query
coverage: all 208 PromQL ANALYZE samples were bare selectors, so range
evaluation could regress without CI noticing. Wire the
promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so
label-triggered runs measure the range path. The case is cheap: a ~0.3s
synthetic fixture and about a minute of query execution per base/candidate
pass.
Signed-off-by: discord9 <discord9@163.com>
* chore(promql): address sliding range review nits
Move test-only imports into their test modules and remove the unused
pre-specialization changes and resets helpers.
Signed-off-by: discord9 <discord9@163.com>
* style(promql): apply pinned rustfmt
Signed-off-by: discord9 <discord9@163.com>
* test(promql): cover sparse range results
Share the changes and resets test scaffolding while keeping their behavior
oracles independent. Add an end-to-end sqlness regression for sparse samples,
empty intermediate windows, and a valid trailing sample.
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>