Commit Graph
6107 Commits
Author SHA1 Message Date
discord9 5e57b00a95 fix(promql): skip empty batches in SeriesDivide stream (#9178)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-16 07:37:15 +00:00
discord9 8af3a04ed7 fix: preserve structured query errors through distributed execution (#9161)
* fix: preserve structured query errors through distributed execution

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update SQL expectations for preserved query error codes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-16 06:28:15 +00:00
Lei, HUANG 7aacc947b3 perf(mito2): lazily extract sparse primary key index values (#9176)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-16 06:15:52 +00:00
fys 1c01b541ec refactor(json2): JSON2 parquet projection and schema alignment (#9137)
* refactor(mito2): make JSON schema alignment targets explicit

Signed-off-by: fys <fengys1996@gmail.com>

* fix: cr

* fix: cargo fmt

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-09-16 04:44:32 +00:00
Lanqing Yang 67a403757d Perf/http sql limit materialization (#9148)
perf(servers): limit HTTP SQL rows before materialization

Signed-off-by: lyang24 <lanqingy93@gmail.com>
2026-09-16 03:46:11 +00:00
jeremyhi d7ada1761d feat: export logical tables from Metric physical scans (#9159)
* feat: add physical Metric table exporter

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: drain Metric export writes before cancellation cleanup

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: construct Metric export error context lazily

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: name the logical table export entry point

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: share Parquet writer for logical table exports

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve Parquet destinations and cancellation boundaries

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: clarify logical table export field names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: clarify logical table export helper responsibilities

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate logical export membership by table ID

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: simplify logical table export coverage and strengthen assertions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-16 02:48:47 +00:00
discord9andNing Sun 94d7e2c7fc feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: migrate DataFusion 55 APIs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve table function planning behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve PostgreSQL query compatibility

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve distributed execution plan behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover DataFusion 55 behavior regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update DataFusion 55 SQLness expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DataFusion 55 test API migration

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address DataFusion 55 CI regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address remaining DataFusion 55 regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: adapt latest base code to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: normalize environment-specific DataFusion 55 plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align final DataFusion 55 expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: isolate DataFusion 55 regression cases

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve empty result schema in timestamp widening

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve JSON source column order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: use released DataFusion 55 integrations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: adapt latest execution plan mock to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: pin DataFusion recursive schema and date repairs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): align dictionary temporality match keys

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: retain Greptime DataFusion fork behaviors on version 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore ordinary function error expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh distributed count compatibility plan

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt last-row cast hint to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh instant last-row empty results for Arrow 59

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: simplify DataFusion expression visitor imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore sorting and PostgreSQL column-order assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): restore primitive numeric coercion signatures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(function): share geo integer signature types

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover timestamp widening overflow boundaries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: fix decimal coercion regression imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): preserve scalar count_hash NULL state semantics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: simplify decimal clamp case type inference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: retain historical count_hash wrapper result

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: restore timestamp widening equality and IN pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream aggregate dynamic filter correctness fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream null and predicate simplification fixes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore baseline JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore histogram JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh empty PromQL range result schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align native timestamp plan with DF55 decimal display

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh native timestamp SQLness results for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regenerate NULL sample empty result headers for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: use DF55 child replacement API in timestamp regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: expose pushed scan dynamic filters to DF55 producers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: encode string-backed PostgreSQL OID aliases in binary results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: verify REGPROC binary and text over PostgreSQL protocol

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: register real PostgreSQL catalogs in server fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DF55 expression inventories for custom query plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: correct RangeSelect expression fixture and column identities

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* ci: wait for Kafka WAL helper deployment rollout

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update custom storage empty result headers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: require exact row counts in scan statistics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: suppress deprecated partition_statistics warning in test

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-09-15 11:42:38 +00:00
jeremyhi c65e8d0962 fix: register region migration procedures before returning IDs (#9163)
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-15 10:33:35 +00:00
LFC 9f6a79da30 fix: support native JSON2 row inserts over gRPC (#9145)
* fix: support native JSON2 row inserts over gRPC

Signed-off-by: luofucong <luofc@foxmail.com>

* test: cover unknown JSON2 schema compatibility

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-15 10:19:48 +00:00
discord9 da7e31842e perf(servers): avoid rebuilding JSON records payload (#9160)
* perf(servers): avoid rebuilding JSON records payload

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: trim json response optimization unit tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* perf: move bulk response items into json payload

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-15 10:02:56 +00:00
Lei, HUANG 6f10c0cd22 perf(metric-engine): avoid deep-cloning physical column metadata in verify_rows (#9162)
verify_rows deep-cloned the whole HashMap<String, ColumnMetadata> of the
physical region on every put batch. On a datanode serving wide physical
tables this showed up as ~18% of total CPU in a CPU flame graph (HashMap
clone + RawTable/ColumnMetadata drop).

Wrap physical_columns in an Arc inside PhysicalRegionState and take a
cheap Arc snapshot instead; add_physical_columns now goes through
Arc::make_mut.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2026-09-15 09:21:53 +00:00
dennis zhuang 7cbd20a053 fix(mysql): strip leading comments before the federated statement filter (#9156)
* fix(mysql): strip leading comments before the federated statement filter

JDBC clients prefix every statement with a comment. DataGrip sends
`/* ApplicationName=DataGrip <version> */` in front of each one, and every
pattern in the federated filter is anchored with `^`, so the prefix makes all
of them miss.

Two failures follow. `SET TRANSACTION READ WRITE` reaches the SQL parser and is
rejected. Worse, the DBeaver-specific entries such as
`^(/\* ApplicationName=(.*)SELECT @@(.*))` do match DataGrip's prefix, so
`SELECT @@GLOBAL.event_scheduler` and `SHOW VARIABLES LIKE ...` are absorbed
into a zero-column output, which the MySQL writer sends as an OK packet. The
JDBC driver reports that the statement returned no cursor.

Strip leading whitespace and comments before matching, and drop the
ApplicationName-specific entries the stripping makes redundant. The three that
had no unprefixed counterpart (`SHOW PLUGINS`, `SHOW ENGINES`, `SHOW @@...`)
keep their behaviour as plain patterns.

Also covers gaps found while probing the same path:

- `BEGIN` is absorbed like `START TRANSACTION`/`COMMIT`/`ROLLBACK`.
- `SHOW [GLOBAL|SESSION|LOCAL] VARIABLES|STATUS` parses; the scope is ignored
  because GreptimeDB keeps no global/session split.
- `USER()`, `CURRENT_USER()`, `SYSTEM_USER()` and `SCHEMA()` are registered.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mysql): keep executable comments, reject multi-statement absorption

Review follow-up on the comment stripping, plus the remaining MySQL
compatibility gaps from #9155.

`/*!...*/` is an executable comment: mysqldump emits its initialization as
`/*!40101 SET NAMES ... */`, and the patterns match those verbatim. Stripping
it left an empty statement, so nothing matched and the original SQL reached the
parser, which rejects `SetNames` and `MultipleAssignments`. Leave executable
comments in place.

Absorbing a request also has to stop at a statement boundary, because every
pattern ends in `(.*)`. `BEGIN; INSERT INTO t VALUES (1)` used to fail on the
unsupported `BEGIN`; once `BEGIN` became absorbable the whole request would
report success and write nothing. A request is now scanned for a second
statement and handed to the query engine if it has one. The scan skips plain
comments and string literals so a `;` inside either is not a boundary, and
treats a `/*!...*/` that is not the request itself as a statement, since it
carries SQL.

`check()` now dispatches on the leading statement keyword, so INSERT, UPDATE,
CREATE and ordinary SELECTs run no regex at all — this replaces the narrower
hand-rolled INSERT shortcut. The statement scan runs only for a request the
patterns already matched, which is always a short one.

New compatibility surface:

- `SELECT CURRENT_USER|SESSION_USER|SYSTEM_USER|USER` without parentheses, and
  `SELECT @var`, are answered here rather than in the parser. Both are anchored
  to the whole statement, so `SELECT user FROM t` still reads the column.
- `information_schema.plugins`, `user_privileges` and `processlist` are
  registered as empty tables, like the other MySQL-shape tables around them.
  Sessions are reported through `information_schema.process_list` and
  `SHOW PROCESSLIST`; `processlist` carries the column shape only.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-15 08:59:30 +00:00
Yingwen 7c85798ad0 feat(mito2): reconcile series indexes in background (#9086)
* feat(mito2): reconcile series indexes in background

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): clean up series indexes published during region drop

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(config): align series index examples with upstream enable flag

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): run series index tasks on compaction runtime

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-15 08:44:18 +00:00
Weny Xu d9b97796a6 fix: preserve count correctness after repartition (#9154)
* fix: preserve count correctness after repartition

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: restore full predicate guard for count statistics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: update count plans for full predicate guard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve count statistics for safe partition scans

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: retain repartition home guard and cover staging flush

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-15 08:26:57 +00:00
Lei, HUANG e0552eee2d fix(mito2): preserve effective sequences during compaction (#9147)
* fix(mito2): preserve effective row sequences during compaction

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): inherit compaction input sequence bounds

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-15 07:43:16 +00:00
jeremyhi 32bf865ffb docs: propose Metric export and import optimizations (#9121)
* docs: propose Metric export and import optimizations

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: colocate Metric RFC supporting materials

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: condense Metric PoC evidence into readable results

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: use the tracking issue as the implementation plan

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: clarify Metric export design and ordering acceptance

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-15 07:18:38 +00:00
Immanuel Tikhonov 5ec679fed9 fix(servers): resolve port from IPv6 bind addresses (#9142)
Signed-off-by: immanuwell <pchpr.00@list.ru>
2026-09-15 06:32:38 +00:00
discord9 ba0f7acd93 feat(mito2): add opt-in byte-stream-split encoding for float SST fields (#9069)
* feat(mito2): add opt-in byte stream split encoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): correct float encoding checks

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(compat): cover float SST encoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): compile float encoding tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): release parquet test writer

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): register float test primary key

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): verify BSS write lifecycles

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(metric-engine): verify BSS physical SST

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): verify bulk BSS lifecycle

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): compile bulk BSS test

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): narrow bulk encoding constructors

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(compat): accept generated float upgrade output

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(compat): accept generated float downgrade output

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): narrow bulk encoding builder

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): add default versus BSS storage comparison

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): align BSS reader benchmarks with prior study

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(perf): parse current read benchmark averages

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(perf): retain default float encoding in direct SST fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): isolate BSS user SSTs and benchmark every file

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(perf): record measured BSS storage and reader tradeoffs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(perf): expose warm scan variability and evidence limits

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(perf): clarify BSS baseline and storage measurement scope

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): model bounded mixed integer and fractional metric series

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(perf): report bounded mixed BSS measurements and query regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(perf): qualify timings affected by concurrent host builds

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): include float BSS comparison in default regression cases

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(perf): omit unsupported float encoding option from baseline setup

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-14 12:05:42 +00:00
discord9 03d4e62a22 fix(mito2): deliver complete WAL entries without waiting for next input (#8916)
* fix(mito2): deliver complete WAL entries without waiting for next input

Entry completeness is self-contained via Entry::is_complete(); no
lookahead to the next entry is needed. Decode and yield complete entries
immediately in both the log-store reader and the entry distributor.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): simplify live WAL entry fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-14 11:16:22 +00:00
Weny Xu 737025760e feat: support request-level WAL skipping for bulk inserts (#9110)
* feat: support request-level WAL skipping for bulk inserts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover bulk insert WAL skipping across protocols

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: align WAL snapshot naming with sequence watermarks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: bump proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-14 09:38:05 +00:00
Yingwen 23a1a5ecaa refactor: remove constant vector and replicate operation (#8999)
* refactor: remove constant vector

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: remove vector replicate operation

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: preserve scalar vector types with optional type hints

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: remove obsolete mutable vector helper and import

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: preserve typed nulls in struct scalar conversion

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-14 09:25:33 +00:00
Yingwen a23fe1c2dc fix: configure series indexes with an enable flag (#9141)
* fix: use join_dir for series index config path

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: configure series indexes with an enable flag

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: omit experimental series index from example configs

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: preserve legacy cache cleanup path behavior

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: remove trivial path joining tests

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: isolate worker group WAL directories on Windows

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-14 09:21:31 +00:00
dennis zhuang 577de012b2 perf(servers): group Prometheus response rows by label runs (#9092)
* perf(servers): group Prometheus response rows by label runs

Query output tends to be clustered by series, but a matrix response read
the same label values out of the tag columns once per row, and allocated
a key vector per row to look the series up.

Use `arrow::compute::partition` to find the runs of rows that share their
labels and build the series key once per run. The key buffer is hoisted
out of the row loop and handed to the map only when the series is new,
through the raw entry API so the key is hashed once either way.

Partitioning does not pay off when rows are not clustered, so a few
adjacent row pairs are probed first to pick between the run path and the
row-by-row path. Both paths produce the same series.

Drop the per-row "same labels as the previous row" check from #8815. Runs
cover the clustered case it was written for, and it now costs more than
it saves: 10% on a result with one row per series, 1-3% on clustered ones.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* refactor(servers): find label runs with cmp::distinct

`arrow::compute::partition` computes the same ranges on the same kernel,
but its contract takes lexicographically sorted columns, and query output
is not sorted: range queries run without the plan's output sort since
#9090, `sort`/`topk` order by value, and the tag column order in the
schema does not have to match any sort key. An implementation that
exploited the precondition would merge `a, b, a` into one run and
attribute one series' samples to another, without failing.

`cmp::distinct` is element-wise, so it holds for any row order, and its
null handling is the one a series key needs: a null label and an empty
one are distinct, two nulls are not. Building the ranges from the
boundary bitmask also folds away the `tag_columns.is_empty()` case, since
no columns means no boundaries means a single run.

Same kernel, so the benchmark does not move: -1.4% to +1.4% across shapes
with no consistent sign, against +-3% run-to-run drift.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

---------

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
2026-09-14 09:01:09 +00:00
jeremyhi 0059508e29 perf: avoid directory listing for explicit COPY input files (#9126)
* perf: avoid directory listing for explicit COPY input files

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve COPY input symlink filtering

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-14 07:52:25 +00:00
discord9 13c69cdaee perf(gc): pack file reference exchange (#9009)
* perf(gc): pack file reference exchange

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(gc): address packed reference review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(gc): stop without retry when maintenance is enabled

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(meta): avoid logging malformed mailbox payloads

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-14 07:30:39 +00:00
dennis zhuang 8e45809b6d perf(query): check Substrait encodability once per plan (#9095)
`PlanRewriter::should_expand` encoded every node it ascended past to
Substrait. Each call clones the plan, runs an analyzer over it and builds
a fresh `SessionState` with all default features, so a deep PromQL plan
pays that cost once per level.

Encode the root once in `PlanRewriter::new` and skip the per-node call
when it succeeds. `to_substrait_plan` recurses from the root to the
leaves, and `should_expand` only receives sub-trees that `f_down` pushed
on the stack untouched, so a root that encodes proves the whole descent
encodes. A root that does not encode falls back to the per-node check,
which is what locates the node that has to stay on the frontend.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-14 06:49:20 +00:00
dennis zhuang 7152ca9264 fix(promql): skip NULL samples and fix counter extrapolation order (#9118)
* fix(promql): treat NULL field values as absent samples in range functions

Range functions read the value column through `Float64Array::values()`,
which returns the raw buffer and ignores the null bitmap. A NULL field
value means the series has no sample at that timestamp, so the padding
under a null slot (0.0 in practice) was counted as a real sample.

`rate`, `increase`, `delta`, `changes`, `resets`, `idelta`, `irate`,
`quantile_over_time` and `avg_over_time` now work on samples instead of
slots. `stddev_over_time` and `stdvar_over_time` used to panic on a NULL
slot, and tokio swallowed the panic so the query returned success with
that series missing.

Null handling is gated on `null_count() == 0` over the whole backing
array, checked once per batch, so tables without NULLs keep the existing
code path. `deriv` and `predict_linear` already had this guard through
`linear_regression_slices` and are untouched.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep other fields when one has no sample in the window

Review follow-up. Two things surfaced once range functions started
returning NULL for a window without samples.

The filter after a function call required every field column to be
non-NULL, so on a multi-field table one field with no samples in a
window would drop the other fields' results with it. It now keeps a row
when any field has a sample, which is the shape a selector already
emits. On a single field column the two predicates are identical.

`quantile_over_time` returned NaN rather than NULL for a window without
samples, so the row survived that filter. Prometheus returns an empty
vector there, so the emptiness check now sits in the range UDF; the
shared quantile kernel still yields NaN for an empty slice, matching
upstream's `quantile()` helper that `quantile_aggr` depends on.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(promql): correct two comments on the null handling

The planner one did not say why `preserve_any_value` is hardcoded at
that call site, which is the question a reader arrives with. The
`quantile_over_time` one described the empty-window behaviour while
sitting on the `has_nulls` line, and that behaviour had moved into
`window_quantile`.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): clamp extrapolation before snapping a counter to zero

Prometheus clamps `durationToStart` to half an average interval once the
first sample is past the extrapolation threshold, and only then lets the
counter zero-snap shorten it further, so the snap can never lengthen the
leading extrapolation. Running the snap first let it rescue a duration
the clamp should have cut, and `rate` and `increase` over-extrapolated
to the left. For samples 1@0s and 2@1s in a 4s window at 1s, upstream
gives 0.375 and this returned 0.5.

`extrapolation_matches_prometheus_on_seeded_windows` carries a
line-by-line port of upstream `extrapolatedRate` as an oracle and diffs
it against the UDF over seeded windows, so the order stays pinned.
`factor` also picked up upstream's guard against a zero sampled
interval, which previously divided by zero.

Two sqlness results move, both verified against the upstream algorithm.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): fold range_presence_null into null_samples

#9104 landed its own NULL-sample case whose data is the same series this
one already used: one host with interior NULLs, one host with nothing
but NULLs. Keeping both means two files asserting the same semantics on
the same rows.

The merged case keeps every query from both, so the presence functions
still cover the trailing-NULL window, the count of two, the empty
left-open window at t=7, and the all-NULL windows.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-14 06:33:36 +00:00
discord9 7f949f48c0 fix(promql): preserve native timestamps through sample selection (#9070)
* fix(promql): preserve native timestamps through sample selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): retain column indices in instant plan ordering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): update native precision plan expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): preserve selector output column order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): verify preserved selector output order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): refresh native timestamp explain expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: apply PromQL offsets without native timestamp overflow

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover negative PromQL offsets at native timestamp bounds

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: allow native precision instant LastRow selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: discard unused bounds for empty range intersections

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve native time bounds independently for LastRow

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: verify native LastRow predicates and overflow through SQL

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs: explain native PromQL selection and scan invariants

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): address timestamp helper and stream review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): pass selector offsets explicitly from planner

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): retain explicit offset in payload overflow regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): record inner-offset subquery SQL results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-14 03:47:57 +00:00
Lei, HUANG 743261f05e fix(mito2): prevent JSON2 SWCS data loss from misaligned Parquet statistics due to projection (#9129)
* fix(mito2): look up row group stats by parquet leaf index for nested columns

On flat-format tables a logical column can expand to multiple parquet
leaf columns (e.g. a JSON2 struct stores the remainder and one leaf per
promoted path). ParquetFlat used the logical column index in the SST
schema directly as the leaf index when reading row group statistics, so
min/max/null stats of every column after a nested column were read from
wrong leaves.

When the misplaced leaf held order-compatible statistics (e.g. a small
Int64 JSON path vs. the timestamp window predicate), min-max pruning
dropped whole row groups by mistake. SWCS compaction reads inputs with a
time window predicate, so it silently lost all rows of such files; plain
queries with time-range predicates were affected as well.

Map each column to its first parquet leaf column and report NoStats for
columns with multiple leaves, which makes pruning conservative for them.
Add a unit test and a sqlness regression case that reproduces the data
loss on the unfixed binary.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): skip nested root stats and correct SWCS regression baseline

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): resolve statistics leaves for primary-key SST readers

Resolve scalar roots against the actual Parquet schema in shared statistics helpers, covering both flat and primary-key readers. Remove flat-side translation to avoid mapping twice and align encoded primary-key statistics as well. Cover dense flat, legacy dense and sparse layouts with statistics and time-pruning regressions.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): distinguish known null counts from unknown statistics

Assert validity before reading timestamp null counts and add a nullable scalar after the nested root with a known nonzero count. Exercise the assertions for flat and primary-key SST layouts.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): check JSON2 time pruning before SWCS compaction

Query the first time window immediately after FLUSH to cover predicate reads on flush-written SSTs independently of compaction outputs. Regenerate the sqlness expectation and retain the post-compaction checks.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): validate all business columns after JSON2 SWCS

Expand the final regression query to all eight business columns so the generated expectation verifies complete rows, including tags and scalar fields, after repeated compaction.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-14 03:44:51 +00:00
Ning Sun ece78a4dad ci(backport): assign backport-failure issue to the original PR author (#9122)
Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-14 03:41:10 +00:00
Lei, HUANG 9696882217 fix(mito): preserve mixed JSON2 types during compaction (#9135)
* fix(mito): preserve mixed JSON2 types during compaction

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito): assert all restored JSON2 rows in unordered merge test

The regression test for aligning JSON2 layouts across unordered bulk
parts only round-tripped the first part's `a` values. A merge that
dropped or corrupted the second source's `b` values would still pass.

Assert the merged batch has four rows and that the restored values of
the second part (`{"b": 3}`, `{"b": 4}`) survive the merge, so the
test covers both opaque sources as its comment claims.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-14 03:38:53 +00:00
discord9 1fce0eb6d8 fix(ci): increase query regression ECS disk to 80 GiB (#9123)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-14 02:45:10 +00:00
dennis zhuang 3d1278b39b docs: restructure README and add a cross-signal SQL example (#9136)
The README claimed signals could be correlated in SQL but contained no SQL
at all, and 68 lines of badges, social buttons and a hand-written TOC pushed
the first section off the opening screen.

- Add a cross-signal join over opentelemetry_traces and opentelemetry_logs
- Drop the hand-written TOC in favour of GitHub's built-in one, and collapse
  the badge block; move CI and coverage badges to Contributing
- Fold the Overview section into What's Supported, keeping the diagram
- Give Benchmarks its own section instead of nesting it under Compatibility
  and Migration, and add Agent RCA Bench
- Split the star request and user logos out of Project Status
- Replace the edge-device bullet, which nothing else in the README supports,
  with Loki query performance

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-14 02:11:59 +00:00
discord9 555485c40e perf(promql): avoid per-window allocations in simple range functions (#9104)
* perf(promql): avoid per-window allocations in simple range functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* perf(promql): avoid copying smoothing window values

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): cover smoothing copy removal across window layouts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): skip null samples in simple range functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-11 11:59:45 +00:00
Lei, HUANG 21aed0371f feat(pprof): switch CPU profiler to framehop unwinder (#9125)
* feat(pprof): switch CPU profiler to framehop unwinder

Replace the default libgcc-based unwinder in pprof-rs with the
framehop unwinder, which is designed to be async-signal-safe:

- framehop performs no heap allocation during unwinding
  (MustNotAllocateDuringUnwind)
- It handles prologue/epilogue interruption correctly
- It falls back to frame-pointer unwinding when CFI is unavailable
- It does not depend on libgcc's unwind implementation, which is
  documented as not signal-safe (see tikv/pprof-rs#36)

Bump pprof from 0.14 to 0.15 in all three consumers (common-pprof,
cmd, servers) to unify on a single version. pprof 0.15 also replaces
parking_lot with spin-rs to avoid a potential profiler deadlock (#268).

This addresses the libgcc_s.so.1 #GP crash observed in production
when CPU profiling is active, by eliminating the libgcc unwinder
from the signal handler path entirely.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(pprof): gate framehop-unwinder to supported targets

framehop-unwinder is only available on x86_64/aarch64 Linux/macOS.
Enabling it unconditionally for all Unix targets breaks the build on
riscv64 and other platforms: pprof disables its backtrace-rs fallback
when framehop-unwinder is set, but the framehop module is not compiled
on unsupported targets, leaving no TraceImpl implementation.

Split the pprof dependency: the base target.'cfg(unix)' block carries
the common features (flamegraph, prost-codec, protobuf), and a separate
target block adds framehop-unwinder only on supported targets.
Cargo unions features from both blocks on matching targets, so x86_64
and aarch64 Linux/macOS get the full feature set while other Unix
targets fall back to the default backtrace-rs implementation.

Addresses review comment discussion_r3987313632.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-11 09:39:13 +00:00
discord9 97648525cf perf(mito2): skip proven all-match prefilters (#9066)
* perf(mito2): skip proven all-match prefilters

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): add manual all-match prefilter reproduction

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): cover all-match prefilter execution paths

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): match prefilter fixture to sparse SST schema

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(mito2): address all-match prefilter lint findings

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): cover all-match prefilters in default regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-11 09:35:57 +00:00
Yingwen b8e0f8e62e feat(mito2): add series index planning and builders (#9085)
* feat(mito2): add series index planning and builders

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): track window coverage and separate index builds

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): normalize index inputs and bound SST window expansion

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): simplify series index source summaries

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore(mito2): assign series index deduplication TODO

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): initialize skip_wal in series index tests

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-11 08:11:36 +00:00
Weny Xu a673e084b2 test: cover request-level insert WAL skipping end to end (#9093)
* test: cover request-level WAL skipping end to end

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover session WAL policy and COPY recovery in sqlness

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): isolate Mito test feature in dev dependencies

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: parameterize WAL protocol cases and make setup explicit

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover skip-WAL hints across streaming messages

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-11 07:48:57 +00:00
Lei, HUANG 9619085310 chore: bump tikv-jemalloc-sys patch to jemalloc dev (ff80bf2d) (#9119)
* chore: bump tikv-jemalloc-sys patch to jemalloc dev (ff80bf2d)

Update the [patch.crates-io] rev for tikv-jemalloc-sys from
e1846d8c (5.3.1 + 54f22c83 backport) to ff444d4 (upstream dev HEAD,
161 commits ahead of 5.3.1).

The dev branch includes additional TSD/tcache fixes beyond the
original backport:
- fb5499aa9c: Handle jemalloc calls after TSD teardown
- 61dc1da395: Fix possible tcache corruption on fiber migration
- 1e92317014: Fix thread-exit TSD cleanup

See GreptimeTeam/jemallocator branch bump-jemalloc-dev and
tikv/jemallocator#182.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main

GreptimeTeam/jemallocator#2 has been merged; reference the merge
commit e254a7ea on main instead of the PR head branch. Jemalloc
source content is unchanged (still upstream dev ff80bf2d).

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-11 07:29:43 +00:00
Weny Xu 1c4d3a1729 fix(ci): check Windows test targets before merge (#9116)
* fix(ci): check Windows test targets before merge

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): use standard Windows runner for checks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): omit dashboard assets from Windows checks

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-11 07:04:41 +00:00
Dhruv Vaishnav 83ff0d8138 feat(meta): record catalog and database reconciliation events (#8896)
* feat(meta): record catalog and database reconciliation events

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): join catalog and database reconciliation events

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
2026-09-11 07:00:44 +00:00
fys f3b29ee700 chore: select cmd package when building greptime (#9113)
Signed-off-by: fys <fengys1996@gmail.com>
2026-09-11 04:58:12 +00:00
Ning Sun 9dabe8ccd2 docs: fix README canary release badge resolving to stable version (#9109)
The canary badge's filter (!*-*-*) only excludes tags with two or more
dashes (nightly and dev builds), so zero-dash stable tags also pass.
Because shields.io picks the newest release by date and v1.2.0 GA was
created after v1.3.0-alpha.1, the canary badge displayed v1.2.0, which
is already covered by the stable badge.

Add sort=semver so shields picks the semver-greatest matching tag
instead: v1.3.0-alpha.1 now ranks above v1.2.0. Also apply sort=semver
to the stable badge to protect it from the same date-order failure when
a patch of an older minor is released after a newer GA.

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-10 14:20:53 +00:00
Lei, HUANG ab0b1f5cce fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix (#9103)
* fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix

Upgrade tikv-jemallocator / tikv-jemalloc-ctl / tikv-jemalloc-sys from
0.6 to 0.7, which embeds jemalloc 5.3.1 (includes a056c20d 'Handle
tcache init failures gracefully').

On top of that, patch tikv-jemalloc-sys to the GreptimeTeam fork that
adds the remaining upstream fix 54f22c83 'Initialize TSD tcache before
enabling it' (GreptimeTeam/jemalloc#1, GreptimeTeam/jemallocator#1).
Without the ordering fix, a reentrant allocation during TSD bootstrap
(e.g. heap-profiling prof_tdata init / sampled backtrace when prof:true
is active) can observe an enabled-but-uninitialized tcache, corrupting
per-thread tcache metadata and crashing the process in arena_stats_merge,
calloc, or the libgcc unwinder.

The patch is pinned by rev and should be removed once tikv/jemallocator
ships a jemalloc snapshot that includes 54f22c83.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: bump tikv-jemalloc-sys patch rev to merged release-5.3.1

GreptimeTeam/jemalloc#1 has been merged; point the patch at the
jemallocator commit referencing the merge commit on release-5.3.1.
Jemalloc source content is unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main

GreptimeTeam/jemallocator#1 has been merged; reference the merge
commit e1846d8c on main instead of the PR head branch. Jemalloc
source content is unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(servers): bump tikv-jemallocator dev-dependency to 0.7

main added a target.'cfg(not(windows))'.dev-dependencies entry on
tikv-jemallocator 0.6 for servers after this branch diverged. On the
merge ref it pulled tikv-jemalloc-sys 0.6 from crates.io, which
conflicts with the patched 0.7 (links = "jemalloc" may only appear
once in the dependency graph), failing version selection in CI.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-10 13:06:43 +00:00
discord9 5d5f0d6d70 fix(query): prevent incomplete aggregate dynamic filtering (#9102)
* fix(query): prevent incomplete aggregate dynamic filtering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): cover mixed expression and column maxima

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): pin upstream aggregate regression backport

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): assert aggregate filter behavior on datanode scans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): pin merged aggregate dynamic filter fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-10 12:52:35 +00:00
Weny Xu fa128adb8e feat: support request-level insert WAL skipping (#9088)
* refactor: add skip_wal fields to internal write requests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore(deps): update greptime-proto for insert skip_wal

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): support request-level WAL skipping

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(metric-engine): handle request-level WAL policies

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: propagate insert WAL policy through query context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: support session-level insert WAL policy via SET

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metric-engine): require uniform WAL policy in batch puts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(metric-engine): simplify WAL policy coverage

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(servers): simplify gRPC hint extraction

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: flatten Mito and Metric WAL scenario orchestration

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: separate WAL and memtable-only mutations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: carry skip-WAL policy in table insert requests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: flatten skip-WAL policy cases

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: clarify WAL notifier naming

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: pin merged skip-WAL proto revision

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-10 11:40:51 +00:00
fys 536f42e4a2 feat(json2): support ALTER syntax for JSON column settings (#9094)
feat(json2): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

This is the commit message #3:
2026-09-10 10:33:41 +00:00
dennis zhuang 0f625a7e92 fix(promql): correct counter reset accumulation in rate windows (#9089)
* fix(promql): correct counter reset accumulation in rate windows

`prom_rate` and `prom_increase` reused the previous window's counter-reset
correction when the next window slid forward by exactly one sample, adding
the entering reset and subtracting the leaving one. Running a sum through
addition and subtraction does not restore the earlier terms in f64: a large
reset absorbs the smaller ones that must survive it, and an expired infinity
leaves a NaN that no later window can clear. `prom_delta` shares the code but
is not a counter function, so it never took that path.

Index the reset positions of the value array once instead, and reduce each
window over the resets it contains, in sample order. The result is
bit-identical to scanning the window directly, so windows keep the direct
reduction when they request fewer sample pairs than the input has.

Also sweep the query step in the rate benchmarks: the cost of the reset
correction depends on how much the windows overlap, which no existing case
varied.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* test(promql): cover counter reset precision over adjacent rate windows

The unit tests build the range windows directly, so they do not show that a
plain PromQL range query produces the window layout that lost the correction.
This case does: with a query step equal to the sample interval, `increase`
over the second window returns 1.333 before the fix and 2.667 after it.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* perf(promql): advance the counter reset bounds instead of searching

Locating a window's resets with two binary searches costs more than the
reduction it replaces once a series resets often enough for the searches to
get deep: on a 20k-sample counter resetting every 37 samples, stepping the
windows by one sample was 2.7x slower than the previous code, against 1.2x
for a counter that never resets.

Windows normally advance, so walk the bounds forward from the previous
window and only search when they move back. The cost then no longer depends
on the reset density.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* perf(promql): cut the per-window cost of the counter reset index

Two costs the index added showed up on a one-sample query step, where the
removed fast path used to answer each window with two comparisons.

Cache the two reset positions that bound the active slice. A window that only
advanced and reached neither of them covers the same resets as the previous
one, so the common case is four integer comparisons and no lookup at all.

Stop summing the requested sample pairs once they exceed one pass over the
values. The sum only decides which side of that comparison the input falls on,
and a query with a short lookback and a long step settles it after a few
windows instead of after every key.

Together these take the one-sample step from 25-32% slower than the previous
code down to 6-11%, measured as before / after / before to bound drift. No
other step value regresses, and a ten-sample step stays about 88% faster.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* fix(promql): accumulate counter resets into the running result

`prom_rate` and `prom_increase` summed a window's counter-reset corrections
on their own and added that sum to `last - first`. Prometheus folds each
reset into the running result instead, and so did this code before #7880.
The two are not interchangeable in f64: over samples `[1e16, 1, 0, 1]` the
isolated sum rounds `1e16 + 1.0` back to `1e16`, which then cancels against
the first sample and reports no increase at all, where folding the resets in
one at a time keeps the 1.0.

Restore the original order. The reset index accumulates into the result the
same way, so it still matches a direct scan of the window bit for bit, but a
window's contribution can no longer be cached as a standalone value and is
re-added from its own difference each time. The bounds are still cached, so
a window that did not cross a reset skips the lookup, and one that holds no
resets returns without touching the index at all.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* test(promql): note which reset boundaries the stride of one walks

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

---------

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
2026-09-10 07:11:39 +00:00
Weny Xu adda50e03f feat: add repartition partition count hint (#9080)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-10 05:04:21 +00:00
discord9 2d78220a4f fix(flow): drain frontend probe response before selecting peer (#9082)
* fix(flow): drain frontend probe response before selecting peer

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(flow): reuse record batch collector for frontend probe

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-10 04:28:18 +00:00