Commit Graph
655 Commits
Author SHA1 Message Date
Weny Xu e520ff8300 feat: decouple error retryability from status codes (#8301)
* feat: add retry hint to common error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: propagate retry hints in core errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: propagate retry hint through RPC metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fallback retry hints to status codes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: add explicit retry hints for retryable errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: remove status code retry fallback

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: classify io retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minior

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve datasource retry hints in query

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve metric engine retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve flow and frontend retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: share rskafka retry hint mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-17 08:27:53 +00:00
Lei, HUANG 4373c77d35 feat: use scan output bytes for read costing (#8276)
* feat: track scan output bytes and use them for read costing

Track the actual byte count of each record batch produced by
`RegionScanExec` and use it in place of the aggregated per-plan
`memory_usage` as the `table_scan` cost input. This avoids double
counting bytes that flow through multiple operators.

A named constant `REGION_SCAN_EXEC_NAME` exposes the plan node name
so downstream metric parsers remain correct if the struct is renamed.

Affected files:
- `src/table/src/table/metrics.rs` -- add `output_bytes` counter
- `src/table/src/table/scan.rs` -- record bytes, export name constant
- `src/query/src/dist_plan/merge_scan.rs` -- consume scan bytes

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: remove misleading `mem_used` gauge from scan metrics

The `mem_used` gauge was used with `add()` on every output batch, but a
RegionScan does not hold memory across polls — each batch is yielded
immediately to the upstream operator. The gauge semantics were
incorrect (cumulative `add()` on a gauge) and the value duplicated
`output_bytes` anyway.

Affected files:
- `src/table/src/table/metrics.rs` — drop `mem_used` field and methods
- `src/table/src/table/scan.rs` — remove `record_mem_usage` call

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: use stable plan names for scan byte metrics

Expose `plan_name` alongside rendered plan text so scan byte extraction no longer depends on EXPLAIN formatting. Keep old serialized metrics compatible by defaulting missing `plan_name`.

Affected files:
- `src/common/recordbatch/src/adapter.rs` -- add `plan_name` to `PlanMetrics` and populate it from `ExecutionPlan::name`
- `src/query/src/dist_plan/merge_scan.rs` -- match `REGION_SCAN_EXEC_NAME` through `plan_name`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: preserve merge scan CPU read cost

Record read cost whenever stream metrics are available, default scan bytes to zero, and aggregate scan output bytes across region scan nodes.

Files: `src/query/src/dist_plan/merge_scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: correct doc comment in `StreamMetrics`

Fix doc comment on `StreamMetrics::new()` to reference the correct
struct name (`StreamMetrics`) instead of the old `MemoryUsageMetrics`.

- `src/table/src/table/metrics.rs` — fix struct name in doc comment

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2026-06-16 02:58:13 +00:00
Yingwen 4a4fe749d8 feat: add per-partition timings to merge scan partition metrics (#8293)
* feat: add per-partition timings to merge scan partition metrics

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: stop global timers for partitions with no region

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-06-12 13:22:49 +00:00
discord9 3608c90b2e feat: pass snapshot read bounds over flight (#8279)
Signed-off-by: discord9 <discord9@163.com>
2026-06-11 12:05:02 +00:00
dennis zhuang ca21c9f048 fix!: correct information_schema index metadata (#8275)
fix: correct information_schema index metadata

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-06-11 01:36:23 +00:00
discord9 8fc5a3b1c7 feat: apply remote dynamic filters on datanode scans (#8262)
* feat: apply rdf

Signed-off-by: discord9 <discord9@163.com>

* chore: clippy

Signed-off-by: discord9 <discord9@163.com>

* fix: drop remote dyn filter fallback exec

Signed-off-by: discord9 <discord9@163.com>

* Revert "fix: drop remote dyn filter fallback exec"

This reverts commit bb757a596c3c2c078a836aa623226b7934103634.

Signed-off-by: discord9 <discord9@163.com>

* refactor: use rdf receiver logical plan instead

Signed-off-by: discord9 <discord9@163.com>

* test: update sqlness

Signed-off-by: discord9 <discord9@163.com>

* feat: rdf disable option

Signed-off-by: discord9 <discord9@163.com>

* tests: large int tests

Signed-off-by: discord9 <discord9@163.com>

* chore: clippy

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* test: update prec fix

Signed-off-by: discord9 <discord9@163.com>

* fix: make receiver node works

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* fix: tql disable rdf

Signed-off-by: discord9 <discord9@163.com>

* chore: rm useless joins

Signed-off-by: discord9 <discord9@163.com>

* fix: also disable in flow tql

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review rm to promql

Signed-off-by: discord9 <discord9@163.com>

* chore: promql ut

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

* test: rm misleading&add some nested/cleanup

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-10 12:57:23 +00:00
dennis zhuang e403133eb2 feat: add information_schema statistics table (#8253)
* feat: add information_schema statistics table

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: use index-local sequence in statistics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: ordinal_position for pk

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: statistics.nullable uses empty string for non-nullable columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-06-08 12:52:02 +00:00
Weny Xu 9a814ca123 chore: add repartition debug logs (#8245)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-08 08:48:23 +00:00
discord9 651cf04525 feat: fan out remote dynamic filter updates from FE (#8241)
* feat: fe send update

Signed-off-by: discord9 <discord9@163.com>

* refactor: after rebase

Signed-off-by: discord9 <discord9@163.com>

* fix: encode at least send bounds

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-08 08:24:57 +00:00
Ruihang Xia 428cd8137d feat: identify noneffective binary modifiers (#8230)
* perf(promql): use tsid for full-label modifier joins

* fix(promql): collect narrow range joins by tsid

* verify on for both sides

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* check if tsid exists

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* correct behavior

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-06-05 08:44:50 +00:00
discord9 6b7772e457 feat: add remote dynamic filter frontend registration (#8148)
* feat: filter id

Signed-off-by: discord9 <discord9@163.com>

* feat: dyn filter registry

Signed-off-by: discord9 <discord9@163.com>

* feat: filter id&refactor to type

Signed-off-by: discord9 <discord9@163.com>

* feat: merge scan register dyn filter(not send yet)

Signed-off-by: discord9 <discord9@163.com>

* feat: init reg dyn filter

Signed-off-by: discord9 <discord9@163.com>

* wip: remote dyn filter task 03

Signed-off-by: discord9 <discord9@163.com>

* fix: resolve remote dyn filter rebase fallout

Signed-off-by: discord9 <discord9@163.com>

* chore: keep remote dyn filter docs local

Signed-off-by: discord9 <discord9@163.com>

* chore: remove stale filter id allow

Signed-off-by: discord9 <discord9@163.com>

* chore: clippy

Signed-off-by: discord9 <discord9@163.com>

* chore: fix remote dyn filter import style

Signed-off-by: discord9 <discord9@163.com>

* chore: fix query metrics test fallout

Signed-off-by: discord9 <discord9@163.com>

* fix: exclude region from remote dyn filter id

Signed-off-by: discord9 <discord9@163.com>

* chore: import

Signed-off-by: discord9 <discord9@163.com>

* refactor: rm some to latter

Signed-off-by: discord9 <discord9@163.com>

* feat: add initial dyn filter snapshot

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* docs: better comment, rm some slop

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-04 12:31:42 +00:00
Ruihang Xia edd563d2e1 feat: join simplifier for promql binary op (#8211)
* basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor, remove shallow methods

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* more sqlness cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clear ctx

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* prefix leaf aliase

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-06-04 08:13:24 +00:00
Ruihang Xia 85ae29cb0c perf: collect narrow binary join (#8193)
* perf(promql): collect narrow binary join build side

* fix projection

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* finalize

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename mod

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-05-28 11:30:30 +00:00
discord9 eccd97b5c7 feat(flow): support incremental read checkpoints (#8179)
* feat: flownode inc mode

Signed-off-by: discord9 <discord9@163.com>

* chore: rename fallback reason

Signed-off-by: discord9 <discord9@163.com>

* fix: harden flow incremental checkpoints

Signed-off-by: discord9 <discord9@163.com>

* fix: address flow watermark lint

Signed-off-by: discord9 <discord9@163.com>

* fix: address flow clippy

Signed-off-by: discord9 <discord9@163.com>

* refactor: clarify incremental plan preparation

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* test: more sqlness test

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-05-28 09:31:46 +00:00
LFC bf7e3551fe test: add jsonbench tests (#8165)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-05-27 08:34:06 +00:00
Yingwen 9487e2c3ca fix: divide series for subquery output (#8173)
* fix: divide series for subquery output

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: propagate time index lookup error in prom_call_manipulate

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-05-27 07:10:24 +00:00
discord9 44f1804b5e feat: add flow query-context plumbing for terminal watermarks (#8154)
* feat: add flow checkpoint plumbing

Signed-off-by: discord9 <discord9@163.com>

* fix: restore when fail

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: clean up some test

Signed-off-by: discord9 <discord9@163.com>

* clippy

Signed-off-by: discord9 <discord9@163.com>

* refactor: move more to pr3b

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-05-26 07:24:18 +00:00
Yingwen a25152664b fix: qualify HistogramFold schema (#8157)
* test: add regression test for binary op on histogram_quantile (#8144)

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: preserve column qualifiers in HistogramFold output schema (#8144)

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-05-25 07:40:48 +00:00
discord9 f1ad472075 fix(mysql): infer LIMIT placeholders in prepare (#8149)
Signed-off-by: discord9 <discord9@163.com>
2026-05-21 23:40:10 +00:00
Ruihang Xiaanddiscord9 3b3f5d628d feat: bump datafusion to 53 (#8107)
* feat: bump datafusion to 53

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use updated datafusion-orc

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* maintain topk in part sort

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* docs: clarify part_sort range detection and Null threshold semantics

* fix limit 0

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: discord9 <discord9@163.com>
2026-05-14 08:33:31 +00:00
Ruihang Xia aa0ff3cce7 feat: some optimistic paths for instant manipulate (#7812)
* feat: some optimistic paths for instant manipulate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use tsid in manipulate plan, resolve_tag_columns walks whole plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tsid reuse

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test assertions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* cap max points

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-05-14 02:36:25 +00:00
Ruihang Xia 9133d0464f feat: pre-cast constants (#7926)
* init impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle no cast

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor using common-expr

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* extend matching pattern

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* more tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplification

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix zero timestamp

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: normalize sqlness partition count output

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: normalize remaining sqlness plan output

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: normalize sqlness repartition details in tql explain

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: tighten const normalization casts

* test: normalize standalone tql explain repartition output

* resolve cr comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-05-12 09:23:43 +00:00
Ruihang Xia 0d90f7407c fix: infer time index from column meta on derived table (#8013)
* rough fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reorganize

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplification

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* enhance default by infer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* supply comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-05-12 08:18:46 +00:00
discord9anddennis zhuang a282b207a6 feat: flow inc query terminal metrics transport (#8045)
* feat: OutputMetrics for inc query

Signed-off-by: discord9 <discord9@163.com>

* chore: rm unused

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* docs: explain should_collect_region_watermark_from_extensions

Signed-off-by: discord9 <discord9@163.com>

* fix: transport flow extensions via metadata

Avoid routing Flow-specific query extensions through comma-separated hints so checkpoint JSON values remain intact over Flight.

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* more per review

Signed-off-by: discord9 <discord9@163.com>

* chore: more review

Signed-off-by: discord9 <discord9@163.com>

* test: compat test

Signed-off-by: discord9 <discord9@163.com>

* fix: standalone also monkey patch

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2026-05-11 11:58:51 +00:00
Ning Sun 5b47ec24ec fix: type inference for sql rewrite (#8052)
fix: type inference for rewrited sql
2026-05-11 08:20:46 +00:00
LFC 7a285c2890 feat: concretize json type from query (#8081)
* feat: concretize json type from query

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* add more tests

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-05-09 07:27:40 +00:00
LFC 160b7e720b feat: json expr planner (#8066)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-05-07 06:44:46 +00:00
discord9 760581b2a0 fix: windows windowed sort ci (#8039)
* fix: windows windowed sort ci

Signed-off-by: discord9 <discord9@163.com>

* chore

Signed-off-by: discord9 <discord9@163.com>

* c

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-04-28 02:35:44 +00:00
discord9 d2d256909f feat(flow): parse defer on miss src table (#7980)
* feat: parse create flow with

Signed-off-by: discord9 <discord9@163.com>

* feat: validate after parse

Signed-off-by: discord9 <discord9@163.com>

* pcr

Signed-off-by: discord9 <discord9@163.com>

* chore: sqlness

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-04-27 03:02:13 +00:00
discord9 9fafd879ed fix: window sort off by one precision TimeRange&better alias track (#8019)
* fix: window sort track alias&off by one precision TimeRange

Signed-off-by: discord9 <discord9@163.com>

* chore: more test

Signed-off-by: discord9 <discord9@163.com>

* refactor: clear helper

Signed-off-by: discord9 <discord9@163.com>

* dedup a bit

Signed-off-by: discord9 <discord9@163.com>

* feat: even more guard

Signed-off-by: discord9 <discord9@163.com>

* fix: case insensitive

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-04-24 13:56:51 +00:00
discord9 1167c36c4e feat: add seq watermark in record batch metrics (#8015)
* feat: add seq watermark in record batch metrics

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-04-24 07:54:10 +00:00
Ning Sun 80c395ee23 refactor: update SqlPlan for more cleaner variants (#7966)
* refactor: update SqlPlan for more cleaner variants

* refactor: change how we check readonly plan

* fix: don't return schema for non-query statement

* chore: reflect review comments

* fix: federated statements
2026-04-21 11:50:47 +00:00
discord9 8123406fae feat(flow): inc query scan bind seq (#7879)
* feat: add sequence scan boundaries and stale detection

Signed-off-by: discord9 <discord9@163.com>

* fix: only scan bind once

Signed-off-by: discord9 <discord9@163.com>

* refactor: flow snapshot decision

Signed-off-by: discord9 <discord9@163.com>

* refactor: one place modify

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: pre review

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-04-20 03:32:31 +00:00
Ruihang Xia 3fe8a61fad perf: join metrics tables on the tsid key whenever possible (#7927)
* feat: prefilter flat parquet scans by primary key

* perf: skip redundant series divide repartitions

* perf: optimize tsid promql join planning

* perf: preserve tsid distribution through merge scans

* perf: remove redundant tsid join repartitions

* fix multi-field join case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Revert "feat: prefilter flat parquet scans by primary key"

This reverts commit 767c3b44c8.

* simplification

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* isolate rule into a dedicated mod

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove rule

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* more sqlness cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix filter join case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: normalize sqlness repartition input count

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: normalize sqlness partition count in promql regression

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: normalize sqlness hash partition fanout

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplification

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-04-14 19:43:58 +00:00
fys c90e4147de refactor: introduce the ProjectInput structure (#7908)
* refactor: introduce the ProjectInput structure

* remove unused import

* fix: cr

* fix: cr

* fix: code review

* add more unit test

* avoid clone of input.projection
2026-04-14 09:29:33 +00:00
Ruihang Xia 09b368c00a feat: tune constants (#7851)
* feat: tune constants

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* cap output batch size

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle empty input

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* one more ut for cr

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-04-08 23:34:13 +00:00
discord9 b623cb1aa2 perf: no longer window sort when limit (#7912)
* perf: no longer window sort when limit

Signed-off-by: discord9 <discord9@163.com>

* test: confusing vector sqlness

Signed-off-by: discord9 <discord9@163.com>

* chore: redact sqlness

Signed-off-by: discord9 <discord9@163.com>

* chore: redact every thing

Signed-off-by: discord9 <discord9@163.com>

* REDACTED

Signed-off-by: discord9 <discord9@163.com>

* what

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-04-08 02:54:22 +00:00
Ning Sun e14404c677 chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21

* chore: new format

* fix: lint

* chore: resolve lint issues

* chore: remove as_millis_f64

* chore: deps up
2026-03-30 12:13:14 +00:00
discord9 d7bc5ad16b feat: add incremental read context and scan boundaries (#7848)
* feat: add incremental read context and scan boundaries

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* docs: explain field

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-03-26 07:00:22 +00:00
jeremyhi 8058ce7cf2 refactor: simplify scan memory tracking (#7827)
* refactor: simplify scan memory tracking

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: make confg-docs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by codex review comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: track_with_policy

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: minor change

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: mem granularity mb to kb

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by review comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by scan_memory_on_exhausted comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by review comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: typo

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-03-26 03:25:50 +00:00
Ning Sun 73b48b14c1 feat: update postgres ParameterDescription size limit (#7861)
* feat: update postgres ParameterDescription size limit

* chore: don't log io error
2026-03-26 03:10:39 +00:00
Ruihang Xia 59dd418629 feat: simplify nested aggr inside count query (#7859)
* as optimizer rule

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* dump changes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* perf: tighten count-count optimizer rewrite

* extend inner op set

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify and more coverage

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove prom-non-null

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* preserve value column through pruning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* more sqlness cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* enforce is not null before inner aggr

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* finalize

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-03-26 00:08:38 +00:00
discord9 30e895abbe fix: prom cast to f64 (#7840)
* fix: cast to f64

Signed-off-by: discord9 <discord9@163.com>

* test: div case

Signed-off-by: discord9 <discord9@163.com>

* test: int test

Signed-off-by: discord9 <discord9@163.com>

* chore: sqlness update

Signed-off-by: discord9 <discord9@163.com>

* chore: test

Signed-off-by: discord9 <discord9@163.com>

* chore: update test

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-03-24 06:24:52 +00:00
Ning Sun d14817bfa6 fix: resolve optimization issue for extended query (#7824)
* fix: resolve optimization issue for extended query

* fix: type cast from subquery

* chore: update error information in sqlness

* chore: switch to released pgwire

* refactor: remove optimize function completely

* chore: add more tests

* test: attempt to fix the fuzz issue

* fix: try to resolve the test issue
2026-03-20 03:58:39 +00:00
discord9 922f9cb3d6 feat: use dyn filter (#7545)
* parent b2074e3863
author discord9 <discord9@163.com> 1767869295 +0800
committer discord9 <discord9@163.com> 1772529023 +0800

feat: use dyn filter

Signed-off-by: discord9 <discord9@163.com>

not supported

Signed-off-by: discord9 <discord9@163.com>

refactor: use make_mut instead

Signed-off-by: discord9 <discord9@163.com>

refactor: rm need to clone stream ctx

Signed-off-by: discord9 <discord9@163.com>

r

Signed-off-by: discord9 <discord9@163.com>

pcr

Signed-off-by: discord9 <discord9@163.com>

test: wait for datafusion update

Signed-off-by: discord9 <discord9@163.com>

refactor: use arc swap for dyn filters

Signed-off-by: discord9 <discord9@163.com>

* test: update sqlness

Signed-off-by: discord9 <discord9@163.com>

* chore: comment out sqlness

Signed-off-by: discord9 <discord9@163.com>

* test: update sqlness

Signed-off-by: discord9 <discord9@163.com>

* test: sqlness fix

Signed-off-by: discord9 <discord9@163.com>

* refactor: predicate without option

Signed-off-by: discord9 <discord9@163.com>

* feat: print dyn filters& more tests

Signed-off-by: discord9 <discord9@163.com>

* test: sqlness vector result update

Signed-off-by: discord9 <discord9@163.com>

* chore: log

Signed-off-by: discord9 <discord9@163.com>

* test: properly redact

Signed-off-by: discord9 <discord9@163.com>

* test: better data dist for non empty dyn filter

Signed-off-by: discord9 <discord9@163.com>

* test: properly redacted

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* properly redact

Signed-off-by: discord9 <discord9@163.com>

* docs:  explain why not  do it

Signed-off-by: discord9 <discord9@163.com>

* chore: rename update to add as its more proper

Signed-off-by: discord9 <discord9@163.com>

* chore: rm no need clone

Signed-off-by: discord9 <discord9@163.com>

* docs: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-03-11 03:06:24 +00:00
Ning SunandRuihang Xia 9b1288f66a feat: add function rewrite rule for json_get with cast (#7631)
* feat: initial function rewriter for json_get

* feat: make sure rewrite rule is applied

* feat: keep analyzer's default rules

* feat: implement rewriter for arrow_cast

* test: add unit test for tht rewriter

* chore: format

* refactor: extract some more functions

* Apply suggestion from @waynexia

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2026-03-09 02:39:48 +00:00
Ruihang Xia 39140058d0 feat: adapt new name of holt winters fn (#7700)
* feat: adapt new name of holt winters fn

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update parser

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* alias old fn name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-03-04 23:32:37 +00:00
b2074e3863 chore: upgrade DataFusion family, again (#7578)
* chore: upgrade DataFusion family

Signed-off-by: luofucong <luofc@foxmail.com>

* chore: switch to released version of datafusion-pg-catalog

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-03-03 07:36:39 +00:00
Yingwen 0c30bf1a10 feat: add a subcommand to bench scan (#7722)
* feat: support scan bench

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: support projection by name

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: support force flat format

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: spawn tasks to poll streams

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: support filter config

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: scan bench support wal

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: support not providing provider in wal

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: skip wal replay

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: wrap EngineComponents

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: add scanbench doc

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: change --skip-wal-replay to --enable-wal

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: remove limit from config

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-02-26 06:37:40 +00:00
dennis zhuangandCopilot 6d998c043e feat: implements anomaly_score_iqr, anomaly_score_mad etc. (#7681)
* feat: implements anomaly_score_iqr, anomaly_score_mad and anomaly_score_zscore functions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: forgot result file

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: return NaN and zero for scale=0

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: comment

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: signature, cast and degenerate case

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: tweak comment

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: format

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: update min samples threshold

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: missing closing quote

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: mock engine

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: oudated comment and test name

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: todo

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: apply suggestions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-25 07:01:22 +00:00