mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-20 01:34:02 +00:00
eb8564ef32a3991fbfc2a49a46b0e08d34db57f3
55
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
+1 |
badceec12a |
fix: backport selected fixes to v1.3 (#10006)
* fix(hdfs): clean up temporary file when an atomic write fails (#9483) * fix(hdfs): clean up temporary file when an atomic write fails Signed-off-by: fengyue-xve <1759519137@qq.com> * style: apply rustfmt to hdfs layer test Signed-off-by: fengyue-xve <1759519137@qq.com> * fix(hdfs): report earlier write failures and log cleanup failures - Later `write`/`close` calls on a writer whose failed write was cleaned up now mention the original failure instead of the bare "unavailable after close or abort" message. - `abort_and_delete` logs abort and deletion failures with the temporary path instead of discarding them; the original write error is still returned to the caller. - The temporary-file cleanup test now lists the directory holding the temporary file, so it fails if the cleanup is reverted. Signed-off-by: 风月 <1759519137@qq.com> Signed-off-by: fengyue-xve <1759519137@qq.com> --------- Signed-off-by: fengyue-xve <1759519137@qq.com> Signed-off-by: 风月 <1759519137@qq.com> (cherry picked from commit |
||
|
|
56c5978647 |
fix(query): stop windowed sort on precision-lowering time casts [Backport release/v1.3] (#10013)
fix(query): stop windowed sort on precision-lowering time casts (#9524)
* test: cover windowed sort cast unit mismatch
* fix(query): stop windowed sort on unit- or timezone-changing time casts
A cast on the time index column keeps the windowed-sort optimization
eligible only when it preserves the underlying timestamp values. Casts
that change the time unit (e.g. ms -> s) or that add a timezone to a
naive timestamp re-interpret the values, so the working ranges derived
from the time index no longer describe the sort column and the scan
fails with a working-range error or silently wrong order. Reject both
cast shapes so such queries fall back to the normal SortExec path.
The integration fixture uses two rows 1000 seconds apart so the cast
output stays distinct and the ordering assertions are not vacuous; the
descending arm gets its own expectation.
* fix(query): stop windowed sort on unit-changing timestamp wrappers
to_timestamp_seconds/millis/... cast their argument to their own unit
inside the function, so the hidden cast bypassed the explicit-cast guard
and kept the windowed-sort optimization eligible on a sort column whose
values no longer match the working ranges. The wrapper branch now applies
the same unit- and value-preserving predicate as the cast branch, with
the predicate extracted into preserves_timestamp_values and shared by
both.
The integration regression gains to_timestamp_seconds ASC/DESC arms and
a to_timestamp_millis value-preserving control; unit tests cover
unit-changing, tz-adding, and non-timestamp wrapper inputs.
* test: regen windowed sort sqlness after wrapper guard
The windowed_sort_nanos and windowed_sort_advance cases sort by
to_timestamp_nanos over a millisecond index and to_timestamp_millis over
a nanosecond index. Both wrappers change the time unit inside the
function, so the windowed-sort rule no longer applies and the plans
render a plain SortExec. Query results are unchanged; only the EXPLAIN
plan line differs. Regenerated by the sqlness runner.
* test(query): simplify windowed sort cast regressions
---------
(cherry picked from commit
|
||
|
|
7a20d38702 |
fix(query): forward scalar subquery filters to region scans [Backport release/v1.3] (#9781)
fix(query): forward scalar subquery filters to region scans (#9738)
* fix(query): forward scalar subquery filters to region scans
Since DataFusion 55, uncorrelated scalar subqueries run in ScalarSubqueryExec
instead of being rewritten into joins, so filters such as
ts = (SELECT ts ... ORDER BY ts DESC LIMIT 1) no longer reach region scans
through a join dynamic filter. MergeScanExec now captures such filters as
remote dynamic filters, keeps them in the parent FilterExec, and replaces the
subquery with its result before querying regions.
* fix(query): check the remote dyn filter budget across pushdown phases
Filters captured in the Pre and Post phases are sent in one request, but each
phase validated the initial registration budget alone. When their sum exceeded
the budget, serialization dropped every registration, including join and TopK
filters. Validate the combined set and keep filters accepted by the other phase.
* fix(query): keep IN-bearing scalar subquery filters at the parent
Oversized remote dyn filter payloads replace IN lists with true. That only
widens a conjunction of bounds, but inside an arbitrary predicate such as
(host IN (...)) = (SELECT flag ...) it can drop rows the parent filter would
keep. Do not forward scalar subquery filters that contain IN lists.
* chore(query): note the IN fallback restriction for scalar subquery filters
* chore(query): document the scalar subquery dyn filter bridge removal condition
* test(query): keep only dyn_filters of region scans in the scalar subquery plan case
#9565 added scan diagnostics to EXPLAIN ANALYZE VERBOSE and partition filter
rendering changed; the case only checks the forwarded dyn filter.
---------
(cherry picked from commit
|
||
|
|
8af3a04ed7 |
fix: preserve structured query errors through distributed execution (#9161)
* fix: preserve structured query errors through distributed execution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update SQL expectations for preserved query error codes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
|
da5cb1a190 |
perf(promql): push down last row for instant queries (#9034)
* perf(promql): push down last row for instant queries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: guard instant last row correctness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update instant query explain results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: apply last row after source deduplication Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: scope post-merge last row selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover instant PromQL last row Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): sort generated SST rows before writing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover instant last row selection in sqlness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): avoid last row hints for lossy timestamp casts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): preserve stale marker semantics across flushes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): decode dictionary labels in stale regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): avoid reserved column name in stale fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): exercise LastRow hints and filtered results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: keep after-merge mode in LastRow selector Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: reject instant LastRow across residual filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: expect after-merge selector in instant vector guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: explain instant LastRow filter eligibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restrict instant LastRow to safe selector nodes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: show LastRow merge mode directly in diagnostics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh LastRow display in explain expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
76924c2d36 |
feat(mito2): introduce two-phase metric series scans (#8826)
* feat(mito2): add two-phase series scan Signed-off-by: evenyag <realevenyag@gmail.com> * docs: regenerate configuration reference Signed-off-by: evenyag <realevenyag@gmail.com> * test(sqlness): update series scan explain results Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config API expectation Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): bound two-phase series discovery Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): avoid candidate distribution deadlock Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): remove obsolete dead code allowances Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): share series scan memory pool Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
764c93bf43 |
perf(query): choose bounded CTE as hash join build side (#8807)
* fix(query): choose bounded CTE as hash join build side Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(query): remove join estimate cap * test(compat): accept repartition in analyze plan Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
0479c05092 |
perf: preserve dictionary-encoded query labels (#8541)
* perf: preserve dictionary-encoded query labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(client): skip dictionary Flight batches Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(servers): decode dictionary labels in HTTP output Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * test(mito2): support dictionary tags in series scans Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(servers): preserve dictionary child nulls in SQL Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(promql): compare dictionary tags by logical nulls Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): keep dictionary tags within query paths Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refactor(query): scope PK dictionary encoding to reads Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): preserve pushdown for dictionary labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): handle dictionary labels in query operators Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): complete label type matching Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): align dictionary query schemas Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): decode incompatible OR labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): preserve dictionary partition pruning Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: handle dictionary query edge cases Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: import dictionary downcast macro Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
b30765e5e9 |
fix: preserve distributed topk merge ordering (#8432)
* fix: preserve distributed topk for scalar latest Signed-off-by: discord9 <discord9@163.com> * fix: preserve scalar latest ordering across merge scan Signed-off-by: discord9 <discord9@163.com> * fix: carry merge scan ordering from planner rewrite Signed-off-by: discord9 <discord9@163.com> * test: cover merge scan ordering metadata Signed-off-by: discord9 <discord9@163.com> * test: cover merge scan partition ordering gate Signed-off-by: discord9 <discord9@163.com> * fix: require adjacent merge sort for ordering metadata Signed-off-by: discord9 <discord9@163.com> * test: update ordering sqlness plans Signed-off-by: discord9 <discord9@163.com> * fix: preserve distributed topk merge ordering Signed-off-by: discord9 <discord9@163.com> * test: cover latest per series queries Signed-off-by: discord9 <discord9@163.com> * test: update distributed merge sort sqlness Signed-off-by: discord9 <discord9@163.com> * test: cover merge scan ordering partition gate Signed-off-by: discord9 <discord9@163.com> * test: cover merge scan over-partition ordering Signed-off-by: discord9 <discord9@163.com> * test: cover distributed latest with low parallelism Signed-off-by: discord9 <discord9@163.com> * fix: keep distributed merge sort opaque Signed-off-by: discord9 <discord9@163.com> * test: assert distributed scalar latest merge sort Signed-off-by: discord9 <discord9@163.com> * test: clarify merge scan ordering helper Signed-off-by: discord9 <discord9@163.com> * fix: complete merge sort exec delegation Signed-off-by: discord9 <discord9@163.com> * docs: explain merge sort limit pushdown Signed-off-by: discord9 <discord9@163.com> * test: cover merge sort optimizer opacity Signed-off-by: discord9 <discord9@163.com> * docs: explain merge sort optimizer hooks Signed-off-by: discord9 <discord9@163.com> * test: cover merge sort optimizer hooks Signed-off-by: discord9 <discord9@163.com> * fix: preserve merge sort child topk Signed-off-by: discord9 <discord9@163.com> * test: update order by topk plan Signed-off-by: discord9 <discord9@163.com> * fix: recognize merge sort global fetch Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
9cf071808a |
fix(query): run optimizer rules before MergeScan (#8339)
* fix(query): push down join filters before MergeScan Signed-off-by: discord9 <discord9@163.com> * fix(query): run optimizer before MergeScan pushdown Signed-off-by: discord9 <discord9@163.com> * fix(query): narrow pre-MergeScan filter pushdown Signed-off-by: discord9 <discord9@163.com> * fix(query): refine pre-MergeScan optimizer prepass Signed-off-by: discord9 <discord9@163.com> * fix(query): satisfy predicate extractor clippy Signed-off-by: discord9 <discord9@163.com> * test(query): cover pre-MergeScan optimizer edges Signed-off-by: discord9 <discord9@163.com> * test(query): cover set comparison prepass Signed-off-by: discord9 <discord9@163.com> * fix(query): guard remote scan filter pushdown Signed-off-by: discord9 <discord9@163.com> * fix(query): preserve subquery planning errors Signed-off-by: discord9 <discord9@163.com> * fix(query): preserve usable scan predicates Signed-off-by: discord9 <discord9@163.com> * fix(query): simplify scan predicate extraction Signed-off-by: discord9 <discord9@163.com> * fix(query): keep scan filter extraction scoped Signed-off-by: discord9 <discord9@163.com> * docs(query): explain pre-MergeScan optimizer Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
9133d0464f |
feat: pre-cast constants (#7926)
* init impl Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * handle no cast Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refactor using common-expr Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * extend matching pattern Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * more tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * simplification Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix zero timestamp Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: normalize sqlness partition count output Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: normalize remaining sqlness plan output Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: normalize sqlness repartition details in tql explain Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: tighten const normalization casts * test: normalize standalone tql explain repartition output * resolve cr comments Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * simplify Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
9fafd879ed |
fix: window sort off by one precision TimeRange&better alias track (#8019)
* fix: window sort track alias&off by one precision TimeRange Signed-off-by: discord9 <discord9@163.com> * chore: more test Signed-off-by: discord9 <discord9@163.com> * refactor: clear helper Signed-off-by: discord9 <discord9@163.com> * dedup a bit Signed-off-by: discord9 <discord9@163.com> * feat: even more guard Signed-off-by: discord9 <discord9@163.com> * fix: case insensitive Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
b623cb1aa2 |
perf: no longer window sort when limit (#7912)
* perf: no longer window sort when limit Signed-off-by: discord9 <discord9@163.com> * test: confusing vector sqlness Signed-off-by: discord9 <discord9@163.com> * chore: redact sqlness Signed-off-by: discord9 <discord9@163.com> * chore: redact every thing Signed-off-by: discord9 <discord9@163.com> * REDACTED Signed-off-by: discord9 <discord9@163.com> * what Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
b2074e3863 |
chore: upgrade DataFusion family, again (#7578)
* chore: upgrade DataFusion family Signed-off-by: luofucong <luofc@foxmail.com> * chore: switch to released version of datafusion-pg-catalog --------- Signed-off-by: luofucong <luofc@foxmail.com> Co-authored-by: Ning Sun <sunning@greptime.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
|
aa3daf7053 |
fix: read filter's column (#7579)
* fix: read columns need for filter Signed-off-by: discord9 <discord9@163.com> * c Signed-off-by: discord9 <discord9@163.com> * feat: add support for explicit read columns in projection mappers Signed-off-by: discord9 <discord9@163.com> * test: add compatibility tests for projection mappers Signed-off-by: discord9 <discord9@163.com> * c Signed-off-by: discord9 <discord9@163.com> * fix: rename variable for clarity and improve column ID retrieval logic Signed-off-by: discord9 <discord9@163.com> * fix: update scan input construction to include read column IDs Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * test: sqlness for projection filter Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * chore: more redacting Signed-off-by: discord9 <discord9@163.com> * chore: more redact Signed-off-by: discord9 <discord9@163.com> * c Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
e64c31e59a |
chore: upgrade DataFusion family (#7558)
* chore: upgrade DataFusion family Signed-off-by: luofucong <luofc@foxmail.com> * use main proto Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
fce1687fa7 |
fix: incorrect timestamp index inference (#7530)
* add sqlness case, but can't reproduce Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * reproduction Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix wildcard rule Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * sort result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
ac6d68aa2d |
fix: simp expr recursively (#7523)
* fix: simp expr recursively Signed-off-by: discord9 <discord9@163.com> * test: some simple constant folding case Signed-off-by: discord9 <discord9@163.com> * fix: literal ts cast to UTC Signed-off-by: discord9 <discord9@163.com> * fix: patch merge scan batch col tz instead Signed-off-by: discord9 <discord9@163.com> * test: fix Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
68247fc9b1 |
fix: count_state use stat to eval&predicate w/out region (#7116)
* fix: count_state use stat to eval Signed-off-by: discord9 <discord9@163.com> * cleanup Signed-off-by: discord9 <discord9@163.com> * fix: use predicate without region Signed-off-by: discord9 <discord9@163.com> * test: diverge standalone/dist impl Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
aca8b690d1 |
fix: step aggr merge phase not order nor filter (#6998)
* fix: not order Signed-off-by: discord9 <discord9@163.com> * test: redacted Signed-off-by: discord9 <discord9@163.com> * feat: fix up state wrapper Signed-off-by: discord9 <discord9@163.com> * df last_value state not as promised! Signed-off-by: discord9 <discord9@163.com> * fix?: could fix better Signed-off-by: discord9 <discord9@163.com> * test: unstable result Signed-off-by: discord9 <discord9@163.com> * fix: work around by fixing state Signed-off-by: discord9 <discord9@163.com> * chore: after rebase fix Signed-off-by: discord9 <discord9@163.com> * chore: finish some todo Signed-off-by: discord9 <discord9@163.com> * chore: per copilot Signed-off-by: discord9 <discord9@163.com> * refactor: not fix but just notify mismatch Signed-off-by: discord9 <discord9@163.com> * chore: warn -> debug state mismatch Signed-off-by: discord9 <discord9@163.com> * chore: refine error msg Signed-off-by: discord9 <discord9@163.com> * test: sqlness add last_value date_bin test Signed-off-by: discord9 <discord9@163.com> * ?: substrait order by decode failure Signed-off-by: discord9 <discord9@163.com> * unit test reproduce that Signed-off-by: discord9 <discord9@163.com> * feat: support state wrapper's order serde in substrait Signed-off-by: discord9 <discord9@163.com> * refactor: stuff Signed-off-by: discord9 <discord9@163.com> * test: standalone/distributed different exec Signed-off-by: discord9 <discord9@163.com> * fmt Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * refactor: closure Signed-off-by: discord9 <discord9@163.com> * test: first value order by Signed-off-by: discord9 <discord9@163.com> * refactor: per cr Signed-off-by: discord9 <discord9@163.com> * feat: ScanHint last_value last row selector Signed-off-by: discord9 <discord9@163.com> * docs: per cr Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
288f69a30f |
fix: plan disorder from upgrading datafusion (#6787)
* fix: plan disorder from upgrading datafusion Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
f9d2a89a0c |
chore: update datafusion family (#6675)
* chore: update datafusion family Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> * use official otel-arrow-rust Signed-off-by: luofucong <luofc@foxmail.com> * rebase Signed-off-by: luofucong <luofc@foxmail.com> * use the official orc-rust Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * remove the empty lines Signed-off-by: luofucong <luofc@foxmail.com> * try following PR comments Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
6501b0b13c |
feat: MergeScan print input (#6563)
* feat: MergeScan print input Signed-off-by: discord9 <discord9@163.com> * test: fix ut Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
af03e89139 |
fix: stricter win sort condition (#6477)
test: sqlness test: fix sqlness redacted Signed-off-by: discord9 <discord9@163.com> |
||
|
|
7468a8ab2a |
feat: organize EXPLAIN ANALYZE VERBOSE's output in JSON format (#6308)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
4b71e493f7 |
feat!: revise compaction picker (#6121)
* - **Refactor `RegionFilePathFactory` to `RegionFilePathProvider`:** Updated references and implementations in `access_layer.rs`, `write_cache.rs`, and related test files to use the new struct name. - **Add `max_file_size` support in compaction:** Introduced `max_file_size` option in `PickerOutput`, `SerializedPickerOutput`, and `WriteOptions` in `compactor.rs`, `picker.rs`, `twcs.rs`, and `window.rs`. - **Enhance Parquet writing logic:** Modified `parquet.rs` and `parquet/writer.rs` to support optional `max_file_size` and added a test case `test_write_multiple_files` to verify writing multiple files based on size constraints. **Refactor Parquet Writer Initialization and File Handling** - Updated `ParquetWriter` in `writer.rs` to handle `current_indexer` as an `Option`, allowing for more flexible initialization and management. - Introduced `finish_current_file` method to encapsulate logic for completing and transitioning between SST files, improving code clarity and maintainability. - Enhanced error handling and logging with `debug` statements for better traceability during file operations. - **Removed Output Size Enforcement in `twcs.rs`:** - Deleted the `enforce_max_output_size` function and related logic to simplify compaction input handling. - **Added Max File Size Option in `parquet.rs`:** - Introduced `max_file_size` in `WriteOptions` to control the maximum size of output files. - **Refactored Indexer Management in `parquet/writer.rs`:** - Changed `current_indexer` from an `Option` to a direct `Indexer` type. - Implemented `roll_to_next_file` to handle file transitions when exceeding `max_file_size`. - Simplified indexer initialization and management logic. - **Refactored SST File Handling**: - Introduced `FilePathProvider` trait and its implementations (`WriteCachePathProvider`, `RegionFilePathFactory`) to manage SST and index file paths. - Updated `AccessLayer`, `WriteCache`, and `ParquetWriter` to use `FilePathProvider` for path management. - Modified `SstWriteRequest` and `SstUploadRequest` to use path providers instead of direct paths. - Files affected: `access_layer.rs`, `write_cache.rs`, `parquet.rs`, `writer.rs`. - **Enhanced Indexer Management**: - Replaced `IndexerBuilder` with `IndexerBuilderImpl` and made it async to support dynamic indexer creation. - Updated `ParquetWriter` to handle multiple indexers and file IDs. - Files affected: `index.rs`, `parquet.rs`, `writer.rs`. - **Removed Redundant File ID Handling**: - Removed `file_id` from `SstWriteRequest` and `CompactionOutput`. - Updated related logic to dynamically generate file IDs where necessary. - Files affected: `compaction.rs`, `flush.rs`, `picker.rs`, `twcs.rs`, `window.rs`. - **Test Adjustments**: - Updated tests to align with new path and indexer management. - Introduced `FixedPathProvider` and `NoopIndexBuilder` for testing purposes. - Files affected: `sst_util.rs`, `version_util.rs`, `parquet.rs`. * chore: rebase main * feat/multiple-compaction-output: ### Add Benchmarking and Refactor Compaction Logic - **Benchmarking**: Added a new benchmark `run_bench` in `Cargo.toml` and implemented benchmarks in `benches/run_bench.rs` using Criterion for `find_sorted_runs` and `reduce_runs` functions. - **Compaction Module Enhancements**: - Made `run.rs` public and refactored the `Ranged` and `Item` traits to be public. - Simplified the logic in `find_sorted_runs` and `reduce_runs` by removing `MergeItems` and related functions. - Introduced `find_overlapping_items` for identifying overlapping items. - **Code Cleanup**: Removed redundant code and tests related to `MergeItems` in `run.rs`. * feat/multiple-compaction-output: ### Enhance Compaction Logic and Add Benchmarks - **Compaction Logic Improvements**: - Updated `reduce_runs` function in `src/mito2/src/compaction/run.rs` to remove the target parameter and improve the logic for selecting files to merge based on minimum penalty. - Enhanced `find_overlapping_items` to handle unsorted inputs and improve overlap detection efficiency. - **Benchmark Enhancements**: - Added `bench_find_overlapping_items` in `src/mito2/benches/run_bench.rs` to benchmark the new `find_overlapping_items` function. - Extended existing benchmarks to include larger data sizes. - **Testing Enhancements**: - Updated tests in `src/mito2/src/compaction/run.rs` to reflect changes in `reduce_runs` and added new tests for `find_overlapping_items`. - **Logging and Debugging**: - Improved logging in `src/mito2/src/compaction/twcs.rs` to provide more detailed information about compaction decisions. * feat/multiple-compaction-output: ### Refactor and Enhance Compaction Logic - **Refactor `find_overlapping_items` Function**: Changed the function signature to accept slices instead of mutable vectors in `run.rs`. - **Rename and Update Struct Fields**: Renamed `penalty` to `size` in `SortedRun` struct and updated related logic in `run.rs`. - **Enhance `reduce_runs` Function**: Improved logic to sort runs by size and limit probe runs to 100 in `run.rs`. - **Add `merge_seq_files` Function**: Introduced a new function `merge_seq_files` in `run.rs` for merging sequential files. - **Modify `TwcsPicker` Logic**: Updated the compaction logic to use `merge_seq_files` when only one run is found in `twcs.rs`. - **Remove `enforce_file_num` Function**: Deleted the `enforce_file_num` function and its related test cases in `twcs.rs`. * feat/multiple-compaction-output: ### Enhance Compaction Logic and Testing - **Add `merge_seq_files` Functionality**: Implemented the `merge_seq_files` function in `run.rs` to optimize file merging based on scoring systems. Updated benchmarks in `run_bench.rs` to include `bench_merge_seq_files`. - **Improve Compaction Strategy in `twcs.rs`**: Modified the compaction logic to handle file merging more effectively, considering file size and overlap. - **Update Tests**: Enhanced test coverage in `compaction_test.rs` and `append_mode_test.rs` to validate new compaction logic and file merging strategies. - **Remove Unused Function**: Deleted `new_file_handles` from `test_util.rs` as it was no longer needed. * feat/multiple-compaction-output: ### Refactor TWCS Compaction Options - **Refactor Compaction Logic**: Simplified the TWCS compaction logic by replacing multiple parameters (`max_active_window_runs`, `max_active_window_files`, `max_inactive_window_runs`, `max_inactive_window_files`) with a single `trigger_file_num` parameter in `picker.rs`, `twcs.rs`, and `options.rs`. - **Update Tests**: Adjusted test cases to reflect the new compaction logic in `append_mode_test.rs`, `compaction_test.rs`, `filter_deleted_test.rs`, `merge_mode_test.rs`, and various test files under `tests/cases`. - **Modify Engine Options**: Updated engine option keys to use `trigger_file_num` in `mito_engine_options.rs` and `region_request.rs`. - **Fuzz Testing**: Updated fuzz test generators and translators to accommodate the new compaction parameter in `alter_expr.rs` and related files. This refactor aims to streamline the compaction configuration by reducing the number of parameters and simplifying the codebase. * chore: add trailing space * fix license header * feat/revise-compaction-picker: **Limit File Processing and Optimize Merge Logic in `run.rs`** - Introduced a limit to process a maximum of 100 files in `merge_seq_files` to control time complexity. - Adjusted logic to calculate `target_size` and iterate over files using the limited set of files. - Updated scoring calculations to use the limited file set, ensuring efficient file merging. * feat/revise-compaction-picker: ### Add Compaction Metrics and Remove Debug Logging - **Compaction Metrics**: Introduced new histograms `COMPACTION_INPUT_BYTES` and `COMPACTION_OUTPUT_BYTES` to track compaction input and output file sizes in `metrics.rs`. Updated `compactor.rs` to observe these metrics during the compaction process. - **Logging Cleanup**: Removed debug logging of file ranges during the merge process in `twcs.rs`. * feat/revise-compaction-picker: ## Enhance Compaction Logic and Metrics - **Compaction Logic Improvements**: - Added methods `input_file_size` and `output_file_size` to `MergeOutput` in `compactor.rs` to streamline file size calculations. - Updated `Compactor` implementation to use these methods for metrics tracking. - Modified `Ranged` trait logic in `run.rs` to improve range comparison. - Enhanced test cases in `run.rs` to reflect changes in compaction logic. - **Metrics Enhancements**: - Changed `COMPACTION_INPUT_BYTES` and `COMPACTION_OUTPUT_BYTES` from histograms to counters in `metrics.rs` for better performance tracking. - **Debugging and Logging**: - Added detailed logging for compaction pick results in `twcs.rs`. - Implemented custom `Debug` trait for `FileMeta` in `file.rs` to improve debugging output. - **Testing Enhancements**: - Added new test `test_compaction_overlapping_files` in `compaction_test.rs` to verify compaction behavior with overlapping files. - Updated `merge_mode_test.rs` to reflect changes in file handling during scans. * feat/revise-compaction-picker: ### Update `FileHandle` Debug Implementation - **Refactor Debug Output**: Simplified the `fmt::Debug` implementation for `FileHandle` in `src/mito2/src/sst/file.rs` by consolidating multiple fields into a single `meta` field using `meta_ref()`. - **Atomic Operations**: Updated the `deleted` field to use atomic loading with `Ordering::Relaxed`. * Trigger CI * feat/revise-compaction-picker: **Update compaction logic and default options** - **`twcs.rs`**: Enhanced logging for compaction pick results by improving the formatting for better readability. - **`options.rs`**: Modified the default `max_output_file_size` in `TwcsOptions` from 2GB to 512MB to optimize file handling and performance. * feat/revise-compaction-picker: Refactor `find_overlapping_items` to use an external result vector - Updated `find_overlapping_items` in `src/mito2/src/compaction/run.rs` to accept a mutable result vector instead of returning a new vector, improving memory efficiency. - Modified benchmarks in `src/mito2/benches/bench_compaction_picker.rs` to accommodate the new function signature. - Adjusted tests in `src/mito2/src/compaction/run.rs` to use the updated function signature, ensuring correct functionality with the new approach. * feat/revise-compaction-picker: Improve file merging logic in `run.rs` - Refactor the loop logic in `merge_seq_files` to simplify the iteration over file groups. - Adjust the range for `end_idx` to include the endpoint, allowing for more flexible group selection. - Remove the condition that skips groups with only one file, enabling more comprehensive processing of file sequences. * feat/revise-compaction-picker: Enhance `find_overlapping_items` with `SortedRun` and Update Tests - Refactor `find_overlapping_items` in `src/mito2/src/compaction/run.rs` to utilize the `SortedRun` struct for improved efficiency and clarity. - Introduce a `sorted` flag in `SortedRun` to optimize sorting operations. - Update test cases in `src/mito2/benches/bench_compaction_picker.rs` to accommodate changes in `find_overlapping_items` by using `SortedRun`. - Add `From<Vec<T>>` implementation for `SortedRun` to facilitate easy conversion from vectors. * feat/revise-compaction-picker: **Enhancements in `compaction/run.rs`:** - Added `ReadableSize` import to handle size calculations. - Modified the logic in `merge_seq_files` to clamp the calculated target size to a maximum of 2GB when `max_file_size` is not provided. * feat/revise-compaction-picker: Add Default Max Output Size Constant for Compaction Introduce DEFAULT_MAX_OUTPUT_SIZE constant to define the default maximum compaction output file size as 2GB. Refactor the merge_seq_files function to utilize this constant, ensuring consistent and maintainable code for handling file size limits during compaction. |
||
|
|
bbb6f8685e |
feat: implement commutativity rule for prom-related plans (#5875)
* feat: implement commutativity rule for prom-related plans Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix range manipulate deserializer Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * blocklist in commutativity rule Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * change dictionary type Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * handle partition and ordering Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * add rate, increase and delta Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * regexp_replace uses empty string instead of null value Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
32fd850c20 |
perf: support in list in simple filter (#5709)
* feat: support in list in simple filter Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
0124a0d156 |
fix: window sort not apply when other column alias to time index name (#5634)
* fix: other col alias to time index column handle * test: update sqlness * chore: per review * test: more sqlness * test: mv some to optimizer folder * fix: resolve alias properly * fix: also retain old name * chore: remove wrong comment * chore: fix sqlness * test: standalone/dist more projection diff |
||
|
|
77223a0f3e |
fix: window sort support alias time index (#5543)
* fix: use alias expr to check commutativity * chore: debug sort * feat: consider alias in window sort optimizer * test: sqlness test * test: update sqlness result |
||
|
|
35b635f639 |
feat!: Bump datafusion, prost, hyper, tonic, tower, axum (#5417)
* change dep Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * feat: adapt to arrow's interval array * chore: fix compile errors in datatypes crate * chore: fix api crate compiler errors * chore: fix compiler errors in common-grpc * chore: fix common-datasource errors * chore: fix deprecated code in common-datasource * fix promql and physical plan related Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * wip: upgrading network deps Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * block on updating `sqlparser` * upgrade sqlparser Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * adapt new df's trait requirements Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix compiler errors in mito2 * chore: fix common-function crate errors * chore: fix catalog errors * change import path Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix some errors in query crate * chore: fix some errors in query crate * aggr expr and some other tiny fixes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix expr related errors in query crate * chore: fix query serializer and admin command * chore: fix grpc services * feat: axum serve * chore: fix http server * remove handle_error handler * refactor timeout layer * serve axum * chore: fix flow aggr functions * chore: fix flow * feat: fix errors in meta-srv * boxed() * use TokioIo * feat!: Remove script crate and python feature (#5321) * feat: exclude script crate * chore: simplify feature * feat: remove the script crate * chore: remove python feature and some comments * chore: fix warning * chore: fix servers tests compiler errors * feat: fix tests-integration errors * chore: fix unused * test: fix catalog test * chore: fix compiler errors for crates using common-meta testing feature is enabled when check with --workspace * test: use display for logical plan test * test: implement rewrite for ScanHintRule * fix: http server build panic * test: fix mito test * fix: sql parser type alias error * test: fix TestClient not listen * test: some flow tests * test(flow): more fix * fix: test_otlp_logs * test: fix promql test that using deprecated method fun() * fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64 * test: fix infer schema test case * test: fix tests related to plan display * chore: fix last flow test * test: fix function format related assertion * test: use larger port range for tests * fix: test_otlp_traces * fix: test_otlp_metrics * fix range query and dist plan Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: flow handle distinct use deprecated field * fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs * test: fix deserialize test * test: reduce split key case num * tests: lower case aggr func name * test: fix some sqlness tests * tests: more sqlness fix * tests: fixed sqlness test * commit non-bug changes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: make our udf correct * fix: implement empty methods of ContextProvider for DfContextProviderAdapter * test: update sqlness test result * chore: remove unused * fix: provide alias name for AggregateExprBuilder in range plan * test: update range query result * fix: implement missing ContextProvider methods for DfContextProviderAdapter * test: update timestamps, cte result * fix: supports empty projection in mito * test: update comment for cte test * fix: support projection for numbers * test: update test cases after projection fix * fix: fix range select first_value/last_value * fix: handle CAST and time index conflict * fix: handle order by correctly in range first_value/last_value * test: update sqlness result * test: update view test result * test: update decimal test wait for https://github.com/apache/datafusion/pull/14126 to fix this * feat: remove redundant physical optimization todo(ruihang): Check if we can remove this. * test: update sqlness test result * chore: range select default sort use nulls_first = false * test: update filter push down test result * test: comment deciaml test to avoid different panic message * test: update some distributed test result * test: update test for distributed count and filter push down * test: update subqueries test * fix: SessionState may overwrite our UDFs * chore: fix compiler errors after merging main * fix: fix elasticsearch and dashboard router panic * chore: fix common-functions tests * chore: update sqlness result * test: fix id keyword and update sqlness result * test: fix flow_null test * fix: enlarge thread size in debug mode to avoid overflow * chore: fix warnings in common-function * chore: fix warning in flow * chore: fix warnings in query crate * chore: remove unused warnings * chore: fix deprecated warnings for parquet * chore: fix deprecated warning in servers crate * style: fix clippy * test: enlarge mito cache tttl test ttl time * chore: fix typo * style: fmt toml * refactor: reimplement PartialOrd for RangeSelect * chore: remove script crate files introduced by merge * fix: return error if sql option is not kv * chore: do not use ..default::default() * chore: per review * chore: update error message in BuildAdminFunctionArgsSnafu Co-authored-by: jeremyhi <jiachun_feng@proton.me> * refactor: typed precision * update sqlness view case Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: flow per review * chore: add example in comment * chore: warn if parquet stats of timestamp is not INT64 * style: add a newline before derive to make the comment more clear * test: update sqlness result * fix: flow from substrait * chore: change update_range_context log to debug level * chore: move axum-extra axum-macros to workspace --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: luofucong <luofc@foxmail.com> Co-authored-by: discord9 <discord9@163.com> Co-authored-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me> |
||
|
|
3e17c09e45 |
feat: skip caching uncompressed pages if they are large (#4705)
* feat: cache each uncompressed page * chore: remove unused function * chore: log * chore: log * chore: row group pages cache kv * feat: also support row group level cache * chore: fix range count * feat: don't cache compressed page for row group cache * feat: use function to get part * chore: log whether scan is from compaction * chore: avoid get column * feat: add timer metrics * chore: Revert "feat: add timer metrics" This reverts commit 4618f57fa2ba13b1e1a8dec83afd01c00ae4c867. * feat: don't cache individual uncompressed page * feat: append in row group level under append mode Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fetch pages cost * perf: yield * Update src/mito2/src/sst/parquet/row_group.rs * refactor: cache key * feat: print file num and row groups num in explain * test: update sqlness test * chore: Update src/mito2/src/sst/parquet/page_reader.rs --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
216bce6973 |
perf: count(*) for append-only tables (#4545)
* feat: support fast count(*) for append-only tables * fix: total_rows stats in time series memtable * fix: sqlness result changes for SinglePartitionScanner -> StreamScanAdapter * fix: some cr comments |
||
|
|
f382a7695f |
perf: reduce lock scope and improve log (#4453)
* feat: refine logs for scan * feat: improve build parts and unordered scan metrics * feat: change to debug log * fix: release lock before reading part * test: replace region id * test: fix sqlness * chore: add todo Co-authored-by: dennis zhuang <killme2008@gmail.com> --------- Co-authored-by: dennis zhuang <killme2008@gmail.com> |
||
|
|
c595a56ac8 |
test(flow): ignore flow tests for now (#4377)
* tests: ignore flow tests for now * chore: typo * test: remove file altogether * fix: last_value sqlness test regex * fix: last_value part REDACTED |
||
|
|
7900367433 |
test: replace unstable output of last value test (#4371)
test: replace unstable output |
||
|
|
20d9c0a345 |
fix: scan hint checks order asc (#4365)
* fix: order by asc check * feat: print selector in explain * test: move last_value opt test to standalone * test: sqlness remove space * test: update regex for datetime * test: fix partitioning * chore: update comment Co-authored-by: Ruihang Xia <waynestxia@gmail.com> --------- Co-authored-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
c2218f8be8 |
build(deps): bump datafusion 20240528 (#4061)
* build(deps): bump datafusion 20240528 Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * another update Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update expected sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix first/last value Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * reformat comment Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix remaining errors Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * revert toml format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix pyo3 feature Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove dead code Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * Apply suggestions from code review Co-authored-by: Jeremyhi <jiachun_feng@proton.me> * format file Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Jeremyhi <jiachun_feng@proton.me> |
||
|
|
179c8c716c |
feat: Adds RegionScanner trait (#3948)
* feat: define region scanner * feat: single partition scanner * feat: use single partition scanner * feat: implement ExecutionPlan wip * feat: mito engine returns single partition scanner * feat: implement DisplayAs for region server * feat: dummy table provider use handle_partitioned_query() * test: update sqlness test * feat: table provider use ReadFromRegion * refactor: remove StreamScanAdapter * chore: update lock * style: fix clippy * refactor: remove handle_query from the RegionEngine trait * chore: address CR comments * refactor: rename methods * refactor: rename ReadFromRegion to RegionScanExec |
||
|
|
aec5cca2c7 |
feat: support distributed EXPLAIN ANALYZE (#3908)
* feat: fetch and pass per-plan metrics Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * impl DistAnalyzeExec Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness results Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix typo Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix typo again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * Update src/query/src/analyze.rs Co-authored-by: Jeremyhi <jiachun_feng@proton.me> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Jeremyhi <jiachun_feng@proton.me> |
||
|
|
314f2704d4 |
build(deps): update datafusion to latest and arrow to 51.0 (#3661)
* chore: update datafusion * update sqlness case of time.sql Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: adjust range query partition * fix: hisogram incorrect result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: ignore filter pushdown temporarily Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: update limit sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: histogram with wrong distribution Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: update negative ordinal sqlness case Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * feat: bump df to cd7a00b Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * resolve conflicts * ignore test_range_filter Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix promql exec panic * fix "select count(*)" exec error * re-enable the "test_range_filter" test since the filter push down seems not necessary to be removed * fix: range query schema error * update sqlness results Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * resolve conflicts * update datafusion, again * fix pyo3 compile error, and update some sqlness results * update decimal sqlness cases Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: promql literal * fix udaf tests * fix filter pushdown sqlness tests * fix?: test_cast * fix: rspy test fail due to datafusion `sin` signature change * rebase main to see if there are any failed tests * debug ci * debug ci * debug ci * enforce input partition Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * debug ci * fix ci * fix ci * debug ci * debug ci * debug ci * fix sqlness * feat: do not return error while creating a filter * chore: remove array from error * chore: replace todo with unimplemented * Update src/flow/clippy.toml Co-authored-by: Yingwen <realevenyag@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: WUJingdi <taylor-lagrange@qq.com> Co-authored-by: discord9 <discord9@163.com> Co-authored-by: evenyag <realevenyag@gmail.com> Co-authored-by: tison <wander4096@gmail.com> |
||
|
|
56fc77e573 |
fix: add missing error display message (#2791)
* fix: add missing error display message Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
5d8b0e8154 |
chore: bump dependencies (#2659)
* fix all hard error Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix nextest Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * trivial changes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix order by Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix sql keyword and data type Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix range exec's input partitioning Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix cover input type Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix explain analyze Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update distributed mode sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix lints Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update locks Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * downgrade dlv-list Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
10ecc30817 |
feat: pushdown aggr, limit and sort plan (#2495)
* check partition for aggr plan Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * handle empty partition rule Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove CheckPartition option Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * add test Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update some valid sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * opt-out promql plan and update sqlness Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix limit Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix insert select subquery Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update unit test result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix typo Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * Update src/query/src/dist_plan/analyzer.rs Co-authored-by: Yingwen <realevenyag@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> |
||
|
|
52ac093110 | fix: drop table 0 rows affected (#2515) | ||
|
|
802229de87 |
fix: type cast bugs found by sqlness (#2438)
* update valid results Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * accomplish datatype Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * cast null Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix unit tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
47bf300869 |
fix: update sqlness result for order_by (#2389)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
db89235474 |
feat: only allow timestamp type as time index (#2281)
* feat: only allow timestamp data type as time index * test: update sqltest cases, todo: need some fixes * fix: sqlness tests * fix: forgot adding back cte test * chore: style |
||
|
|
44f3ed2f74 |
chore(deps): bump datafusion to the latest commit (#1967)
* bump deps Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix compile except pyo3 backend Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix promql-parser metric name matcher Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix pyarrow convert Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix pyo3 compiling Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove deadcode Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update stream adapter display format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix physical optimizer rule Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |