Commit Graph
144 Commits
Author SHA1 Message Date
discord9 8af3a04ed7 fix: preserve structured query errors through distributed execution (#9161)
* fix: preserve structured query errors through distributed execution

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update SQL expectations for preserved query error codes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-16 06:28:15 +00:00
discord9andNing Sun 94d7e2c7fc feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: migrate DataFusion 55 APIs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve table function planning behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve PostgreSQL query compatibility

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve distributed execution plan behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover DataFusion 55 behavior regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update DataFusion 55 SQLness expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DataFusion 55 test API migration

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address DataFusion 55 CI regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address remaining DataFusion 55 regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: adapt latest base code to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: normalize environment-specific DataFusion 55 plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align final DataFusion 55 expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: isolate DataFusion 55 regression cases

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve empty result schema in timestamp widening

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve JSON source column order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: use released DataFusion 55 integrations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: adapt latest execution plan mock to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: pin DataFusion recursive schema and date repairs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): align dictionary temporality match keys

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: retain Greptime DataFusion fork behaviors on version 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore ordinary function error expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh distributed count compatibility plan

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt last-row cast hint to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh instant last-row empty results for Arrow 59

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: simplify DataFusion expression visitor imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore sorting and PostgreSQL column-order assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): restore primitive numeric coercion signatures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(function): share geo integer signature types

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover timestamp widening overflow boundaries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: fix decimal coercion regression imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): preserve scalar count_hash NULL state semantics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: simplify decimal clamp case type inference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: retain historical count_hash wrapper result

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: restore timestamp widening equality and IN pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream aggregate dynamic filter correctness fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream null and predicate simplification fixes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore baseline JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore histogram JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh empty PromQL range result schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align native timestamp plan with DF55 decimal display

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh native timestamp SQLness results for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regenerate NULL sample empty result headers for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: use DF55 child replacement API in timestamp regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: expose pushed scan dynamic filters to DF55 producers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: encode string-backed PostgreSQL OID aliases in binary results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: verify REGPROC binary and text over PostgreSQL protocol

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: register real PostgreSQL catalogs in server fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DF55 expression inventories for custom query plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: correct RangeSelect expression fixture and column identities

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* ci: wait for Kafka WAL helper deployment rollout

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update custom storage empty result headers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: require exact row counts in scan statistics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: suppress deprecated partition_statistics warning in test

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-09-15 11:42:38 +00:00
discord9 9c135ebcb3 feat!: stabilize streaming analyze metrics (#8966)
* feat: stabilize streaming analyze metrics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: expose analyze memory usage

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: simplify analyze stream handling

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve analyze stream sequence on panic

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: log analyze stream worker panic

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-02 10:28:36 +00:00
Ning Sun 6f1dd0cb76 feat: allow widening the time index column's timestamp unit via ALTER TABLE, mito2 table only (#8894)
* feat: allow widening the time index column's timestamp unit via ALTER TABLE ... MODIFY COLUMN

Previously MODIFY COLUMN rejected the time index column outright. Now the
time index unit can be widened (Second -> Milli -> Micro -> Nano), which is
lossless for data that fits the target unit: historical data in old SSTs is
cast to the new unit on read by the existing schema-compat layer, and
compaction rewrites it lazily. Narrowing and non-timestamp targets remain
rejected; tag columns keep being rejected. Widening is rejected if any
SST's time range would overflow the target unit's i64 range (e.g.
millisecond -> nanosecond beyond year 2262), since the cast would silently
null those values.

Read-path correctness for old-unit SSTs (verified by new engine e2e tests
and sqlness WHERE queries):

- row-group min/max pruning: parquet statistics of a timestamp column are
  raw integers in the file's unit; when the region metadata's type differs
  (also the case for altered field columns), stats are now interpreted in
  the file's type and converted to the expected type before pruning.
  Without this, a new-unit predicate silently pruned whole row groups of
  old-unit files (wrong results, rows missing).
- SST-level simple filters are skipped for columns whose file type differs
  from the expected type; the predicate is applied by the query layer's
  residual filter above the region scan. Also drop the stale
  "timestamp columns cannot change type" debug_assert.
- retry idempotency: a same-type ModifyColumnType on the time index
  validates as a no-op and `need_alter` returns false, so a retried alter
  procedure (region already altered before the previous attempt failed)
  converges instead of aborting forever.

- add TimeUnit ordering and ConcreteDataType::is_timestamp_unit_widening_to
- relax ModifyColumnType validation in store-api and table metadata
- tests: unit tests in datatypes/store-api/table; mito2 engine e2e tests
  (flushed SST + new writes + reopen + retry + overflow + predicate scans,
  cross-unit dedup, mixed-unit SSTs, compaction); sqlness cases incl.
  partitioned table and WHERE filters over old-unit data

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: resolve parquet filter issue

* test: provide sqlness tests

* refactor: drop trivial test cases and shorten comments

Review pass over the branch's additions:

- datatypes: keep a representative subset of the widening-matrix asserts
- table: collapse the three single-branch rejection blocks into one loop
- mito2: drop the boundary gt_eq and post-compaction predicate asserts
  (covered by the exact-filter regression test and sqlness); drop the
  engine-level gt_eq/lt_eq casts (full operator matrix stays in the
  cast_timestamp_unit unit tests)
- sqlness: drop a bare full scan already covered by the filter above it
- shorten function doc comments across datatypes/store-api/table/mito2/
  recordbatch to the essential semantics

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: drop physical prefilter for columns whose file type differs

Follow-up to the review feedback on CompatBatch/prune reader/filter
handling for widened time index units.

Between/InList/IsNull predicates are prefiltered by PhysicalFilterContext,
which builds its physical expression against the FILE's schema while the
predicate literals are in the expected (post-alter) unit. Evaluating them
against an old-unit SST raised a cross-unit comparison error (Timestamp(ms)
>= Timestamp(µs)) that failed the whole scan. Physical prefilter predicates
are best-effort pruning hints (the query layer re-applies them above the
scan), so drop the prefilter when the column's file type differs from the
expected type, mirroring the simple-filter strategy.

Verified: Between and a non-rewritten (large) InList on old-unit data no
longer error and filter exactly end-to-end (sqlness), and no matching rows
are lost at the engine level (engine test).

Signed-off-by: Ning Sun <sunning@greptime.com>

* test: add direct unit tests for stats cast and prefilter drop

The two-step stats cast (reinterpret raw Int64 stats in the file's
timestamp type, then rescale to the expected type) and the physical
prefilter drop on file/expected type mismatch were only covered
end-to-end; add localized unit tests so a regression fails at the
exact site:

- stats.rs (previously no tests): RowGroupPruningStats min/max over a
  hand-built RowGroupMetaData — passthrough with no expected metadata,
  passthrough on same type, and rescale (1000ms -> 1_000_000us, not
  1000us) on a widened expected unit
- reader.rs: PhysicalFilterContext::new_opt keeps a Between prefilter
  when file and expected types match and drops it on unit mismatch

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: tolerate mixed time units range cache key coverage check

* docs: flag mixed-unit hazard in the (unwired) series index

The series index stores per-series min/max ts as raw Int64 in the unit
of the region metadata at write time, and the searcher builds its range
predicates from a single per-region metadata. After a time index unit
widen, files of one region would carry mixed units, so a per-file unit
(or an index rebuild on such alters) is required before this index is
wired into scans. Leave notes at both sites.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test: cover mixed-unit compaction for sparse encoding and strict windows

Compaction-path audit follow-up. The compat cast and window math were
already covered for dense regions; add the two remaining e2e scenarios:

- sparse primary key encoding (used by metric-engine physical regions):
  widening then compacting mixed-unit files rewrites the old-unit time
  index correctly through the sparse compaction compat path
- strict-window manual compaction: each window output trims rows with a
  predicate built in the region's new unit against an old-unit file;
  every instant must survive exactly once (no loss, no cross-window
  duplication), rescaled

Also documents the audit finding that Regular ranged (manual)
compaction never trims rows: TwcsPicker sets output_time_range to None
and the request time range only selects candidate windows.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test: cover time index unit change in FlatCompatBatch directly

The compat layer's rescaling of a widened time index was only verified
end-to-end; add direct unit tests for both paths:

- dense: identical units skip compat entirely; a widened unit rescales
  the time index column (1000ms -> 1_000_000us, not reinterpreted) while
  other columns pass through and the output schema matches the expected
  metadata
- compact sparse (the metric-engine compaction path): same rescaling

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor: move timestamp unit division into common-time

The exact unit division (UnitQuotient + div_mod_units) is time
semantics, not filter logic; move it next to TimeUnit in common-time
with a compact test covering representable/non-representable values,
negative (floor) instants, and quotient overflow. The ScalarValue
helpers stay in filter.rs since common-time has no datafusion
dependency.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test: compat rescales a widened time index and fills an added column together

The realistic multi-alter sequence (widen at T1, add column at T2,
read a T0 SST) exercises cast and default-fill in the same
compute_index_and_fields pass; assert both in one output batch.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: preflight time index widening overflow before any region alters

Address review feedback on the overflow guard:

- preflight: when the frontend operator receives a widening alter on the
  time index, run an existence scan (ts outside the target unit's i64
  range, LIMIT 1, via the query engine so it covers every region of the
  table in both standalone and distributed modes) BEFORE any DDL task is
  submitted. A region that fits can no longer commit the new schema
  while another region rejects the alter with a non-retryable error.
  File and row-group pruning keep the scan cheap when nothing overflows.
  The per-region check in mito2 stays as the final guard for data
  written after the preflight (the remaining race window); without a
  validate-only wire field (region.proto lives in the external
  greptime-proto repo) a fully atomic two-phase validate/commit is out
  of scope here.

- fast path: cast_timestamp_unit returns the filter unchanged when the
  literal is already in the target unit, skipping the div-mod rebuild.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments

* fix: address auto review comments

* fix: remove time index widening overflow preflight

Overflow needs timestamps beyond the target unit's i64 range (~year
2262 for nanoseconds), which real workloads never write, so the two
existence scans before every widening alter are not worth the cost.
Region validation already rejects the alter when an SST's time range
overflows the target unit; it now logs the rejection (with the
offending file) and returns a deterministic client-facing message.

Signed-off-by: Ning Sun <sunning@greptime.com>

* test: make sqlness test stable

* fix: log instead of rejecting time index widening overflow

Overflowing values cast to NULL on read but do not otherwise affect
reads or writes, so the alter is allowed; the region-level check now
only logs (with the offending file) when an SST's time range exceeds
the target unit's i64 range.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-02 04:29:32 +00:00
Ning Sun d32cd77505 fix: postgres describe for more statements (#8974)
* fix: postgres describe for more statements

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: cover more show statements

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments

- add missing `clippy::too_many_arguments` allow on
  `query_from_information_schema_dataframe` (CI clippy failure)
- take `&ShowKind` in the information-schema dataframe helper so `kind`
  is no longer cloned at every call site; only the WHERE arm (which needs
  an owned expression for `sql_to_expr`) clones internally
- document why re-applying TQL explain formats never overwrites an
  existing value (per-query context state)

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: trim comments to essentials

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-08-30 13:24:17 +00:00
Ning Sun b31f05eb59 fix: update tokio-postgres and correct explain/fetch cursor output schema (#8955)
* chore(deps): update tokio-postgres

* fix: describing fetch cursor and analyze
2026-08-27 03:37:59 +00:00
LFC 95d9d92e42 refactor: separate a json2 extension type (#8745)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-08-05 12:15:05 +00:00
LFC 4e04f92591 fix: make select whole json2 column worked (#8683)
* fix: make select whole json2 column worked

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* x

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-07-31 13:14:33 +00:00
discord9 9776d88dc8 fix: preserve dictionary regex filter semantics (#8688)
* fix: preserve dictionary regex filter semantics

Signed-off-by: discord9 <discord9@163.com>

* test(query): update dictionary regex plan results

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-07-30 06:20:57 +00:00
fys 3a904f332f fix(json2): standardize widening and projection cast semantics (#8661)
* fix(json2): standardize widening and projection cast semantics

Signed-off-by: fys <fengys1996@gmail.com>

* fix: cargo clippy

Signed-off-by: fys <fengys1996@gmail.com>

* fix: typos

Signed-off-by: fys <fengys1996@gmail.com>

* fix: unit test

Signed-off-by: fys <fengys1996@gmail.com>

* feat: add fast-path

Signed-off-by: fys <fengys1996@gmail.com>

* remove unsed code

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(json2): project nested json_get paths with JsonArray

Signed-off-by: fys <fengys1996@gmail.com>

* fix(query): reject projecting whole JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(query): reject whole-column JSON2 reads

Signed-off-by: fys <fengys1996@gmail.com>

* test(json2): organize limitation sqlness cases

Signed-off-by: fys <fengys1996@gmail.com>

* fix: sqlness test

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-07-30 04:08:53 +00:00
discord9 56addd0623 fix: stream remote analyze metrics while pending (#8405)
* fix: stream remote analyze metrics while pending

Signed-off-by: discord9 <discord9@163.com>

* test: verify flight metrics preserve pending batch

Signed-off-by: discord9 <discord9@163.com>

* fix: preserve direct SST perf queries in plans

Signed-off-by: discord9 <discord9@163.com>

* fix: bind flight metrics capability to query

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-07-21 04:17:50 +00:00
Ruihang Xia 0479c05092 perf: preserve dictionary-encoded query labels (#8541)
* perf: preserve dictionary-encoded query labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(client): skip dictionary Flight batches

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(servers): decode dictionary labels in HTTP output

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test(mito2): support dictionary tags in series scans

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(servers): preserve dictionary child nulls in SQL

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(promql): compare dictionary tags by logical nulls

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): keep dictionary tags within query paths

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor(query): scope PK dictionary encoding to reads

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): preserve pushdown for dictionary labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): handle dictionary labels in query operators

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): complete label type matching

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): align dictionary query schemas

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): decode incompatible OR labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): preserve dictionary partition pruning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: handle dictionary query edge cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: import dictionary downcast macro

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-07-17 13:57:10 +00:00
jeremyhi 9cde31a647 fix: account logical record batch slice memory (#8480)
* fix: account logical record batch slice memory

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: account nested view payload memory

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: align scan memory expectation with logical size

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: name Arrow inline view limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: account visible list child memory

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: exclude null list child memory

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: exclude null struct child memory

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: account fixed-size list slices

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: limit scan accounting to flat slices

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: clarify nested view accounting scope

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: reduce view memory accounting overhead

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: fix benchmark license header

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: optimize view slice accounting

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover sliced mixed-null view accounting

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: narrow slice accounting to arrow buffers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: note view accounting assumption

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-07-14 09:35:17 +00:00
Lei, HUANG af51651b64 fix: collect lightweight query-load metrics (#8437)
- Performance: add collect_lightweight_query_load_metrics to walk the
  physical plan and read raw metric values without invoking MetricCollector's
  plan-node formatting on the normal query hot path before EOF.
- Refactor: extract collect_full_metrics and keep full aggregation/formatting
  for verbose analyze output and terminal metrics.
- Test: cover lightweight partial metrics and drop-time query stats reuse.

Files: src/common/recordbatch/src/adapter.rs

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-07-08 07:07:38 +00:00
jeremyhi 5bcc2e9d6b fix: global limit for distributed inspect streams (#8412)
* fix: global limit for distributed inspect streams

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: honor tighter filter fetch under global limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: restore distribution after global limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve inherited distribution for global limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: propagate inherited hash partitioning

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-07-06 07:02:54 +00:00
Weny Xu 2dc6addb66 feat: report region query stats in heartbeat (#8401)
* feat: report region read load in heartbeat

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: expose region query stats in information schema

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update sqlness result

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: record region query stats on stream drop

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: keep region query cpu stats in nanoseconds

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-07-03 06:11:57 +00:00
Lei, HUANGandLei, HUANG 9b1672316c fix(metric-engine): report query load under physical region id (#8355)
* fix(metric-engine): report query load under physical region id

Propagate the physical region ID through the scanner, record batch
stream, and query engine so that query-load metrics (CPU time, scanned
bytes) are attributed to the correct physical region rather than always
to the logical region.

- `src/store-api/src/region_engine.rs` — add `query_load_region_id` to
  `ScannerProperties` and `set_query_load_region_id` to `RegionScanner`
  trait
- `src/mito2/src/read/seq_scan.rs`,
  `src/mito2/src/read/series_scan.rs`,
  `src/mito2/src/read/unordered_scan.rs` — implement the new trait
  method on each scanner
- `src/common/recordbatch/src/adapter.rs` — carry region id through
  `RecordBatchStreamAdapter` into `RecordBatchMetrics`
- `src/table/src/table/scan.rs` — expose region id on `RegionScanExec`
- `src/query/src/datafusion.rs` — extract region id from the physical
  plan and set it on the output stream
- `src/query/src/dist_plan/merge_scan.rs` — use metrics-contained region
  id in query-load reporting, falling back to the logical region id
- `src/metric-engine/src/engine/read.rs` — set region id on the metric
  engine scanner

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(query): satisfy clippy for query load region id

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(query): ignore missing query load region ids

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
2026-06-25 06:49:28 +00:00
Copilot 58c85dd1ce fix: guard structured JSON alignment paths against legacy JSONB columns (#8323)
* Initial plan

* fix: guard structured json alignment to fix Clippy CI failure

- Add `is_structured_json_field` function that only returns true for
  fields with both JSON extension type AND Struct Arrow data type
- Replace all usages of `is_json_extension_type` / `has_json_extension_field`
  with `is_structured_json_field` to prevent legacy JSONB binary columns
  from entering structured JSON alignment paths
- Fix logic in `FlatProjectionMapper::new_with_read_columns` to guard
  JSON type hint concretization for JSON2 columns only
- Fix `create_column` in show_create_table.rs to only emit JSON structure
  settings for JSON2 columns
- Move `mod tests` to end of flat_projection.rs to fix clippy::items_after_test_module
- Add tests for legacy JSON behavior

* fix: guard narrow_read_columns_by_json_type_hint with is_structured_json_field check

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-06-18 05:27:56 +00:00
Weny Xu e520ff8300 feat: decouple error retryability from status codes (#8301)
* feat: add retry hint to common error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: propagate retry hints in core errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: propagate retry hint through RPC metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fallback retry hints to status codes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: add explicit retry hints for retryable errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: remove status code retry fallback

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: classify io retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minior

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve datasource retry hints in query

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve metric engine retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve flow and frontend retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: share rskafka retry hint mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-17 08:27:53 +00:00
Lei, HUANG 4373c77d35 feat: use scan output bytes for read costing (#8276)
* feat: track scan output bytes and use them for read costing

Track the actual byte count of each record batch produced by
`RegionScanExec` and use it in place of the aggregated per-plan
`memory_usage` as the `table_scan` cost input. This avoids double
counting bytes that flow through multiple operators.

A named constant `REGION_SCAN_EXEC_NAME` exposes the plan node name
so downstream metric parsers remain correct if the struct is renamed.

Affected files:
- `src/table/src/table/metrics.rs` -- add `output_bytes` counter
- `src/table/src/table/scan.rs` -- record bytes, export name constant
- `src/query/src/dist_plan/merge_scan.rs` -- consume scan bytes

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: remove misleading `mem_used` gauge from scan metrics

The `mem_used` gauge was used with `add()` on every output batch, but a
RegionScan does not hold memory across polls — each batch is yielded
immediately to the upstream operator. The gauge semantics were
incorrect (cumulative `add()` on a gauge) and the value duplicated
`output_bytes` anyway.

Affected files:
- `src/table/src/table/metrics.rs` — drop `mem_used` field and methods
- `src/table/src/table/scan.rs` — remove `record_mem_usage` call

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: use stable plan names for scan byte metrics

Expose `plan_name` alongside rendered plan text so scan byte extraction no longer depends on EXPLAIN formatting. Keep old serialized metrics compatible by defaulting missing `plan_name`.

Affected files:
- `src/common/recordbatch/src/adapter.rs` -- add `plan_name` to `PlanMetrics` and populate it from `ExecutionPlan::name`
- `src/query/src/dist_plan/merge_scan.rs` -- match `REGION_SCAN_EXEC_NAME` through `plan_name`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: preserve merge scan CPU read cost

Record read cost whenever stream metrics are available, default scan bytes to zero, and aggregate scan output bytes across region scan nodes.

Files: `src/query/src/dist_plan/merge_scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: correct doc comment in `StreamMetrics`

Fix doc comment on `StreamMetrics::new()` to reference the correct
struct name (`StreamMetrics`) instead of the old `MemoryUsageMetrics`.

- `src/table/src/table/metrics.rs` — fix struct name in doc comment

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2026-06-16 02:58:13 +00:00
LFC 92a915b5a0 feat: json2 field access pushdown to parquet (#8267)
* feat: json2 field access pushdown to parquet

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-06-11 02:10:30 +00:00
discord9 1167c36c4e feat: add seq watermark in record batch metrics (#8015)
* feat: add seq watermark in record batch metrics

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-04-24 07:54:10 +00:00
LFC 209880b991 feat: json2 flush (#8011)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-04-23 03:03:37 +00:00
Ning Sun e14404c677 chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21

* chore: new format

* fix: lint

* chore: resolve lint issues

* chore: remove as_millis_f64

* chore: deps up
2026-03-30 12:13:14 +00:00
Yingwen 92e2d71f48 feat: implement prefilter framework and primary key prefilter (#7862)
* feat: prefilter basic framework

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: move arguments to RowGroupBuildContext

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: skip prefiltered exprs in FlatPruneReader

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: remove unused functions

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: update comment

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: handle partition columns in prefilter

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: apply prefiltered selection by and_then

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: handle last row cache

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: don't ignore error in PrimaryKeyFilter

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-03-30 04:24:20 +00:00
jeremyhi 3904df5397 chore: refine track memory metrics semantics (#7874)
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-03-30 03:32:49 +00:00
jeremyhi 8058ce7cf2 refactor: simplify scan memory tracking (#7827)
* refactor: simplify scan memory tracking

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: make confg-docs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by codex review comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: track_with_policy

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: minor change

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: mem granularity mb to kb

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by review comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by scan_memory_on_exhausted comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by review comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: typo

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-03-26 03:25:50 +00:00
LFC 5eac4f10aa chore: remove dependency on "atty" (#7725)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-02-26 09:58:01 +00:00
Weny Xu 5bfc728d32 fix(repartition): improve physical region allocation and compaction read path correctness (#7621)
* fix: fix metadata region

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: adjust repartition flow and compaction read compatibility

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: rename compaction mapper and pk projection

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: rename `CompactionProjectionMapper`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: clarify compaction projection naming

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: fmt

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: allow create physical table with internal columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix template logic

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit test

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update sqlness result

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-01-28 04:04:05 +00:00
Ruihang Xia 3ce7e7bc81 perf: accelerate pk filter (#7584)
* perf: accelerate pk filter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add benchmark

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* unit tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* support compare ops

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor to reduce dense decode/encode duplication

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor if-else chain

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-01-19 06:41:51 +00:00
Ruihang Xia d39895a970 feat: tune query traces (#7524)
* feat: add partition and region id

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* wip: instrument mito

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* connect region scan span

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* instrument streams

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tweak

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-01-07 08:11:09 +00:00
LFC fdab75ce27 feat: simple read write new json type values (#7175)
feat: basic json read and write

Signed-off-by: luofucong <luofc@foxmail.com>
2025-11-27 12:40:35 +00:00
LFC 4a7c16586b refactor: remove Vectors from RecordBatch completely (#7184)
* refactor: remove `Vector`s from `RecordBatch` completely

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2025-11-21 08:53:35 +00:00
jeremyhi c7fded29ee feat: query mem limiter (#7078)
* feat: query mem limiter

* feat: config docs

* feat: frontend query limit config

* fix: unused imports

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: add metrics for query memory tracker

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: right postion for tracker

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: avoid race condition

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: soft and hard limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: docs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: when soft_limit == 0

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: upgrade limit algorithm

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: remove batch window

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: batch mem size

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: refine limit algorithm

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: get sys mem

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: minor change

* feat: up tracker to the top stream

* feat: estimated_size for batch

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: minor refactor

* feat: scan_memory_limit connect to max_concurrent_queries

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: make callback clearly

* feat: add unlimted enum

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by review comment

* chore: comment on recursion_limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: refactor and put permit into RegionScanExec

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: multiple lazy static blocks

* chore: minor change

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2025-11-11 07:47:55 +00:00
LFC b7e834ab92 refactor: convert to influxdb values directly from arrow (#7163)
* refactor: convert to influxdb values directly from arrow

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2025-11-03 07:52:37 +00:00
LFC 855eb54ded refactor: convert to mysql values directly from arrow (#7096)
Signed-off-by: luofucong <luofc@foxmail.com>
2025-10-20 11:09:24 +00:00
Ning SunandLFC 145c1024d1 feat: add Value::Json value type (#7083)
* feat: struct value

Signed-off-by: Ning Sun <sunning@greptime.com>

* feat: update for proto module

* feat: wip struct type

* feat: implement more vector operations

* feat: make datatype and api

* feat: reoslve some compilation issues

* feat: resolve all compilation issues

* chore: format update

* test: resolve tests

* test: test and refactor value-to-pb

* feat: add more tests and fix for value types

* chore: remove dbg

* feat: test and fix iterator

* fix: resolve struct_type issue

* feat: pgwire 0.33 update

* refactor: use vec for struct items

* feat: conversion from json to value

* feat: add decode function

* fix: lint issue

* feat: update how we encode raw data

* feat: add convertion to fully strcutured StructValue

* refactor: take owned value in all encode/decode functions

* feat: add pg serialization of structvalue

* chore: toml format

* refactor: adopt new and try_new from struct value

* chore: cleanup residual issues

* docs: docs up

* fix lint issue

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* Apply suggestion from @MichaelScofield

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* chore: address review comment especially collection capacity

* refactor: remove unneeded processed keys collection

* feat: Value::Json type

* chore: add some work in progress changes

* feat: adopt new json type

* refactor: limit scope json conversion functions

* fix: self review update

* test: provide tests for value::json

* test: add tests for api/helper

* switch proto to main branch

* fix: implement is_null for ValueRef::Json

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
2025-10-15 20:13:12 +00:00
LFCandNing Sun 8fe17d43d5 chore: update rust to nightly 2025-10-01 (#7069)
* chore: update rust to nightly 2025-10-01

Signed-off-by: luofucong <luofc@foxmail.com>

* chore: nix update

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
2025-10-11 07:30:52 +00:00
Yingwen 47c1ef672a fix: support dictionary in regex match (#7055)
* fix: support dictionary in regex match

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: get key from keys buffer directly

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2025-10-10 03:03:34 +00:00
LFC 6d0dd2540e refactor: rewrite some UDFs to DataFusion style (part 4) (#7011)
Signed-off-by: luofucong <luofc@foxmail.com>
2025-09-24 19:50:58 +00:00
Ruihang Xia c9377e7c5a build: bump rust edition to 2024 (#6920)
* bump edition

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* gen keyword

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* lifetime and env var

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* one more gen fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* lifetime of temporaries in tail expressions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clippy nested if

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clippy let and return

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-09-08 02:37:18 +00:00
Ruihang Xia faae1db066 feat: humanize analyze numbers (#6889)
* feat: humanize analyze numbers

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update cargo lock

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tailing space

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use readable size

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-09-05 03:42:38 +00:00
LFC f9d2a89a0c chore: update datafusion family (#6675)
* chore: update datafusion family

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

* use official otel-arrow-rust

Signed-off-by: luofucong <luofc@foxmail.com>

* rebase

Signed-off-by: luofucong <luofc@foxmail.com>

* use the official orc-rust

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* remove the empty lines

Signed-off-by: luofucong <luofc@foxmail.com>

* try following PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2025-08-15 12:41:49 +00:00
Arshdeep 2e571e351f fix: add map datatype conversion in copy_table_from (#6185) (#6422)
Signed-off-by: Arshdeep54 <balarsh535@gmail.com>
2025-07-28 03:53:10 +00:00
Ruihang Xia 639b3ddc3e feat: update partial execution metrics (#6499)
* feat: update partial execution metrics

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* send data with metrics in distributed mode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* only send partial metrics under VERBOSE flag

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* loop to while

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-07-16 16:59:10 +00:00
Yingwen b1b0d0136f fix: correct MAX_EXECUTION_TIME timeout calculation (#6444)
* feat: implement statement timeout in frontend instance

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: fail fast when timeout is 0

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: update start time

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2025-07-02 08:31:40 +00:00
Lei, HUANG 6da633e70d feat: support killing process (#6309)
* feat/kill-process:
 ### Add Cancellation Support and Enhance Process Management

 - **Cancellation Handle Implementation**: Introduced `CancellationHandle` in `cancellation_handle.rs` to facilitate cancellation of futures and streams.
 - **Process Management Enhancements**:
   - Updated `ProcessManager` in `process_manager.rs` to support cancellable processes using `CancellableProcess`.
   - Added `kill_process` method for terminating processes.
 - **Stream Wrapper Update**:
   - Replaced `StreamWrapper` with `CancellableStreamWrapper` in `stream_wrapper.rs` and `instance.rs` to handle stream cancellation.
 - **Error Handling**:
   - Added `StreamCancelled` error variant in `error.rs` to handle stream cancellation scenarios.
 - **gRPC Handler Update**:
   - Added `kill_process` gRPC method in `frontend_grpc_handler.rs` to allow external process termination.
 - **Dependency Updates**:
   - Updated `Cargo.lock` and `Cargo.toml` to include `common-base` and `tokio-util`.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 **Enhancements and Bug Fixes**

 - **Dependency Update**: Updated `greptime-proto` dependency in `Cargo.lock` and `Cargo.toml` to a new revision.
 - **Error Handling Improvements**:
   - Modified error variants in `src/catalog/src/error.rs` and `src/common/frontend/src/error.rs` to improve error messages and handling.
   - Added `FrontendNotFound` error variant for better error specificity.
 - **Process Management Enhancements**:
   - Updated `ProcessManager` in `src/catalog/src/process_manager.rs` to include `kill_process` functionality with server address validation.
   - Enhanced `FrontendClient` trait in `src/common/frontend/src/selector.rs` to support `kill_process` requests.
 - **gRPC Handler Update**:
   - Refactored `FrontendGrpcHandler` in `src/servers/src/grpc/frontend_grpc_handler.rs` to handle `kill_process` requests asynchronously and return process status.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 ### Add Kill Process Functionality

 - **`Cargo.lock`, `Cargo.toml`**: Added `common-frontend` as a dependency.
 - **`server.rs`, `builder.rs`, `instance.rs`**: Updated `FrontendInvoker` and `FrontendBuilder` to support process management.
 - **`error.rs`**: Introduced `InvalidProcessId` error for handling invalid process IDs.
 - **`statement.rs`, `kill.rs`**: Implemented `execute_kill` method in `StatementExecutor` to handle the `KILL` statement.
 - **`parser.rs`, `statement.rs`**: Updated SQL parser to recognize and parse the `KILL` statement.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 ## Add Cancellation Support to Query Execution

 - **`process_manager.rs`**: Updated `CancellationHandle` initialization to use `default()` method.
 - **`cancellation_handle.rs`**: Implemented `Debug` trait for `CancellationHandle` and added `Cancellation` and `CancellableFuture` structs to support cancellable futures.
 - **`error.rs`**: Introduced `Cancelled` error variant to handle query cancellations.
 - **`instance.rs`**: Integrated `CancellableFuture` to manage query execution with cancellation support.
 - **`stream_wrapper.rs`**: Modified `CancellableStreamWrapper` to use the new `waker()` method for cancellation handling.
 - **`statement.rs`**: Added `#[allow(clippy::too_many_arguments)]` to `StatementExecutor::new` to suppress clippy warnings.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 - **Add `MetaClientMissing` Error**: Introduced a new error variant `MetaClientMissing` in `error.rs` to handle missing meta client scenarios.
 - **Refactor Cancellation Handling**: Merged `cancellation_handle.rs` into `cancellation.rs` and updated related logic in `process_manager.rs`, `instance.rs`, and `stream_wrapper.rs`.
 - **Enhance Process Management**: Improved process management logic in `process_manager.rs` to handle process cancellation more effectively.
 - **Update Tests**: Added and updated tests in `cancellation.rs` and `stream_wrapper.rs` to cover new cancellation logic and error handling.
 - **Cargo.toml Update**: Adjusted workspace settings in `Cargo.toml` for `common-frontend`.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 - **Add Tests for Process Management**: Introduced multiple async tests in `process_manager.rs` to verify query registration, deregistration, cancellation, and process killing functionalities.
 - **Update Error Message in SQL Parser**: Modified the expected error message in `parser.rs` to clarify the expected token as a "process id string literal".

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 ### Add Process Count Metrics to Catalog

 - **`metrics.rs`**: Introduced a new metric `PROCESS_LIST_COUNT` to track the count of running processes per catalog using `IntGaugeVec`.
 - **`process_manager.rs`**: Updated `CancellableProcess` to increment and decrement `PROCESS_LIST_COUNT` upon creation and destruction, respectively. Added a `Drop` implementation for `CancellableProcess` to handle metric updates.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 ### Fix process removal logic in `process_manager.rs`

 - Corrected the condition for removing an entry from the catalog in `ProcessManager` by using `o.get()` instead of `o.get_mut()`.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 - **Error Handling Improvements**:
   - Updated status codes for `Error::FrontendNotFound` and `Error::MetaClientMissing` to `StatusCode::Unexpected` in `src/catalog/src/error.rs`.
   - Changed `InvokeFrontend` error display message and status code in `src/common/frontend/src/error.rs`.
   - Added `ProcessManagerMissing` error in `src/operator/src/error.rs` and updated its handling in `src/operator/src/statement/kill.rs`.

 - **Process Management Enhancements**:
   - Added documentation for `ProcessManager` and `register_query` in `src/catalog/src/process_manager.rs`.
   - Modified `kill_process` response handling in `src/servers/src/grpc/frontend_grpc_handler.rs`.

 - **Cancellation Logic Update**:
   - Improved cancellation logic in `src/common/base/src/cancellation.rs` to use `compare_exchange` for atomic operations.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 ### Add Process Kill Count Metric and Refactor Cancellation Handle

 - **Metrics Update**: Added a new metric `PROCESS_KILL_COUNT` in `metrics.rs` to track the count of completed kill process requests per catalog.
 - **Refactor Cancellation Handle**: Renamed `cancellation_handler` to `cancellation_handle` across multiple files for consistency:
   - `process_manager.rs`
   - `instance.rs`
   - `stream_wrapper.rs`
 - **Process Management**: Updated process management logic in `process_manager.rs` to increment the `PROCESS_KILL_COUNT` metric upon successful process termination.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 Update metric description in `metrics.rs`

 - Changed the description of `PROCESS_KILL_COUNT` to reflect the count of killed processes instead of running processes in `metrics.rs`.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/kill-process:
 Update `greptime-proto` Dependency and Fix Response Field

 - **Updated Dependency**: Changed the `greptime-proto` Git revision in `Cargo.lock` and `Cargo.toml` to `f0913f1`.
 - **Code Fix**: Modified `frontend_grpc_handler.rs` to correct the response field from `found` to `success` in `KillProcessResponse`.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2025-06-13 13:30:25 +00:00
Weny Xu 0d4f27a699 fix: convert JSON type to JSON string in COPY TABLE TO statment (#6255)
* fix: convert JSON type to JSON string in COPY TABLE TO statement

* chore: apply suggestions from CR

* chore: apply suggestions from CR
2025-06-06 02:23:57 +00:00
Yingwen a5c443f734 perf: keep compiled regex in SimpleFilterEvaluator to avoid re-compiling (#5919)
* feat: cache regex in evaluator

* chore: fix warnings

* chore: add reference

* refactor: address CR comments

* Add negative to state
* Don't create the evaluator if the regex is invalid

* test: add test for maybe_build_regex
2025-04-18 09:36:28 +00:00
LFC 71255b3cbd refactor: avoid empty display in errors (#5858)
* refactor: avoid empty display in errors

* fix: resolve PR comments
2025-04-10 10:08:45 +00:00