Commit Graph
241 Commits
Author SHA1 Message Date
Weny Xu e9ad585975 feat(event): add event context to procedure events (#8734)
* feat(event): add trigger context to procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix trigger context event contracts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve trigger context origins

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve lifecycle trigger contexts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): gate enterprise trigger imports

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): assert trigger contexts exactly

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): restore migration test literals

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event): add trigger context to non-table ddl

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): simplify trigger context encoding

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve auto alter trigger reason

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): require explicit trigger context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): derive trigger context at ddl boundary

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): propagate DDL trigger reasons

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): resolve alter table rebase conflict

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): fix alter table trigger context setup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): pass trigger context to region migration

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix unknown trigger context assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): centralize trigger context protocol mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix migration trigger context assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(procedure): rename event runtime context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): rename trigger context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix migration event context assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 1f1c9270a8)
2026-08-21 11:51:11 +08:00
discord9 c4baa56b23 fix(flow): fix flow stats aggregation and df_plan_to_sql quoting (#8729)
* fix(flow): fix flow stats aggregation and df_plan_to_sql quoting

1. Distributed-mode flow stats last-writer-wins overwrite:
   Each flownode heartbeat put its local flow state map into the single
   global __flow/state key, so reports from different nodes overwrote
   each other. Store per-flownode reports under
   __flow/state/node/{node_id} in the in-memory KV and aggregate on each
   heartbeat (last_exec_time_map/state_size/start_time_map take the max
   across nodes) into the global key. FlowStateHandler derives node
   identity from header.member_id (fallback peer.id) and ignores
   identity-less reports. Per-node keys clear automatically on leader
   change KV reset. Adapts to FlowStateValue.start_time_map added in
   #8392.

2. df_plan_to_sql unquoted special characters break flush/scheduled
   execution: ForceQuoteIdentifiers only quoted uppercase identifiers,
   so Prometheus-style table names with ':' (e.g. cpu_cores:sum) were
   left unquoted, producing invalid SQL ('keyword: :'). Quote any
   identifier with non-[a-z0-9_] chars using double quotes
   (dialect-neutral).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): address review comments on quoting and logging

- df_plan_to_sql: also quote digit-leading identifiers (e.g. 123metrics)
  which would produce invalid SQL when re-parsed. SQL keywords are
  intentionally not checked (ALL_KEYWORDS would over-quote common column
  names like number; the unparse failure path has an InsertIntoPlan
  fallback).
- flow_state_handler: downgrade identity-less report log from warn! to
  debug! to avoid an anomalous sender spamming warn every heartbeat.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 375dd569482235914ef09ef5c21e15cc79fcd052)
2026-08-21 11:51:11 +08:00
discord9 f9135380b5 feat(flow): add flow statistics and SHOW FLOW STATUS (release slice)
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 523c60c53bdc21fd1b1fa43b857e78de710f0d58)
2026-08-21 11:51:11 +08:00
jeremyhi 448f973593 fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address local file sandbox review findings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: support Windows local copy paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: improve sandbox path errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: simplify local path error context

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: stream secure filesystem listings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* style: derive local file access default

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: improve local file access errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address local file access review findings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: simplify local file access coverage

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: harden sandboxed local file backends

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject directory copy targets before creation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: avoid implicit string clone in file table listing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-07-31 13:23:15 +00:00
discord9 a0e6f0b4ca fix(flow): avoid duplicate incremental planning warnings (#8611)
fix(flow): reduce duplicate incremental fallback warning

Signed-off-by: discord9 <discord9@163.com>
2026-07-24 07:40:57 +00:00
discord9 4c5be47400 fix(flow): downgrade disabled incremental checkpoint log (#8572)
Signed-off-by: discord9 <discord9@163.com>
2026-07-22 10:41:09 +00:00
discord9 9f5a11e82a fix(flow): lower routine batching messages to debug (#8592)
fix(flow): reduce expected batching log noise

Signed-off-by: discord9 <discord9@163.com>
2026-07-22 08:46:42 +00:00
Lei, HUANG ed8a4990fe feat(flow): handle time_ranges in DirtyWindowRequest (#8582)
* feat(flow): handle time_ranges in DirtyWindowRequest

Bump greptime-proto to include the new `time_ranges` field on
DirtyWindowRequest (GreptimeTeam/greptime-proto#330) and mark the
corresponding aligned time windows as dirty in the batching engine,
in addition to the existing per-timestamp dirty marking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(flow): fix doc comment spacing in align_time_window

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(flow): cover time_ranges in handle_mark_dirty_time_window

Verify a valid [start_inclusive, end_exclusive) range is aligned to
time window boundaries and stored with an explicit end, and that empty
or reversed ranges are skipped.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(flow): union merged dirty windows with the larger end

Merging a bounded dirty range with a window contained in it (e.g.
[0s, 15s) with nested [5s, 10s), or an unbounded dirty window inside a
bounded range) previously assigned the contained window's upper bound,
shrinking the merged window and permanently dropping the tail range
from re-computation. Keep max(prev_upper, cur_upper) instead.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(flow): clip bounded dirty ranges at the expire bound

merge_dirty_time_windows dropped every window whose start is before
expire_lower_bound, so a bounded dirty range crossing the bound (e.g.
[0s, 15s) with expire 10s) lost its still-live suffix [10s, 15s). Now
bounded ranges are dropped only when their end is at/before the expire
bound, and crossing ranges are clipped to the bound (which the caller
aligns to the time window boundary). Unbounded windows keep the
existing start-based behavior.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(flow): fall back to full dirty on dirty-window alignment failure

An eval/alignment error previously aborted the per-task dirty-marking
closure, losing every dirty timestamp and range accumulated for that
task, while the RPC still returned Ok so the producer would not retry.
On alignment failure now log a warning and mark the whole task dirty
(set_dirty) instead, so the affected data is conservatively
recomputed.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(flow): apply rustfmt to new dirty-window merge tests

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(flow): cover time index units in dirty window marking

Document that DirtyWindowRequest timestamps/time_ranges are bare i64s
interpreted in the source table's time index native unit, and add a
test expressing the same [3s, 11s) range in second/millisecond/
microsecond/nanosecond units across four tables, asserting all align
to the same dirty window [0s, 15s).

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: bump greptime-proto to 8127f179

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(flow): return dirty-window alignment errors to callers

Do not acknowledge a DirtyWindowRequest when a time-windowed task cannot
align a timestamp or range. The previous conservative fallback used
set_dirty(), but that marker only represents a single epoch-start window
for time-windowed flows, so it could still lose the affected dirty
range. Propagate task errors through the join loop instead so producers
can retry.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: update proto to commits on main

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-07-21 12:56:01 +00:00
Ruihang Xia 0479c05092 perf: preserve dictionary-encoded query labels (#8541)
* perf: preserve dictionary-encoded query labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(client): skip dictionary Flight batches

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(servers): decode dictionary labels in HTTP output

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test(mito2): support dictionary tags in series scans

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(servers): preserve dictionary child nulls in SQL

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(promql): compare dictionary tags by logical nulls

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): keep dictionary tags within query paths

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor(query): scope PK dictionary encoding to reads

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): preserve pushdown for dictionary labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): handle dictionary labels in query operators

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): complete label type matching

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): align dictionary query schemas

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): decode incompatible OR labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): preserve dictionary partition pruning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: handle dictionary query edge cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: import dictionary downcast macro

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-07-17 13:57:10 +00:00
QuakeWang 5d9dddffa7 fix(flow): convert streaming expiration to milliseconds (#8481)
* fix(flow): convert streaming expiration to milliseconds

Flow EXPIRE AFTER values are stored in seconds, while streaming flow timestamps and durations use milliseconds. Passing the value through unchanged expires state and limits refill scans 1000 times too early.

Convert the value at streaming create and refill boundaries, reject overflow, and document the CreateFlowArgs unit.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* fix(flow): reject negative streaming expiration

Reject negative EXPIRE AFTER values at the streaming adapter boundary so refill ranges cannot move into the future. Keep zero valid and retain checked seconds-to-milliseconds conversion.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
2026-07-14 04:42:23 +00:00
discord9 b6438d3ea2 fix(flow): rebind stale snapshot fence (#8409)
* fix(flow): rebind stale snapshot fence

Signed-off-by: discord9 <discord9@163.com>

* refactor(flow): avoid repeated stale fence error formatting

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-07-07 08:36:39 +00:00
discord9 9936a83384 feat(query): add runtime provider interface (#8386)
* feat(query): add runtime provider interface

Signed-off-by: discord9 <discord9@163.com>

* docs(query): document runtime provider interface

Signed-off-by: discord9 <discord9@163.com>

* feat(query): pass runtime builder to provider

Signed-off-by: discord9 <discord9@163.com>

* fix(query): make runtime provider fallible

Signed-off-by: discord9 <discord9@163.com>

* fix(query): propagate runtime provider errors

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-07-07 07:18:23 +00:00
discord9 d3cc1b1888 feat(flow): stabilize eval interval scheduling (#8360)
* feat(flow): stabilize eval interval scheduling

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): satisfy eval schedule clippy

Signed-off-by: discord9 <discord9@163.com>

* test(flow): trim eval schedule coverage

Signed-off-by: discord9 <discord9@163.com>

* test(flow): cover stable eval scheduling

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): reserve scheduled runtime hint

Signed-off-by: discord9 <discord9@163.com>

* test(flow): trim sqlness result eof

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): harden eval schedule edges

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): address scheduled flow review

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): clean scheduled config handling

Signed-off-by: discord9 <discord9@163.com>

* test(flow): add eval interval compat case

Signed-off-by: discord9 <discord9@163.com>

* test(flow): cover show create flow in compat

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): drop scheduled time from flow context

Signed-off-by: discord9 <discord9@163.com>

* test(flow): assert scheduled now binding

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-29 09:55:27 +00:00
dennis zhuang 3b8f55e490 docs(agents): add per-crate guides, architecture invariants, and generated-files list (#8346)
* docs(agents): add per-crate guides, architecture invariants, and generated-files list

Add agent/contributor navigation docs modeled on the AGENTS.md convention:

- Per-crate AGENTS.md for hot crates (mito2, metric-engine, flow, frontend,
  meta-srv): module map, read/write paths, change-coupling points, test
  commands, and gotchas.
- .agents/architecture-invariants.md: repo-wide rules that clippy and the
  style guide do not cover (format compatibility, crate layering, async
  runtimes, error handling, experimental gating, the DataFusion fork).
- .agents/generated-files.md: tool-generated artifacts that must not be
  hand-edited (sqlness .result, config.md, dashboards, build.rs output, proto).
- Anchor the .gitignore CLAUDE.md/AGENTS.md rules to the repo root so per-crate
  AGENTS.md files are tracked while root-level personal config stays ignored.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: update crate AGENTS.md and fix config.md path

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(agents): fix DataFusion patch layout and SQL query lifecycle order

Address review feedback on #8346:

- architecture-invariants: the DataFusion sub-crates pin an exact crates.io
  version in [workspace.dependencies] and are redirected to the fork rev in
  [patch.crates-io]; the two sections hold different forms, not the same rev.
- frontend: the SQL query lifecycle runs the pre_parsing/post_parsing
  interceptors around parsing, before the per-statement permission check.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-06-24 09:14:09 +00:00
Lei, HUANG a3461caf9d feat: expose region read load metrics (#8316)
* feat: expose region read load through Prometheus metrics and heartbeat

Introduce region-level query load tracking (CPU time and scanned bytes)
collected by `RegionScanExec`, exposed via Prometheus metrics and optionally
reported through heartbeat region stats.

- **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add
  `greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`,
  and `greptime_mito_region_written_bytes_since_open` gauge metrics.
- **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`,
  `src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with
  `IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle
  management (init, reset, remove-on-drop).
- **RegionStatistic** (`src/store-api/src/region_engine.rs`,
  `src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and
  `query_scanned_bytes` fields.
- **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from
  metadata and data regions.
- **Heartbeat** (`src/datanode/src/heartbeat.rs`,
  `src/common/meta/src/datanode.rs`): Relay region query load via heartbeat
  `RegionStat`; add test.
- **Query engine** (`src/query/src/options.rs`,
  `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`,
  `src/query/src/dist_plan/merge_scan.rs`,
  `src/query/src/dist_plan/analyzer.rs`,
  `src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config;
  wire `RegionScanExec` to accumulate CPU time and scanned bytes.
- **Table scan** (`src/table/src/table/scan.rs`,
  `src/table/src/table/metrics.rs`): Wire table scan metrics.
- **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`,
  `config/frontend.example.toml`, `config/config.md`): Add example config and
  documentation for `enable_region_query_load_report`.
- **Tests** (`src/mito2/src/engine/basic_test.rs`,
  `src/mito2/src/engine/close_test.rs`,
  `src/cmd/tests/load_config_test.rs`,
  `src/flow/src/adapter.rs`): Add unit tests for region query load reporting
  and metric cleanup on region close; set default config values.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: move region read load report config from query layer to mito engine

Move the `enable_region_query_load_report` setting from query-level config
(`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config
(`MitoConfig`), and expose it through the `RegionScanner` trait instead
of `ScanRequest`/`PrepareRequest`.

- Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs`
- Scan region plumbing: `src/mito2/src/read/scan_region.rs`
- RegionScanner trait: `src/store-api/src/region_engine.rs`
- Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs`
- RegionScanExec: `src/table/src/table/scan.rs`
- Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs`
- Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs`
- Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: move region query load report config from MitoConfig to LoggingOptions

Relocate the `enable_region_query_load_report` setting from
`MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`),
and thread it into `MitoEngineBuilder` instead of reading from
the engine config directly. This makes the region read-load
reporting a per-node logging/observability concern rather than
a per-engine storage setting.

- `config/config.md`
- `config/datanode.example.toml`
- `config/standalone.example.toml`
- `src/common/telemetry/src/logging.rs`
- `src/datanode/src/datanode.rs`
- `src/mito2/src/config.rs`
- `src/mito2/src/engine.rs`
- `src/mito2/src/region.rs`

Signed-off-by: Lei Huang <lei@huang.to>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: report region query load on stream drop instead of stream end

Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()`
to `Drop::drop()` so that region query load is reported even when the
stream is dropped prematurely (not just when fully consumed).

Affected files:
- `src/table/src/table/scan.rs`

Signed-off-by: Lei, Huang <huanglei@qiyi.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: make region query load reporting configurable

Introduce `enable_region_query_load_report` flag to optionally report
per-region `query_cpu_time` and `query_scanned_bytes` metrics instead
of always creating them. When disabled, the Prometheus gauges are not
created (`None`), avoiding metric churn for workloads that do not
need query-level load tracking.

- `src/common/meta/src/datanode.rs` — Placeholder fields for query load
- `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset
- `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener`
- `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop`
- `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open
- `src/mito2/src/worker/handle_create.rs` — Pass flag on region create
- `src/mito2/src/worker/handle_open.rs` — Pass flag on region open
- `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder`
- `src/mito2/src/test_util.rs` — Test helpers for both modes
- `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases
- `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: remove elapsed_compute metric from scan stream

The elapsed_compute metric conflated poll-wait time with actual CPU
computation, making it misleading. Removed the metric and its
recording path from StreamMetrics and StreamWithMetricWrapper.

Added a test asserting that poll duration is not reported as
elapsed_compute.

- `src/table/src/table/metrics.rs` — removed elapsed_compute field,
  builder, and record_elapsed_compute method
- `src/table/src/table/scan.rs` — removed record_elapsed_compute
  call; added SlowRecordBatchStream test helper and
  wrapper_poll_time_is_not_elapsed_compute test

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: disable region query load report for compaction scans

Compaction scans are internal operations initiated by the engine,
not user queries. Disable region query load reporting when the
scan input is marked as compaction to avoid misleading load metrics.

- `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report`
  to `false` when compaction is enabled; add unit test

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* test: add `enable_per_region_metrics` config to HTTP integration test

- Enable per-region metrics config in HTTP test setup

\`tests-integration/tests/http.rs\`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: remove region query load reporting tests and helpers

Remove the region query load reporting feature from the codebase,
including tests, test utilities, and helper infrastructure that were
part of this now-deprecated functionality.

Specifically:

- Remove region query load reporting tests from
  `src/mito2/src/engine/basic_test.rs` and
  `src/table/src/table/scan.rs`, and the region close metrics test
  from `src/mito2/src/engine/close_test.rs`

- Remove region query load report test utilities and simplify engine
  construction helpers in `src/mito2/src/test_util.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* perf: avoid disabled region query load timing

Summary:
- Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled.
- Keep region query-load CPU accounting active only when reporting is enabled.

Files:
- `src/table/src/table/scan.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: move per-region query load reporting from storage to query engine

Move `enable_per_region_metrics` from datanode to frontend config and
migrate query load tracking (CPU time, scanned bytes) from mito2
storage engine to the query engine's distributed scan planner. The
storage-level metrics plumbing and `enable_region_query_load_report`
flag are removed from mito2, `ScanInput`, `ScanRegion`, and
`RegionScanner`. Query-level metrics are now collected in
`merge_scan.rs` via `scan_region_load`.

- `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes`
  metrics, `enable_region_query_load_report` plumbing from engine,
  region, opener, scanner types, workers
- `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes`
  from `RegionStatistic`
- `src/metric-engine/` -- Remove query load fields from
  `get_region_statistic`
- `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`;
  wire through planner, optimizer, merge scan with `scan_region_load`
  metrics
- `src/frontend/` -- Pass `enable_per_region_metrics` into
  `QueryOptions`
- `src/common/meta/` -- Remove TODO for query load fields
- `config/` -- Move `enable_per_region_metrics` from datanode to
  frontend and standalone example configs
- `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode
  config test
- `src/flow/` -- Add `enable_per_region_metrics` default to flownode
  options
- `src/table/` -- Remove unused query load fields from scan
- `src/datanode/` -- Remove
  `with_enable_region_query_load_report` calls

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: remove obsolete mito write load metric

Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path.

Related files:
- \`src/mito2/src/metrics.rs\`
- \`src/mito2/src/region.rs\`
- \`src/mito2/src/region/opener.rs\`
- \`src/mito2/src/region_write_ctx.rs\`
- \`src/mito2/src/engine/basic_test.rs\`
- \`src/mito2/src/worker.rs\`
- \`src/mito2/src/config.rs\`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: change region query load metrics from gauge to counter

Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from
`IntGaugeVec` to `IntCounterVec` since these values are monotonically
increasing and do not need gauge semantics. Update corresponding `add`
calls to `inc_by` in merge scan reporting.

Files:
- `src/store-api/src/metrics.rs` — metric type and label changes
- `src/query/src/dist_plan/merge_scan.rs` — caller adaptation

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: pass ReadItem directly to report_region_query_load

Move `region_scan_load` call to the caller, so `report_region_query_load`
accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`.

- `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call,
  remove stale test

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: ensure region query load is reported on MergeScanExec drop

Remove the `enable_per_region_metrics` parameter from `report_region_query_load`
so region load metrics are always emitted. Add a `Drop` impl for
`MergeScanExec` that reports sub-stage metrics when the executor is
dropped, covering edge cases where per-region metric emission was
missed. Add a unit test verifying CPU time and scanned bytes are
recorded on drop.

Affected file: `src/query/src/dist_plan/merge_scan.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: gate region query load reporting

Guard drop-time region query load reporting with the configured per-region metrics flag.

Related files:
- \`src/query/src/dist_plan/merge_scan.rs\`

Symbols:
- \`MergeScanExec::drop\`
- \`enable_per_region_metrics\`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: clean region query load metrics on drop

Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry.

Related files:
- \`src/mito2/src/region.rs\`

Symbols:
- \`MitoRegion::drop\`
- \`REGION_QUERY_CPU_TIME\`
- \`REGION_QUERY_SCANNED_BYTES\`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei Huang <lei@huang.to>
Signed-off-by: Lei, Huang <huanglei@qiyi.com>
2026-06-17 16:03:09 +00:00
Weny Xu e520ff8300 feat: decouple error retryability from status codes (#8301)
* feat: add retry hint to common error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: propagate retry hints in core errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: propagate retry hint through RPC metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fallback retry hints to status codes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: add explicit retry hints for retryable errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: remove status code retry fallback

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: classify io retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minior

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve datasource retry hints in query

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve metric engine retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve flow and frontend retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: share rskafka retry hint mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-17 08:27:53 +00:00
discord9 f6f56e6bc5 fix!: fence scoped flow repair snapshots (#8277)
* fix: fence scoped flow repair snapshots

Signed-off-by: discord9 <discord9@163.com>

* test: trim duplicate flow task tests

Signed-off-by: discord9 <discord9@163.com>

* test: moreless

Signed-off-by: discord9 <discord9@163.com>

* refactor: simplify flow query failure fallback

Signed-off-by: discord9 <discord9@163.com>

* test: update need eval interval

Signed-off-by: discord9 <discord9@163.com>

* docs: for helper fn

Signed-off-by: discord9 <discord9@163.com>

* chore: less test

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* test: update for eval interval

Signed-off-by: discord9 <discord9@163.com>

* fix: consume dirty windows after successful query

Signed-off-by: discord9 <discord9@163.com>

* test: rm useless tests

Signed-off-by: discord9 <discord9@163.com>

* test: standalone seq&rm dead if

Signed-off-by: discord9 <discord9@163.com>

* chore: move to pending window instead

Signed-off-by: discord9 <discord9@163.com>

* chore: mark full also call abandon_fenced_repair

Signed-off-by: discord9 <discord9@163.com>

* chore: instant for pending fenced repair

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-12 13:27:12 +00:00
discord9 3608c90b2e feat: pass snapshot read bounds over flight (#8279)
Signed-off-by: discord9 <discord9@163.com>
2026-06-11 12:05:02 +00:00
discord9 1c7b4c11fc fix(flow): support complex SQL full-query flows (#8242)
* fix(flow): skip TWE detection for complex plans

Signed-off-by: discord9 <discord9@163.com>

* chore: helper

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): allow SQL CTE flow parsing

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): execute unscoped SQL snapshots as SQL

Signed-off-by: discord9 <discord9@163.com>

* fix: column list

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-08 13:07:43 +00:00
shuiyisong 5fd5b91b29 chore: remove auth in flownode (#8244)
* chore: remove auth in flownode

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update docs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add flow startup check

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: clippy

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-06-08 08:15:19 +00:00
Ning Sun fd64ced4da feat: introduce plugin setup functions with richer context (#8256)
feat: enrich plugin setup context
2026-06-08 06:53:08 +00:00
discord9 d304df6e75 fix: run eval-interval flow without time window (#8231)
* fix: run eval-interval flow without time window

Signed-off-by: discord9 <discord9@163.com>

* test: cover eval-interval flow join query

Signed-off-by: discord9 <discord9@163.com>

* fix: address eval interval flow review comments

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-03 13:06:22 +00:00
discord9 7f76ad5439 feat: validate batching flow sink schema on create (#8176)
* feat: check schema on create

Signed-off-by: discord9 <discord9@163.com>

* chore: update sqlness

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): avoid duplicate fields when matching sink schema

Signed-off-by: discord9 <discord9@163.com>

* fix: null handling

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: debug log

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-02 08:24:50 +00:00
discord9 28fd796f4e fix(flow): harden incremental read correctness (#8196)
* fix(flow): harden incremental read correctness

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): propagate dirty window options

Signed-off-by: discord9 <discord9@163.com>

* test: more

Signed-off-by: discord9 <discord9@163.com>

* chore: test config api

Signed-off-by: discord9 <discord9@163.com>

* refactor: split gen

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* fix: allowlist key

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-01 02:48:00 +00:00
dennis zhuang ed9312f8e3 feat: global switch for creating tables automatically (#8203)
* feat: global switch for creating table automatically

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: make auto_create_table as comment by default

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: respect gloabl switch for metric engine

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-05-31 23:51:14 +00:00
discord9 eccd97b5c7 feat(flow): support incremental read checkpoints (#8179)
* feat: flownode inc mode

Signed-off-by: discord9 <discord9@163.com>

* chore: rename fallback reason

Signed-off-by: discord9 <discord9@163.com>

* fix: harden flow incremental checkpoints

Signed-off-by: discord9 <discord9@163.com>

* fix: address flow watermark lint

Signed-off-by: discord9 <discord9@163.com>

* fix: address flow clippy

Signed-off-by: discord9 <discord9@163.com>

* refactor: clarify incremental plan preparation

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* test: more sqlness test

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-05-28 09:31:46 +00:00
shuiyisong 17815830ed chore: add LeaderServicesContext control to standalone (#8164)
* chore: add refresh hook

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: merge start_with_context and start

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: place reset in recover

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: revert stop changes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-05-28 09:23:30 +00:00
discord9 44f1804b5e feat: add flow query-context plumbing for terminal watermarks (#8154)
* feat: add flow checkpoint plumbing

Signed-off-by: discord9 <discord9@163.com>

* fix: restore when fail

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: clean up some test

Signed-off-by: discord9 <discord9@163.com>

* clippy

Signed-off-by: discord9 <discord9@163.com>

* refactor: move more to pr3b

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-05-26 07:24:18 +00:00
discord9 454344ab4d feat: inc query join rewrite helper (#8108)
* feat: inc query join rewrite helper

Signed-off-by: discord9 <discord9@163.com>

* chore: rm unwrap

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* chore

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: pre review

Signed-off-by: discord9 <discord9@163.com>

* more per review

Signed-off-by: discord9 <discord9@163.com>

* test: add expected plan test

Signed-off-by: discord9 <discord9@163.com>

* fix: disallow having

Signed-off-by: discord9 <discord9@163.com>

* refactor: allow list

Signed-off-by: discord9 <discord9@163.com>

* fix: also reject groupings

Signed-off-by: discord9 <discord9@163.com>

* fix: rm global aggr

Signed-off-by: discord9 <discord9@163.com>

* refactor: split test

Signed-off-by: discord9 <discord9@163.com>

* fix: computed shadow expr

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-05-21 07:04:57 +00:00
Lei, HUANG 2fdbe6c8c3 feat: expose node info for placement selectors (#8095)
* feat: expose node info for placement selectors

Return `NodeInfo` from `PeerDiscovery` methods and keep OSS selectors mapping back to `Peer`.

Carry `__greptime_origin_frontend.addr` from frontend create-table DDLs into selector `extensions`, and thread `PeerAllocContext` through table-route allocation.

Persist datanode `NodeInfo` when heartbeat stats are absent so collected env vars remain available after restart.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: skip datanode node info without stats

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: avoid unnecessary workload clones

Skip workload cloning for inactive nodes and for active node-info lookups without workload filters.

Files: `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: require frontend origin address

Require `StatementExecutor` to carry a concrete frontend origin address and always attach it to meta DDL query contexts.

Files: `src/operator/src/statement.rs`, `src/operator/src/statement/ddl.rs`, `src/operator/src/utils.rs`, `src/frontend/src/instance/builder.rs`, `src/frontend/src/heartbeat.rs`, `src/flow/src/server.rs`, `src/cmd/src/standalone.rs`, `src/cmd/src/flownode.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: reuse resolved frontend address

Resolve the frontend peer address once in the frontend builder, store it on the instance, and reuse it for heartbeat and flow invoker origins.

Files: `src/frontend/src/instance/builder.rs`, `src/frontend/src/instance.rs`, `src/frontend/src/heartbeat.rs`, `src/cmd/src/frontend.rs`, `src/cmd/src/standalone.rs`, `src/frontend/src/frontend.rs`, `src/frontend/src/heartbeat/tests.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: preserve datanode lease liveness

Filter active datanode node infos through lease timestamps and workloads while preserving node info fields such as reported env vars.

Files: `src/meta-srv/src/discovery/utils.rs`, `src/meta-srv/src/discovery/lease.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* Remove stale datanode lease helper

- `discovery`: remove the obsolete `alive_datanodes` helper and related tests in `src/meta-srv/src/discovery/utils.rs` and `src/meta-srv/src/discovery/lease.rs`
- `integration`: update cluster and standalone setup paths in `tests-integration/src/cluster.rs` and `tests-integration/src/standalone.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/env-based-region-selector-oss: simplify lease discovery

- `lease-discovery`: simplify logic and remove unused utilities in `src/meta-srv/src/discovery/lease.rs` and `src/meta-srv/src/discovery/utils.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2026-05-15 07:35:18 +00:00
discord9anddennis zhuang a282b207a6 feat: flow inc query terminal metrics transport (#8045)
* feat: OutputMetrics for inc query

Signed-off-by: discord9 <discord9@163.com>

* chore: rm unused

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* docs: explain should_collect_region_watermark_from_extensions

Signed-off-by: discord9 <discord9@163.com>

* fix: transport flow extensions via metadata

Avoid routing Flow-specific query extensions through comma-separated hints so checkpoint JSON values remain intact over Flight.

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* more per review

Signed-off-by: discord9 <discord9@163.com>

* chore: more review

Signed-off-by: discord9 <discord9@163.com>

* test: compat test

Signed-off-by: discord9 <discord9@163.com>

* fix: standalone also monkey patch

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2026-05-11 11:58:51 +00:00
Weny Xu 0c942fc23a refactor: unify frontend discovery with active peer discovery (#8031)
* feat: add MetaClient peer discovery support

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: reuse peer discovery for flow frontend discovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: use active frontend peers in process manager

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-04-27 06:40:37 +00:00
Ning Sun e14404c677 chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21

* chore: new format

* fix: lint

* chore: resolve lint issues

* chore: remove as_millis_f64

* chore: deps up
2026-03-30 12:13:14 +00:00
discord9 3beb538aa8 fix: rm useless analyzer (#7797)
* fix: rm useless analyzer

Signed-off-by: discord9 <discord9@163.com>

* test: rm related test

Signed-off-by: discord9 <discord9@163.com>

* test: flow tql avg

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-03-12 10:53:47 +00:00
discord9 d1151b665b feat: flow tql cte (#7702)
* feat: flow tql cte

Signed-off-by: discord9 <discord9@163.com>

* fix: creating flow TQL CTE source tables lose cte part in query

Signed-off-by: discord9 <discord9@163.com>

* test: update sqlness result

Signed-off-by: discord9 <discord9@163.com>

* chore

Signed-off-by: discord9 <discord9@163.com>

* fix: properly canonicalize ident

Signed-off-by: discord9 <discord9@163.com>

* feat: even stricter check

Signed-off-by: discord9 <discord9@163.com>

* chore: sqlness update

Signed-off-by: discord9 <discord9@163.com>

* chore: after rebase fix

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-03-06 03:36:42 +00:00
Ruihang Xia 77013d9085 feat: report flow stats from streaming and batching engines (#7701)
* fix: report flow stats from streaming and batching engines

* handle restart report handler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename fn name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-02-11 07:38:09 +00:00
Ning Sun 43afb7962a refactor: remove session from common meta (#7698)
* refactor: remove session dependency from common-meta

* chore: add udeps

* chore: format

* fix: lint issues

* chore: update oneshot

* chore: update unused deps
2026-02-11 03:04:45 +00:00
Weny Xu 0ed3b83099 refactor: rename partition rule version to partition expr version (#7696)
* refactor: rename partition rule version to partition expr version

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-02-10 10:12:47 +00:00
LFC 8c23b29725 refactor: remove the RawTableMeta and RawTableInfo to make codes more concise (#7626)
* refactor: remove the `RawTableMeta` and `RawTableInfo` to make codes more concise

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-02-10 07:10:04 +00:00
Weny Xu 8026b23834 feat: partition rule version validation for writes and staging (#7628)
* feat: verify partition rule

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add partition version cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: header check

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: fmt toml

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minor refactor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: header

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: fix clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minor refactor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: nit

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: nit

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-02-06 12:16:34 +00:00
discord9 a6e69bc180 feat: flow last non null (#7646)
* feat: flow last non null support

Signed-off-by: discord9 <discord9@163.com>

* clippy

Signed-off-by: discord9 <discord9@163.com>

* test: report error when sink is not last non null

Signed-off-by: discord9 <discord9@163.com>

* fix: error if query column matches nothing if partial

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-01-30 12:34:33 +00:00
Ruihang Xia 38b5df574c refactor: SQL and gRPC server handlers (#7637)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-01-30 07:19:45 +00:00
discord9andRuihang Xia 085e9dfc7b fix: use phy table when need fitler by tsid/table id (#7609)
* fix: use phy table when need fitler by tsid/table id

Signed-off-by: discord9 <discord9@163.com>

* feat: add subquery alias

Signed-off-by: discord9 <discord9@163.com>

* test: update after alias

Signed-off-by: discord9 <discord9@163.com>

* test: sort&redact

Signed-off-by: discord9 <discord9@163.com>

* test: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2026-01-26 06:11:16 +00:00
discord9 38e4a94956 feat: ser/de for HistogramFold&UnionDistinctOn (#7601)
* feat: ser/de for histogramFold&UnionDistinctOn

Signed-off-by: discord9 <discord9@163.com>

* chore: update proto

Signed-off-by: discord9 <discord9@163.com>

* fix: skip find time window when in tql

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-01-21 15:02:20 +00:00
Weny Xu 25687bb282 feat: add ddl timeout/wait options, repartition WITH parsing, meta-client startup refactor (#7589)
* feat: add ddl request timeouts and unify meta client startup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: omplement ALTER TABLE repartition DDL options parsing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add sqlness tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: pass timeout argument to procedure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: refine comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: assert timeout

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-01-20 09:26:53 +00:00
Weny Xu ada4666e10 refactor: remove region_numbers from TableMeta and TableInfo (#7519)
* refactor: remove `region_numbers` from `TableMeta` and `TableInfo`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: create partitions from region route

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix build

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-01-06 13:21:36 +00:00
jeremyhiandCopilot 898e84898c feat!: make heartbeat config only in metasrv (#7510)
* feat: make heartbeat config only in metasrv

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* feat: refine config doc

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: make the heartbeat setup simple

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: revert config

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: proto update

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: fix sqlness wrong cfg

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-06 09:43:36 +00:00
jeremyhi 6a6b34c709 feat!: memory limiter unification write path (#7437)
* feat: remove option max_in_flight_write_bytes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: replace RequestMemoryLimiter

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: add integration test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: fix test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: global permit pool on writing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by ai comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2025-12-23 02:18:49 +00:00
fys 0bc5a305be chore: add wait_initialized method for frontend client (#7414)
* chore: add wait_initialized method for frontend client

* fix: some

* fix: cargo fmt

* add comment

* add unit test

* rename

* fix: cargo check

* fix: cr by copilot
2025-12-17 08:13:36 +00:00
fys 913ac325e5 chore: add is_initialized method for frontend client (#7409)
chore: add `is_initialized` for frontend client
2025-12-15 12:51:09 +00:00