Commit Graph
60 Commits
Author SHA1 Message Date
+4 cabc2f6cc6 feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (#7987) (#8392)
* feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test(flow): add sqlness golden result for flow_status

Signed-off-by: Palak Jha <palakjha916@gmail.com>

* fix(catalog): remove unused OptionExt import in flow_statistics

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* docs(flow): fix stale 'recent errors' comment on QueryFlowExecStats

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test: regenerate golden results for flow_statistics table

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* style: apply rustfmt to flow_statistics changes

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor(catalog): hoist current_time_millis out of flow loop and clamp uptime

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore: remove accidentally committed fmt_check.log

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* Update flow_status.result

del eof trailing blank line as per review

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test(flow): restore runner-generated trailing blank line for sqlness

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* postgres: include SHOW FLOW STATUS in extended-query describe (return flow_statistics fields)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: address reviewer feedback on flow_statistics PR

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): resolve merge conflicts with main

Signed-off-by: polar <palakjha916@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* sqlness check post gen (information_schema.result)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): record start_time after req/snapshot_seqs built, before dispatch

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(sql): handle ShowFlowStatus in match statement at util.rs

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: review patch implementation

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore: remove accidentally committed local tool output files and fix fmt

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix worker.rs return type formatting

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): apply rustfmt to get_full_flow_stat return type

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix worker.rs return type formatting

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): re-apply rustfmt to get_full_flow_stat return type after merge

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: merge conflicts

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(auth): warn when credential load disables Postgres SCRAM or drops a line (#8652)

* fix(auth): warn when credential load disables Postgres SCRAM or drops a line

Static and watch user providers degraded silently in two ways:

- A single non-SCRAM verifier (mysql_native_password, or a legacy
  pbkdf2_sha256 hash that predates SCRAM) disables Postgres SCRAM for
  every user and falls back to cleartext, with no signal to the operator.
- A malformed credential line (commonly a plaintext password containing
  '=', which splits into more than two parts) was dropped without a trace.

Emit a warning at each credential load for both cases so operators don't
unknowingly serve cleartext passwords over Postgres or lose a user. This
is logging only; authentication behavior is unchanged. The SCRAM check
never logs secrets, and the malformed-line warning logs the line number
and file, never the line content.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): warn on credential file read error before truncating

A read error from lines() (I/O failure or invalid UTF-8) ends the
iterator via map_while, silently dropping every remaining credential.
Warn with the line number and file before truncating, matching the
malformed-line handling, so the drop is observable.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test(object-store): fix racy SecureFs abort test (#8720)

test_writer_abort_is_unsupported_without_atomic_write asserted the file
content immediately after abort() returned Unsupported. SecureFsWriter
writes through tokio::fs::File, whose write_all() only enqueues a blocking
write task (tokio's poll_write returns Ready before the write completes),
so the data may not be visible yet when the test reads the file. Drop the
race-prone content assertion and only verify the Unsupported contract.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(query): plan native histogram functions (#8705)

* feat(query): plan native histogram functions

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue & add tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* perf(promql): avoid repeated scans in sliding range evaluation (#8646)

* perf(promql): use two pointers for sliding range boundaries

Replace the stale cursor heuristic in RangeManipulateStream::calculate_range
with monotonic left/right cursors. The old path rescanned each evaluation
window (O(E x samples-per-window)) and could lose valid samples after sparse
gaps or trailing empty windows. The two pointers keep strict monotonic
progress, reducing boundary generation to O(N + E) while preserving
(curr-range, curr] semantics, start/end shortening, and empty-window output.

Controlled release benchmarks (fixed CPU, ABBA):
- Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s,
  ~96% at 1h/15s.
- Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end;
  shorter windows stayed within run-order noise.

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): specialize changes/resets with adaptive edge counting

The generic range_fn macro slices, downcasts, and rescans every overlapping
window for changes() and resets(). Replace the macro path for these two
functions with hand-written UDF wrappers backed by a shared private
edge-count kernel: direct raw-offset scans when requested edges are few,
otherwise one global u64 edge prefix so each window is answered by a prefix
difference.

Behavior is preserved bit-for-bit, including raw null-buffer values, NaN
semantics, signed zero, infinities, empty/singleton windows, independent
timestamp/value offsets, arbitrary window layouts, and exact DataFusion
error messages. The shared proc macro, planner, serializer, and other range
functions are untouched.

Controlled release benchmarks (fixed CPU, ABBA):
- Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time.
- Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster.
- Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client
  and 12.0-20.9% server latency improvement; controls stayed within drift.

Signed-off-by: discord9 <discord9@163.com>

* ci(query-regression): include PromQL range boundary case in defaults

An audit of historical query-regression runs found zero range-query
coverage: all 208 PromQL ANALYZE samples were bare selectors, so range
evaluation could regress without CI noticing. Wire the
promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so
label-triggered runs measure the range path. The case is cheap: a ~0.3s
synthetic fixture and about a minute of query execution per base/candidate
pass.

Signed-off-by: discord9 <discord9@163.com>

* chore(promql): address sliding range review nits

Move test-only imports into their test modules and remove the unused
pre-specialization changes and resets helpers.

Signed-off-by: discord9 <discord9@163.com>

* style(promql): apply pinned rustfmt

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover sparse range results

Share the changes and resets test scaffolding while keeping their behavior
oracles independent. Add an end-to-end sqlness regression for sparse samples,
empty intermediate windows, and a valid trailing sample.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* ci: optimize fuzz and split workflows (#8710)

* ci: batch fuzz targets in GitHub Actions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: improve fuzz test observability

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): preserve fuzz setup failure artifacts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(ci): keep fuzz mock output in logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: optimize fuzz worker cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: warm fuzz target binaries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: isolate fuzz workflow

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: centralize fuzz target preparation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: split general workflows

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: streamline docs required checks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: transfer fuzz targets as artifacts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve fuzz binary permissions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: streamline fuzz workers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: cache PR build dependencies

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: retain main build cache policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: address fuzz review feedback

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: add public constructor for compactor (#8724)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(logging): add enable_file_logging option to disable file logging (#8721)

Signed-off-by: xhwhis <hi@whis.me>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* avoid cloning final Prometheus remote write row (#8733)

perf: avoid cloning final Prometheus remote write row

Signed-off-by: lyang24 <lanqingy93@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(function): add json_object_keys scalar function (#8722)

Expose JSON object key listing for outermost objects, with sqlness coverage.

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor(mito2): revise compaction trigger behavior (#8706)

* refactor(mito2): revise compaction trigger behavior

Distinguish automatic and manual triggers, coalesce explicit automatic follow-ups, and reject concurrent manual compactions.

Remove implicit post-execution continuation and transient idle statuses so scheduler entries always represent an active lifecycle.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): track automatic compaction follow-ups

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): fix compaction transition rustdoc

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): mark manual compaction conflict retryable

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor(mito2): drop unused RequestCancelResult::NotRunning variant

request_cancel is only called in tests where the region is guaranteed to be
running, so the NotRunning case was dead code. Simplify to unwrap() and
remove the variant.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): gate test-only cancellation import

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): prioritize DDL after compaction planning

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: update dashboard to v0.13.11 (#8737)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(object-store): skip removed-entry lister test on Windows (#8735)

DirEntry on Windows is a snapshot from FindFirstFileW: file_type() and
metadata() keep returning cached data after the file is removed, so
read_list_entry() cannot observe the deletion. The test asserts the
Unix behavior (lstat returns ENOENT) and fails deterministically on
Windows nightly CI (4/4 tries). Gate it with #[cfg(not(windows))].

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(query): preserve remote dynamic filter target (#8615)

* fix(query): preserve remote dynamic filter target

Signed-off-by: discord9 <discord9@163.com>

* fix(query): check RDF subscriber registration

Signed-off-by: discord9 <discord9@163.com>

* fix(query): refresh initial dyn filter snapshot before dispatch and handle RDF unregister

The remote dynamic filter dispatch ordering regression: freeze the target,
pre-register subscribers, refresh the initial snapshot, then dispatch.
Also implement handle_remote_dyn_filter_unregister to keep unregister
targets consistent with do_get/update.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): update test-only RegionQueryHandler impl to new trait signatures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* docs: rework README release badges, drop star history, fix grpc flag (#8743)

* docs: show stable, latest and nightly version badges in README

The single release badge rendered whatever GitHub considered newest, so a
pre-release such as v1.2.0-beta.1 looked like the recommended version.

Split it into three self-updating badges using the shields.io `filter`
parameter, keyed off the existing tag naming:

- stable: `!*-*` matches tags without a hyphen (v1.1.4)
- latest: `!*-*-*` excludes nightly and dev builds (v1.2.0-beta.1)
- nightly: `*-nightly-*` matches the weekly build (v1.2.0-nightly-20260706)

No workflow changes are needed; the badges track new releases on their own.
A one-line caption below them says which channel to pick. The release-date
badge is dropped as the three version badges already carry that signal.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: remove star history chart from README

The chart carried a sealed_token in three URLs and added a large
third-party image to the Project Status section without saying anything
the badges and case studies do not already cover.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: use --grpc-bind-addr in README quickstart

--rpc-bind-addr is now only a hidden alias of --grpc-bind-addr and no
longer shows up in --help.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* Update README.md

Co-authored-by: Ning Sun <classicning@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Ning Sun <classicning@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore: check enterprise-gated files are listed in both license configs (#8750)

* chore: check enterprise-gated files are listed in both license configs

A file reachable only through `#[cfg(feature = "enterprise")] mod ...;` is
governed by the GreptimeDB Enterprise License, so it must appear in the
`includes` of licenserc-enterprise.toml and the `excludes` of licenserc.toml.
hawkeye stays silent when it does not: the file keeps its Apache-2.0 header and
passes the default check precisely because it was never excluded from it.

scripts/check-enterprise-license.py walks enterprise-gated `mod` declarations,
resolves them to files (submodules included) and diffs that set against both
configs, also reporting stale entries. It runs in the license job in CI and as
`make check-enterprise-license`.

Documents the split it cannot decide for you — whole enterprise features get
their own file, a gated match arm stays inline — in
.agents/architecture-invariants.md.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: tighten enterprise license checks

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(operator): invalidate local cache after dropping view (#8748)

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore!: gate soft-drop table behind the enterprise feature (#8747)

* chore: gate soft-drop table behind the enterprise feature

Soft-drop table becomes an enterprise-only feature:

- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
  non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
  without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
  registration, and information_schema.recycle_bin registration are
  compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
  unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
  example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
  (moving to the enterprise repo); affected information_schema .result
  files are regenerated

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: limit unused_variables allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: include the config key in the soft-drop enterprise gate error

Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: limit unused_mut allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: reject soft-drop DDL submissions in non-enterprise builds

Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: stop --enable-gc from enabling soft drop in the sqlness template

Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate fresh soft-drop procedures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: gate soft-drop fallback coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate soft-drop procedure implementation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate drop table soft-drop behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate expired soft-drop gc behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: test enterprise table ddl lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark purge_table as enterprise licensed

The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark recycle_bin as enterprise licensed

The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark soft-drop procedure sources as enterprise licensed

The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: make frontend heartbeat extensible and lifecycle-safe (#8726)

* feat: make frontend heartbeat extensible and lifecycle-safe

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: isolate heartbeat extension response handlers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: cancel in-flight heartbeat response handling

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover heartbeat wire compatibility

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up failed heartbeat startup

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address frontend heartbeat review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(meta): release region guards after drop rollback (#8751)

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor!: move native histogram config and `prom_validation_mode` to prom_store (#8744)

* chore: adjust the position of experimental_enable_prometheus_native_histogram

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: move prom_validation_mode as well

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: add health-aware gRPC client routing (#8684)

* feat: add gRPC client health routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: harden gRPC client health routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: defer gRPC client health checks until first use

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(mito2): discard unflushed region data safely (#8600)

* feat: support discarding unflushed region data

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): wake stalled writers after discard

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): drop redundant manifest check for discarding unflushed data

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* docs: align wal.sync_period documented default with actual fallback (5s) (#8753)

The example TOMLs and generated config.md documented the default of
wal.sync_period as "10s", but since #5677 moved the WAL sync task to a
background RepeatedTask, an unset sync_period falls back to 5s in
RaftEngineLogStore. The two paths therefore had different fsync
periods: deployments based on the example configs used 10s while bare
configs used 5s.

Align the documentation with the actual code behavior (5s) instead of
changing the code fallback to 10s, so that no existing deployment
silently gets a larger data-loss window on host power loss.

- config/datanode.example.toml, config/standalone.example.toml: 10s -> 5s
- config/config.md: regenerated via make config-docs
- src/cmd/tests/load_config_test.rs: update assertions accordingly

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: support Utf8View labels in Prometheus response (#8754)

Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(query): validate merge scan remote schema (#8579)

* fix(query): validate merge scan remote schema

Signed-off-by: discord9 <discord9@163.com>

* fix(query): treat JSON columns as schema-compatible across wire/decode forms

CI (Sqlness json2_limit standalone + distributed) failed on the new
remote-schema validation: a JSON column is Binary + extension metadata
(ARROW:extension:name=greptime.json, greptime:type=Json) on the wire but
decodes to Struct(...) with the extension metadata — validate_remote_schema
compared raw arrow data_type and rejected it as a mismatch.

Adds json_fields_compatible(): JSON fields are equal when name and
nullability match, greptime:type matches, and the JSON2 settings
(ARROW:extension:metadata type hints) match, ignoring the physical arrow
type. Only JSON fields may bypass the raw-type comparison; non-JSON
validation stays strict.

Adds 4 regression tests mirroring the CI failure (wire-binary vs
decoded-struct accepted both directions; different JSON2 settings
rejected; JSON vs plain Binary rejected).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(grafana): add events dashboard (#8725)

* feat(grafana): add events dashboard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): tolerate evolving event schemas

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): address events dashboard review

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): restore events dashboard panels

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): bound events dashboard queries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): include historical event catalogs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): preserve events drill-down context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): correct events lifecycle outcomes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): handle empty event type ranges

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): scope event catalogs to submissions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): handle empty events dashboard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): refresh event schema variables

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor: separate a json2 extension type (#8745)

Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: add admin function registrar (#8762)

* feat: add admin function registrar

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject admin function name collisions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: fix typo in admin function test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: information_schema.rs table initialization issue

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* rustfmt fix

Signed-off-by: onepizzateam <palakjha916@gmail.com>

---------

Signed-off-by: onepizzateam <palakjha916@gmail.com>
Signed-off-by: Palak Jha <palakjha916@gmail.com>
Signed-off-by: polar <palakjha916@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: xhwhis <hi@whis.me>
Signed-off-by: lyang24 <lanqingy93@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: Whis Liao <xhwhis@gmail.com>
Co-authored-by: Lanqing Yang <lanqingy93@gmail.com>
Co-authored-by: sun <sunchang_long@163.com>
Co-authored-by: Ning Sun <classicning@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
2026-08-06 07:24:59 +00:00
discord9 5f6b88b782 fix(flow): bind scheduled now in dist plan (#8389)
Signed-off-by: discord9 <discord9@163.com>
2026-07-01 10:56:30 +00:00
discord9 f6f56e6bc5 fix!: fence scoped flow repair snapshots (#8277)
* fix: fence scoped flow repair snapshots

Signed-off-by: discord9 <discord9@163.com>

* test: trim duplicate flow task tests

Signed-off-by: discord9 <discord9@163.com>

* test: moreless

Signed-off-by: discord9 <discord9@163.com>

* refactor: simplify flow query failure fallback

Signed-off-by: discord9 <discord9@163.com>

* test: update need eval interval

Signed-off-by: discord9 <discord9@163.com>

* docs: for helper fn

Signed-off-by: discord9 <discord9@163.com>

* chore: less test

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* test: update for eval interval

Signed-off-by: discord9 <discord9@163.com>

* fix: consume dirty windows after successful query

Signed-off-by: discord9 <discord9@163.com>

* test: rm useless tests

Signed-off-by: discord9 <discord9@163.com>

* test: standalone seq&rm dead if

Signed-off-by: discord9 <discord9@163.com>

* chore: move to pending window instead

Signed-off-by: discord9 <discord9@163.com>

* chore: mark full also call abandon_fenced_repair

Signed-off-by: discord9 <discord9@163.com>

* chore: instant for pending fenced repair

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-12 13:27:12 +00:00
discord9 d304df6e75 fix: run eval-interval flow without time window (#8231)
* fix: run eval-interval flow without time window

Signed-off-by: discord9 <discord9@163.com>

* test: cover eval-interval flow join query

Signed-off-by: discord9 <discord9@163.com>

* fix: address eval interval flow review comments

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-03 13:06:22 +00:00
discord9 7f76ad5439 feat: validate batching flow sink schema on create (#8176)
* feat: check schema on create

Signed-off-by: discord9 <discord9@163.com>

* chore: update sqlness

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): avoid duplicate fields when matching sink schema

Signed-off-by: discord9 <discord9@163.com>

* fix: null handling

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: debug log

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-02 08:24:50 +00:00
discord9 28fd796f4e fix(flow): harden incremental read correctness (#8196)
* fix(flow): harden incremental read correctness

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): propagate dirty window options

Signed-off-by: discord9 <discord9@163.com>

* test: more

Signed-off-by: discord9 <discord9@163.com>

* chore: test config api

Signed-off-by: discord9 <discord9@163.com>

* refactor: split gen

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* fix: allowlist key

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-01 02:48:00 +00:00
discord9 ba15a9c056 feat: support pending flow metadata with defer_on_missing_source (#8124)
* feat: support defer_on_missing_source for pending flow creation

Add `defer_on_missing_source` flow option that allows creating flows
even when source tables do not yet exist. The flow enters a pending
state and is automatically activated when source tables become available.

Key changes:
- New `FlowStatus::PendingSources` and fields in `FlowInfoValue` for
  unresolved source table names and last activation error
- `defer_on_missing_source` create-time-only option: stripped from
  runtime/flownode `CreateRequest` but preserved in metadata for
  SQL round-trip (`SHOW CREATE FLOW`, `information_schema.flows`)
- `CreateFlowProcedure` creates pending metadata when sources are
  missing and `defer_on_missing_source=true`; falls back to
  `FlowType::Batching` for missing-source flows
- `PendingFlowReconcileManager` in meta-srv periodically checks
  pending flows and activates them when source tables resolve
- `ActivatePendingFlowProcedure` handles activation: allocates peers,
  creates flows on flownodes, updates metadata, invalidates cache
- `OR REPLACE` properly handles pending<->active transitions,
  including peer allocation and flownode flow teardown
- `FlowMetadataAllocator::alloc_peers` for peer allocation at
  activation time
- Validated flow options: only `defer_on_missing_source` allowed;
  unknown options rejected
- Known issue: standalone mode does not support flownodes, so
  pending flow flush/sink behavior covered only in distributed
  sqlness; operator and meta unit tests cover activation logic

Tests:
- operator `determine_flow_type_for_source_state` (3 passed)
- common-meta `create_flow` (19 passed) including replacement
- common-meta `activate_flow` (4 passed)
- meta-srv `flow` (11 passed)
- sqlness: `flow_pending` covers create/replace/round-trip

Signed-off-by: discord9 <discord9@163.com>

* chore: simplify pending flow PR scope

Reduce PR #8124 to the metadata-only MVP after complexity review.

Changes:
- Remove automatic activation procedure and meta-srv reconcile wiring
- Remove activation tests and activation-only metadata fields
- Reject cross-state pending<->active `OR REPLACE` transitions for now
- Keep pending metadata creation and SQL round-trip behavior
- Allow `DROP FLOW` for pending flows without routes
- Reduce flow_pending sqlness to metadata/round-trip/drop coverage only

Deferred follow-ups are documented locally in `.tmp/tasks/pending-defer-semantics/deferred-followups.md` and intentionally not committed.

Tests:
- `cargo test -p operator determine_flow_type_for_source_state`
- `cargo test -p common-meta create_flow`
- `cargo test -p common-meta drop_flow`
- `cargo sqlness bare --test-filter flow_pending --bins-dir /mnt/nvme_rust/rust-targets/pending_defer/debug`

Signed-off-by: discord9 <discord9@163.com>

* test: cover pending flow metadata edge cases

Signed-off-by: discord9 <discord9@163.com>

* test: fix pending flow metadata test lint

Signed-off-by: discord9 <discord9@163.com>

* docs: document pending flow metadata fields

Signed-off-by: discord9 <discord9@163.com>

* chore: more sleep when test

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-05-29 06:59:21 +00:00
discord9 eccd97b5c7 feat(flow): support incremental read checkpoints (#8179)
* feat: flownode inc mode

Signed-off-by: discord9 <discord9@163.com>

* chore: rename fallback reason

Signed-off-by: discord9 <discord9@163.com>

* fix: harden flow incremental checkpoints

Signed-off-by: discord9 <discord9@163.com>

* fix: address flow watermark lint

Signed-off-by: discord9 <discord9@163.com>

* fix: address flow clippy

Signed-off-by: discord9 <discord9@163.com>

* refactor: clarify incremental plan preparation

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* test: more sqlness test

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-05-28 09:31:46 +00:00
discord9 d2d256909f feat(flow): parse defer on miss src table (#7980)
* feat: parse create flow with

Signed-off-by: discord9 <discord9@163.com>

* feat: validate after parse

Signed-off-by: discord9 <discord9@163.com>

* pcr

Signed-off-by: discord9 <discord9@163.com>

* chore: sqlness

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-04-27 03:02:13 +00:00
yxrxy 6c924efc2f fix: remove redundant error messages in admin functions (#7953)
Closes #7938

Signed-off-by: yxrxy <yxrxytrigger@gmail.com>
2026-04-17 08:21:22 +00:00
discord9 3beb538aa8 fix: rm useless analyzer (#7797)
* fix: rm useless analyzer

Signed-off-by: discord9 <discord9@163.com>

* test: rm related test

Signed-off-by: discord9 <discord9@163.com>

* test: flow tql avg

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-03-12 10:53:47 +00:00
b2074e3863 chore: upgrade DataFusion family, again (#7578)
* chore: upgrade DataFusion family

Signed-off-by: luofucong <luofc@foxmail.com>

* chore: switch to released version of datafusion-pg-catalog

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-03-03 07:36:39 +00:00
Ning Sun bcfbd01582 fix: use full DDL of flow in information_schema.flows.flow_definition (#7704)
* fix: use full DDL of flow in information_schema.flows.flow_definition

* fix: add schema name in sink table
2026-02-12 00:09:40 +00:00
Ruihang Xia 77013d9085 feat: report flow stats from streaming and batching engines (#7701)
* fix: report flow stats from streaming and batching engines

* handle restart report handler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename fn name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-02-11 07:38:09 +00:00
discord9 a6e69bc180 feat: flow last non null (#7646)
* feat: flow last non null support

Signed-off-by: discord9 <discord9@163.com>

* clippy

Signed-off-by: discord9 <discord9@163.com>

* test: report error when sink is not last non null

Signed-off-by: discord9 <discord9@163.com>

* fix: error if query column matches nothing if partial

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-01-30 12:34:33 +00:00
564cc0c750 feat: table/column/flow COMMENT (#7060)
* initial impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sqlness test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* avoid unimplemented panic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* validate flow

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix table column comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* table level comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify table info serde

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* don't txn

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove empty trait

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* wip: procedure

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update proto

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* grpc support

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* try from pb struct

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* doc comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* check unchanged fast case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tune errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix merge error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use try_as_raw_value

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
2025-12-10 15:08:47 +00:00
dennis zhuangandCopilot 1f91422bae feat!: improve mysql/pg compatibility (#7315)
* feat(mysql): add SHOW WARNINGS support and return warnings for unsupported SET variables

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(function): add MySQL IF() function and PostgreSQL description functions for connector compatibility

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: show tables for mysql

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: partitions table in information_schema and add starrocks external catalog compatibility

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: async udf

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: set warnings

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: impl pg_my_temp_schema and make description functions simple

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: add test for issue 7313

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: apply suggestions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: partition_expression and partition_description

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: test

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: unit tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: saerch_path only works for pg

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: improve warnings processing

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: warnings while writing affected rows and refactor

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: improve ShobjDescriptionFunction signature

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: array_to_boolean

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-01 20:41:14 +00:00
discord9 8452a9d579 feat(flow): add eval interval option (#6623)
* feat: add flow eval interval

Signed-off-by: discord9 <discord9@163.com>

* feat: tql flow must have eval interval

Signed-off-by: discord9 <discord9@163.com>

* chore: clippy

Signed-off-by: discord9 <discord9@163.com>

* test: update sqlness

Signed-off-by: discord9 <discord9@163.com>

* wip

Signed-off-by: discord9 <discord9@163.com>

* wip

Signed-off-by: discord9 <discord9@163.com>

* feat: check for now func

Signed-off-by: discord9 <discord9@163.com>

* refactor: use ms instead

Signed-off-by: discord9 <discord9@163.com>

* fix: not panic&proper simplifier

Signed-off-by: discord9 <discord9@163.com>

* test: update to fix

Signed-off-by: discord9 <discord9@163.com>

* feat: not allow month in interval

Signed-off-by: discord9 <discord9@163.com>

* test: update remov months

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: after rebase fix

Signed-off-by: discord9 <discord9@163.com>

* feat: use seconds and add to field instead

Signed-off-by: discord9 <discord9@163.com>

* chore: aft rebase fix

Signed-off-by: discord9 <discord9@163.com>

* fix: add check for month

Signed-off-by: discord9 <discord9@163.com>

* chore: fmt

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: rm clone per review

Signed-off-by: discord9 <discord9@163.com>

* chore: update proto

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2025-08-27 09:44:32 +00:00
discord9andLei, HUANG 9b5baa965c feat: truly limit time range by split window (#6295)
* feat: actually split window to limit time range

feat: truly limit time range by split window

Update src/flow/src/batching_mode/state.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>

* chore: added stalled time window range

Signed-off-by: discord9 <discord9@163.com>

* fix: not flush all time range as too expensive

Signed-off-by: discord9 <discord9@163.com>

* test: make it more robust

Signed-off-by: discord9 <discord9@163.com>

* what

Signed-off-by: discord9 <discord9@163.com>

* feat: denfensively handle surplus

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review,explain flush flow

Signed-off-by: discord9 <discord9@163.com>

* chore: per bugbot

Signed-off-by: discord9 <discord9@163.com>

* fix: a temp fix to make mirror insert go first(still need better fix to sync with mirror insert that happens before

Signed-off-by: discord9 <discord9@163.com>

* chore: add todo

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2025-07-04 03:37:43 +00:00
Ruihang Xia 2b0f27da51 feat: don't allow creating flow with the same sink and source table (#6435)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-07-01 11:33:09 +00:00
Lei, HUANG 69870e2762 fix(mito): use 1day as default time partition duration (#6202)
* fix unit tests

* fix: sqlness

* fix/default-time-window:
 ## Add Helper Functions and Enhance Compaction Tests

 - **Refactor Compaction Logic**: Introduced helper functions `flush` and `compact` in `compaction_test.rs` to streamline compaction operations.
 - **Enhance Compaction Tests**: Added a new test `test_infer_compaction_time_window` in `compaction_test.rs` to verify compaction time window inference.
 - **Testing Improvements**: Added `#[cfg(test)]` attribute to `new_multi_partitions` in `time_partition.rs` to ensure it's only included in test builds.

* fix/default-time-window:
 - **Refactor `TimePartition` Struct**: Removed unnecessary comments regarding `time_range` in `time_partition.rs`.
 - **Enhance `TimePartitions` Functionality**: Added a method `part_duration_or_default` to provide a default partition duration in `time_partition.rs`.
 - **Update SQL Test Cases**: Modified SQL operations and expected results in `scan_big_varchar.result` and `scan_big_varchar.sql` to reflect changes in data manipulation logic.

* fix/default-time-window:
 ### Update Time Partition Default Duration

 - **Refactor Default Duration**: Introduced `INITIAL_TIME_WINDOW` constant to define the default time window duration as `Duration::from_days(1)`. This change replaces multiple instances of the hardcoded default duration across the `time_partition.rs` file.
 - **Files Affected**: `time_partition.rs`

* fix/default-time-window:
 ## Update Partition Duration Handling

 - **`time_partition.rs`**: Refactored `part_duration` to be non-optional, removing `Option` wrapper. Updated logic to use `unwrap_or` with `INITIAL_TIME_WINDOW` where necessary. Adjusted related methods and tests to accommodate this change.
 - **`version.rs` (memtable and region)**: Updated handling of `part_duration` to align with changes in `time_partition.rs`, ensuring consistent use of non-optional `Duration`.

* fix/default-time-window:
 ### Improve Error Context in `time_partition.rs`

 - Enhanced error context message in `time_partition.rs` to provide clearer information on partition time range issues, including bucket size details.

Signed-off-by: Lei, HUANG <lhuang@greptime.com>

---------

Signed-off-by: Lei, HUANG <lhuang@greptime.com>
2025-06-08 16:20:26 +00:00
discord9 cbafb6e00b feat(flow): flow streaming mode in list expr support (#6229)
* feat: flow streaming in list support

* chore: per review

* chore: per review

* fix: expr correct type
2025-06-04 08:05:20 +00:00
discord9 fc6300a2ba feat(flow): support prom ql(in tql) in flow (#6063)
* feat: support parse prom ql in create flow

* refactor

* fix: just run tql unmodified

* refactor: determine type faster

* fix: pass original query

* tests: sqlness

* test: fix format&chore

* fix: get raw query

* test: fix sqlness randomness

* chore: what's the box for?

* test: location_to_index

* test: make sqlness more determinstic

* fix: tmp add sleep 1s after flush_flow

* undo test sleep 1s&rm done todo

* chore: more tests
2025-05-22 03:06:09 +00:00
discord9 b0ad3f0bb4 fix: force streaming mode for instant source table (#6031)
* fix: force streaming mode for instant source table

* tests: sqlness test&refactor: get table

* refactor: per review
2025-05-01 08:43:06 +00:00
discord9 bdd44fd7ec chore: only retry when retry-able in flow (#5987)
* chore: only retry when retry-able

* chore: revert dbg change

* refactor: per review

* fix: check for available frontend first

* docs: more explain&longer timeout&feat: more retry at every level&try send select 1

* fix: use `sql` method for "SELECT 1"

* fix: also put recover flows in spawned task and a dead loop

* test: update transient error in flow rebuild test

* chore: sleep after sqlness sleep

* chore: add a warning

* chore: wait even more time after reboot
2025-04-28 09:49:49 +00:00
Yingwen 86aae6733d fix: prune primary key with multiple columns may use default value as statistics (#5996)
* test: incorrect test result when filtering pk with multiple columns

* fix: prune non first tag correctly

Distinguish no column and no stats and only use default value when no
column

* test: update test result

* refactor: rename test file

* test: add test for null filter

* fix: use StatValues for null counts

* test: drop table

* test: fix unstable flow test
2025-04-28 04:53:30 +00:00
discord9 21044c7339 feat: uddsketch_merge udaf (#5992) 2025-04-27 12:43:21 +00:00
discord9anddennis zhuang 66e2242e46 fix: conn timeout&refactor: better err msg (#5974)
* fix: conn timeout&refactor: better err msg

* chore: clippy

* chore: make test work

* chore: comment

* todo: fix null cast

* fix: retry conn&udd_calc

* chore: comment

* chore: apply suggestion

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2025-04-25 19:12:30 +00:00
discord9 a0900f5b90 feat(flow): use batching mode&fix sqlness (#5903)
* feat: use flow batching engine

broken: try using logical plan

fix: use dummy catalog for logical plan

fix: insert plan exec&sqlness grpc addr

feat: use frontend instance in flownode in standalone

feat: flow type in metasrv&fix: flush flow out of sync& column name alias

tests: sqlness update

tests: sqlness flow rebuild udpate

chore: per review

refactor: keep chnl mgr

refactor: use catalog mgr for get table

tests: use valid sql

fix: add more check

refactor: put flow type determine to frontend

* chore: update proto

* chore: update proto to main branch

* fix: add locks for create/drop flow&docs: update docs

* feat: flush_flow flush all ranges now

* test: add align time window test

* docs: explain `nodeid` use in check task

* refactor: AddAutoColumnRewriter check for Projection

* refactor: per review

* fix: query without time window also clean dirty time window

* chore: better logging

* chore: add comments per review

* refactor: per review

* chore: per review

* chore: per review rename args

* refactor: per review partially

* chore: update docs

* chore: use better error variant

* chore: better error variant

* refactor: rename FlowWorkerManager to FlowStreamingEngine

* rename again

* refactor: per review

* chore: rebase after #5963 merged

* refactor: rename all flow_worker_manager occurs

* docs: rm resolved TODO
2025-04-23 15:12:16 +00:00
discord9 79ed7bbc44 fix: store flow query ctx on creation (#5963)
* fix: store flow schema on creation

* chore: update sqlness

* refactor: save the entire query context to flow info

* chore: sqlness update

* chore: rm pub

* fix: keep old version compatibility
2025-04-23 09:59:09 +00:00
discord9 3aebfc1716 test: looser condition (#5816) 2025-04-02 07:38:05 +00:00
discord9 5ad2d8b3b8 fix: handle nullable default value (#5747)
* fix: handle nullable default value

* chore: update sqlness
2025-03-24 02:38:26 +00:00
b8070adc3a feat: enhancement information_schema.flows (#5623)
* feat: enhancement information_schema.flows

* feat: enhancement information_schema.flows

* u

* u

* u

* u

* u

* u

* u

* u

* u

* update

* update

* update

* delete unused code

* u

* u

* Update src/flow/src/adapter/worker.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update src/common/meta/src/key/flow/flow_state.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update src/common/meta/src/key/flow/flow_info.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update src/common/meta/src/key/flow/flow_state.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update src/common/meta/src/key/flow/flow_info.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* u

* u

* u

* u

* u

* u

* chore: fix sqlness

* chore: update proto

* fix: remove date time

* fix: update result of information_schema test

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: discord9 <discord9@163.com>
2025-03-11 15:08:10 +00:00
Ruihang Xiaandjeremyhi 8026b1d72c feat!: unify all index creation grammars (#5486)
* column options

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle table constrain

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test assertions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change inverted index table constrain usage

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* don't create inverted index for pk on alter table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove remaining pk-as-inverted-index

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* more inverted index magic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/sql/src/statements.rs

Co-authored-by: jeremyhi <jiachun_feng@proton.me>

* drop support for index def in table constrain

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
2025-02-12 06:54:09 +00:00
yihong adb5c3743c fix: flush table panic when table has interval column close #3235 (#5422)
* fix: flash table panic when table has interval column close #3235

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* Revert "fix: flash table panic when table has interval column close #3235"

This reverts commit ffc63efda39cd6ef525313b60ede061c5ec24b12.

* fix: create table do not support interval type for now close #3235

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: sqlness

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments fix conflict and more tests

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address final comments drop useless sqlness tests

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-01-25 08:09:55 +00:00
35b635f639 feat!: Bump datafusion, prost, hyper, tonic, tower, axum (#5417)
* change dep

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: adapt to arrow's interval array

* chore: fix compile errors in datatypes crate

* chore: fix api crate compiler errors

* chore: fix compiler errors in common-grpc

* chore: fix common-datasource errors

* chore: fix deprecated code in common-datasource

* fix promql and physical plan related

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* wip: upgrading network deps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* block on updating `sqlparser`

* upgrade sqlparser

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adapt new df's trait requirements

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fix compiler errors in mito2

* chore: fix common-function crate errors

* chore: fix catalog errors

* change import path

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fix some errors in query crate

* chore: fix some errors in query crate

* aggr expr and some other tiny fixes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fix expr related errors in query crate

* chore: fix query serializer and admin command

* chore: fix grpc services

* feat: axum serve

* chore: fix http server

* remove handle_error handler
* refactor timeout layer
* serve axum

* chore: fix flow aggr functions

* chore: fix flow

* feat: fix errors in meta-srv

* boxed()
* use TokioIo

* feat!: Remove script crate and python feature (#5321)

* feat: exclude script crate

* chore: simplify feature

* feat: remove the script crate

* chore: remove python feature and some comments

* chore: fix warning

* chore: fix servers tests compiler errors

* feat: fix tests-integration errors

* chore: fix unused

* test: fix catalog test

* chore: fix compiler errors for crates using common-meta

testing feature is enabled when check with --workspace

* test: use display for logical plan test

* test: implement rewrite for ScanHintRule

* fix: http server build panic

* test: fix mito test

* fix: sql parser type alias error

* test: fix TestClient not listen

* test: some flow tests

* test(flow): more fix

* fix: test_otlp_logs

* test: fix promql test that using deprecated method fun()

* fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64

* test: fix infer schema test case

* test: fix tests related to plan display

* chore: fix last flow test

* test: fix function format related assertion

* test: use larger port range for tests

* fix: test_otlp_traces

* fix: test_otlp_metrics

* fix range query and dist plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: flow handle distinct use deprecated field

* fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs

* test: fix deserialize test

* test: reduce split key case num

* tests: lower case aggr func name

* test: fix some sqlness tests

* tests: more sqlness fix

* tests: fixed sqlness test

* commit non-bug changes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: make our udf correct

* fix: implement empty methods of ContextProvider for DfContextProviderAdapter

* test: update sqlness test result

* chore: remove unused

* fix: provide alias name for AggregateExprBuilder in range plan

* test: update range query result

* fix: implement missing ContextProvider methods for DfContextProviderAdapter

* test: update timestamps, cte result

* fix: supports empty projection in mito

* test: update comment for cte test

* fix: support projection for numbers

* test: update test cases after projection fix

* fix: fix range select first_value/last_value

* fix: handle CAST and time index conflict

* fix: handle order by correctly in range first_value/last_value

* test: update sqlness result

* test: update view test result

* test: update decimal test

wait for https://github.com/apache/datafusion/pull/14126 to fix this

* feat: remove redundant physical optimization

todo(ruihang): Check if we can remove this.

* test: update sqlness test result

* chore: range select default sort use nulls_first = false

* test: update filter push down test result

* test: comment deciaml test to avoid different panic message

* test: update some distributed test result

* test: update test for distributed count and filter push down

* test: update subqueries test

* fix: SessionState may overwrite our UDFs

* chore: fix compiler errors after merging main

* fix: fix elasticsearch and dashboard router panic

* chore: fix common-functions tests

* chore: update sqlness result

* test: fix id keyword and update sqlness result

* test: fix flow_null test

* fix: enlarge thread size in debug mode to avoid overflow

* chore: fix warnings in common-function

* chore: fix warning in flow

* chore: fix warnings in query crate

* chore: remove unused warnings

* chore: fix deprecated warnings for parquet

* chore: fix deprecated warning in servers crate

* style: fix clippy

* test: enlarge mito cache tttl test ttl time

* chore: fix typo

* style: fmt toml

* refactor: reimplement PartialOrd for RangeSelect

* chore: remove script crate files introduced by merge

* fix: return error if sql option is not kv

* chore: do not use ..default::default()

* chore: per review

* chore: update error message in BuildAdminFunctionArgsSnafu

Co-authored-by: jeremyhi <jiachun_feng@proton.me>

* refactor: typed precision

* update sqlness view case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: flow per review

* chore: add example in comment

* chore: warn if parquet stats of timestamp is not INT64

* style: add a newline before derive to make the comment more clear

* test: update sqlness result

* fix: flow from substrait

* chore: change update_range_context log to debug level

* chore: move axum-extra axum-macros to workspace

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: luofucong <luofc@foxmail.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
2025-01-23 06:15:40 +00:00
discord9 95b20592ac fix: handle insert default value (#5307)
* fix: handle flow inserts with default values

* test: sqlness

* chore: typo

* chore: newline

* feat(WIP): impure default filler

* feat: fill impure default values

* test: add test for default fill impure

* feat: check for impure

* fix: also handle stmt to region

* refactor: per review

* refactor: per review

* chore: rebase fix

* chore: clippy

* chore: per review
2025-01-14 09:06:53 +00:00
discord9andYingwen 353c8230db refactor: flow replace check&better error msg (#5277)
* chore: better error msg

* chore eof newline

* refactor: move replace check to flow worker

* chore: add ctx to insert flow failure

* chore: Update src/flow/src/adapter/flownode_impl.rs

* test: add order by for deterministic

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
2025-01-03 10:59:58 +00:00
discord9 d20b592fe8 fix: flow handle reordered inserts (#5275)
* fix: reorder correct schema

* tests: reorder insert handled correctly

* chore: rm unused

* refactor: per review

* chore: more comment

* chore: per review
2025-01-03 06:25:39 +00:00
discord9 4d6fe31fff fix(flow): flow's table schema cache (#5251)
* fix: flow schema cache

* refactor: location for `to_meta_err`

* chore: endfile emptyline

* chore: review(partially)

* chore: per review

* refactor: per review

* refactor: per review
2025-01-02 10:33:23 +00:00
discord9 5ba8bd09fb fix: flow compare null values (#5234)
* fix: flow compare null values

* fix: fix again ck ty before cmp

* chore: rm comment

* fix: handle null

* chore: typo

* docs: update comment

* refactor: per review

* tests: more sqlness

* tests: sqlness not show create table
2024-12-25 15:31:27 +00:00
discord9 abf34b845c feat(flow): check sink table mismatch on flow creation (#5112)
* tests: more mismatch errors

* feat: check sink table schema if exists&prompt nice err msg

* chore: rm unused variant

* chore: fmt

* chore: cargo clippy

* feat: check schema on create

* feat: better err msg when mismatch

* tests: fix a schema mismatch

* todo: create sink table

* feat: create sink table

* fix: find time index

* tests: auto created sink table

* fix: remove empty keys

* refactor: per review

* chore: fmt

* test: sqlness

* chore: after rebase
2024-12-25 13:42:37 +00:00
discord9 a4d61bcaf1 fix(flow): batch builder with type (#5195)
* fix: typed builder

* chore: clippy

* chore: rename

* fix: unit tests

* refactor: per review
2024-12-19 09:16:56 +00:00
discord9 2d6f63a504 feat: show flow's mem usage in INFORMATION_SCHEMA.FLOWS (#4890)
* feat: add flow mem size to sys table

* chore: rm dup def

* chore: remove unused variant

* chore: minor refactor

* refactor: per review
2024-12-19 08:24:04 +00:00
discord9 043d0bd7c2 test: flow rebuild (#5162)
* tests: rebuild flow

* tests: more rebuild

* tests: restart

* chore: drop clean
2024-12-16 12:25:23 +00:00
discord9 8b944268da feat: ttl=0/instant/forever/humantime&ttl refactor (#5089)
* feat: ttl zero filter

* refactor: use TimeToLive enum

* fix: unit test

* tests: sqlness

* refactor: Option<TTL> None means UNSET

* tests: sqlness

* fix: 10000 years --> forever

* chore: minor refactor from reviews

* chore: rename back TimeToLive

* refactor: split imme request from normal requests

* fix: use correct lifetime

* refactor: rename immediate to instant

* tests: flow sink table default ttl

* refactor: per review

* tests: sqlness

* fix: ttl alter to instant

* tests: sqlness

* refactor: per review

* chore: per review

* feat: add db ttl type&forbid instant for db

* tests: more unit test
2024-12-06 09:20:42 +00:00
discord9 d931389a4c fix(flow): minor fix about count(*)&sink keyword (#5061)
* fix: SiNk

* fix: sink&count(*)

* tests: SiNk

* refactor: per review
2024-11-29 03:06:27 +00:00
discord9 4d8fe29ea8 feat: CREATE OR REPLACE FLOW (#5001)
* feat: Replace flow

* refactor: better show create flow&tests: better check

* tests: sqlness result update

* tests: unit test for update

* refactor: cmp with raw bytes

* refactor: rename

* refactor: per review
2024-11-19 08:44:57 +00:00
discord9 35898f0b2e test: more sqlness tests for flow (#4988)
* tests: more flow testcase

* tests(WIP): more tests

* tests: more flow tests

* test: wired regex for sqlness

* refactor: put blog&example to two files
2024-11-14 07:40:14 +00:00
discord9 ff38abde2e chore: better column schema check for flow (#4855)
* chore: better column schema check for flow

* chore: better msg

* tests: clean up after tests

* chore: better msg

* chore: per review

* tests: sqlness
2024-10-24 09:43:32 +00:00