Palak Jha cabc2f6cc6 feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (#7987) (#8392)
* feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test(flow): add sqlness golden result for flow_status

Signed-off-by: Palak Jha <palakjha916@gmail.com>

* fix(catalog): remove unused OptionExt import in flow_statistics

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* docs(flow): fix stale 'recent errors' comment on QueryFlowExecStats

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test: regenerate golden results for flow_statistics table

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* style: apply rustfmt to flow_statistics changes

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor(catalog): hoist current_time_millis out of flow loop and clamp uptime

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore: remove accidentally committed fmt_check.log

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* Update flow_status.result

del eof trailing blank line as per review

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test(flow): restore runner-generated trailing blank line for sqlness

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* postgres: include SHOW FLOW STATUS in extended-query describe (return flow_statistics fields)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: address reviewer feedback on flow_statistics PR

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): resolve merge conflicts with main

Signed-off-by: polar <palakjha916@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* sqlness check post gen (information_schema.result)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): record start_time after req/snapshot_seqs built, before dispatch

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(sql): handle ShowFlowStatus in match statement at util.rs

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: review patch implementation

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore: remove accidentally committed local tool output files and fix fmt

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix worker.rs return type formatting

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): apply rustfmt to get_full_flow_stat return type

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix worker.rs return type formatting

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): re-apply rustfmt to get_full_flow_stat return type after merge

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: merge conflicts

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(auth): warn when credential load disables Postgres SCRAM or drops a line (#8652)

* fix(auth): warn when credential load disables Postgres SCRAM or drops a line

Static and watch user providers degraded silently in two ways:

- A single non-SCRAM verifier (mysql_native_password, or a legacy
  pbkdf2_sha256 hash that predates SCRAM) disables Postgres SCRAM for
  every user and falls back to cleartext, with no signal to the operator.
- A malformed credential line (commonly a plaintext password containing
  '=', which splits into more than two parts) was dropped without a trace.

Emit a warning at each credential load for both cases so operators don't
unknowingly serve cleartext passwords over Postgres or lose a user. This
is logging only; authentication behavior is unchanged. The SCRAM check
never logs secrets, and the malformed-line warning logs the line number
and file, never the line content.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): warn on credential file read error before truncating

A read error from lines() (I/O failure or invalid UTF-8) ends the
iterator via map_while, silently dropping every remaining credential.
Warn with the line number and file before truncating, matching the
malformed-line handling, so the drop is observable.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test(object-store): fix racy SecureFs abort test (#8720)

test_writer_abort_is_unsupported_without_atomic_write asserted the file
content immediately after abort() returned Unsupported. SecureFsWriter
writes through tokio::fs::File, whose write_all() only enqueues a blocking
write task (tokio's poll_write returns Ready before the write completes),
so the data may not be visible yet when the test reads the file. Drop the
race-prone content assertion and only verify the Unsupported contract.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(query): plan native histogram functions (#8705)

* feat(query): plan native histogram functions

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue & add tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* perf(promql): avoid repeated scans in sliding range evaluation (#8646)

* perf(promql): use two pointers for sliding range boundaries

Replace the stale cursor heuristic in RangeManipulateStream::calculate_range
with monotonic left/right cursors. The old path rescanned each evaluation
window (O(E x samples-per-window)) and could lose valid samples after sparse
gaps or trailing empty windows. The two pointers keep strict monotonic
progress, reducing boundary generation to O(N + E) while preserving
(curr-range, curr] semantics, start/end shortening, and empty-window output.

Controlled release benchmarks (fixed CPU, ABBA):
- Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s,
  ~96% at 1h/15s.
- Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end;
  shorter windows stayed within run-order noise.

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): specialize changes/resets with adaptive edge counting

The generic range_fn macro slices, downcasts, and rescans every overlapping
window for changes() and resets(). Replace the macro path for these two
functions with hand-written UDF wrappers backed by a shared private
edge-count kernel: direct raw-offset scans when requested edges are few,
otherwise one global u64 edge prefix so each window is answered by a prefix
difference.

Behavior is preserved bit-for-bit, including raw null-buffer values, NaN
semantics, signed zero, infinities, empty/singleton windows, independent
timestamp/value offsets, arbitrary window layouts, and exact DataFusion
error messages. The shared proc macro, planner, serializer, and other range
functions are untouched.

Controlled release benchmarks (fixed CPU, ABBA):
- Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time.
- Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster.
- Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client
  and 12.0-20.9% server latency improvement; controls stayed within drift.

Signed-off-by: discord9 <discord9@163.com>

* ci(query-regression): include PromQL range boundary case in defaults

An audit of historical query-regression runs found zero range-query
coverage: all 208 PromQL ANALYZE samples were bare selectors, so range
evaluation could regress without CI noticing. Wire the
promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so
label-triggered runs measure the range path. The case is cheap: a ~0.3s
synthetic fixture and about a minute of query execution per base/candidate
pass.

Signed-off-by: discord9 <discord9@163.com>

* chore(promql): address sliding range review nits

Move test-only imports into their test modules and remove the unused
pre-specialization changes and resets helpers.

Signed-off-by: discord9 <discord9@163.com>

* style(promql): apply pinned rustfmt

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover sparse range results

Share the changes and resets test scaffolding while keeping their behavior
oracles independent. Add an end-to-end sqlness regression for sparse samples,
empty intermediate windows, and a valid trailing sample.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* ci: optimize fuzz and split workflows (#8710)

* ci: batch fuzz targets in GitHub Actions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: improve fuzz test observability

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): preserve fuzz setup failure artifacts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(ci): keep fuzz mock output in logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: optimize fuzz worker cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: warm fuzz target binaries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: isolate fuzz workflow

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: centralize fuzz target preparation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: split general workflows

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: streamline docs required checks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: transfer fuzz targets as artifacts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve fuzz binary permissions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: streamline fuzz workers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: cache PR build dependencies

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: retain main build cache policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: address fuzz review feedback

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: add public constructor for compactor (#8724)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(logging): add enable_file_logging option to disable file logging (#8721)

Signed-off-by: xhwhis <hi@whis.me>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* avoid cloning final Prometheus remote write row (#8733)

perf: avoid cloning final Prometheus remote write row

Signed-off-by: lyang24 <lanqingy93@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(function): add json_object_keys scalar function (#8722)

Expose JSON object key listing for outermost objects, with sqlness coverage.

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor(mito2): revise compaction trigger behavior (#8706)

* refactor(mito2): revise compaction trigger behavior

Distinguish automatic and manual triggers, coalesce explicit automatic follow-ups, and reject concurrent manual compactions.

Remove implicit post-execution continuation and transient idle statuses so scheduler entries always represent an active lifecycle.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): track automatic compaction follow-ups

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): fix compaction transition rustdoc

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): mark manual compaction conflict retryable

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor(mito2): drop unused RequestCancelResult::NotRunning variant

request_cancel is only called in tests where the region is guaranteed to be
running, so the NotRunning case was dead code. Simplify to unwrap() and
remove the variant.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): gate test-only cancellation import

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): prioritize DDL after compaction planning

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: update dashboard to v0.13.11 (#8737)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(object-store): skip removed-entry lister test on Windows (#8735)

DirEntry on Windows is a snapshot from FindFirstFileW: file_type() and
metadata() keep returning cached data after the file is removed, so
read_list_entry() cannot observe the deletion. The test asserts the
Unix behavior (lstat returns ENOENT) and fails deterministically on
Windows nightly CI (4/4 tries). Gate it with #[cfg(not(windows))].

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(query): preserve remote dynamic filter target (#8615)

* fix(query): preserve remote dynamic filter target

Signed-off-by: discord9 <discord9@163.com>

* fix(query): check RDF subscriber registration

Signed-off-by: discord9 <discord9@163.com>

* fix(query): refresh initial dyn filter snapshot before dispatch and handle RDF unregister

The remote dynamic filter dispatch ordering regression: freeze the target,
pre-register subscribers, refresh the initial snapshot, then dispatch.
Also implement handle_remote_dyn_filter_unregister to keep unregister
targets consistent with do_get/update.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): update test-only RegionQueryHandler impl to new trait signatures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* docs: rework README release badges, drop star history, fix grpc flag (#8743)

* docs: show stable, latest and nightly version badges in README

The single release badge rendered whatever GitHub considered newest, so a
pre-release such as v1.2.0-beta.1 looked like the recommended version.

Split it into three self-updating badges using the shields.io `filter`
parameter, keyed off the existing tag naming:

- stable: `!*-*` matches tags without a hyphen (v1.1.4)
- latest: `!*-*-*` excludes nightly and dev builds (v1.2.0-beta.1)
- nightly: `*-nightly-*` matches the weekly build (v1.2.0-nightly-20260706)

No workflow changes are needed; the badges track new releases on their own.
A one-line caption below them says which channel to pick. The release-date
badge is dropped as the three version badges already carry that signal.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: remove star history chart from README

The chart carried a sealed_token in three URLs and added a large
third-party image to the Project Status section without saying anything
the badges and case studies do not already cover.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: use --grpc-bind-addr in README quickstart

--rpc-bind-addr is now only a hidden alias of --grpc-bind-addr and no
longer shows up in --help.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* Update README.md

Co-authored-by: Ning Sun <classicning@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Ning Sun <classicning@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore: check enterprise-gated files are listed in both license configs (#8750)

* chore: check enterprise-gated files are listed in both license configs

A file reachable only through `#[cfg(feature = "enterprise")] mod ...;` is
governed by the GreptimeDB Enterprise License, so it must appear in the
`includes` of licenserc-enterprise.toml and the `excludes` of licenserc.toml.
hawkeye stays silent when it does not: the file keeps its Apache-2.0 header and
passes the default check precisely because it was never excluded from it.

scripts/check-enterprise-license.py walks enterprise-gated `mod` declarations,
resolves them to files (submodules included) and diffs that set against both
configs, also reporting stale entries. It runs in the license job in CI and as
`make check-enterprise-license`.

Documents the split it cannot decide for you — whole enterprise features get
their own file, a gated match arm stays inline — in
.agents/architecture-invariants.md.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: tighten enterprise license checks

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(operator): invalidate local cache after dropping view (#8748)

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore!: gate soft-drop table behind the enterprise feature (#8747)

* chore: gate soft-drop table behind the enterprise feature

Soft-drop table becomes an enterprise-only feature:

- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
  non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
  without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
  registration, and information_schema.recycle_bin registration are
  compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
  unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
  example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
  (moving to the enterprise repo); affected information_schema .result
  files are regenerated

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: limit unused_variables allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: include the config key in the soft-drop enterprise gate error

Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: limit unused_mut allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: reject soft-drop DDL submissions in non-enterprise builds

Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: stop --enable-gc from enabling soft drop in the sqlness template

Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate fresh soft-drop procedures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: gate soft-drop fallback coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate soft-drop procedure implementation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate drop table soft-drop behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate expired soft-drop gc behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: test enterprise table ddl lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark purge_table as enterprise licensed

The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark recycle_bin as enterprise licensed

The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark soft-drop procedure sources as enterprise licensed

The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: make frontend heartbeat extensible and lifecycle-safe (#8726)

* feat: make frontend heartbeat extensible and lifecycle-safe

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: isolate heartbeat extension response handlers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: cancel in-flight heartbeat response handling

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover heartbeat wire compatibility

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up failed heartbeat startup

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address frontend heartbeat review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(meta): release region guards after drop rollback (#8751)

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor!: move native histogram config and `prom_validation_mode` to prom_store (#8744)

* chore: adjust the position of experimental_enable_prometheus_native_histogram

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: move prom_validation_mode as well

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: add health-aware gRPC client routing (#8684)

* feat: add gRPC client health routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: harden gRPC client health routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: defer gRPC client health checks until first use

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(mito2): discard unflushed region data safely (#8600)

* feat: support discarding unflushed region data

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): wake stalled writers after discard

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): drop redundant manifest check for discarding unflushed data

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* docs: align wal.sync_period documented default with actual fallback (5s) (#8753)

The example TOMLs and generated config.md documented the default of
wal.sync_period as "10s", but since #5677 moved the WAL sync task to a
background RepeatedTask, an unset sync_period falls back to 5s in
RaftEngineLogStore. The two paths therefore had different fsync
periods: deployments based on the example configs used 10s while bare
configs used 5s.

Align the documentation with the actual code behavior (5s) instead of
changing the code fallback to 10s, so that no existing deployment
silently gets a larger data-loss window on host power loss.

- config/datanode.example.toml, config/standalone.example.toml: 10s -> 5s
- config/config.md: regenerated via make config-docs
- src/cmd/tests/load_config_test.rs: update assertions accordingly

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: support Utf8View labels in Prometheus response (#8754)

Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(query): validate merge scan remote schema (#8579)

* fix(query): validate merge scan remote schema

Signed-off-by: discord9 <discord9@163.com>

* fix(query): treat JSON columns as schema-compatible across wire/decode forms

CI (Sqlness json2_limit standalone + distributed) failed on the new
remote-schema validation: a JSON column is Binary + extension metadata
(ARROW:extension:name=greptime.json, greptime:type=Json) on the wire but
decodes to Struct(...) with the extension metadata — validate_remote_schema
compared raw arrow data_type and rejected it as a mismatch.

Adds json_fields_compatible(): JSON fields are equal when name and
nullability match, greptime:type matches, and the JSON2 settings
(ARROW:extension:metadata type hints) match, ignoring the physical arrow
type. Only JSON fields may bypass the raw-type comparison; non-JSON
validation stays strict.

Adds 4 regression tests mirroring the CI failure (wire-binary vs
decoded-struct accepted both directions; different JSON2 settings
rejected; JSON vs plain Binary rejected).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(grafana): add events dashboard (#8725)

* feat(grafana): add events dashboard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): tolerate evolving event schemas

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): address events dashboard review

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): restore events dashboard panels

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): bound events dashboard queries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): include historical event catalogs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): preserve events drill-down context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): correct events lifecycle outcomes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): handle empty event type ranges

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): scope event catalogs to submissions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): handle empty events dashboard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): refresh event schema variables

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor: separate a json2 extension type (#8745)

Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: add admin function registrar (#8762)

* feat: add admin function registrar

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject admin function name collisions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: fix typo in admin function test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: information_schema.rs table initialization issue

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* rustfmt fix

Signed-off-by: onepizzateam <palakjha916@gmail.com>

---------

Signed-off-by: onepizzateam <palakjha916@gmail.com>
Signed-off-by: Palak Jha <palakjha916@gmail.com>
Signed-off-by: polar <palakjha916@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: xhwhis <hi@whis.me>
Signed-off-by: lyang24 <lanqingy93@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: Whis Liao <xhwhis@gmail.com>
Co-authored-by: Lanqing Yang <lanqingy93@gmail.com>
Co-authored-by: sun <sunchang_long@163.com>
Co-authored-by: Ning Sun <classicning@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
2026-08-06 07:24:59 +00:00
2023-08-10 08:08:37 +00:00
2023-06-25 11:05:46 +08:00
2023-11-09 10:38:12 +00:00
2023-03-28 19:14:29 +08:00

GreptimeDB Logo

One database for metrics, logs, and traces
replacing Prometheus, Loki, and Elasticsearch

The unified OpenTelemetry backend — with SQL + PromQL on object storage.

User Guide | API Docs | Roadmap 2026

Stable Canary Nightly

stable for production  ·  latest includes pre-releases  ·  nightly is a weekly snapshot of main

Docker Pulls GitHub Actions Codecov License
Slack Twitter LinkedIn

Introduction

GreptimeDB is an open-source observability database built for Observability 2.0 — treating metrics, logs, and traces as one unified data model (wide events) instead of three separate pillars.

Use it as the single OpenTelemetry backend — replacing Prometheus, Loki, and Elasticsearch with one database built on object storage. Query with SQL and PromQL, scale without pain, cut costs up to 50×.

Overview

A quick overview of what GreptimeDB ingests, how it connects to other systems, and what its distributed engine lets you do.

GreptimeDB Overview

Features

Feature Description
Observability 2.0 native Logs, metrics, and traces in one engine with SQL + PromQL. Native OpenTelemetry, Prometheus remote write, and Jaeger. Migrate one signal at a time, or use as a single backend.
Elastic compute-storage separation Scale reads independently with horizontal replicas. Serve high-concurrency workloads from dashboards, alerting, and AI agents — without resharding or data migration.
Sub-second on PBEB-scale data Columnar engine with fulltext, inverted, and skipping indexes. Written in Rust. Designed for high-concurrency point queries, not just analytical scans.
50× lower cost Object storage (S3, GCS, Azure Blob) as primary storage, with a tiered cache (memory + local disk) to keep writes and queries fast.

Perfect for:

  • Replacing Prometheus + Loki + Elasticsearch with a single observability backend
  • Scaling past Prometheus — high cardinality, long-term storage, no Thanos/Mimir overhead
  • AI/agent workloads — store GenAI telemetry (OTel GenAI conventions), and serve high-concurrency reads from SRE/developer agents via horizontal read replicas
  • Cutting observability costs with object storage (up to 50× savings on traces, 30% on logs)
  • Edge-to-cloud observability with unified APIs on resource-constrained devices

Why Observability 2.0? Three separate databases for metrics, logs, and traces means three storage layers, three query languages, and three sets of dashboards. GreptimeDB stores all three as timestamped wide events in one columnar engine — JOIN across signals in SQL, run one stack instead of three, and ingest AI agent telemetry the same way. Read more: Observability 2.0 and the Database for It.

Learn more in Why GreptimeDB.

How GreptimeDB Compares

Capability GreptimeDB Prometheus / Thanos / Mimir Grafana Loki Elasticsearch
Data types Metrics, logs, traces Metrics only Logs only Logs, traces
Query language SQL + PromQL PromQL LogQL Query DSL
Storage Native object storage (S3, etc.) Local disk + object storage (Thanos/Mimir) Object storage (chunks) Local disk
Scaling Compute-storage separation, stateless nodes Federation / Thanos / Mimir — multi-component, ops heavy Stateless + object storage Shard-based, ops heavy
Cost efficiency Up to 50× lower storage cost High at scale Moderate High (inverted index overhead)
OpenTelemetry Native (metrics + logs + traces) Partial (metrics only) Partial (logs only) Via instrumentation

Benchmarks:

Architecture

GreptimeDB can run in two modes:

  • Standalone — single binary for development and small deployments.
  • Distributed — four components, each independently scalable:
    • Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
    • Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
    • Metasrv — metadata, routing, repartitioning, autopilot, and security. Backed by a pluggable KV layer (etcd or RDS).
    • Flownode (optional) — continuous flow computation (streaming and materialized views).

For deeper coverage, see the architecture doc or DeepWiki.

GreptimeDB System Overview

Try GreptimeDB

For AI agents — paste this prompt into your agent:

Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
  -v "$(pwd)/greptimedb_data:/greptimedb_data" \
  --name greptime --rm \
  greptime/greptimedb:latest standalone start \
  --http-addr 0.0.0.0:4000 \
  --grpc-bind-addr 0.0.0.0:4001 \
  --mysql-addr 0.0.0.0:4002 \
  --postgres-addr 0.0.0.0:4003

Dashboard: http://localhost:4000/dashboard

Read more in the full Install Guide.

Troubleshooting:

  • Cannot connect to the database? Ensure that ports 4000, 4001, 4002, and 4003 are not blocked by a firewall or used by other services.
  • Failed to start? Check the container logs with docker logs greptime for further details.

Getting Started

Build From Source

Prerequisites:

  • Rust toolchain — nightly, pinned by rust-toolchain.toml
  • Protobuf compiler (>= 3.15)
  • C/C++ building essentials: gcc / g++ / autoconf and the glibc dev package (libc6-dev on Ubuntu, glibc-devel on Fedora)
  • Python toolchain (optional, only for some test scripts)

Build and run:

make                          # build greptime binary
cargo run -- standalone start # start in standalone mode

Common dev commands:

make fmt            # format Rust code
make clippy         # lint (fails on warnings)
make test           # unit + integration tests (uses cargo-nextest)
make sqlness-test   # SQL regression tests

See the Contribution Guidelines for the full developer workflow.

Tools & Extensions

Project Status

GreptimeDB is at v1.0 GA with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60% after migrating from Grafana Loki. See more in case studies.

Read the v1.0 highlights and 2026 roadmap, or browse the version reference.

If GreptimeDB is useful to you, please star the repo.

Known Users

Community

We invite you to engage and contribute!

License

GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.

A small set of peripheral, enterprise-only features are gated behind the enterprise Cargo feature (not built by default) and are governed by the separate GreptimeDB Enterprise License. Source files under that license carry an explicit Enterprise License header.

Commercial Support

Running GreptimeDB in your organization? We offer enterprise add-ons, services, training, and consulting. Contact us for details.

Contributing

Acknowledgement

Special thanks to all contributors! See AUTHOR.md.


All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.

S
Description
Languages
Rust 98.8%
Python 0.6%
Shell 0.3%
JavaScript 0.1%