Commit Graph
408 Commits
Author SHA1 Message Date
discord9 be0985cc39 fix(frontend): remove gRPC DDL panics for DropView and non-timestamp time index (#8739)
* fix(frontend): remove gRPC DDL panics for DropView and non-timestamp time index

Direct gRPC DDL bypasses the SQL parser, so two client-controlled DDL
payloads could panic a request handler:

- QX-152: DdlExpr::DropView hit todo!() (instance/grpc.rs:247-248).
  Wire it to the real drop-view implementation (drop_view was
  pub(crate); widened to pub) so a DropView DDL returns a structured
  error (e.g. TableNotFound) instead of panicking.
- QX-153: a CreateTableExpr whose time_index column is not a timestamp
  reached Schema::new's unwrap (ddl.rs:2346 -> schema.rs:114-119).
  create_table_info now uses Schema::try_new with ConvertSchemaSnafu
  context (InvalidArguments), and the direct gRPC CreateTable arm
  validates the request via validate_create_expr (which now also checks
  the time-index column type is a timestamp) before any catalog work.

SQL/HTTP paths were already protected by the parser; unchanged.
Tests: qx_152_drop_view_via_grpc_ddl_returns_error_not_panic,
qx_153_create_table_with_non_timestamp_time_index_via_grpc_returns_error
(asserts InvalidArguments), test_create_table_info_rejects_non_timestamp_time_index.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(frontend): add gRPC DDL happy-path coverage for DropView and CreateTable

Per review: the initial tests only asserted error paths. Add:
- drop_if_exists=true on a missing view succeeds (no error)
- dropping an existing view via gRPC DDL succeeds end-to-end
- a valid CreateTableExpr with a timestamp time index still succeeds
  (guards validate_create_expr against rejecting good requests)
- qx_152 test now asserts the TableNotFound status instead of is_err()

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 11e785ed5680dbb07b7cf88ec1b9642cde5ba1e1)
2026-08-21 11:51:11 +08:00
discord9 f9135380b5 feat(flow): add flow statistics and SHOW FLOW STATUS (release slice)
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 523c60c53bdc21fd1b1fa43b857e78de710f0d58)
2026-08-21 11:51:11 +08:00
Yingwen 6618427fc6 feat: add admin function to discard unflushed data (#8768)
* feat: add admin function to discard unflushed data

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: cover discarding unflushed data by table

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix license header

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: reject discarding logical metric table data

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: defer table name formatting in error paths

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore(deps): update greptime-proto revision

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: rename discard unflushed admin function

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 68605aa64d25a24e14b3bbf0b62e453d97179947)
2026-08-21 11:51:11 +08:00
a08d3b7e63 feat: support enabling skip_wal with ALTER TABLE (#8817)
* feat: support enabling skip_wal with ALTER TABLE (#8730)

* feat: support enabling skip_wal with alter table

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover skip wal on metric physical region

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: retry skip wal alter on route changes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address skip wal review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve skip wal create options

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: prefer typed skip wal option

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address skip wal review comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: canonicalize legacy skip wal option

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep typed skip wal canonical

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: simplify skip wal option tracking

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>

* test(object-store): fix racy SecureFs abort test (#8720)

test_writer_abort_is_unsupported_without_atomic_write asserted the file
content immediately after abort() returned Unsupported. SecureFsWriter
writes through tokio::fs::File, whose write_all() only enqueues a blocking
write task (tokio's poll_write returns Ready before the write completes),
so the data may not be visible yet when the test reads the file. Drop the
race-prone content assertion and only verify the Unsupported contract.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2026-08-10 15:45:41 +08:00
Lei, HUANG 7a2479496f chore!: gate soft-drop table behind the enterprise feature (#8747)
* chore: gate soft-drop table behind the enterprise feature

Soft-drop table becomes an enterprise-only feature:

- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
  non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
  without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
  registration, and information_schema.recycle_bin registration are
  compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
  unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
  example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
  (moving to the enterprise repo); affected information_schema .result
  files are regenerated

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: limit unused_variables allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: include the config key in the soft-drop enterprise gate error

Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: limit unused_mut allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: reject soft-drop DDL submissions in non-enterprise builds

Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: stop --enable-gc from enabling soft drop in the sqlness template

Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate fresh soft-drop procedures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: gate soft-drop fallback coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate soft-drop procedure implementation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate drop table soft-drop behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate expired soft-drop gc behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: test enterprise table ddl lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark purge_table as enterprise licensed

The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark recycle_bin as enterprise licensed

The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark soft-drop procedure sources as enterprise licensed

The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-08-06 15:10:54 +08:00
jeremyhi 448f973593 fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address local file sandbox review findings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: support Windows local copy paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: improve sandbox path errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: simplify local path error context

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: stream secure filesystem listings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* style: derive local file access default

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: improve local file access errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address local file access review findings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: simplify local file access coverage

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: harden sandboxed local file backends

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject directory copy targets before creation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: avoid implicit string clone in file table listing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-07-31 13:23:15 +00:00
LFC 4e04f92591 fix: make select whole json2 column worked (#8683)
* fix: make select whole json2 column worked

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* x

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-07-31 13:14:33 +00:00
Lei, HUANG f524a0b5b4 feat: support time range in manual compaction (#8669)
* feat: support time range in manual compaction

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: reject overflowing compaction range alignment

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: preserve range across compaction continuations

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf: use graph traversal for compaction windows

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: bump proto to commit on main

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain compaction window dependency closure

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-07-29 13:33:10 +00:00
discord9 b32a1de2fd fix(partition): avoid panic on missing route columns (#8645)
Signed-off-by: discord9 <discord9@163.com>
2026-07-27 11:52:06 +00:00
shuiyisong b462d5d19e fix: honor default prefix for all metric columns (#8640)
* fix: honor default prefix for metric columns

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-07-27 07:35:28 +00:00
yimeng cbabf8a53d Support unquoted postgres intervalstyle (#8637)
* Support unquoted postgres intervalstyle

Signed-off-by: yimeng <1982337+yimeng@users.noreply.github.com>

* test: verify unquoted intervalstyle output

Signed-off-by: yimeng <1982337+yimeng@users.noreply.github.com>

---------

Signed-off-by: yimeng <1982337+yimeng@users.noreply.github.com>
2026-07-26 15:02:18 +00:00
discord9 f66f99f102 fix: enforce COPY FROM row limit (#8551)
Signed-off-by: discord9 <discord9@163.com>
2026-07-24 07:02:01 +00:00
Lei, HUANG c8f65c7b99 feat: update flow windows after metric batch flush (#8544)
* feat: update flow windows after metric batch flush

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: move batch rows into flow notifier

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: preserve timestamp index in metric batches

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: avoid blocking flow notification lookups

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: add context to timestamp extraction logs

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: include peer in flow notification errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: compact flow notifications with time ranges

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: bound pending flow notification queue

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: send raw timestamps in flow notifications

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: update config API snapshot

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-07-22 13:09:30 +00:00
Lei, HUANG ed8a4990fe feat(flow): handle time_ranges in DirtyWindowRequest (#8582)
* feat(flow): handle time_ranges in DirtyWindowRequest

Bump greptime-proto to include the new `time_ranges` field on
DirtyWindowRequest (GreptimeTeam/greptime-proto#330) and mark the
corresponding aligned time windows as dirty in the batching engine,
in addition to the existing per-timestamp dirty marking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(flow): fix doc comment spacing in align_time_window

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(flow): cover time_ranges in handle_mark_dirty_time_window

Verify a valid [start_inclusive, end_exclusive) range is aligned to
time window boundaries and stored with an explicit end, and that empty
or reversed ranges are skipped.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(flow): union merged dirty windows with the larger end

Merging a bounded dirty range with a window contained in it (e.g.
[0s, 15s) with nested [5s, 10s), or an unbounded dirty window inside a
bounded range) previously assigned the contained window's upper bound,
shrinking the merged window and permanently dropping the tail range
from re-computation. Keep max(prev_upper, cur_upper) instead.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(flow): clip bounded dirty ranges at the expire bound

merge_dirty_time_windows dropped every window whose start is before
expire_lower_bound, so a bounded dirty range crossing the bound (e.g.
[0s, 15s) with expire 10s) lost its still-live suffix [10s, 15s). Now
bounded ranges are dropped only when their end is at/before the expire
bound, and crossing ranges are clipped to the bound (which the caller
aligns to the time window boundary). Unbounded windows keep the
existing start-based behavior.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(flow): fall back to full dirty on dirty-window alignment failure

An eval/alignment error previously aborted the per-task dirty-marking
closure, losing every dirty timestamp and range accumulated for that
task, while the RPC still returned Ok so the producer would not retry.
On alignment failure now log a warning and mark the whole task dirty
(set_dirty) instead, so the affected data is conservatively
recomputed.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(flow): apply rustfmt to new dirty-window merge tests

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(flow): cover time index units in dirty window marking

Document that DirtyWindowRequest timestamps/time_ranges are bare i64s
interpreted in the source table's time index native unit, and add a
test expressing the same [3s, 11s) range in second/millisecond/
microsecond/nanosecond units across four tables, asserting all align
to the same dirty window [0s, 15s).

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: bump greptime-proto to 8127f179

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(flow): return dirty-window alignment errors to callers

Do not acknowledge a DirtyWindowRequest when a time-windowed task cannot
align a timestamp or range. The previous conservative fallback used
set_dirty(), but that marker only represents a single epoch-start window
for time-windowed flows, so it could still lose the affected dirty
range. Propagate task errors through the join loop instead so producers
can retry.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: update proto to commits on main

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-07-21 12:56:01 +00:00
shuiyisong 5065dfcf2f feat: grant creators access to newly created databases (#8566)
* feat: implement auto database acl after create database

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: timeout conversion:

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove register loaders parameter in ddl_manager initialization

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-07-21 11:29:49 +00:00
Lei, HUANG 6a53bb053c feat: enable soft-drop table lifecycle (#8554)
* feat: add purge_table admin function

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: restrict purge_table to admin

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep purge tombstone lookup consistent

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: verify soft-drop table lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: run soft-drop lifecycle sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: read purge tombstones authoritatively

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): invalidate soft-drop marker caches

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): enable configurable table soft drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(ci): configure gc for soft-drop sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): close regions before soft-drop metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): avoid rollback after soft-drop tombstone

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): validate admin single-row calls

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): log purge dropped table target

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): mark soft-drop config experimental

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: sqlness test base

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-07-20 08:28:35 +00:00
Ruihang Xia 0479c05092 perf: preserve dictionary-encoded query labels (#8541)
* perf: preserve dictionary-encoded query labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(client): skip dictionary Flight batches

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(servers): decode dictionary labels in HTTP output

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test(mito2): support dictionary tags in series scans

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(servers): preserve dictionary child nulls in SQL

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(promql): compare dictionary tags by logical nulls

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): keep dictionary tags within query paths

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor(query): scope PK dictionary encoding to reads

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): preserve pushdown for dictionary labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): handle dictionary labels in query operators

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): complete label type matching

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): align dictionary query schemas

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): decode incompatible OR labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): preserve dictionary partition pruning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: handle dictionary query edge cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: import dictionary downcast macro

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-07-17 13:57:10 +00:00
Lei, HUANG bbf989ac36 feat: support soft-drop recycle bin and UNDROP TABLE (#8546)
* feat: support full WAL retirement

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: complete close request migration

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: guard Kafka provider setup behind index collector check

Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.

Affected file:
- `src/log-store/src/kafka/log_store.rs`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: avoid to_vec

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: clean up soft-dropped regions offline

Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.

Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): reject file-engine soft drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: preserve soft-drop cleanup split state

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: remove obsolete CleanUp match arm from RegionRequest

The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): clean every soft-dropped region replica

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): order soft-drop replica cleanup

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* Revert "fix(meta): order soft-drop replica cleanup"

This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* Revert "fix(meta): clean every soft-dropped region replica"

This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(catalog): expose soft drops in recycle bin

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(sql): add UNDROP TABLE

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(sql): cover UNDROP TABLE execution

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(sql): keep successful UNDROP result

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: reject stale undrop by name-based lookup after tombstone consumed

When a table is dropped, recreated under the same name (consuming the name
tombstone), and then the recreated table is dropped, an undrop procedure
that was built before the first drop and holds a stale original table name
should fail with TableNotFound instead of silently matching a different
table.

Changed `UndropTableProcedure::on_prepare` to perform a name-based lookup
when `table_name` is available and filter by table ID, ensuring that a
dropped table can only be recovered when its name tombstone still maps to
the expected ID.

- `src/common/meta/src/ddl/undrop_table.rs`: name-first lookup in on_prepare
- `src/common/meta/src/ddl/tests/drop_table.rs`: test for the stale-id
  rejection case

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(log-store): keep Kafka obsolete_all as no-op

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(catalog): hide purging tables from recycle bin

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(catalog): scope recycle bin scans by catalog

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(sqlness): update recycle bin expectations

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: avoid redundant recycle bin allocations

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-07-17 08:15:21 +00:00
shuiyisong 3ae8d7d46a refactor: reconcile OTLP trace schemas request-wide (#8485)
* refactor: global trace schema

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: fmt

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: split files

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments and doc

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-07-16 10:56:27 +00:00
shuiyisong 589ae56a6f fix: close database ACL gaps in permission checks (#8492)
* fix: database ACL

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: reduce code

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-07-14 14:16:08 +00:00
QuakeWang 4e14ac8676 feat: add strict CSV header validation (#8426)
* feat: add strict CSV header validation

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* style: simplify strict header parse error

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
2026-07-10 02:15:08 +00:00
fys 3bf8b9722d feat(json2): validate append mode for tables with JSON2 columns (#8434)
* feat: validate append mode for JSON2 tables

* fix: cr by codex

* fix: unit test

* fix: validate JSON2 append_mode for all ALTER TABLE operations
2026-07-08 03:33:36 +00:00
shuiyisong 90752f5649 feat: persist Prometheus remote write v2 native histograms (#8382)
* chore: add histogram decode

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add convert func

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat: implement native histogram persistency

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: refactor

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add README for v2

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat: implement metric engine

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: store nh in one struct

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues and add feature gate

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: add config option

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-07-06 11:51:22 +00:00
Weny Xu 3aa82567c4 fix: invalidate comment DDL cache and lock by object ID (#8390)
* fix: invalidate comment ddl cache locally

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix typos

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-07-01 10:02:57 +00:00
fys aa563628fb feat(json2): type hint (#8247)
* feat(json2): type hint

* test(datatypes): add JsonSettings serde round-trip test

* minor refactor

This reverts commit 7ff5a5249a09be5396536284fe822b5761ef4e6a.

* fix: code review
2026-06-30 07:37:31 +00:00
discord9 d3cc1b1888 feat(flow): stabilize eval interval scheduling (#8360)
* feat(flow): stabilize eval interval scheduling

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): satisfy eval schedule clippy

Signed-off-by: discord9 <discord9@163.com>

* test(flow): trim eval schedule coverage

Signed-off-by: discord9 <discord9@163.com>

* test(flow): cover stable eval scheduling

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): reserve scheduled runtime hint

Signed-off-by: discord9 <discord9@163.com>

* test(flow): trim sqlness result eof

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): harden eval schedule edges

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): address scheduled flow review

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): clean scheduled config handling

Signed-off-by: discord9 <discord9@163.com>

* test(flow): add eval interval compat case

Signed-off-by: discord9 <discord9@163.com>

* test(flow): cover show create flow in compat

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): drop scheduled time from flow context

Signed-off-by: discord9 <discord9@163.com>

* test(flow): assert scheduled now binding

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-29 09:55:27 +00:00
shuiyisong 59d40c39b1 feat: prw_v2 initial commit with sample ingestion (#8361)
* chore: add v2 entrance

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: decode request

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: implement remote write v2 for samples

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove hand-written proto

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: refactor

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: merge tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add source version field

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update proto

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update proto

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-06-29 03:15:41 +00:00
dennis zhuang e89d591cb4 chore: declare GreptimeDB Enterprise License for enterprise-gated sources (#8364)
* chore: declare GreptimeDB Enterprise License for enterprise-gated sources

The `enterprise`-feature-gated sources (triggers, mito2 extension) were
excluded from the Apache-2.0 header check but carried no license of their
own. Declare a separate GreptimeDB Enterprise License and enforce it.

- Add LICENSE-ENTERPRISE (open-core split; core stays Apache-2.0).
- Add an Enterprise License header to each enterprise source file.
- Add licenserc-enterprise.toml and a second hawkeye step in CI to enforce
  the Enterprise header on exactly those files.
- Cross-reference the two complementary file lists; document the layout in
  licenses/README.md and the README License section.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: reference per-customer Enterprise Agreement instead of a terms URL

There is no public enterprise-terms page; each customer signs an individually
negotiated agreement. Point the license at a "separate written commercial
agreement with GrepTime Inc." and direct readers to the existing contact page.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-06-26 06:31:02 +00:00
Yash Agrawal daed55e836 feat: Add support for splunk HEC endpoints (#8321)
* feat: add health endpoint for splunk HEC

Signed-off-by: yash <yasha4658@gmail.com>

* feat: add event endpoint and handler + tests

Signed-off-by: yash <yasha4658@gmail.com>

* feat: add pipeline override test

Signed-off-by: yash <yasha4658@gmail.com>

* feat: add error response for auth errors

Signed-off-by: yash <yasha4658@gmail.com>

* fix: cargo fmt + doc example test

Signed-off-by: yash <yasha4658@gmail.com>

* fix: clippy warnings

Signed-off-by: yash <yasha4658@gmail.com>

* fix: address comments

Signed-off-by: yash <yasha4658@gmail.com>

* fix: address copilot comments and default table name inlining

Signed-off-by: yash <yasha4658@gmail.com>

* fix: cargo fmt

Signed-off-by: yash <yasha4658@gmail.com>

* fix: add error code 12, 13 and fix the timestamp override bug

Signed-off-by: yash <yasha4658@gmail.com>

* fix: add host/source/sourcetype ordering

Signed-off-by: yash <yasha4658@gmail.com>

* fix: fix typos

Signed-off-by: yash <yasha4658@gmail.com>

---------

Signed-off-by: yash <yasha4658@gmail.com>
2026-06-26 04:03:41 +00:00
Weny Xu e520ff8300 feat: decouple error retryability from status codes (#8301)
* feat: add retry hint to common error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: propagate retry hints in core errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: propagate retry hint through RPC metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fallback retry hints to status codes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: add explicit retry hints for retryable errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: remove status code retry fallback

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: classify io retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minior

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve datasource retry hints in query

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve metric engine retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve flow and frontend retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: share rskafka retry hint mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-17 08:27:53 +00:00
Weny Xu b01ee594f3 feat: add repartition column hint (#8291)
* feat: add repartition column hint option

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: support altering repartition column hint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update sqlness result

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reject time index repartition hint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: treat rename table as metadata-only alter

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-16 09:04:52 +00:00
Weny Xu 15597047c9 feat: pass Kafka pruned entry id when creating regions (#8282)
* refactor: use typed region wal options

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: pass kafka pruned entry id to regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove unused error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: clarify wal options serialization errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover legacy region wal options encoding

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: recover legacy create table wal options

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: lock remote wal during table creation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests-s

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: refresh remote wal prune hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-15 08:51:50 +00:00
Weny Xu 61f802cdf2 feat: support repartition with target partition columns (#8278)
* feat(sql): support repartition on columns syntax

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): support repartition target partition columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(operator): wire repartition target columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): validate repartition target columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover repartition on columns integration

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): validate duplicate target partition columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: upgrade proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-12 07:36:53 +00:00
dennis zhuang 31c2c1f6db feat: table semantic layer per-table enrichment (Phase 2) (#8218)
* feat: table semantic layer per-table enrichment (Phase 2)

Phase 2 of the table semantic layer, plus a vocabulary trim so the layer only
records what a machine consumer cannot cheaply recover on its own.

Per-table metric enrichment (OTLP), via an internal per-table channel:
- A `SemanticIndex` accumulator records, per emitted table, the declared metric
  keys: type / unit / temporality / metadata_quality=declared / original_name.
  Conflicting single-valued keys collapse to `mixed`/`unknown`.
- Recording happens at the `encode_metrics` level where the base name, metric
  type, and proto fields are all in scope, so histogram/summary fan-out gets the
  correct per-subtable type (`_bucket`=histogram, `_sum`/`_count`=counter)
  without threading state through every encoder.
- The index is serialized onto the `greptime.internal.semantic.per_table_index`
  context extension; `apply_per_table_semantic_options` folds each table's keys
  into its options at auto-create time.
- `trace.conventions` is refined from the request's resource/scope `schema_url`s
  (concrete when uniform, else `mixed`/`unknown`).

Vocabulary trimmed to only meaningful keys. Kept: signal_type, source, pipeline,
trace.conventions, metric.{type,unit,temporality,metadata_quality,original_name}.
Dropped: metric.monotonic (a function of type), trace.has_events/has_links
(constant + derivable from columns), log.severity_scheme/body_format (constant /
derivable, and body_format cost an O(rows) scan), resource/scope lineage
(restates columns / collector-config concern), source_version (no cheap
non-constant value today). Prometheus carries type/unit in the metric name by
convention, so it gets identity only — no inferred enrichment.

Identity (signal_type + source) extended to the remaining ingest protocols so
the discovery view is complete: InfluxDB and OpenTSDB (metric), Loki and
Elasticsearch (log). These protocols carry no type/unit metadata, so identity is
all that applies.

Tests: unit coverage for the accumulator, per-metric-type fan-out, and trace
conventions; integration goldens updated for the OTLP metric/trace SHOW CREATE
output and the new Loki identity.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: validate the option value

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-06-04 12:52:00 +00:00
QuakeWang 6ed67167bb feat: support headerless CSV copy from (#8233)
* feat: support headerless CSV copy from

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* fix: update csv copy sqlness result

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* test: cover headerless CSV copy from

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* test: cover headerless CSV column count mismatch

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
2026-06-04 12:04:15 +00:00
Lei, HUANG da9d314fa4 fix: align bulk insert schema handling (#8222)
* fix: align bulk insert schema handling

Carry `aligned_schema_version` through bulk insert requests so datanode can recheck stale schemas only when needed. Use the physical data region schema version for metric logical bulk inserts, while external bulk insert callers leave alignment unknown.

Related files:
- `Cargo.toml`
- `Cargo.lock`
- `src/store-api/src/region_request.rs`
- `src/metric-engine/src/engine/bulk_insert.rs`
- `src/mito2/src/request.rs`
- `src/mito2/src/worker/handle_bulk_insert.rs`
- `src/mito2/src/worker/handle_write.rs`
- `src/operator/src/bulk_insert.rs`
- `src/servers/src/pending_rows_batcher.rs`
- `src/mito2/src/engine/edit_region_test.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* test: cover bulk insert missing columns

Add Flight bulk insert coverage for batches that omit nullable columns and verify the database fills the missing values with nulls.

Related files:
- `tests-integration/src/grpc/flight.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore: update proto to main branch commit

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: align physical metric bulk insert schema

Pass the physical schema version through metric bulk insert requests so
physical and logical region writes share the same schema alignment behavior.

Files:
- \`src/metric-engine/src/engine/bulk_insert.rs\` — updates \`bulk_insert_physical_region\` and reuses \`physical_schema_version\` for bulk insert schema alignment.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore: add some doc

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2026-06-04 02:55:09 +00:00
QuakeWang ca07a53deb feat: support CSV copy skip bad records (#8198)
* feat: support CSV copy headers and skip bad records

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* refactor: reuse CSV skippable error helper

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* fix: address CSV skip bad records review

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* docs: clarify CSV skip bad records performance tradeoff

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
2026-06-03 12:11:02 +00:00
dennis zhuang f25ab67af1 feat: table semantic layer identity (Phase 1) (#8210)
* feat: table semantic layer identity (Phase 1)

Attach a thin layer of semantic metadata to ingested tables via the existing
`table_options` slot, so machine consumers (LLM agents, alert/dashboard builders,
MCP servers, ETL) can align a table with the observability concept it stands for
without guessing from column names. See docs/rfcs/2026-05-28-table-semantic-layer.md.

Phase 1 (identity) only:
- New `table::requests::semantic` module: the `greptime.semantic.*` vocabulary
  (signal/source/source_version/pipeline + trace/metric/log/resource-scope keys,
  defined now, populated by later phases), value constants, the internal
  `greptime.internal.semantic.per_table_index` transport key (reserved for Phase 2,
  deliberately outside the public namespace), and `is_semantic_option_key`.
- `validate_table_option` accepts the `greptime.semantic.*` prefix, so the keys are
  valid both on the auto-create path and on explicit `CREATE TABLE ... WITH (...)`.
- `fill_table_options_for_create` copies every semantic ctx extension into the new
  table's options (prefix passthrough alongside the fixed allowlist).
- Frontend stamps identity on the context at each ingest entry: OTLP metrics
  (metric/opentelemetry), traces (+pipeline, has_events/has_links/conventions for
  the v1 model), logs (log/opentelemetry), and Prometheus remote write
  (metric/prometheus, metadata_quality=inferred). OTLP metric metadata_quality is
  left for Phase 2 (declared).
- Trace identity is stamped only on the main span table; the derived
  `_services` / `_operations` lookup tables keep the unstamped context and carry no
  semantic identity (cross-table relationships are out of scope).

Semantic options appear in SHOW CREATE TABLE (like table_data_model /
otlp_metric_compat) and in information_schema, so an LLM inspecting a table sees its
semantics directly.

Tests: unit (validation prefix + internal-key rejection, ctx passthrough) and
integration assertions that the common keys land for OTLP metrics (metric-engine
logical table), traces, logs, and Prometheus remote write; SHOW CREATE goldens
updated.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: prom batcher not cover and white list for semantic keys/values

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: typo

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-06-02 01:56:29 +00:00
discord9 28fd796f4e fix(flow): harden incremental read correctness (#8196)
* fix(flow): harden incremental read correctness

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): propagate dirty window options

Signed-off-by: discord9 <discord9@163.com>

* test: more

Signed-off-by: discord9 <discord9@163.com>

* chore: test config api

Signed-off-by: discord9 <discord9@163.com>

* refactor: split gen

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* fix: allowlist key

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-06-01 02:48:00 +00:00
dennis zhuang ed9312f8e3 feat: global switch for creating tables automatically (#8203)
* feat: global switch for creating table automatically

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: make auto_create_table as comment by default

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: respect gloabl switch for metric engine

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-05-31 23:51:14 +00:00
discord9 ba15a9c056 feat: support pending flow metadata with defer_on_missing_source (#8124)
* feat: support defer_on_missing_source for pending flow creation

Add `defer_on_missing_source` flow option that allows creating flows
even when source tables do not yet exist. The flow enters a pending
state and is automatically activated when source tables become available.

Key changes:
- New `FlowStatus::PendingSources` and fields in `FlowInfoValue` for
  unresolved source table names and last activation error
- `defer_on_missing_source` create-time-only option: stripped from
  runtime/flownode `CreateRequest` but preserved in metadata for
  SQL round-trip (`SHOW CREATE FLOW`, `information_schema.flows`)
- `CreateFlowProcedure` creates pending metadata when sources are
  missing and `defer_on_missing_source=true`; falls back to
  `FlowType::Batching` for missing-source flows
- `PendingFlowReconcileManager` in meta-srv periodically checks
  pending flows and activates them when source tables resolve
- `ActivatePendingFlowProcedure` handles activation: allocates peers,
  creates flows on flownodes, updates metadata, invalidates cache
- `OR REPLACE` properly handles pending<->active transitions,
  including peer allocation and flownode flow teardown
- `FlowMetadataAllocator::alloc_peers` for peer allocation at
  activation time
- Validated flow options: only `defer_on_missing_source` allowed;
  unknown options rejected
- Known issue: standalone mode does not support flownodes, so
  pending flow flush/sink behavior covered only in distributed
  sqlness; operator and meta unit tests cover activation logic

Tests:
- operator `determine_flow_type_for_source_state` (3 passed)
- common-meta `create_flow` (19 passed) including replacement
- common-meta `activate_flow` (4 passed)
- meta-srv `flow` (11 passed)
- sqlness: `flow_pending` covers create/replace/round-trip

Signed-off-by: discord9 <discord9@163.com>

* chore: simplify pending flow PR scope

Reduce PR #8124 to the metadata-only MVP after complexity review.

Changes:
- Remove automatic activation procedure and meta-srv reconcile wiring
- Remove activation tests and activation-only metadata fields
- Reject cross-state pending<->active `OR REPLACE` transitions for now
- Keep pending metadata creation and SQL round-trip behavior
- Allow `DROP FLOW` for pending flows without routes
- Reduce flow_pending sqlness to metadata/round-trip/drop coverage only

Deferred follow-ups are documented locally in `.tmp/tasks/pending-defer-semantics/deferred-followups.md` and intentionally not committed.

Tests:
- `cargo test -p operator determine_flow_type_for_source_state`
- `cargo test -p common-meta create_flow`
- `cargo test -p common-meta drop_flow`
- `cargo sqlness bare --test-filter flow_pending --bins-dir /mnt/nvme_rust/rust-targets/pending_defer/debug`

Signed-off-by: discord9 <discord9@163.com>

* test: cover pending flow metadata edge cases

Signed-off-by: discord9 <discord9@163.com>

* test: fix pending flow metadata test lint

Signed-off-by: discord9 <discord9@163.com>

* docs: document pending flow metadata fields

Signed-off-by: discord9 <discord9@163.com>

* chore: more sleep when test

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-05-29 06:59:21 +00:00
Weny Xu f513b77ccc feat: support alter table partition syntax (#8177)
* feat(sql): support alter table partition syntax

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: support repartition source proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update greptime-proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-05-26 15:06:14 +00:00
Lei, HUANG f8df016623 feat: add InfluxDB default merge mode config (#8134)
* feat/influxdb-default-merge-mode: add InfluxDB merge mode config

- `influxdb` config: add `default_merge_mode` parsing and defaults in `src/frontend/src/service_config/influxdb.rs` and `src/frontend/src/service_config.rs`
- auto-create behavior: apply configured `merge_mode` for InfluxDB ingestion in `src/frontend/src/instance.rs`, `src/frontend/src/instance/builder.rs`, `src/frontend/src/instance/influxdb.rs`, and `src/operator/src/insert.rs`
- config docs: document `influxdb.default_merge_mode` in `config/frontend.example.toml`, `config/standalone.example.toml`, and `config/config.md`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/influxdb-default-merge-mode: derive merge mode default

- `influxdb` config: derive `Default` for `InfluxdbMergeMode` in `src/frontend/src/service_config/influxdb.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/influxdb-default-merge-mode: update config API snapshot

- `config API`: include `default_merge_mode` in `tests-integration/tests/http.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/influxdb-default-merge-mode: avoid default context clone

- `InfluxDB merge mode`: avoid cloning `QueryContext` for default `last_non_null` in `src/frontend/src/instance/influxdb.rs`
- `InfluxDB merge mode`: cover default, configured, and explicit `MERGE_MODE_KEY` paths in `src/frontend/src/instance/influxdb.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2026-05-19 16:54:36 +00:00
Lei, HUANG 2fdbe6c8c3 feat: expose node info for placement selectors (#8095)
* feat: expose node info for placement selectors

Return `NodeInfo` from `PeerDiscovery` methods and keep OSS selectors mapping back to `Peer`.

Carry `__greptime_origin_frontend.addr` from frontend create-table DDLs into selector `extensions`, and thread `PeerAllocContext` through table-route allocation.

Persist datanode `NodeInfo` when heartbeat stats are absent so collected env vars remain available after restart.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: skip datanode node info without stats

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: avoid unnecessary workload clones

Skip workload cloning for inactive nodes and for active node-info lookups without workload filters.

Files: `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: require frontend origin address

Require `StatementExecutor` to carry a concrete frontend origin address and always attach it to meta DDL query contexts.

Files: `src/operator/src/statement.rs`, `src/operator/src/statement/ddl.rs`, `src/operator/src/utils.rs`, `src/frontend/src/instance/builder.rs`, `src/frontend/src/heartbeat.rs`, `src/flow/src/server.rs`, `src/cmd/src/standalone.rs`, `src/cmd/src/flownode.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: reuse resolved frontend address

Resolve the frontend peer address once in the frontend builder, store it on the instance, and reuse it for heartbeat and flow invoker origins.

Files: `src/frontend/src/instance/builder.rs`, `src/frontend/src/instance.rs`, `src/frontend/src/heartbeat.rs`, `src/cmd/src/frontend.rs`, `src/cmd/src/standalone.rs`, `src/frontend/src/frontend.rs`, `src/frontend/src/heartbeat/tests.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: preserve datanode lease liveness

Filter active datanode node infos through lease timestamps and workloads while preserving node info fields such as reported env vars.

Files: `src/meta-srv/src/discovery/utils.rs`, `src/meta-srv/src/discovery/lease.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* Remove stale datanode lease helper

- `discovery`: remove the obsolete `alive_datanodes` helper and related tests in `src/meta-srv/src/discovery/utils.rs` and `src/meta-srv/src/discovery/lease.rs`
- `integration`: update cluster and standalone setup paths in `tests-integration/src/cluster.rs` and `tests-integration/src/standalone.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat/env-based-region-selector-oss: simplify lease discovery

- `lease-discovery`: simplify logic and remove unused utilities in `src/meta-srv/src/discovery/lease.rs` and `src/meta-srv/src/discovery/utils.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2026-05-15 07:35:18 +00:00
Lei, HUANG 796aae3d9f feat(operator): allow last_row merge mode with append mode (#8065)
* feat(operator): allow last_row merge_mode when append_mode is enabled

- Update RegionOptions::validate to allow last_row merge_mode with append_mode.
- Update fill_table_options_for_create to automatically set merge_mode to last_row when append_mode is enabled for LastNonNull table type.
- Add unit tests in mito2 and operator to verify options validation and table creation.
- Add integration test for InfluxDB write with append mode hint.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix(operator): simplify append mode options

Group `LastNonNull` auto-create options in a single append-mode branch.

Files:

- `src/operator/src/insert.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: sqlness

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2026-05-07 07:21:37 +00:00
discord9 d2d256909f feat(flow): parse defer on miss src table (#7980)
* feat: parse create flow with

Signed-off-by: discord9 <discord9@163.com>

* feat: validate after parse

Signed-off-by: discord9 <discord9@163.com>

* pcr

Signed-off-by: discord9 <discord9@163.com>

* chore: sqlness

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-04-27 03:02:13 +00:00
shuiyisongandevenyag 0effc30778 chore: update the opendal to 0.56 rc2 (#8003)
* chore: update opendal version

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update opendal version

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: grpc init

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: dep versions

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: remove aws-lc-rs in reqwest

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: rebase main and fix compile

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: remove unused deps

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* Revert "fix: remove aws-lc-rs in reqwest"

This reverts commit 90bfafca06f877befb36f3e54bb72fcfc4c56778.

* chore: remove aws-lc-sys from blacklist

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add tls deps

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: idemptent install in rds

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: use aws-lc-sys as possible

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: lint

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: address comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: address CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: sync opendal compat adapter with upstream

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: address compat clippy warnings

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
2026-04-26 09:59:48 +00:00
LFC c7427fa6d2 feat: json2 insert (#7981)
* feat: json2 insert

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* use main greptime-proto

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-04-20 09:45:32 +00:00
yxrxy 6c924efc2f fix: remove redundant error messages in admin functions (#7953)
Closes #7938

Signed-off-by: yxrxy <yxrxytrigger@gmail.com>
2026-04-17 08:21:22 +00:00
fys 62013217c7 fix: cargo check -p common-meta (#7964)
fix: moka feature
2026-04-14 08:27:22 +00:00