Commit Graph
18 Commits
Author SHA1 Message Date
discord9 bb6d55a99f feat: support old-stage datanode config overlays (#8647)
* feat: support old-stage datanode config overlays

Signed-off-by: discord9 <discord9@163.com>

* fix: derive compat overlay policy from WAL config

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-08-07 08:40:46 +00:00
Lei, HUANG c55f297dec chore!: gate soft-drop table behind the enterprise feature (#8747)
* chore: gate soft-drop table behind the enterprise feature

Soft-drop table becomes an enterprise-only feature:

- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
  non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
  without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
  registration, and information_schema.recycle_bin registration are
  compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
  unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
  example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
  (moving to the enterprise repo); affected information_schema .result
  files are regenerated

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: limit unused_variables allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: include the config key in the soft-drop enterprise gate error

Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: limit unused_mut allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: reject soft-drop DDL submissions in non-enterprise builds

Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: stop --enable-gc from enabling soft drop in the sqlness template

Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate fresh soft-drop procedures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: gate soft-drop fallback coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate soft-drop procedure implementation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate drop table soft-drop behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate expired soft-drop gc behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: test enterprise table ddl lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark purge_table as enterprise licensed

The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark recycle_bin as enterprise licensed

The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark soft-drop procedure sources as enterprise licensed

The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-08-05 04:28:30 +00:00
jeremyhi 448f973593 fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address local file sandbox review findings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: support Windows local copy paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: improve sandbox path errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: simplify local path error context

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: stream secure filesystem listings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* style: derive local file access default

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: improve local file access errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address local file access review findings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: simplify local file access coverage

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: harden sandboxed local file backends

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject directory copy targets before creation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: avoid implicit string clone in file table listing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-07-31 13:23:15 +00:00
Lei, HUANG 6a53bb053c feat: enable soft-drop table lifecycle (#8554)
* feat: add purge_table admin function

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: restrict purge_table to admin

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep purge tombstone lookup consistent

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: verify soft-drop table lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: run soft-drop lifecycle sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: read purge tombstones authoritatively

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): invalidate soft-drop marker caches

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): enable configurable table soft drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(ci): configure gc for soft-drop sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): close regions before soft-drop metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): avoid rollback after soft-drop tombstone

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): validate admin single-row calls

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): log purge dropped table target

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): mark soft-drop config experimental

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: sqlness test base

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-07-20 08:28:35 +00:00
discord9 88d20d8c93 test: choose compat runner cli args by version (#8376)
* test: use legacy rpc args in compat runner

Signed-off-by: discord9 <discord9@163.com>

* test: detect compat runner grpc cli args

Signed-off-by: discord9 <discord9@163.com>

* ci: run compat on ready pr open

Signed-off-by: discord9 <discord9@163.com>

* test: choose compat cli args by version

Signed-off-by: discord9 <discord9@163.com>

* test: address compat arg test review

Signed-off-by: discord9 <discord9@163.com>

* test: fix compat runner comment typo

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-07-01 07:29:32 +00:00
QuakeWang 8825ea3fdf fix!: align gRPC CLI option names with config naming (#8021)
* fix: align gRPC CLI option names with config naming

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* fix: warn on deprecated metasrv grpc config

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
2026-04-24 09:51:01 +00:00
LFC 74ff5c37ea refactor: customize standalone instance build (#7807)
* refactor: customize standalone instance build

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-03-13 09:25:21 +00:00
Yingwen 0939dc1d32 test: run sqlness for flat format (#7178)
* test: support flat format in sqlness

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: replace region stats test result with NUM

Signed-off-by: evenyag <realevenyag@gmail.com>

* ci: add flat format to sqlness ci

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2025-11-05 11:23:12 +00:00
Weny Xu 40e9ce90a7 refactor: restructure sqlness to support multiple envs and extract common utils (#7066)
* refactor: restructure sqlness to support multiple envs and extract common utils

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore(ci): update sqlness cmd

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: error fmt

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: only reconnect mysql and pg client

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2025-10-11 06:34:17 +00:00
LFC 979c8be51b feat: able to pass external service for sqlness test (#7032)
feat: able to pass external service instead of creating inside for sqlness test

Signed-off-by: luofucong <luofc@foxmail.com>
2025-10-09 07:02:19 +00:00
Ruihang Xia c9377e7c5a build: bump rust edition to 2024 (#6920)
* bump edition

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* gen keyword

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* lifetime and env var

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* one more gen fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* lifetime of temporaries in tail expressions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clippy nested if

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clippy let and return

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-09-08 02:37:18 +00:00
discord9 d774996e89 chore: make internal grpc optional (#6789)
* chore: make internal grpc optional

Signed-off-by: discord9 <discord9@163.com>

* revert sqlness runner too

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2025-08-22 04:22:49 +00:00
discord9andNing Sun eaceae4c91 feat: frontend internal grpc port (#6784)
* feat: frontend internal grpc port

Signed-off-by: discord9 <discord9@163.com>

* fix: grpc server naming

Signed-off-by: discord9 <discord9@163.com>

* test: sqlness test fix

Signed-off-by: discord9 <discord9@163.com>

* fix: internal not use process manager

Signed-off-by: discord9 <discord9@163.com>

* test: test integration port alloc

Signed-off-by: discord9 <discord9@163.com>

* feat: skip auth for internal grpc

Signed-off-by: discord9 <discord9@163.com>

* test: is distributed

Signed-off-by: discord9 <discord9@163.com>

* what:

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
2025-08-22 02:46:35 +00:00
discord9 a0900f5b90 feat(flow): use batching mode&fix sqlness (#5903)
* feat: use flow batching engine

broken: try using logical plan

fix: use dummy catalog for logical plan

fix: insert plan exec&sqlness grpc addr

feat: use frontend instance in flownode in standalone

feat: flow type in metasrv&fix: flush flow out of sync& column name alias

tests: sqlness update

tests: sqlness flow rebuild udpate

chore: per review

refactor: keep chnl mgr

refactor: use catalog mgr for get table

tests: use valid sql

fix: add more check

refactor: put flow type determine to frontend

* chore: update proto

* chore: update proto to main branch

* fix: add locks for create/drop flow&docs: update docs

* feat: flush_flow flush all ranges now

* test: add align time window test

* docs: explain `nodeid` use in check task

* refactor: AddAutoColumnRewriter check for Projection

* refactor: per review

* fix: query without time window also clean dirty time window

* chore: better logging

* chore: add comments per review

* refactor: per review

* chore: per review

* chore: per review rename args

* refactor: per review partially

* chore: update docs

* chore: use better error variant

* chore: better error variant

* refactor: rename FlowWorkerManager to FlowStreamingEngine

* rename again

* refactor: per review

* chore: rebase after #5963 merged

* refactor: rename all flow_worker_manager occurs

* docs: rm resolved TODO
2025-04-23 15:12:16 +00:00
Yohan Wal 1dc4a196bf feat: add mysql election logic (#5694)
* feat: add mysql election

* feat: add mysql election

* chore: fix deps

* chore: fix deps

* fix: duplicate container

* fix: duplicate setup for sqlness

* fix: call once

* fix: do not use NOWAIT for mysql 5.7

* chore: apply comments

* fix: no parallel sqlness for mysql

* chore: comments and minor revert

* chore: apply comments

* chore: apply comments

* chore: add  to table name

* ci: use 2 metasrv to detect election bugs

* refactor: better election logic

* chore: apply comments

* chore: apply comments

* feat: version check before startup
2025-03-19 11:31:18 +00:00
yihong 408dd55a2f fix: flaky test in sqlness by fix random port (#5657)
* fix: flaky test in sqlness by fix random port

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: typo

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: panic insead of forever loop

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-03-07 00:41:22 +00:00
Ruihang Xia 1c04ace4b0 feat: skip printing full config content in sqlness (#5618)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-03-03 09:43:55 +00:00
Ruihang Xia d7b6718be0 feat: run sqlness in parallel (#5499)
* define server mode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* bump sqlness

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* all good

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor: Move config generation logic from Env to ServerMode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* finalize

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change license header

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename variables

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* override parallelism

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename more variables

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-02-21 07:05:19 +00:00