rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-09-12 16:32:16 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,910 Commits 197 Branches 272 Tags
backport/v1.2/patch-correctness
Commit Graph
102 Commits
Author SHA1 Message Date
Qiang-Liu 8f8f42965b fix(cli): sanitize store_addrs in kvbackend build log (#8967) 
* fix(cli): sanitize store_addrs in kvbackend build log

Close #7525. The CLI's kvbackend construction log was printing raw
store_addrs which could contain sensitive connection strings (e.g.
PostgreSQL DSNs with passwords).

Changes:
- Add sanitize_store_addrs() helper that reuses
  common_meta::kv_backend::util::sanitize_connection_string(),
  consistent with MetasrvOptions and StartCommand patterns.
- Replace raw store_addrs in the info! log with sanitized version.
- Add unit tests covering MySQL URLs, PostgreSQL DSNs, etcd addresses,
  and empty store_addrs cases.

Signed-off-by: qiang_liu
Signed-off-by: qiang_liu <qiang_liu@trendmicro.com>
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>

* fix(cli): drop redundant sanitize tests per review

sanitize_connection_string in common_meta already covers MySQL URLs,
PostgreSQL DSNs and credential-free etcd addresses with its own tests.
The added tests only exercised a trivial map+collect wrapper, so remove
them per review nit.

Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>

---------

Signed-off-by: qiang_liu
Signed-off-by: qiang_liu <qiang_liu@trendmicro.com>
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
(cherry picked from commit ad5ccc98ec)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-09-11 03:06:27 +08:00
discord9 f9135380b5 feat(flow): add flow statistics and SHOW FLOW STATUS (release slice) 
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 523c60c53bdc21fd1b1fa43b857e78de710f0d58)
 2026-08-21 11:51:11 +08:00
Lei, HUANG 4873fc4f18 feat(meta): add retention GC for soft-dropped tables (#8526) 
* feat(meta): persist soft-drop retention metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): complete soft-drop retention coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): finalize soft-drop metadata after prepare

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): purge expired soft-dropped tables

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): keep soft-drop GC responsive

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): deduplicate soft-drop purge tasks

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): release purge reservations safely

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(meta): simplify soft-drop GC wiring

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): persist soft-drop retention deadlines

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): retain soft-drop config for recovery

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: complete standalone DDL test context

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): recheck retention before automatic purge

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): keep experimental soft drop disabled

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): rotate soft-drop purge candidates

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): address soft-drop retention review

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): protect retained table tombstones

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): preserve post-cleanup purge state

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): fence purge by drop generation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-16 08:47:03 +00:00
jeremyhi 619460f742 test(cli): add minio export-import v2 e2e (#8314) 
* test(cli): add minio export-import v2 e2e

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): reuse e2e connection helper in minio test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): avoid stderr in minio e2e cleanup

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): report minio snapshot cleanup failures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* ci: run import-v2 minio e2e

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): cleanup minio snapshot on failure

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-24 09:25:10 +00:00
jeremyhi f74dfacfad test(cli): harden incomplete import-v2 snapshot e2e (#8313) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-23 06:55:21 +00:00
jeremyhi 47581acbd3 test(cli): harden import-v2 schema filter e2e (#8312) 
* test(cli): harden import-v2 schema filter e2e

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): assert filtered export includes both schemas

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): assert import-v2 schema filter result

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-22 12:23:28 +00:00
jeremyhi a3e01e0018 test(cli): add import-v2 resume e2e coverage (#8311) 
* test(cli): harden export import v2 e2e roundtrip

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): add import-v2 resume e2e coverage

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): make import-v2 resume replay check non-idempotent

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): share export import e2e setup helpers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): assert import-v2 resume row identities

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-18 09:02:24 +00:00
jeremyhi 3e36778b0e test(cli): harden export/import-v2 e2e roundtrip (#8310) 
test(cli): harden export import v2 e2e roundtrip

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-18 07:41:54 +00:00
Weny Xu e520ff8300 feat: decouple error retryability from status codes (#8301) 
* feat: add retry hint to common error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: propagate retry hints in core errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: propagate retry hint through RPC metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fallback retry hints to status codes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: add explicit retry hints for retryable errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: remove status code retry fallback

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: classify io retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minior

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve datasource retry hints in query

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve metric engine retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve flow and frontend retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: share rskafka retry hint mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-06-17 08:27:53 +00:00
jeremyhi fd6c9e710d feat(cli): allow overriding import-v2 state path (#8302) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-16 09:53:19 +00:00
jeremyhi c22f5d741b feat(cli): add import-v2 task parallelism (#8300) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-16 02:55:52 +00:00
jeremyhi 90606af070 feat(cli): add export-v2 progress reporting (#8294) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-15 09:56:31 +00:00
Weny Xu 15597047c9 feat: pass Kafka pruned entry id when creating regions (#8282) 
* refactor: use typed region wal options

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: pass kafka pruned entry id to regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove unused error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: clarify wal options serialization errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover legacy region wal options encoding

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: recover legacy create table wal options

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: lock remote wal during table creation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests-s

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: refresh remote wal prune hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-06-15 08:51:50 +00:00
jeremyhi 336e4cdb84 feat(cli): add export-v2 chunk parallelism (#8292) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-15 06:00:53 +00:00
jeremyhi 34c4cae9bb feat(cli): add import-v2 progress UI (#8289) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-12 09:45:45 +00:00
jeremyhi c567a6562a feat(cli): add import-v2 progress mode (#8283) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-12 06:13:27 +00:00
jeremyhiandfengjiachun 0a2dc6d3f2 feat(cli): add export import progress abstraction (#8270) 
Signed-off-by: fengjiachun <3860496+fengjiachun@users.noreply.github.com>
Co-authored-by: fengjiachun <3860496+fengjiachun@users.noreply.github.com>
 2026-06-10 11:35:32 +00:00
jeremyhi 6ceae4498f refactor(cli): stream export-v2 verify data listing (#8268) 
Signed-off-by: fengjiachun <fengjiachun@gmail.com>
 2026-06-10 02:59:27 +00:00
jeremyhi aaffe1a048 refactor(cli): split export-v2 verify into plan/scan/reconcile (#8206) 
* refactor(cli): split export-v2 verify into plan/scan/reconcile

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-06-03 03:10:10 +00:00
jeremyhi c84462bdc1 feat(cli): add export-v2 delete command (#8162) 
* feat(cli): add export-v2 celete command

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat(cli): refine delete confirmation flag

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-05-26 04:28:23 +00:00
jeremyhi 72434ee5d7 feat: add export-v2 snapshot verification (#8111) 
* feat: add export-v2 snapshot verification

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject full export snapshots without chunks

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: enforce strict export snapshot data mainifest

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-05-19 07:15:51 +00:00
jeremyhi 0518567e5a feat: add export-v2 snapshot listing (#8096) 
* feat: add export-v2 snapshot listing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: allow export-v2 list at storage roots

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-05-13 03:04:04 +00:00
jeremyhi d709fd29ef feat: import resume part2 (#8070) 
* feat: import resume part2

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: condig docs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-05-12 02:56:52 +00:00
shuiyisong bdaad9db77 refactor: extract functions for building mysql/pg's kvbackend and electionref (#8076) 
chore: extract functions for building mysql/pg's kvbackend and electionref

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
 2026-05-07 08:48:49 +00:00
jeremyhi 266734d7ef fix: win cli lock (#8049) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-04-29 10:02:06 +00:00
shuiyisongandevenyag 0effc30778 chore: update the opendal to 0.56 rc2 (#8003) 
* chore: update opendal version

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update opendal version

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: grpc init

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: dep versions

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: remove aws-lc-rs in reqwest

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: rebase main and fix compile

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: remove unused deps

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* Revert "fix: remove aws-lc-rs in reqwest"

This reverts commit 90bfafca06f877befb36f3e54bb72fcfc4c56778.

* chore: remove aws-lc-sys from blacklist

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add tls deps

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: idemptent install in rds

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: use aws-lc-sys as possible

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: lint

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: address comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: address CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: sync opendal compat adapter with upstream

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: address compat clippy warnings

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
 2026-04-26 09:59:48 +00:00
jeremyhi 10f3f59a01 test(cli): harden import state lock test (#8030) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-04-24 06:38:05 +00:00
jeremyhi e316797fff feat(cli): add retry and import state fundations (#8007) 
* feat(cli): add retry and import state fiundations

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by human comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-04-23 04:38:36 +00:00
Weny Xu 39f9c7b4a4 feat(cli): add metadata put key and table commands (#7989) 
* feat: add validated metadata put cli

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add table metadata put commands

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: share metadata table selector

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(cli): read metadata put values from stdin

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-04-23 04:34:05 +00:00
Weny Xu 24e3364142 fix: relax azblob validation requirements (#7970) 2026-04-15 08:08:39 +00:00
jeremyhi e3f7ea8783 feat(cli): implement import-v2 data import pipeline (#7898) 
* feat(cli): implement import-v2 data import pipeline

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: cargo fmt

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(cli): harden import-v2 snapshot validation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: excape sql

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(cli): redact escaped secrets in copy sql logs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cli): tighten export v2 e2e helpers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(cli): log execution time

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-04-14 03:15:36 +00:00
jeremyhi f0ea87f52f fix: windows ci (#7905) 
* fix: windows ci

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: typo

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: use common create_temp_dir

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-04-03 02:17:42 +00:00
jeremyhi ab10696657 feat: implement export-v2 chunked data export flow (#7841) 
* feat: implement export-v2 chunked data export flow

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by codex comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by gemini comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clippy

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: handle empty export ranges consistently

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate resume config

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: file-uri paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: check args on schema-only mode

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-03-31 22:43:40 +00:00
Ning Sun e14404c677 chore: update rust toolchain to 2026-03-21 (#7849) 
* chore: update rust toolchain to 2026-03-21

* chore: new format

* fix: lint

* chore: resolve lint issues

* chore: remove as_millis_f64

* chore: deps up
 2026-03-30 12:13:14 +00:00
jeremyhi 805536aed1 fix: windows file path (#7839) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-03-20 08:19:41 +00:00
jeremyhi 16fcbb2729 feat: export import v2 pr1 (#7785) 
* feat: v2 schema handling

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: impl m1.5 ddl export/import and schema tests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: git ignore update

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: add license header

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: make fmt-check happy

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: Run imported DDL against the intended schema

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: Canonicalize schema names after case-insensitive check

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: escape sql funcs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: Fixed by carrying explicit execution_schema in DdlStatement instead of parsing schema from SQL

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: Fixed by encoding schema names as safe path segments in shared DDL path helpers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(cli): make export/import v2 schema recovery DDL-only

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by clippy

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: follow our styling

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(cli): reject remote snapshot URIs with empty root

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(cli): dedupe schema filters after canonicalization

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(cli): schema-scoped detection to cover external tables

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-03-19 21:26:41 +00:00
Weny Xu b17947b5d8 feat: add COPY support for GCS/Azblob with OSS/GCS/Azblob integration coverage (#7743) 
* feat: add gcs and azblob backends to copy datasource

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add copy to/from oss integration coverage

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add copy to/from gcs/azure integration coverage

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: use existing layers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: hide `credential_path` in cli

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-03-04 04:21:21 +00:00
dependabot[bot]andNing Sun b0fb4abbdf chore(deps): bump tracing-subscriber from 0.3.19 to 0.3.20 (#7720) 
* chore(deps): bump tracing-subscriber from 0.3.19 to 0.3.20

Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.19 to 0.3.20.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.19...tracing-subscriber-0.3.20)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-version: 0.3.20
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: remove ansi term depdenncy

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
 2026-02-25 23:56:55 +00:00
Weny Xu df04267c54 fix(repartition): reject writes on deallocating regions during region merge (#7694) 
* feat(meta): add write route policy to region route with backward compatibility

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): use partition_expr compatibility accessor in repartition matching

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): introduce staging partition rule enum for repartition instructions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(datanode): plumb staging partition rule enum through heartbeat handlers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): mark pending-deallocate regions as reject-all during merge staging

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(partition): exclude reject-all regions from write partitioning

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): store staging partition rule enum in region state

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): reject writes in staging when partition rule is reject-all

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): send enter staging instruction with reject-all

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(repartition): preserve reject-all on exit, merge enter-staging instructions, and allow staged bulk writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: refactor to ignore all writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: rename StagingPartitionRule to StagingPartitionDirective across staging flow

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: nit

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: rename

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-02-25 07:04:38 +00:00
LFC 8c23b29725 refactor: remove the RawTableMeta and RawTableInfo to make codes more concise (#7626) 
* refactor: remove the `RawTableMeta` and `RawTableInfo` to make codes more concise

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
 2026-02-10 07:10:04 +00:00
LFCandgemini-code-assist[bot] fcfab97d8c feat: cli tool to repair partition columns mismatch (#7605) 
* feat: cli tool to repair partition columns mismatch

Signed-off-by: luofucong <luofc@foxmail.com>

* Apply suggestions from code review

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* fix: harden partition column repair

Signed-off-by: MichaelScofield <MichaelScofield@users.noreply.github.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: MichaelScofield <MichaelScofield@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-02-09 02:58:25 +00:00
fys 8f48d233ef fix: unit tests for export module (#7607) 
fix: unit test about export
 2026-01-23 06:11:33 +00:00
Weny Xu 25687bb282 feat: add ddl timeout/wait options, repartition WITH parsing, meta-client startup refactor (#7589) 
* feat: add ddl request timeouts and unify meta client startup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: omplement ALTER TABLE repartition DDL options parsing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add sqlness tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: pass timeout argument to procedure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: refine comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: assert timeout

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-20 09:26:53 +00:00
Weny Xu 4e35028e28 test(cli): fix snapshot test paths on Windows (#7574) 
* test(cli): fix snapshot test paths on Windows

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix bugs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: styling

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-16 06:54:24 +00:00
Weny Xu ed171d9264 feat(cli): support RaftEngine store backend for metadata snapshot operations (#7467) 
* feat(cli): support RaftEngine store backend for metadata snapshot operations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: allow empty aksk

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-14 02:24:36 +00:00
Weny Xu f3e2d333e4 feat(repartition): implement region allocation for repartition procedure (#7534) 
* refactor: rename WalOptionsAllocator to WalProvider

The name "WalOptionsAllocator" was misleading because:
- For RaftEngine variant, it doesn't actually allocate anything
- The actual allocation logic lives in KafkaTopicPool

"WalProvider" better describes its role as providing WAL options
based on the configured WAL backend (RaftEngine or Kafka).

Changes:
- Rename `WalOptionsAllocator` to `WalProvider`
- Rename `WalOptionsAllocatorRef` to `WalProviderRef`
- Rename `build_wal_options_allocator` to `build_wal_provider`
- Rename module `wal_options_allocator` to `wal_provider`
- Rename error types: `BuildWalOptionsAllocator` -> `BuildWalProvider`,
  `StartWalOptionsAllocator` -> `StartWalProvider`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): extract allocator traits from TableMetadataAllocator

Refactor TableMetadataAllocator to use trait-based dependency injection
for better testability and separation of concerns.

Changes:
- Add `ResourceIdAllocator` trait to abstract ID allocation
- Add `WalOptionsAllocator` trait to abstract WAL options allocation
- Implement traits for `Sequence` and `WalProvider`
- Remove duplicate `allocate_region_wal_options` function
- Rename `table_id_sequence` to `table_id_allocator` for consistency
- Rename `TableIdSequenceHandler` to `TableIdAllocatorHandler`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): add max_region_number tracking to PhysicalTableRouteValue

Add `max_region_number` field to track the highest region number ever
allocated for a table. This value only increases when regions are added
and never decreases when regions are dropped, ensuring unique region
numbers across the table's lifetime.

Changes:
- Add `max_region_number` field to `PhysicalTableRouteValue`
- Implement custom `Deserialize` for backward compatibility
- Update `update_region_routes` to maintain max_region_number
- Calculate max_region_number from region_routes in `new()`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: extract TableRouteAllocator trait from TableMetadataAllocator

- Add TableRouteAllocator trait for abstracting region route allocation
- Implement blanket impl for all PeerAllocator types
- Add PeerAllocator impl for Arc<T> to support trait object delegation
- Update TableMetadataAllocator to use TableRouteAllocatorRef

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: rename TableRouteAllocator to RegionRoutesAllocator

- Rename table_route.rs to region_routes.rs
- Rename TableRouteAllocator trait to RegionRoutesAllocator
- Rename wal_option.rs to wal_options.rs for consistency
- Update TableMetadataAllocator to use new naming

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta-srv): implement region allocation for repartition procedure

This commit implements the region allocation phase of the repartition procedure,
which handles allocating new regions when a table needs to be split into more partitions.

Key changes:
- Refactor `RegionRoutesAllocator::allocate` to accept `(region_number, partition_expr)` tuples
  for more flexible region number assignment
- Simplify `AllocationPlanEntry` by removing `regions_to_allocate` and `regions_to_deallocate`
  fields (now derived from source/target counts)
- Add `convert_allocation_plan_to_repartition_plan` function to handle allocation, equal,
  and deallocation cases
- Fix `RepartitionPlanEntry::allocate_regions()` to return target regions (was incorrectly
  returning source regions)
- Implement complete `AllocateRegion` state with:
  - Region route allocation via `RegionRoutesAllocator`
  - WAL options allocation via `WalOptionsAllocator`
  - Operating region registration for concurrency control
  - Region creation on datanodes via `CreateTableExecutor`
  - Table route metadata update
- Add `TableRouteValue::max_region_number()` helper method
- Add comprehensive unit tests for plan conversion and allocation logic

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-08 11:03:58 +00:00
Weny Xu ada4666e10 refactor: remove region_numbers from TableMeta and TableInfo (#7519) 
* refactor: remove `region_numbers` from `TableMeta` and `TableInfo`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: create partitions from region route

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix build

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-06 13:21:36 +00:00
Weny Xu 9343da7fe8 feat(meta-srv): fallback to non-TLS connection when etcd TLS prefer mode fail (#7507) 
* feat(meta-srv): fallback to non-TLS connection when etcd TLS prefer mode fail

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore(ci): set timeout for deploy cluster

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: simplify etcd TLS prefer mode handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-12-31 10:03:34 +00:00
Lanqing Yang 8d9afc83e3 feat: allow auto schema creation for pg (#7459) 
Signed-off-by: lyang24 <lanqingy93@gmail.com>
 2025-12-23 08:55:24 +00:00
Weny Xu 4d66bd96b8 feat: make distributed time constants and client timeouts configurable (#7433) 
Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-12-19 02:23:20 +00:00