fix: re-scan stream-backed tables in recursive CTEs (#9039)
* fix: re-scan stream-backed tables in recursive CTEs
A recursive CTE re-executes its recursive term on every iteration, but
DfTableProviderAdapter hands StreamScanAdapter a single-use stream built at
planning time. The second iteration failed with "Stream already exhausted"
for every table served through DataSource::get_stream — information_schema,
pg_catalog, the computed entity-graph tables and numbers.
Keep that stream for the first execution and open a new one over the same
scan request for later executions.
Closes#9037
* refactor: drop redundant binding in stream factory
---------
(cherry picked from commit bb9b7e8778)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* fix(mito2): fence checkpoints during region transitions (#8847)
* fix: fence checkpoints during region transitions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(datanode): fix transient downgrade setup
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(mito2): fix checkpoint lifecycle test setup
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(mito2): cover cancelled downgrade waiter retry
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): fence direct follower transitions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: trim checkpoint transition coverage
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(mito2): clarify checkpoint task lifecycle
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit a502dfdefd)
* fix(mito2): split SSTs at primary key series boundaries (#8888)
* fix(mito2): split SSTs at series boundaries
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover SST splitting without primary key
Also document the sortedness precondition and the series boundary
split semantics on write_all_flat/write_all_flat_as_primary_key and
the new split helpers.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): avoid per-row Vec allocation for empty primary key
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Update src/mito2/src/sst/parquet/writer.rs
Co-authored-by: Yingwen <realevenyag@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
(cherry picked from commit 4ac3423261)
* fix(meta): allow manual migration from offline datanodes (#8934)
* fix(meta): allow migration from offline datanodes
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: fix offline migration event actor
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: read migration routes from metadata
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 8a473c5bf0)
* fix(flight): bound DoGet response wait (#8943)
* fix(flight): defer datanode query initialization
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(client): retain Flight stream peer context
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(client): improve Flight stream diagnostics
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 28398138ec)
* fix(query): keep INSERT timestamp conversion out of the source query (#8911)
* fix(query): keep INSERT timestamp conversion out of the source query
Interpreting an INSERT's string timestamps used to work by pushing the
conversion down into the source query, which changed what that query
means. Two consequences:
- Pushing through a UNION's DISTINCT moved the dedup key from the raw
strings to parsed instants, so rows spelling the same instant
differently collapsed into one. On an append-only table that is a
silently dropped row.
- A UNION branch that needed no conversion (a NULL, or an explicit cast)
made the whole column give up, leaving sibling branches on UTC while
the rest of the row used the session timezone.
Convert at the assignment instead, by routing its cast through a
timezone-carrying timestamp type and back. Arrow applies the timezone
when a cast target carries one, and stripping it afterwards preserves
the value. The source query is no longer touched, so both cases go away
and the tree-walking rewrite (roughly 160 lines) is deleted.
The rewrite reads source types, so it now runs TypeCoercion first: a
UNION still carries its loose per-branch schema before coercion, and
retargeting a cast whose input later becomes a timestamp would shift the
value rather than reinterpret it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): address review on INSERT assignment rewrite
- Clone the input `Arc` instead of the whole subtree, and only rebuild it
when a `Values` row actually changes.
- Defer cloning the cast source until the literal-folding path has been
ruled out.
- Move the UTC check onto `Timezone::is_utc`, replacing a bare string
compare.
- Cover a prepared `INSERT ... VALUES (?)`: an untyped placeholder types
as `Null`, so the assignment cast is left for parameter substitution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 1851f6bf4d)
* test: renew etcd TLS certificates (#8956)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 144f83528d)
* fix: update tokio-postgres and correct explain/fetch cursor output schema (#8955)
* chore(deps): update tokio-postgres
* fix: describing fetch cursor and analyze
(cherry picked from commit b31f05eb59)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): avoid insert select HTTP/2 stalls
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit d4a3d88cd7)
* fix(query): share record batch forwarding for DML
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit c34bec197c)
* fix(mysql): interpret prepared statement datetime params in session timezone (#8923)
* fix(mysql): interpret prepared statement datetime params in session timezone
Binary DATETIME parameters of server-side prepared statements were
converted as if UTC, ignoring the session timezone set via SET time_zone.
Convert them with the session timezone and add an integration test
covering prepared inserts and predicates under Asia/Shanghai.
Signed-off-by: wy471x <wy471x@gmail.com>
* refactor: share naive datetime timezone policy via common-time
Address review feedback on the prepared-statement timezone fix:
- Expose Timestamp::from_naive_datetime in common-time so the DST policy
(gap -> error, ambiguous -> earlier instant) lives in one place, shared
by the text protocol (Timestamp::from_str) and the MySQL binary protocol.
- Route the MySQL prepared-statement datetime conversion through it.
- Match the target type before converting datetime params so
PreparedStmtTypeMismatch fails fast without wasted conversion.
- Use the short Timezone import form for consistency with the rest of servers.
Signed-off-by: wy471x <wy471x@gmail.com>
---------
Signed-off-by: wy471x <wy471x@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
(cherry picked from commit aaa843104b)
* fix(mito2): use target sequence for foreign SSTs (#8946)
* fix(mito2): use target sequence for foreign SSTs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore(mito2): address foreign SST review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit bd7d2c1dfa)
* fix: postgres describe for more statements (#8974)
* fix: postgres describe for more statements
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: cover more show statements
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
- add missing `clippy::too_many_arguments` allow on
`query_from_information_schema_dataframe` (CI clippy failure)
- take `&ShowKind` in the information-schema dataframe helper so `kind`
is no longer cloned at every call site; only the WHERE arm (which needs
an owned expression for `sql_to_expr`) clones internally
- document why re-applying TQL explain formats never overwrites an
existing value (per-query context state)
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: trim comments to essentials
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
(cherry picked from commit d32cd77505)
* fix(promql): resolve derived labels in aggregation arithmetic (#8994)
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit c4dafb5826)
* fix(deps): narrow postgres lockfile update
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(deps): normalize remaining x509 lock reference
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(client): isolate query and control transports
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(frontend): clarify adaptive window purpose
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(grpc): satisfy clippy in transport isolation coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(pipeline): coalesce concurrent pipeline cache misses (#9022)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "test(grpc): satisfy clippy in transport isolation coverage"
This reverts commit 888bcedc96.
(cherry picked from commit e0216b4074485bcffaae53348a17091532cb3600)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "docs(frontend): clarify adaptive window purpose"
This reverts commit 84cf69d099.
(cherry picked from commit da30e605ad7ebfb2a3fa86ea44b67405e5afde98)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "fix(client): isolate query and control transports"
This reverts commit a1d2dbb050.
(cherry picked from commit 03f5c31f544d3d3852beddfa87cff34d04a646ae)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(json2): add bounded auto-expansion to the JSON2 vector builder (#8909)
* refactor(json2): add bounded auto-expansion to the JSON2 vector builder
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 76f08d2b3f)
(cherry picked from commit 640fa63bd8ee098fac8e8d62d6fb7ca5af4ddcd0)
* refactor(json2): optimize JSON2 building without auto-expanded paths (#8928)
* refactor(json2): optimize JSON2 building without auto-expanded paths
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* avoid panicking memtable write
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 7fd0a7bb98)
(cherry picked from commit 63c68ee60101e2a09e14420051f8f889b41a2a3b)
* refactor(json2): support querying v2 storage layout (#8940)
* feat(json2): support querying v2 storage layout
- route missing JSON2 paths to the v2 remainder field
- reconstruct complete values from explicit fields and remainder data
- preserve root JSON2 columns across projections and filters
- support nested JSON values in json_get string results
- add and reorganize JSON2 sqlness coverage
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 932f87f7a8)
(cherry picked from commit 5af1394a1f319bb99cafa9474b8a5a075be1e805)
* feat(pipeline): support table-aware JSON2 transforms (#8964)
* feat(pipeline): support table-aware JSON2 transforms
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* feat(pipeline): support JSON2 type hints in transforms
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(pipeline): default failed JSON2 transforms to null
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* refactor(json2): distinguish invalid settings from layout errors
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit fb86f6573e)
(cherry picked from commit 18a21a9ce017b6887c857a3f3bbb9a83aa7f968b)
* refactor: json2 v2 storage layout (#8979)
* refactor: json2 v2 storage layout
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
* rethinking when "needs_remainder"
Signed-off-by: luofucong <luofc@foxmail.com>
* restore "ReadColumns"
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 529f046110)
(cherry picked from commit 7628d795197c786872730c3a5ef36c5c662d4322)
* feat(json2): support JSON2 paths in SQL functions (#9007)
feat(query): support JSON2 paths in SQL functions
Update the DataFusion fork to expose scalar function planning hooks.
Infer JSON2 path output types from scalar, aggregate, and window function signatures, while preserving the default Utf8View behavior for functions that accept arbitrary inputs.
Add unit and sqlness coverage for type conflicts, mixed typed and untyped JSON paths, filters, aggregates, and window functions.
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 15317a131b)
(cherry picked from commit a945cbc7a4398e6bee74d4d2c67528772faf398d)
* feat(json2): support empty and null JSON2 value (#9010)
* feat(json2): support empty and null JSON2 value
Signed-off-by: fys <fengys1996@gmail.com>
* test(json2): cover explicit NULL and omitted-column inserts
Signed-off-by: fys <fengys1996@gmail.com>
* fix: cargo fmt
Signed-off-by: fys <fengys1996@gmail.com>
* fix: infer empty JSON object as object type
Signed-off-by: fys <fengys1996@gmail.com>
---------
Signed-off-by: fys <fengys1996@gmail.com>
(cherry picked from commit 05c65f54a8)
(cherry picked from commit 05c27452ea5e958fab85e0ba4b82f5df205a5d08)
* feat(json2): support list indexing for JSON2 columns (#9013)
feat(query): support list indexing for JSON2 columns
Extend JSON2 paths through DataFusion field-access planning, including nested list indexes and object fields following an index.
Preserve Variant reads for bracket JSONPath expressions and normalize dot accesses after subscripts to work around the current DataFusion planner limitation.
Add unit and sqlness coverage for nested indexes, type conflicts, missing paths, flushes, and compacted SSTs.
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit d62a5a990a)
(cherry picked from commit c11d8647feb30ecdd5ba1bc8ed4df78999801672)
* refactor(json2): concretize JSON2 schemas at merge scan boundaries (#9016)
* refactor(json2): concretize JSON2 schemas at merge scan boundaries
Infer concrete JSON2 output types from remote plans and expose them on MergeScanLogicalPlan before physical planning. Recompute affected local schemas and remove the JSON2-specific rewrite from MergeScanExec.
Add SQLness coverage for whole JSON2 columns in windows and joins.
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 84bd993131)
(cherry picked from commit 51c98a7e31a929fc33ff5133714bb47108f7539b)
* fix(json2): keep empty structs in remainder (#9027)
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit d67d3501a9)
(cherry picked from commit b3f2c1cc5bf0cfddb7c0670ac8f3522eb67c35a0)
* fix(mito2): remove stale scan test imports
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit d63bbe6aef298f5b8e960bb02eed08743a6684ee)
* test(query): expect bounded merge scan partitions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 7bcef46b1c010d2e73ec49be7581c216e6b5b1b6)
* style(mito2): remove trailing whitespace
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(sqlness): refresh bounded partition plans
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(sqlness): normalize CPU-dependent partition plans
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: wy471x <wy471x@gmail.com>
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: wy471x <wy471x@gmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
Co-authored-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: luofucong <luofc@foxmail.com>
Co-authored-by: fys <fengys1996@gmail.com>
* fix(query): follow timestamp insert assignment lineage
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): fold constant insert timestamp literals at the assignment
Following lineage by retyping the source column changed every output
column that reads it: a string column sharing the literal was silently
rewritten to a formatted timestamp, and a nanosecond column was
truncated to the precision of whichever column was converted first.
Resolve the constant read-only and fold it into the assignment
expression instead, which leaves the source query untouched and also
covers literals behind WHERE, ORDER BY and DISTINCT. VALUES rows and
UNION branches carry per-row values, so they keep the in-place rewrite,
now guarded against columns with more than one consumer.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): strengthen insert lineage regression
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): trim redundant insert coverage
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): rebuild insert unions loosely and cover UNION distinct
Per review: rebuilding a rewritten union with the strict constructor
rejected legal pre-coercion plans whose untouched columns still differ
across branches. Use try_new_with_loose_types, matching the SQL planner.
Distinct::All joins the rewrite passthrough so UNION (distinct) literals
get session-timezone parsing like UNION ALL; deduplication then keys on
parsed instants instead of raw strings. The top-level rewrite path gains
the same single-consumer guard as rewrite_projection for hand-built DML
plans that share a source column between targets.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* style(query): tighten insert assignment comments
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit b2fc4b9cba4c0e137cc6f704d88dfe7f187b57aa)
(cherry picked from commit 1775e31a7916cf09a7da8238cebea5e3d74817f7)
* fix(query): avoid unsafe count wildcard rewrites
Signed-off-by: discord9 <discord9@163.com>
* fix(query): preserve outer count alias
Signed-off-by: discord9 <discord9@163.com>
* fix(query): address review comments on count wildcard rewrite
- Remove the has_projection check: the row count is correct regardless
of whether a projection exists (per review).
- Explain why checking the first input is equivalent to checking all
inputs (a plan with zero inputs falls back to count(1)).
- Rename qa_ prefixed tests to follow the module convention.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(tql): update tql-cte expectations for count wildcard rewrite
The QP-026 count-wildcard fix rewrites count(*) -> count(time_index), so
the EXPLAIN output for the filtered/final CTE aggregates names the
time-index column. Aligns tql-cte.result with the actual output (CI
failure).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 7223f2a9b80a476448e3bded312e072cd1585e2a)
* chore: gate soft-drop table behind the enterprise feature
Soft-drop table becomes an enterprise-only feature:
- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
registration, and information_schema.recycle_bin registration are
compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
(moving to the enterprise repo); affected information_schema .result
files are regenerated
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: limit unused_variables allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: include the config key in the soft-drop enterprise gate error
Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: limit unused_mut allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: reject soft-drop DDL submissions in non-enterprise builds
Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: stop --enable-gc from enabling soft drop in the sqlness template
Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate fresh soft-drop procedures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: gate soft-drop fallback coverage
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate soft-drop procedure implementation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate drop table soft-drop behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate expired soft-drop gc behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: test enterprise table ddl lifecycle
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark purge_table as enterprise licensed
The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark recycle_bin as enterprise licensed
The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark soft-drop procedure sources as enterprise licensed
The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support full WAL retirement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: complete close request migration
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: guard Kafka provider setup behind index collector check
Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.
Affected file:
- `src/log-store/src/kafka/log_store.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: avoid to_vec
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: clean up soft-dropped regions offline
Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.
Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): reject file-engine soft drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: preserve soft-drop cleanup split state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove obsolete CleanUp match arm from RegionRequest
The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): clean every soft-dropped region replica
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): order soft-drop replica cleanup
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): order soft-drop replica cleanup"
This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): clean every soft-dropped region replica"
This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(catalog): expose soft drops in recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(sql): add UNDROP TABLE
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sql): cover UNDROP TABLE execution
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(sql): keep successful UNDROP result
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: reject stale undrop by name-based lookup after tombstone consumed
When a table is dropped, recreated under the same name (consuming the name
tombstone), and then the recreated table is dropped, an undrop procedure
that was built before the first drop and holds a stale original table name
should fail with TableNotFound instead of silently matching a different
table.
Changed `UndropTableProcedure::on_prepare` to perform a name-based lookup
when `table_name` is available and filter by table ID, ensuring that a
dropped table can only be recovered when its name tombstone still maps to
the expected ID.
- `src/common/meta/src/ddl/undrop_table.rs`: name-first lookup in on_prepare
- `src/common/meta/src/ddl/tests/drop_table.rs`: test for the stale-id
rejection case
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(log-store): keep Kafka obsolete_all as no-op
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): hide purging tables from recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): scope recycle bin scans by catalog
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sqlness): update recycle bin expectations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: avoid redundant recycle bin allocations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: timestamp display precision should respect column schema (#8227)
Previously, the MySQL writer converted timestamps to NaiveDateTime via
to_chrono_datetime_with_timezone(), then passed the NaiveDateTime to
write_col(). NaiveDateTime::Display uses a fixed 6-digit fractional-
second format, causing:
- TIMESTAMP(3) to show '.195000' instead of '.195'
- TIMESTAMP(9) to show '.195123' instead of '.195123456'
Fix: use Timestamp::to_timezone_aware_string() directly, which formats
with chrono's '%.f' specifier — it strips trailing zeros and preserves
full nanosecond fidelity based on the actual stored unit.
Also add comprehensive unit tests in mysql_writer_test.rs covering:
- All concrete data type → MySQL column type mappings
- UNSIGNED_FLAG propagation for unsigned integer types
- Timestamp precision for all four units (0/3/6/9 decimal places)
- Edge cases: zero subseconds, trailing-zero stripping, Unix epoch,
negative timestamps, and timezone offset shifts
- Column-def ordering, empty schema, and decimal variants
Signed-off-by: Divyansh <anshmcs@gmail.com>
* perf: reuse format buffer for timestamp serialization to avoid per-row heap allocation
Signed-off-by: Divyansh <anshmcs@gmail.com>
* refactor: rename test variable and update visibility of create_mysql_column function
Signed-off-by: Divyansh <anshmcs@gmail.com>
* test: fix timestamp test constants and add sqlness case for display precision
- Correct the base epoch constant: 1_748_836_200 is 2025-06-02, not
2026-06-02; use 1_780_372_200 to match the documented instant.
- Fix subsecond expectations: chrono's %.f renders fractional digits in
groups of 3 (.100 / .010), it does not strip to .1 / .01.
- Add sqlness case timestamp_precision_display reproducing issue #8227
over the MySQL protocol (TIMESTAMP(0/3/6/9) rendering).
- cargo fmt reflow of create_mysql_column signature.
Signed-off-by: Divyansh <anshmcs@gmail.com>
* fix: support binary protocol for timestamp column
Signed-off-by: Divyansh <anshmcs@gmail.com>
* test: update sqlness results for new timestamp display precision
Signed-off-by: Divyansh <anshmcs@gmail.com>
---------
Signed-off-by: Divyansh <anshmcs@gmail.com>
* feat: support ALTER TABLE SET auto_flush_interval
Closes#8394.
Add a new SetRegionOption::AutoFlushInterval variant so that the
per-table auto flush interval can be changed on existing tables
via 'ALTER TABLE t SET ...', following up the CREATE TABLE path
from #8357.
- region_request.rs: parse 'auto_flush_interval' with humantime and
map it to the new variant.
- metadata.rs: persist the value (or remove it, if None) in
TableOptions.extra_options using the same humantime string format
the engine already expects.
- handle_alter.rs: apply the new interval in
handle_alter_region_options_fast (no memtable flush needed, same
pattern as Ttl) and group the variant with Ttl/Twsc in
new_region_options_on_empty_memtable.
Tests:
- Two unit tests in metadata.rs covering set and unset-to-None.
- A new sqlness case alter_auto_flush_interval.sql covering
create-then-alter, alter-then-alter, invalid duration, and
alter on a table that already had auto_flush_interval at create
time.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: validate auto_flush_interval > 0 in ALTER SET path
Gemini code assist flagged that the request parser accepted a zero
duration, leaving the rejection to the downstream RegionOptions
validation which only fires on next flush. Reject it at parse time
so users get the error immediately at the ALTER TABLE statement.
Also add a '0s' error case to the sqlness test.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: handle SET 'auto_flush_interval' = NULL and add checked-in .result
Address the rest of fengjiachun's review on #8403:
1. Empty value in ALTER SET clears the override (parallels Ttl).
'ALTER TABLE t SET ... = NULL' comes through as value = ''; we
now return AutoFlushInterval(None) so the override is removed
from TableOptions.extra_options, matching the Ttl pattern.
2. Add a unit test in region_request.rs covering the four cases
(valid, empty-clears, zero-rejected, garbage-rejected).
3. Generate and check in alter_auto_flush_interval.result via
'cargo sqlness-runner bare -t alter_auto_flush_interval'. Both
the standalone and distributed sqlness jobs now pass locally,
and the test extension covers the NULL-clears path end to end.
Signed-off-by: srivtx <crypticcc101@gmail.com>
---------
Signed-off-by: srivtx <crypticcc101@gmail.com>
* feat: report region read load in heartbeat
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: expose region query stats in information schema
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update sqlness result
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record region query stats on stream drop
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: keep region query cpu stats in nanoseconds
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: disable non object json write
* remove unused code
* fix: sqlness test
* simply code
* add comment
* remove unused unit test
* fix: cargo fmt
* fix: unit test
* fix: unit test
LFCanddennis zhuang
Morax
fys
Ruihang Xia
Div
shuiyisong
sribatsha dash
