mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-09-07 22:18:57 +00:00
+1
c60311973d
c60311973d
* fix(mito2): fence checkpoints during region transitions (#8847) * fix: fence checkpoints during region transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): fix transient downgrade setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): fix checkpoint lifecycle test setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): cover cancelled downgrade waiter retry Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): fence direct follower transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test: trim checkpoint transition coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): clarify checkpoint task lifecycle Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commita502dfdefd) * fix(mito2): split SSTs at primary key series boundaries (#8888) * fix(mito2): split SSTs at series boundaries Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover SST splitting without primary key Also document the sortedness precondition and the series boundary split semantics on write_all_flat/write_all_flat_as_primary_key and the new split helpers. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): avoid per-row Vec allocation for empty primary key Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Update src/mito2/src/sst/parquet/writer.rs Co-authored-by: Yingwen <realevenyag@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> (cherry picked from commit4ac3423261) * fix(meta): allow manual migration from offline datanodes (#8934) * fix(meta): allow migration from offline datanodes Signed-off-by: WenyXu <wenymedia@gmail.com> * test: fix offline migration event actor Signed-off-by: WenyXu <wenymedia@gmail.com> * test: read migration routes from metadata Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit8a473c5bf0) * fix(flight): bound DoGet response wait (#8943) * fix(flight): defer datanode query initialization Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): retain Flight stream peer context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): improve Flight stream diagnostics Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit28398138ec) * fix(query): keep INSERT timestamp conversion out of the source query (#8911) * fix(query): keep INSERT timestamp conversion out of the source query Interpreting an INSERT's string timestamps used to work by pushing the conversion down into the source query, which changed what that query means. Two consequences: - Pushing through a UNION's DISTINCT moved the dedup key from the raw strings to parsed instants, so rows spelling the same instant differently collapsed into one. On an append-only table that is a silently dropped row. - A UNION branch that needed no conversion (a NULL, or an explicit cast) made the whole column give up, leaving sibling branches on UTC while the rest of the row used the session timezone. Convert at the assignment instead, by routing its cast through a timezone-carrying timestamp type and back. Arrow applies the timezone when a cast target carries one, and stripping it afterwards preserves the value. The source query is no longer touched, so both cases go away and the tree-walking rewrite (roughly 160 lines) is deleted. The rewrite reads source types, so it now runs TypeCoercion first: a UNION still carries its loose per-branch schema before coercion, and retargeting a cast whose input later becomes a timestamp would shift the value rather than reinterpret it. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): address review on INSERT assignment rewrite - Clone the input `Arc` instead of the whole subtree, and only rebuild it when a `Values` row actually changes. - Defer cloning the cast source until the literal-folding path has been ruled out. - Move the UTC check onto `Timezone::is_utc`, replacing a bare string compare. - Cover a prepared `INSERT ... VALUES (?)`: an untyped placeholder types as `Null`, so the assignment cast is left for parameter substitution. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> (cherry picked from commit1851f6bf4d) * test: renew etcd TLS certificates (#8956) Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit144f83528d) * fix: update tokio-postgres and correct explain/fetch cursor output schema (#8955) * chore(deps): update tokio-postgres * fix: describing fetch cursor and analyze (cherry picked from commitb31f05eb59) Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): avoid insert select HTTP/2 stalls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commitd4a3d88cd7) * fix(query): share record batch forwarding for DML Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commitc34bec197c) * fix(mysql): interpret prepared statement datetime params in session timezone (#8923) * fix(mysql): interpret prepared statement datetime params in session timezone Binary DATETIME parameters of server-side prepared statements were converted as if UTC, ignoring the session timezone set via SET time_zone. Convert them with the session timezone and add an integration test covering prepared inserts and predicates under Asia/Shanghai. Signed-off-by: wy471x <wy471x@gmail.com> * refactor: share naive datetime timezone policy via common-time Address review feedback on the prepared-statement timezone fix: - Expose Timestamp::from_naive_datetime in common-time so the DST policy (gap -> error, ambiguous -> earlier instant) lives in one place, shared by the text protocol (Timestamp::from_str) and the MySQL binary protocol. - Route the MySQL prepared-statement datetime conversion through it. - Match the target type before converting datetime params so PreparedStmtTypeMismatch fails fast without wasted conversion. - Use the short Timezone import form for consistency with the rest of servers. Signed-off-by: wy471x <wy471x@gmail.com> --------- Signed-off-by: wy471x <wy471x@gmail.com> Co-authored-by: Ning Sun <sunng@protonmail.com> (cherry picked from commitaaa843104b) * fix(mito2): use target sequence for foreign SSTs (#8946) * fix(mito2): use target sequence for foreign SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): address foreign SST review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commitbd7d2c1dfa) * fix: postgres describe for more statements (#8974) * fix: postgres describe for more statements Signed-off-by: Ning Sun <sunning@greptime.com> * fix: cover more show statements Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments - add missing `clippy::too_many_arguments` allow on `query_from_information_schema_dataframe` (CI clippy failure) - take `&ShowKind` in the information-schema dataframe helper so `kind` is no longer cloned at every call site; only the WHERE arm (which needs an owned expression for `sql_to_expr`) clones internally - document why re-applying TQL explain formats never overwrites an existing value (per-query context state) Signed-off-by: Ning Sun <sunning@greptime.com> * chore: trim comments to essentials Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> (cherry picked from commitd32cd77505) * fix(promql): resolve derived labels in aggregation arithmetic (#8994) Signed-off-by: shuiyisong <xixing.sys@gmail.com> (cherry picked from commitc4dafb5826) * fix(deps): narrow postgres lockfile update Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(deps): normalize remaining x509 lock reference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(client): isolate query and control transports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(frontend): clarify adaptive window purpose Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(grpc): satisfy clippy in transport isolation coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(pipeline): coalesce concurrent pipeline cache misses (#9022) Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * Revert "test(grpc): satisfy clippy in transport isolation coverage" This reverts commit888bcedc96. (cherry picked from commit e0216b4074485bcffaae53348a17091532cb3600) Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * Revert "docs(frontend): clarify adaptive window purpose" This reverts commit84cf69d099. (cherry picked from commit da30e605ad7ebfb2a3fa86ea44b67405e5afde98) Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * Revert "fix(client): isolate query and control transports" This reverts commita1d2dbb050. (cherry picked from commit 03f5c31f544d3d3852beddfa87cff34d04a646ae) Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(json2): add bounded auto-expansion to the JSON2 vector builder (#8909) * refactor(json2): add bounded auto-expansion to the JSON2 vector builder Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> (cherry picked from commit76f08d2b3f) (cherry picked from commit 640fa63bd8ee098fac8e8d62d6fb7ca5af4ddcd0) * refactor(json2): optimize JSON2 building without auto-expanded paths (#8928) * refactor(json2): optimize JSON2 building without auto-expanded paths Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * avoid panicking memtable write Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> (cherry picked from commit7fd0a7bb98) (cherry picked from commit 63c68ee60101e2a09e14420051f8f889b41a2a3b) * refactor(json2): support querying v2 storage layout (#8940) * feat(json2): support querying v2 storage layout - route missing JSON2 paths to the v2 remainder field - reconstruct complete values from explicit fields and remainder data - preserve root JSON2 columns across projections and filters - support nested JSON values in json_get string results - add and reorganize JSON2 sqlness coverage Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> (cherry picked from commit932f87f7a8) (cherry picked from commit 5af1394a1f319bb99cafa9474b8a5a075be1e805) * feat(pipeline): support table-aware JSON2 transforms (#8964) * feat(pipeline): support table-aware JSON2 transforms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat(pipeline): support JSON2 type hints in transforms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(pipeline): default failed JSON2 transforms to null Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(json2): distinguish invalid settings from layout errors Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> (cherry picked from commitfb86f6573e) (cherry picked from commit 18a21a9ce017b6887c857a3f3bbb9a83aa7f968b) * refactor: json2 v2 storage layout (#8979) * refactor: json2 v2 storage layout Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> * rethinking when "needs_remainder" Signed-off-by: luofucong <luofc@foxmail.com> * restore "ReadColumns" Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> (cherry picked from commit529f046110) (cherry picked from commit 7628d795197c786872730c3a5ef36c5c662d4322) * feat(json2): support JSON2 paths in SQL functions (#9007) feat(query): support JSON2 paths in SQL functions Update the DataFusion fork to expose scalar function planning hooks. Infer JSON2 path output types from scalar, aggregate, and window function signatures, while preserving the default Utf8View behavior for functions that accept arbitrary inputs. Add unit and sqlness coverage for type conflicts, mixed typed and untyped JSON paths, filters, aggregates, and window functions. Signed-off-by: luofucong <luofc@foxmail.com> (cherry picked from commit15317a131b) (cherry picked from commit a945cbc7a4398e6bee74d4d2c67528772faf398d) * feat(json2): support empty and null JSON2 value (#9010) * feat(json2): support empty and null JSON2 value Signed-off-by: fys <fengys1996@gmail.com> * test(json2): cover explicit NULL and omitted-column inserts Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo fmt Signed-off-by: fys <fengys1996@gmail.com> * fix: infer empty JSON object as object type Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> (cherry picked from commit05c65f54a8) (cherry picked from commit 05c27452ea5e958fab85e0ba4b82f5df205a5d08) * feat(json2): support list indexing for JSON2 columns (#9013) feat(query): support list indexing for JSON2 columns Extend JSON2 paths through DataFusion field-access planning, including nested list indexes and object fields following an index. Preserve Variant reads for bracket JSONPath expressions and normalize dot accesses after subscripts to work around the current DataFusion planner limitation. Add unit and sqlness coverage for nested indexes, type conflicts, missing paths, flushes, and compacted SSTs. Signed-off-by: luofucong <luofc@foxmail.com> (cherry picked from commitd62a5a990a) (cherry picked from commit c11d8647feb30ecdd5ba1bc8ed4df78999801672) * refactor(json2): concretize JSON2 schemas at merge scan boundaries (#9016) * refactor(json2): concretize JSON2 schemas at merge scan boundaries Infer concrete JSON2 output types from remote plans and expose them on MergeScanLogicalPlan before physical planning. Recompute affected local schemas and remove the JSON2-specific rewrite from MergeScanExec. Add SQLness coverage for whole JSON2 columns in windows and joins. Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> (cherry picked from commit84bd993131) (cherry picked from commit 51c98a7e31a929fc33ff5133714bb47108f7539b) * fix(json2): keep empty structs in remainder (#9027) Signed-off-by: luofucong <luofc@foxmail.com> (cherry picked from commitd67d3501a9) (cherry picked from commit b3f2c1cc5bf0cfddb7c0670ac8f3522eb67c35a0) * fix(mito2): remove stale scan test imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit d63bbe6aef298f5b8e960bb02eed08743a6684ee) * test(query): expect bounded merge scan partitions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit 7bcef46b1c010d2e73ec49be7581c216e6b5b1b6) * style(mito2): remove trailing whitespace Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(sqlness): refresh bounded partition plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(sqlness): normalize CPU-dependent partition plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: wy471x <wy471x@gmail.com> Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: luofucong <luofc@foxmail.com> Signed-off-by: fys <fengys1996@gmail.com> Co-authored-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Lei, HUANG <ratuthomm@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: Ning Sun <sunng@protonmail.com> Co-authored-by: wy471x <wy471x@gmail.com> Co-authored-by: Ning Sun <sunning@greptime.com> Co-authored-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: luofucong <luofc@foxmail.com> Co-authored-by: fys <fengys1996@gmail.com>