* fix(mito): preserve mixed JSON2 types during compaction
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito): assert all restored JSON2 rows in unordered merge test
The regression test for aligning JSON2 layouts across unordered bulk
parts only round-tripped the first part's `a` values. A merge that
dropped or corrupted the second source's `b` values would still pass.
Assert the merged batch has four rows and that the restored values of
the second part (`{"b": 3}`, `{"b": 4}`) survive the merge, so the
test covers both opaque sources as its comment claims.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit 9696882217)
(cherry picked from commit 6ae3d829e4)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit e0c87713993b156c64c71d94940848afa87d54fb)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(pprof): switch CPU profiler to framehop unwinder
Replace the default libgcc-based unwinder in pprof-rs with the
framehop unwinder, which is designed to be async-signal-safe:
- framehop performs no heap allocation during unwinding
(MustNotAllocateDuringUnwind)
- It handles prologue/epilogue interruption correctly
- It falls back to frame-pointer unwinding when CFI is unavailable
- It does not depend on libgcc's unwind implementation, which is
documented as not signal-safe (see tikv/pprof-rs#36)
Bump pprof from 0.14 to 0.15 in all three consumers (common-pprof,
cmd, servers) to unify on a single version. pprof 0.15 also replaces
parking_lot with spin-rs to avoid a potential profiler deadlock (#268).
This addresses the libgcc_s.so.1 #GP crash observed in production
when CPU profiling is active, by eliminating the libgcc unwinder
from the signal handler path entirely.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(pprof): gate framehop-unwinder to supported targets
framehop-unwinder is only available on x86_64/aarch64 Linux/macOS.
Enabling it unconditionally for all Unix targets breaks the build on
riscv64 and other platforms: pprof disables its backtrace-rs fallback
when framehop-unwinder is set, but the framehop module is not compiled
on unsupported targets, leaving no TraceImpl implementation.
Split the pprof dependency: the base target.'cfg(unix)' block carries
the common features (flamegraph, prost-codec, protobuf), and a separate
target block adds framehop-unwinder only on supported targets.
Cargo unions features from both blocks on matching targets, so x86_64
and aarch64 Linux/macOS get the full feature set while other Unix
targets fall back to the default backtrace-rs implementation.
Addresses review comment discussion_r3987313632.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit 21aed0371f)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump tikv-jemalloc-sys patch to jemalloc dev (ff80bf2d)
Update the [patch.crates-io] rev for tikv-jemalloc-sys from
e1846d8c (5.3.1 + 54f22c83 backport) to ff444d4 (upstream dev HEAD,
161 commits ahead of 5.3.1).
The dev branch includes additional TSD/tcache fixes beyond the
original backport:
- fb5499aa9c: Handle jemalloc calls after TSD teardown
- 61dc1da395: Fix possible tcache corruption on fiber migration
- 1e92317014: Fix thread-exit TSD cleanup
See GreptimeTeam/jemallocator branch bump-jemalloc-dev and
tikv/jemallocator#182.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main
GreptimeTeam/jemallocator#2 has been merged; reference the merge
commit e254a7ea on main instead of the PR head branch. Jemalloc
source content is unchanged (still upstream dev ff80bf2d).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit 9619085310)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix
Upgrade tikv-jemallocator / tikv-jemalloc-ctl / tikv-jemalloc-sys from
0.6 to 0.7, which embeds jemalloc 5.3.1 (includes a056c20d 'Handle
tcache init failures gracefully').
On top of that, patch tikv-jemalloc-sys to the GreptimeTeam fork that
adds the remaining upstream fix 54f22c83 'Initialize TSD tcache before
enabling it' (GreptimeTeam/jemalloc#1, GreptimeTeam/jemallocator#1).
Without the ordering fix, a reentrant allocation during TSD bootstrap
(e.g. heap-profiling prof_tdata init / sampled backtrace when prof:true
is active) can observe an enabled-but-uninitialized tcache, corrupting
per-thread tcache metadata and crashing the process in arena_stats_merge,
calloc, or the libgcc unwinder.
The patch is pinned by rev and should be removed once tikv/jemallocator
ships a jemalloc snapshot that includes 54f22c83.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump tikv-jemalloc-sys patch rev to merged release-5.3.1
GreptimeTeam/jemalloc#1 has been merged; point the patch at the
jemallocator commit referencing the merge commit on release-5.3.1.
Jemalloc source content is unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main
GreptimeTeam/jemallocator#1 has been merged; reference the merge
commit e1846d8c on main instead of the PR head branch. Jemalloc
source content is unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(servers): bump tikv-jemallocator dev-dependency to 0.7
main added a target.'cfg(not(windows))'.dev-dependencies entry on
tikv-jemallocator 0.6 for servers after this branch diverged. On the
merge ref it pulled tikv-jemalloc-sys 0.6 from crates.io, which
conflicts with the patched 0.7 (links = "jemalloc" may only appear
once in the dependency graph), failing version selection in CI.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit ab0b1f5cce)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): look up row group stats by parquet leaf index for nested columns
On flat-format tables a logical column can expand to multiple parquet
leaf columns (e.g. a JSON2 struct stores the remainder and one leaf per
promoted path). ParquetFlat used the logical column index in the SST
schema directly as the leaf index when reading row group statistics, so
min/max/null stats of every column after a nested column were read from
wrong leaves.
When the misplaced leaf held order-compatible statistics (e.g. a small
Int64 JSON path vs. the timestamp window predicate), min-max pruning
dropped whole row groups by mistake. SWCS compaction reads inputs with a
time window predicate, so it silently lost all rows of such files; plain
queries with time-range predicates were affected as well.
Map each column to its first parquet leaf column and report NoStats for
columns with multiple leaves, which makes pruning conservative for them.
Add a unit test and a sqlness regression case that reproduces the data
loss on the unfixed binary.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): skip nested root stats and correct SWCS regression baseline
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): resolve statistics leaves for primary-key SST readers
Resolve scalar roots against the actual Parquet schema in shared statistics helpers, covering both flat and primary-key readers. Remove flat-side translation to avoid mapping twice and align encoded primary-key statistics as well. Cover dense flat, legacy dense and sparse layouts with statistics and time-pruning regressions.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): distinguish known null counts from unknown statistics
Assert validity before reading timestamp null counts and add a nullable scalar after the nested root with a known nonzero count. Exercise the assertions for flat and primary-key SST layouts.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): check JSON2 time pruning before SWCS compaction
Query the first time window immediately after FLUSH to cover predicate reads on flush-written SSTs independently of compaction outputs. Regenerate the sqlness expectation and retain the post-compaction checks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): validate all business columns after JSON2 SWCS
Expand the final regression query to all eight business columns so the generated expectation verifies complete rows, including tags and scalar fields, after repeated compaction.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit 743261f05e)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 113db823e8)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(prometheus): align batch flush deadline with creation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(servers): sync pending worker submission with explicit ack
Replace the mpsc capacity() polling in the pending rows batcher deadline
test with a test-only WorkerCommand::Ack round trip. The FIFO channel
guarantees the worker has dequeued and processed the submission (and
anchored the flush deadline) before the test advances virtual time,
removing reliance on an implementation detail that can be flaky under
scheduling variance.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(servers): await pending worker flush results with bounded timeout
Replace the try_recv() yield-polling loop in the pending rows batcher
deadline test with a direct await bounded by tokio::time::timeout. Under
paused time the timeout auto-advances the clock and fires
deterministically, so the test fails reliably instead of intermittently
missing the result on slower CI or under contention.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit 4fd35462d6)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(cli): sanitize store_addrs in kvbackend build log
Close#7525. The CLI's kvbackend construction log was printing raw
store_addrs which could contain sensitive connection strings (e.g.
PostgreSQL DSNs with passwords).
Changes:
- Add sanitize_store_addrs() helper that reuses
common_meta::kv_backend::util::sanitize_connection_string(),
consistent with MetasrvOptions and StartCommand patterns.
- Replace raw store_addrs in the info! log with sanitized version.
- Add unit tests covering MySQL URLs, PostgreSQL DSNs, etcd addresses,
and empty store_addrs cases.
Signed-off-by: qiang_liu
Signed-off-by: qiang_liu <qiang_liu@trendmicro.com>
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>
* fix(cli): drop redundant sanitize tests per review
sanitize_connection_string in common_meta already covers MySQL URLs,
PostgreSQL DSNs and credential-free etcd addresses with its own tests.
The added tests only exercised a trivial map+collect wrapper, so remove
them per review nit.
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>
---------
Signed-off-by: qiang_liu
Signed-off-by: qiang_liu <qiang_liu@trendmicro.com>
Signed-off-by: LiuQhahah <liuqiang9596@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
(cherry picked from commit ad5ccc98ec)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(promql): correct counter reset accumulation in rate windows
`prom_rate` and `prom_increase` reused the previous window's counter-reset
correction when the next window slid forward by exactly one sample, adding
the entering reset and subtracting the leaving one. Running a sum through
addition and subtraction does not restore the earlier terms in f64: a large
reset absorbs the smaller ones that must survive it, and an expired infinity
leaves a NaN that no later window can clear. `prom_delta` shares the code but
is not a counter function, so it never took that path.
Index the reset positions of the value array once instead, and reduce each
window over the resets it contains, in sample order. The result is
bit-identical to scanning the window directly, so windows keep the direct
reduction when they request fewer sample pairs than the input has.
Also sweep the query step in the rate benchmarks: the cost of the reset
correction depends on how much the windows overlap, which no existing case
varied.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* test(promql): cover counter reset precision over adjacent rate windows
The unit tests build the range windows directly, so they do not show that a
plain PromQL range query produces the window layout that lost the correction.
This case does: with a query step equal to the sample interval, `increase`
over the second window returns 1.333 before the fix and 2.667 after it.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* perf(promql): advance the counter reset bounds instead of searching
Locating a window's resets with two binary searches costs more than the
reduction it replaces once a series resets often enough for the searches to
get deep: on a 20k-sample counter resetting every 37 samples, stepping the
windows by one sample was 2.7x slower than the previous code, against 1.2x
for a counter that never resets.
Windows normally advance, so walk the bounds forward from the previous
window and only search when they move back. The cost then no longer depends
on the reset density.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* perf(promql): cut the per-window cost of the counter reset index
Two costs the index added showed up on a one-sample query step, where the
removed fast path used to answer each window with two comparisons.
Cache the two reset positions that bound the active slice. A window that only
advanced and reached neither of them covers the same resets as the previous
one, so the common case is four integer comparisons and no lookup at all.
Stop summing the requested sample pairs once they exceed one pass over the
values. The sum only decides which side of that comparison the input falls on,
and a query with a short lookback and a long step settles it after a few
windows instead of after every key.
Together these take the one-sample step from 25-32% slower than the previous
code down to 6-11%, measured as before / after / before to bound drift. No
other step value regresses, and a ten-sample step stays about 88% faster.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* fix(promql): accumulate counter resets into the running result
`prom_rate` and `prom_increase` summed a window's counter-reset corrections
on their own and added that sum to `last - first`. Prometheus folds each
reset into the running result instead, and so did this code before #7880.
The two are not interchangeable in f64: over samples `[1e16, 1, 0, 1]` the
isolated sum rounds `1e16 + 1.0` back to `1e16`, which then cancels against
the first sample and reports no increase at all, where folding the resets in
one at a time keeps the 1.0.
Restore the original order. The reset index accumulates into the result the
same way, so it still matches a direct scan of the window bit for bit, but a
window's contribution can no longer be cached as a standalone value and is
re-added from its own difference each time. The bounds are still cached, so
a window that did not cross a reset skips the lookup, and one that holds no
resets returns without touching the index at all.
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
* test(promql): note which reset boundaries the stride of one walks
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
---------
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
(cherry picked from commit 0f625a7e92)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Complete the configuration scope of upstream #9022, including frontend and standalone propagation, examples, generated docs, and config serialization coverage.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
fix: re-scan stream-backed tables in recursive CTEs (#9039)
* fix: re-scan stream-backed tables in recursive CTEs
A recursive CTE re-executes its recursive term on every iteration, but
DfTableProviderAdapter hands StreamScanAdapter a single-use stream built at
planning time. The second iteration failed with "Stream already exhausted"
for every table served through DataSource::get_stream — information_schema,
pg_catalog, the computed entity-graph tables and numbers.
Keep that stream for the first execution and open a new one over the same
scan request for later executions.
Closes#9037
* refactor: drop redundant binding in stream factory
---------
(cherry picked from commit bb9b7e8778)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
fix: match system schema names case-insensitively (#9040)
* fix: match system schema names case-insensitively
Database names that arrive over a protocol (the MySQL handshake and
COM_INIT_DB, the Postgres startup parameter, the HTTP `db` parameter, the
gRPC dbname header) never reach the SQL parser, which is what lowercases
unquoted identifiers. Since #8062 stopped lowercasing them wholesale,
connecting to `INFORMATION_SCHEMA` in any spelling but the canonical one
fails with "Unknown database" -- including the `USE <db>` that a MySQL
client turns into COM_INIT_DB.
Fold only system schema names to their canonical spelling, so user schema
names keep the case they were created with. `is_reserved_schema_name` uses
the same match, otherwise a quoted `CREATE DATABASE "INFORMATION_SCHEMA"`
creates a schema shadowed by the system one.
* refactor: hoist system schema names into a const
---------
(cherry picked from commit fa794fae7a)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* fix(mito2): fence checkpoints during region transitions (#8847)
* fix: fence checkpoints during region transitions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(datanode): fix transient downgrade setup
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(mito2): fix checkpoint lifecycle test setup
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(mito2): cover cancelled downgrade waiter retry
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): fence direct follower transitions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: trim checkpoint transition coverage
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(mito2): clarify checkpoint task lifecycle
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit a502dfdefd)
* fix(mito2): split SSTs at primary key series boundaries (#8888)
* fix(mito2): split SSTs at series boundaries
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover SST splitting without primary key
Also document the sortedness precondition and the series boundary
split semantics on write_all_flat/write_all_flat_as_primary_key and
the new split helpers.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): avoid per-row Vec allocation for empty primary key
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Update src/mito2/src/sst/parquet/writer.rs
Co-authored-by: Yingwen <realevenyag@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
(cherry picked from commit 4ac3423261)
* fix(meta): allow manual migration from offline datanodes (#8934)
* fix(meta): allow migration from offline datanodes
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: fix offline migration event actor
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: read migration routes from metadata
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 8a473c5bf0)
* fix(flight): bound DoGet response wait (#8943)
* fix(flight): defer datanode query initialization
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(client): retain Flight stream peer context
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(client): improve Flight stream diagnostics
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 28398138ec)
* fix(query): keep INSERT timestamp conversion out of the source query (#8911)
* fix(query): keep INSERT timestamp conversion out of the source query
Interpreting an INSERT's string timestamps used to work by pushing the
conversion down into the source query, which changed what that query
means. Two consequences:
- Pushing through a UNION's DISTINCT moved the dedup key from the raw
strings to parsed instants, so rows spelling the same instant
differently collapsed into one. On an append-only table that is a
silently dropped row.
- A UNION branch that needed no conversion (a NULL, or an explicit cast)
made the whole column give up, leaving sibling branches on UTC while
the rest of the row used the session timezone.
Convert at the assignment instead, by routing its cast through a
timezone-carrying timestamp type and back. Arrow applies the timezone
when a cast target carries one, and stripping it afterwards preserves
the value. The source query is no longer touched, so both cases go away
and the tree-walking rewrite (roughly 160 lines) is deleted.
The rewrite reads source types, so it now runs TypeCoercion first: a
UNION still carries its loose per-branch schema before coercion, and
retargeting a cast whose input later becomes a timestamp would shift the
value rather than reinterpret it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): address review on INSERT assignment rewrite
- Clone the input `Arc` instead of the whole subtree, and only rebuild it
when a `Values` row actually changes.
- Defer cloning the cast source until the literal-folding path has been
ruled out.
- Move the UTC check onto `Timezone::is_utc`, replacing a bare string
compare.
- Cover a prepared `INSERT ... VALUES (?)`: an untyped placeholder types
as `Null`, so the assignment cast is left for parameter substitution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 1851f6bf4d)
* test: renew etcd TLS certificates (#8956)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 144f83528d)
* fix: update tokio-postgres and correct explain/fetch cursor output schema (#8955)
* chore(deps): update tokio-postgres
* fix: describing fetch cursor and analyze
(cherry picked from commit b31f05eb59)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): avoid insert select HTTP/2 stalls
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit d4a3d88cd7)
* fix(query): share record batch forwarding for DML
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit c34bec197c)
* fix(mysql): interpret prepared statement datetime params in session timezone (#8923)
* fix(mysql): interpret prepared statement datetime params in session timezone
Binary DATETIME parameters of server-side prepared statements were
converted as if UTC, ignoring the session timezone set via SET time_zone.
Convert them with the session timezone and add an integration test
covering prepared inserts and predicates under Asia/Shanghai.
Signed-off-by: wy471x <wy471x@gmail.com>
* refactor: share naive datetime timezone policy via common-time
Address review feedback on the prepared-statement timezone fix:
- Expose Timestamp::from_naive_datetime in common-time so the DST policy
(gap -> error, ambiguous -> earlier instant) lives in one place, shared
by the text protocol (Timestamp::from_str) and the MySQL binary protocol.
- Route the MySQL prepared-statement datetime conversion through it.
- Match the target type before converting datetime params so
PreparedStmtTypeMismatch fails fast without wasted conversion.
- Use the short Timezone import form for consistency with the rest of servers.
Signed-off-by: wy471x <wy471x@gmail.com>
---------
Signed-off-by: wy471x <wy471x@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
(cherry picked from commit aaa843104b)
* fix(mito2): use target sequence for foreign SSTs (#8946)
* fix(mito2): use target sequence for foreign SSTs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore(mito2): address foreign SST review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit bd7d2c1dfa)
* fix: postgres describe for more statements (#8974)
* fix: postgres describe for more statements
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: cover more show statements
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
- add missing `clippy::too_many_arguments` allow on
`query_from_information_schema_dataframe` (CI clippy failure)
- take `&ShowKind` in the information-schema dataframe helper so `kind`
is no longer cloned at every call site; only the WHERE arm (which needs
an owned expression for `sql_to_expr`) clones internally
- document why re-applying TQL explain formats never overwrites an
existing value (per-query context state)
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: trim comments to essentials
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
(cherry picked from commit d32cd77505)
* fix(promql): resolve derived labels in aggregation arithmetic (#8994)
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit c4dafb5826)
* fix(deps): narrow postgres lockfile update
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(deps): normalize remaining x509 lock reference
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(client): isolate query and control transports
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(frontend): clarify adaptive window purpose
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(grpc): satisfy clippy in transport isolation coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(pipeline): coalesce concurrent pipeline cache misses (#9022)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "test(grpc): satisfy clippy in transport isolation coverage"
This reverts commit 888bcedc96.
(cherry picked from commit e0216b4074485bcffaae53348a17091532cb3600)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "docs(frontend): clarify adaptive window purpose"
This reverts commit 84cf69d099.
(cherry picked from commit da30e605ad7ebfb2a3fa86ea44b67405e5afde98)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "fix(client): isolate query and control transports"
This reverts commit a1d2dbb050.
(cherry picked from commit 03f5c31f544d3d3852beddfa87cff34d04a646ae)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(json2): add bounded auto-expansion to the JSON2 vector builder (#8909)
* refactor(json2): add bounded auto-expansion to the JSON2 vector builder
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 76f08d2b3f)
(cherry picked from commit 640fa63bd8ee098fac8e8d62d6fb7ca5af4ddcd0)
* refactor(json2): optimize JSON2 building without auto-expanded paths (#8928)
* refactor(json2): optimize JSON2 building without auto-expanded paths
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* avoid panicking memtable write
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 7fd0a7bb98)
(cherry picked from commit 63c68ee60101e2a09e14420051f8f889b41a2a3b)
* refactor(json2): support querying v2 storage layout (#8940)
* feat(json2): support querying v2 storage layout
- route missing JSON2 paths to the v2 remainder field
- reconstruct complete values from explicit fields and remainder data
- preserve root JSON2 columns across projections and filters
- support nested JSON values in json_get string results
- add and reorganize JSON2 sqlness coverage
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 932f87f7a8)
(cherry picked from commit 5af1394a1f319bb99cafa9474b8a5a075be1e805)
* feat(pipeline): support table-aware JSON2 transforms (#8964)
* feat(pipeline): support table-aware JSON2 transforms
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* feat(pipeline): support JSON2 type hints in transforms
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(pipeline): default failed JSON2 transforms to null
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* refactor(json2): distinguish invalid settings from layout errors
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit fb86f6573e)
(cherry picked from commit 18a21a9ce017b6887c857a3f3bbb9a83aa7f968b)
* refactor: json2 v2 storage layout (#8979)
* refactor: json2 v2 storage layout
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
* rethinking when "needs_remainder"
Signed-off-by: luofucong <luofc@foxmail.com>
* restore "ReadColumns"
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 529f046110)
(cherry picked from commit 7628d795197c786872730c3a5ef36c5c662d4322)
* feat(json2): support JSON2 paths in SQL functions (#9007)
feat(query): support JSON2 paths in SQL functions
Update the DataFusion fork to expose scalar function planning hooks.
Infer JSON2 path output types from scalar, aggregate, and window function signatures, while preserving the default Utf8View behavior for functions that accept arbitrary inputs.
Add unit and sqlness coverage for type conflicts, mixed typed and untyped JSON paths, filters, aggregates, and window functions.
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 15317a131b)
(cherry picked from commit a945cbc7a4398e6bee74d4d2c67528772faf398d)
* feat(json2): support empty and null JSON2 value (#9010)
* feat(json2): support empty and null JSON2 value
Signed-off-by: fys <fengys1996@gmail.com>
* test(json2): cover explicit NULL and omitted-column inserts
Signed-off-by: fys <fengys1996@gmail.com>
* fix: cargo fmt
Signed-off-by: fys <fengys1996@gmail.com>
* fix: infer empty JSON object as object type
Signed-off-by: fys <fengys1996@gmail.com>
---------
Signed-off-by: fys <fengys1996@gmail.com>
(cherry picked from commit 05c65f54a8)
(cherry picked from commit 05c27452ea5e958fab85e0ba4b82f5df205a5d08)
* feat(json2): support list indexing for JSON2 columns (#9013)
feat(query): support list indexing for JSON2 columns
Extend JSON2 paths through DataFusion field-access planning, including nested list indexes and object fields following an index.
Preserve Variant reads for bracket JSONPath expressions and normalize dot accesses after subscripts to work around the current DataFusion planner limitation.
Add unit and sqlness coverage for nested indexes, type conflicts, missing paths, flushes, and compacted SSTs.
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit d62a5a990a)
(cherry picked from commit c11d8647feb30ecdd5ba1bc8ed4df78999801672)
* refactor(json2): concretize JSON2 schemas at merge scan boundaries (#9016)
* refactor(json2): concretize JSON2 schemas at merge scan boundaries
Infer concrete JSON2 output types from remote plans and expose them on MergeScanLogicalPlan before physical planning. Recompute affected local schemas and remove the JSON2-specific rewrite from MergeScanExec.
Add SQLness coverage for whole JSON2 columns in windows and joins.
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 84bd993131)
(cherry picked from commit 51c98a7e31a929fc33ff5133714bb47108f7539b)
* fix(json2): keep empty structs in remainder (#9027)
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit d67d3501a9)
(cherry picked from commit b3f2c1cc5bf0cfddb7c0670ac8f3522eb67c35a0)
* fix(mito2): remove stale scan test imports
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit d63bbe6aef298f5b8e960bb02eed08743a6684ee)
* test(query): expect bounded merge scan partitions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 7bcef46b1c010d2e73ec49be7581c216e6b5b1b6)
* style(mito2): remove trailing whitespace
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(sqlness): refresh bounded partition plans
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(sqlness): normalize CPU-dependent partition plans
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: wy471x <wy471x@gmail.com>
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: wy471x <wy471x@gmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
Co-authored-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: luofucong <luofc@foxmail.com>
Co-authored-by: fys <fengys1996@gmail.com>
Bump the pinned DataFusion fork from 452cb4b786 to its linear successor
4c8a6bf283. The successor preserves the existing Dictionary-literal
Substrait fix, includes the intervening join-cardinality overflow fix,
and adds DictionaryGroupValuesColumn support so dictionary-encoded group
keys can use the columnar grouping path instead of GroupValuesRows.
Add an end-to-end integration test that writes 1,200 rows to an
append-mode flat SST, groups its dictionary-encoded hostname tag by hour,
and checks all six ordered result tuples exactly. AggregateExec does not
expose the concrete GroupValues implementation through EXPLAIN or metrics;
direct DictionaryGroupValuesColumn path coverage remains in the pinned
DataFusion fork tests.
(cherry picked from commit d99b0df374)
Signed-off-by: discord9 <discord9@163.com>
Lei, HUANG
discord9
Qiang-Liu
discord9
dennis zhuang
discord9
shuiyisong
