fix: match system schema names case-insensitively (#9040)
* fix: match system schema names case-insensitively
Database names that arrive over a protocol (the MySQL handshake and
COM_INIT_DB, the Postgres startup parameter, the HTTP `db` parameter, the
gRPC dbname header) never reach the SQL parser, which is what lowercases
unquoted identifiers. Since #8062 stopped lowercasing them wholesale,
connecting to `INFORMATION_SCHEMA` in any spelling but the canonical one
fails with "Unknown database" -- including the `USE <db>` that a MySQL
client turns into COM_INIT_DB.
Fold only system schema names to their canonical spelling, so user schema
names keep the case they were created with. `is_reserved_schema_name` uses
the same match, otherwise a quoted `CREATE DATABASE "INFORMATION_SCHEMA"`
creates a schema shadowed by the system one.
* refactor: hoist system schema names into a const
---------
(cherry picked from commit fa794fae7a)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* fix(mito2): fence checkpoints during region transitions (#8847)
* fix: fence checkpoints during region transitions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(datanode): fix transient downgrade setup
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(mito2): fix checkpoint lifecycle test setup
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test(mito2): cover cancelled downgrade waiter retry
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(mito2): fence direct follower transitions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: trim checkpoint transition coverage
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(mito2): clarify checkpoint task lifecycle
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit a502dfdefd)
* fix(mito2): split SSTs at primary key series boundaries (#8888)
* fix(mito2): split SSTs at series boundaries
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover SST splitting without primary key
Also document the sortedness precondition and the series boundary
split semantics on write_all_flat/write_all_flat_as_primary_key and
the new split helpers.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): avoid per-row Vec allocation for empty primary key
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Update src/mito2/src/sst/parquet/writer.rs
Co-authored-by: Yingwen <realevenyag@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
(cherry picked from commit 4ac3423261)
* fix(meta): allow manual migration from offline datanodes (#8934)
* fix(meta): allow migration from offline datanodes
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: fix offline migration event actor
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: read migration routes from metadata
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 8a473c5bf0)
* fix(flight): bound DoGet response wait (#8943)
* fix(flight): defer datanode query initialization
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(client): retain Flight stream peer context
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(client): improve Flight stream diagnostics
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 28398138ec)
* fix(query): keep INSERT timestamp conversion out of the source query (#8911)
* fix(query): keep INSERT timestamp conversion out of the source query
Interpreting an INSERT's string timestamps used to work by pushing the
conversion down into the source query, which changed what that query
means. Two consequences:
- Pushing through a UNION's DISTINCT moved the dedup key from the raw
strings to parsed instants, so rows spelling the same instant
differently collapsed into one. On an append-only table that is a
silently dropped row.
- A UNION branch that needed no conversion (a NULL, or an explicit cast)
made the whole column give up, leaving sibling branches on UTC while
the rest of the row used the session timezone.
Convert at the assignment instead, by routing its cast through a
timezone-carrying timestamp type and back. Arrow applies the timezone
when a cast target carries one, and stripping it afterwards preserves
the value. The source query is no longer touched, so both cases go away
and the tree-walking rewrite (roughly 160 lines) is deleted.
The rewrite reads source types, so it now runs TypeCoercion first: a
UNION still carries its loose per-branch schema before coercion, and
retargeting a cast whose input later becomes a timestamp would shift the
value rather than reinterpret it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): address review on INSERT assignment rewrite
- Clone the input `Arc` instead of the whole subtree, and only rebuild it
when a `Values` row actually changes.
- Defer cloning the cast source until the literal-folding path has been
ruled out.
- Move the UTC check onto `Timezone::is_utc`, replacing a bare string
compare.
- Cover a prepared `INSERT ... VALUES (?)`: an untyped placeholder types
as `Null`, so the assignment cast is left for parameter substitution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 1851f6bf4d)
* test: renew etcd TLS certificates (#8956)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 144f83528d)
* fix: update tokio-postgres and correct explain/fetch cursor output schema (#8955)
* chore(deps): update tokio-postgres
* fix: describing fetch cursor and analyze
(cherry picked from commit b31f05eb59)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): avoid insert select HTTP/2 stalls
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit d4a3d88cd7)
* fix(query): share record batch forwarding for DML
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit c34bec197c)
* fix(mysql): interpret prepared statement datetime params in session timezone (#8923)
* fix(mysql): interpret prepared statement datetime params in session timezone
Binary DATETIME parameters of server-side prepared statements were
converted as if UTC, ignoring the session timezone set via SET time_zone.
Convert them with the session timezone and add an integration test
covering prepared inserts and predicates under Asia/Shanghai.
Signed-off-by: wy471x <wy471x@gmail.com>
* refactor: share naive datetime timezone policy via common-time
Address review feedback on the prepared-statement timezone fix:
- Expose Timestamp::from_naive_datetime in common-time so the DST policy
(gap -> error, ambiguous -> earlier instant) lives in one place, shared
by the text protocol (Timestamp::from_str) and the MySQL binary protocol.
- Route the MySQL prepared-statement datetime conversion through it.
- Match the target type before converting datetime params so
PreparedStmtTypeMismatch fails fast without wasted conversion.
- Use the short Timezone import form for consistency with the rest of servers.
Signed-off-by: wy471x <wy471x@gmail.com>
---------
Signed-off-by: wy471x <wy471x@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
(cherry picked from commit aaa843104b)
* fix(mito2): use target sequence for foreign SSTs (#8946)
* fix(mito2): use target sequence for foreign SSTs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore(mito2): address foreign SST review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit bd7d2c1dfa)
* fix: postgres describe for more statements (#8974)
* fix: postgres describe for more statements
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: cover more show statements
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
- add missing `clippy::too_many_arguments` allow on
`query_from_information_schema_dataframe` (CI clippy failure)
- take `&ShowKind` in the information-schema dataframe helper so `kind`
is no longer cloned at every call site; only the WHERE arm (which needs
an owned expression for `sql_to_expr`) clones internally
- document why re-applying TQL explain formats never overwrites an
existing value (per-query context state)
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: trim comments to essentials
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
(cherry picked from commit d32cd77505)
* fix(promql): resolve derived labels in aggregation arithmetic (#8994)
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit c4dafb5826)
* fix(deps): narrow postgres lockfile update
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(deps): normalize remaining x509 lock reference
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(client): isolate query and control transports
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(frontend): clarify adaptive window purpose
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(grpc): satisfy clippy in transport isolation coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(pipeline): coalesce concurrent pipeline cache misses (#9022)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "test(grpc): satisfy clippy in transport isolation coverage"
This reverts commit 888bcedc96.
(cherry picked from commit e0216b4074485bcffaae53348a17091532cb3600)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "docs(frontend): clarify adaptive window purpose"
This reverts commit 84cf69d099.
(cherry picked from commit da30e605ad7ebfb2a3fa86ea44b67405e5afde98)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "fix(client): isolate query and control transports"
This reverts commit a1d2dbb050.
(cherry picked from commit 03f5c31f544d3d3852beddfa87cff34d04a646ae)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(json2): add bounded auto-expansion to the JSON2 vector builder (#8909)
* refactor(json2): add bounded auto-expansion to the JSON2 vector builder
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 76f08d2b3f)
(cherry picked from commit 640fa63bd8ee098fac8e8d62d6fb7ca5af4ddcd0)
* refactor(json2): optimize JSON2 building without auto-expanded paths (#8928)
* refactor(json2): optimize JSON2 building without auto-expanded paths
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* avoid panicking memtable write
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 7fd0a7bb98)
(cherry picked from commit 63c68ee60101e2a09e14420051f8f889b41a2a3b)
* refactor(json2): support querying v2 storage layout (#8940)
* feat(json2): support querying v2 storage layout
- route missing JSON2 paths to the v2 remainder field
- reconstruct complete values from explicit fields and remainder data
- preserve root JSON2 columns across projections and filters
- support nested JSON values in json_get string results
- add and reorganize JSON2 sqlness coverage
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 932f87f7a8)
(cherry picked from commit 5af1394a1f319bb99cafa9474b8a5a075be1e805)
* feat(pipeline): support table-aware JSON2 transforms (#8964)
* feat(pipeline): support table-aware JSON2 transforms
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* feat(pipeline): support JSON2 type hints in transforms
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* fix(pipeline): default failed JSON2 transforms to null
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* refactor(json2): distinguish invalid settings from layout errors
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit fb86f6573e)
(cherry picked from commit 18a21a9ce017b6887c857a3f3bbb9a83aa7f968b)
* refactor: json2 v2 storage layout (#8979)
* refactor: json2 v2 storage layout
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
* rethinking when "needs_remainder"
Signed-off-by: luofucong <luofc@foxmail.com>
* restore "ReadColumns"
Signed-off-by: luofucong <luofc@foxmail.com>
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 529f046110)
(cherry picked from commit 7628d795197c786872730c3a5ef36c5c662d4322)
* feat(json2): support JSON2 paths in SQL functions (#9007)
feat(query): support JSON2 paths in SQL functions
Update the DataFusion fork to expose scalar function planning hooks.
Infer JSON2 path output types from scalar, aggregate, and window function signatures, while preserving the default Utf8View behavior for functions that accept arbitrary inputs.
Add unit and sqlness coverage for type conflicts, mixed typed and untyped JSON paths, filters, aggregates, and window functions.
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 15317a131b)
(cherry picked from commit a945cbc7a4398e6bee74d4d2c67528772faf398d)
* feat(json2): support empty and null JSON2 value (#9010)
* feat(json2): support empty and null JSON2 value
Signed-off-by: fys <fengys1996@gmail.com>
* test(json2): cover explicit NULL and omitted-column inserts
Signed-off-by: fys <fengys1996@gmail.com>
* fix: cargo fmt
Signed-off-by: fys <fengys1996@gmail.com>
* fix: infer empty JSON object as object type
Signed-off-by: fys <fengys1996@gmail.com>
---------
Signed-off-by: fys <fengys1996@gmail.com>
(cherry picked from commit 05c65f54a8)
(cherry picked from commit 05c27452ea5e958fab85e0ba4b82f5df205a5d08)
* feat(json2): support list indexing for JSON2 columns (#9013)
feat(query): support list indexing for JSON2 columns
Extend JSON2 paths through DataFusion field-access planning, including nested list indexes and object fields following an index.
Preserve Variant reads for bracket JSONPath expressions and normalize dot accesses after subscripts to work around the current DataFusion planner limitation.
Add unit and sqlness coverage for nested indexes, type conflicts, missing paths, flushes, and compacted SSTs.
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit d62a5a990a)
(cherry picked from commit c11d8647feb30ecdd5ba1bc8ed4df78999801672)
* refactor(json2): concretize JSON2 schemas at merge scan boundaries (#9016)
* refactor(json2): concretize JSON2 schemas at merge scan boundaries
Infer concrete JSON2 output types from remote plans and expose them on MergeScanLogicalPlan before physical planning. Recompute affected local schemas and remove the JSON2-specific rewrite from MergeScanExec.
Add SQLness coverage for whole JSON2 columns in windows and joins.
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 84bd993131)
(cherry picked from commit 51c98a7e31a929fc33ff5133714bb47108f7539b)
* fix(json2): keep empty structs in remainder (#9027)
Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit d67d3501a9)
(cherry picked from commit b3f2c1cc5bf0cfddb7c0670ac8f3522eb67c35a0)
* fix(mito2): remove stale scan test imports
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit d63bbe6aef298f5b8e960bb02eed08743a6684ee)
* test(query): expect bounded merge scan partitions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 7bcef46b1c010d2e73ec49be7581c216e6b5b1b6)
* style(mito2): remove trailing whitespace
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(sqlness): refresh bounded partition plans
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(sqlness): normalize CPU-dependent partition plans
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: wy471x <wy471x@gmail.com>
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: wy471x <wy471x@gmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
Co-authored-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: luofucong <luofc@foxmail.com>
Co-authored-by: fys <fengys1996@gmail.com>
Bump the pinned DataFusion fork from 452cb4b786 to its linear successor
4c8a6bf283. The successor preserves the existing Dictionary-literal
Substrait fix, includes the intervening join-cardinality overflow fix,
and adds DictionaryGroupValuesColumn support so dictionary-encoded group
keys can use the columnar grouping path instead of GroupValuesRows.
Add an end-to-end integration test that writes 1,200 rows to an
append-mode flat SST, groups its dictionary-encoded hostname tag by hour,
and checks all six ordered result tuples exactly. AggregateExec does not
expose the concrete GroupValues implementation through EXPLAIN or metrics;
direct DictionaryGroupValuesColumn path coverage remains in the pinned
DataFusion fork tests.
(cherry picked from commit d99b0df374)
Signed-off-by: discord9 <discord9@163.com>
Use the JSON2 extension API introduced by the replayed #8745 patch.
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 6690189d6a07d0bdc03566fed5f2a06306df5cff)
* fix(metric-engine): handle Utf8View tag/label columns without panicking
label_replace (planned as DataFusion regexp_replace) coerces to Utf8View,
so label columns materialize as StringViewArray; build_tag_arrays'
StringArray downcast then panicked ('tag column must be utf8') — e.g. for
OTLP/json2 ingest. TSID computation, sparse-PK encoding and tag
extraction now accept generic ArrayRef tag columns (Utf8/LargeUtf8/
Utf8View/Dictionary) via string_array_value_at_index, and build_tag_arrays
errors instead of panicking on non-string columns. The mito2 time-series
memtable string-field paths are hardened the same way.
Adds label_replace_with_utf8view_labels_does_not_panic (issue #8732).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(metric-engine): add is_string_null_at helper for tag null checks
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(datatypes): use is_none_or to satisfy clippy unnecessary-map-or
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix: reject oversized string batches before memtable append
Distinguish a full active string builder from a batch that cannot fit an
empty Arrow string builder at all. Scan every string field so a later
intrinsically oversized field cannot be skipped after an earlier field
requests a freeze. Return InvalidBatch instead of reaching Arrow's offset
overflow panic.
Also cover Utf8View tags with nulls through the metric-engine tag, TSID,
and sparse-primary-key path.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 31a4061552eaa3456faef12b290ed1e10e722804)
(cherry picked from commit 5cef6fe4e14c2847b816009f5838e8e41c9dea94)
* fix(query): follow timestamp insert assignment lineage
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): fold constant insert timestamp literals at the assignment
Following lineage by retyping the source column changed every output
column that reads it: a string column sharing the literal was silently
rewritten to a formatted timestamp, and a nanosecond column was
truncated to the precision of whichever column was converted first.
Resolve the constant read-only and fold it into the assignment
expression instead, which leaves the source query untouched and also
covers literals behind WHERE, ORDER BY and DISTINCT. VALUES rows and
UNION branches carry per-row values, so they keep the in-place rewrite,
now guarded against columns with more than one consumer.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): strengthen insert lineage regression
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): trim redundant insert coverage
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): rebuild insert unions loosely and cover UNION distinct
Per review: rebuilding a rewritten union with the strict constructor
rejected legal pre-coercion plans whose untouched columns still differ
across branches. Use try_new_with_loose_types, matching the SQL planner.
Distinct::All joins the rewrite passthrough so UNION (distinct) literals
get session-timezone parsing like UNION ALL; deduplication then keys on
parsed instants instead of raw strings. The top-level rewrite path gains
the same single-consumer guard as rewrite_projection for hand-built DML
plans that share a source column between targets.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* style(query): tighten insert assignment comments
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit b2fc4b9cba4c0e137cc6f704d88dfe7f187b57aa)
(cherry picked from commit 1775e31a7916cf09a7da8238cebea5e3d74817f7)
* feat(native-histogram): store counts and span lengths as signed integers
Native histograms are unreleased, so the on-disk integer payload columns
are switched from unsigned to signed types without backward-compat:
- count_u64 / zero_count_u64: uint64 -> int64
- positive_span_lengths / negative_span_lengths: list(uint32) -> list(int32)
- Span.length (query-time model): u32 -> i32
The Prometheus remote-write v2 source carries these as uint64/uint32, so
the unsigned->signed conversion at the ingestion boundary is overflow
checked: an integer count >= 2^63 or a span length >= 2^31 is rejected
with an explicit error rather than silently wrapping to a negative value.
read_spans additionally rejects negative stored lengths to keep the
non-negative invariant sound for downstream `as usize` casts.
The UDAF accumulator's own observation counter (transient aggregation
state, not part of the persisted histogram value) is intentionally left
as uint64.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(native-histogram): rename count/zero_count fields to _i64
Now that the integer payload columns are stored as int64, rename the
field constants and persisted names to match:
COUNT_U64_FIELD ("count_u64") -> COUNT_I64_FIELD ("count_i64")
ZERO_COUNT_U64_FIELD ("zero_count_u64") -> ZERO_COUNT_I64_FIELD ("zero_count_i64")
The local builder variables and the docs/JSON snapshot are updated to
match. No backward-compat (unreleased feature).
Signed-off-by: Ning Sun <sunning@greptime.com>
* test(native-histogram): refresh planner plan snapshot for signed types
The mixed native-histogram range test embeds the full histogram Struct
type in its expected plan string, which still carried the pre-rename
unsigned fields. Update the snapshot to match the signed schema:
positive/negative_span_lengths: List(UInt32) -> List(Int32)
count_u64/zero_count_u64: UInt64 -> count_i64/zero_count_i64: Int64
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
(cherry picked from commit 3510ef7d4c)
(cherry picked from commit d636d6fd9e9474b262982f54dc5805f80127087a)
When a bulk insert request carries a stale schema, the worker fills the
missing columns via BulkPart::fill_missing_columns before writing. The
method replaced the batch but kept raw_data (the original Arrow Flight
IPC bytes), while BulkWalEntry::try_from(&BulkPart) prefers raw_data, so
the memtable received the filled batch but the WAL recorded the pre-fill
bytes.
Replaying such an entry restores a batch that misses the filled columns:
- Bulk memtable (flat format): convert_bulk_part fails with
ColumnNotFound; the error is swallowed by the no-op write notifier and
the rows are silently lost after restart.
- Time series memtable: BulkPart::to_mutation builds rows shorter than
the declared schema and the region worker panics with index out of
bounds during replay, hanging the region open.
Fixes:
- fill_missing_columns clears raw_data so the WAL entry is re-encoded
from the filled batch.
- replay_memtable fills missing columns for replayed bulk parts of dense
regions, so entries already written by affected versions replay
correctly.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 1f4a8ad71cdfc135cf571ea7bca2cb87ed90ad72)
(cherry picked from commit 7fae38e667a17b3cd3de3f0540cb3a7b6d5a277a)
* fix(mito2): fail open when Bloom IN predicate has non-literal or unencodable members
collect_in_list previously filtered out non-literal and encoding-failed IN
members and could build a partial hard-pruning predicate, so scanning with
the Bloom filter enabled could prune rows that actually match the query
(false-negative results).
Now any non-literal member or any encoding failure disables Bloom pruning
for the whole IN expression (fail open), while independent AND
subpredicates such as `col = 42` are still extracted as before. Encoding
errors are logged and ignored, so queries never fail.
Adds builder unit tests covering pure-literal, pure-nonliteral, mixed
literal+null, all-null, mixed-nonliteral-with-AND, and encoding-failure
cases, plus an engine-level witness test asserting Bloom-on and Bloom-off
scans return identical rows for a mixed IN filter.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): rename bloom filter tests to semantic names without bug-id prefixes
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 70ddf1685eb63d6faca2d0009fe157369c950f73)
(cherry picked from commit 5cbab008ffcdd83c0c5603b81448e1a9b2790254)
* fix(query): avoid unsafe count wildcard rewrites
Signed-off-by: discord9 <discord9@163.com>
* fix(query): preserve outer count alias
Signed-off-by: discord9 <discord9@163.com>
* fix(query): address review comments on count wildcard rewrite
- Remove the has_projection check: the row count is correct regardless
of whether a projection exists (per review).
- Explain why checking the first input is equivalent to checking all
inputs (a plan with zero inputs falls back to count(1)).
- Rename qa_ prefixed tests to follow the module convention.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(tql): update tql-cte expectations for count wildcard rewrite
The QP-026 count-wildcard fix rewrites count(*) -> count(time_index), so
the EXPLAIN output for the filtered/final CTE aggregates names the
time-index column. Aligns tql-cte.result with the actual output (CI
failure).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 7223f2a9b80a476448e3bded312e072cd1585e2a)
* fix(prometheus): custom column remote reads
Resolve timestamp and value column names from the table schema and carry
them through query planning and result conversion. Add a remote-read
regression test covering custom_ts and custom_value.
Signed-off-by: grezzko <me@gauravshokeen.com>
* fix: resolve remote-read value columns safely
Prefer the sole field for custom schemas and greptime_value for
multi-field tables. Reject ambiguous schemas and add regression tests.
Signed-off-by: grezzko <me@gauravshokeen.com>
---------
Signed-off-by: grezzko <me@gauravshokeen.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>
(cherry picked from commit 3d7d7e916c5c8643d57d61616c180396193f8911)
* fix(frontend): remove gRPC DDL panics for DropView and non-timestamp time index
Direct gRPC DDL bypasses the SQL parser, so two client-controlled DDL
payloads could panic a request handler:
- QX-152: DdlExpr::DropView hit todo!() (instance/grpc.rs:247-248).
Wire it to the real drop-view implementation (drop_view was
pub(crate); widened to pub) so a DropView DDL returns a structured
error (e.g. TableNotFound) instead of panicking.
- QX-153: a CreateTableExpr whose time_index column is not a timestamp
reached Schema::new's unwrap (ddl.rs:2346 -> schema.rs:114-119).
create_table_info now uses Schema::try_new with ConvertSchemaSnafu
context (InvalidArguments), and the direct gRPC CreateTable arm
validates the request via validate_create_expr (which now also checks
the time-index column type is a timestamp) before any catalog work.
SQL/HTTP paths were already protected by the parser; unchanged.
Tests: qx_152_drop_view_via_grpc_ddl_returns_error_not_panic,
qx_153_create_table_with_non_timestamp_time_index_via_grpc_returns_error
(asserts InvalidArguments), test_create_table_info_rejects_non_timestamp_time_index.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(frontend): add gRPC DDL happy-path coverage for DropView and CreateTable
Per review: the initial tests only asserted error paths. Add:
- drop_if_exists=true on a missing view succeeds (no error)
- dropping an existing view via gRPC DDL succeeds end-to-end
- a valid CreateTableExpr with a timestamp time index still succeeds
(guards validate_create_expr against rejecting good requests)
- qx_152 test now asserts the TableNotFound status instead of is_err()
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 11e785ed5680dbb07b7cf88ec1b9642cde5ba1e1)
* fix(flow): fix flow stats aggregation and df_plan_to_sql quoting
1. Distributed-mode flow stats last-writer-wins overwrite:
Each flownode heartbeat put its local flow state map into the single
global __flow/state key, so reports from different nodes overwrote
each other. Store per-flownode reports under
__flow/state/node/{node_id} in the in-memory KV and aggregate on each
heartbeat (last_exec_time_map/state_size/start_time_map take the max
across nodes) into the global key. FlowStateHandler derives node
identity from header.member_id (fallback peer.id) and ignores
identity-less reports. Per-node keys clear automatically on leader
change KV reset. Adapts to FlowStateValue.start_time_map added in
#8392.
2. df_plan_to_sql unquoted special characters break flush/scheduled
execution: ForceQuoteIdentifiers only quoted uppercase identifiers,
so Prometheus-style table names with ':' (e.g. cpu_cores:sum) were
left unquoted, producing invalid SQL ('keyword: :'). Quote any
identifier with non-[a-z0-9_] chars using double quotes
(dialect-neutral).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): address review comments on quoting and logging
- df_plan_to_sql: also quote digit-leading identifiers (e.g. 123metrics)
which would produce invalid SQL when re-parsed. SQL keywords are
intentionally not checked (ALL_KEYWORDS would over-quote common column
names like number; the unparse failure path has an InsertIntoPlan
fallback).
- flow_state_handler: downgrade identity-less report log from warn! to
debug! to avoid an anomalous sender spamming warn every heartbeat.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 375dd569482235914ef09ef5c21e15cc79fcd052)
LFCanddennis zhuang
discord9
discord9
shuiyisong
Morax
jeremyhi
fys
sun
Gaurav Shokeen
