Use the JSON2 extension API introduced by the replayed #8745 patch.
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 6690189d6a07d0bdc03566fed5f2a06306df5cff)
* fix(metric-engine): handle Utf8View tag/label columns without panicking
label_replace (planned as DataFusion regexp_replace) coerces to Utf8View,
so label columns materialize as StringViewArray; build_tag_arrays'
StringArray downcast then panicked ('tag column must be utf8') — e.g. for
OTLP/json2 ingest. TSID computation, sparse-PK encoding and tag
extraction now accept generic ArrayRef tag columns (Utf8/LargeUtf8/
Utf8View/Dictionary) via string_array_value_at_index, and build_tag_arrays
errors instead of panicking on non-string columns. The mito2 time-series
memtable string-field paths are hardened the same way.
Adds label_replace_with_utf8view_labels_does_not_panic (issue #8732).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(metric-engine): add is_string_null_at helper for tag null checks
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(datatypes): use is_none_or to satisfy clippy unnecessary-map-or
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix: reject oversized string batches before memtable append
Distinguish a full active string builder from a batch that cannot fit an
empty Arrow string builder at all. Scan every string field so a later
intrinsically oversized field cannot be skipped after an earlier field
requests a freeze. Return InvalidBatch instead of reaching Arrow's offset
overflow panic.
Also cover Utf8View tags with nulls through the metric-engine tag, TSID,
and sparse-primary-key path.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 31a4061552eaa3456faef12b290ed1e10e722804)
(cherry picked from commit 5cef6fe4e14c2847b816009f5838e8e41c9dea94)
* fix(query): follow timestamp insert assignment lineage
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): fold constant insert timestamp literals at the assignment
Following lineage by retyping the source column changed every output
column that reads it: a string column sharing the literal was silently
rewritten to a formatted timestamp, and a nanosecond column was
truncated to the precision of whichever column was converted first.
Resolve the constant read-only and fold it into the assignment
expression instead, which leaves the source query untouched and also
covers literals behind WHERE, ORDER BY and DISTINCT. VALUES rows and
UNION branches carry per-row values, so they keep the in-place rewrite,
now guarded against columns with more than one consumer.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): strengthen insert lineage regression
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): trim redundant insert coverage
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): rebuild insert unions loosely and cover UNION distinct
Per review: rebuilding a rewritten union with the strict constructor
rejected legal pre-coercion plans whose untouched columns still differ
across branches. Use try_new_with_loose_types, matching the SQL planner.
Distinct::All joins the rewrite passthrough so UNION (distinct) literals
get session-timezone parsing like UNION ALL; deduplication then keys on
parsed instants instead of raw strings. The top-level rewrite path gains
the same single-consumer guard as rewrite_projection for hand-built DML
plans that share a source column between targets.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* style(query): tighten insert assignment comments
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit b2fc4b9cba4c0e137cc6f704d88dfe7f187b57aa)
(cherry picked from commit 1775e31a7916cf09a7da8238cebea5e3d74817f7)
* feat(native-histogram): store counts and span lengths as signed integers
Native histograms are unreleased, so the on-disk integer payload columns
are switched from unsigned to signed types without backward-compat:
- count_u64 / zero_count_u64: uint64 -> int64
- positive_span_lengths / negative_span_lengths: list(uint32) -> list(int32)
- Span.length (query-time model): u32 -> i32
The Prometheus remote-write v2 source carries these as uint64/uint32, so
the unsigned->signed conversion at the ingestion boundary is overflow
checked: an integer count >= 2^63 or a span length >= 2^31 is rejected
with an explicit error rather than silently wrapping to a negative value.
read_spans additionally rejects negative stored lengths to keep the
non-negative invariant sound for downstream `as usize` casts.
The UDAF accumulator's own observation counter (transient aggregation
state, not part of the persisted histogram value) is intentionally left
as uint64.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(native-histogram): rename count/zero_count fields to _i64
Now that the integer payload columns are stored as int64, rename the
field constants and persisted names to match:
COUNT_U64_FIELD ("count_u64") -> COUNT_I64_FIELD ("count_i64")
ZERO_COUNT_U64_FIELD ("zero_count_u64") -> ZERO_COUNT_I64_FIELD ("zero_count_i64")
The local builder variables and the docs/JSON snapshot are updated to
match. No backward-compat (unreleased feature).
Signed-off-by: Ning Sun <sunning@greptime.com>
* test(native-histogram): refresh planner plan snapshot for signed types
The mixed native-histogram range test embeds the full histogram Struct
type in its expected plan string, which still carried the pre-rename
unsigned fields. Update the snapshot to match the signed schema:
positive/negative_span_lengths: List(UInt32) -> List(Int32)
count_u64/zero_count_u64: UInt64 -> count_i64/zero_count_i64: Int64
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
(cherry picked from commit 3510ef7d4c)
(cherry picked from commit d636d6fd9e9474b262982f54dc5805f80127087a)
When a bulk insert request carries a stale schema, the worker fills the
missing columns via BulkPart::fill_missing_columns before writing. The
method replaced the batch but kept raw_data (the original Arrow Flight
IPC bytes), while BulkWalEntry::try_from(&BulkPart) prefers raw_data, so
the memtable received the filled batch but the WAL recorded the pre-fill
bytes.
Replaying such an entry restores a batch that misses the filled columns:
- Bulk memtable (flat format): convert_bulk_part fails with
ColumnNotFound; the error is swallowed by the no-op write notifier and
the rows are silently lost after restart.
- Time series memtable: BulkPart::to_mutation builds rows shorter than
the declared schema and the region worker panics with index out of
bounds during replay, hanging the region open.
Fixes:
- fill_missing_columns clears raw_data so the WAL entry is re-encoded
from the filled batch.
- replay_memtable fills missing columns for replayed bulk parts of dense
regions, so entries already written by affected versions replay
correctly.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 1f4a8ad71cdfc135cf571ea7bca2cb87ed90ad72)
(cherry picked from commit 7fae38e667a17b3cd3de3f0540cb3a7b6d5a277a)
* fix(mito2): fail open when Bloom IN predicate has non-literal or unencodable members
collect_in_list previously filtered out non-literal and encoding-failed IN
members and could build a partial hard-pruning predicate, so scanning with
the Bloom filter enabled could prune rows that actually match the query
(false-negative results).
Now any non-literal member or any encoding failure disables Bloom pruning
for the whole IN expression (fail open), while independent AND
subpredicates such as `col = 42` are still extracted as before. Encoding
errors are logged and ignored, so queries never fail.
Adds builder unit tests covering pure-literal, pure-nonliteral, mixed
literal+null, all-null, mixed-nonliteral-with-AND, and encoding-failure
cases, plus an engine-level witness test asserting Bloom-on and Bloom-off
scans return identical rows for a mixed IN filter.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): rename bloom filter tests to semantic names without bug-id prefixes
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 70ddf1685eb63d6faca2d0009fe157369c950f73)
(cherry picked from commit 5cbab008ffcdd83c0c5603b81448e1a9b2790254)
* fix(query): avoid unsafe count wildcard rewrites
Signed-off-by: discord9 <discord9@163.com>
* fix(query): preserve outer count alias
Signed-off-by: discord9 <discord9@163.com>
* fix(query): address review comments on count wildcard rewrite
- Remove the has_projection check: the row count is correct regardless
of whether a projection exists (per review).
- Explain why checking the first input is equivalent to checking all
inputs (a plan with zero inputs falls back to count(1)).
- Rename qa_ prefixed tests to follow the module convention.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(tql): update tql-cte expectations for count wildcard rewrite
The QP-026 count-wildcard fix rewrites count(*) -> count(time_index), so
the EXPLAIN output for the filtered/final CTE aggregates names the
time-index column. Aligns tql-cte.result with the actual output (CI
failure).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 7223f2a9b80a476448e3bded312e072cd1585e2a)
* fix(prometheus): custom column remote reads
Resolve timestamp and value column names from the table schema and carry
them through query planning and result conversion. Add a remote-read
regression test covering custom_ts and custom_value.
Signed-off-by: grezzko <me@gauravshokeen.com>
* fix: resolve remote-read value columns safely
Prefer the sole field for custom schemas and greptime_value for
multi-field tables. Reject ambiguous schemas and add regression tests.
Signed-off-by: grezzko <me@gauravshokeen.com>
---------
Signed-off-by: grezzko <me@gauravshokeen.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>
(cherry picked from commit 3d7d7e916c5c8643d57d61616c180396193f8911)
* fix(frontend): remove gRPC DDL panics for DropView and non-timestamp time index
Direct gRPC DDL bypasses the SQL parser, so two client-controlled DDL
payloads could panic a request handler:
- QX-152: DdlExpr::DropView hit todo!() (instance/grpc.rs:247-248).
Wire it to the real drop-view implementation (drop_view was
pub(crate); widened to pub) so a DropView DDL returns a structured
error (e.g. TableNotFound) instead of panicking.
- QX-153: a CreateTableExpr whose time_index column is not a timestamp
reached Schema::new's unwrap (ddl.rs:2346 -> schema.rs:114-119).
create_table_info now uses Schema::try_new with ConvertSchemaSnafu
context (InvalidArguments), and the direct gRPC CreateTable arm
validates the request via validate_create_expr (which now also checks
the time-index column type is a timestamp) before any catalog work.
SQL/HTTP paths were already protected by the parser; unchanged.
Tests: qx_152_drop_view_via_grpc_ddl_returns_error_not_panic,
qx_153_create_table_with_non_timestamp_time_index_via_grpc_returns_error
(asserts InvalidArguments), test_create_table_info_rejects_non_timestamp_time_index.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(frontend): add gRPC DDL happy-path coverage for DropView and CreateTable
Per review: the initial tests only asserted error paths. Add:
- drop_if_exists=true on a missing view succeeds (no error)
- dropping an existing view via gRPC DDL succeeds end-to-end
- a valid CreateTableExpr with a timestamp time index still succeeds
(guards validate_create_expr against rejecting good requests)
- qx_152 test now asserts the TableNotFound status instead of is_err()
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 11e785ed5680dbb07b7cf88ec1b9642cde5ba1e1)
* fix(flow): fix flow stats aggregation and df_plan_to_sql quoting
1. Distributed-mode flow stats last-writer-wins overwrite:
Each flownode heartbeat put its local flow state map into the single
global __flow/state key, so reports from different nodes overwrote
each other. Store per-flownode reports under
__flow/state/node/{node_id} in the in-memory KV and aggregate on each
heartbeat (last_exec_time_map/state_size/start_time_map take the max
across nodes) into the global key. FlowStateHandler derives node
identity from header.member_id (fallback peer.id) and ignores
identity-less reports. Per-node keys clear automatically on leader
change KV reset. Adapts to FlowStateValue.start_time_map added in
#8392.
2. df_plan_to_sql unquoted special characters break flush/scheduled
execution: ForceQuoteIdentifiers only quoted uppercase identifiers,
so Prometheus-style table names with ':' (e.g. cpu_cores:sum) were
left unquoted, producing invalid SQL ('keyword: :'). Quote any
identifier with non-[a-z0-9_] chars using double quotes
(dialect-neutral).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): address review comments on quoting and logging
- df_plan_to_sql: also quote digit-leading identifiers (e.g. 123metrics)
which would produce invalid SQL when re-parsed. SQL keywords are
intentionally not checked (ALL_KEYWORDS would over-quote common column
names like number; the unparse failure path has an InsertIntoPlan
fallback).
- flow_state_handler: downgrade identity-less report log from warn! to
debug! to avoid an anomalous sender spamming warn every heartbeat.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 375dd569482235914ef09ef5c21e15cc79fcd052)
* feat: support discarding unflushed region data
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): wake stalled writers after discard
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): drop redundant manifest check for discarding unflushed data
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 62d2c43b54c6bc45de5d49305073e396d6daf434)
* chore(deps): bump datafusion to 452cb4b (support Dictionary literals in substrait)
Backport of #8839 to release/v1.2.
Bump the GreptimeTeam/datafusion fork rev from 6d6ae9a to 452cb4b,
which includes fix(substrait): support Dictionary literals in producer.
This fixes flow queries against dictionary-encoded PK string columns
(metric tables) failing with 'NotImplemented("Unsupported literal:
Dictionary(UInt32, Utf8(...))")'.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito): make do_region_gc pub(crate) to fix private_interfaces warning
clippy runs with -D warnings and this private_interfaces warning fails the
backport CI on release/v1.2.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: gate soft-drop table behind the enterprise feature
Soft-drop table becomes an enterprise-only feature:
- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
registration, and information_schema.recycle_bin registration are
compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
(moving to the enterprise repo); affected information_schema .result
files are regenerated
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: limit unused_variables allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: include the config key in the soft-drop enterprise gate error
Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: limit unused_mut allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: reject soft-drop DDL submissions in non-enterprise builds
Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: stop --enable-gc from enabling soft drop in the sqlness template
Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate fresh soft-drop procedures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: gate soft-drop fallback coverage
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate soft-drop procedure implementation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate drop table soft-drop behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate expired soft-drop gc behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: test enterprise table ddl lifecycle
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark purge_table as enterprise licensed
The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark recycle_bin as enterprise licensed
The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark soft-drop procedure sources as enterprise licensed
The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): fence async index builds by schema generation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): retry stale index builds after schema changes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Weny Xu
shuiyisong
discord9
dennis zhuang
Morax
Ning Sun
jeremyhi
fys
LFC
sun
Gaurav Shokeen
Yingwen
Lei, HUANG
discord9