* fix(prometheus): custom column remote reads
Resolve timestamp and value column names from the table schema and carry
them through query planning and result conversion. Add a remote-read
regression test covering custom_ts and custom_value.
Signed-off-by: grezzko <me@gauravshokeen.com>
* fix: resolve remote-read value columns safely
Prefer the sole field for custom schemas and greptime_value for
multi-field tables. Reject ambiguous schemas and add regression tests.
Signed-off-by: grezzko <me@gauravshokeen.com>
---------
Signed-off-by: grezzko <me@gauravshokeen.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>
(cherry picked from commit 3d7d7e916c5c8643d57d61616c180396193f8911)
* fix(frontend): remove gRPC DDL panics for DropView and non-timestamp time index
Direct gRPC DDL bypasses the SQL parser, so two client-controlled DDL
payloads could panic a request handler:
- QX-152: DdlExpr::DropView hit todo!() (instance/grpc.rs:247-248).
Wire it to the real drop-view implementation (drop_view was
pub(crate); widened to pub) so a DropView DDL returns a structured
error (e.g. TableNotFound) instead of panicking.
- QX-153: a CreateTableExpr whose time_index column is not a timestamp
reached Schema::new's unwrap (ddl.rs:2346 -> schema.rs:114-119).
create_table_info now uses Schema::try_new with ConvertSchemaSnafu
context (InvalidArguments), and the direct gRPC CreateTable arm
validates the request via validate_create_expr (which now also checks
the time-index column type is a timestamp) before any catalog work.
SQL/HTTP paths were already protected by the parser; unchanged.
Tests: qx_152_drop_view_via_grpc_ddl_returns_error_not_panic,
qx_153_create_table_with_non_timestamp_time_index_via_grpc_returns_error
(asserts InvalidArguments), test_create_table_info_rejects_non_timestamp_time_index.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(frontend): add gRPC DDL happy-path coverage for DropView and CreateTable
Per review: the initial tests only asserted error paths. Add:
- drop_if_exists=true on a missing view succeeds (no error)
- dropping an existing view via gRPC DDL succeeds end-to-end
- a valid CreateTableExpr with a timestamp time index still succeeds
(guards validate_create_expr against rejecting good requests)
- qx_152 test now asserts the TableNotFound status instead of is_err()
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 11e785ed5680dbb07b7cf88ec1b9642cde5ba1e1)
* fix(flow): fix flow stats aggregation and df_plan_to_sql quoting
1. Distributed-mode flow stats last-writer-wins overwrite:
Each flownode heartbeat put its local flow state map into the single
global __flow/state key, so reports from different nodes overwrote
each other. Store per-flownode reports under
__flow/state/node/{node_id} in the in-memory KV and aggregate on each
heartbeat (last_exec_time_map/state_size/start_time_map take the max
across nodes) into the global key. FlowStateHandler derives node
identity from header.member_id (fallback peer.id) and ignores
identity-less reports. Per-node keys clear automatically on leader
change KV reset. Adapts to FlowStateValue.start_time_map added in
#8392.
2. df_plan_to_sql unquoted special characters break flush/scheduled
execution: ForceQuoteIdentifiers only quoted uppercase identifiers,
so Prometheus-style table names with ':' (e.g. cpu_cores:sum) were
left unquoted, producing invalid SQL ('keyword: :'). Quote any
identifier with non-[a-z0-9_] chars using double quotes
(dialect-neutral).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): address review comments on quoting and logging
- df_plan_to_sql: also quote digit-leading identifiers (e.g. 123metrics)
which would produce invalid SQL when re-parsed. SQL keywords are
intentionally not checked (ALL_KEYWORDS would over-quote common column
names like number; the unparse failure path has an InsertIntoPlan
fallback).
- flow_state_handler: downgrade identity-less report log from warn! to
debug! to avoid an anomalous sender spamming warn every heartbeat.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 375dd569482235914ef09ef5c21e15cc79fcd052)
* feat: support discarding unflushed region data
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): wake stalled writers after discard
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): drop redundant manifest check for discarding unflushed data
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 62d2c43b54c6bc45de5d49305073e396d6daf434)
* chore(deps): bump datafusion to 452cb4b (support Dictionary literals in substrait)
Backport of #8839 to release/v1.2.
Bump the GreptimeTeam/datafusion fork rev from 6d6ae9a to 452cb4b,
which includes fix(substrait): support Dictionary literals in producer.
This fixes flow queries against dictionary-encoded PK string columns
(metric tables) failing with 'NotImplemented("Unsupported literal:
Dictionary(UInt32, Utf8(...))")'.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito): make do_region_gc pub(crate) to fix private_interfaces warning
clippy runs with -D warnings and this private_interfaces warning fails the
backport CI on release/v1.2.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: gate soft-drop table behind the enterprise feature
Soft-drop table becomes an enterprise-only feature:
- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
registration, and information_schema.recycle_bin registration are
compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
(moving to the enterprise repo); affected information_schema .result
files are regenerated
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: limit unused_variables allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: include the config key in the soft-drop enterprise gate error
Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: limit unused_mut allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: reject soft-drop DDL submissions in non-enterprise builds
Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: stop --enable-gc from enabling soft drop in the sqlness template
Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate fresh soft-drop procedures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: gate soft-drop fallback coverage
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate soft-drop procedure implementation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate drop table soft-drop behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate expired soft-drop gc behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: test enterprise table ddl lifecycle
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark purge_table as enterprise licensed
The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark recycle_bin as enterprise licensed
The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark soft-drop procedure sources as enterprise licensed
The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): fence async index builds by schema generation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): retry stale index builds after schema changes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito2): split compaction module into scheduler/reader submodules
Extract the compaction scheduler lifecycle (scheduler, status, phases,
execution, SST reservations, pending requests) and its tests out of
compaction.rs into compaction/scheduler.rs and compaction/scheduler_test.rs.
Split the remaining helpers by responsibility:
- estimate_compaction_bytes/refresh_picker_output move to scheduler.rs,
the only call site
- get_expired_ssts moves to picker.rs, shared by the TWCS and window
pickers
- CompactionSstReaderBuilder/time_range_to_predicate/ts_to_lit move to
the new compaction/reader.rs
The root compaction.rs keeps the shared output types and
find_dynamic_options, and re-exports the moved types so existing call
paths stay unchanged. Pure code motion, no behavior change.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): separate compaction scheduling from execution details
Turn compaction/scheduler.rs into a directory module to make the
scheduling flow easier to review:
- scheduler.rs keeps the pure scheduling core: the CompactionScheduler
state machine, scheduling entry points, termination chaining, DDL
coordination and region lifecycle events
- scheduler/planning.rs holds the execution-facing parts: background
planning dispatch, picker invocation, plan acceptance, remote/local
submission and memory estimation
- scheduler/state.rs holds the per-region lifecycle types:
CompactionStatus, ActiveCompaction, CompactionPhase, CompactingFiles,
LocalCompactionState, CompactionExecution and PendingCompaction
Child modules keep access to the scheduler's private methods, so the
split is pure code motion with minimal visibility changes (pub(super)
only where the parent module or tests reach into child items).
No behavior change.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): use absolute scheduler imports
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): document compaction scheduler modules
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): make async index publication conditional
Check the captured SST generation and commit index metadata under the same manifest write lock. Revalidate the committed metadata before applying it to the in-memory version, and clean exact-version artifacts when either publication stage becomes stale.
Add deterministic compaction and overlapping-index tests covering reopen consistency, duplicate rows, cache cleanup, and both file purgers.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito2): centralize manifest update finalization
Share the locked update, lock release, follower check, and hook firing path between regular manifest updates and conditional index publication.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): retain index build leases across reopen
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): avoid retiring scheduler on index failure
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): handle cross-region index publication
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): use physical region for index paths
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(cmd): update noop index builder
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): avoid reusing published index versions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito2): log untracked index build stops
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): set compaction time range in index test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: expose MitoRegion::all_manifest_files for metadata rebuild
Adds a public read-only accessor that returns all live SST file metas
and the current manifest version from the region manifest. Used by the
downstream project admin path to enumerate the authoritative live file
set without going through the worker loop.
* fix(mito2): merge staging manifest files in all_manifest_files
The original implementation only read the normal manifest
(manifest_ctx.manifest()) and skipped staging_manifest(). While the
region is in staging mode (e.g. region copy/migration), the authoritative
live file set lives in the staging manifest, so callers would silently
miss those files.
Now matches the semantics of manifest_sst_entries() (~L771), which
explicitly merges manifest().files with staging_manifest().files via a
HashMap collect (dedup by FileId). The returned manifest version is the
staging version when a staging manifest is present, otherwise the normal
manifest version.
Also removed downstream-specific references from the rustdoc comments.
discord9
sun
Ning Sun
Gaurav Shokeen
Yingwen
jeremyhi
shuiyisong
Lei, HUANG
discord9
Weny Xu
LFC
dennis zhuang
fys