* feat(runtime): add weighted workload scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics
Use the GreptimeTeam/catio fork (pinned c20eafc) which adds
ClassStats::total_admission_wait and ClassStats::admitted, recorded
at each QUEUED -> ADMITTED transition. This exposes the scheduler's
own admission delay (excluding Tokio queueing and poll execution),
enabling admission-wait based fairness gates.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to dynamic-config revision
Bump the catio scheduler fork to 9f4b028 which adds
Scheduler::set_weight and Scheduler::set_max_concurrent_polls for
runtime configuration.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(perf): runtime-adjustable workload scheduler parameters
Expose dynamic adjustment of the experimental workload scheduler at
runtime:
- common-runtime: set_workload_scheduler_weights and
set_workload_scheduler_max_concurrent_polls, which forward to the
catio scheduler's set_weight/set_max_concurrent_polls when the
scheduler is enabled and reject zero values.
- servers: /debug/workload_scheduler/weights and
/debug/workload_scheduler/max_concurrent_polls POST handlers, so
operators can rebalance query/write shares or admission concurrency
without restarting the datanode.
Both endpoints return 400 with a clear reason when the scheduler is
disabled or the requested value is invalid.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(perf): add GET /debug/workload_scheduler status endpoint
Returns the current weights (per class), max_concurrent_polls,
active_polls and per-class counters (queued, tasks, wakes, polls,
completed, cancelled, admitted, total_admission_wait) as JSON. When the
scheduler is disabled, returns enabled=false with the other fields
omitted, so operators can distinguish 'disabled' from an error.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to time-accounting revision
Bump the catio scheduler fork to 257ba56 which replaces
admission-count accounting with real execution-time accounting
(pass += exec_time / (weight * concurrency)), so CPU share follows the
configured weights regardless of poll length.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to lock-free sampling revision
Bump the catio scheduler fork to efdc0a4 which adds an optional
downsampled clock sampling mode (SchedulerBuilder::sample_every_polls,
default off) with a lock-free per-class atomic counter, so the
downsampled path costs one fetch_add per poll instead of a global
mutex.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin catio to scheduler PR head
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): add scheduler bypass control
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: advance catio scheduler fixes
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin merged catio scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: regenerate config docs for workload scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin catio scheduler test fix
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(http): satisfy scheduler lifecycle clippy
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: add distributed scheduler toggle coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat: finalize workload scheduler runtime controls
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin merged catio atomic weights
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: preserve unrelated lockfile resolution
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(runtime): downsample scheduler time accounting
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(runtime): verify cross-runtime scheduler progress
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): configure scheduler poll sampling
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(runtime): clarify scheduler activation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(runtime): explain scheduler use case
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
The k8s.pod runs_on k8s.node rule can only fire from kube_pod_info: it is
the only table declaring both endpoints, and co-declared edges require
both on the same row. A deployment sending only OTLP has no
kube-state-metrics tables, so its node layer is invisible and that rule
has no source at all, even though the resource attributes carry
k8s.node.name.
Declare k8s.node in otlp_trace_entities and in the synthesized resource
descriptor, and project k8s.node.name in the descriptor writer so the
column the declaration needs exists. Identity is the name rather than
k8s.node.uid: kube-state-metrics carries no node UID, so the name is the
only identity both sources can agree on.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: allow widening the time index column's timestamp unit via ALTER TABLE ... MODIFY COLUMN
Previously MODIFY COLUMN rejected the time index column outright. Now the
time index unit can be widened (Second -> Milli -> Micro -> Nano), which is
lossless for data that fits the target unit: historical data in old SSTs is
cast to the new unit on read by the existing schema-compat layer, and
compaction rewrites it lazily. Narrowing and non-timestamp targets remain
rejected; tag columns keep being rejected. Widening is rejected if any
SST's time range would overflow the target unit's i64 range (e.g.
millisecond -> nanosecond beyond year 2262), since the cast would silently
null those values.
Read-path correctness for old-unit SSTs (verified by new engine e2e tests
and sqlness WHERE queries):
- row-group min/max pruning: parquet statistics of a timestamp column are
raw integers in the file's unit; when the region metadata's type differs
(also the case for altered field columns), stats are now interpreted in
the file's type and converted to the expected type before pruning.
Without this, a new-unit predicate silently pruned whole row groups of
old-unit files (wrong results, rows missing).
- SST-level simple filters are skipped for columns whose file type differs
from the expected type; the predicate is applied by the query layer's
residual filter above the region scan. Also drop the stale
"timestamp columns cannot change type" debug_assert.
- retry idempotency: a same-type ModifyColumnType on the time index
validates as a no-op and `need_alter` returns false, so a retried alter
procedure (region already altered before the previous attempt failed)
converges instead of aborting forever.
- add TimeUnit ordering and ConcreteDataType::is_timestamp_unit_widening_to
- relax ModifyColumnType validation in store-api and table metadata
- tests: unit tests in datatypes/store-api/table; mito2 engine e2e tests
(flushed SST + new writes + reopen + retry + overflow + predicate scans,
cross-unit dedup, mixed-unit SSTs, compaction); sqlness cases incl.
partitioned table and WHERE filters over old-unit data
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: resolve parquet filter issue
* test: provide sqlness tests
* refactor: drop trivial test cases and shorten comments
Review pass over the branch's additions:
- datatypes: keep a representative subset of the widening-matrix asserts
- table: collapse the three single-branch rejection blocks into one loop
- mito2: drop the boundary gt_eq and post-compaction predicate asserts
(covered by the exact-filter regression test and sqlness); drop the
engine-level gt_eq/lt_eq casts (full operator matrix stays in the
cast_timestamp_unit unit tests)
- sqlness: drop a bare full scan already covered by the filter above it
- shorten function doc comments across datatypes/store-api/table/mito2/
recordbatch to the essential semantics
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: drop physical prefilter for columns whose file type differs
Follow-up to the review feedback on CompatBatch/prune reader/filter
handling for widened time index units.
Between/InList/IsNull predicates are prefiltered by PhysicalFilterContext,
which builds its physical expression against the FILE's schema while the
predicate literals are in the expected (post-alter) unit. Evaluating them
against an old-unit SST raised a cross-unit comparison error (Timestamp(ms)
>= Timestamp(µs)) that failed the whole scan. Physical prefilter predicates
are best-effort pruning hints (the query layer re-applies them above the
scan), so drop the prefilter when the column's file type differs from the
expected type, mirroring the simple-filter strategy.
Verified: Between and a non-rewritten (large) InList on old-unit data no
longer error and filter exactly end-to-end (sqlness), and no matching rows
are lost at the engine level (engine test).
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: add direct unit tests for stats cast and prefilter drop
The two-step stats cast (reinterpret raw Int64 stats in the file's
timestamp type, then rescale to the expected type) and the physical
prefilter drop on file/expected type mismatch were only covered
end-to-end; add localized unit tests so a regression fails at the
exact site:
- stats.rs (previously no tests): RowGroupPruningStats min/max over a
hand-built RowGroupMetaData — passthrough with no expected metadata,
passthrough on same type, and rescale (1000ms -> 1_000_000us, not
1000us) on a widened expected unit
- reader.rs: PhysicalFilterContext::new_opt keeps a Between prefilter
when file and expected types match and drops it on unit mismatch
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: tolerate mixed time units range cache key coverage check
* docs: flag mixed-unit hazard in the (unwired) series index
The series index stores per-series min/max ts as raw Int64 in the unit
of the region metadata at write time, and the searcher builds its range
predicates from a single per-region metadata. After a time index unit
widen, files of one region would carry mixed units, so a per-file unit
(or an index rebuild on such alters) is required before this index is
wired into scans. Leave notes at both sites.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: cover mixed-unit compaction for sparse encoding and strict windows
Compaction-path audit follow-up. The compat cast and window math were
already covered for dense regions; add the two remaining e2e scenarios:
- sparse primary key encoding (used by metric-engine physical regions):
widening then compacting mixed-unit files rewrites the old-unit time
index correctly through the sparse compaction compat path
- strict-window manual compaction: each window output trims rows with a
predicate built in the region's new unit against an old-unit file;
every instant must survive exactly once (no loss, no cross-window
duplication), rescaled
Also documents the audit finding that Regular ranged (manual)
compaction never trims rows: TwcsPicker sets output_time_range to None
and the request time range only selects candidate windows.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: cover time index unit change in FlatCompatBatch directly
The compat layer's rescaling of a widened time index was only verified
end-to-end; add direct unit tests for both paths:
- dense: identical units skip compat entirely; a widened unit rescales
the time index column (1000ms -> 1_000_000us, not reinterpreted) while
other columns pass through and the output schema matches the expected
metadata
- compact sparse (the metric-engine compaction path): same rescaling
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: move timestamp unit division into common-time
The exact unit division (UnitQuotient + div_mod_units) is time
semantics, not filter logic; move it next to TimeUnit in common-time
with a compact test covering representable/non-representable values,
negative (floor) instants, and quotient overflow. The ScalarValue
helpers stay in filter.rs since common-time has no datafusion
dependency.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: compat rescales a widened time index and fills an added column together
The realistic multi-alter sequence (widen at T1, add column at T2,
read a T0 SST) exercises cast and default-fill in the same
compute_index_and_fields pass; assert both in one output batch.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: preflight time index widening overflow before any region alters
Address review feedback on the overflow guard:
- preflight: when the frontend operator receives a widening alter on the
time index, run an existence scan (ts outside the target unit's i64
range, LIMIT 1, via the query engine so it covers every region of the
table in both standalone and distributed modes) BEFORE any DDL task is
submitted. A region that fits can no longer commit the new schema
while another region rejects the alter with a non-retryable error.
File and row-group pruning keep the scan cheap when nothing overflows.
The per-region check in mito2 stays as the final guard for data
written after the preflight (the remaining race window); without a
validate-only wire field (region.proto lives in the external
greptime-proto repo) a fully atomic two-phase validate/commit is out
of scope here.
- fast path: cast_timestamp_unit returns the filter unchanged when the
literal is already in the target unit, skipping the div-mod rebuild.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
* fix: address auto review comments
* fix: remove time index widening overflow preflight
Overflow needs timestamps beyond the target unit's i64 range (~year
2262 for nanoseconds), which real workloads never write, so the two
existence scans before every widening alter are not worth the cost.
Region validation already rejects the alter when an SST's time range
overflows the target unit; it now logs the rejection (with the
offending file) and returns a deterministic client-facing message.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: make sqlness test stable
* fix: log instead of rejecting time index widening overflow
Overflowing values cast to NULL on read but do not otherwise affect
reads or writes, so the alter is allowed; the region-level check now
only logs (with the offending file) when an SST's time range exceeds
the target unit's i64 range.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Add `WriteCacheUploadStoreWrapper` in `src/mito2/src/cache/write_cache.rs` and wire it through `src/mito2/src/worker.rs`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(semantic-graph): let the generic container yield to k8s.container
A container inside a Kubernetes pod reaches the graph twice: as the
k8s.container entity kube-state-metrics describes, identified by
[pod uid, container name], and as the generic container the OTel resource
attributes describe, identified by container.id. One physical container,
two nodes.
Kubernetes is the primary scenario, so k8s.container keeps its identity
and the generic type stands down where it applies. Conventions gain a
row-level condition for that: `suppressed_by` withdraws a declaration on
rows where any of the named columns has a value. The test has to be per
row, not per table — one descriptor table holds both pod rows and
bare-runtime rows.
Every branch that turns a declaration into rows now shares one guard
(`declaration_predicate`), so the condition cannot apply to entities but
not to the edges they carry.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(semantic-graph): report derived entity declarations in table_semantics
information_schema.table_semantics only read table options, so the
declarations the built-in conventions derive — for trace tables, for
whitelisted Prometheus and OTel descriptor metrics — were invisible.
"Why is my table not in the graph?" was answerable only from debug logs,
which is not a self-service path.
A new `entity_declarations` column reports the entities a table actually
contributes: each one's identity, whether it came from an option or from
the conventions, and any row-level condition attached to it. An expected
entity missing from the list is the answer — the table name is not
whitelisted, the source stamp is wrong, an id column is absent.
The row filter widens to match: a table that declares nothing by option
but derives entities by convention now appears, since it is in the graph
and the view has to say so. The provider reaches the derivation through a
new metadata-only trait method, keeping catalog below operator.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(semantic-graph): never yield a container to an entity nothing derives
The generic container withdrew on any row carrying a pod UID, on the
assumption that k8s.container would cover it. Nothing guaranteed that:
k8s.container came only from kube-state-metrics, so an OTLP-only
deployment — or one whose KSM data had expired or fell outside the query
window — lost the container node and its edges entirely instead of
gaining a more specific one.
The rule now names the superseding type rather than a trigger column, and
withdraws only where that type's full identity is on the row. Both OTel
sources declare k8s.container themselves, under the identity
kube-state-metrics gives it ([pod uid, container name]), so the two
sources name one node; `k8s.container.name` joins the descriptor's
projected attributes to carry it.
Resolution runs once every declaration for the table is known, so a
superseding type that ends up undeclared — its columns are gone, or an
explicit declaration of it was skipped — leaves the guard empty and the
generic container standing. A container can change type; it cannot
disappear.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(semantic-graph): build the declaration JSON by serializing a type
The column was assembled entry by entry into a serde_json::Map, cloning
every value and allocating a String per key. A Serialize struct that
consumes the declaration moves the same data instead, and the field order
now reads type, origin, identity, then description.
The scan path around it was doing the same kind of avoidable work:
declarations were derived before the predicate could discard the table,
the supersession pass cloned every declaration's identity to look up one,
and the option parse claimed the time index for tables that turn out to
declare nothing.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(semantic-graph): keep the structured identity for single-column ids
`entity_id_attrs` was NULL whenever the identity came from one column, so a
consumer holding `host` = `a3f2...` had no way to tell which column produced
it, and no way back to the source table. Most identities are single-column —
host, k8s.pod, k8s.node, service — so the common case was the opaque one.
Build the JSON object unconditionally. Entity equality still reads `entity_id`
alone, so this changes no merging: it records which attributes the id was
assembled from, beside an id that deliberately omits them.
Also corrects the `entity_id` column doc, which still described the `k=v,k=v`
rendering replaced in #8904 by values joined in declared order.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(semantic-graph): keep what a container carried when it yields
Yielding to k8s.container cost the row two things it had as a generic
container. The runtime container id, which kube_pod_container_info keeps
descriptive precisely because it is the handle back to runtime logs and
metrics, went missing entirely: neither an id nor an attribute of any
node. And the edge vocabulary knew only the generic type, so a row with
the more complete labels ended up with fewer connections than one
without — the container layer no longer reached its host.
Both OTel sources now keep the runtime id and name descriptive on
k8s.container, and the vocabulary gains the two edges that mirror the
generic type's. Nothing checks a supersession against the edge
vocabulary, so that requirement is written down where the rule is.
Also: the conventions-failure path now reports the explicit half as its
comment always claimed, entity_declarations reports scope columns, and
identifies() is private again now that only declaration_predicate calls
it. The two RFCs catch up with entity_id_attrs being unconditional, the
view listing convention-derived tables, and supersession existing.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(semantic-graph): report unmatched clients and the longest request
`real_wins` pins a `calls` edge's RED metrics to the observed span pairs:
when a window's edge key holds any pair, the unmatched client spans are
suppressed so one (window, edge) yields one row. That leaves no way to
tell a callee that stopped responding from traffic that stopped arriving —
both show up as a lower request_count.
Add two columns to `semantic_relationships`:
- `unmatched_count` — client spans with no server span, counted outside
`real_wins` on the same row, so the suppressed population stays visible
without splitting the edge into two rows. NULL for agent calls, whose
inner join leaves nothing unmatched, and for declared edges.
- `duration_max` — the longest single request. It goes through
`real_wins`: a pair is timed by the server span while an unmatched
client is timed by its own (network wait included), so mixing them would
make the max describe a different population than duration_sum and
duration_count. Agent calls compute it from the child spans they already
aggregate.
The projection contract goes from 16 to 18 columns; every branch projects
both. Explicit column queries are unaffected, `SELECT *` and
ordinal-based readers see the new shape.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(semantic-graph): drop the duplicated source-gate case
The whitelist gate on `source=opentelemetry` is already covered by
`otel_implicit_declarations_are_gated` and by the wrong-source case in
`table_semantics`; here it only paid for another table create, insert and
full graph derivation.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* style(semantic-graph): trim the comments back to what the code cannot say
Several comments restated the code, repeated a rationale already stated at
the type or in the RFC, or explained a test in more words than the test
body.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: postgres describe for more statements
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: cover more show statements
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
- add missing `clippy::too_many_arguments` allow on
`query_from_information_schema_dataframe` (CI clippy failure)
- take `&ShowKind` in the information-schema dataframe helper so `kind`
is no longer cloned at every call site; only the WHERE arm (which needs
an owned expression for `sql_to_expr`) clones internally
- document why re-applying TQL explain formats never overwrites an
existing value (per-query context state)
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: trim comments to essentials
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: Add built-in daemon mode for standalone service
Add an optional --daemon/-d flag to `greptimedb standalone start` to run the server as a background daemon detached from the shell session, similar to Redis `daemonize yes`.
- Use the daemonize crate (Unix only) to fork/setsid and detach from the controlling terminal before the Tokio runtime is built.
- stdin/stdout/stderr are redirected to /dev/null; logs continue to go to data_home/logs/ via the existing tracing appender.
- On non-Unix platforms, --daemon is a no-op with a warning (runs in foreground).
Closes#7314
Signed-off-by: tian1220A <1573612521@qq.com>
* test(cmd): add integration test for standalone daemon mode
Verify that standalone start --daemon detaches from the shell and brings up the HTTP listener, guarding against regressions where the daemon blocks in the foreground or crashes after forking.
Signed-off-by: tian1220A <1573612521@qq.com>
* fix(cmd): gate --daemon flag with #[cfg(unix)] and fix rustfmt
Address review feedback to not expose --daemon/-d on non-Unix platforms. The daemon field is now gated with #[cfg(unix)], and a cross-platform is_daemon() accessor returns false on non-Unix so maybe_daemonize() stays unconditional.
Also fix the rustfmt formatting issues that caused fmt-check to fail.
Signed-off-by: tian1220A <1573612521@qq.com>
* chore(cmd): declare daemonize dependency directly in cmd
daemonize is only used by the cmd crate. Move it out of the workspace
dependencies and declare it directly in the unix-only dependencies of
src/cmd/Cargo.toml.
Signed-off-by: tian1220A <1573612521@qq.com>
---------
Signed-off-by: tian1220A <1573612521@qq.com>
* refactor(mito2): prioritize file count in TWCS picker
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): migrate TWCS tests to async picker API
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): remove legacy reduce_runs and merge_seq_files pickers
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): balance TWCS picks by file group
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): treat multi-SST groups as barriers in TWCS pick_count_first
Previously pick_count_first filtered out multi-SST file groups and could
pick singleton groups across them in one interval. Now multi-SST groups
split the candidates into independent segments, so a single pick never
crosses a multi-SST group.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): remove redundant overlaps_files_left_behind check
The rebase onto main kept both the upstream fix (#8872) and the branch's
selected_overlaps_unselected check for the same deletion-marker problem.
The upstream check is fully subsumed: files_to_merge only differs from
the window files in append mode, where filter_deleted is already false,
and selected_overlaps_unselected treats partially-selected groups as
unselected, making it strictly stronger.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): pre-filter by selection time span in selected_overlaps_unselected
A window group outside the overall time span of the selected groups
cannot overlap any of them, so skip the precise overlap check (and the
per-group file id set lookup) for it. In typical TWCS windows the pick
is clustered in one segment, so most unselected groups are rejected in
O(1).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): remove FileGroup abstraction from TWCS picker
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): drain compactable backlog after successful compaction
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): score TWCS candidates by predicted progress
Redefine what a pick candidate is worth: an interval is only eligible if
compacting it makes progress on at least one axis — it reduces the
physical file count given the max_output_file_size split threshold
(predicted output = ceil(input bytes / threshold)), or it resolves at
least one overlap between sorted runs. A pure rewrite that achieves
neither (e.g. 32 large balanced files whose output would split back into
just as many SSTs) is skipped instead of burning I/O.
The candidate metrics are accumulated in a Candidate struct as the
interval expands, and the score ranks by predicted file reduction, then
overlap participants, then smaller input bytes. With no output size
limit the behavior degenerates to the previous count-first rule.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): make TWCS input limit configurable
Allow operators to tune the maximum SST inputs through a hidden environment variable while keeping 32 as the validated default.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): apply TWCS trigger at window level
Check the total physical SST count before candidate selection so trigger values above the per-task input limit still allow bounded compactions.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Provide valid distributed frontend options in `src/cmd/src/frontend.rs` so enterprise plugin setup can preserve prefilled heartbeat extensions without weakening production validation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mysql): interpret prepared statement datetime params in session timezone
Binary DATETIME parameters of server-side prepared statements were
converted as if UTC, ignoring the session timezone set via SET time_zone.
Convert them with the session timezone and add an integration test
covering prepared inserts and predicates under Asia/Shanghai.
Signed-off-by: wy471x <wy471x@gmail.com>
* refactor: share naive datetime timezone policy via common-time
Address review feedback on the prepared-statement timezone fix:
- Expose Timestamp::from_naive_datetime in common-time so the DST policy
(gap -> error, ambiguous -> earlier instant) lives in one place, shared
by the text protocol (Timestamp::from_str) and the MySQL binary protocol.
- Route the MySQL prepared-statement datetime conversion through it.
- Match the target type before converting datetime params so
PreparedStmtTypeMismatch fails fast without wasted conversion.
- Use the short Timezone import form for consistency with the rest of servers.
Signed-off-by: wy471x <wy471x@gmail.com>
---------
Signed-off-by: wy471x <wy471x@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Add a typed `Command::build_with_heartbeat_extensions` seam in `src/cmd/src/frontend.rs`.
Freeze and harden `FrontendHeartbeatExtensions` in `src/frontend/src/heartbeat.rs`, with lifecycle and race coverage in `src/frontend/src/heartbeat/tests.rs`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(servers): compose OTLP metrics job from service.namespace/service.name
The OTel Prometheus compatibility spec defines job as
"<service.namespace>/<service.name>" when the namespace is present.
The OTLP metrics path only used the bare service.name, so the job tag
diverged from target_info produced by Prometheus-side exporters for the
same resource. Compose the namespace form, and keep not fabricating a
job when service.name is absent.
Behavior change: resources carrying service.namespace now get
"namespace/name" as their job tag value.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(otlp): synthesize otel_resource_info at OTLP metrics ingestion
Ordinary OTLP metrics scatter filtered resource attributes as tags over
every logical metric table, so metrics-only services contribute nothing
to the semantic entity graph. Each request now also projects its
distinct resources into one info-metric-shaped mito table,
otel_resource_info: a fixed allowlist of identity-relevant attributes
under their raw OTel keys (independent of the label translation
strategy and the promote/ignore headers) plus derived job/instance
compatibility columns, value 1.0, and the newest data-point timestamp.
The descriptor is written after the main insert is committed; a failure
there (conflicting pre-existing table, auto-create disabled) degrades
to an OTLP partial_success warning with rejected_data_points = 0
instead of failing the request and triggering client retries of
already-accepted data. A request writing a metric named
otel_resource_info suppresses synthesis. Legacy mode is unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): otel info-metric conventions with host/container entities
Whitelist the ingestion-synthesized otel_resource_info descriptor via a
new otel_info_metrics conventions map, gated on source=opentelemetry
(the existing gate hardcoded source=prometheus). Its declarations use
explicit descriptive lists instead of descriptive_rest so identifying
attributes of other entities do not leak into service.instance.
Conventions tightened per the Astronomy Shop findings: host identity is
host.id with host.name descriptive only (host.name is not stable across
SDKs and resource detectors), a generic container entity (new entity
type) is declared only when container.id is present, and trace-v1
tables now synthesize host/container from their flattened resource
attributes too. New co-declared edges: service.instance runs_on
container, container runs_on host.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): cover the resource descriptor in integration tests
Covers the descriptor's raw-key columns and info-metric options through
the HTTP path, the namespace/name job composition end-to-end, column
names being independent of the translation strategy, the allowlist
excluding unlisted resource attributes, auto-create after a drop, the
metric-name collision suppressing synthesis, and the partial-success
warning (rejected_data_points = 0) when a pre-existing incompatible
table fails the descriptor write while metric data is accepted.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: cargo fmt
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(frontend): degrade descriptor permission denial to a warning
A table-level permission policy denying otel_resource_info would have
failed the whole OTLP metrics request because the descriptor's
permission check ran before the main insert. The descriptor is derived
enrichment: check its permission in the degrade path so a denial skips
the write and surfaces as the partial-success warning, like any other
descriptor write failure.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): guard descriptor writes with semantic ownership markers
A pre-existing schema-compatible table named otel_resource_info would
silently receive descriptor rows while its missing semantic stamps kept
it out of the entity graph. The descriptor write now requires the
auto-created table's ownership markers (mito engine + signal_type +
source + metric.type=info + metadata_quality=declared) and otherwise
degrades to the partial-success warning; the entity-graph gate for the
otel whitelist likewise requires metric.type=info, so a user table
stamped with only signal/source no longer picks up implicit
declarations.
Also fold the descriptor write cost into the response and surface the
degrade warning through the otel-arrow BatchStatus status_message.
Integration tests pin the full marker set on auto-create and that an
existing owned descriptor keeps accepting writes without degrading —
a missing marker would otherwise silently stop every descriptor write
after the first request.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(otlp): build descriptor rows without the per-resource BTreeMap
Projecting a resource allocated a BTreeMap and then collected it into the
row key, and every attribute was matched against the allowlist by linear
scan. Collect the tags into a Vec and sort once, and match the allowlist
instead of scanning it. Measured on the conversion path: descriptor work
drops 16-18%, from 10.6% to 8.9% of conversion CPU on the worst shape
(1000 resources with 4 data points each), where the cost tracks resource
count rather than data-point count.
Also trims the comments and tests added with the descriptor to what
carries information.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): pin the descriptor permission-denial degrade path
A policy denying the descriptor table must not fail the metrics request,
which the fix in 2401b3dd9c does but nothing covered. Verified as a
regression guard by mutation: moving the permission check back before
the main write makes this test fail.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): keep legacy mode covered after trimming the unit tests
Trimming the descriptor tests dropped the only assertion that legacy
mode skips the job/instance remap and the promote filter. Both alter
the columns of tables already in use, so fold the check into the legacy
conversion test rather than leaving it uncovered.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(semantic-graph): stop encoding column names into composite entity ids
A composite entity id rendered the identifying columns as sorted
`col=value` pairs, so the same identity split into one entity per
signal: a trace table names its columns service_name and
resource_attributes.service.instance.id where a metric table names them
job and instance. One service instance became two nodes with two
parallel edge sets, breaking the walk from a trace to that instance's
metrics.
Render an id as its values in declared order instead, escaping the
separator so components stay distinguishable, which is what single-column
ids already did by keeping only the value. entity_id_attrs still carries
the structured form.
Values alone are not enough for a namespaced service: the metric side
folds service.namespace into job while traces keep the bare name. Add
qualified_by to the conventions so the trace declarations compose the
namespace the same way, per the OTel rule that job is
<service.namespace>/<service.name> or the bare name when the namespace
is empty. A table without the namespace column keeps the unqualified
identity rather than losing the declaration.
Conventions validation now rejects one entity type declared with a
different number of id columns by two sources, which would silently
produce ids that can never match.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* style(otlp): import the parent module by crate path
check-super-imports.py, part of the CI format gate, rejects a
file-level `use super::`.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(otlp): gate the resource descriptor, and fix what review found
Synthesizing greptime_otel_resource_info creates and writes a table the
user never sent, so it is now off unless
otlp.experimental_enable_resource_info says otherwise. With it off the
request costs exactly what it did before the descriptor existed: nothing
is projected, no table is created, no write and no permission check
happen. Tests run with it on. StandaloneOptions carried no otlp field,
so the whole [otlp] section was silently dropped in standalone mode; map
it through, or the new option (and trace_ingest_chunk_size before it)
would do nothing there.
Renamed from otel_resource_info: the greptime_ prefix marks the table as
engine-managed and makes a collision with a user metric unlikely, which
is what the pre-existing-table ownership check and its per-request
catalog lookup were defending against. Both are gone.
A request may carry data for several graph windows, but the descriptor
folded every data-point time into one row at the newest of them, leaving
the earlier windows with metric rows and no entities. Key the rows by
window as well, and take the times from the data points the encoder
actually writes: it drops exponential histograms, and a resource
carrying nothing else was being described as an entity with no
measurements.
Projecting a resource cloned its attributes once per data point. Nest
the windows under the attributes instead, so they are moved once per
resource, and walk the data-point times through a visitor rather than
collecting a Vec per metric.
Also documents what the two maps key and hold, and lifts the projected
attribute names to constants beside KEY_SERVICE_NAME.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): skip the descriptor's work entirely when it is disabled
The collision scan over the request's output tables ran even with the
feature off. Short-circuit on the option instead, and update the config
snapshot the new [otlp] section changed.
Also drops the doc comment orphaned by the deleted ownership check: it
had attached itself to the trait impl and described a check that no
longer exists.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(semantic-graph): drop the expect and name the service identity
The CASE is built through Case directly rather than the fallible
when().otherwise() builder, so the non-test path no longer carries an
expect (architecture-invariants $4).
service_identity returned two same-typed Options that both call sites
destructured positionally; a named struct makes a swap fail to compile.
Also records that id-column order is part of the identity, where the
option docs and the conventions authors will read it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(semantic-graph): drop comments that narrate the code
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(semantic-graph): cast duration_nano before the trace-table union
Trace tables written before the signed-integer ingest change hold
duration_nano as UInt64 and later ones as Int64. The calls derivation
unions the per-table selects, and the two have no common integer type,
so a deployment holding both shapes could not build the plan. The
cross-table test now spans both.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(semantic-graph): drop the redundant duration_nano casts
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): decide exponential histogram acceptance in one place
The resource descriptor mirrored only the experimental gate, so with both
experimental flags on a resource whose only metric is a delta exponential
histogram was described as an entity with no measurements. The encoder's
whole-metric rules move into exponential_histogram_gate, which both call,
and the descriptor takes its timestamps through exponential_histogram_value
so per-point rejections drop out too.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): keep INSERT timestamp conversion out of the source query
Interpreting an INSERT's string timestamps used to work by pushing the
conversion down into the source query, which changed what that query
means. Two consequences:
- Pushing through a UNION's DISTINCT moved the dedup key from the raw
strings to parsed instants, so rows spelling the same instant
differently collapsed into one. On an append-only table that is a
silently dropped row.
- A UNION branch that needed no conversion (a NULL, or an explicit cast)
made the whole column give up, leaving sibling branches on UTC while
the rest of the row used the session timezone.
Convert at the assignment instead, by routing its cast through a
timezone-carrying timestamp type and back. Arrow applies the timezone
when a cast target carries one, and stripping it afterwards preserves
the value. The source query is no longer touched, so both cases go away
and the tree-walking rewrite (roughly 160 lines) is deleted.
The rewrite reads source types, so it now runs TypeCoercion first: a
UNION still carries its loose per-branch schema before coercion, and
retargeting a cast whose input later becomes a timestamp would shift the
value rather than reinterpret it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): address review on INSERT assignment rewrite
- Clone the input `Arc` instead of the whole subtree, and only rebuild it
when a `Values` row actually changes.
- Defer cloning the cast source until the literal-folding path has been
ruled out.
- Move the UTC check onto `Timezone::is_utc`, replacing a bare string
compare.
- Cover a prepared `INSERT ... VALUES (?)`: an untyped placeholder types
as `Null`, so the assignment cast is left for parameter substitution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(mito2): add series index searcher
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): use parquet push decoder for series index
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): handle evolved series index schemas
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(perf): align direct-SST CREATE TABLE with baked index metadata
The offline fixture generator (query_perf_fixture::direct_sst::
build_region_metadata) bakes greptime:inverted_index /
greptime:skipping_index field metadata into the region manifest for
tag/field columns, but create_table_sql emitted a bare CREATE TABLE
without those declarations. MergeScan's remote-schema validation then
failed on any tag/field projection (HTTP 500 'advertised remote stream
schema field mismatch'), breaking direct_readable_sst perf cases.
CREATE TABLE now declares the matching SKIPPING INDEX WITH
(granularity='1') / INVERTED INDEX column options. A round-trip test
proves the emitted SQL is parser-valid and yields the exact catalog
metadata.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): speed up Prometheus JSON response building with ryu and per-series entry reuse
PrometheusJsonResponse::record_batches_to_data spends ~47% of its CPU
in f64::to_string() per sample and ~32% in IndexMap::entry() per row
(60s profile of concurrent query_range workloads, ~800k series).
- Replace f64::to_string() with ryu::Buffer::format_finite for finite
values (shortest round-trip, 3-5x faster); NaN/+Inf/-Inf keep the
previous std formatting so wire output is unchanged.
- Remember the previous row's label vector and entry index; query output
is clustered by series, so consecutive rows reuse the same IndexMap
entry via get_index_mut instead of rebuilding and hashing the label
vector (worst case adds one Vec comparison per series transition).
Also adds a query-regression case (prom_json_response) that measures the
real Prometheus HTTP range API path (/v1/prometheus/api/v1/query_range),
which is the only frontend path that builds the Prometheus JSON response
(TQL ANALYZE formats the SQL JSON shape instead), plus a prom_http query
kind in the regression runner.
Perf (aligned base d90cca4b75, 256 series x 481 points):
- prom_range_2h (JSON response path): 29.31ms -> 21.46ms (-26.8%)
- tql_range_2h_control (non-JSON path): +1.87% (noise)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(servers): keep Prometheus wire format for integral floats with ryu
ryu::Buffer::format_finite prints integral values as "1.0", but the
Prometheus JSON wire format (matching std f64::to_string) expects "1".
Strip the trailing ".0" that ryu only emits for integral values; extreme
values keep ryu scientific notation, and NaN/Inf keep std output. Adds
wire-format tests covering 1.0, 0.0, -0.0, 1.5, 0.1, 1e21, 1e30, 1e-7,
f64::MAX, NaN, ±Inf.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(servers): address Prometheus response review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers)!: use ryu for Prometheus sample values
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(cmd): skip Prometheus execution time extraction
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(servers): add signed→unsigned int coercion to OTLP ingest path
Phase 0 of transitioning built-in data models from unsigned to signed
integers (#8793): add lossless Int64→UInt64 and Int32→UInt32 coercion
arms so existing UInt64/UInt32 columns (e.g. trace `duration_nano`, log
`trace_flags`) keep accepting new signed ingest without an ALTER TABLE.
The OTLP ingest path already reconciles every incoming column against the
existing table schema and treats it as authoritative. With these arms,
`choose_trace_reconcile_decision` returns `UseExisting(UInt64/Uint32)`
for an existing unsigned column receiving signed data: the table keeps
its type byte-for-byte and the request value is coerced. No persisted
format is mutated; existing data stays readable as-is. This is the safety
net that makes the actual schema flip (Phase 1) safe.
Only the signed→unsigned direction is supported — the reverse would be
lossy for values above the signed range and is intentionally rejected.
Tests cover both new arms plus an end-to-end log test proving an existing
UInt64 column coerces an incoming Int64 value while keeping its type.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: add compatibility layer for uint trace/log fields
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: jaeger test
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: keep trace v0 unsigned, reject negative span durations
- Keep the frozen v0 data model on UInt64 duration_nano: the signed
ingest compatibility layer only runs on the v1 path, so flipping v0
would break writes into every pre-existing v0 table at mito's schema
check. Pin the schema with unit and integration tests.
- Reject spans whose end precedes their start (or whose duration does
not fit i64) on the v1 path instead of wrapping: new Int64 tables and
existing UInt64 tables now fail identically, rather than storing
negative durations that break the Jaeger query API.
- Extract is_supported_signed_to_unsigned_coercion so the trace and log
ingest paths share one supported-pair predicate and cannot drift.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: clamp negative span durations to zero, revert semantic_graph comment
- Record duration 0 for spans whose end precedes their start instead of
erroring: a malformed span no longer fails the request, and the value
written is always a non-negative, in-range i64 so new Int64 tables and
existing UInt64 tables (via the checked coercion) behave identically.
Durations above i64::MAX saturate rather than wrap.
- Revert the doc-comment tweak on the semantic_graph test fixture; the
file is untouched by this PR again.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
discord9
dennis zhuang
Ning Sun
LFC
shuiyisong
Lei, HUANG
Yingwen
zhaiAohan
Weny Xu
Dhruv Vaishnav
wy471xandNing Sun
sun