* feat(servers): add signed→unsigned int coercion to OTLP ingest path
Phase 0 of transitioning built-in data models from unsigned to signed
integers (#8793): add lossless Int64→UInt64 and Int32→UInt32 coercion
arms so existing UInt64/UInt32 columns (e.g. trace `duration_nano`, log
`trace_flags`) keep accepting new signed ingest without an ALTER TABLE.
The OTLP ingest path already reconciles every incoming column against the
existing table schema and treats it as authoritative. With these arms,
`choose_trace_reconcile_decision` returns `UseExisting(UInt64/Uint32)`
for an existing unsigned column receiving signed data: the table keeps
its type byte-for-byte and the request value is coerced. No persisted
format is mutated; existing data stays readable as-is. This is the safety
net that makes the actual schema flip (Phase 1) safe.
Only the signed→unsigned direction is supported — the reverse would be
lossy for values above the signed range and is intentionally rejected.
Tests cover both new arms plus an end-to-end log test proving an existing
UInt64 column coerces an incoming Int64 value while keeping its type.
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: add compatibility layer for uint trace/log fields
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: jaeger test
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: keep trace v0 unsigned, reject negative span durations
- Keep the frozen v0 data model on UInt64 duration_nano: the signed
ingest compatibility layer only runs on the v1 path, so flipping v0
would break writes into every pre-existing v0 table at mito's schema
check. Pin the schema with unit and integration tests.
- Reject spans whose end precedes their start (or whose duration does
not fit i64) on the v1 path instead of wrapping: new Int64 tables and
existing UInt64 tables now fail identically, rather than storing
negative durations that break the Jaeger query API.
- Extract is_supported_signed_to_unsigned_coercion so the trace and log
ingest paths share one supported-pair predicate and cannot drift.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: clamp negative span durations to zero, revert semantic_graph comment
- Record duration 0 for spans whose end precedes their start instead of
erroring: a malformed span no longer fails the request, and the value
written is always a non-negative, in-range i64 so new Int64 tables and
existing UInt64 tables (via the checked coercion) behave identically.
Durations above i64::MAX saturate rather than wrap.
- Revert the doc-comment tweak on the semantic_graph test fixture; the
file is untouched by this PR again.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(query): restore columnar group-by for dictionary-encoded tags
Bump the DataFusion fork to be93ffd85 (feat/dict-group-column-53), which
backports apache/datafusion #23187: DictionaryGroupValuesColumn lets
dictionary-encoded group keys use the columnar GroupValuesColumn fast
path (hash distinct dictionary values once per batch, resolve rows by
key index) instead of falling back to row-based GroupValuesRows.
This fixes the TSBS double-groupby regression introduced by #8541
(preserve dictionary-encoded query labels): v1.2.0-beta.1 scan output
changed tag columns to Dictionary(UInt32, Utf8), which DataFusion 53.1.0
did not support in GroupValuesColumn's supported_type allow-list, so
GROUP BY queries silently dropped to the ~60% slower row path
(time_calculating_group_ids +57%, peak_mem +50%, end-to-end +38%).
Adds an end-to-end integration test (dict_groupby_sst) that flushes a
flat-format SST with dictionary-encoded hostname, runs the tsbs-style
double-groupby query, and asserts correct results with no CastExec
inserted before the aggregate.
Signed-off-by: discord9 <discord9@greptime.dev>
* chore(deps): pin merged dictionary group-by support
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@greptime.dev>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: discord9 <discord9@greptime.dev>
* feat(otlp): report the cause of rejected trace spans
When trace-v1 ingestion cannot coerce an attribute value, it falls back to
single-span writes and rejects the bad span. That behavior is correct, but the
OTLP partial-success message only carried `Rejected span <trace_id>:<span_id>
(InvalidArguments)`: the column, the source value, the source type and the
target type were all dropped, so locating the bad attribute required adding a
detailed exporter on the collector side and replaying traffic.
Two places lost the information. `prepare_trace_column_rewrites` built a message
without the failing value, and the span rejection path kept only the status code
from the error.
Coercion errors now name the failing value, e.g.
failed to coerce trace column 'span_attributes.http.response.body.size'
in table 'opentelemetry_traces' from String("") to Int64
and the rejection detail carries that cause. Values are user data, so a string
keeps at most 16 characters, is escaped, and binary payloads report only their
length; the cause itself is bounded at 256 characters. Both truncations cut on a
char boundary.
Failure details now deduplicate: repeats of the same (site, cause) collapse into
one entry with an occurrence count, keyed on the untruncated cause so two
failures that differ past the display limit stay separate. Only four distinct
entries are retained and the rest are counted, which keeps the state bounded no
matter how many distinct bad values a request carries. A fully rejected request
is logged at warn level and a partial success at debug level, since the latter
repeats every export interval; the detail goes out as a Debug field so a newline
in an attribute key cannot forge log lines.
Rejection semantics are unchanged: partial success, same accepted and rejected
counts, same HTTP status mapping.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(otlp): compare failure keys directly instead of hashing
The dedup identity was a DefaultHasher fingerprint of `(label, key)`, which
bought a fixed 8 bytes per entry at the cost of an import, four lines, and a
collision argument the reader has to make. Entries are capped at four and a
cause runs a couple of hundred characters, so the saving is about a kilobyte
per in-flight request while the column name it avoids retaining is already held
several times over by the request itself.
Compare the strings instead, keeping the untruncated cause as the key so
failures differing past the display limit still stay apart. Labels are metric
label values and always static, so the entry borrows them.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: embed the derivation conventions as data and adopt gen_ai entity naming
Move the co-declared edge vocabulary, the agent-edge vocabulary and the
virtual-destination candidates from Rust consts into an embedded
conventions.yaml (include_str!), parsed once behind a LazyLock and
validated against the entity-type grammar and the closed rel_type set; a
broken file propagates as a plan error instead of panicking. The agent
vocabulary entity types follow the GenAI semantic-convention namespace
as written: gen_ai.agent / gen_ai.model / gen_ai.tool.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: drop the tag requirement for entity identity columns
Entity declarations no longer require id columns to be tag/primary-key
columns; only column existence is validated. Trace pipelines flatten the
identifying attributes (span_attributes.gen_ai.agent.id, ...) into field
columns, so the tag rule locked real trace tables out of declaring
entities while buying no correctness — the read-time derivation works on
any column.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: implicit declarations for well-known prometheus info metrics
Tables stamped signal_type=metric + source=prometheus whose name matches
the conventions.yaml whitelist (kube_pod_info, kube_node_info,
kube_pod_owner, target_info) get implicit entity declarations: k8s.pod /
k8s.node / k8s.workload with name-based identity and target_info's
service / service.instance with the remaining tags as the descriptive
snapshot. The existing co-declared vocabulary then derives runs_on and
part_of from the same rows, so no new edge branch is needed. Explicit
declarations of a type always suppress the implicit one, and the metric
engine's physical table is excluded (it aggregates every logical
table's columns).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover the prometheus conventions in sqlness and compact the graph cases
Add the whitelisted-info-metric scenario (kube_pod_info, kube_pod_owner,
target_info deriving runs_on / part_of, a non-whitelisted metric
contributing nothing), fold the single-table calls, cross-table pairing
and virtual-node cases into one trace scenario (they exercise the same
union-before-join path), merge the two declaring-metric-table cases, and
reuse one rename probe for both reserved names.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject entity id columns without a stable string form
Review follow-ups: the DDL check now validates against the schema and
rejects binary-backed and nested types for identity columns (the
derivation renders ids via CAST to Utf8, so the failure used to surface
only when the graph was scanned); the agent sqlness case keeps its
identity columns as fields to cover the relaxed tag rule end to end;
stale tag-rule comments and a dangling const reference are cleaned up.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: type-check every entity column role, not only ids
The registry renders scope and descriptive values through the same
CAST-to-string path as ids, so a binary-backed column in any role fails
at scan time; the DDL check is now role-independent (and simpler).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: name the code-anchored vocabulary constants
Entity types and edge attributes the derivation code itself anchors on
(service, gen_ai.agent, calls, trace/attribute provenance) become
constants in the conventions module; the rest of the vocabulary stays
YAML-only data. ImplicitEntity is renamed PromImplicitEntity, and the
implicit-declaration path logs each skip of a whitelisted info metric
(wrong stamps, suppressed by an explicit declaration, missing id
column) so a missing graph entity is diagnosable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: single-source the graph constants
The graph tables' column names move to common-catalog (the schemas
catalog exposes and the plans operator builds must match column by
column), and the conventions module now carries the complete built-in
vocabulary — entity types, rel_types, provenances and connection types —
with the embedded YAML validated by membership against it, so an edit
drifting outside the vocabulary fails the conventions test instead of
deriving nothing.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: treat empty identity components as absent
kube-state-metrics emits empty-string labels an entity id must not be
built from: an unscheduled pod's node and an owner-less pod's owner_kind
/ owner_name. Standard Prometheus drops empty labels (they arrive as
NULL and the existing predicate handles them), but other remote-write
agents may keep them, which produced ghost entities with empty ids and
false runs_on / part_of edges. Every identity predicate (registry,
co-declared edges, span endpoints) now requires non-NULL and non-empty
components through one shared helper.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: tighten the conventions DSL semantics
Rename the co-declaration rule lists to what they are (co_declared_edges
/ trace_co_declared_edges — derivation rules, not a relation
vocabulary), stop overstating the GenAI entity types (Greptime types
derived from GenAI attributes; OTel defines no model/tool entities),
move target_info's descriptive snapshot to service.instance (the
remaining labels are the target's resource attributes, and instances
would write conflicting snapshots onto the logical service), and extend
the descriptor whitelist with the stable KSM sources: container info
metrics (closing the k8s.pod contains k8s.container rule),
kube_service_info (new k8s.service entity type) and the fuller
descriptive label sets.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: guard entity column types on ALTER as well
ALTER MODIFY COLUMN could change a declared entity column to a type
without a stable string form, deferring the failure to graph scan time;
verify_alter now checks the post-alter schema. Dropping a declared
column stays allowed — the read-time derivation skips the stale
declaration, and semantic options cannot be altered off yet.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: bridge traces and kube-state-metrics on the pod UID
Trace-v1 tables now get implicit declarations from their flattened
resource attributes (otlp_trace_entities in conventions.yaml): the
service identity — replacing the hardcoded fallback — plus
service.instance and k8s.pod, each applied only when its columns exist.
A new co-declared rule derives service.instance runs_on k8s.pod, and
the whitelisted kube-state-metrics pod identity switches from
namespace+pod names to the UID, so the trace-side pod and every KSM
descriptor land on one entity while names stay descriptive. This also
removes pod identity from the multi-cluster same-name collision.
The conventions rejection tests were passing for the wrong reason (a
half-renamed fixture key failed deserialization before reaching any
validation rule); they now assert the specific error each case targets.
Sqlness covers the UID merge across descriptor tables, pod-contains-
container, the k8s.service node, and the empty-uid/empty-node rows
deriving nothing.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover the OTLP-to-graph chain end to end
One real OTLP export must come out of semantic_relationships as the
zero-configuration chain: service calls service, instance part_of
service, instance runs_on pod (bridged by k8s.pod.uid). Resources
without service.instance.id or k8s.pod.uid derive nothing extra.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: identify k8s.service by UID
Same reasoning as pods: a recreated same-name service must not merge
into the old entity and same-named services across clusters must not
collide; kube_service_info carries a stable uid and nothing joins on the
service's name. Also drop a stale tag-rule mention from the option
validation docs.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: cut duplicated test coverage and redundant comments
The trace service-fallback test collapsed into the resource-entities
test (same synthesis path since the fallback moved to YAML; only the
invalid-explicit-no-fallback case was distinct), role-duplicate and
subsumed DDL cases are gone, the embedded-conventions test is just the
parse (its assertions were decorative), and the YAML section comments no
longer restate the struct docs.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(native-histogram): store counts and span lengths as signed integers
Native histograms are unreleased, so the on-disk integer payload columns
are switched from unsigned to signed types without backward-compat:
- count_u64 / zero_count_u64: uint64 -> int64
- positive_span_lengths / negative_span_lengths: list(uint32) -> list(int32)
- Span.length (query-time model): u32 -> i32
The Prometheus remote-write v2 source carries these as uint64/uint32, so
the unsigned->signed conversion at the ingestion boundary is overflow
checked: an integer count >= 2^63 or a span length >= 2^31 is rejected
with an explicit error rather than silently wrapping to a negative value.
read_spans additionally rejects negative stored lengths to keep the
non-negative invariant sound for downstream `as usize` casts.
The UDAF accumulator's own observation counter (transient aggregation
state, not part of the persisted histogram value) is intentionally left
as uint64.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(native-histogram): rename count/zero_count fields to _i64
Now that the integer payload columns are stored as int64, rename the
field constants and persisted names to match:
COUNT_U64_FIELD ("count_u64") -> COUNT_I64_FIELD ("count_i64")
ZERO_COUNT_U64_FIELD ("zero_count_u64") -> ZERO_COUNT_I64_FIELD ("zero_count_i64")
The local builder variables and the docs/JSON snapshot are updated to
match. No backward-compat (unreleased feature).
Signed-off-by: Ning Sun <sunning@greptime.com>
* test(native-histogram): refresh planner plan snapshot for signed types
The mixed native-histogram range test embeds the full histogram Struct
type in its expected plan string, which still carried the pre-rename
unsigned fields. Update the snapshot to match the signed schema:
positive/negative_span_lengths: List(UInt32) -> List(Int32)
count_u64/zero_count_u64: UInt64 -> count_i64/zero_count_i64: Int64
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat(servers): stamp prometheus remote write v2 metadata as semantic table options
Remote write 2.0 carries per-series metadata (type, unit, help) that the
v2 ingest decoded and dropped; tables kept the name-based 'inferred'
quality. Wire it into the semantic layer:
- generalize the OTLP per-table semantic index into a shared, schema-
aware servers::semantic module: v2 lets each series override its
target schema, so the index is keyed {schema -> table -> options} and
the same metric name in two schemas no longer collapses;
- into_write_requests records metric type and unit per written table;
an explicit type upgrades the table's metadata quality to declared,
UNSPECIFIED series keep the request-level inferred stamp, and units
are canonicalised from OpenMetrics words to the UCUM codes the
vocabulary is defined in (unknown units are dropped, help text is not
persisted);
- the consumer folds the index in on both auto-create paths: the
operator row-insert path and the pending-rows batched create, which
bypasses the former.
Table options are stamped at auto-create only; updating existing tables
from later metadata is future work (a metadata registry, see the native
histograms RFC).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim over-commenting in the remote write metadata path
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: parse the per-table semantic index once per create round
The index was re-parsed from JSON for every table being created — a
first write creating N tables (a fleet's first scrape) paid
O(N x index size). Parsing now happens lazily once per create-planning
round, on both consumers: the operator row-insert auto-create (also
serving OTLP metrics) and the pending-rows batched create.
Also validate every non-zero metadata symbol reference up front, as the
remote write 2.0 spec requires: help_ref was never checked, and
unit_ref escaped checking when the metric type was UNSPECIFIED.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): stamp remote-write v2 unit independently of metric type
OpenMetrics models TYPE and UNIT as independent MetricFamily metadata, and
the Prometheus v2 sender emits UNSPECIFIED-type series that still carry a
unit. The early return on UNSPECIFIED dropped that unit, which is
unrecoverable after table auto-create (units are not stored in rows).
Stamp the mapped unit whenever present; the type and the
metadata_quality=declared upgrade still require an explicit type.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim restating comments in the v2 metadata path
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(table): add entity semantic declarations
Define open-ended greptime.semantic.entity.* options, validate entity columns at DDL time, and stamp OTLP trace tables with the service entity declaration.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add read-time entity relationships graph
Add computed semantic graph tables, typed DataFusion derivation plans for entity registry and trace calls edges, and streaming read-time execution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: exclude semantic graph tables from table constraints
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(operator): name the plan-builder source groupings
Review feedback: build_registry_plan / build_calls_plan took anonymous
(declarations, DataFrame) tuples while the caller already grouped the same
fields. Introduce RegistrySource { declarations, scan } and CallsSource
{ service, scan } next to the builders and flow them through the frontend
caller and tests. The frontend-side EntitySource keeps holding a TableRef
(the operator builders stay pure over already-built scans), so the named
structs live in operator rather than reusing that type. No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: adjust the position of experimental_enable_prometheus_native_histogram
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* chore: move prom_validation_mode as well
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* chore: gate soft-drop table behind the enterprise feature
Soft-drop table becomes an enterprise-only feature:
- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
registration, and information_schema.recycle_bin registration are
compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
(moving to the enterprise repo); affected information_schema .result
files are regenerated
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: limit unused_variables allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: include the config key in the soft-drop enterprise gate error
Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: limit unused_mut allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: reject soft-drop DDL submissions in non-enterprise builds
Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: stop --enable-gc from enabling soft drop in the sqlness template
Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate fresh soft-drop procedures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: gate soft-drop fallback coverage
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate soft-drop procedure implementation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate drop table soft-drop behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate expired soft-drop gc behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: test enterprise table ddl lifecycle
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark purge_table as enterprise licensed
The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark recycle_bin as enterprise licensed
The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark soft-drop procedure sources as enterprise licensed
The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add a dedicated http api server port
* fix: integration test
* refactor: make http-api-port opt-in
* refactor: rename attribute to http-api-server
* feat: use middleware to check different http server port
* refactor: rename config option
Skip scheduled meta GC while cluster maintenance mode is enabled and reject manual GC requests explicitly instead of returning an empty success report.
Also increase mito GC's default lingering time to 1h and update generated config docs and config API expectations.
Signed-off-by: discord9 <discord9@163.com>
Ning Sun
discord9anddiscord9
dennis zhuang
Weny Xu
Yingwen
shuiyisong
Lei, HUANG
Whis Liao
yimeng
Yash Agrawal