Files
greptimedb/tests/cases/standalone/common/information_schema/table_semantics.sql
T
dennis zhuang 109257505e feat: report what the graph derives and fix two duplicate-node bugs (#8936)
* feat(semantic-graph): let the generic container yield to k8s.container

A container inside a Kubernetes pod reaches the graph twice: as the
k8s.container entity kube-state-metrics describes, identified by
[pod uid, container name], and as the generic container the OTel resource
attributes describe, identified by container.id. One physical container,
two nodes.

Kubernetes is the primary scenario, so k8s.container keeps its identity
and the generic type stands down where it applies. Conventions gain a
row-level condition for that: `suppressed_by` withdraws a declaration on
rows where any of the named columns has a value. The test has to be per
row, not per table — one descriptor table holds both pod rows and
bare-runtime rows.

Every branch that turns a declaration into rows now shares one guard
(`declaration_predicate`), so the condition cannot apply to entities but
not to the edges they carry.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(semantic-graph): report derived entity declarations in table_semantics

information_schema.table_semantics only read table options, so the
declarations the built-in conventions derive — for trace tables, for
whitelisted Prometheus and OTel descriptor metrics — were invisible.
"Why is my table not in the graph?" was answerable only from debug logs,
which is not a self-service path.

A new `entity_declarations` column reports the entities a table actually
contributes: each one's identity, whether it came from an option or from
the conventions, and any row-level condition attached to it. An expected
entity missing from the list is the answer — the table name is not
whitelisted, the source stamp is wrong, an id column is absent.

The row filter widens to match: a table that declares nothing by option
but derives entities by convention now appears, since it is in the graph
and the view has to say so. The provider reaches the derivation through a
new metadata-only trait method, keeping catalog below operator.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(semantic-graph): never yield a container to an entity nothing derives

The generic container withdrew on any row carrying a pod UID, on the
assumption that k8s.container would cover it. Nothing guaranteed that:
k8s.container came only from kube-state-metrics, so an OTLP-only
deployment — or one whose KSM data had expired or fell outside the query
window — lost the container node and its edges entirely instead of
gaining a more specific one.

The rule now names the superseding type rather than a trigger column, and
withdraws only where that type's full identity is on the row. Both OTel
sources declare k8s.container themselves, under the identity
kube-state-metrics gives it ([pod uid, container name]), so the two
sources name one node; `k8s.container.name` joins the descriptor's
projected attributes to carry it.

Resolution runs once every declaration for the table is known, so a
superseding type that ends up undeclared — its columns are gone, or an
explicit declaration of it was skipped — leaves the guard empty and the
generic container standing. A container can change type; it cannot
disappear.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(semantic-graph): build the declaration JSON by serializing a type

The column was assembled entry by entry into a serde_json::Map, cloning
every value and allocating a String per key. A Serialize struct that
consumes the declaration moves the same data instead, and the field order
now reads type, origin, identity, then description.

The scan path around it was doing the same kind of avoidable work:
declarations were derived before the predicate could discard the table,
the supersession pass cloned every declaration's identity to look up one,
and the option parse claimed the time index for tables that turn out to
declare nothing.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(semantic-graph): keep the structured identity for single-column ids

`entity_id_attrs` was NULL whenever the identity came from one column, so a
consumer holding `host` = `a3f2...` had no way to tell which column produced
it, and no way back to the source table. Most identities are single-column —
host, k8s.pod, k8s.node, service — so the common case was the opaque one.

Build the JSON object unconditionally. Entity equality still reads `entity_id`
alone, so this changes no merging: it records which attributes the id was
assembled from, beside an id that deliberately omits them.

Also corrects the `entity_id` column doc, which still described the `k=v,k=v`
rendering replaced in #8904 by values joined in declared order.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(semantic-graph): keep what a container carried when it yields

Yielding to k8s.container cost the row two things it had as a generic
container. The runtime container id, which kube_pod_container_info keeps
descriptive precisely because it is the handle back to runtime logs and
metrics, went missing entirely: neither an id nor an attribute of any
node. And the edge vocabulary knew only the generic type, so a row with
the more complete labels ended up with fewer connections than one
without — the container layer no longer reached its host.

Both OTel sources now keep the runtime id and name descriptive on
k8s.container, and the vocabulary gains the two edges that mirror the
generic type's. Nothing checks a supersession against the edge
vocabulary, so that requirement is written down where the rule is.

Also: the conventions-failure path now reports the explicit half as its
comment always claimed, entity_declarations reports scope columns, and
identifies() is private again now that only declaration_predicate calls
it. The two RFCs catch up with entity_id_attrs being unconditional, the
view listing convention-derived tables, and supersession existing.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(semantic-graph): report unmatched clients and the longest request

`real_wins` pins a `calls` edge's RED metrics to the observed span pairs:
when a window's edge key holds any pair, the unmatched client spans are
suppressed so one (window, edge) yields one row. That leaves no way to
tell a callee that stopped responding from traffic that stopped arriving —
both show up as a lower request_count.

Add two columns to `semantic_relationships`:

- `unmatched_count` — client spans with no server span, counted outside
  `real_wins` on the same row, so the suppressed population stays visible
  without splitting the edge into two rows. NULL for agent calls, whose
  inner join leaves nothing unmatched, and for declared edges.
- `duration_max` — the longest single request. It goes through
  `real_wins`: a pair is timed by the server span while an unmatched
  client is timed by its own (network wait included), so mixing them would
  make the max describe a different population than duration_sum and
  duration_count. Agent calls compute it from the child spans they already
  aggregate.

The projection contract goes from 16 to 18 columns; every branch projects
both. Explicit column queries are unaffected, `SELECT *` and
ordinal-based readers see the new shape.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(semantic-graph): drop the duplicated source-gate case

The whitelist gate on `source=opentelemetry` is already covered by
`otel_implicit_declarations_are_gated` and by the wrong-source case in
`table_semantics`; here it only paid for another table create, insert and
full graph derivation.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* style(semantic-graph): trim the comments back to what the code cannot say

Several comments restated the code, repeated a rationale already stated at
the type or in the RFC, or explained a test in more words than the test
body.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-08-31 10:39:17 +00:00

207 lines
6.5 KiB
SQL

DESC TABLE information_schema.table_semantics;
CREATE TABLE metrics_tagged (
ts TIMESTAMP TIME INDEX,
val DOUBLE,
)
WITH (
'greptime.semantic.signal_type' = 'metric',
'greptime.semantic.source' = 'opentelemetry',
'greptime.semantic.source_version' = '2.0',
'greptime.semantic.pipeline' = 'greptime_metric_v1',
'greptime.semantic.metric.metadata_quality' = 'declared',
'greptime.semantic.metric.type' = 'counter',
'greptime.semantic.metric.unit' = 'By'
);
CREATE TABLE traces_tagged (
ts TIMESTAMP TIME INDEX,
span_name STRING,
)
WITH (
'greptime.semantic.signal_type' = 'trace',
'greptime.semantic.source' = 'opentelemetry',
'greptime.semantic.trace.conventions' = 'https://opentelemetry.io/schemas/1.27.0'
);
-- A table with no semantic options must not appear in the view.
CREATE TABLE plain_table (
ts TIMESTAMP TIME INDEX,
val DOUBLE,
);
SELECT table_schema, table_name, signal_type, source, source_version, pipeline, metadata_quality, semantic_options
FROM information_schema.table_semantics
ORDER BY table_name;
-- Predicate pushdown on a promoted column.
SELECT table_name, signal_type
FROM information_schema.table_semantics
WHERE signal_type = 'metric'
ORDER BY table_name;
DROP TABLE metrics_tagged;
DROP TABLE traces_tagged;
DROP TABLE plain_table;
-- ALTER TABLE manages semantic declarations on existing tables: SET appears in
-- the view, UNSET disappears from it.
CREATE TABLE altered_semantics (
ts TIMESTAMP TIME INDEX,
svc STRING,
payload BINARY,
val DOUBLE,
);
ALTER TABLE altered_semantics SET 'greptime.semantic.signal_type' = 'metric', 'greptime.semantic.entity.service.id' = 'svc';
SELECT table_name, signal_type, semantic_options
FROM information_schema.table_semantics
WHERE table_name = 'altered_semantics';
-- Semantic options never share a statement with regular table options.
ALTER TABLE altered_semantics SET 'greptime.semantic.source' = 'prometheus', 'ttl' = '7d';
-- Unknown semantic keys are rejected on SET.
ALTER TABLE altered_semantics SET 'greptime.semantic.nonsense' = 'x';
-- Values outside the key's domain are rejected.
ALTER TABLE altered_semantics SET 'greptime.semantic.signal_type' = 'garbage';
-- Entity columns must exist.
ALTER TABLE altered_semantics SET 'greptime.semantic.entity.host.id' = 'no_such_column';
-- Entity columns must render as strings.
ALTER TABLE altered_semantics SET 'greptime.semantic.entity.host.id' = 'payload';
ALTER TABLE altered_semantics UNSET 'greptime.semantic.entity.service.id';
SELECT table_name, signal_type, semantic_options
FROM information_schema.table_semantics
WHERE table_name = 'altered_semantics';
DROP TABLE altered_semantics;
-- Logical metric tables take the same metadata-only path.
CREATE TABLE phy_sem (ts TIMESTAMP TIME INDEX, val DOUBLE) engine=metric with ("physical_metric_table" = "");
CREATE TABLE logical_sem (ts TIMESTAMP TIME INDEX, val DOUBLE, host STRING PRIMARY KEY) engine=metric with ("on_physical_table" = "phy_sem");
ALTER TABLE logical_sem SET 'greptime.semantic.signal_type' = 'metric', 'greptime.semantic.entity.host.id' = 'host';
SELECT table_name, signal_type, semantic_options
FROM information_schema.table_semantics
WHERE table_name = 'logical_sem';
-- Regular options still cannot be altered on logical tables.
ALTER TABLE logical_sem SET 'ttl' = '7d';
ALTER TABLE logical_sem UNSET 'greptime.semantic.entity.host.id';
SELECT table_name, signal_type, semantic_options
FROM information_schema.table_semantics
WHERE table_name = 'logical_sem';
DROP TABLE logical_sem;
DROP TABLE phy_sem;
-- entity_declarations reports what a table actually contributes to the graph,
-- including the declarations the built-in conventions derive. A whitelisted
-- OTel descriptor table declares its entities without any entity option, and
-- the generic container records the identity it yields to.
CREATE TABLE greptime_otel_resource_info (
greptime_timestamp TIMESTAMP TIME INDEX,
greptime_value DOUBLE,
"job" STRING,
"instance" STRING,
"service.name" STRING,
"host.id" STRING,
"container.id" STRING,
"container.name" STRING,
"k8s.pod.uid" STRING,
"k8s.pod.name" STRING,
"k8s.container.name" STRING,
PRIMARY KEY ("job", "instance", "service.name", "host.id", "container.id", "container.name", "k8s.pod.uid", "k8s.pod.name", "k8s.container.name")
)
WITH (
'greptime.semantic.signal_type' = 'metric',
'greptime.semantic.source' = 'opentelemetry',
'greptime.semantic.metric.type' = 'info'
);
SELECT table_name, entity_declarations
FROM information_schema.table_semantics
WHERE table_name = 'greptime_otel_resource_info';
-- The same table under the wrong source stamp derives nothing: the column is
-- empty and `source` names the reason.
CREATE TABLE kube_pod_info (
greptime_timestamp TIMESTAMP TIME INDEX,
greptime_value DOUBLE,
"namespace" STRING,
"pod" STRING,
"uid" STRING,
"node" STRING,
PRIMARY KEY ("namespace", "pod", "uid", "node")
)
WITH (
'greptime.semantic.signal_type' = 'metric',
'greptime.semantic.source' = 'influxdb'
);
SELECT table_name, source, entity_declarations
FROM information_schema.table_semantics
WHERE table_name = 'kube_pod_info';
ALTER TABLE kube_pod_info SET 'greptime.semantic.source' = 'prometheus';
SELECT table_name, source, entity_declarations
FROM information_schema.table_semantics
WHERE table_name = 'kube_pod_info';
DROP TABLE greptime_otel_resource_info;
DROP TABLE kube_pod_info;
-- A missing id column drops that entity alone: without `uid` there is no
-- k8s.pod, and the k8s.node next to it still lands.
CREATE TABLE kube_pod_info (
greptime_timestamp TIMESTAMP TIME INDEX,
greptime_value DOUBLE,
"namespace" STRING,
"pod" STRING,
"node" STRING,
PRIMARY KEY ("namespace", "pod", "node")
)
WITH (
'greptime.semantic.signal_type' = 'metric',
'greptime.semantic.source' = 'prometheus'
);
SELECT table_name, entity_declarations
FROM information_schema.table_semantics
WHERE table_name = 'kube_pod_info';
DROP TABLE kube_pod_info;
-- A trace table carries no semantic option at all, yet the conventions derive
-- its entities: it must still be visible here, or the view cannot answer why
-- it is in the graph.
CREATE TABLE traces_untagged (
"timestamp" TIMESTAMP(9) TIME INDEX,
trace_id STRING,
span_id STRING,
service_name STRING,
"resource_attributes.host.id" STRING,
PRIMARY KEY (service_name)
) WITH ('table_data_model' = 'greptime_trace_v1', 'append_mode' = 'true');
SELECT table_name, signal_type, semantic_options, entity_declarations
FROM information_schema.table_semantics
WHERE table_name = 'traces_untagged';
DROP TABLE traces_untagged;