* feat(operator): pair calls edges across trace tables and derive virtual-node edges
Union the normalized client and server spans of all trace tables before the
join, so a client span pairs with a server span stored in a different table.
A client span with no matching server span becomes an edge to a virtual node
named by span attributes (peer.service / db.name / server.address), with
confidence < 1.0 and attributes.connection_type; a window's real pairs win
over virtual candidates for the same edge key.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): derive same-row co-declared edges from the built-in vocabulary
A table declaring both entity types of a vocabulary pair witnesses the edge
on every row carrying both identities: runs_on / contains / part_of for any
declaring table (provenance 'attribute'), agent uses model / agent invoked
tool only for trace sources (span-structure observations, provenance
'trace').
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): derive parent_agent-calls-agent edges from span structure
Trace tables declaring an agent entity pair each span with its child span
across tables (no span-kind filter), keep pairs whose agent identities
differ, and aggregate RED metrics per window, anchored on the parent span
like the service derivation is anchored on the client.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(frontend): feed co-declared and agent sources into the relationships scan
scan_relationships now passes every declaring table (with its trace-ness)
to the co-declared branch and the trace tables' agent declarations to the
agent-calls derivation. enumerate validates the fixed trace-v1 columns and
derives around a malformed trace table instead of failing the whole scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover cross-table pairing, virtual nodes, co-declared and agent edges
sqlness exercises the new derivations end to end (including a malformed
trace-model table being skipped); the integration authorization test now
also pins that a pair split across tables derives no edge when the caller
cannot read one side.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: update the relationships module doc for the new branches
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: import shared derivation helpers via crate paths
The fmt CI gate rejects module-level 'use super::' imports.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: fold co-declared duplicates, decouple agent calls, verify the trace time index
Review findings: the co-declared branch lacked a cross-source DISTINCT, so
two tables witnessing the same edge in one window emitted duplicate rows;
the agent-calls derivation was gated on a usable service declaration; the
trace schema guard accepted a table whose time index is not the column the
derivations bucket by. The empty-trace-table test asserted a union
invariant with no information and is dropped.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: rename the agent-tool edge to invokes and track current OTel peer attributes
The vocabulary's other relation names are present tense; semconv 1.39/1.26
replaced peer.service and db.name with service.peer.name and db.namespace,
so the virtual-node candidates now check the current names first and keep
the deprecated ones for existing telemetry.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: trust the trace-v1 table option instead of matching the fixed schema
The option is only ever stamped by the ingest path, which guarantees the
fixed span columns; matching column types here couples the graph to every
trace schema evolution (e.g. #8816) for a case that cannot occur.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(frontend): run entity-graph derivation as the caller
The derivation contract requires the computed graph tables to run under
the outer query's identity. Capture the caller's QueryContext when the
computed table is resolved, thread it through EntityGraphProvider, and:
- authorize every contributing source table against the caller via the
new semantic_graph.query permission action, silently excluding denied
sources (entities, edges and source_tables never appear);
- execute the derivation plan under the caller's context so it inherits
permissions, cancellation and deadline instead of a fresh default.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): derive the entity-graph window from the scan's time predicate
Implements the RFC window contract for the computed graph tables:
- table: add extract_time_range_strict, a strict variant of the lenient
time-range extraction that distinguishes an absent observed_at filter
from one that cannot be safely turned into a range;
- operator: replace GraphWindow with GraphQueryWindow, splitting the
queried observed_at range from the source-scan range widened to whole
60s buckets, so boundary buckets aggregate over their full extent;
- frontend: resolve the window from ScanRequest filters — no predicate
keeps the last-hour default, a missing upper bound means now, and a
missing lower bound or unextractable shape is an explicit error, never
a silent fallback.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): system-defined declared-edge table for the entity graph
Reintroduces greptime_private.semantic_relationships_declared with a
canonical, system-owned definition:
- the CREATE TABLE expr (8-tag primary key, business validity columns,
RED fields, 30d TTL); attributes is now a json column so the future
union branch matches the computed table without a per-scan parse;
- created on first use on every write path: SQL INSERT creates it
before executing, and the gRPC row-insert auto-create substitutes the
canonical expr instead of deriving a schema from the request;
- user DDL (CREATE/ALTER/DROP/RENAME/TRUNCATE) and write-path
auto-ALTER are rejected via the new is_ddl_reserved_table guard,
while INSERT/DELETE stay allowed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): union declared edges into semantic_relationships
Adds the declared-edge branch to the relationship derivation
(build_relationships_plan replaces build_calls_plan):
- latest revision per edge key first (mito dedups on primary key plus
observed_at, so a re-asserted edge stores a new revision), then the
business-validity overlap against the queried window; valid_from
defaults to the declaration time and a NULL valid_until means the
edge holds while its row exists;
- the projected observed_at is synthesized inside the queried range
(Inexact pushdown re-applies the scan's filters above the computed
table, which would drop rows keyed by the physical revision time);
window_end/fresh_until of open-ended edges take the window's upper
bound so 'fresh_until >= now() - ...' queries see them;
- tag columns are cast out of dictionary encoding, and the union is
re-projected to the 16-column contract;
- the frontend feeds the branch only when the physical table exists,
the caller may read it, and its schema still matches the canonical
definition (mismatch is an explicit error, not a silent drop).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover declared edges, window contract and caller authorization
- sqlness: system auto-create on first INSERT, latest-revision reads,
open-ended vs retired validity, explicit/lower-only/upper-only window
behavior, user-DDL rejection, rename-into rejection, DELETE cleanup;
- integration: a permission checker denying one trace table excludes it
from both semantic_relationships and semantic_entities.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: allow DROP/TRUNCATE on the declared-edge table and fix CI lints
The definition guard rejected every DDL, which left sqlness (and any
shared deployment) no way to remove the table the semantic_graph case
creates — its extra region then broke unrelated region/partition case
expectations. Narrow the guard to what actually protects the canonical
definition: user CREATE, ALTER, RENAME-into and repartition stay
rejected, while DROP and TRUNCATE are allowed — dropping loses nothing
structural, the next INSERT recreates the table canonically, and DROP
doubles as the recovery path if the canonical definition ever changes.
The sqlness case now verifies drop-then-recreate and cleans up after
itself.
Also: rustfmt for the catalog crate and two typo fixes.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: adapt canonical declared-table create to TriggerReason
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: address review on the declared-edge table lifecycle and revision reads
- gRPC first writes actually work now: the reserved table's creation
went through the generic create_table_inner, which the definition
guard itself rejects; both branches of create_or_alter_tables_on_demand
route it to create_declared_relationships_table instead, and being a
system action it also bypasses the auto_create_table config/hint;
- revision selection is as-of the queried window: revisions recorded
after the window's end, or whose validity starts after it, no longer
outrank (and hide) the revision that was in effect inside it;
- the canonical-schema check validates the whole definition the union
semantics lean on — time index, primary key, engine, append/merge
mode — not just column names and types;
- UNDROP TABLE of the reserved name is rejected like CREATE: it could
resurrect a pre-canonical shape, and the next INSERT recreates the
table anyway.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim over-commenting in the entity-graph code
Comments that restated adjacent code or narrated justification are cut;
the ones stating non-obvious contracts and gotchas stay.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject CREATE VIEW against DDL-reserved table names
A view named greptime_private.semantic_relationships_declared would
squat the reserved name: the first INSERT then skips the canonical
create (an object already exists) and graph reads fail on the schema
mismatch. CREATE VIEW now passes the same definition guard as CREATE
TABLE.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: debug-log authorization exclusions; declared-edge TTL to 90d
Sources the derivation contract silently excludes (per-table denial,
whole-scan denial, the declared-edge table) are invisible from outside;
a debug log at each names what was excluded and why.
The declared-edge table's default TTL becomes 90d, overridable at
creation time via GREPTIMEDB_DECLARED_RELATIONSHIPS_TTL (a proper
configuration option is a TODO).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: rank declared-edge revisions by the visible edge identity
Ranking partitioned by the full primary key, but the projection drops
scope and generation_id: two assertions of the same visible edge under
different generations both ranked first and came out as duplicate,
indistinguishable rows. Rank by the exposed identity (endpoints,
rel_type, provenance) instead, with generation_id/scope as
deterministic tie-breakers for same-timestamp assertions.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: drop redundant declared-edge tests
The generations regression is already asserted by the revision and
as-of tests; the DDL shape test restated the declarative builder
against itself. Its one non-tautological check (attributes maps to the
json type) moves into the schema-matcher test.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject disjunctive graph windows and unmatchable future windows
- OR/IN over observed_at collapse disjoint ranges into their convex
hull; a declared edge's synthesized timestamp can land in a gap and
be dropped by the re-applied filter even though the edge is valid at
a requested instant. The strict extractor now rejects those shapes.
- A lower bound in the future inverts against the implicit up-to-now
upper bound; the declared branch then fabricated an edge observed at
the future bound. Such windows now derive nothing.
- The reserved-table gRPC create path classifies an instant-TTL table
like every sibling path.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: provide query information for post_execute interceptor
* test: update for tests-integration
* feat: make interceptor available to prometheus serialization
* feat: revert post_execute change
* feat: add expr to pre_execute and remove serialization interceptor
* chore: lint
* fix: table resolving logic related to pg_catalog
refer to
https://github.com/GreptimeTeam/greptimedb/issues/3560#issuecomment-2287794348
and #4543
* refactor: remove CatalogProtocol type
* fix: sqlness
* fix: forbid create database pg_catalog with mysql client
* refactor: use QueryContext as arguments rather than Channel
* refactor: pass None as default behaviour in information_schema
* test: fix test
* refactor: bring metrics to http output
* chore: remove unwrap
* chore: make walk plan accumulate
* chore: change field name and comment
* chore: add metrics to http resp header
* chore: move PrometheusJsonResponse to a separate file and impl IntoResponse
* chore: put metrics in prometheus resp header too
* feat: add random weigted choose in load_based selector
* fix: meta cannot save heartbeats when cluster have no region
* chore: print some log
* chore: remove unused code
* cr
* add some logs when filter result is empty
dennis zhuang
Ning Sun
Ruihang Xia
Weny Xu
fys
LFC
JohnsonLee
tison
shuiyisong
ClSlaid
AntiTopQuark
JeremyHi
LFC