* feat(operator): pair calls edges across trace tables and derive virtual-node edges
Union the normalized client and server spans of all trace tables before the
join, so a client span pairs with a server span stored in a different table.
A client span with no matching server span becomes an edge to a virtual node
named by span attributes (peer.service / db.name / server.address), with
confidence < 1.0 and attributes.connection_type; a window's real pairs win
over virtual candidates for the same edge key.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): derive same-row co-declared edges from the built-in vocabulary
A table declaring both entity types of a vocabulary pair witnesses the edge
on every row carrying both identities: runs_on / contains / part_of for any
declaring table (provenance 'attribute'), agent uses model / agent invoked
tool only for trace sources (span-structure observations, provenance
'trace').
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): derive parent_agent-calls-agent edges from span structure
Trace tables declaring an agent entity pair each span with its child span
across tables (no span-kind filter), keep pairs whose agent identities
differ, and aggregate RED metrics per window, anchored on the parent span
like the service derivation is anchored on the client.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(frontend): feed co-declared and agent sources into the relationships scan
scan_relationships now passes every declaring table (with its trace-ness)
to the co-declared branch and the trace tables' agent declarations to the
agent-calls derivation. enumerate validates the fixed trace-v1 columns and
derives around a malformed trace table instead of failing the whole scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover cross-table pairing, virtual nodes, co-declared and agent edges
sqlness exercises the new derivations end to end (including a malformed
trace-model table being skipped); the integration authorization test now
also pins that a pair split across tables derives no edge when the caller
cannot read one side.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: update the relationships module doc for the new branches
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: import shared derivation helpers via crate paths
The fmt CI gate rejects module-level 'use super::' imports.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: fold co-declared duplicates, decouple agent calls, verify the trace time index
Review findings: the co-declared branch lacked a cross-source DISTINCT, so
two tables witnessing the same edge in one window emitted duplicate rows;
the agent-calls derivation was gated on a usable service declaration; the
trace schema guard accepted a table whose time index is not the column the
derivations bucket by. The empty-trace-table test asserted a union
invariant with no information and is dropped.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: rename the agent-tool edge to invokes and track current OTel peer attributes
The vocabulary's other relation names are present tense; semconv 1.39/1.26
replaced peer.service and db.name with service.peer.name and db.namespace,
so the virtual-node candidates now check the current names first and keep
the deprecated ones for existing telemetry.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: trust the trace-v1 table option instead of matching the fixed schema
The option is only ever stamped by the ingest path, which guarantees the
fixed span columns; matching column types here couples the graph to every
trace schema evolution (e.g. #8816) for a case that cannot occur.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(native-histogram): store counts and span lengths as signed integers
Native histograms are unreleased, so the on-disk integer payload columns
are switched from unsigned to signed types without backward-compat:
- count_u64 / zero_count_u64: uint64 -> int64
- positive_span_lengths / negative_span_lengths: list(uint32) -> list(int32)
- Span.length (query-time model): u32 -> i32
The Prometheus remote-write v2 source carries these as uint64/uint32, so
the unsigned->signed conversion at the ingestion boundary is overflow
checked: an integer count >= 2^63 or a span length >= 2^31 is rejected
with an explicit error rather than silently wrapping to a negative value.
read_spans additionally rejects negative stored lengths to keep the
non-negative invariant sound for downstream `as usize` casts.
The UDAF accumulator's own observation counter (transient aggregation
state, not part of the persisted histogram value) is intentionally left
as uint64.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(native-histogram): rename count/zero_count fields to _i64
Now that the integer payload columns are stored as int64, rename the
field constants and persisted names to match:
COUNT_U64_FIELD ("count_u64") -> COUNT_I64_FIELD ("count_i64")
ZERO_COUNT_U64_FIELD ("zero_count_u64") -> ZERO_COUNT_I64_FIELD ("zero_count_i64")
The local builder variables and the docs/JSON snapshot are updated to
match. No backward-compat (unreleased feature).
Signed-off-by: Ning Sun <sunning@greptime.com>
* test(native-histogram): refresh planner plan snapshot for signed types
The mixed native-histogram range test embeds the full histogram Struct
type in its expected plan string, which still carried the pre-rename
unsigned fields. Update the snapshot to match the signed schema:
positive/negative_span_lengths: List(UInt32) -> List(Int32)
count_u64/zero_count_u64: UInt64 -> count_i64/zero_count_i64: Int64
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat(servers): stamp prometheus remote write v2 metadata as semantic table options
Remote write 2.0 carries per-series metadata (type, unit, help) that the
v2 ingest decoded and dropped; tables kept the name-based 'inferred'
quality. Wire it into the semantic layer:
- generalize the OTLP per-table semantic index into a shared, schema-
aware servers::semantic module: v2 lets each series override its
target schema, so the index is keyed {schema -> table -> options} and
the same metric name in two schemas no longer collapses;
- into_write_requests records metric type and unit per written table;
an explicit type upgrades the table's metadata quality to declared,
UNSPECIFIED series keep the request-level inferred stamp, and units
are canonicalised from OpenMetrics words to the UCUM codes the
vocabulary is defined in (unknown units are dropped, help text is not
persisted);
- the consumer folds the index in on both auto-create paths: the
operator row-insert path and the pending-rows batched create, which
bypasses the former.
Table options are stamped at auto-create only; updating existing tables
from later metadata is future work (a metadata registry, see the native
histograms RFC).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim over-commenting in the remote write metadata path
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: parse the per-table semantic index once per create round
The index was re-parsed from JSON for every table being created — a
first write creating N tables (a fleet's first scrape) paid
O(N x index size). Parsing now happens lazily once per create-planning
round, on both consumers: the operator row-insert auto-create (also
serving OTLP metrics) and the pending-rows batched create.
Also validate every non-zero metadata symbol reference up front, as the
remote write 2.0 spec requires: help_ref was never checked, and
unit_ref escaped checking when the metric type was UNSPECIFIED.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): stamp remote-write v2 unit independently of metric type
OpenMetrics models TYPE and UNIT as independent MetricFamily metadata, and
the Prometheus v2 sender emits UNSPECIFIED-type series that still carry a
unit. The early return on UNSPECIFIED dropped that unit, which is
unrecoverable after table auto-create (units are not stored in rows).
Stamp the mapped unit whenever present; the type and the
metadata_quality=declared upgrade still require an explicit type.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim restating comments in the v2 metadata path
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(frontend): run entity-graph derivation as the caller
The derivation contract requires the computed graph tables to run under
the outer query's identity. Capture the caller's QueryContext when the
computed table is resolved, thread it through EntityGraphProvider, and:
- authorize every contributing source table against the caller via the
new semantic_graph.query permission action, silently excluding denied
sources (entities, edges and source_tables never appear);
- execute the derivation plan under the caller's context so it inherits
permissions, cancellation and deadline instead of a fresh default.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): derive the entity-graph window from the scan's time predicate
Implements the RFC window contract for the computed graph tables:
- table: add extract_time_range_strict, a strict variant of the lenient
time-range extraction that distinguishes an absent observed_at filter
from one that cannot be safely turned into a range;
- operator: replace GraphWindow with GraphQueryWindow, splitting the
queried observed_at range from the source-scan range widened to whole
60s buckets, so boundary buckets aggregate over their full extent;
- frontend: resolve the window from ScanRequest filters — no predicate
keeps the last-hour default, a missing upper bound means now, and a
missing lower bound or unextractable shape is an explicit error, never
a silent fallback.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): system-defined declared-edge table for the entity graph
Reintroduces greptime_private.semantic_relationships_declared with a
canonical, system-owned definition:
- the CREATE TABLE expr (8-tag primary key, business validity columns,
RED fields, 30d TTL); attributes is now a json column so the future
union branch matches the computed table without a per-scan parse;
- created on first use on every write path: SQL INSERT creates it
before executing, and the gRPC row-insert auto-create substitutes the
canonical expr instead of deriving a schema from the request;
- user DDL (CREATE/ALTER/DROP/RENAME/TRUNCATE) and write-path
auto-ALTER are rejected via the new is_ddl_reserved_table guard,
while INSERT/DELETE stay allowed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): union declared edges into semantic_relationships
Adds the declared-edge branch to the relationship derivation
(build_relationships_plan replaces build_calls_plan):
- latest revision per edge key first (mito dedups on primary key plus
observed_at, so a re-asserted edge stores a new revision), then the
business-validity overlap against the queried window; valid_from
defaults to the declaration time and a NULL valid_until means the
edge holds while its row exists;
- the projected observed_at is synthesized inside the queried range
(Inexact pushdown re-applies the scan's filters above the computed
table, which would drop rows keyed by the physical revision time);
window_end/fresh_until of open-ended edges take the window's upper
bound so 'fresh_until >= now() - ...' queries see them;
- tag columns are cast out of dictionary encoding, and the union is
re-projected to the 16-column contract;
- the frontend feeds the branch only when the physical table exists,
the caller may read it, and its schema still matches the canonical
definition (mismatch is an explicit error, not a silent drop).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover declared edges, window contract and caller authorization
- sqlness: system auto-create on first INSERT, latest-revision reads,
open-ended vs retired validity, explicit/lower-only/upper-only window
behavior, user-DDL rejection, rename-into rejection, DELETE cleanup;
- integration: a permission checker denying one trace table excludes it
from both semantic_relationships and semantic_entities.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: allow DROP/TRUNCATE on the declared-edge table and fix CI lints
The definition guard rejected every DDL, which left sqlness (and any
shared deployment) no way to remove the table the semantic_graph case
creates — its extra region then broke unrelated region/partition case
expectations. Narrow the guard to what actually protects the canonical
definition: user CREATE, ALTER, RENAME-into and repartition stay
rejected, while DROP and TRUNCATE are allowed — dropping loses nothing
structural, the next INSERT recreates the table canonically, and DROP
doubles as the recovery path if the canonical definition ever changes.
The sqlness case now verifies drop-then-recreate and cleans up after
itself.
Also: rustfmt for the catalog crate and two typo fixes.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: adapt canonical declared-table create to TriggerReason
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: address review on the declared-edge table lifecycle and revision reads
- gRPC first writes actually work now: the reserved table's creation
went through the generic create_table_inner, which the definition
guard itself rejects; both branches of create_or_alter_tables_on_demand
route it to create_declared_relationships_table instead, and being a
system action it also bypasses the auto_create_table config/hint;
- revision selection is as-of the queried window: revisions recorded
after the window's end, or whose validity starts after it, no longer
outrank (and hide) the revision that was in effect inside it;
- the canonical-schema check validates the whole definition the union
semantics lean on — time index, primary key, engine, append/merge
mode — not just column names and types;
- UNDROP TABLE of the reserved name is rejected like CREATE: it could
resurrect a pre-canonical shape, and the next INSERT recreates the
table anyway.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim over-commenting in the entity-graph code
Comments that restated adjacent code or narrated justification are cut;
the ones stating non-obvious contracts and gotchas stay.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject CREATE VIEW against DDL-reserved table names
A view named greptime_private.semantic_relationships_declared would
squat the reserved name: the first INSERT then skips the canonical
create (an object already exists) and graph reads fail on the schema
mismatch. CREATE VIEW now passes the same definition guard as CREATE
TABLE.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: debug-log authorization exclusions; declared-edge TTL to 90d
Sources the derivation contract silently excludes (per-table denial,
whole-scan denial, the declared-edge table) are invisible from outside;
a debug log at each names what was excluded and why.
The declared-edge table's default TTL becomes 90d, overridable at
creation time via GREPTIMEDB_DECLARED_RELATIONSHIPS_TTL (a proper
configuration option is a TODO).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: rank declared-edge revisions by the visible edge identity
Ranking partitioned by the full primary key, but the projection drops
scope and generation_id: two assertions of the same visible edge under
different generations both ranked first and came out as duplicate,
indistinguishable rows. Rank by the exposed identity (endpoints,
rel_type, provenance) instead, with generation_id/scope as
deterministic tie-breakers for same-timestamp assertions.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: drop redundant declared-edge tests
The generations regression is already asserted by the revision and
as-of tests; the DDL shape test restated the declarative builder
against itself. Its one non-tautological check (attributes maps to the
json type) moves into the schema-matcher test.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject disjunctive graph windows and unmatchable future windows
- OR/IN over observed_at collapse disjoint ranges into their convex
hull; a declared edge's synthesized timestamp can land in a gap and
be dropped by the re-applied filter even though the edge is valid at
a requested instant. The strict extractor now rejects those shapes.
- A lower bound in the future inverts against the implicit up-to-now
upper bound; the declared branch then fabricated an edge observed at
the future bound. Such windows now derive nothing.
- The reserved-table gRPC create path classifies an instant-TTL table
like every sibling path.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(table): add entity semantic declarations
Define open-ended greptime.semantic.entity.* options, validate entity columns at DDL time, and stamp OTLP trace tables with the service entity declaration.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add read-time entity relationships graph
Add computed semantic graph tables, typed DataFusion derivation plans for entity registry and trace calls edges, and streaming read-time execution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: exclude semantic graph tables from table constraints
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(operator): name the plan-builder source groupings
Review feedback: build_registry_plan / build_calls_plan took anonymous
(declarations, DataFrame) tuples while the caller already grouped the same
fields. Introduce RegistrySource { declarations, scan } and CallsSource
{ service, scan } next to the builders and flow them through the frontend
caller and tests. The frontend-side EntitySource keeps holding a TableRef
(the operator builders stay pure over already-built scans), so the named
structs live in operator rather than reusing that type. No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: adjust the position of experimental_enable_prometheus_native_histogram
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* chore: move prom_validation_mode as well
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* chore: gate soft-drop table behind the enterprise feature
Soft-drop table becomes an enterprise-only feature:
- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
registration, and information_schema.recycle_bin registration are
compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
(moving to the enterprise repo); affected information_schema .result
files are regenerated
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: limit unused_variables allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: include the config key in the soft-drop enterprise gate error
Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: limit unused_mut allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: reject soft-drop DDL submissions in non-enterprise builds
Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: stop --enable-gc from enabling soft drop in the sqlness template
Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate fresh soft-drop procedures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: gate soft-drop fallback coverage
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate soft-drop procedure implementation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate drop table soft-drop behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate expired soft-drop gc behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: test enterprise table ddl lifecycle
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark purge_table as enterprise licensed
The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark recycle_bin as enterprise licensed
The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark soft-drop procedure sources as enterprise licensed
The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add a dedicated http api server port
* fix: integration test
* refactor: make http-api-port opt-in
* refactor: rename attribute to http-api-server
* feat: use middleware to check different http server port
* refactor: rename config option
shuiyisong
dennis zhuang
Weny Xu
Ning Sun
Yingwen
discord9
Lei, HUANG
Whis Liao
jeremyhi
yimeng
Yash Agrawal
QuakeWang