* feat: embed the derivation conventions as data and adopt gen_ai entity naming
Move the co-declared edge vocabulary, the agent-edge vocabulary and the
virtual-destination candidates from Rust consts into an embedded
conventions.yaml (include_str!), parsed once behind a LazyLock and
validated against the entity-type grammar and the closed rel_type set; a
broken file propagates as a plan error instead of panicking. The agent
vocabulary entity types follow the GenAI semantic-convention namespace
as written: gen_ai.agent / gen_ai.model / gen_ai.tool.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: drop the tag requirement for entity identity columns
Entity declarations no longer require id columns to be tag/primary-key
columns; only column existence is validated. Trace pipelines flatten the
identifying attributes (span_attributes.gen_ai.agent.id, ...) into field
columns, so the tag rule locked real trace tables out of declaring
entities while buying no correctness — the read-time derivation works on
any column.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: implicit declarations for well-known prometheus info metrics
Tables stamped signal_type=metric + source=prometheus whose name matches
the conventions.yaml whitelist (kube_pod_info, kube_node_info,
kube_pod_owner, target_info) get implicit entity declarations: k8s.pod /
k8s.node / k8s.workload with name-based identity and target_info's
service / service.instance with the remaining tags as the descriptive
snapshot. The existing co-declared vocabulary then derives runs_on and
part_of from the same rows, so no new edge branch is needed. Explicit
declarations of a type always suppress the implicit one, and the metric
engine's physical table is excluded (it aggregates every logical
table's columns).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover the prometheus conventions in sqlness and compact the graph cases
Add the whitelisted-info-metric scenario (kube_pod_info, kube_pod_owner,
target_info deriving runs_on / part_of, a non-whitelisted metric
contributing nothing), fold the single-table calls, cross-table pairing
and virtual-node cases into one trace scenario (they exercise the same
union-before-join path), merge the two declaring-metric-table cases, and
reuse one rename probe for both reserved names.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject entity id columns without a stable string form
Review follow-ups: the DDL check now validates against the schema and
rejects binary-backed and nested types for identity columns (the
derivation renders ids via CAST to Utf8, so the failure used to surface
only when the graph was scanned); the agent sqlness case keeps its
identity columns as fields to cover the relaxed tag rule end to end;
stale tag-rule comments and a dangling const reference are cleaned up.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: type-check every entity column role, not only ids
The registry renders scope and descriptive values through the same
CAST-to-string path as ids, so a binary-backed column in any role fails
at scan time; the DDL check is now role-independent (and simpler).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: name the code-anchored vocabulary constants
Entity types and edge attributes the derivation code itself anchors on
(service, gen_ai.agent, calls, trace/attribute provenance) become
constants in the conventions module; the rest of the vocabulary stays
YAML-only data. ImplicitEntity is renamed PromImplicitEntity, and the
implicit-declaration path logs each skip of a whitelisted info metric
(wrong stamps, suppressed by an explicit declaration, missing id
column) so a missing graph entity is diagnosable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: single-source the graph constants
The graph tables' column names move to common-catalog (the schemas
catalog exposes and the plans operator builds must match column by
column), and the conventions module now carries the complete built-in
vocabulary — entity types, rel_types, provenances and connection types —
with the embedded YAML validated by membership against it, so an edit
drifting outside the vocabulary fails the conventions test instead of
deriving nothing.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: treat empty identity components as absent
kube-state-metrics emits empty-string labels an entity id must not be
built from: an unscheduled pod's node and an owner-less pod's owner_kind
/ owner_name. Standard Prometheus drops empty labels (they arrive as
NULL and the existing predicate handles them), but other remote-write
agents may keep them, which produced ghost entities with empty ids and
false runs_on / part_of edges. Every identity predicate (registry,
co-declared edges, span endpoints) now requires non-NULL and non-empty
components through one shared helper.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: tighten the conventions DSL semantics
Rename the co-declaration rule lists to what they are (co_declared_edges
/ trace_co_declared_edges — derivation rules, not a relation
vocabulary), stop overstating the GenAI entity types (Greptime types
derived from GenAI attributes; OTel defines no model/tool entities),
move target_info's descriptive snapshot to service.instance (the
remaining labels are the target's resource attributes, and instances
would write conflicting snapshots onto the logical service), and extend
the descriptor whitelist with the stable KSM sources: container info
metrics (closing the k8s.pod contains k8s.container rule),
kube_service_info (new k8s.service entity type) and the fuller
descriptive label sets.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: guard entity column types on ALTER as well
ALTER MODIFY COLUMN could change a declared entity column to a type
without a stable string form, deferring the failure to graph scan time;
verify_alter now checks the post-alter schema. Dropping a declared
column stays allowed — the read-time derivation skips the stale
declaration, and semantic options cannot be altered off yet.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: bridge traces and kube-state-metrics on the pod UID
Trace-v1 tables now get implicit declarations from their flattened
resource attributes (otlp_trace_entities in conventions.yaml): the
service identity — replacing the hardcoded fallback — plus
service.instance and k8s.pod, each applied only when its columns exist.
A new co-declared rule derives service.instance runs_on k8s.pod, and
the whitelisted kube-state-metrics pod identity switches from
namespace+pod names to the UID, so the trace-side pod and every KSM
descriptor land on one entity while names stay descriptive. This also
removes pod identity from the multi-cluster same-name collision.
The conventions rejection tests were passing for the wrong reason (a
half-renamed fixture key failed deserialization before reaching any
validation rule); they now assert the specific error each case targets.
Sqlness covers the UID merge across descriptor tables, pod-contains-
container, the k8s.service node, and the empty-uid/empty-node rows
deriving nothing.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover the OTLP-to-graph chain end to end
One real OTLP export must come out of semantic_relationships as the
zero-configuration chain: service calls service, instance part_of
service, instance runs_on pod (bridged by k8s.pod.uid). Resources
without service.instance.id or k8s.pod.uid derive nothing extra.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: identify k8s.service by UID
Same reasoning as pods: a recreated same-name service must not merge
into the old entity and same-named services across clusters must not
collide; kube_service_info carries a stable uid and nothing joins on the
service's name. Also drop a stale tag-rule mention from the option
validation docs.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: cut duplicated test coverage and redundant comments
The trace service-fallback test collapsed into the resource-entities
test (same synthesis path since the fallback moved to YAML; only the
invalid-explicit-no-fallback case was distinct), role-duplicate and
subsumed DDL cases are gone, the embedded-conventions test is just the
parse (its assertions were decorative), and the YAML section comments no
longer restate the struct docs.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(frontend): run entity-graph derivation as the caller
The derivation contract requires the computed graph tables to run under
the outer query's identity. Capture the caller's QueryContext when the
computed table is resolved, thread it through EntityGraphProvider, and:
- authorize every contributing source table against the caller via the
new semantic_graph.query permission action, silently excluding denied
sources (entities, edges and source_tables never appear);
- execute the derivation plan under the caller's context so it inherits
permissions, cancellation and deadline instead of a fresh default.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): derive the entity-graph window from the scan's time predicate
Implements the RFC window contract for the computed graph tables:
- table: add extract_time_range_strict, a strict variant of the lenient
time-range extraction that distinguishes an absent observed_at filter
from one that cannot be safely turned into a range;
- operator: replace GraphWindow with GraphQueryWindow, splitting the
queried observed_at range from the source-scan range widened to whole
60s buckets, so boundary buckets aggregate over their full extent;
- frontend: resolve the window from ScanRequest filters — no predicate
keeps the last-hour default, a missing upper bound means now, and a
missing lower bound or unextractable shape is an explicit error, never
a silent fallback.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): system-defined declared-edge table for the entity graph
Reintroduces greptime_private.semantic_relationships_declared with a
canonical, system-owned definition:
- the CREATE TABLE expr (8-tag primary key, business validity columns,
RED fields, 30d TTL); attributes is now a json column so the future
union branch matches the computed table without a per-scan parse;
- created on first use on every write path: SQL INSERT creates it
before executing, and the gRPC row-insert auto-create substitutes the
canonical expr instead of deriving a schema from the request;
- user DDL (CREATE/ALTER/DROP/RENAME/TRUNCATE) and write-path
auto-ALTER are rejected via the new is_ddl_reserved_table guard,
while INSERT/DELETE stay allowed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): union declared edges into semantic_relationships
Adds the declared-edge branch to the relationship derivation
(build_relationships_plan replaces build_calls_plan):
- latest revision per edge key first (mito dedups on primary key plus
observed_at, so a re-asserted edge stores a new revision), then the
business-validity overlap against the queried window; valid_from
defaults to the declaration time and a NULL valid_until means the
edge holds while its row exists;
- the projected observed_at is synthesized inside the queried range
(Inexact pushdown re-applies the scan's filters above the computed
table, which would drop rows keyed by the physical revision time);
window_end/fresh_until of open-ended edges take the window's upper
bound so 'fresh_until >= now() - ...' queries see them;
- tag columns are cast out of dictionary encoding, and the union is
re-projected to the 16-column contract;
- the frontend feeds the branch only when the physical table exists,
the caller may read it, and its schema still matches the canonical
definition (mismatch is an explicit error, not a silent drop).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover declared edges, window contract and caller authorization
- sqlness: system auto-create on first INSERT, latest-revision reads,
open-ended vs retired validity, explicit/lower-only/upper-only window
behavior, user-DDL rejection, rename-into rejection, DELETE cleanup;
- integration: a permission checker denying one trace table excludes it
from both semantic_relationships and semantic_entities.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: allow DROP/TRUNCATE on the declared-edge table and fix CI lints
The definition guard rejected every DDL, which left sqlness (and any
shared deployment) no way to remove the table the semantic_graph case
creates — its extra region then broke unrelated region/partition case
expectations. Narrow the guard to what actually protects the canonical
definition: user CREATE, ALTER, RENAME-into and repartition stay
rejected, while DROP and TRUNCATE are allowed — dropping loses nothing
structural, the next INSERT recreates the table canonically, and DROP
doubles as the recovery path if the canonical definition ever changes.
The sqlness case now verifies drop-then-recreate and cleans up after
itself.
Also: rustfmt for the catalog crate and two typo fixes.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: adapt canonical declared-table create to TriggerReason
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: address review on the declared-edge table lifecycle and revision reads
- gRPC first writes actually work now: the reserved table's creation
went through the generic create_table_inner, which the definition
guard itself rejects; both branches of create_or_alter_tables_on_demand
route it to create_declared_relationships_table instead, and being a
system action it also bypasses the auto_create_table config/hint;
- revision selection is as-of the queried window: revisions recorded
after the window's end, or whose validity starts after it, no longer
outrank (and hide) the revision that was in effect inside it;
- the canonical-schema check validates the whole definition the union
semantics lean on — time index, primary key, engine, append/merge
mode — not just column names and types;
- UNDROP TABLE of the reserved name is rejected like CREATE: it could
resurrect a pre-canonical shape, and the next INSERT recreates the
table anyway.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim over-commenting in the entity-graph code
Comments that restated adjacent code or narrated justification are cut;
the ones stating non-obvious contracts and gotchas stay.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject CREATE VIEW against DDL-reserved table names
A view named greptime_private.semantic_relationships_declared would
squat the reserved name: the first INSERT then skips the canonical
create (an object already exists) and graph reads fail on the schema
mismatch. CREATE VIEW now passes the same definition guard as CREATE
TABLE.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: debug-log authorization exclusions; declared-edge TTL to 90d
Sources the derivation contract silently excludes (per-table denial,
whole-scan denial, the declared-edge table) are invisible from outside;
a debug log at each names what was excluded and why.
The declared-edge table's default TTL becomes 90d, overridable at
creation time via GREPTIMEDB_DECLARED_RELATIONSHIPS_TTL (a proper
configuration option is a TODO).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: rank declared-edge revisions by the visible edge identity
Ranking partitioned by the full primary key, but the projection drops
scope and generation_id: two assertions of the same visible edge under
different generations both ranked first and came out as duplicate,
indistinguishable rows. Rank by the exposed identity (endpoints,
rel_type, provenance) instead, with generation_id/scope as
deterministic tie-breakers for same-timestamp assertions.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: drop redundant declared-edge tests
The generations regression is already asserted by the revision and
as-of tests; the DDL shape test restated the declarative builder
against itself. Its one non-tautological check (attributes maps to the
json type) moves into the schema-matcher test.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject disjunctive graph windows and unmatchable future windows
- OR/IN over observed_at collapse disjoint ranges into their convex
hull; a declared edge's synthesized timestamp can land in a gap and
be dropped by the re-applied filter even though the edge is valid at
a requested instant. The strict extractor now rejects those shapes.
- A lower bound in the future inverts against the implicit up-to-now
upper bound; the declared branch then fabricated an edge observed at
the future bound. Such windows now derive nothing.
- The reserved-table gRPC create path classifies an instant-TTL table
like every sibling path.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(table): add entity semantic declarations
Define open-ended greptime.semantic.entity.* options, validate entity columns at DDL time, and stamp OTLP trace tables with the service entity declaration.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add read-time entity relationships graph
Add computed semantic graph tables, typed DataFusion derivation plans for entity registry and trace calls edges, and streaming read-time execution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: exclude semantic graph tables from table constraints
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(operator): name the plan-builder source groupings
Review feedback: build_registry_plan / build_calls_plan took anonymous
(declarations, DataFrame) tuples while the caller already grouped the same
fields. Introduce RegistrySource { declarations, scan } and CallsSource
{ service, scan } next to the builders and flow them through the frontend
caller and tests. The frontend-side EntitySource keeps holding a TableRef
(the operator builders stay pure over already-built scans), so the named
structs live in operator rather than reusing that type. No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: support ALTER TABLE SET auto_flush_interval
Closes#8394.
Add a new SetRegionOption::AutoFlushInterval variant so that the
per-table auto flush interval can be changed on existing tables
via 'ALTER TABLE t SET ...', following up the CREATE TABLE path
from #8357.
- region_request.rs: parse 'auto_flush_interval' with humantime and
map it to the new variant.
- metadata.rs: persist the value (or remove it, if None) in
TableOptions.extra_options using the same humantime string format
the engine already expects.
- handle_alter.rs: apply the new interval in
handle_alter_region_options_fast (no memtable flush needed, same
pattern as Ttl) and group the variant with Ttl/Twsc in
new_region_options_on_empty_memtable.
Tests:
- Two unit tests in metadata.rs covering set and unset-to-None.
- A new sqlness case alter_auto_flush_interval.sql covering
create-then-alter, alter-then-alter, invalid duration, and
alter on a table that already had auto_flush_interval at create
time.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: validate auto_flush_interval > 0 in ALTER SET path
Gemini code assist flagged that the request parser accepted a zero
duration, leaving the rejection to the downstream RegionOptions
validation which only fires on next flush. Reject it at parse time
so users get the error immediately at the ALTER TABLE statement.
Also add a '0s' error case to the sqlness test.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: handle SET 'auto_flush_interval' = NULL and add checked-in .result
Address the rest of fengjiachun's review on #8403:
1. Empty value in ALTER SET clears the override (parallels Ttl).
'ALTER TABLE t SET ... = NULL' comes through as value = ''; we
now return AutoFlushInterval(None) so the override is removed
from TableOptions.extra_options, matching the Ttl pattern.
2. Add a unit test in region_request.rs covering the four cases
(valid, empty-clears, zero-rejected, garbage-rejected).
3. Generate and check in alter_auto_flush_interval.result via
'cargo sqlness-runner bare -t alter_auto_flush_interval'. Both
the standalone and distributed sqlness jobs now pass locally,
and the test extension covers the NULL-clears path end to end.
Signed-off-by: srivtx <crypticcc101@gmail.com>
---------
Signed-off-by: srivtx <crypticcc101@gmail.com>
* feat: report region read load in heartbeat
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: expose region query stats in information schema
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update sqlness result
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record region query stats on stream drop
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: keep region query cpu stats in nanoseconds
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(metric-engine): report query load under physical region id
Propagate the physical region ID through the scanner, record batch
stream, and query engine so that query-load metrics (CPU time, scanned
bytes) are attributed to the correct physical region rather than always
to the logical region.
- `src/store-api/src/region_engine.rs` — add `query_load_region_id` to
`ScannerProperties` and `set_query_load_region_id` to `RegionScanner`
trait
- `src/mito2/src/read/seq_scan.rs`,
`src/mito2/src/read/series_scan.rs`,
`src/mito2/src/read/unordered_scan.rs` — implement the new trait
method on each scanner
- `src/common/recordbatch/src/adapter.rs` — carry region id through
`RecordBatchStreamAdapter` into `RecordBatchMetrics`
- `src/table/src/table/scan.rs` — expose region id on `RegionScanExec`
- `src/query/src/datafusion.rs` — extract region id from the physical
plan and set it on the output stream
- `src/query/src/dist_plan/merge_scan.rs` — use metrics-contained region
id in query-load reporting, falling back to the logical region id
- `src/metric-engine/src/engine/read.rs` — set region id on the metric
engine scanner
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(query): satisfy clippy for query load region id
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(query): ignore missing query load region ids
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: track scan output bytes and use them for read costing
Track the actual byte count of each record batch produced by
`RegionScanExec` and use it in place of the aggregated per-plan
`memory_usage` as the `table_scan` cost input. This avoids double
counting bytes that flow through multiple operators.
A named constant `REGION_SCAN_EXEC_NAME` exposes the plan node name
so downstream metric parsers remain correct if the struct is renamed.
Affected files:
- `src/table/src/table/metrics.rs` -- add `output_bytes` counter
- `src/table/src/table/scan.rs` -- record bytes, export name constant
- `src/query/src/dist_plan/merge_scan.rs` -- consume scan bytes
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove misleading `mem_used` gauge from scan metrics
The `mem_used` gauge was used with `add()` on every output batch, but a
RegionScan does not hold memory across polls — each batch is yielded
immediately to the upstream operator. The gauge semantics were
incorrect (cumulative `add()` on a gauge) and the value duplicated
`output_bytes` anyway.
Affected files:
- `src/table/src/table/metrics.rs` — drop `mem_used` field and methods
- `src/table/src/table/scan.rs` — remove `record_mem_usage` call
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: use stable plan names for scan byte metrics
Expose `plan_name` alongside rendered plan text so scan byte extraction no longer depends on EXPLAIN formatting. Keep old serialized metrics compatible by defaulting missing `plan_name`.
Affected files:
- `src/common/recordbatch/src/adapter.rs` -- add `plan_name` to `PlanMetrics` and populate it from `ExecutionPlan::name`
- `src/query/src/dist_plan/merge_scan.rs` -- match `REGION_SCAN_EXEC_NAME` through `plan_name`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: preserve merge scan CPU read cost
Record read cost whenever stream metrics are available, default scan bytes to zero, and aggregate scan output bytes across region scan nodes.
Files: `src/query/src/dist_plan/merge_scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: correct doc comment in `StreamMetrics`
Fix doc comment on `StreamMetrics::new()` to reference the correct
struct name (`StreamMetrics`) instead of the old `MemoryUsageMetrics`.
- `src/table/src/table/metrics.rs` — fix struct name in doc comment
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: table semantic layer per-table enrichment (Phase 2)
Phase 2 of the table semantic layer, plus a vocabulary trim so the layer only
records what a machine consumer cannot cheaply recover on its own.
Per-table metric enrichment (OTLP), via an internal per-table channel:
- A `SemanticIndex` accumulator records, per emitted table, the declared metric
keys: type / unit / temporality / metadata_quality=declared / original_name.
Conflicting single-valued keys collapse to `mixed`/`unknown`.
- Recording happens at the `encode_metrics` level where the base name, metric
type, and proto fields are all in scope, so histogram/summary fan-out gets the
correct per-subtable type (`_bucket`=histogram, `_sum`/`_count`=counter)
without threading state through every encoder.
- The index is serialized onto the `greptime.internal.semantic.per_table_index`
context extension; `apply_per_table_semantic_options` folds each table's keys
into its options at auto-create time.
- `trace.conventions` is refined from the request's resource/scope `schema_url`s
(concrete when uniform, else `mixed`/`unknown`).
Vocabulary trimmed to only meaningful keys. Kept: signal_type, source, pipeline,
trace.conventions, metric.{type,unit,temporality,metadata_quality,original_name}.
Dropped: metric.monotonic (a function of type), trace.has_events/has_links
(constant + derivable from columns), log.severity_scheme/body_format (constant /
derivable, and body_format cost an O(rows) scan), resource/scope lineage
(restates columns / collector-config concern), source_version (no cheap
non-constant value today). Prometheus carries type/unit in the metric name by
convention, so it gets identity only — no inferred enrichment.
Identity (signal_type + source) extended to the remaining ingest protocols so
the discovery view is complete: InfluxDB and OpenTSDB (metric), Loki and
Elasticsearch (log). These protocols carry no type/unit metadata, so identity is
all that applies.
Tests: unit coverage for the accumulator, per-metric-type fan-out, and trace
conventions; integration goldens updated for the OTLP metric/trace SHOW CREATE
output and the new Loki identity.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: validate the option value
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: table semantic layer identity (Phase 1)
Attach a thin layer of semantic metadata to ingested tables via the existing
`table_options` slot, so machine consumers (LLM agents, alert/dashboard builders,
MCP servers, ETL) can align a table with the observability concept it stands for
without guessing from column names. See docs/rfcs/2026-05-28-table-semantic-layer.md.
Phase 1 (identity) only:
- New `table::requests::semantic` module: the `greptime.semantic.*` vocabulary
(signal/source/source_version/pipeline + trace/metric/log/resource-scope keys,
defined now, populated by later phases), value constants, the internal
`greptime.internal.semantic.per_table_index` transport key (reserved for Phase 2,
deliberately outside the public namespace), and `is_semantic_option_key`.
- `validate_table_option` accepts the `greptime.semantic.*` prefix, so the keys are
valid both on the auto-create path and on explicit `CREATE TABLE ... WITH (...)`.
- `fill_table_options_for_create` copies every semantic ctx extension into the new
table's options (prefix passthrough alongside the fixed allowlist).
- Frontend stamps identity on the context at each ingest entry: OTLP metrics
(metric/opentelemetry), traces (+pipeline, has_events/has_links/conventions for
the v1 model), logs (log/opentelemetry), and Prometheus remote write
(metric/prometheus, metadata_quality=inferred). OTLP metric metadata_quality is
left for Phase 2 (declared).
- Trace identity is stamped only on the main span table; the derived
`_services` / `_operations` lookup tables keep the unstamped context and carry no
semantic identity (cross-table relationships are out of scope).
Semantic options appear in SHOW CREATE TABLE (like table_data_model /
otlp_metric_compat) and in information_schema, so an LLM inspecting a table sees its
semantics directly.
Tests: unit (validation prefix + internal-key rejection, ctx passthrough) and
integration assertions that the common keys land for OTLP metrics (metric-engine
logical table), traces, logs, and Prometheus remote write; SHOW CREATE goldens
updated.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: prom batcher not cover and white list for semantic keys/values
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: typo
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: switch partition tree to bulk
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: keep partition tree memtable for migration test
Restore PartitionTreeMemtable construction when memtable.type=partition_tree
is explicit, and move the sparse-encoding bulk override into the default
(no explicit memtable.type) arm so phase 2's memtable.type=bulk wins on
reopen. Rewrite test_reopen_time_series_sparse_memtable_with_bulk to use a
metric-engine-shaped schema and sparse-encoded rows with WriteHint::Sparse,
so the test actually exercises a PartitionTreeMemtable in phase 1 and
verifies WAL replay into the new BulkMemtable on reopen without flushing.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: drop partition tree memtable from runtime
Re-apply the unconditional sparse-encoding override in
`MemtableBuilderProvider::builder_for_options` and route the
`MemtableOptions::PartitionTree` arm to `BulkMemtable` with a deprecation
warning. After this change, `PartitionTreeMemtableBuilder` is no longer
reachable from the engine runtime; benchmarks still reference the type.
Remove `test_reopen_time_series_sparse_memtable_with_bulk` and the
`put_sparse_rows` helper added in the previous commit — that test only
existed to validate the PartitionTree -> Bulk reopen migration and is
unnecessary now that the override is in place.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): move timestamp_array_to_i64_slice into read module
Relocate the timestamp_array_to_i64_slice helper from
memtable/partition_tree/data.rs to the read module so that the read
path no longer depends on the partition_tree internals. All call sites
(both inside and outside the partition_tree module) now import from
crate::read.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): use TimeSeriesMemtableBuilder in time_partition tests
The time_partition tests use the memtable builder purely as a generic
backend for the TimePartitions write/scan paths; nothing in them is
specific to the partition-tree memtable. Switch the seven affected
tests to TimeSeriesMemtableBuilder so the tests no longer depend on
PartitionTreeMemtableBuilder.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): delete PartitionTreeMemtable implementation
The runtime already falls back to BulkMemtable for the PartitionTree
variant. Drop the now-unreachable implementation, its metrics, the
partition_tree benchmarks, the metric-engine Unsupported fallback in
bulk_insert.rs, and the test helpers that only existed for the deleted
module.
MemtableOptions::PartitionTree, its parsing, the runtime fallback, the
store-api MEMTABLE_PARTITION_TREE_* constants, and the SQL fixtures
remain so existing region options keep round-tripping.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito-codec): drop skip_partition_column parameter
PartitionTreeMemtable was the only caller passing
skip_partition_column=true; every other caller passes false. Now that
the partition_tree module is gone, the parameter is uniformly false
and the guard branch is dead. Drop the parameter from the trait method
and both impls, remove the guard and the is_partition_column helper,
and update the four remaining call sites in mito2 plus the bench.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): remove unused MemtableConfig enum
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: fmt code
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: remove unused variant
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update test_config_api
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: remove unused memtable test helpers
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: address review comment
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: support bulk memtable options
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: sanitize config
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: remove partition tree options from region options
Move primary_key_encoding to the top level
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: make ssts test datetime replaced text stable
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update sqlness result
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: validate_enum_options consider bulk memtable
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: pass region id when parsing region options
Replace the `TryFrom<&HashMap>` impl for `RegionOptions` with
`try_from_options(region_id, options_map)` so the legacy partition_tree
fallback can log the affected region. The fallback now also overrides
the SST format to flat in addition to clearing the memtable type.
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: align sst_format with bulk memtable on parse and open
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: remove the `RawTableMeta` and `RawTableInfo` to make codes more concise
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
* feat: impl vector index query
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: remove VectorSearchRule and merge it into scan hint rule
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: vector search hint
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: join and subquery
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: clippy when feature disabled
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: push hint only when column is non-nullable or an explicit IS NOT NULL filter exists
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: transformed = true
Co-authored-by: Yingwen <realevenyag@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: remove adpater vector hint
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: revert transformed
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
dennis zhuang
jeremyhi
Lei, HUANG
fys
Yingwen
Ruihang Xia
sribatsha dash
Weny Xu
shuiyisong
discord9
Ning Sun
discord9
Sicong Hu