Reuse the Inserter auto-create policy before pending-row batch DDL. Keep existing-schema writes working while rejecting disabled table creation and schema evolution.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Share the pending-row batching contract and lifecycle across Prometheus, OTLP HTTP, and OTel Arrow in `src/servers/` and `src/frontend/`.
Add OTLP batching configuration, protocol-aware submit observability, integration coverage in `tests-integration/`, and generated config/dashboard updates.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(servers): compose OTLP metrics job from service.namespace/service.name
The OTel Prometheus compatibility spec defines job as
"<service.namespace>/<service.name>" when the namespace is present.
The OTLP metrics path only used the bare service.name, so the job tag
diverged from target_info produced by Prometheus-side exporters for the
same resource. Compose the namespace form, and keep not fabricating a
job when service.name is absent.
Behavior change: resources carrying service.namespace now get
"namespace/name" as their job tag value.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(otlp): synthesize otel_resource_info at OTLP metrics ingestion
Ordinary OTLP metrics scatter filtered resource attributes as tags over
every logical metric table, so metrics-only services contribute nothing
to the semantic entity graph. Each request now also projects its
distinct resources into one info-metric-shaped mito table,
otel_resource_info: a fixed allowlist of identity-relevant attributes
under their raw OTel keys (independent of the label translation
strategy and the promote/ignore headers) plus derived job/instance
compatibility columns, value 1.0, and the newest data-point timestamp.
The descriptor is written after the main insert is committed; a failure
there (conflicting pre-existing table, auto-create disabled) degrades
to an OTLP partial_success warning with rejected_data_points = 0
instead of failing the request and triggering client retries of
already-accepted data. A request writing a metric named
otel_resource_info suppresses synthesis. Legacy mode is unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): otel info-metric conventions with host/container entities
Whitelist the ingestion-synthesized otel_resource_info descriptor via a
new otel_info_metrics conventions map, gated on source=opentelemetry
(the existing gate hardcoded source=prometheus). Its declarations use
explicit descriptive lists instead of descriptive_rest so identifying
attributes of other entities do not leak into service.instance.
Conventions tightened per the Astronomy Shop findings: host identity is
host.id with host.name descriptive only (host.name is not stable across
SDKs and resource detectors), a generic container entity (new entity
type) is declared only when container.id is present, and trace-v1
tables now synthesize host/container from their flattened resource
attributes too. New co-declared edges: service.instance runs_on
container, container runs_on host.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): cover the resource descriptor in integration tests
Covers the descriptor's raw-key columns and info-metric options through
the HTTP path, the namespace/name job composition end-to-end, column
names being independent of the translation strategy, the allowlist
excluding unlisted resource attributes, auto-create after a drop, the
metric-name collision suppressing synthesis, and the partial-success
warning (rejected_data_points = 0) when a pre-existing incompatible
table fails the descriptor write while metric data is accepted.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: cargo fmt
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(frontend): degrade descriptor permission denial to a warning
A table-level permission policy denying otel_resource_info would have
failed the whole OTLP metrics request because the descriptor's
permission check ran before the main insert. The descriptor is derived
enrichment: check its permission in the degrade path so a denial skips
the write and surfaces as the partial-success warning, like any other
descriptor write failure.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): guard descriptor writes with semantic ownership markers
A pre-existing schema-compatible table named otel_resource_info would
silently receive descriptor rows while its missing semantic stamps kept
it out of the entity graph. The descriptor write now requires the
auto-created table's ownership markers (mito engine + signal_type +
source + metric.type=info + metadata_quality=declared) and otherwise
degrades to the partial-success warning; the entity-graph gate for the
otel whitelist likewise requires metric.type=info, so a user table
stamped with only signal/source no longer picks up implicit
declarations.
Also fold the descriptor write cost into the response and surface the
degrade warning through the otel-arrow BatchStatus status_message.
Integration tests pin the full marker set on auto-create and that an
existing owned descriptor keeps accepting writes without degrading —
a missing marker would otherwise silently stop every descriptor write
after the first request.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(otlp): build descriptor rows without the per-resource BTreeMap
Projecting a resource allocated a BTreeMap and then collected it into the
row key, and every attribute was matched against the allowlist by linear
scan. Collect the tags into a Vec and sort once, and match the allowlist
instead of scanning it. Measured on the conversion path: descriptor work
drops 16-18%, from 10.6% to 8.9% of conversion CPU on the worst shape
(1000 resources with 4 data points each), where the cost tracks resource
count rather than data-point count.
Also trims the comments and tests added with the descriptor to what
carries information.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): pin the descriptor permission-denial degrade path
A policy denying the descriptor table must not fail the metrics request,
which the fix in 2401b3dd9c does but nothing covered. Verified as a
regression guard by mutation: moving the permission check back before
the main write makes this test fail.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): keep legacy mode covered after trimming the unit tests
Trimming the descriptor tests dropped the only assertion that legacy
mode skips the job/instance remap and the promote filter. Both alter
the columns of tables already in use, so fold the check into the legacy
conversion test rather than leaving it uncovered.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(semantic-graph): stop encoding column names into composite entity ids
A composite entity id rendered the identifying columns as sorted
`col=value` pairs, so the same identity split into one entity per
signal: a trace table names its columns service_name and
resource_attributes.service.instance.id where a metric table names them
job and instance. One service instance became two nodes with two
parallel edge sets, breaking the walk from a trace to that instance's
metrics.
Render an id as its values in declared order instead, escaping the
separator so components stay distinguishable, which is what single-column
ids already did by keeping only the value. entity_id_attrs still carries
the structured form.
Values alone are not enough for a namespaced service: the metric side
folds service.namespace into job while traces keep the bare name. Add
qualified_by to the conventions so the trace declarations compose the
namespace the same way, per the OTel rule that job is
<service.namespace>/<service.name> or the bare name when the namespace
is empty. A table without the namespace column keeps the unqualified
identity rather than losing the declaration.
Conventions validation now rejects one entity type declared with a
different number of id columns by two sources, which would silently
produce ids that can never match.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* style(otlp): import the parent module by crate path
check-super-imports.py, part of the CI format gate, rejects a
file-level `use super::`.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(otlp): gate the resource descriptor, and fix what review found
Synthesizing greptime_otel_resource_info creates and writes a table the
user never sent, so it is now off unless
otlp.experimental_enable_resource_info says otherwise. With it off the
request costs exactly what it did before the descriptor existed: nothing
is projected, no table is created, no write and no permission check
happen. Tests run with it on. StandaloneOptions carried no otlp field,
so the whole [otlp] section was silently dropped in standalone mode; map
it through, or the new option (and trace_ingest_chunk_size before it)
would do nothing there.
Renamed from otel_resource_info: the greptime_ prefix marks the table as
engine-managed and makes a collision with a user metric unlikely, which
is what the pre-existing-table ownership check and its per-request
catalog lookup were defending against. Both are gone.
A request may carry data for several graph windows, but the descriptor
folded every data-point time into one row at the newest of them, leaving
the earlier windows with metric rows and no entities. Key the rows by
window as well, and take the times from the data points the encoder
actually writes: it drops exponential histograms, and a resource
carrying nothing else was being described as an entity with no
measurements.
Projecting a resource cloned its attributes once per data point. Nest
the windows under the attributes instead, so they are moved once per
resource, and walk the data-point times through a visitor rather than
collecting a Vec per metric.
Also documents what the two maps key and hold, and lifts the projected
attribute names to constants beside KEY_SERVICE_NAME.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): skip the descriptor's work entirely when it is disabled
The collision scan over the request's output tables ran even with the
feature off. Short-circuit on the option instead, and update the config
snapshot the new [otlp] section changed.
Also drops the doc comment orphaned by the deleted ownership check: it
had attached itself to the trait impl and described a check that no
longer exists.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(semantic-graph): drop the expect and name the service identity
The CASE is built through Case directly rather than the fallible
when().otherwise() builder, so the non-test path no longer carries an
expect (architecture-invariants $4).
service_identity returned two same-typed Options that both call sites
destructured positionally; a named struct makes a swap fail to compile.
Also records that id-column order is part of the identity, where the
option docs and the conventions authors will read it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(semantic-graph): drop comments that narrate the code
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(semantic-graph): cast duration_nano before the trace-table union
Trace tables written before the signed-integer ingest change hold
duration_nano as UInt64 and later ones as Int64. The calls derivation
unions the per-table selects, and the two have no common integer type,
so a deployment holding both shapes could not build the plan. The
cross-table test now spans both.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(semantic-graph): drop the redundant duration_nano casts
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): decide exponential histogram acceptance in one place
The resource descriptor mirrored only the experimental gate, so with both
experimental flags on a resource whose only metric is a delta exponential
histogram was described as an entity with no measurements. The encoder's
whole-metric rules move into exponential_histogram_gate, which both call,
and the descriptor takes its timestamps through exponential_histogram_value
so per-point rejections drop out too.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): pair calls edges across trace tables and derive virtual-node edges
Union the normalized client and server spans of all trace tables before the
join, so a client span pairs with a server span stored in a different table.
A client span with no matching server span becomes an edge to a virtual node
named by span attributes (peer.service / db.name / server.address), with
confidence < 1.0 and attributes.connection_type; a window's real pairs win
over virtual candidates for the same edge key.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): derive same-row co-declared edges from the built-in vocabulary
A table declaring both entity types of a vocabulary pair witnesses the edge
on every row carrying both identities: runs_on / contains / part_of for any
declaring table (provenance 'attribute'), agent uses model / agent invoked
tool only for trace sources (span-structure observations, provenance
'trace').
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): derive parent_agent-calls-agent edges from span structure
Trace tables declaring an agent entity pair each span with its child span
across tables (no span-kind filter), keep pairs whose agent identities
differ, and aggregate RED metrics per window, anchored on the parent span
like the service derivation is anchored on the client.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(frontend): feed co-declared and agent sources into the relationships scan
scan_relationships now passes every declaring table (with its trace-ness)
to the co-declared branch and the trace tables' agent declarations to the
agent-calls derivation. enumerate validates the fixed trace-v1 columns and
derives around a malformed trace table instead of failing the whole scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover cross-table pairing, virtual nodes, co-declared and agent edges
sqlness exercises the new derivations end to end (including a malformed
trace-model table being skipped); the integration authorization test now
also pins that a pair split across tables derives no edge when the caller
cannot read one side.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: update the relationships module doc for the new branches
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: import shared derivation helpers via crate paths
The fmt CI gate rejects module-level 'use super::' imports.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: fold co-declared duplicates, decouple agent calls, verify the trace time index
Review findings: the co-declared branch lacked a cross-source DISTINCT, so
two tables witnessing the same edge in one window emitted duplicate rows;
the agent-calls derivation was gated on a usable service declaration; the
trace schema guard accepted a table whose time index is not the column the
derivations bucket by. The empty-trace-table test asserted a union
invariant with no information and is dropped.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: rename the agent-tool edge to invokes and track current OTel peer attributes
The vocabulary's other relation names are present tense; semconv 1.39/1.26
replaced peer.service and db.name with service.peer.name and db.namespace,
so the virtual-node candidates now check the current names first and keep
the deprecated ones for existing telemetry.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: trust the trace-v1 table option instead of matching the fixed schema
The option is only ever stamped by the ingest path, which guarantees the
fixed span columns; matching column types here couples the graph to every
trace schema evolution (e.g. #8816) for a case that cannot occur.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(frontend): run entity-graph derivation as the caller
The derivation contract requires the computed graph tables to run under
the outer query's identity. Capture the caller's QueryContext when the
computed table is resolved, thread it through EntityGraphProvider, and:
- authorize every contributing source table against the caller via the
new semantic_graph.query permission action, silently excluding denied
sources (entities, edges and source_tables never appear);
- execute the derivation plan under the caller's context so it inherits
permissions, cancellation and deadline instead of a fresh default.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): derive the entity-graph window from the scan's time predicate
Implements the RFC window contract for the computed graph tables:
- table: add extract_time_range_strict, a strict variant of the lenient
time-range extraction that distinguishes an absent observed_at filter
from one that cannot be safely turned into a range;
- operator: replace GraphWindow with GraphQueryWindow, splitting the
queried observed_at range from the source-scan range widened to whole
60s buckets, so boundary buckets aggregate over their full extent;
- frontend: resolve the window from ScanRequest filters — no predicate
keeps the last-hour default, a missing upper bound means now, and a
missing lower bound or unextractable shape is an explicit error, never
a silent fallback.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): system-defined declared-edge table for the entity graph
Reintroduces greptime_private.semantic_relationships_declared with a
canonical, system-owned definition:
- the CREATE TABLE expr (8-tag primary key, business validity columns,
RED fields, 30d TTL); attributes is now a json column so the future
union branch matches the computed table without a per-scan parse;
- created on first use on every write path: SQL INSERT creates it
before executing, and the gRPC row-insert auto-create substitutes the
canonical expr instead of deriving a schema from the request;
- user DDL (CREATE/ALTER/DROP/RENAME/TRUNCATE) and write-path
auto-ALTER are rejected via the new is_ddl_reserved_table guard,
while INSERT/DELETE stay allowed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): union declared edges into semantic_relationships
Adds the declared-edge branch to the relationship derivation
(build_relationships_plan replaces build_calls_plan):
- latest revision per edge key first (mito dedups on primary key plus
observed_at, so a re-asserted edge stores a new revision), then the
business-validity overlap against the queried window; valid_from
defaults to the declaration time and a NULL valid_until means the
edge holds while its row exists;
- the projected observed_at is synthesized inside the queried range
(Inexact pushdown re-applies the scan's filters above the computed
table, which would drop rows keyed by the physical revision time);
window_end/fresh_until of open-ended edges take the window's upper
bound so 'fresh_until >= now() - ...' queries see them;
- tag columns are cast out of dictionary encoding, and the union is
re-projected to the 16-column contract;
- the frontend feeds the branch only when the physical table exists,
the caller may read it, and its schema still matches the canonical
definition (mismatch is an explicit error, not a silent drop).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover declared edges, window contract and caller authorization
- sqlness: system auto-create on first INSERT, latest-revision reads,
open-ended vs retired validity, explicit/lower-only/upper-only window
behavior, user-DDL rejection, rename-into rejection, DELETE cleanup;
- integration: a permission checker denying one trace table excludes it
from both semantic_relationships and semantic_entities.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: allow DROP/TRUNCATE on the declared-edge table and fix CI lints
The definition guard rejected every DDL, which left sqlness (and any
shared deployment) no way to remove the table the semantic_graph case
creates — its extra region then broke unrelated region/partition case
expectations. Narrow the guard to what actually protects the canonical
definition: user CREATE, ALTER, RENAME-into and repartition stay
rejected, while DROP and TRUNCATE are allowed — dropping loses nothing
structural, the next INSERT recreates the table canonically, and DROP
doubles as the recovery path if the canonical definition ever changes.
The sqlness case now verifies drop-then-recreate and cleans up after
itself.
Also: rustfmt for the catalog crate and two typo fixes.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: adapt canonical declared-table create to TriggerReason
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: address review on the declared-edge table lifecycle and revision reads
- gRPC first writes actually work now: the reserved table's creation
went through the generic create_table_inner, which the definition
guard itself rejects; both branches of create_or_alter_tables_on_demand
route it to create_declared_relationships_table instead, and being a
system action it also bypasses the auto_create_table config/hint;
- revision selection is as-of the queried window: revisions recorded
after the window's end, or whose validity starts after it, no longer
outrank (and hide) the revision that was in effect inside it;
- the canonical-schema check validates the whole definition the union
semantics lean on — time index, primary key, engine, append/merge
mode — not just column names and types;
- UNDROP TABLE of the reserved name is rejected like CREATE: it could
resurrect a pre-canonical shape, and the next INSERT recreates the
table anyway.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim over-commenting in the entity-graph code
Comments that restated adjacent code or narrated justification are cut;
the ones stating non-obvious contracts and gotchas stay.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject CREATE VIEW against DDL-reserved table names
A view named greptime_private.semantic_relationships_declared would
squat the reserved name: the first INSERT then skips the canonical
create (an object already exists) and graph reads fail on the schema
mismatch. CREATE VIEW now passes the same definition guard as CREATE
TABLE.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: debug-log authorization exclusions; declared-edge TTL to 90d
Sources the derivation contract silently excludes (per-table denial,
whole-scan denial, the declared-edge table) are invisible from outside;
a debug log at each names what was excluded and why.
The declared-edge table's default TTL becomes 90d, overridable at
creation time via GREPTIMEDB_DECLARED_RELATIONSHIPS_TTL (a proper
configuration option is a TODO).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: rank declared-edge revisions by the visible edge identity
Ranking partitioned by the full primary key, but the projection drops
scope and generation_id: two assertions of the same visible edge under
different generations both ranked first and came out as duplicate,
indistinguishable rows. Rank by the exposed identity (endpoints,
rel_type, provenance) instead, with generation_id/scope as
deterministic tie-breakers for same-timestamp assertions.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: drop redundant declared-edge tests
The generations regression is already asserted by the revision and
as-of tests; the DDL shape test restated the declarative builder
against itself. Its one non-tautological check (attributes maps to the
json type) moves into the schema-matcher test.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject disjunctive graph windows and unmatchable future windows
- OR/IN over observed_at collapse disjoint ranges into their convex
hull; a declared edge's synthesized timestamp can land in a gap and
be dropped by the re-applied filter even though the edge is valid at
a requested instant. The strict extractor now rejects those shapes.
- A lower bound in the future inverts against the implicit up-to-now
upper bound; the declared branch then fabricated an edge observed at
the future bound. Such windows now derive nothing.
- The reserved-table gRPC create path classifies an instant-TTL table
like every sibling path.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: adjust the position of experimental_enable_prometheus_native_histogram
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* chore: move prom_validation_mode as well
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* feat: add close-first soft-drop table flow
Soft-drop now tombstones table metadata and closes datanode regions instead of
issuing physical drop requests, while preserving hard-drop cleanup semantics and
blocking conflicting drops of recreated table names.
Files:
- `src/common/meta/src/ddl.rs`
- `src/common/meta/src/ddl/drop_table.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/ddl_manager.rs`
- `src/meta-srv/src/metasrv/builder.rs`
- `src/cmd/src/standalone.rs`
- `src/common/meta/src/test_util.rs`
- `src/meta-srv/src/procedure/utils.rs`
- `tests-integration/src/standalone.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add undrop and purge table procedures
Add soft-drop recovery and cleanup procedures, wire their DDL task handling, and update \`greptime-proto\` so the new tasks can round-trip through protobuf.
Files:
- \`Cargo.toml\`
- \`Cargo.lock\`
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl_manager.rs\`
- \`src/common/meta/src/rpc/ddl.rs\`
- \`src/common/meta/src/key.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
- \`src/mito2/src/engine/open_test.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: reopen soft-dropped regions before purge
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: disable soft-drop operations for metric logical tables
Prevent soft-dropping, undropping, and purging of metric engine logical tables by explicitly returning unsupported errors. This introduces `is_metric_engine_logical_table` to identify metric logical tables and adds corresponding test cases.
Files:
- `src/common/meta/src/ddl/drop_table/metadata.rs`
- `src/common/meta/src/ddl/purge_dropped_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: sync failure detectors during soft-drop lifecycle
Keep region failure detector state aligned as soft-dropped tables close, reopen, and purge regions so stale detectors do not trigger failover for unavailable or deleted regions.
Files:
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/drop_table.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: simplify soft-drop table tests
Consolidate redundant soft-drop lifecycle assertions into existing end-to-end tests and share dropped-table metadata setup to keep the branch coverage focused.
Files:
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/key.rs`
- `src/mito2/src/engine/open_test.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: reopen follower regions during undrop
Reopen all replicas when restoring dropped physical tables so recovered replicated tables do not leave follower regions closed.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: use dropped table ddl expr protos
Update greptime-proto and adapt dropped table DDL task conversions to the shared expression wrappers required by the proto API.
Files:
- `Cargo.toml`
- `Cargo.lock`
- `src/api/src/helper.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/rpc/ddl.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* revert: use inline dropped table task protos
Point GreptimeDB at the proto revision that restores direct dropped table task fields and remove wrapper-expression conversion code.
Files:
- `Cargo.toml`
- `Cargo.lock`
- `src/api/src/helper.rs`
- `src/common/meta/src/rpc/ddl.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: various soft-drop improvements
- Bump `greptime-proto` dependency revision
(`Cargo.toml`, `Cargo.lock`)
- Pass `region_wal_options` directly without pre-serialization in undrop flow
(`src/common/meta/src/ddl/undrop_table.rs`, `src/common/meta/src/key.rs`)
- Remove unused `RegionNumber` import
(`src/common/meta/src/ddl/utils.rs`)
- Add `reset_failure_detectors` to test mock
(`src/common/meta/src/ddl/tests/drop_table.rs`)
- Add JSON roundtrip tests for `UndropTableTask` and `PurgeDroppedTableTask`
(`src/common/meta/src/rpc/ddl.rs`)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: validate table name match in undrop procedure
Add a guard in `UndropTableProcedure::on_prepare()` to check that the
dropped table name matches the undrop task name, returning `TableNotFound`
on mismatch. This prevents undropping a table by a different name when
only the table ID is known.
- `src/common/meta/src/ddl/undrop_table.rs` — add table-name validation
- `src/common/meta/src/ddl/tests/drop_table.rs` — add test for name mismatch
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: simplify UndropTableTask to use table_id only
Remove catalog, schema, and table name fields from `UndropTableTask`
since the table name can be derived from the dropped table metadata
in the procedure itself. This eliminates redundant fields and the
associated name-validation test.
Simplify locking in `UndropTableProcedure` to only use `TableLock`.
Update `greptime-proto` dependency revision.
- `Cargo.toml`, `Cargo.lock`
- `src/common/meta/src/rpc/ddl.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: detect table name conflict during tombstone restore in undrop
- Added `require_dest_not_exists` parameter to tombstone `move_values` to check destination key existence during restore
- Added `TombstoneTargetAlreadyExists` error variant
- Map tombstone conflict to `TableAlreadyExists` in undrop procedure
- Added test for undrop failing when live name created after prepare
Files:
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/key/tombstone.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: make PurgeDroppedTableTask table_id-only
Streamline the purge-dropped-table flow by requiring a table_id
instead of allowing name-based fallback.
- Refactored `PurgeDroppedTableTask` to hold only `table_id` in `src/common/meta/src/rpc/ddl.rs`
- Simplified purge procedure in `src/common/meta/src/ddl/purge_dropped_table.rs`
- Adapted tests in `src/common/meta/src/ddl/tests/drop_table.rs`
- Bumped `greptime-proto` dependency
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tombstone): chunk values by per-key txn ops instead of fixed divisor
Replaced the fixed `max_txn_ops() / 2` chunk size with operation-aware
constants (`MOVE_VALUE_TXN_OPS_PER_KEY=4`, `RESTORE_VALUE_TXN_OPS_PER_KEY=6`)
to correctly account for per-key transaction operations. Added
`TxnOpLimitKvBackend` test helper and two new tests
(`test_restore_chunks_by_total_txn_ops_limit`,
`test_create_chunks_by_total_txn_ops_limit`) verifying chunking under
tight txn op limits.
Affected file:
- `src/common/meta/src/key/tombstone.rs` — chunk size fix,
`TxnOpLimitKvBackend` helper, two new tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(soft-drop): deregister failure detectors and handle replayed open-regions during purge
- `src/common/meta/src/ddl/drop_table.rs`: deregister failure detectors before
transitioning to DeleteTombstone state
- `src/common/meta/src/ddl/undrop_table.rs`: refactor `open_regions` into
`open_regions_inner` with an `ignore_region_not_found` flag; expose
`open_regions_ignore_region_not_found` for purge replayer
- `src/common/meta/src/ddl/purge_dropped_table.rs`: use
`open_regions_ignore_region_not_found` in replayed purge procedures
- `src/common/meta/src/ddl/tests/drop_table.rs`: add tests for undrop
idempotency and purge replay tolerance of dropped regions
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: fix clippy
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(soft-drop): open regions before restoring undrop metadata
Restore undropped table metadata only after physical regions have been reopened, keeping the table hidden while regions are still closed. Preserve the live-name conflict check before opening regions and cover the ordering with a regression test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tombstone): fail fast on invalid txn op budget
Fail before issuing `TombstoneManager` transactions when the configured `max_txn_ops` cannot fit one key. Add coverage for undersized restore budgets in `src/common/meta/src/key/tombstone.rs`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to main branch commit
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(json2): type hint
* test(datatypes): add JsonSettings serde round-trip test
* minor refactor
This reverts commit 7ff5a5249a09be5396536284fe822b5761ef4e6a.
* fix: code review
* feat: table semantic layer identity (Phase 1)
Attach a thin layer of semantic metadata to ingested tables via the existing
`table_options` slot, so machine consumers (LLM agents, alert/dashboard builders,
MCP servers, ETL) can align a table with the observability concept it stands for
without guessing from column names. See docs/rfcs/2026-05-28-table-semantic-layer.md.
Phase 1 (identity) only:
- New `table::requests::semantic` module: the `greptime.semantic.*` vocabulary
(signal/source/source_version/pipeline + trace/metric/log/resource-scope keys,
defined now, populated by later phases), value constants, the internal
`greptime.internal.semantic.per_table_index` transport key (reserved for Phase 2,
deliberately outside the public namespace), and `is_semantic_option_key`.
- `validate_table_option` accepts the `greptime.semantic.*` prefix, so the keys are
valid both on the auto-create path and on explicit `CREATE TABLE ... WITH (...)`.
- `fill_table_options_for_create` copies every semantic ctx extension into the new
table's options (prefix passthrough alongside the fixed allowlist).
- Frontend stamps identity on the context at each ingest entry: OTLP metrics
(metric/opentelemetry), traces (+pipeline, has_events/has_links/conventions for
the v1 model), logs (log/opentelemetry), and Prometheus remote write
(metric/prometheus, metadata_quality=inferred). OTLP metric metadata_quality is
left for Phase 2 (declared).
- Trace identity is stamped only on the main span table; the derived
`_services` / `_operations` lookup tables keep the unstamped context and carry no
semantic identity (cross-table relationships are out of scope).
Semantic options appear in SHOW CREATE TABLE (like table_data_model /
otlp_metric_compat) and in information_schema, so an LLM inspecting a table sees its
semantics directly.
Tests: unit (validation prefix + internal-key rejection, ctx passthrough) and
integration assertions that the common keys land for OTLP metrics (metric-engine
logical table), traces, logs, and Prometheus remote write; SHOW CREATE goldens
updated.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: prom batcher not cover and white list for semantic keys/values
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: typo
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: global switch for creating table automatically
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: make auto_create_table as comment by default
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: respect gloabl switch for metric engine
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: provide query information for post_execute interceptor
* test: update for tests-integration
* feat: make interceptor available to prometheus serialization
* feat: revert post_execute change
* feat: add expr to pre_execute and remove serialization interceptor
* chore: lint
* feat: expose node info for placement selectors
Return `NodeInfo` from `PeerDiscovery` methods and keep OSS selectors mapping back to `Peer`.
Carry `__greptime_origin_frontend.addr` from frontend create-table DDLs into selector `extensions`, and thread `PeerAllocContext` through table-route allocation.
Persist datanode `NodeInfo` when heartbeat stats are absent so collected env vars remain available after restart.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: skip datanode node info without stats
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: avoid unnecessary workload clones
Skip workload cloning for inactive nodes and for active node-info lookups without workload filters.
Files: `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: require frontend origin address
Require `StatementExecutor` to carry a concrete frontend origin address and always attach it to meta DDL query contexts.
Files: `src/operator/src/statement.rs`, `src/operator/src/statement/ddl.rs`, `src/operator/src/utils.rs`, `src/frontend/src/instance/builder.rs`, `src/frontend/src/heartbeat.rs`, `src/flow/src/server.rs`, `src/cmd/src/standalone.rs`, `src/cmd/src/flownode.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: reuse resolved frontend address
Resolve the frontend peer address once in the frontend builder, store it on the instance, and reuse it for heartbeat and flow invoker origins.
Files: `src/frontend/src/instance/builder.rs`, `src/frontend/src/instance.rs`, `src/frontend/src/heartbeat.rs`, `src/cmd/src/frontend.rs`, `src/cmd/src/standalone.rs`, `src/frontend/src/frontend.rs`, `src/frontend/src/heartbeat/tests.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: preserve datanode lease liveness
Filter active datanode node infos through lease timestamps and workloads while preserving node info fields such as reported env vars.
Files: `src/meta-srv/src/discovery/utils.rs`, `src/meta-srv/src/discovery/lease.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Remove stale datanode lease helper
- `discovery`: remove the obsolete `alive_datanodes` helper and related tests in `src/meta-srv/src/discovery/utils.rs` and `src/meta-srv/src/discovery/lease.rs`
- `integration`: update cluster and standalone setup paths in `tests-integration/src/cluster.rs` and `tests-integration/src/standalone.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/env-based-region-selector-oss: simplify lease discovery
- `lease-discovery`: simplify logic and remove unused utilities in `src/meta-srv/src/discovery/lease.rs` and `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Lei, HUANG
XuanwoandWenyXu
shuiyisong
discord9
LFC
dennis zhuang
Lei, HUANG
Yingwen
jeremyhi
QuakeWang
Ruihang Xia
Ning Sun