789 Commits

Include renames

Author	SHA1	Message	Date
dennis zhuang	02278d301b	feat: derive k8s.node from OTLP resource attributes (#9002) ... The k8s.pod runs_on k8s.node rule can only fire from kube_pod_info: it is the only table declaring both endpoints, and co-declared edges require both on the same row. A deployment sending only OTLP has no kube-state-metrics tables, so its node layer is invisible and that rule has no source at all, even though the resource attributes carry k8s.node.name. Declare k8s.node in otlp_trace_entities and in the synthesized resource descriptor, and project k8s.node.name in the descriptor writer so the column the declaration needs exists. Identity is the name rather than k8s.node.uid: kube-state-metrics carries no node UID, so the name is the only identity both sources can agree on. Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-09-02 05:12:30 +00:00
dennis zhuang	109257505e	feat: report what the graph derives and fix two duplicate-node bugs (#8936) ... * feat(semantic-graph): let the generic container yield to k8s.container A container inside a Kubernetes pod reaches the graph twice: as the k8s.container entity kube-state-metrics describes, identified by [pod uid, container name], and as the generic container the OTel resource attributes describe, identified by container.id. One physical container, two nodes. Kubernetes is the primary scenario, so k8s.container keeps its identity and the generic type stands down where it applies. Conventions gain a row-level condition for that: `suppressed_by` withdraws a declaration on rows where any of the named columns has a value. The test has to be per row, not per table — one descriptor table holds both pod rows and bare-runtime rows. Every branch that turns a declaration into rows now shares one guard (`declaration_predicate`), so the condition cannot apply to entities but not to the edges they carry. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): report derived entity declarations in table_semantics information_schema.table_semantics only read table options, so the declarations the built-in conventions derive — for trace tables, for whitelisted Prometheus and OTel descriptor metrics — were invisible. "Why is my table not in the graph?" was answerable only from debug logs, which is not a self-service path. A new `entity_declarations` column reports the entities a table actually contributes: each one's identity, whether it came from an option or from the conventions, and any row-level condition attached to it. An expected entity missing from the list is the answer — the table name is not whitelisted, the source stamp is wrong, an id column is absent. The row filter widens to match: a table that declares nothing by option but derives entities by convention now appears, since it is in the graph and the view has to say so. The provider reaches the derivation through a new metadata-only trait method, keeping catalog below operator. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(semantic-graph): never yield a container to an entity nothing derives The generic container withdrew on any row carrying a pod UID, on the assumption that k8s.container would cover it. Nothing guaranteed that: k8s.container came only from kube-state-metrics, so an OTLP-only deployment — or one whose KSM data had expired or fell outside the query window — lost the container node and its edges entirely instead of gaining a more specific one. The rule now names the superseding type rather than a trigger column, and withdraws only where that type's full identity is on the row. Both OTel sources declare k8s.container themselves, under the identity kube-state-metrics gives it ([pod uid, container name]), so the two sources name one node; `k8s.container.name` joins the descriptor's projected attributes to carry it. Resolution runs once every declaration for the table is known, so a superseding type that ends up undeclared — its columns are gone, or an explicit declaration of it was skipped — leaves the guard empty and the generic container standing. A container can change type; it cannot disappear. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(semantic-graph): build the declaration JSON by serializing a type The column was assembled entry by entry into a serde_json::Map, cloning every value and allocating a String per key. A Serialize struct that consumes the declaration moves the same data instead, and the field order now reads type, origin, identity, then description. The scan path around it was doing the same kind of avoidable work: declarations were derived before the predicate could discard the table, the supersession pass cloned every declaration's identity to look up one, and the option parse claimed the time index for tables that turn out to declare nothing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): keep the structured identity for single-column ids `entity_id_attrs` was NULL whenever the identity came from one column, so a consumer holding `host` = `a3f2...` had no way to tell which column produced it, and no way back to the source table. Most identities are single-column — host, k8s.pod, k8s.node, service — so the common case was the opaque one. Build the JSON object unconditionally. Entity equality still reads `entity_id` alone, so this changes no merging: it records which attributes the id was assembled from, beside an id that deliberately omits them. Also corrects the `entity_id` column doc, which still described the `k=v,k=v` rendering replaced in #8904 by values joined in declared order. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(semantic-graph): keep what a container carried when it yields Yielding to k8s.container cost the row two things it had as a generic container. The runtime container id, which kube_pod_container_info keeps descriptive precisely because it is the handle back to runtime logs and metrics, went missing entirely: neither an id nor an attribute of any node. And the edge vocabulary knew only the generic type, so a row with the more complete labels ended up with fewer connections than one without — the container layer no longer reached its host. Both OTel sources now keep the runtime id and name descriptive on k8s.container, and the vocabulary gains the two edges that mirror the generic type's. Nothing checks a supersession against the edge vocabulary, so that requirement is written down where the rule is. Also: the conventions-failure path now reports the explicit half as its comment always claimed, entity_declarations reports scope columns, and identifies() is private again now that only declaration_predicate calls it. The two RFCs catch up with entity_id_attrs being unconditional, the view listing convention-derived tables, and supersession existing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): report unmatched clients and the longest request `real_wins` pins a `calls` edge's RED metrics to the observed span pairs: when a window's edge key holds any pair, the unmatched client spans are suppressed so one (window, edge) yields one row. That leaves no way to tell a callee that stopped responding from traffic that stopped arriving — both show up as a lower request_count. Add two columns to `semantic_relationships`: - `unmatched_count` — client spans with no server span, counted outside `real_wins` on the same row, so the suppressed population stays visible without splitting the edge into two rows. NULL for agent calls, whose inner join leaves nothing unmatched, and for declared edges. - `duration_max` — the longest single request. It goes through `real_wins`: a pair is timed by the server span while an unmatched client is timed by its own (network wait included), so mixing them would make the max describe a different population than duration_sum and duration_count. Agent calls compute it from the child spans they already aggregate. The projection contract goes from 16 to 18 columns; every branch projects both. Explicit column queries are unaffected, `SELECT *` and ordinal-based readers see the new shape. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(semantic-graph): drop the duplicated source-gate case The whitelist gate on `source=opentelemetry` is already covered by `otel_implicit_declarations_are_gated` and by the wrong-source case in `table_semantics`; here it only paid for another table create, insert and full graph derivation. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * style(semantic-graph): trim the comments back to what the code cannot say Several comments restated the code, repeated a rationale already stated at the type or in the RFC, or explained a test in more words than the test body. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-08-31 10:39:17 +00:00
Ning Sun	d32cd77505	fix: postgres describe for more statements (#8974) ... * fix: postgres describe for more statements Signed-off-by: Ning Sun <sunning@greptime.com> * fix: cover more show statements Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments - add missing `clippy::too_many_arguments` allow on `query_from_information_schema_dataframe` (CI clippy failure) - take `&ShowKind` in the information-schema dataframe helper so `kind` is no longer cloned at every call site; only the WHERE arm (which needs an owned expression for `sql_to_expr`) clones internally - document why re-applying TQL explain formats never overwrites an existing value (per-query context state) Signed-off-by: Ning Sun <sunning@greptime.com> * chore: trim comments to essentials Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com>	2026-08-30 13:24:17 +00:00
Lei, HUANG	a2f39ecf7b	feat: harden frontend heartbeat extensions (#8803) ... Add a typed `Command::build_with_heartbeat_extensions` seam in `src/cmd/src/frontend.rs`. Freeze and harden `FrontendHeartbeatExtensions` in `src/frontend/src/heartbeat.rs`, with lifecycle and race coverage in `src/frontend/src/heartbeat/tests.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>	2026-08-27 17:27:52 +00:00
dennis zhuang	6d86e6ff06	feat: synthesize OTLP resource descriptor for the semantic entity graph (#8904) ... * fix(servers): compose OTLP metrics job from service.namespace/service.name The OTel Prometheus compatibility spec defines job as "<service.namespace>/<service.name>" when the namespace is present. The OTLP metrics path only used the bare service.name, so the job tag diverged from target_info produced by Prometheus-side exporters for the same resource. Compose the namespace form, and keep not fabricating a job when service.name is absent. Behavior change: resources carrying service.namespace now get "namespace/name" as their job tag value. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(otlp): synthesize otel_resource_info at OTLP metrics ingestion Ordinary OTLP metrics scatter filtered resource attributes as tags over every logical metric table, so metrics-only services contribute nothing to the semantic entity graph. Each request now also projects its distinct resources into one info-metric-shaped mito table, otel_resource_info: a fixed allowlist of identity-relevant attributes under their raw OTel keys (independent of the label translation strategy and the promote/ignore headers) plus derived job/instance compatibility columns, value 1.0, and the newest data-point timestamp. The descriptor is written after the main insert is committed; a failure there (conflicting pre-existing table, auto-create disabled) degrades to an OTLP partial_success warning with rejected_data_points = 0 instead of failing the request and triggering client retries of already-accepted data. A request writing a metric named otel_resource_info suppresses synthesis. Legacy mode is unchanged. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(operator): otel info-metric conventions with host/container entities Whitelist the ingestion-synthesized otel_resource_info descriptor via a new otel_info_metrics conventions map, gated on source=opentelemetry (the existing gate hardcoded source=prometheus). Its declarations use explicit descriptive lists instead of descriptive_rest so identifying attributes of other entities do not leak into service.instance. Conventions tightened per the Astronomy Shop findings: host identity is host.id with host.name descriptive only (host.name is not stable across SDKs and resource detectors), a generic container entity (new entity type) is declared only when container.id is present, and trace-v1 tables now synthesize host/container from their flattened resource attributes too. New co-declared edges: service.instance runs_on container, container runs_on host. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(otlp): cover the resource descriptor in integration tests Covers the descriptor's raw-key columns and info-metric options through the HTTP path, the namespace/name job composition end-to-end, column names being independent of the translation strategy, the allowlist excluding unlisted resource attributes, auto-create after a drop, the metric-name collision suppressing synthesis, and the partial-success warning (rejected_data_points = 0) when a pre-existing incompatible table fails the descriptor write while metric data is accepted. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: cargo fmt Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(frontend): degrade descriptor permission denial to a warning A table-level permission policy denying otel_resource_info would have failed the whole OTLP metrics request because the descriptor's permission check ran before the main insert. The descriptor is derived enrichment: check its permission in the degrade path so a denial skips the write and surfaces as the partial-success warning, like any other descriptor write failure. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(otlp): guard descriptor writes with semantic ownership markers A pre-existing schema-compatible table named otel_resource_info would silently receive descriptor rows while its missing semantic stamps kept it out of the entity graph. The descriptor write now requires the auto-created table's ownership markers (mito engine + signal_type + source + metric.type=info + metadata_quality=declared) and otherwise degrades to the partial-success warning; the entity-graph gate for the otel whitelist likewise requires metric.type=info, so a user table stamped with only signal/source no longer picks up implicit declarations. Also fold the descriptor write cost into the response and surface the degrade warning through the otel-arrow BatchStatus status_message. Integration tests pin the full marker set on auto-create and that an existing owned descriptor keeps accepting writes without degrading — a missing marker would otherwise silently stop every descriptor write after the first request. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(otlp): build descriptor rows without the per-resource BTreeMap Projecting a resource allocated a BTreeMap and then collected it into the row key, and every attribute was matched against the allowlist by linear scan. Collect the tags into a Vec and sort once, and match the allowlist instead of scanning it. Measured on the conversion path: descriptor work drops 16-18%, from 10.6% to 8.9% of conversion CPU on the worst shape (1000 resources with 4 data points each), where the cost tracks resource count rather than data-point count. Also trims the comments and tests added with the descriptor to what carries information. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(otlp): pin the descriptor permission-denial degrade path A policy denying the descriptor table must not fail the metrics request, which the fix in 2401b3dd9c does but nothing covered. Verified as a regression guard by mutation: moving the permission check back before the main write makes this test fail. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(otlp): keep legacy mode covered after trimming the unit tests Trimming the descriptor tests dropped the only assertion that legacy mode skips the job/instance remap and the promote filter. Both alter the columns of tables already in use, so fold the check into the legacy conversion test rather than leaving it uncovered. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(semantic-graph): stop encoding column names into composite entity ids A composite entity id rendered the identifying columns as sorted `col=value` pairs, so the same identity split into one entity per signal: a trace table names its columns service_name and resource_attributes.service.instance.id where a metric table names them job and instance. One service instance became two nodes with two parallel edge sets, breaking the walk from a trace to that instance's metrics. Render an id as its values in declared order instead, escaping the separator so components stay distinguishable, which is what single-column ids already did by keeping only the value. entity_id_attrs still carries the structured form. Values alone are not enough for a namespaced service: the metric side folds service.namespace into job while traces keep the bare name. Add qualified_by to the conventions so the trace declarations compose the namespace the same way, per the OTel rule that job is <service.namespace>/<service.name> or the bare name when the namespace is empty. A table without the namespace column keeps the unqualified identity rather than losing the declaration. Conventions validation now rejects one entity type declared with a different number of id columns by two sources, which would silently produce ids that can never match. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * style(otlp): import the parent module by crate path check-super-imports.py, part of the CI format gate, rejects a file-level `use super::`. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(otlp): gate the resource descriptor, and fix what review found Synthesizing greptime_otel_resource_info creates and writes a table the user never sent, so it is now off unless otlp.experimental_enable_resource_info says otherwise. With it off the request costs exactly what it did before the descriptor existed: nothing is projected, no table is created, no write and no permission check happen. Tests run with it on. StandaloneOptions carried no otlp field, so the whole [otlp] section was silently dropped in standalone mode; map it through, or the new option (and trace_ingest_chunk_size before it) would do nothing there. Renamed from otel_resource_info: the greptime_ prefix marks the table as engine-managed and makes a collision with a user metric unlikely, which is what the pre-existing-table ownership check and its per-request catalog lookup were defending against. Both are gone. A request may carry data for several graph windows, but the descriptor folded every data-point time into one row at the newest of them, leaving the earlier windows with metric rows and no entities. Key the rows by window as well, and take the times from the data points the encoder actually writes: it drops exponential histograms, and a resource carrying nothing else was being described as an entity with no measurements. Projecting a resource cloned its attributes once per data point. Nest the windows under the attributes instead, so they are moved once per resource, and walk the data-point times through a visitor rather than collecting a Vec per metric. Also documents what the two maps key and hold, and lifts the projected attribute names to constants beside KEY_SERVICE_NAME. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(otlp): skip the descriptor's work entirely when it is disabled The collision scan over the request's output tables ran even with the feature off. Short-circuit on the option instead, and update the config snapshot the new [otlp] section changed. Also drops the doc comment orphaned by the deleted ownership check: it had attached itself to the trait impl and described a check that no longer exists. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(semantic-graph): drop the expect and name the service identity The CASE is built through Case directly rather than the fallible when().otherwise() builder, so the non-test path no longer carries an expect (architecture-invariants $4). service_identity returned two same-typed Options that both call sites destructured positionally; a named struct makes a swap fail to compile. Also records that id-column order is part of the identity, where the option docs and the conventions authors will read it. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(semantic-graph): drop comments that narrate the code Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(semantic-graph): cast duration_nano before the trace-table union Trace tables written before the signed-integer ingest change hold duration_nano as UInt64 and later ones as Int64. The calls derivation unions the per-table selects, and the two have no common integer type, so a deployment holding both shapes could not build the plan. The cross-table test now spans both. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(semantic-graph): drop the redundant duration_nano casts Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(otlp): decide exponential histogram acceptance in one place The resource descriptor mirrored only the experimental gate, so with both experimental flags on a resource whose only metric is a delta exponential histogram was described as an entity with no measurements. The encoder's whole-metric rules move into exponential_histogram_gate, which both call, and the descriptor takes its timestamps through exponential_histogram_value so per-point rejections drop out too. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-08-25 13:08:44 +00:00
shuiyisong	1c5eabcbbf	feat(otlp): support cumulative exponential histograms (#8900) ... * feat: implement exponential histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove duplicate tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): enforce exponential histogram ingestion safety Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update rfc Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): remove protocol-coupled histogram checks Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): reuse native histogram schema across data points Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: merge repeated OTLP histogram fragments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): build rejection messages lazily Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: add doc Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-08-24 12:59:44 +00:00
Ning Sun	174577164a	feat: otlp duration_nano and trace_flag signed integer coercion (#8816) ... * feat(servers): add signed→unsigned int coercion to OTLP ingest path Phase 0 of transitioning built-in data models from unsigned to signed integers (#8793): add lossless Int64→UInt64 and Int32→UInt32 coercion arms so existing UInt64/UInt32 columns (e.g. trace `duration_nano`, log `trace_flags`) keep accepting new signed ingest without an ALTER TABLE. The OTLP ingest path already reconciles every incoming column against the existing table schema and treats it as authoritative. With these arms, `choose_trace_reconcile_decision` returns `UseExisting(UInt64/Uint32)` for an existing unsigned column receiving signed data: the table keeps its type byte-for-byte and the request value is coerced. No persisted format is mutated; existing data stays readable as-is. This is the safety net that makes the actual schema flip (Phase 1) safe. Only the signed→unsigned direction is supported — the reverse would be lossy for values above the signed range and is intentionally rejected. Tests cover both new arms plus an end-to-end log test proving an existing UInt64 column coerces an incoming Int64 value while keeping its type. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: add compatibility layer for uint trace/log fields Signed-off-by: Ning Sun <sunning@greptime.com> * fix: jaeger test Signed-off-by: Ning Sun <sunning@greptime.com> * fix: keep trace v0 unsigned, reject negative span durations - Keep the frozen v0 data model on UInt64 duration_nano: the signed ingest compatibility layer only runs on the v1 path, so flipping v0 would break writes into every pre-existing v0 table at mito's schema check. Pin the schema with unit and integration tests. - Reject spans whose end precedes their start (or whose duration does not fit i64) on the v1 path instead of wrapping: new Int64 tables and existing UInt64 tables now fail identically, rather than storing negative durations that break the Jaeger query API. - Extract is_supported_signed_to_unsigned_coercion so the trace and log ingest paths share one supported-pair predicate and cannot drift. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: clamp negative span durations to zero, revert semantic_graph comment - Record duration 0 for spans whose end precedes their start instead of erroring: a malformed span no longer fails the request, and the value written is always a non-negative, in-range i64 so new Int64 tables and existing UInt64 tables (via the checked coercion) behave identically. Durations above i64::MAX saturate rather than wrap. - Revert the doc-comment tweak on the semantic_graph test fixture; the file is untouched by this PR again. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com>	2026-08-24 09:28:49 +00:00
dennis zhuang	3de4feddd5	feat(otlp): report the cause of rejected trace spans (#8897) ... * feat(otlp): report the cause of rejected trace spans When trace-v1 ingestion cannot coerce an attribute value, it falls back to single-span writes and rejects the bad span. That behavior is correct, but the OTLP partial-success message only carried `Rejected span <trace_id>:<span_id> (InvalidArguments)`: the column, the source value, the source type and the target type were all dropped, so locating the bad attribute required adding a detailed exporter on the collector side and replaying traffic. Two places lost the information. `prepare_trace_column_rewrites` built a message without the failing value, and the span rejection path kept only the status code from the error. Coercion errors now name the failing value, e.g. failed to coerce trace column 'span_attributes.http.response.body.size' in table 'opentelemetry_traces' from String("") to Int64 and the rejection detail carries that cause. Values are user data, so a string keeps at most 16 characters, is escaped, and binary payloads report only their length; the cause itself is bounded at 256 characters. Both truncations cut on a char boundary. Failure details now deduplicate: repeats of the same (site, cause) collapse into one entry with an occurrence count, keyed on the untruncated cause so two failures that differ past the display limit stay separate. Only four distinct entries are retained and the rest are counted, which keeps the state bounded no matter how many distinct bad values a request carries. A fully rejected request is logged at warn level and a partial success at debug level, since the latter repeats every export interval; the detail goes out as a Debug field so a newline in an attribute key cannot forge log lines. Rejection semantics are unchanged: partial success, same accepted and rejected counts, same HTTP status mapping. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(otlp): compare failure keys directly instead of hashing The dedup identity was a DefaultHasher fingerprint of `(label, key)`, which bought a fixed 8 bytes per entry at the cost of an import, four lines, and a collision argument the reader has to make. Entries are capped at four and a cause runs a couple of hundred characters, so the saving is about a kilobyte per in-flight request while the column name it avoids retaining is already held several times over by the request itself. Compare the strings instead, keeping the untruncated cause as the key so failures differing past the display limit still stay apart. Labels are metric label values and always static, so the entry borrows them. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-08-19 04:32:23 +00:00
Ning Sun	8dbfbee611	refactor: remove open metrics parser (#8905)	2026-08-19 03:47:44 +00:00
Lei, HUANG	7539e60139	refactor: remove trivial tests (#8877) ... * refactor: remove trivial tests Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: remove unused trace test import Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>	2026-08-13 12:10:49 +00:00
Weny Xu	1af4c33524	refactor(event): separate procedure submission context (#8856) ... * refactor(event): separate procedure submission context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): map extensions and forward GC context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(gc): initialize integration test context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(test): pass procedure context to DDL helpers Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: simplify procedure submission contexts Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): separate procedure and query contexts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): clarify procedure context propagation Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve procedure submission context Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): move DDL context by value Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(test): retain manual GC event context Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): tighten procedure context API Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update greptime-proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2026-08-13 07:29:00 +00:00
dennis zhuang	546625c45a	feat: embedded convention pack for the entity graph (prom/k8s, gen_ai naming) (#8854) ... * feat: embed the derivation conventions as data and adopt gen_ai entity naming Move the co-declared edge vocabulary, the agent-edge vocabulary and the virtual-destination candidates from Rust consts into an embedded conventions.yaml (include_str!), parsed once behind a LazyLock and validated against the entity-type grammar and the closed rel_type set; a broken file propagates as a plan error instead of panicking. The agent vocabulary entity types follow the GenAI semantic-convention namespace as written: gen_ai.agent / gen_ai.model / gen_ai.tool. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: drop the tag requirement for entity identity columns Entity declarations no longer require id columns to be tag/primary-key columns; only column existence is validated. Trace pipelines flatten the identifying attributes (span_attributes.gen_ai.agent.id, ...) into field columns, so the tag rule locked real trace tables out of declaring entities while buying no correctness — the read-time derivation works on any column. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: implicit declarations for well-known prometheus info metrics Tables stamped signal_type=metric + source=prometheus whose name matches the conventions.yaml whitelist (kube_pod_info, kube_node_info, kube_pod_owner, target_info) get implicit entity declarations: k8s.pod / k8s.node / k8s.workload with name-based identity and target_info's service / service.instance with the remaining tags as the descriptive snapshot. The existing co-declared vocabulary then derives runs_on and part_of from the same rows, so no new edge branch is needed. Explicit declarations of a type always suppress the implicit one, and the metric engine's physical table is excluded (it aggregates every logical table's columns). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover the prometheus conventions in sqlness and compact the graph cases Add the whitelisted-info-metric scenario (kube_pod_info, kube_pod_owner, target_info deriving runs_on / part_of, a non-whitelisted metric contributing nothing), fold the single-table calls, cross-table pairing and virtual-node cases into one trace scenario (they exercise the same union-before-join path), merge the two declaring-metric-table cases, and reuse one rename probe for both reserved names. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject entity id columns without a stable string form Review follow-ups: the DDL check now validates against the schema and rejects binary-backed and nested types for identity columns (the derivation renders ids via CAST to Utf8, so the failure used to surface only when the graph was scanned); the agent sqlness case keeps its identity columns as fields to cover the relaxed tag rule end to end; stale tag-rule comments and a dangling const reference are cleaned up. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: type-check every entity column role, not only ids The registry renders scope and descriptive values through the same CAST-to-string path as ids, so a binary-backed column in any role fails at scan time; the DDL check is now role-independent (and simpler). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: name the code-anchored vocabulary constants Entity types and edge attributes the derivation code itself anchors on (service, gen_ai.agent, calls, trace/attribute provenance) become constants in the conventions module; the rest of the vocabulary stays YAML-only data. ImplicitEntity is renamed PromImplicitEntity, and the implicit-declaration path logs each skip of a whitelisted info metric (wrong stamps, suppressed by an explicit declaration, missing id column) so a missing graph entity is diagnosable. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: single-source the graph constants The graph tables' column names move to common-catalog (the schemas catalog exposes and the plans operator builds must match column by column), and the conventions module now carries the complete built-in vocabulary — entity types, rel_types, provenances and connection types — with the embedded YAML validated by membership against it, so an edit drifting outside the vocabulary fails the conventions test instead of deriving nothing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: treat empty identity components as absent kube-state-metrics emits empty-string labels an entity id must not be built from: an unscheduled pod's node and an owner-less pod's owner_kind / owner_name. Standard Prometheus drops empty labels (they arrive as NULL and the existing predicate handles them), but other remote-write agents may keep them, which produced ghost entities with empty ids and false runs_on / part_of edges. Every identity predicate (registry, co-declared edges, span endpoints) now requires non-NULL and non-empty components through one shared helper. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: tighten the conventions DSL semantics Rename the co-declaration rule lists to what they are (co_declared_edges / trace_co_declared_edges — derivation rules, not a relation vocabulary), stop overstating the GenAI entity types (Greptime types derived from GenAI attributes; OTel defines no model/tool entities), move target_info's descriptive snapshot to service.instance (the remaining labels are the target's resource attributes, and instances would write conflicting snapshots onto the logical service), and extend the descriptor whitelist with the stable KSM sources: container info metrics (closing the k8s.pod contains k8s.container rule), kube_service_info (new k8s.service entity type) and the fuller descriptive label sets. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: guard entity column types on ALTER as well ALTER MODIFY COLUMN could change a declared entity column to a type without a stable string form, deferring the failure to graph scan time; verify_alter now checks the post-alter schema. Dropping a declared column stays allowed — the read-time derivation skips the stale declaration, and semantic options cannot be altered off yet. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: bridge traces and kube-state-metrics on the pod UID Trace-v1 tables now get implicit declarations from their flattened resource attributes (otlp_trace_entities in conventions.yaml): the service identity — replacing the hardcoded fallback — plus service.instance and k8s.pod, each applied only when its columns exist. A new co-declared rule derives service.instance runs_on k8s.pod, and the whitelisted kube-state-metrics pod identity switches from namespace+pod names to the UID, so the trace-side pod and every KSM descriptor land on one entity while names stay descriptive. This also removes pod identity from the multi-cluster same-name collision. The conventions rejection tests were passing for the wrong reason (a half-renamed fixture key failed deserialization before reaching any validation rule); they now assert the specific error each case targets. Sqlness covers the UID merge across descriptor tables, pod-contains- container, the k8s.service node, and the empty-uid/empty-node rows deriving nothing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover the OTLP-to-graph chain end to end One real OTLP export must come out of semantic_relationships as the zero-configuration chain: service calls service, instance part_of service, instance runs_on pod (bridged by k8s.pod.uid). Resources without service.instance.id or k8s.pod.uid derive nothing extra. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: identify k8s.service by UID Same reasoning as pods: a recreated same-name service must not merge into the old entity and same-named services across clusters must not collide; kube_service_info carries a stable uid and nothing joins on the service's name. Also drop a stale tag-rule mention from the option validation docs. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: cut duplicated test coverage and redundant comments The trace service-fallback test collapsed into the resource-entities test (same synthesis path since the fallback moved to YAML; only the invalid-explicit-no-fallback case was distinct), role-duplicate and subsumed DDL cases are gone, the embedded-conventions test is just the parse (its assertions were decorative), and the YAML section comments no longer restate the struct docs. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-08-13 02:31:56 +00:00
dennis zhuang	e778a72829	feat: complete the derived-edge vocabulary of the entity graph (#8836) ... * feat(operator): pair calls edges across trace tables and derive virtual-node edges Union the normalized client and server spans of all trace tables before the join, so a client span pairs with a server span stored in a different table. A client span with no matching server span becomes an edge to a virtual node named by span attributes (peer.service / db.name / server.address), with confidence < 1.0 and attributes.connection_type; a window's real pairs win over virtual candidates for the same edge key. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(operator): derive same-row co-declared edges from the built-in vocabulary A table declaring both entity types of a vocabulary pair witnesses the edge on every row carrying both identities: runs_on / contains / part_of for any declaring table (provenance 'attribute'), agent uses model / agent invoked tool only for trace sources (span-structure observations, provenance 'trace'). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(operator): derive parent_agent-calls-agent edges from span structure Trace tables declaring an agent entity pair each span with its child span across tables (no span-kind filter), keep pairs whose agent identities differ, and aggregate RED metrics per window, anchored on the parent span like the service derivation is anchored on the client. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(frontend): feed co-declared and agent sources into the relationships scan scan_relationships now passes every declaring table (with its trace-ness) to the co-declared branch and the trace tables' agent declarations to the agent-calls derivation. enumerate validates the fixed trace-v1 columns and derives around a malformed trace table instead of failing the whole scan. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover cross-table pairing, virtual nodes, co-declared and agent edges sqlness exercises the new derivations end to end (including a malformed trace-model table being skipped); the integration authorization test now also pins that a pair split across tables derives no edge when the caller cannot read one side. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: update the relationships module doc for the new branches Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: import shared derivation helpers via crate paths The fmt CI gate rejects module-level 'use super::' imports. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: fold co-declared duplicates, decouple agent calls, verify the trace time index Review findings: the co-declared branch lacked a cross-source DISTINCT, so two tables witnessing the same edge in one window emitted duplicate rows; the agent-calls derivation was gated on a usable service declaration; the trace schema guard accepted a table whose time index is not the column the derivations bucket by. The empty-trace-table test asserted a union invariant with no information and is dropped. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: rename the agent-tool edge to invokes and track current OTel peer attributes The vocabulary's other relation names are present tense; semconv 1.39/1.26 replaced peer.service and db.name with service.peer.name and db.namespace, so the virtual-node candidates now check the current names first and keep the deprecated ones for existing telemetry. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: trust the trace-v1 table option instead of matching the fixed schema The option is only ever stamped by the ingest path, which guarantees the fixed span columns; matching column types here couples the graph to every trace schema evolution (e.g. #8816) for a case that cannot occur. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-08-12 04:06:31 +00:00
shuiyisong	154f90b365	fix: harden permission checks and process visibility (#8852) ... Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-08-12 03:12:04 +00:00
Weny Xu	943eee852f	feat(event): record admin function executions (#8835) ... * feat(event): record admin function executions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): handle admin function recording edge cases Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(event): record actor for admin functions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve admin function event values Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve non-finite admin results Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2026-08-11 12:50:50 +00:00
dennis zhuang	4eeb4052d6	feat(servers): stamp prometheus remote write v2 metadata as semantic table options (#8797) ... * feat(servers): stamp prometheus remote write v2 metadata as semantic table options Remote write 2.0 carries per-series metadata (type, unit, help) that the v2 ingest decoded and dropped; tables kept the name-based 'inferred' quality. Wire it into the semantic layer: - generalize the OTLP per-table semantic index into a shared, schema- aware servers::semantic module: v2 lets each series override its target schema, so the index is keyed {schema -> table -> options} and the same metric name in two schemas no longer collapses; - into_write_requests records metric type and unit per written table; an explicit type upgrades the table's metadata quality to declared, UNSPECIFIED series keep the request-level inferred stamp, and units are canonicalised from OpenMetrics words to the UCUM codes the vocabulary is defined in (unknown units are dropped, help text is not persisted); - the consumer folds the index in on both auto-create paths: the operator row-insert path and the pending-rows batched create, which bypasses the former. Table options are stamped at auto-create only; updating existing tables from later metadata is future work (a metadata registry, see the native histograms RFC). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: trim over-commenting in the remote write metadata path Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: parse the per-table semantic index once per create round The index was re-parsed from JSON for every table being created — a first write creating N tables (a fleet's first scrape) paid O(N x index size). Parsing now happens lazily once per create-planning round, on both consumers: the operator row-insert auto-create (also serving OTLP metrics) and the pending-rows batched create. Also validate every non-zero metadata symbol reference up front, as the remote write 2.0 spec requires: help_ref was never checked, and unit_ref escaped checking when the metric type was UNSPECIFIED. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(servers): stamp remote-write v2 unit independently of metric type OpenMetrics models TYPE and UNIT as independent MetricFamily metadata, and the Prometheus v2 sender emits UNSPECIFIED-type series that still carry a unit. The early return on UNSPECIFIED dropped that unit, which is unrecoverable after table auto-create (units are not stored in rows). Stamp the mapped unit whenever present; the type and the metadata_quality=declared upgrade still require an explicit type. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: trim restating comments in the v2 metadata path Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-08-11 06:20:08 +00:00
dennis zhuang	335a95a369	feat: declared edges and the derivation contract for the entity graph (#8794) ... * feat(frontend): run entity-graph derivation as the caller The derivation contract requires the computed graph tables to run under the outer query's identity. Capture the caller's QueryContext when the computed table is resolved, thread it through EntityGraphProvider, and: - authorize every contributing source table against the caller via the new semantic_graph.query permission action, silently excluding denied sources (entities, edges and source_tables never appear); - execute the derivation plan under the caller's context so it inherits permissions, cancellation and deadline instead of a fresh default. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(operator): derive the entity-graph window from the scan's time predicate Implements the RFC window contract for the computed graph tables: - table: add extract_time_range_strict, a strict variant of the lenient time-range extraction that distinguishes an absent observed_at filter from one that cannot be safely turned into a range; - operator: replace GraphWindow with GraphQueryWindow, splitting the queried observed_at range from the source-scan range widened to whole 60s buckets, so boundary buckets aggregate over their full extent; - frontend: resolve the window from ScanRequest filters — no predicate keeps the last-hour default, a missing upper bound means now, and a missing lower bound or unextractable shape is an explicit error, never a silent fallback. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(operator): system-defined declared-edge table for the entity graph Reintroduces greptime_private.semantic_relationships_declared with a canonical, system-owned definition: - the CREATE TABLE expr (8-tag primary key, business validity columns, RED fields, 30d TTL); attributes is now a json column so the future union branch matches the computed table without a per-scan parse; - created on first use on every write path: SQL INSERT creates it before executing, and the gRPC row-insert auto-create substitutes the canonical expr instead of deriving a schema from the request; - user DDL (CREATE/ALTER/DROP/RENAME/TRUNCATE) and write-path auto-ALTER are rejected via the new is_ddl_reserved_table guard, while INSERT/DELETE stay allowed. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(operator): union declared edges into semantic_relationships Adds the declared-edge branch to the relationship derivation (build_relationships_plan replaces build_calls_plan): - latest revision per edge key first (mito dedups on primary key plus observed_at, so a re-asserted edge stores a new revision), then the business-validity overlap against the queried window; valid_from defaults to the declaration time and a NULL valid_until means the edge holds while its row exists; - the projected observed_at is synthesized inside the queried range (Inexact pushdown re-applies the scan's filters above the computed table, which would drop rows keyed by the physical revision time); window_end/fresh_until of open-ended edges take the window's upper bound so 'fresh_until >= now() - ...' queries see them; - tag columns are cast out of dictionary encoding, and the union is re-projected to the 16-column contract; - the frontend feeds the branch only when the physical table exists, the caller may read it, and its schema still matches the canonical definition (mismatch is an explicit error, not a silent drop). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover declared edges, window contract and caller authorization - sqlness: system auto-create on first INSERT, latest-revision reads, open-ended vs retired validity, explicit/lower-only/upper-only window behavior, user-DDL rejection, rename-into rejection, DELETE cleanup; - integration: a permission checker denying one trace table excludes it from both semantic_relationships and semantic_entities. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: allow DROP/TRUNCATE on the declared-edge table and fix CI lints The definition guard rejected every DDL, which left sqlness (and any shared deployment) no way to remove the table the semantic_graph case creates — its extra region then broke unrelated region/partition case expectations. Narrow the guard to what actually protects the canonical definition: user CREATE, ALTER, RENAME-into and repartition stay rejected, while DROP and TRUNCATE are allowed — dropping loses nothing structural, the next INSERT recreates the table canonically, and DROP doubles as the recovery path if the canonical definition ever changes. The sqlness case now verifies drop-then-recreate and cleans up after itself. Also: rustfmt for the catalog crate and two typo fixes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: adapt canonical declared-table create to TriggerReason Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: address review on the declared-edge table lifecycle and revision reads - gRPC first writes actually work now: the reserved table's creation went through the generic create_table_inner, which the definition guard itself rejects; both branches of create_or_alter_tables_on_demand route it to create_declared_relationships_table instead, and being a system action it also bypasses the auto_create_table config/hint; - revision selection is as-of the queried window: revisions recorded after the window's end, or whose validity starts after it, no longer outrank (and hide) the revision that was in effect inside it; - the canonical-schema check validates the whole definition the union semantics lean on — time index, primary key, engine, append/merge mode — not just column names and types; - UNDROP TABLE of the reserved name is rejected like CREATE: it could resurrect a pre-canonical shape, and the next INSERT recreates the table anyway. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: trim over-commenting in the entity-graph code Comments that restated adjacent code or narrated justification are cut; the ones stating non-obvious contracts and gotchas stay. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject CREATE VIEW against DDL-reserved table names A view named greptime_private.semantic_relationships_declared would squat the reserved name: the first INSERT then skips the canonical create (an object already exists) and graph reads fail on the schema mismatch. CREATE VIEW now passes the same definition guard as CREATE TABLE. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: debug-log authorization exclusions; declared-edge TTL to 90d Sources the derivation contract silently excludes (per-table denial, whole-scan denial, the declared-edge table) are invisible from outside; a debug log at each names what was excluded and why. The declared-edge table's default TTL becomes 90d, overridable at creation time via GREPTIMEDB_DECLARED_RELATIONSHIPS_TTL (a proper configuration option is a TODO). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: rank declared-edge revisions by the visible edge identity Ranking partitioned by the full primary key, but the projection drops scope and generation_id: two assertions of the same visible edge under different generations both ranked first and came out as duplicate, indistinguishable rows. Rank by the exposed identity (endpoints, rel_type, provenance) instead, with generation_id/scope as deterministic tie-breakers for same-timestamp assertions. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: drop redundant declared-edge tests The generations regression is already asserted by the revision and as-of tests; the DDL shape test restated the declarative builder against itself. Its one non-tautological check (attributes maps to the json type) moves into the schema-matcher test. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject disjunctive graph windows and unmatchable future windows - OR/IN over observed_at collapse disjoint ranges into their convex hull; a declared edge's synthesized timestamp can land in a gap and be dropped by the re-applied filter even though the edge is valid at a requested instant. The strict extractor now rejects those shapes. - A lower bound in the future inverts against the implicit up-to-now upper bound; the declared branch then fabricated an edge observed at the future bound. Such windows now derive nothing. - The reserved-table gRPC create path classifies an instant-TTL table like every sibling path. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-08-10 12:18:28 +00:00
shuiyisongandfys	d4af650ec0	perf: reduce cold workspace compile time (#8801) ... * refactor: remove datanode and meta-srv dep from frontend Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: use on-device protoc if possible Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: remove unused dep Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: version and docs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update logs Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>	2026-08-10 07:18:16 +00:00
Ning Sun	606e5fd6a0	feat: update opentelemetry family to 0.32 series (#8776) ... * feat: update openetelemtry family to 0.32 series * chore: resolve warning * fix: update tests	2026-08-10 03:42:53 +00:00
dennis zhuang	5d4699db1c	docs: refine coding agent maps (#8790) ... * docs: refine coding agent maps Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: trim license header guidance Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: update README links and project status Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-08-10 02:45:37 +00:00
Gaurav ShokeenandLei, HUANG	d90cca4b75	fix(prometheus): custom column remote reads (#8659) ... * fix(prometheus): custom column remote reads Resolve timestamp and value column names from the table schema and carry them through query planning and result conversion. Add a remote-read regression test covering custom_ts and custom_value. Signed-off-by: grezzko <me@gauravshokeen.com> * fix: resolve remote-read value columns safely Prefer the sole field for custom schemas and greptime_value for multi-field tables. Reject ambiguous schemas and add regression tests. Signed-off-by: grezzko <me@gauravshokeen.com> --------- Signed-off-by: grezzko <me@gauravshokeen.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>	2026-08-07 13:37:00 +00:00
Weny Xu	1f1c9270a8	feat(event): add event context to procedure events (#8734) ... * feat(event): add trigger context to procedure events Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix trigger context event contracts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve trigger context origins Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve lifecycle trigger contexts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): gate enterprise trigger imports Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): assert trigger contexts exactly Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(test): restore migration test literals Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(event): add trigger context to non-table ddl Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): simplify trigger context encoding Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve auto alter trigger reason Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): require explicit trigger context Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): derive trigger context at ddl boundary Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): propagate DDL trigger reasons Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): resolve alter table rebase conflict Signed-off-by: WenyXu <wenymedia@gmail.com> * test(meta): fix alter table trigger context setup Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): pass trigger context to region migration Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix unknown trigger context assertions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): centralize trigger context protocol mapping Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix migration trigger context assertion Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(procedure): rename event runtime context Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): rename trigger context Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix migration event context assertion Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2026-08-07 12:37:58 +00:00
discord9	97ca104129	fix(frontend): remove gRPC DDL panics for DropView and non-timestamp time index (#8739) ... * fix(frontend): remove gRPC DDL panics for DropView and non-timestamp time index Direct gRPC DDL bypasses the SQL parser, so two client-controlled DDL payloads could panic a request handler: - QX-152: DdlExpr::DropView hit todo!() (instance/grpc.rs:247-248). Wire it to the real drop-view implementation (drop_view was pub(crate); widened to pub) so a DropView DDL returns a structured error (e.g. TableNotFound) instead of panicking. - QX-153: a CreateTableExpr whose time_index column is not a timestamp reached Schema::new's unwrap (ddl.rs:2346 -> schema.rs:114-119). create_table_info now uses Schema::try_new with ConvertSchemaSnafu context (InvalidArguments), and the direct gRPC CreateTable arm validates the request via validate_create_expr (which now also checks the time-index column type is a timestamp) before any catalog work. SQL/HTTP paths were already protected by the parser; unchanged. Tests: qx_152_drop_view_via_grpc_ddl_returns_error_not_panic, qx_153_create_table_with_non_timestamp_time_index_via_grpc_returns_error (asserts InvalidArguments), test_create_table_info_rejects_non_timestamp_time_index. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(frontend): add gRPC DDL happy-path coverage for DropView and CreateTable Per review: the initial tests only asserted error paths. Add: - drop_if_exists=true on a missing view succeeds (no error) - dropping an existing view via gRPC DDL succeeds end-to-end - a valid CreateTableExpr with a timestamp time index still succeeds (guards validate_create_expr against rejecting good requests) - qx_152 test now asserts the TableNotFound status instead of is_err() Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-08-07 06:48:15 +00:00
dennis zhuang	3d12273c84	feat: read-time entity relationships graph over telemetry (M0+M1) (#8614) ... * feat(table): add entity semantic declarations Define open-ended greptime.semantic.entity.* options, validate entity columns at DDL time, and stamp OTLP trace tables with the service entity declaration. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: add read-time entity relationships graph Add computed semantic graph tables, typed DataFusion derivation plans for entity registry and trace calls edges, and streaming read-time execution. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: exclude semantic graph tables from table constraints Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(operator): name the plan-builder source groupings Review feedback: build_registry_plan / build_calls_plan took anonymous (declarations, DataFrame) tuples while the caller already grouped the same fields. Introduce RegistrySource { declarations, scan } and CallsSource { service, scan } next to the builders and flow them through the frontend caller and tests. The frontend-side EntitySource keeps holding a TableRef (the operator builders stay pure over already-built scans), so the named structs live in operator rather than reusing that type. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-08-07 03:37:14 +00:00
+414 people	cabc2f6cc6	feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (#7987) (#8392) ... * feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987) Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987) Signed-off-by: onepizzateam <palakjha916@gmail.com> * test(flow): add sqlness golden result for flow_status Signed-off-by: Palak Jha <palakjha916@gmail.com> * fix(catalog): remove unused OptionExt import in flow_statistics Signed-off-by: onepizzateam <palakjha916@gmail.com> * docs(flow): fix stale 'recent errors' comment on QueryFlowExecStats Signed-off-by: onepizzateam <palakjha916@gmail.com> * test: regenerate golden results for flow_statistics table Signed-off-by: onepizzateam <palakjha916@gmail.com> * style: apply rustfmt to flow_statistics changes Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor(catalog): hoist current_time_millis out of flow loop and clamp uptime Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore: remove accidentally committed fmt_check.log Signed-off-by: onepizzateam <palakjha916@gmail.com> * Update flow_status.result del eof trailing blank line as per review Signed-off-by: onepizzateam <palakjha916@gmail.com> * test(flow): restore runner-generated trailing blank line for sqlness Signed-off-by: onepizzateam <palakjha916@gmail.com> * postgres: include SHOW FLOW STATUS in extended-query describe (return flow_statistics fields) Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: address reviewer feedback on flow_statistics PR Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): resolve merge conflicts with main Signed-off-by: polar <palakjha916@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * sqlness check post gen (information_schema.result) Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): record start_time after req/snapshot_seqs built, before dispatch Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(sql): handle ShowFlowStatus in match statement at util.rs Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: review patch implementation Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore: remove accidentally committed local tool output files and fix fmt Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix worker.rs return type formatting Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): apply rustfmt to get_full_flow_stat return type Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix worker.rs return type formatting Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): re-apply rustfmt to get_full_flow_stat return type after merge Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: merge conflicts Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(auth): warn when credential load disables Postgres SCRAM or drops a line (#8652) * fix(auth): warn when credential load disables Postgres SCRAM or drops a line Static and watch user providers degraded silently in two ways: - A single non-SCRAM verifier (mysql_native_password, or a legacy pbkdf2_sha256 hash that predates SCRAM) disables Postgres SCRAM for every user and falls back to cleartext, with no signal to the operator. - A malformed credential line (commonly a plaintext password containing '=', which splits into more than two parts) was dropped without a trace. Emit a warning at each credential load for both cases so operators don't unknowingly serve cleartext passwords over Postgres or lose a user. This is logging only; authentication behavior is unchanged. The SCRAM check never logs secrets, and the malformed-line warning logs the line number and file, never the line content. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): warn on credential file read error before truncating A read error from lines() (I/O failure or invalid UTF-8) ends the iterator via map_while, silently dropping every remaining credential. Warn with the line number and file before truncating, matching the malformed-line handling, so the drop is observable. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * test(object-store): fix racy SecureFs abort test (#8720) test_writer_abort_is_unsupported_without_atomic_write asserted the file content immediately after abort() returned Unsupported. SecureFsWriter writes through tokio::fs::File, whose write_all() only enqueues a blocking write task (tokio's poll_write returns Ready before the write completes), so the data may not be visible yet when the test reads the file. Drop the race-prone content assertion and only verify the Unsupported contract. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(query): plan native histogram functions (#8705) * feat(query): plan native histogram functions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue & add tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * perf(promql): avoid repeated scans in sliding range evaluation (#8646) * perf(promql): use two pointers for sliding range boundaries Replace the stale cursor heuristic in RangeManipulateStream::calculate_range with monotonic left/right cursors. The old path rescanned each evaluation window (O(E x samples-per-window)) and could lose valid samples after sparse gaps or trailing empty windows. The two pointers keep strict monotonic progress, reducing boundary generation to O(N + E) while preserving (curr-range, curr] semantics, start/end shortening, and empty-window output. Controlled release benchmarks (fixed CPU, ABBA): - Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s, ~96% at 1h/15s. - Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end; shorter windows stayed within run-order noise. Signed-off-by: discord9 <discord9@163.com> * perf(promql): specialize changes/resets with adaptive edge counting The generic range_fn macro slices, downcasts, and rescans every overlapping window for changes() and resets(). Replace the macro path for these two functions with hand-written UDF wrappers backed by a shared private edge-count kernel: direct raw-offset scans when requested edges are few, otherwise one global u64 edge prefix so each window is answered by a prefix difference. Behavior is preserved bit-for-bit, including raw null-buffer values, NaN semantics, signed zero, infinities, empty/singleton windows, independent timestamp/value offsets, arbitrary window layouts, and exact DataFusion error messages. The shared proc macro, planner, serializer, and other range functions are untouched. Controlled release benchmarks (fixed CPU, ABBA): - Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time. - Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster. - Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client and 12.0-20.9% server latency improvement; controls stayed within drift. Signed-off-by: discord9 <discord9@163.com> * ci(query-regression): include PromQL range boundary case in defaults An audit of historical query-regression runs found zero range-query coverage: all 208 PromQL ANALYZE samples were bare selectors, so range evaluation could regress without CI noticing. Wire the promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so label-triggered runs measure the range path. The case is cheap: a ~0.3s synthetic fixture and about a minute of query execution per base/candidate pass. Signed-off-by: discord9 <discord9@163.com> * chore(promql): address sliding range review nits Move test-only imports into their test modules and remove the unused pre-specialization changes and resets helpers. Signed-off-by: discord9 <discord9@163.com> * style(promql): apply pinned rustfmt Signed-off-by: discord9 <discord9@163.com> * test(promql): cover sparse range results Share the changes and resets test scaffolding while keeping their behavior oracles independent. Add an end-to-end sqlness regression for sparse samples, empty intermediate windows, and a valid trailing sample. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * ci: optimize fuzz and split workflows (#8710) * ci: batch fuzz targets in GitHub Actions Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: improve fuzz test observability Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): preserve fuzz setup failure artifacts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(ci): keep fuzz mock output in logs Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: optimize fuzz worker cache Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: warm fuzz target binaries Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: isolate fuzz workflow Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: centralize fuzz target preparation Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: split general workflows Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: streamline docs required checks Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: transfer fuzz targets as artifacts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve fuzz binary permissions Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: streamline fuzz workers Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: cache PR build dependencies Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: retain main build cache policy Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: address fuzz review feedback Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: add public constructor for compactor (#8724) Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(logging): add enable_file_logging option to disable file logging (#8721) Signed-off-by: xhwhis <hi@whis.me> Signed-off-by: onepizzateam <palakjha916@gmail.com> * avoid cloning final Prometheus remote write row (#8733) perf: avoid cloning final Prometheus remote write row Signed-off-by: lyang24 <lanqingy93@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(function): add json_object_keys scalar function (#8722) Expose JSON object key listing for outermost objects, with sqlness coverage. Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor(mito2): revise compaction trigger behavior (#8706) * refactor(mito2): revise compaction trigger behavior Distinguish automatic and manual triggers, coalesce explicit automatic follow-ups, and reject concurrent manual compactions. Remove implicit post-execution continuation and transient idle statuses so scheduler entries always represent an active lifecycle. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): track automatic compaction follow-ups Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): fix compaction transition rustdoc Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark manual compaction conflict retryable Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(mito2): drop unused RequestCancelResult::NotRunning variant request_cancel is only called in tests where the region is guaranteed to be running, so the NotRunning case was dead code. Simplify to unwrap() and remove the variant. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): gate test-only cancellation import Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize DDL after compaction planning Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: update dashboard to v0.13.11 (#8737) Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(object-store): skip removed-entry lister test on Windows (#8735) DirEntry on Windows is a snapshot from FindFirstFileW: file_type() and metadata() keep returning cached data after the file is removed, so read_list_entry() cannot observe the deletion. The test asserts the Unix behavior (lstat returns ENOENT) and fails deterministically on Windows nightly CI (4/4 tries). Gate it with #[cfg(not(windows))]. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(query): preserve remote dynamic filter target (#8615) * fix(query): preserve remote dynamic filter target Signed-off-by: discord9 <discord9@163.com> * fix(query): check RDF subscriber registration Signed-off-by: discord9 <discord9@163.com> * fix(query): refresh initial dyn filter snapshot before dispatch and handle RDF unregister The remote dynamic filter dispatch ordering regression: freeze the target, pre-register subscribers, refresh the initial snapshot, then dispatch. Also implement handle_remote_dyn_filter_unregister to keep unregister targets consistent with do_get/update. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): update test-only RegionQueryHandler impl to new trait signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * docs: rework README release badges, drop star history, fix grpc flag (#8743) * docs: show stable, latest and nightly version badges in README The single release badge rendered whatever GitHub considered newest, so a pre-release such as v1.2.0-beta.1 looked like the recommended version. Split it into three self-updating badges using the shields.io `filter` parameter, keyed off the existing tag naming: - stable: `!*-*` matches tags without a hyphen (v1.1.4) - latest: `!*-*-*` excludes nightly and dev builds (v1.2.0-beta.1) - nightly: `*-nightly-*` matches the weekly build (v1.2.0-nightly-20260706) No workflow changes are needed; the badges track new releases on their own. A one-line caption below them says which channel to pick. The release-date badge is dropped as the three version badges already carry that signal. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: remove star history chart from README The chart carried a sealed_token in three URLs and added a large third-party image to the Project Status section without saying anything the badges and case studies do not already cover. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: use --grpc-bind-addr in README quickstart --rpc-bind-addr is now only a hidden alias of --grpc-bind-addr and no longer shows up in --help. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * Update README.md Co-authored-by: Ning Sun <classicning@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Ning Sun <classicning@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore: check enterprise-gated files are listed in both license configs (#8750) * chore: check enterprise-gated files are listed in both license configs A file reachable only through `#[cfg(feature = "enterprise")] mod ...;` is governed by the GreptimeDB Enterprise License, so it must appear in the `includes` of licenserc-enterprise.toml and the `excludes` of licenserc.toml. hawkeye stays silent when it does not: the file keeps its Apache-2.0 header and passes the default check precisely because it was never excluded from it. scripts/check-enterprise-license.py walks enterprise-gated `mod` declarations, resolves them to files (submodules included) and diffs that set against both configs, also reporting stale entries. It runs in the license job in CI and as `make check-enterprise-license`. Documents the split it cannot decide for you — whole enterprise features get their own file, a gated match arm stays inline — in .agents/architecture-invariants.md. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: tighten enterprise license checks Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(operator): invalidate local cache after dropping view (#8748) Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore!: gate soft-drop table behind the enterprise feature (#8747) * chore: gate soft-drop table behind the enterprise feature Soft-drop table becomes an enterprise-only feature: - metasrv rejects gc.experimental_soft_drop.enable=true at startup in non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled without the enterprise feature as a second line of defense - the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table() registration, and information_schema.recycle_bin registration are compiled out unless the enterprise feature is enabled - common-meta procedures, tombstone keys, and DdlTask serde stay unconditional for persisted-procedure recovery and wire compatibility - the [gc.experimental_soft_drop] section is removed from the OSS example config and generated docs (moving to the enterprise repo) - the soft-drop sqlness cases and their CI job are removed from OSS (moving to the enterprise repo); affected information_schema .result files are regenerated Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: limit unused_variables allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch accidental unused variables in register_admin_only. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: include the config key in the soft-drop enterprise gate error Addresses review comment: name gc.experimental_soft_drop.enable in the startup validation error so users can locate the setting quickly when it is set via env vars or layered config. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: limit unused_mut allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch unused mut in the table_ddl_event test setup. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: reject soft-drop DDL submissions in non-enterprise builds Addresses review comment: clients could bypass the SQL-level gates by submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly to the procedure service. Reject fresh submissions at the DdlManager boundary in non-enterprise builds while keeping the procedure loaders registered for crash recovery and wire compatibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: stop --enable-gc from enabling soft drop in the sqlness template Addresses review comment: the metasrv test template rendered [gc.experimental_soft_drop] enable = true under the generic --enable-gc flag, which non-enterprise metasrv now rejects at startup, making the documented --enable-gc mode unusable in OSS. Keep the flag scoped to plain GC; enterprise soft-drop coverage moves to the enterprise repo. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate fresh soft-drop procedures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: gate soft-drop fallback coverage Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate soft-drop procedure implementation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate drop table soft-drop behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate expired soft-drop gc behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: test enterprise table ddl lifecycle Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark purge_table as enterprise licensed The purge_table module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark recycle_bin as enterprise licensed The recycle_bin module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark soft-drop procedure sources as enterprise licensed The purge and undrop procedure implementations plus the recycle-bin test module compile only with the enterprise feature. Apply the Enterprise License header and register them with both license configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: make frontend heartbeat extensible and lifecycle-safe (#8726) * feat: make frontend heartbeat extensible and lifecycle-safe Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: isolate heartbeat extension response handlers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: cancel in-flight heartbeat response handling Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover heartbeat wire compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up failed heartbeat startup Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address frontend heartbeat review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(meta): release region guards after drop rollback (#8751) Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor!: move native histogram config and `prom_validation_mode` to prom_store (#8744) * chore: adjust the position of experimental_enable_prometheus_native_histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: move prom_validation_mode as well Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: add health-aware gRPC client routing (#8684) * feat: add gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: harden gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: defer gRPC client health checks until first use Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(mito2): discard unflushed region data safely (#8600) * feat: support discarding unflushed region data Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): wake stalled writers after discard Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): drop redundant manifest check for discarding unflushed data Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * docs: align wal.sync_period documented default with actual fallback (5s) (#8753) The example TOMLs and generated config.md documented the default of wal.sync_period as "10s", but since #5677 moved the WAL sync task to a background RepeatedTask, an unset sync_period falls back to 5s in RaftEngineLogStore. The two paths therefore had different fsync periods: deployments based on the example configs used 10s while bare configs used 5s. Align the documentation with the actual code behavior (5s) instead of changing the code fallback to 10s, so that no existing deployment silently gets a larger data-loss window on host power loss. - config/datanode.example.toml, config/standalone.example.toml: 10s -> 5s - config/config.md: regenerated via make config-docs - src/cmd/tests/load_config_test.rs: update assertions accordingly Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: support Utf8View labels in Prometheus response (#8754) Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(query): validate merge scan remote schema (#8579) * fix(query): validate merge scan remote schema Signed-off-by: discord9 <discord9@163.com> * fix(query): treat JSON columns as schema-compatible across wire/decode forms CI (Sqlness json2_limit standalone + distributed) failed on the new remote-schema validation: a JSON column is Binary + extension metadata (ARROW:extension:name=greptime.json, greptime:type=Json) on the wire but decodes to Struct(...) with the extension metadata — validate_remote_schema compared raw arrow data_type and rejected it as a mismatch. Adds json_fields_compatible(): JSON fields are equal when name and nullability match, greptime:type matches, and the JSON2 settings (ARROW:extension:metadata type hints) match, ignoring the physical arrow type. Only JSON fields may bypass the raw-type comparison; non-JSON validation stays strict. Adds 4 regression tests mirroring the CI failure (wire-binary vs decoded-struct accepted both directions; different JSON2 settings rejected; JSON vs plain Binary rejected). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(grafana): add events dashboard (#8725) * feat(grafana): add events dashboard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): tolerate evolving event schemas Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): address events dashboard review Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): restore events dashboard panels Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): bound events dashboard queries Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): include historical event catalogs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): preserve events drill-down context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): correct events lifecycle outcomes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): handle empty event type ranges Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): scope event catalogs to submissions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): handle empty events dashboard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): refresh event schema variables Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor: separate a json2 extension type (#8745) Signed-off-by: luofucong <luofc@foxmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: add admin function registrar (#8762) * feat: add admin function registrar Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject admin function name collisions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: fix typo in admin function test Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: information_schema.rs table initialization issue Signed-off-by: onepizzateam <palakjha916@gmail.com> * rustfmt fix Signed-off-by: onepizzateam <palakjha916@gmail.com> --------- Signed-off-by: onepizzateam <palakjha916@gmail.com> Signed-off-by: Palak Jha <palakjha916@gmail.com> Signed-off-by: polar <palakjha916@gmail.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: discord9 <discord9@163.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: xhwhis <hi@whis.me> Signed-off-by: lyang24 <lanqingy93@gmail.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: luofucong <luofc@foxmail.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com> Co-authored-by: discord9 <discord9@163.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> Co-authored-by: Ning Sun <sunng@protonmail.com> Co-authored-by: Whis Liao <xhwhis@gmail.com> Co-authored-by: Lanqing Yang <lanqingy93@gmail.com> Co-authored-by: sun <sunchang_long@163.com> Co-authored-by: Ning Sun <classicning@gmail.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me> Co-authored-by: Yingwen <realevenyag@gmail.com> Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>	2026-08-06 07:24:59 +00:00
Weny Xu	8026064659	feat: add health-aware gRPC client routing (#8684) ... * feat: add gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: harden gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: defer gRPC client health checks until first use Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2026-08-05 06:16:51 +00:00
shuiyisong	aa72563783	refactor!: move native histogram config and prom_validation_mode to prom_store (#8744) ... * chore: adjust the position of experimental_enable_prometheus_native_histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: move prom_validation_mode as well Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-08-05 06:16:09 +00:00
jeremyhi	9f724aa5f6	feat: make frontend heartbeat extensible and lifecycle-safe (#8726) ... * feat: make frontend heartbeat extensible and lifecycle-safe Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: isolate heartbeat extension response handlers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: cancel in-flight heartbeat response handling Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover heartbeat wire compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up failed heartbeat startup Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address frontend heartbeat review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com>	2026-08-05 04:38:54 +00:00
Lei, HUANG	c55f297dec	chore!: gate soft-drop table behind the enterprise feature (#8747) ... * chore: gate soft-drop table behind the enterprise feature Soft-drop table becomes an enterprise-only feature: - metasrv rejects gc.experimental_soft_drop.enable=true at startup in non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled without the enterprise feature as a second line of defense - the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table() registration, and information_schema.recycle_bin registration are compiled out unless the enterprise feature is enabled - common-meta procedures, tombstone keys, and DdlTask serde stay unconditional for persisted-procedure recovery and wire compatibility - the [gc.experimental_soft_drop] section is removed from the OSS example config and generated docs (moving to the enterprise repo) - the soft-drop sqlness cases and their CI job are removed from OSS (moving to the enterprise repo); affected information_schema .result files are regenerated Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: limit unused_variables allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch accidental unused variables in register_admin_only. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: include the config key in the soft-drop enterprise gate error Addresses review comment: name gc.experimental_soft_drop.enable in the startup validation error so users can locate the setting quickly when it is set via env vars or layered config. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: limit unused_mut allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch unused mut in the table_ddl_event test setup. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: reject soft-drop DDL submissions in non-enterprise builds Addresses review comment: clients could bypass the SQL-level gates by submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly to the procedure service. Reject fresh submissions at the DdlManager boundary in non-enterprise builds while keeping the procedure loaders registered for crash recovery and wire compatibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: stop --enable-gc from enabling soft drop in the sqlness template Addresses review comment: the metasrv test template rendered [gc.experimental_soft_drop] enable = true under the generic --enable-gc flag, which non-enterprise metasrv now rejects at startup, making the documented --enable-gc mode unusable in OSS. Keep the flag scoped to plain GC; enterprise soft-drop coverage moves to the enterprise repo. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate fresh soft-drop procedures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: gate soft-drop fallback coverage Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate soft-drop procedure implementation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate drop table soft-drop behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate expired soft-drop gc behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: test enterprise table ddl lifecycle Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark purge_table as enterprise licensed The purge_table module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark recycle_bin as enterprise licensed The recycle_bin module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark soft-drop procedure sources as enterprise licensed The purge and undrop procedure implementations plus the recycle-bin test module compile only with the enterprise feature. Apply the Enterprise License header and register them with both license configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>	2026-08-05 04:28:30 +00:00
discord9	9b1e60511d	fix(query): preserve remote dynamic filter target (#8615) ... * fix(query): preserve remote dynamic filter target Signed-off-by: discord9 <discord9@163.com> * fix(query): check RDF subscriber registration Signed-off-by: discord9 <discord9@163.com> * fix(query): refresh initial dyn filter snapshot before dispatch and handle RDF unregister The remote dynamic filter dispatch ordering regression: freeze the target, pre-register subscribers, refresh the initial snapshot, then dispatch. Also implement handle_remote_dyn_filter_unregister to keep unregister targets consistent with do_get/update. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): update test-only RegionQueryHandler impl to new trait signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-08-04 13:56:34 +00:00
jeremyhi	448f973593	fix: sandbox SQL local filesystem access (#8708) ... * fix: sandbox SQL local filesystem access Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file sandbox review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support Windows local copy paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve sandbox path errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: simplify local path error context Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: stream secure filesystem listings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: derive local file access default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve local file access errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file access review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: simplify local file access coverage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: harden sandboxed local file backends Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject directory copy targets before creation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: avoid implicit string clone in file table listing Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com>	2026-07-31 13:23:15 +00:00
shuiyisong	00f0dcc663	refactor: replace endpoint permission variants with named actions (#8692) ... * refactor: perm enum Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: use typed actions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: pub structs Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-07-30 10:19:48 +00:00
shuiyisong	2afcd4a558	fix: enforce permissions for restricted HTTP endpoints (#8672) ... * chore: add perm check Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-07-29 08:51:17 +00:00
Lei, HUANG	1e9d70fa5c	feat: record metrics for timed out explain analyze (#8668) ... * feat: record metrics for timed out explain analyze Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: update event recorder test implementations Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep timeout metrics payload shape stable Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>	2026-07-29 06:47:40 +00:00
Ning Sun	deb688f572	feat: add a dedicated http api server port (#8657) ... * feat: add a dedicated http api server port * fix: integration test * refactor: make http-api-port opt-in * refactor: rename attribute to http-api-server * feat: use middleware to check different http server port * refactor: rename config option	2026-07-29 03:31:49 +00:00
Lei, HUANG	5ad4e71007	fix(prometheus): make remote write timeout retryable (#8639) ... * fix(prometheus): make remote write timeout retryable Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(prometheus): enforce pending row timeout budget Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(prometheus): skip pending-row timeout fallback when batcher is disabled PendingRowsBatcher::try_new returns None when max_batch_rows, max_concurrent_flushes, worker_channel_capacity or max_inflight_requests is zero, meaning remote writes bypass batching entirely. The timeout fallback predicate now mirrors these enablement conditions so the HTTP timeout is not raised when no request can wait for a pending-row flush. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(prometheus): skip pending-row timeout fallback in async batch mode With PENDING_ROWS_BATCH_SYNC=false, pending-row submissions return right after enqueue and no request waits for a flush, so raising the global HTTP timeout only delays unrelated routes. Export the batch sync mode predicate from the servers crate and consult it in the frontend's effective_http_options so the fallback is skipped in asynchronous mode. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>	2026-07-27 13:39:09 +00:00
Weny Xu	7344d47756	feat(event-recorder): configure lifecycle event recording (#8648) ... * refactor(event-recorder): centralize event table helpers Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(procedure): wire lifecycle event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(event-recorder): filter events by type Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): derive event type filter default Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): decouple frontend filtering Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove docs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): complete configuration support Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event-recorder): centralize filter ownership Signed-off-by: WenyXu <wenymedia@gmail.com> * test(config): update event recorder snapshot Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(frontend): decouple slow query event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2026-07-27 13:05:56 +00:00
shuiyisong	d9122ece3c	perf: optimize OTLP trace ingestion (#8604) ... perf: optimize trace ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-07-22 13:16:19 +00:00
Lei, HUANG	c8f65c7b99	feat: update flow windows after metric batch flush (#8544) ... * feat: update flow windows after metric batch flush Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: move batch rows into flow notifier Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: preserve timestamp index in metric batches Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: avoid blocking flow notification lookups Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: add context to timestamp extraction logs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: include peer in flow notification errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: compact flow notifications with time ranges Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: bound pending flow notification queue Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: send raw timestamps in flow notifications Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: update config API snapshot Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>	2026-07-22 13:09:30 +00:00
discord9andLei, HUANG	0564269426	fix: bind Prom remote read schema per query (#8591) ... Signed-off-by: discord9 <discord9@163.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>	2026-07-22 10:14:39 +00:00
shuiyisong	d4d9e199e7	fix: enforce table-aware permissions across query and ingest protocols (#8552) ... * feat: extend permission req check Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: preserve Prometheus query and write semantics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-07-21 12:15:53 +00:00
shuiyisong	5065dfcf2f	feat: grant creators access to newly created databases (#8566) ... * feat: implement auto database acl after create database Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: timeout conversion: Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove register loaders parameter in ddl_manager initialization Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-07-21 11:29:49 +00:00
Lei, HUANG	6a53bb053c	feat: enable soft-drop table lifecycle (#8554) ... * feat: add purge_table admin function Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: restrict purge_table to admin Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep purge tombstone lookup consistent Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: verify soft-drop table lifecycle Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: run soft-drop lifecycle sqlness Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: read purge tombstones authoritatively Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): invalidate soft-drop marker caches Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(meta): enable configurable table soft drop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(ci): configure gc for soft-drop sqlness Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): close regions before soft-drop metadata Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): avoid rollback after soft-drop tombstone Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): validate admin single-row calls Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): log purge dropped table target Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): mark soft-drop config experimental Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: sqlness test base Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>	2026-07-20 08:28:35 +00:00
Ruihang Xia	0479c05092	perf: preserve dictionary-encoded query labels (#8541) ... * perf: preserve dictionary-encoded query labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(client): skip dictionary Flight batches Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(servers): decode dictionary labels in HTTP output Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * test(mito2): support dictionary tags in series scans Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(servers): preserve dictionary child nulls in SQL Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(promql): compare dictionary tags by logical nulls Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): keep dictionary tags within query paths Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refactor(query): scope PK dictionary encoding to reads Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): preserve pushdown for dictionary labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): handle dictionary labels in query operators Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): complete label type matching Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): align dictionary query schemas Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): decode incompatible OR labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): preserve dictionary partition pruning Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: handle dictionary query edge cases Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: import dictionary downcast macro Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2026-07-17 13:57:10 +00:00
Lei, HUANG	bbf989ac36	feat: support soft-drop recycle bin and UNDROP TABLE (#8546) ... * feat: support full WAL retirement Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: complete close request migration Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: guard Kafka provider setup behind index collector check Move Kafka provider initialization and `get_or_insert` inside the existing `if let Some(collector)` block so these operations are skipped when no global index collector is configured. Affected file: - `src/log-store/src/kafka/log_store.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: avoid to_vec Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: clean up soft-dropped regions offline Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them. Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): reject file-engine soft drop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: preserve soft-drop cleanup split state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove obsolete CleanUp match arm from RegionRequest The `CleanUp` variant in the `region_request::Body` match is now handled exclusively by `RegionServer` via a separate path. This arm would have returned an unexpected error, so removing it eliminates dead code. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): clean every soft-dropped region replica Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): order soft-drop replica cleanup Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): order soft-drop replica cleanup" This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): clean every soft-dropped region replica" This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(catalog): expose soft drops in recycle bin Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(sql): add UNDROP TABLE Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(sql): cover UNDROP TABLE execution Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(sql): keep successful UNDROP result Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: reject stale undrop by name-based lookup after tombstone consumed When a table is dropped, recreated under the same name (consuming the name tombstone), and then the recreated table is dropped, an undrop procedure that was built before the first drop and holds a stale original table name should fail with TableNotFound instead of silently matching a different table. Changed `UndropTableProcedure::on_prepare` to perform a name-based lookup when `table_name` is available and filter by table ID, ensuring that a dropped table can only be recovered when its name tombstone still maps to the expected ID. - `src/common/meta/src/ddl/undrop_table.rs`: name-first lookup in on_prepare - `src/common/meta/src/ddl/tests/drop_table.rs`: test for the stale-id rejection case Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(log-store): keep Kafka obsolete_all as no-op Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(catalog): hide purging tables from recycle bin Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(catalog): scope recycle bin scans by catalog Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(sqlness): update recycle bin expectations Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: avoid redundant recycle bin allocations Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>	2026-07-17 08:15:21 +00:00
shuiyisong	3ae8d7d46a	refactor: reconcile OTLP trace schemas request-wide (#8485) ... * refactor: global trace schema Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: split files Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments and doc Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-07-16 10:56:27 +00:00
shuiyisong	589ae56a6f	fix: close database ACL gaps in permission checks (#8492) ... * fix: database ACL Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: reduce code Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-07-14 14:16:08 +00:00
discord9	b513dbaf4a	fix: reject datanode startup on GC config mismatch (#8509) ... * fix: reject datanode gc config mismatch Signed-off-by: discord9 <discord9@163.com> * refactor: minimize datanode gc startup check Signed-off-by: discord9 <discord9@163.com> * chore: update greptime-proto revision Signed-off-by: discord9 <discord9@163.com> * chore: use merged greptime-proto revision Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2026-07-14 12:48:13 +00:00
shuiyisong	7764d2f054	chore: make OTLP trace ingest chunk size configurable (#8455) ... * chore: expose trace chunk to config Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: change default value to 128 Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2026-07-09 08:22:40 +00:00
discord9	9936a83384	feat(query): add runtime provider interface (#8386) ... * feat(query): add runtime provider interface Signed-off-by: discord9 <discord9@163.com> * docs(query): document runtime provider interface Signed-off-by: discord9 <discord9@163.com> * feat(query): pass runtime builder to provider Signed-off-by: discord9 <discord9@163.com> * fix(query): make runtime provider fallible Signed-off-by: discord9 <discord9@163.com> * fix(query): propagate runtime provider errors Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2026-07-07 07:18:23 +00:00