mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-08-18 20:18:30 +00:00
feat/flow-eval-offset
97 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dennis zhuang
|
546625c45a |
feat: embedded convention pack for the entity graph (prom/k8s, gen_ai naming) (#8854)
* feat: embed the derivation conventions as data and adopt gen_ai entity naming Move the co-declared edge vocabulary, the agent-edge vocabulary and the virtual-destination candidates from Rust consts into an embedded conventions.yaml (include_str!), parsed once behind a LazyLock and validated against the entity-type grammar and the closed rel_type set; a broken file propagates as a plan error instead of panicking. The agent vocabulary entity types follow the GenAI semantic-convention namespace as written: gen_ai.agent / gen_ai.model / gen_ai.tool. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: drop the tag requirement for entity identity columns Entity declarations no longer require id columns to be tag/primary-key columns; only column existence is validated. Trace pipelines flatten the identifying attributes (span_attributes.gen_ai.agent.id, ...) into field columns, so the tag rule locked real trace tables out of declaring entities while buying no correctness — the read-time derivation works on any column. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: implicit declarations for well-known prometheus info metrics Tables stamped signal_type=metric + source=prometheus whose name matches the conventions.yaml whitelist (kube_pod_info, kube_node_info, kube_pod_owner, target_info) get implicit entity declarations: k8s.pod / k8s.node / k8s.workload with name-based identity and target_info's service / service.instance with the remaining tags as the descriptive snapshot. The existing co-declared vocabulary then derives runs_on and part_of from the same rows, so no new edge branch is needed. Explicit declarations of a type always suppress the implicit one, and the metric engine's physical table is excluded (it aggregates every logical table's columns). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover the prometheus conventions in sqlness and compact the graph cases Add the whitelisted-info-metric scenario (kube_pod_info, kube_pod_owner, target_info deriving runs_on / part_of, a non-whitelisted metric contributing nothing), fold the single-table calls, cross-table pairing and virtual-node cases into one trace scenario (they exercise the same union-before-join path), merge the two declaring-metric-table cases, and reuse one rename probe for both reserved names. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject entity id columns without a stable string form Review follow-ups: the DDL check now validates against the schema and rejects binary-backed and nested types for identity columns (the derivation renders ids via CAST to Utf8, so the failure used to surface only when the graph was scanned); the agent sqlness case keeps its identity columns as fields to cover the relaxed tag rule end to end; stale tag-rule comments and a dangling const reference are cleaned up. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: type-check every entity column role, not only ids The registry renders scope and descriptive values through the same CAST-to-string path as ids, so a binary-backed column in any role fails at scan time; the DDL check is now role-independent (and simpler). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: name the code-anchored vocabulary constants Entity types and edge attributes the derivation code itself anchors on (service, gen_ai.agent, calls, trace/attribute provenance) become constants in the conventions module; the rest of the vocabulary stays YAML-only data. ImplicitEntity is renamed PromImplicitEntity, and the implicit-declaration path logs each skip of a whitelisted info metric (wrong stamps, suppressed by an explicit declaration, missing id column) so a missing graph entity is diagnosable. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: single-source the graph constants The graph tables' column names move to common-catalog (the schemas catalog exposes and the plans operator builds must match column by column), and the conventions module now carries the complete built-in vocabulary — entity types, rel_types, provenances and connection types — with the embedded YAML validated by membership against it, so an edit drifting outside the vocabulary fails the conventions test instead of deriving nothing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: treat empty identity components as absent kube-state-metrics emits empty-string labels an entity id must not be built from: an unscheduled pod's node and an owner-less pod's owner_kind / owner_name. Standard Prometheus drops empty labels (they arrive as NULL and the existing predicate handles them), but other remote-write agents may keep them, which produced ghost entities with empty ids and false runs_on / part_of edges. Every identity predicate (registry, co-declared edges, span endpoints) now requires non-NULL and non-empty components through one shared helper. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: tighten the conventions DSL semantics Rename the co-declaration rule lists to what they are (co_declared_edges / trace_co_declared_edges — derivation rules, not a relation vocabulary), stop overstating the GenAI entity types (Greptime types derived from GenAI attributes; OTel defines no model/tool entities), move target_info's descriptive snapshot to service.instance (the remaining labels are the target's resource attributes, and instances would write conflicting snapshots onto the logical service), and extend the descriptor whitelist with the stable KSM sources: container info metrics (closing the k8s.pod contains k8s.container rule), kube_service_info (new k8s.service entity type) and the fuller descriptive label sets. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: guard entity column types on ALTER as well ALTER MODIFY COLUMN could change a declared entity column to a type without a stable string form, deferring the failure to graph scan time; verify_alter now checks the post-alter schema. Dropping a declared column stays allowed — the read-time derivation skips the stale declaration, and semantic options cannot be altered off yet. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: bridge traces and kube-state-metrics on the pod UID Trace-v1 tables now get implicit declarations from their flattened resource attributes (otlp_trace_entities in conventions.yaml): the service identity — replacing the hardcoded fallback — plus service.instance and k8s.pod, each applied only when its columns exist. A new co-declared rule derives service.instance runs_on k8s.pod, and the whitelisted kube-state-metrics pod identity switches from namespace+pod names to the UID, so the trace-side pod and every KSM descriptor land on one entity while names stay descriptive. This also removes pod identity from the multi-cluster same-name collision. The conventions rejection tests were passing for the wrong reason (a half-renamed fixture key failed deserialization before reaching any validation rule); they now assert the specific error each case targets. Sqlness covers the UID merge across descriptor tables, pod-contains- container, the k8s.service node, and the empty-uid/empty-node rows deriving nothing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover the OTLP-to-graph chain end to end One real OTLP export must come out of semantic_relationships as the zero-configuration chain: service calls service, instance part_of service, instance runs_on pod (bridged by k8s.pod.uid). Resources without service.instance.id or k8s.pod.uid derive nothing extra. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: identify k8s.service by UID Same reasoning as pods: a recreated same-name service must not merge into the old entity and same-named services across clusters must not collide; kube_service_info carries a stable uid and nothing joins on the service's name. Also drop a stale tag-rule mention from the option validation docs. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: cut duplicated test coverage and redundant comments The trace service-fallback test collapsed into the resource-entities test (same synthesis path since the fallback moved to YAML; only the invalid-explicit-no-fallback case was distinct), role-duplicate and subsumed DDL cases are gone, the embedded-conventions test is just the parse (its assertions were decorative), and the YAML section comments no longer restate the struct docs. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
335a95a369 |
feat: declared edges and the derivation contract for the entity graph (#8794)
* feat(frontend): run entity-graph derivation as the caller The derivation contract requires the computed graph tables to run under the outer query's identity. Capture the caller's QueryContext when the computed table is resolved, thread it through EntityGraphProvider, and: - authorize every contributing source table against the caller via the new semantic_graph.query permission action, silently excluding denied sources (entities, edges and source_tables never appear); - execute the derivation plan under the caller's context so it inherits permissions, cancellation and deadline instead of a fresh default. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(operator): derive the entity-graph window from the scan's time predicate Implements the RFC window contract for the computed graph tables: - table: add extract_time_range_strict, a strict variant of the lenient time-range extraction that distinguishes an absent observed_at filter from one that cannot be safely turned into a range; - operator: replace GraphWindow with GraphQueryWindow, splitting the queried observed_at range from the source-scan range widened to whole 60s buckets, so boundary buckets aggregate over their full extent; - frontend: resolve the window from ScanRequest filters — no predicate keeps the last-hour default, a missing upper bound means now, and a missing lower bound or unextractable shape is an explicit error, never a silent fallback. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(operator): system-defined declared-edge table for the entity graph Reintroduces greptime_private.semantic_relationships_declared with a canonical, system-owned definition: - the CREATE TABLE expr (8-tag primary key, business validity columns, RED fields, 30d TTL); attributes is now a json column so the future union branch matches the computed table without a per-scan parse; - created on first use on every write path: SQL INSERT creates it before executing, and the gRPC row-insert auto-create substitutes the canonical expr instead of deriving a schema from the request; - user DDL (CREATE/ALTER/DROP/RENAME/TRUNCATE) and write-path auto-ALTER are rejected via the new is_ddl_reserved_table guard, while INSERT/DELETE stay allowed. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(operator): union declared edges into semantic_relationships Adds the declared-edge branch to the relationship derivation (build_relationships_plan replaces build_calls_plan): - latest revision per edge key first (mito dedups on primary key plus observed_at, so a re-asserted edge stores a new revision), then the business-validity overlap against the queried window; valid_from defaults to the declaration time and a NULL valid_until means the edge holds while its row exists; - the projected observed_at is synthesized inside the queried range (Inexact pushdown re-applies the scan's filters above the computed table, which would drop rows keyed by the physical revision time); window_end/fresh_until of open-ended edges take the window's upper bound so 'fresh_until >= now() - ...' queries see them; - tag columns are cast out of dictionary encoding, and the union is re-projected to the 16-column contract; - the frontend feeds the branch only when the physical table exists, the caller may read it, and its schema still matches the canonical definition (mismatch is an explicit error, not a silent drop). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover declared edges, window contract and caller authorization - sqlness: system auto-create on first INSERT, latest-revision reads, open-ended vs retired validity, explicit/lower-only/upper-only window behavior, user-DDL rejection, rename-into rejection, DELETE cleanup; - integration: a permission checker denying one trace table excludes it from both semantic_relationships and semantic_entities. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: allow DROP/TRUNCATE on the declared-edge table and fix CI lints The definition guard rejected every DDL, which left sqlness (and any shared deployment) no way to remove the table the semantic_graph case creates — its extra region then broke unrelated region/partition case expectations. Narrow the guard to what actually protects the canonical definition: user CREATE, ALTER, RENAME-into and repartition stay rejected, while DROP and TRUNCATE are allowed — dropping loses nothing structural, the next INSERT recreates the table canonically, and DROP doubles as the recovery path if the canonical definition ever changes. The sqlness case now verifies drop-then-recreate and cleans up after itself. Also: rustfmt for the catalog crate and two typo fixes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: adapt canonical declared-table create to TriggerReason Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: address review on the declared-edge table lifecycle and revision reads - gRPC first writes actually work now: the reserved table's creation went through the generic create_table_inner, which the definition guard itself rejects; both branches of create_or_alter_tables_on_demand route it to create_declared_relationships_table instead, and being a system action it also bypasses the auto_create_table config/hint; - revision selection is as-of the queried window: revisions recorded after the window's end, or whose validity starts after it, no longer outrank (and hide) the revision that was in effect inside it; - the canonical-schema check validates the whole definition the union semantics lean on — time index, primary key, engine, append/merge mode — not just column names and types; - UNDROP TABLE of the reserved name is rejected like CREATE: it could resurrect a pre-canonical shape, and the next INSERT recreates the table anyway. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: trim over-commenting in the entity-graph code Comments that restated adjacent code or narrated justification are cut; the ones stating non-obvious contracts and gotchas stay. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject CREATE VIEW against DDL-reserved table names A view named greptime_private.semantic_relationships_declared would squat the reserved name: the first INSERT then skips the canonical create (an object already exists) and graph reads fail on the schema mismatch. CREATE VIEW now passes the same definition guard as CREATE TABLE. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: debug-log authorization exclusions; declared-edge TTL to 90d Sources the derivation contract silently excludes (per-table denial, whole-scan denial, the declared-edge table) are invisible from outside; a debug log at each names what was excluded and why. The declared-edge table's default TTL becomes 90d, overridable at creation time via GREPTIMEDB_DECLARED_RELATIONSHIPS_TTL (a proper configuration option is a TODO). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: rank declared-edge revisions by the visible edge identity Ranking partitioned by the full primary key, but the projection drops scope and generation_id: two assertions of the same visible edge under different generations both ranked first and came out as duplicate, indistinguishable rows. Rank by the exposed identity (endpoints, rel_type, provenance) instead, with generation_id/scope as deterministic tie-breakers for same-timestamp assertions. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: drop redundant declared-edge tests The generations regression is already asserted by the revision and as-of tests; the DDL shape test restated the declarative builder against itself. Its one non-tautological check (attributes maps to the json type) moves into the schema-matcher test. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject disjunctive graph windows and unmatchable future windows - OR/IN over observed_at collapse disjoint ranges into their convex hull; a declared edge's synthesized timestamp can land in a gap and be dropped by the re-applied filter even though the edge is valid at a requested instant. The strict extractor now rejects those shapes. - A lower bound in the future inverts against the implicit up-to-now upper bound; the declared branch then fabricated an edge observed at the future bound. Such windows now derive nothing. - The reserved-table gRPC create path classifies an instant-TTL table like every sibling path. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
3d12273c84 |
feat: read-time entity relationships graph over telemetry (M0+M1) (#8614)
* feat(table): add entity semantic declarations Define open-ended greptime.semantic.entity.* options, validate entity columns at DDL time, and stamp OTLP trace tables with the service entity declaration. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: add read-time entity relationships graph Add computed semantic graph tables, typed DataFusion derivation plans for entity registry and trace calls edges, and streaming read-time execution. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: exclude semantic graph tables from table constraints Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(operator): name the plan-builder source groupings Review feedback: build_registry_plan / build_calls_plan took anonymous (declarations, DataFrame) tuples while the caller already grouped the same fields. Introduce RegistrySource { declarations, scan } and CallsSource { service, scan } next to the builders and flow them through the frontend caller and tests. The frontend-side EntitySource keeps holding a TableRef (the operator builders stay pure over already-built scans), so the named structs live in operator rather than reusing that type. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Palak Jha
|
cabc2f6cc6 |
feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (#7987) (#8392)
* feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987) Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987) Signed-off-by: onepizzateam <palakjha916@gmail.com> * test(flow): add sqlness golden result for flow_status Signed-off-by: Palak Jha <palakjha916@gmail.com> * fix(catalog): remove unused OptionExt import in flow_statistics Signed-off-by: onepizzateam <palakjha916@gmail.com> * docs(flow): fix stale 'recent errors' comment on QueryFlowExecStats Signed-off-by: onepizzateam <palakjha916@gmail.com> * test: regenerate golden results for flow_statistics table Signed-off-by: onepizzateam <palakjha916@gmail.com> * style: apply rustfmt to flow_statistics changes Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor(catalog): hoist current_time_millis out of flow loop and clamp uptime Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore: remove accidentally committed fmt_check.log Signed-off-by: onepizzateam <palakjha916@gmail.com> * Update flow_status.result del eof trailing blank line as per review Signed-off-by: onepizzateam <palakjha916@gmail.com> * test(flow): restore runner-generated trailing blank line for sqlness Signed-off-by: onepizzateam <palakjha916@gmail.com> * postgres: include SHOW FLOW STATUS in extended-query describe (return flow_statistics fields) Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: address reviewer feedback on flow_statistics PR Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): resolve merge conflicts with main Signed-off-by: polar <palakjha916@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * sqlness check post gen (information_schema.result) Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): record start_time after req/snapshot_seqs built, before dispatch Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(sql): handle ShowFlowStatus in match statement at util.rs Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: review patch implementation Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore: remove accidentally committed local tool output files and fix fmt Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix worker.rs return type formatting Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): apply rustfmt to get_full_flow_stat return type Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix worker.rs return type formatting Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): re-apply rustfmt to get_full_flow_stat return type after merge Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: merge conflicts Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(auth): warn when credential load disables Postgres SCRAM or drops a line (#8652) * fix(auth): warn when credential load disables Postgres SCRAM or drops a line Static and watch user providers degraded silently in two ways: - A single non-SCRAM verifier (mysql_native_password, or a legacy pbkdf2_sha256 hash that predates SCRAM) disables Postgres SCRAM for every user and falls back to cleartext, with no signal to the operator. - A malformed credential line (commonly a plaintext password containing '=', which splits into more than two parts) was dropped without a trace. Emit a warning at each credential load for both cases so operators don't unknowingly serve cleartext passwords over Postgres or lose a user. This is logging only; authentication behavior is unchanged. The SCRAM check never logs secrets, and the malformed-line warning logs the line number and file, never the line content. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): warn on credential file read error before truncating A read error from lines() (I/O failure or invalid UTF-8) ends the iterator via map_while, silently dropping every remaining credential. Warn with the line number and file before truncating, matching the malformed-line handling, so the drop is observable. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * test(object-store): fix racy SecureFs abort test (#8720) test_writer_abort_is_unsupported_without_atomic_write asserted the file content immediately after abort() returned Unsupported. SecureFsWriter writes through tokio::fs::File, whose write_all() only enqueues a blocking write task (tokio's poll_write returns Ready before the write completes), so the data may not be visible yet when the test reads the file. Drop the race-prone content assertion and only verify the Unsupported contract. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(query): plan native histogram functions (#8705) * feat(query): plan native histogram functions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue & add tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * perf(promql): avoid repeated scans in sliding range evaluation (#8646) * perf(promql): use two pointers for sliding range boundaries Replace the stale cursor heuristic in RangeManipulateStream::calculate_range with monotonic left/right cursors. The old path rescanned each evaluation window (O(E x samples-per-window)) and could lose valid samples after sparse gaps or trailing empty windows. The two pointers keep strict monotonic progress, reducing boundary generation to O(N + E) while preserving (curr-range, curr] semantics, start/end shortening, and empty-window output. Controlled release benchmarks (fixed CPU, ABBA): - Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s, ~96% at 1h/15s. - Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end; shorter windows stayed within run-order noise. Signed-off-by: discord9 <discord9@163.com> * perf(promql): specialize changes/resets with adaptive edge counting The generic range_fn macro slices, downcasts, and rescans every overlapping window for changes() and resets(). Replace the macro path for these two functions with hand-written UDF wrappers backed by a shared private edge-count kernel: direct raw-offset scans when requested edges are few, otherwise one global u64 edge prefix so each window is answered by a prefix difference. Behavior is preserved bit-for-bit, including raw null-buffer values, NaN semantics, signed zero, infinities, empty/singleton windows, independent timestamp/value offsets, arbitrary window layouts, and exact DataFusion error messages. The shared proc macro, planner, serializer, and other range functions are untouched. Controlled release benchmarks (fixed CPU, ABBA): - Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time. - Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster. - Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client and 12.0-20.9% server latency improvement; controls stayed within drift. Signed-off-by: discord9 <discord9@163.com> * ci(query-regression): include PromQL range boundary case in defaults An audit of historical query-regression runs found zero range-query coverage: all 208 PromQL ANALYZE samples were bare selectors, so range evaluation could regress without CI noticing. Wire the promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so label-triggered runs measure the range path. The case is cheap: a ~0.3s synthetic fixture and about a minute of query execution per base/candidate pass. Signed-off-by: discord9 <discord9@163.com> * chore(promql): address sliding range review nits Move test-only imports into their test modules and remove the unused pre-specialization changes and resets helpers. Signed-off-by: discord9 <discord9@163.com> * style(promql): apply pinned rustfmt Signed-off-by: discord9 <discord9@163.com> * test(promql): cover sparse range results Share the changes and resets test scaffolding while keeping their behavior oracles independent. Add an end-to-end sqlness regression for sparse samples, empty intermediate windows, and a valid trailing sample. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * ci: optimize fuzz and split workflows (#8710) * ci: batch fuzz targets in GitHub Actions Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: improve fuzz test observability Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): preserve fuzz setup failure artifacts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(ci): keep fuzz mock output in logs Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: optimize fuzz worker cache Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: warm fuzz target binaries Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: isolate fuzz workflow Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: centralize fuzz target preparation Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: split general workflows Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: streamline docs required checks Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: transfer fuzz targets as artifacts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve fuzz binary permissions Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: streamline fuzz workers Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: cache PR build dependencies Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: retain main build cache policy Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: address fuzz review feedback Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: add public constructor for compactor (#8724) Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(logging): add enable_file_logging option to disable file logging (#8721) Signed-off-by: xhwhis <hi@whis.me> Signed-off-by: onepizzateam <palakjha916@gmail.com> * avoid cloning final Prometheus remote write row (#8733) perf: avoid cloning final Prometheus remote write row Signed-off-by: lyang24 <lanqingy93@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(function): add json_object_keys scalar function (#8722) Expose JSON object key listing for outermost objects, with sqlness coverage. Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor(mito2): revise compaction trigger behavior (#8706) * refactor(mito2): revise compaction trigger behavior Distinguish automatic and manual triggers, coalesce explicit automatic follow-ups, and reject concurrent manual compactions. Remove implicit post-execution continuation and transient idle statuses so scheduler entries always represent an active lifecycle. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): track automatic compaction follow-ups Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): fix compaction transition rustdoc Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark manual compaction conflict retryable Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(mito2): drop unused RequestCancelResult::NotRunning variant request_cancel is only called in tests where the region is guaranteed to be running, so the NotRunning case was dead code. Simplify to unwrap() and remove the variant. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): gate test-only cancellation import Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize DDL after compaction planning Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: update dashboard to v0.13.11 (#8737) Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(object-store): skip removed-entry lister test on Windows (#8735) DirEntry on Windows is a snapshot from FindFirstFileW: file_type() and metadata() keep returning cached data after the file is removed, so read_list_entry() cannot observe the deletion. The test asserts the Unix behavior (lstat returns ENOENT) and fails deterministically on Windows nightly CI (4/4 tries). Gate it with #[cfg(not(windows))]. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(query): preserve remote dynamic filter target (#8615) * fix(query): preserve remote dynamic filter target Signed-off-by: discord9 <discord9@163.com> * fix(query): check RDF subscriber registration Signed-off-by: discord9 <discord9@163.com> * fix(query): refresh initial dyn filter snapshot before dispatch and handle RDF unregister The remote dynamic filter dispatch ordering regression: freeze the target, pre-register subscribers, refresh the initial snapshot, then dispatch. Also implement handle_remote_dyn_filter_unregister to keep unregister targets consistent with do_get/update. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): update test-only RegionQueryHandler impl to new trait signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * docs: rework README release badges, drop star history, fix grpc flag (#8743) * docs: show stable, latest and nightly version badges in README The single release badge rendered whatever GitHub considered newest, so a pre-release such as v1.2.0-beta.1 looked like the recommended version. Split it into three self-updating badges using the shields.io `filter` parameter, keyed off the existing tag naming: - stable: `!*-*` matches tags without a hyphen (v1.1.4) - latest: `!*-*-*` excludes nightly and dev builds (v1.2.0-beta.1) - nightly: `*-nightly-*` matches the weekly build (v1.2.0-nightly-20260706) No workflow changes are needed; the badges track new releases on their own. A one-line caption below them says which channel to pick. The release-date badge is dropped as the three version badges already carry that signal. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: remove star history chart from README The chart carried a sealed_token in three URLs and added a large third-party image to the Project Status section without saying anything the badges and case studies do not already cover. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: use --grpc-bind-addr in README quickstart --rpc-bind-addr is now only a hidden alias of --grpc-bind-addr and no longer shows up in --help. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * Update README.md Co-authored-by: Ning Sun <classicning@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Ning Sun <classicning@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore: check enterprise-gated files are listed in both license configs (#8750) * chore: check enterprise-gated files are listed in both license configs A file reachable only through `#[cfg(feature = "enterprise")] mod ...;` is governed by the GreptimeDB Enterprise License, so it must appear in the `includes` of licenserc-enterprise.toml and the `excludes` of licenserc.toml. hawkeye stays silent when it does not: the file keeps its Apache-2.0 header and passes the default check precisely because it was never excluded from it. scripts/check-enterprise-license.py walks enterprise-gated `mod` declarations, resolves them to files (submodules included) and diffs that set against both configs, also reporting stale entries. It runs in the license job in CI and as `make check-enterprise-license`. Documents the split it cannot decide for you — whole enterprise features get their own file, a gated match arm stays inline — in .agents/architecture-invariants.md. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: tighten enterprise license checks Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(operator): invalidate local cache after dropping view (#8748) Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore!: gate soft-drop table behind the enterprise feature (#8747) * chore: gate soft-drop table behind the enterprise feature Soft-drop table becomes an enterprise-only feature: - metasrv rejects gc.experimental_soft_drop.enable=true at startup in non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled without the enterprise feature as a second line of defense - the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table() registration, and information_schema.recycle_bin registration are compiled out unless the enterprise feature is enabled - common-meta procedures, tombstone keys, and DdlTask serde stay unconditional for persisted-procedure recovery and wire compatibility - the [gc.experimental_soft_drop] section is removed from the OSS example config and generated docs (moving to the enterprise repo) - the soft-drop sqlness cases and their CI job are removed from OSS (moving to the enterprise repo); affected information_schema .result files are regenerated Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: limit unused_variables allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch accidental unused variables in register_admin_only. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: include the config key in the soft-drop enterprise gate error Addresses review comment: name gc.experimental_soft_drop.enable in the startup validation error so users can locate the setting quickly when it is set via env vars or layered config. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: limit unused_mut allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch unused mut in the table_ddl_event test setup. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: reject soft-drop DDL submissions in non-enterprise builds Addresses review comment: clients could bypass the SQL-level gates by submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly to the procedure service. Reject fresh submissions at the DdlManager boundary in non-enterprise builds while keeping the procedure loaders registered for crash recovery and wire compatibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: stop --enable-gc from enabling soft drop in the sqlness template Addresses review comment: the metasrv test template rendered [gc.experimental_soft_drop] enable = true under the generic --enable-gc flag, which non-enterprise metasrv now rejects at startup, making the documented --enable-gc mode unusable in OSS. Keep the flag scoped to plain GC; enterprise soft-drop coverage moves to the enterprise repo. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate fresh soft-drop procedures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: gate soft-drop fallback coverage Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate soft-drop procedure implementation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate drop table soft-drop behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate expired soft-drop gc behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: test enterprise table ddl lifecycle Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark purge_table as enterprise licensed The purge_table module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark recycle_bin as enterprise licensed The recycle_bin module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark soft-drop procedure sources as enterprise licensed The purge and undrop procedure implementations plus the recycle-bin test module compile only with the enterprise feature. Apply the Enterprise License header and register them with both license configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: make frontend heartbeat extensible and lifecycle-safe (#8726) * feat: make frontend heartbeat extensible and lifecycle-safe Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: isolate heartbeat extension response handlers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: cancel in-flight heartbeat response handling Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover heartbeat wire compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up failed heartbeat startup Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address frontend heartbeat review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(meta): release region guards after drop rollback (#8751) Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor!: move native histogram config and `prom_validation_mode` to prom_store (#8744) * chore: adjust the position of experimental_enable_prometheus_native_histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: move prom_validation_mode as well Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: add health-aware gRPC client routing (#8684) * feat: add gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: harden gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: defer gRPC client health checks until first use Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(mito2): discard unflushed region data safely (#8600) * feat: support discarding unflushed region data Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): wake stalled writers after discard Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): drop redundant manifest check for discarding unflushed data Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * docs: align wal.sync_period documented default with actual fallback (5s) (#8753) The example TOMLs and generated config.md documented the default of wal.sync_period as "10s", but since #5677 moved the WAL sync task to a background RepeatedTask, an unset sync_period falls back to 5s in RaftEngineLogStore. The two paths therefore had different fsync periods: deployments based on the example configs used 10s while bare configs used 5s. Align the documentation with the actual code behavior (5s) instead of changing the code fallback to 10s, so that no existing deployment silently gets a larger data-loss window on host power loss. - config/datanode.example.toml, config/standalone.example.toml: 10s -> 5s - config/config.md: regenerated via make config-docs - src/cmd/tests/load_config_test.rs: update assertions accordingly Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: support Utf8View labels in Prometheus response (#8754) Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(query): validate merge scan remote schema (#8579) * fix(query): validate merge scan remote schema Signed-off-by: discord9 <discord9@163.com> * fix(query): treat JSON columns as schema-compatible across wire/decode forms CI (Sqlness json2_limit standalone + distributed) failed on the new remote-schema validation: a JSON column is Binary + extension metadata (ARROW:extension:name=greptime.json, greptime:type=Json) on the wire but decodes to Struct(...) with the extension metadata — validate_remote_schema compared raw arrow data_type and rejected it as a mismatch. Adds json_fields_compatible(): JSON fields are equal when name and nullability match, greptime:type matches, and the JSON2 settings (ARROW:extension:metadata type hints) match, ignoring the physical arrow type. Only JSON fields may bypass the raw-type comparison; non-JSON validation stays strict. Adds 4 regression tests mirroring the CI failure (wire-binary vs decoded-struct accepted both directions; different JSON2 settings rejected; JSON vs plain Binary rejected). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(grafana): add events dashboard (#8725) * feat(grafana): add events dashboard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): tolerate evolving event schemas Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): address events dashboard review Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): restore events dashboard panels Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): bound events dashboard queries Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): include historical event catalogs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): preserve events drill-down context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): correct events lifecycle outcomes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): handle empty event type ranges Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): scope event catalogs to submissions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): handle empty events dashboard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): refresh event schema variables Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor: separate a json2 extension type (#8745) Signed-off-by: luofucong <luofc@foxmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: add admin function registrar (#8762) * feat: add admin function registrar Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject admin function name collisions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: fix typo in admin function test Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: information_schema.rs table initialization issue Signed-off-by: onepizzateam <palakjha916@gmail.com> * rustfmt fix Signed-off-by: onepizzateam <palakjha916@gmail.com> --------- Signed-off-by: onepizzateam <palakjha916@gmail.com> Signed-off-by: Palak Jha <palakjha916@gmail.com> Signed-off-by: polar <palakjha916@gmail.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: discord9 <discord9@163.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: xhwhis <hi@whis.me> Signed-off-by: lyang24 <lanqingy93@gmail.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: luofucong <luofc@foxmail.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com> Co-authored-by: discord9 <discord9@163.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> Co-authored-by: Ning Sun <sunng@protonmail.com> Co-authored-by: Whis Liao <xhwhis@gmail.com> Co-authored-by: Lanqing Yang <lanqingy93@gmail.com> Co-authored-by: sun <sunchang_long@163.com> Co-authored-by: Ning Sun <classicning@gmail.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me> Co-authored-by: Yingwen <realevenyag@gmail.com> Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com> |
||
 Lei, HUANG
|
bbf989ac36 |
feat: support soft-drop recycle bin and UNDROP TABLE (#8546)
* feat: support full WAL retirement Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: complete close request migration Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: guard Kafka provider setup behind index collector check Move Kafka provider initialization and `get_or_insert` inside the existing `if let Some(collector)` block so these operations are skipped when no global index collector is configured. Affected file: - `src/log-store/src/kafka/log_store.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: avoid to_vec Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: clean up soft-dropped regions offline Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them. Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): reject file-engine soft drop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: preserve soft-drop cleanup split state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove obsolete CleanUp match arm from RegionRequest The `CleanUp` variant in the `region_request::Body` match is now handled exclusively by `RegionServer` via a separate path. This arm would have returned an unexpected error, so removing it eliminates dead code. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): clean every soft-dropped region replica Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): order soft-drop replica cleanup Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): order soft-drop replica cleanup" This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): clean every soft-dropped region replica" This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(catalog): expose soft drops in recycle bin Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(sql): add UNDROP TABLE Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(sql): cover UNDROP TABLE execution Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(sql): keep successful UNDROP result Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: reject stale undrop by name-based lookup after tombstone consumed When a table is dropped, recreated under the same name (consuming the name tombstone), and then the recreated table is dropped, an undrop procedure that was built before the first drop and holds a stale original table name should fail with TableNotFound instead of silently matching a different table. Changed `UndropTableProcedure::on_prepare` to perform a name-based lookup when `table_name` is available and filter by table ID, ensuring that a dropped table can only be recovered when its name tombstone still maps to the expected ID. - `src/common/meta/src/ddl/undrop_table.rs`: name-first lookup in on_prepare - `src/common/meta/src/ddl/tests/drop_table.rs`: test for the stale-id rejection case Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(log-store): keep Kafka obsolete_all as no-op Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(catalog): hide purging tables from recycle bin Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(catalog): scope recycle bin scans by catalog Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(sqlness): update recycle bin expectations Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: avoid redundant recycle bin allocations Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
e403133eb2 |
feat: add information_schema statistics table (#8253)
* feat: add information_schema statistics table Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: use index-local sequence in statistics Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: ordinal_position for pk Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: statistics.nullable uses empty string for non-nullable columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
d6c37778ae |
feat: table semantic layer information_schema view (Phase 3) (#8240)
* feat: table semantic layer information_schema view (Phase 3) Add `information_schema.table_semantics`, a queryable view over the table semantic layer. One row per table that carries at least one `greptime.semantic.*` option: the signal-agnostic keys (signal_type/source/pipeline/metadata_quality) are promoted to columns and the remaining signal-specific keys are folded into a `semantic_options` JSON string. Tables with no semantic key are excluded. Stacked on Phase 2. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: address PR review on table_semantics - fold JSON serialization failure into None instead of unwrap/panic - drop per-row Vec allocation in predicate eval; use a fixed array - align RFC view name with the shipped `table_semantics` Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: update results Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
9a4e5e8457 |
chore: expose region info inspection table (#8178)
* chore/region-sync-diff: add region info inspection core - `store-api`: add `RegionInfoEntry` schema and plan builder in `src/store-api/src/region_info.rs` and export it from `src/store-api/src/lib.rs` - `mito2`: collect region runtime metadata with `MitoEngine::all_region_infos` and `RegionRoleState::as_str` in `src/mito2/src/engine.rs`, `src/mito2/src/region.rs`, `src/mito2/src/engine/basic_test.rs`, `src/mito2/Cargo.toml`, and `Cargo.lock` - `datanode`: expose the reserved `InspectRegionInfo` provider in `src/datanode/src/region_server/catalog.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: expose region info schema table - `information_schema.region_info`: add frontend table wiring in `src/catalog/src/system_schema/information_schema.rs`, `src/catalog/src/system_schema/information_schema/region_info.rs`, `src/catalog/src/system_schema/information_schema/table_names.rs`, and `src/common/catalog/src/consts.rs` - `region_group` removal: drop `region_group` from `src/store-api/src/region_info.rs`, `src/mito2/src/region.rs`, and `src/mito2/src/engine/basic_test.rs` - `SQLness coverage`: add standalone coverage in `tests/cases/standalone/common/information_schema/region_info.sql` and `tests/cases/standalone/common/information_schema/region_info.result` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: restore region group info - `region_info` schema: restore `region_group` alongside `region_sequence` in `src/store-api/src/region_info.rs`, `src/mito2/src/region.rs`, `src/mito2/src/engine/basic_test.rs`, and `tests/cases/standalone/common/information_schema/region_info.result` - `MitoEngine::all_region_infos`: remove redundant iterator conversion in `src/mito2/src/engine.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sqlness Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sqlness Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: clarify region sequence columns - `region_info` schema: rename `sequence` to `committed_sequence` and add nullable `flushed_sequence` in `src/store-api/src/region_info.rs` and `src/mito2/src/region.rs` - `region_info` coverage: update sequence assertions and expected metadata in `src/mito2/src/engine/basic_test.rs`, `tests/cases/standalone/common/information_schema/region_info.sql`, `tests/cases/standalone/common/information_schema/region_info.result`, and `tests/cases/standalone/common/system/information_schema.result` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: report region options errors - `region_info` output: preserve `region_options` serialization failures as JSON error objects in `src/mito2/src/region.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Lei, HUANG
|
f5c1d5d9bc |
fix: preserve case in database name from connection string (#8062)
`parse_optional_catalog_and_schema_from_db_string` unconditionally lowercased database/schema names, causing quoted database names (e.g. `CREATE DATABASE "TestQuery"`) to be stored with preserved case but looked up as lowercase on connection, resulting in "Database not found". Fixes #8059 Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
 fys
|
d180cc8f4b |
chore: add INFORMATION_SCHEMA_ALERTS_TABLE_ID const value (#7288)
|
||
 WaterWhisperer
|
de9ae6066f |
refactor: remove export_metrics and related configuration (#7236)
Signed-off-by: WaterWhisperer <waterwhisperer24@qq.com> |
||
 shuiyisong
|
11c0381fc1 |
chore: set default catalog using build env (#7156)
* chore: update reference to const Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: use option_env to set default catalog Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: use const_format Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update reference in cli Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: introduce a build.rs to set default catalog Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove unused feature gate Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 zyy17
|
0a3961927d |
refactor!: add a opentelemetry_traces_operations table to aggregate (service_name, span_name, span_kind) to improve query performance (#7144)
refactor: add a `*_operations` table to aggregate `(service_name, span_name, span_kind)` to improve query performance Signed-off-by: zyy17 <zyylsxm@gmail.com> |
||
 Zhenchi
|
938d757523 |
feat: expose SST index metadata via information schema (#7044)
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> |
||
 Ning Sun
|
964dc254aa |
feat: upgraded pg_catalog support (#6918)
* refactor: add datafusion-postgres dependency * refactor: move and include pg_catalog udfs * chore: update upstream * feat: register table function pg_get_keywords * feat: bridge CatalogInfo for our CatalogManager Signed-off-by: Ning Sun <sunning@greptime.com> * feat: convert pg_catalog table to our system table * feat: bridge system catalog with datafusion-postgres Signed-off-by: Ning Sun <sunning@greptime.com> * feat: add more udfs * feat: add compatibility rewriter to postgres handler * fix: various fix * fmt: fix * fix: use functions from pg_catalog library * fmt * fix: sqlness runner Signed-off-by: Ning Sun <sunning@greptime.com> * test: adopt arrow 56.0 to 56.1 memory size change * fix: add additional udfs * chore: format * refactor: return None when creating system table failed Signed-off-by: Ning Sun <sunning@greptime.com> * chore: provide safety comments about expect usage --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Zhenchi
|
80c8ab42b0 |
feat: add ssts releated system table (#6924)
* feat: add InformationExtension.inspect_datanode for datanode inspection Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * aggregate results from all datanodes Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix fmt Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * feat: add ssts releated system table Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * update sst entry Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix sqlness Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix sqlness Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> |
||
 LFC
|
370d27587a |
refactor: make greptimedb's tests ran as a submodule (#6544)
fix: failed to run a test when as a submodule Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
fys
|
c5360601f5 |
feat: information table extension (#6434)
* feat: information table extension * avoid use std HashMap behind cfg feature |
||
|
Lei, HUANG
|
05b708ed2e |
feat: implement process manager and information_schema.process_list (#5865)
* ### Add Process List Management - **Error Handling Enhancements**: * refactor: Update test IP addresses to include ports in ProcessKey * feat/show-process-list: Refactor Process Management in Meta Module - Introduced `ProcessManager` for handling process registration and deregistration. - Added methods for managing and querying process states, including `register_query`, `deregister_query`, and `list_all_processes`. - Removed redundant process management code from the query module. - Updated error handling to reflect changes in process management. - Enhanced test coverage for process management functionalities. * chore: rebase main * add information schema process list table * integrate process list table to system catalog * build ProcessManager on frontend and standalone mode * feat/show-process-list: **Add Process Management Enhancements** - **`manager.rs`**: Introduced `process_manager` to `SystemCatalog` and `KvBackendCatalogManager` for improved process handling. - **`information_schema.rs`**: Updated table insertion logic to conditionally include `PROCESS_LIST`. - **`frontend.rs`, `standalone.rs`**: Enhanced `StartCommand` to clone `process_manager` for better resource management. - **`instance.rs`, `builder.rs`**: Integrated `ProcessManager` into `Instance` and `FrontendBuilder` to manage query * feat/show-process-list: ### Add Process Listing and Error Handling Enhancements - **Error Handling**: Introduced a new error variant `ListProcess` in `error.rs` to handle failures when listing running processes. - **Process List Implementation**: Enhanced `InformationSchemaProcessList` in `process_list.rs` to track running queries, including defining column names and implementing the `make_process_list` function to build the process list. - **Frontend Builder**: Added a `#[allow(clippy::too_many_arguments)]` attribute in `builder.rs` to suppress Clippy warnings for the `FrontendBuilder::new` function. These changes improve error handling and process tracking capabilities within the system. * feat/show-process-list: Refactor imports in `process_list.rs` - Updated import paths for `Predicates` and `InformationTable` in `process_list.rs` to align with the new module structure. * feat/show-process-list: Refactor process list generation in `process_list.rs` - Simplified the process list generation by removing intermediate row storage and directly building vectors. - Updated `process_to_row` function to use a mutable vector for current row data, improving memory efficiency. - Removed `rows_to_record_batch` function, integrating its logic directly into the main loop for streamlined processing. * wip: move ProcessManager to catalog crate * feat/show-process-list: - **Refactor Row Construction**: Updated row construction in multiple files to use references for `Value` objects, improving memory efficiency. Affected files include: - `cluster_info.rs` - `columns.rs` - `flows.rs` - `key_column_usage.rs` - `partitions.rs` - `procedure_info.rs` - `process_list.rs` - `region_peers.rs` - `region_statistics.rs` - `schemata.rs` - `table_constraints.rs` - `tables.rs` - `views.rs` - `pg_class.rs` - `pg_database.rs` - `pg_namespace.rs` - **Remove Unused Code**: Deleted unused functions and error variants related to process management in `process_list.rs` and `error.rs`. - **Predicate Evaluation Update**: Modified predicate evaluation functions in `predicate.rs` to work with references, enhancing performance. * feat/show-process-list: ### Implement Process Management Enhancements - **Error Handling Enhancements**: - Added new error variants `BumpSequence`, `StartReportTask`, `ReportProcess`, and `BuildProcessManager` in `error.rs` to improve error handling for process management tasks. - Updated `ErrorExt` implementations to handle new error types. - **Process Manager Improvements**: - Introduced `ProcessManager` enhancements in `process_manager.rs` to manage process states using `ProcessWithState` and `ProcessState` enums. - Implemented periodic task `ReportTask` to report running queries to the KV backend. - Modified `register_query` and `deregister_query` methods to use the new state management system. - **Testing and Validation**: - Updated tests in `process_manager.rs` to validate new process management logic. - Replaced `dump` method with `list_all_processes` for listing processes. - **Integration with Frontend and Standalone**: - Updated `frontend.rs` and `standalone.rs` to handle `ProcessManager` initialization errors using `BuildProcessManager` error variant. - **Schema Adjustments**: - Modified `process_list.rs` in `system_schema/information_schema` to use the updated process listing method. - **Key-Value Conversion**: - Added `TryFrom` implementation for converting `Process` to `KeyValue` in `process_list.rs`. * chore: remove register * fix: sqlness tests * merge main Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: - **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency in `Cargo.lock` and `Cargo.toml` to a new revision. - **Refactor `ProcessManager`**: Simplified the `ProcessManager` implementation by removing the use of `KvBackendRef` and `SequenceRef`, and replaced them with `AtomicU64` and `RwLock` for managing process IDs and catalogs in `process_manager.rs`. - **Remove Process List Metadata**: Deleted the `process_list.rs` file and removed related metadata key definitions in `key.rs`. - **Update Process List Logic**: Modified the process list logic in `process_list.rs` to use the new `ProcessManager` structure. - **Adjust Frontend and Standalone Start Commands**: Updated `frontend.rs` and `standalone.rs` to use the new `ProcessManager` constructor. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: - **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency version in `Cargo.lock` and `Cargo.toml` to a new commit hash. - **Refactor Error Handling**: Removed unused error variants and added a new `ParseProcessId` error in `src/catalog/src/error.rs`. - **Enhance Process Management**: Introduced `DisplayProcessId` struct for better process ID representation and parsing in `src/catalog/src/process_manager.rs`. - **Revise Process List Schema**: Updated the schema and logic for process listing in `src/catalog/src/system_schema/information_schema/process_list.rs` to include new fields like `client` and `frontend`. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Commit Message **Enhancements and Refactoring** - **Process Management:** - Refactored `ProcessManager` to list local processes with an optional catalog filter in `process_manager.rs`. - Updated related tests in `process_manager.rs` and `process_list.rs`. - **Client Enhancements:** - Added `frontend_client` method in `client.rs` to support gRPC communication with the frontend. - **Error Handling:** - Extended error handling in `error.rs` to include gRPC and Meta errors. - **Frontend Module:** - Introduced `selector.rs` for frontend client selection and process listing. - Updated `Cargo.toml` to include new dependencies and dev-dependencies. - **gRPC Server:** - Integrated `FrontendServer` in `builder.rs` for enhanced gRPC server capabilities. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Commit Message **Refactor Process Management and Frontend Integration** - **Add `common-frontend` Dependency**: - Updated `Cargo.lock`, `Cargo.toml` files to include `common-frontend` as a dependency. - **Refactor Process Management**: - Moved `ProcessManager` trait and `DisplayProcessId` struct to `common-frontend`. - Updated `process_manager.rs` to use `MetaProcessManager` and `ProcessManagerRef`. - Removed `ParseProcessId` error variant from `error.rs` in `catalog` and `frontend`. - **Frontend gRPC Service**: - Added `frontend_grpc_handler.rs` to handle gRPC requests for frontend processes. - Updated `grpc.rs` and `builder.rs` to integrate `FrontendGrpcHandler`. - **Update Tests**: - Modified tests in `process_manager.rs` to align with new `ProcessManager` implementation. - **Remove Unused Code**: - Removed `DisplayProcessId` and related parsing logic from `process_manager.rs`. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add `MetaClientRef` to `MetaProcessManager` and Update Instantiation - **Files Modified**: - `src/catalog/src/process_manager.rs` - `src/cmd/src/frontend.rs` - `src/cmd/src/standalone.rs` - **Key Changes**: - Added `MetaClientRef` as an optional parameter to the `MetaProcessManager::new` method. - Updated instantiation of `MetaProcessManager` to include `MetaClientRef` where applicable. ### Update `ProcessManagerRef` Usage - **Files Modified**: - `src/catalog/src/kvbackend/manager.rs` - `src/catalog/src/system_schema/information_schema.rs` - `src/catalog/src/system_schema/information_schema/process_list.rs` - `src/frontend/src/instance.rs` - `src/frontend/src/instance/builder.rs` - **Key Changes**: - Ensured consistent usage of `ProcessManagerRef` across various modules. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ## Refactor Process Management - **Unified Process Manager**: - Replaced `MetaProcessManager` with `ProcessManager` across the codebase. - Updated `ProcessManager` to use `Arc` for shared references and introduced a `Ticket` struct for query registration and deregistration. - Affected files: `manager.rs`, `process_manager.rs`, `frontend.rs`, `standalone.rs`, `frontend_grpc_handler.rs`, `instance.rs`, `builder.rs`, `cluster.rs`, `standalone.rs`. - **Stream Wrapper Implementation**: - Added `StreamWrapper` to handle record batch streams with process management. - Affected file: `stream_wrapper.rs`. - **Test Adjustments**: - Updated tests to align with the new `ProcessManager` implementation. - Affected file: `tests-integration/src/cluster.rs`, `tests-integration/src/standalone.rs`. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add Error Handling and Process Management - **Error Handling Enhancements**: - Added new error variants `ListProcess` and `CreateChannel` in `error.rs` to handle specific gRPC service invocation failures. - Updated error handling in `selector.rs` to use the new error variants for better context and error propagation. - **Process Management Integration**: - Introduced `process_manager` method in `instance.rs` to access the process manager. - Integrated `FrontendGrpcHandler` with process management in `server.rs` to handle gRPC requests related to process management. - **gRPC Server Enhancements**: - Made `frontend_grpc_handler` public in `grpc.rs` to allow external access and integration with other modules. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: Update `greptime-proto` dependency and enhance process management - **Dependency Update**: Updated `greptime-proto` in `Cargo.lock` and `Cargo.toml` to a new revision. - **Process Management**: - Modified `process_manager.rs` to include catalog filtering in `list_process`. - Updated `frontend_grpc_handler.rs` to handle catalog filtering in `list_process` requests. - **System Schema**: Added a TODO comment in `process_list.rs` for future user catalog filtering implementation. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: - **Update Workspace Dependencies**: - Modified `Cargo.toml` files in `src/catalog`, `src/common/frontend`, and `src/servers` to adjust workspace dependencies. - **Refactor `ProcessManager` Logic**: - Updated `process_manager.rs` to simplify the condition in the `select` method. - **Remove Unused Error Variants**: - Deleted `BuildProcessManager` error variant from `error.rs` in `src/cmd`. - Removed `InvalidProcessKey` error variant from `error.rs` in `src/common/meta`. - **Add License Header**: - Added Apache License header to `stream_wrapper.rs` in `src/frontend`. - **Update Test Results**: - Adjusted expected results in `information_schema.result` to reflect changes in the schema. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add Error Handling for Process Listing - **`src/catalog/src/error.rs`**: Introduced a new error variant `ListProcess` to handle failures in listing frontend nodes. - **`src/catalog/src/process_manager.rs`**: Updated `local_processes` and `list_all_processes` methods to return the new error type, adding context for error handling. - **`src/catalog/src/system_schema/information_schema/process_list.rs`**: Modified `make_process_list` to propagate errors using the new error handling mechanism. - **`src/servers/src/grpc/frontend_grpc_handler.rs`**: Enhanced error handling in the `list_process` method to log errors and return appropriate gRPC status codes. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: Update `greptime-proto` Dependency and Remove `frontend_client` Method - **Cargo.lock** and **Cargo.toml**: Updated the `greptime-proto` dependency to a new revision (`5f6119ac7952878d39dcde0343c4bf828d18ffc8`). - **src/client/src/client.rs**: Removed the `frontend_client` method from the `Client` implementation. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add Query Registration with Pre-Generated ID - **`process_manager.rs`**: Introduced `register_query_with_id` method to allow registering queries with a pre-generated ID. This includes creating a `ProcessInfo` instance and inserting it into the catalog. Added `next_id` method to generate the next process ID. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Update Process List Retrieval Method - **File**: `process_list.rs` - Updated the method for retrieving process lists from `local_processes` to `list_all_processes` to support asynchronous operations. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Update error handling in `error.rs` - Refined status code handling for `CreateChannel` error by delegating to `source.status_code()`. - Separated `ListProcess` and `CreateChannel` error handling for clarity. Signed-off-by: Lei, HUANG <lhuang@greptime.com> --------- Signed-off-by: Lei, HUANG <lhuang@greptime.com> |
||
|
zyy17
|
ee4fe9d273 |
refactor: improve performance for Jaeger APIs (#5838)
* refactor: improve jaeger '/api/services' performance by adding the trace services table * chore: refine some logic * chore: compatible v0 * test: add integration test * chore: expand default limit from 100 to 2000 * test: fix integration test * refactor: make trace service table configurable * refactor: use a timestamp(2100-01-01 00:00:00) as large as possible * refactor: use '<trace_table>_services' as trace services table name |
||
|
Ning Sun
|
2260782c12 |
refactor: update jaeger api implementation for new trace modeling (#5655)
* refactor: update jaeger api implementation * test: add tests for v1 data model * feat: customize trace table name * fix: update column requirements to use Column type instead of String Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: lint fix * refactor: accumulate resource attributes for v1 * fix: add empty check for additional string * feat: add table option to mark data model version * fix: do not overwrite all tags * feat: use table option to mark table data model version and process accordingly * chore: update comments to reflect query changes * feat: use header for jaeger table name * feat: update index for service_name, drop index for span_name --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: zyy17 <zyylsxm@gmail.com> |
||
|
Ning Sun
|
37f8341963 |
feat: opentelemetry trace new data modeling (#5622)
* feat: include trace v1 encoding * feat: add trace ingestion in inserter * feat: add partition rules and index for trace_id * chore: format * chore: fmt * fix: issue introduced with merge * feat: adjust index and add integration test for v1 * refactor: remove comment key * fix: update default value of skip index granularity * fix: update default value of skip index granularity * refactor: rename some functions * feat: remove skipping index from span_id * refactor: made span_id part of primary key for potential dedup purpose * feat: move the special attribute resource_attribute.service.name to top level --------- Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com> |
||
 yihong
|
7e61d1ae27 |
feat: support pg_database for DBeaver. (#5362)
This patch support pg_database for pg_catalog, also add query replace, in fixtures.rs for the reason that datafusion do not support sql like 'select 1,1;' more can check issue #5344. Signed-off-by: yihong0618 <zouzou0208@gmail.com> |
||
|
shuiyisong
|
9d7fea902e |
chore: remove unused dep (#5163)
* chore: remove unused dep * chore: remove more unused dep |
||
|
Lei, HUANG
|
e328c7067c |
chore: udapte Rust toolchain to 2024-10-19 (#4857)
* update rust toolchain * change toolchain to 2024-10-17 * fix: clippy * fix: ut * bump shadow-rs * fix: use nightly-2024-10-19 * fix: clippy * chore/udapte-toolchain-2024-10-17: Update DEV_BUILDER_IMAGE_TAG to 2024-10-19-a5c00e85-20241024184445 in Makefile |
||
 Weny Xu
|
4045298cb2 |
feat: add region_statistics table (#4771)
* refactor: introduce `region_statistic` * refactor: move DatanodeStat related structs to common_meta * chore: add comments * feat: implement `list_region_stats` for `ClusterInfo` trait * feat: add `region_statistics` table * feat: add table_id and region_number fields * chore: rename unused snafu * chore: udpate sqlness results * chore: avoid to print source in error msg * chore: move `procedure_info` under `greptime` catalog * chore: apply suggestions from CR * Update src/common/meta/src/datanode.rs Co-authored-by: jeremyhi <jiachun_feng@proton.me> --------- Co-authored-by: jeremyhi <jiachun_feng@proton.me> |
||
 taobo
|
0c9b8eb0d2 |
feat: improve observability for procedure (#4675)
* feat: improve observability for procedure * fix: test error * test: add sqlness test for information_schema.procedure_info * fix: sqlness test error * fix: cr comment * chore: update proto version * fix: apply cr comment * update version * fix: cr comment * optimize procedure type output format * upgrade dep version * fix: clippy error * fix: `procedure` borrowed error * fix: optimize code |
||
 Ruihang Xia
|
93f202694c |
refactor: remove unused error variants (#4666)
* add python script Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove unused errors Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix all negative cases Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * setup CI Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * add license header Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Weny Xu
|
25cd61b310 |
chore: upgrade toolchain to nightly-2024-08-07 (#4549)
* chore: upgrade toolchain to `nightly-2024-08-07` * chore(ci): upgrade toolchain * fix: fix unit test |
||
 JohnsonLee
|
9a5fa49955 |
feat: support pg_namespace, pg_class and related psql command (#4428)
* feat: add function 'pg_catalog.pg_table_is_visible'q * feat: add 'pg_class' and 'pg_namespace', now we can run '\d' and '\dt'! * refactor: move memory_table::tables to utils::tables * refactor: move out predicate to system_schema to reuse it * feat: predicates pushdown * test: add pg_namespace, pg_class related sqlness test * fix: typos and license header * fix: sqlness test * refactor: use `expect` instead of `unwrap` here * refactor: remove the `information_schema::utils` mod * doc: make the comment in pg_get_userbyid more precise * doc: add TODO and comment in pg_catalog * fix: typo * fix: sqlness * doc: change to comment on PGClassBuilder to TODO |
||
 discord9
|
9fa9156bde |
feat: FLOWS table in information_schema&SHOW FLOWS (#4386)
* feat(WIP): flow info table refactor: better err handling&log feat: add flow metadata to info schema provider feat(WIP): info_schema.flows feat: info_schema.flows table * fix: err after rebase * fix: wrong comparsion op * feat: SHOW FLOWS&tests * refactor: per review * chore: unused * refactor: json error * chore: per review * test: sqlness * chore: rm inline error * refactor: per review |
||
|
JohnsonLee
|
072d7c2022 |
feat: introduce 'pg_catalog.pg_type' (#4332)
* WIP: pg_catalog * refactor: move memory_table to crate public level to reuse it in pgcatalog * refactor: new system_schema mod to manage implementation of information_schema and pg_catalog * feat: pg_catalog.pg_type * fix: remove unused code to avoid warning * test: add pg_catalog sqlness test * feat: pg_catalog_cache in system_catalog * fix: integration test * test: rollback unit test * refactor: mix pg_catalog table_id with old ones * fix: add todo information * tests: rerun sqlness --------- Co-authored-by: johnsonlee <johnsonlee@localhost.localdomain> |
||
 Lanqing Yang
|
15ac8116ea |
feat: adding information_schema.views table (#4342)
This commit introduces information_schema.views table. The VIEWS table provides information about views in databases. |
||
 Yohan Wal
|
c4db9e8aa7 |
fix!: forbid to change information_schema (#4233)
* fix: forbid to change tables in information_schema * refactor: use unified read-only check function * test: add more sqlness tests for information_schema * refactor: move is_readonly_schema to common_catalog |
||
|
Ruihang Xia
|
115c74791d |
build(deps): bump snafu to 0.8 (#3911)
* change Cargo.toml Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * global replace Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * handle alias in script engine Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Weny Xu
|
5545a8b023 |
feat: implement drop flow procedure (#3877)
* feat: implement `destroy_flow_metadata` method * chore: bump proto to 65c1364 * feat: implement the drop flow procedure * feat: add `MockFlownodeManager` * tests: add tests for create flow & drop flow procedure * chore: apply suggestions from CR * chore: use `ClusterId` |
||
|
dennis zhuang
|
65d47bab56 |
feat: adds information_schema cluster_info table (#3832)
* feat: adds server running mode to KvBackendCatalogManager * feat: adds MetaClient to KvBackendCatalogManager * feat: impl information_schema.cluster_info table * fix: forgot files * test: update information_schema result * feat: adds start_time and uptime to cluster_info * chore: tweak cargo and comment * feat: rename greptime_region_peers to region_peers * fix: cluster_info result * chore: simplify sqlness commands * chore: set peer_id to -1 for frontends * fix: move cluster_info to greptime catalog * chore: use official proto * feat: adds active_time * chore: apply suggestion Co-authored-by: Jeremyhi <jiachun_feng@proton.me> * chore: STANDALONE for runtime_metrics --------- Co-authored-by: Jeremyhi <jiachun_feng@proton.me> Co-authored-by: tison <wander4096@gmail.com> |
||
|
Weny Xu
|
701aba9cdb |
refactor: rename flow task to flow (#3833)
* refactor: rename to `MIN_USER_FLOW_ID` * refactor: rename to `FLOW_ID_SEQ` * refactor: rename to `flow_id_sequence` * refactor: rename to `FlowMetadataManager` * refactor: rename flow_task.rs to flow.rs * refactor: rename to FlowInfoManager * refactor: rename to FlowName * refactor: rename to FlownodeFlow * refactor: rename to TableFlow * refactor: remove TASK * refactor: rename to __flow * refactor: rename to flow_id * refactor: rename to flow_name * refactor: update comments * refactor: rename to flow_metadata_manager * refactor: rename to flow_metadata_allocator * refactor: rename to FlowMetadataAllocator * refactor: rename task suffix * refactor: rename FlowTask to FlowInfo * refactor: rename FlowTaskScoped to FlowScoped * refactor: rename FlowTaskId to FlowId * chore: bump proto to b5412f7 * chore: apply suggestions from CR * chore: apply suggestions from CR * chore: apply suggestions from CR |
||
|
Weny Xu
|
b493ea1b38 |
feat: implement the CreateFlowProcedure (#3810)
* feat: implement `FlowTaskMetadataAllocator` * feat: add `FlowTaskMetadataManagerRef` and `FlowTaskMetadataAllocatorRef` * chore: fix clippy * feat: add `FlowTaskNameLock` * feat: implement the `CreateFlowTaskProcedure` * chore: rename to `CreateFlowProcedure` * chore: apply suggestions from CR * feat: invoke create flow procedure * chore: apply suggestions from CR * refactor: rename TYPE_NAME * feat: register the procedure * chore: apply suggestions from CR * feat: acquire the lock of sink table name |
||
|
dennis zhuang
|
75d85f9915 |
feat: impl table_constraints table for information_schema (#3698)
* feat: impl table_constraints table for information_schema * test: update information_schema sqlness test * test: adds table_constraints sqlness test |
||
|
dennis zhuang
|
e3b37ee2c9 | fix: canonicalize catalog and schema names (#3600) | ||
 SteveLauC
|
e9a2b0a9ee |
chore: use workspace-wide lints (#3352)
* chore: use workspace-wide lints * respond to review |
||
|
dennis zhuang
|
8b73067815 |
feat: impl partitions and region_peers information schema (#3278)
* feat: impl partitions table * fix: typo * feat: impl region_peers information schema * chore: rename region_peers to greptime_region_peers * chore: rename statuses to upper case * fix: comments * chore: update partition result * chore: remove redundant checking * refactor: replace 42 with constant * feat: fetch region routes in batch |
||
|
shuiyisong
|
4cbdf64d52 |
chore: start plugins during standalone startup & comply with current catalog while changing database (#3282)
* chore: start plugins in standalone * chore: respect current catalog in use statement for mysql * chore: reduce unnecessory convert to string * chore: reduce duplicate code |
||
|
dennis zhuang
|
fd3f23ea15 |
feat: adds runtime_metrics (#3127)
* feat: adds runtime_metrics * fix: comment * feat: refactor metrics table * chore: ensure build_info and runtime_metrics only avaiable in greptime catalog * feat: adds timestamp column |
||
|
Weny Xu
|
ec8266b969 |
refactor: refactor the locks in the procedure (#3126)
* feat: add lock key * refactor: procedure lock keys * chore: apply suggestions from CR |
||
 dimbtp
|
c4d7b0d91d |
feat: add some tables for information_schema (#3060)
* feat: add information_schema.optimizer_trace * feat: add information_schema.parameters * feat: add information_schema.profiling * feat: add information_schema.referential_constraints * feat: add information_schema.routines * feat: add information_schema.schema_privileges * feat: add information_schema.table_privileges * feat: add information_schema.triggers * fix: update sql test result * feat: add information_schema.global_status * feat: add information_schema.session_status * fix: update sql test result * fix: add TODO for some tables * Update src/catalog/src/information_schema/memory_table/tables.rs Co-authored-by: Yingwen <realevenyag@gmail.com> --------- Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> |
||
|
dimbtp
|
f735f739e5 |
feat: add information_schema.key_column_usage (#3057)
* feat: add information_schema.key_column_usage * fix: follow #3057 review comments * fix: add sql test for `key_column_usage` table * fix: fix spell typo * fix: resolve conflict in sql test result |
||
|
dimbtp
|
6070e88077 |
feat: add information_schema.files (#3054)
* feat: add information_schema.files * fix: update information_schema.result * fix: change `EXTRA` field type to string |
||
|
dennis zhuang
|
11ae85b1cd |
feat: adds information_schema.schemata (#3051)
* feat: improve information_schema.columns * feat: adds information_schema.schemata * fix: instance test * fix: comment |