* docs: add entity relationships and graph query RFC
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: revise entity-graph RFC after review
- Zero-configuration declarations: the Prometheus-on-Kubernetes convention
pack (job/instance per the OTel-Prometheus compatibility spec, k8s SD
labels, *_info descriptors, target_info enrichment) alongside the OTLP
trace auto-stamp, plus Remote Write 2.0 inline metadata.
- Calls endpoints follow the service entity declaration; self-calls
compare full endpoint ids; no silent identity fallback.
- Strict time-window contract: the source window is never narrower than
the query's observed_at range; unsafe-to-extract predicates error
instead of silently defaulting.
- scope removed from the relationship schema (kept on entities as a
display property); entity row contract restated per projected
observation; endpoint encoding documented as the v1 storage-level key
with its known collision limitation.
- Sampling caveats corrected (ratios are representative only under
unbiased sampling); snapshot relation synthesizes endpoint-only
vertices; shared attributes provide join keys while co-declaration
provides relationship semantics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: align entity-graph RFC contracts and tighten prose
- Metric/trace service unification is promised only when service.namespace
is empty and job is not relabeled (the compatibility spec renders job as
<namespace>/<name>); otherwise alignment needs pipeline normalization or
explicit declarations.
- Entity row contract stated once (per projected observation); the calls
defining SQL is marked as the single-column simplification of the
declaration-derived endpoint ids; Remote Write 2.0 metadata intake and
the Prometheus implicit declarations are listed as M1 work.
- Compress survey/example/reference prose.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: pin down the k8s convention pack rules and edge directions
- kube_pod_owner implicitly declares k8s.workload with id
(namespace, owner_kind, owner_name) and derives pod part_of workload;
target_info's non-job/instance labels are implicit service descriptive
columns — fixed rules, no new declaration syntax.
- One direction for pod placement: k8s.pod runs_on k8s.node (pod added to
runs_on sources; node->pod removed from contains).
- Drop the remaining 'canonical' wording for the v1 storage-level id.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: make contains/part_of a true inverse pair
part_of covers service.instance->service and k8s.pod->k8s.workload with
contains as its inverse; has_instance is dropped.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: rewrite entity-graph RFC as a design narrative
Restructure for human review: mainline first, cases illustrate the
design instead of specifying it.
- New Architecture and Benefits and Drawbacks sections; the calls
derivation stays expanded as the flagship example while schema
enumerations, window-rule listings, and executor edge-case handling
move out of the document.
- The cross-signal promise is stated honestly: neighbours and their
source tables are discovered first, their telemetry is the next
query — one engine, two statements; the worked example shows the
full declaration -> entity -> edge -> telemetry flow.
- Default materialisation added as the most direct alternative to
read-time derivation, with its costs.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: final wording pass on the entity-graph RFC
Mark the property-graph DDL as illustrative rather than settled M2
syntax, credit standards as foundations rather than claiming wholesale
alignment, and clean up punctuation-heavy prose.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: keep the RFC at design altitude
Demote the convention-pack rule details and snapshot property-merge
semantics to the implementing changes; correct the single-trace-table
assumption (traces can be routed to multiple tables); record
attribute-key participation in entity equality as an open question.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: correct service graph terminology
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: table semantic layer information_schema view (Phase 3)
Add `information_schema.table_semantics`, a queryable view over the table
semantic layer. One row per table that carries at least one
`greptime.semantic.*` option: the signal-agnostic keys
(signal_type/source/pipeline/metadata_quality) are promoted to columns and
the remaining signal-specific keys are folded into a `semantic_options`
JSON string. Tables with no semantic key are excluded.
Stacked on Phase 2.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: address PR review on table_semantics
- fold JSON serialization failure into None instead of unwrap/panic
- drop per-row Vec allocation in predicate eval; use a fixed array
- align RFC view name with the shipped `table_semantics`
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: update results
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: table semantic layer per-table enrichment (Phase 2)
Phase 2 of the table semantic layer, plus a vocabulary trim so the layer only
records what a machine consumer cannot cheaply recover on its own.
Per-table metric enrichment (OTLP), via an internal per-table channel:
- A `SemanticIndex` accumulator records, per emitted table, the declared metric
keys: type / unit / temporality / metadata_quality=declared / original_name.
Conflicting single-valued keys collapse to `mixed`/`unknown`.
- Recording happens at the `encode_metrics` level where the base name, metric
type, and proto fields are all in scope, so histogram/summary fan-out gets the
correct per-subtable type (`_bucket`=histogram, `_sum`/`_count`=counter)
without threading state through every encoder.
- The index is serialized onto the `greptime.internal.semantic.per_table_index`
context extension; `apply_per_table_semantic_options` folds each table's keys
into its options at auto-create time.
- `trace.conventions` is refined from the request's resource/scope `schema_url`s
(concrete when uniform, else `mixed`/`unknown`).
Vocabulary trimmed to only meaningful keys. Kept: signal_type, source, pipeline,
trace.conventions, metric.{type,unit,temporality,metadata_quality,original_name}.
Dropped: metric.monotonic (a function of type), trace.has_events/has_links
(constant + derivable from columns), log.severity_scheme/body_format (constant /
derivable, and body_format cost an O(rows) scan), resource/scope lineage
(restates columns / collector-config concern), source_version (no cheap
non-constant value today). Prometheus carries type/unit in the metric name by
convention, so it gets identity only — no inferred enrichment.
Identity (signal_type + source) extended to the remaining ingest protocols so
the discovery view is complete: InfluxDB and OpenTSDB (metric), Loki and
Elasticsearch (log). These protocols carry no type/unit metadata, so identity is
all that applies.
Tests: unit coverage for the accumulator, per-metric-type fan-out, and trace
conventions; integration goldens updated for the OTLP metric/trace SHOW CREATE
output and the new Loki identity.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: validate the option value
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: cast filters type for scanbench
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: pub file_range mod
So we can use the pub struct FileRange in other places
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: add api as dev-dependency to cmd for clippy
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: support profiling after warmup
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat/gdump:
### Add Support for Jemalloc Gdump Flag
- **`jemalloc.rs`**: Introduced `PROF_GDUMP` constant and added functions `set_gdump_active` and `is_gdump_active` to manage the gdump flag.
- **`error.rs`**: Added error handling for reading and updating the jemalloc gdump flag with `ReadGdump` and `UpdateGdump` errors.
- **`lib.rs`**: Exposed `is_gdump_active` and `set_gdump_active` functions for non-Windows platforms.
- **`http.rs`**: Added HTTP routes for checking and toggling the jemalloc gdump flag status.
- **`mem_prof.rs`**: Implemented handlers `gdump_toggle_handler` and `gdump_status_handler` for managing gdump flag via HTTP requests.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Update docs/how-to/how-to-profile-memory.md
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
* fix: typo in docs
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
* refactor: use DataFusion's UDAF implementation directly
Signed-off-by: luofucong <luofc@foxmail.com>
* remove: delete how-to guide for writing aggregate functions
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
* refactor: port json_encode_path to datafusion udaf
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
* feat: enable prof by default
* docs: don't need to build with features
* feat: add common-pprof as optional dep for pprof feature
* build: remove optional
* feat: use dump_text
shuiyisong
jeremyhi
dennis zhuang
discord9
Yingwen
Ruihang Xia
Lei, HUANG
Ning Sun
Sicong Hu
LFC
yihong
Yiran
liyang
omahs
fys
Yuhan Wang
Weny Xu
Yohan Wal