mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-09-06 13:38:59 +00:00
fb86f6573e244ce4cb3cbbce2140f032ef509ff2
371
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Yingwen
|
ba3c5a939e |
chore(mito2): reduce default auto flush interval (#8971)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Weny Xu
|
1409e66837 |
refactor(flight): add request builder and defer DoGet execution (#8953)
* refactor: add flight request builder Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use flight request builder in flow Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: defer frontend flight query execution Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grpc): avoid cloning requests during auth Signed-off-by: WenyXu <wenymedia@gmail.com> * test(grpc): cover flight request timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(client): share Flight message reader Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(flow): add Flight DoGet timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(client): gate Flight DDL helpers for testing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): restore Flight stream error semantics Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(grpc): document Flight stream input constructors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(flight): preserve deferred stream context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): use Flight stream SNAFU context Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 dennis zhuang
|
6d86e6ff06 |
feat: synthesize OTLP resource descriptor for the semantic entity graph (#8904)
* fix(servers): compose OTLP metrics job from service.namespace/service.name
The OTel Prometheus compatibility spec defines job as
"<service.namespace>/<service.name>" when the namespace is present.
The OTLP metrics path only used the bare service.name, so the job tag
diverged from target_info produced by Prometheus-side exporters for the
same resource. Compose the namespace form, and keep not fabricating a
job when service.name is absent.
Behavior change: resources carrying service.namespace now get
"namespace/name" as their job tag value.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(otlp): synthesize otel_resource_info at OTLP metrics ingestion
Ordinary OTLP metrics scatter filtered resource attributes as tags over
every logical metric table, so metrics-only services contribute nothing
to the semantic entity graph. Each request now also projects its
distinct resources into one info-metric-shaped mito table,
otel_resource_info: a fixed allowlist of identity-relevant attributes
under their raw OTel keys (independent of the label translation
strategy and the promote/ignore headers) plus derived job/instance
compatibility columns, value 1.0, and the newest data-point timestamp.
The descriptor is written after the main insert is committed; a failure
there (conflicting pre-existing table, auto-create disabled) degrades
to an OTLP partial_success warning with rejected_data_points = 0
instead of failing the request and triggering client retries of
already-accepted data. A request writing a metric named
otel_resource_info suppresses synthesis. Legacy mode is unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): otel info-metric conventions with host/container entities
Whitelist the ingestion-synthesized otel_resource_info descriptor via a
new otel_info_metrics conventions map, gated on source=opentelemetry
(the existing gate hardcoded source=prometheus). Its declarations use
explicit descriptive lists instead of descriptive_rest so identifying
attributes of other entities do not leak into service.instance.
Conventions tightened per the Astronomy Shop findings: host identity is
host.id with host.name descriptive only (host.name is not stable across
SDKs and resource detectors), a generic container entity (new entity
type) is declared only when container.id is present, and trace-v1
tables now synthesize host/container from their flattened resource
attributes too. New co-declared edges: service.instance runs_on
container, container runs_on host.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): cover the resource descriptor in integration tests
Covers the descriptor's raw-key columns and info-metric options through
the HTTP path, the namespace/name job composition end-to-end, column
names being independent of the translation strategy, the allowlist
excluding unlisted resource attributes, auto-create after a drop, the
metric-name collision suppressing synthesis, and the partial-success
warning (rejected_data_points = 0) when a pre-existing incompatible
table fails the descriptor write while metric data is accepted.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: cargo fmt
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(frontend): degrade descriptor permission denial to a warning
A table-level permission policy denying otel_resource_info would have
failed the whole OTLP metrics request because the descriptor's
permission check ran before the main insert. The descriptor is derived
enrichment: check its permission in the degrade path so a denial skips
the write and surfaces as the partial-success warning, like any other
descriptor write failure.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): guard descriptor writes with semantic ownership markers
A pre-existing schema-compatible table named otel_resource_info would
silently receive descriptor rows while its missing semantic stamps kept
it out of the entity graph. The descriptor write now requires the
auto-created table's ownership markers (mito engine + signal_type +
source + metric.type=info + metadata_quality=declared) and otherwise
degrades to the partial-success warning; the entity-graph gate for the
otel whitelist likewise requires metric.type=info, so a user table
stamped with only signal/source no longer picks up implicit
declarations.
Also fold the descriptor write cost into the response and surface the
degrade warning through the otel-arrow BatchStatus status_message.
Integration tests pin the full marker set on auto-create and that an
existing owned descriptor keeps accepting writes without degrading —
a missing marker would otherwise silently stop every descriptor write
after the first request.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(otlp): build descriptor rows without the per-resource BTreeMap
Projecting a resource allocated a BTreeMap and then collected it into the
row key, and every attribute was matched against the allowlist by linear
scan. Collect the tags into a Vec and sort once, and match the allowlist
instead of scanning it. Measured on the conversion path: descriptor work
drops 16-18%, from 10.6% to 8.9% of conversion CPU on the worst shape
(1000 resources with 4 data points each), where the cost tracks resource
count rather than data-point count.
Also trims the comments and tests added with the descriptor to what
carries information.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): pin the descriptor permission-denial degrade path
A policy denying the descriptor table must not fail the metrics request,
which the fix in
|
||
 discord9
|
3493d2d0fb |
perf(servers)!: speed up Prometheus JSON response building with ryu and per-series entry reuse (#8815)
* fix(perf): align direct-SST CREATE TABLE with baked index metadata
The offline fixture generator (query_perf_fixture::direct_sst::
build_region_metadata) bakes greptime:inverted_index /
greptime:skipping_index field metadata into the region manifest for
tag/field columns, but create_table_sql emitted a bare CREATE TABLE
without those declarations. MergeScan's remote-schema validation then
failed on any tag/field projection (HTTP 500 'advertised remote stream
schema field mismatch'), breaking direct_readable_sst perf cases.
CREATE TABLE now declares the matching SKIPPING INDEX WITH
(granularity='1') / INVERTED INDEX column options. A round-trip test
proves the emitted SQL is parser-valid and yields the exact catalog
metadata.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): speed up Prometheus JSON response building with ryu and per-series entry reuse
PrometheusJsonResponse::record_batches_to_data spends ~47% of its CPU
in f64::to_string() per sample and ~32% in IndexMap::entry() per row
(60s profile of concurrent query_range workloads, ~800k series).
- Replace f64::to_string() with ryu::Buffer::format_finite for finite
values (shortest round-trip, 3-5x faster); NaN/+Inf/-Inf keep the
previous std formatting so wire output is unchanged.
- Remember the previous row's label vector and entry index; query output
is clustered by series, so consecutive rows reuse the same IndexMap
entry via get_index_mut instead of rebuilding and hashing the label
vector (worst case adds one Vec comparison per series transition).
Also adds a query-regression case (prom_json_response) that measures the
real Prometheus HTTP range API path (/v1/prometheus/api/v1/query_range),
which is the only frontend path that builds the Prometheus JSON response
(TQL ANALYZE formats the SQL JSON shape instead), plus a prom_http query
kind in the regression runner.
Perf (aligned base
|
||
 shuiyisong
|
1c5eabcbbf |
feat(otlp): support cumulative exponential histograms (#8900)
* feat: implement exponential histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove duplicate tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): enforce exponential histogram ingestion safety Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update rfc Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): remove protocol-coupled histogram checks Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): reuse native histogram schema across data points Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: merge repeated OTLP histogram fragments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): build rejection messages lazily Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: add doc Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 Ning Sun
|
174577164a |
feat: otlp duration_nano and trace_flag signed integer coercion (#8816)
* feat(servers): add signed→unsigned int coercion to OTLP ingest path Phase 0 of transitioning built-in data models from unsigned to signed integers (#8793): add lossless Int64→UInt64 and Int32→UInt32 coercion arms so existing UInt64/UInt32 columns (e.g. trace `duration_nano`, log `trace_flags`) keep accepting new signed ingest without an ALTER TABLE. The OTLP ingest path already reconciles every incoming column against the existing table schema and treats it as authoritative. With these arms, `choose_trace_reconcile_decision` returns `UseExisting(UInt64/Uint32)` for an existing unsigned column receiving signed data: the table keeps its type byte-for-byte and the request value is coerced. No persisted format is mutated; existing data stays readable as-is. This is the safety net that makes the actual schema flip (Phase 1) safe. Only the signed→unsigned direction is supported — the reverse would be lossy for values above the signed range and is intentionally rejected. Tests cover both new arms plus an end-to-end log test proving an existing UInt64 column coerces an incoming Int64 value while keeping its type. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: add compatibility layer for uint trace/log fields Signed-off-by: Ning Sun <sunning@greptime.com> * fix: jaeger test Signed-off-by: Ning Sun <sunning@greptime.com> * fix: keep trace v0 unsigned, reject negative span durations - Keep the frozen v0 data model on UInt64 duration_nano: the signed ingest compatibility layer only runs on the v1 path, so flipping v0 would break writes into every pre-existing v0 table at mito's schema check. Pin the schema with unit and integration tests. - Reject spans whose end precedes their start (or whose duration does not fit i64) on the v1 path instead of wrapping: new Int64 tables and existing UInt64 tables now fail identically, rather than storing negative durations that break the Jaeger query API. - Extract is_supported_signed_to_unsigned_coercion so the trace and log ingest paths share one supported-pair predicate and cannot drift. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: clamp negative span durations to zero, revert semantic_graph comment - Record duration 0 for spans whose end precedes their start instead of erroring: a malformed span no longer fails the request, and the value written is always a non-negative, in-range i64 so new Int64 tables and existing UInt64 tables (via the checked coercion) behave identically. Durations above i64::MAX saturate rather than wrap. - Revert the doc-comment tweak on the semantic_graph test fixture; the file is untouched by this PR again. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
dennis zhuang
|
3de4feddd5 |
feat(otlp): report the cause of rejected trace spans (#8897)
* feat(otlp): report the cause of rejected trace spans
When trace-v1 ingestion cannot coerce an attribute value, it falls back to
single-span writes and rejects the bad span. That behavior is correct, but the
OTLP partial-success message only carried `Rejected span <trace_id>:<span_id>
(InvalidArguments)`: the column, the source value, the source type and the
target type were all dropped, so locating the bad attribute required adding a
detailed exporter on the collector side and replaying traffic.
Two places lost the information. `prepare_trace_column_rewrites` built a message
without the failing value, and the span rejection path kept only the status code
from the error.
Coercion errors now name the failing value, e.g.
failed to coerce trace column 'span_attributes.http.response.body.size'
in table 'opentelemetry_traces' from String("") to Int64
and the rejection detail carries that cause. Values are user data, so a string
keeps at most 16 characters, is escaped, and binary payloads report only their
length; the cause itself is bounded at 256 characters. Both truncations cut on a
char boundary.
Failure details now deduplicate: repeats of the same (site, cause) collapse into
one entry with an occurrence count, keyed on the untruncated cause so two
failures that differ past the display limit stay separate. Only four distinct
entries are retained and the rest are counted, which keeps the state bounded no
matter how many distinct bad values a request carries. A fully rejected request
is logged at warn level and a partial success at debug level, since the latter
repeats every export interval; the detail goes out as a Debug field so a newline
in an attribute key cannot forge log lines.
Rejection semantics are unchanged: partial success, same accepted and rejected
counts, same HTTP status mapping.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(otlp): compare failure keys directly instead of hashing
The dedup identity was a DefaultHasher fingerprint of `(label, key)`, which
bought a fixed 8 bytes per entry at the cost of an import, four lines, and a
collision argument the reader has to make. Entries are capped at four and a
cause runs a couple of hundred characters, so the saving is about a kilobyte
per in-flight request while the column name it avoids retaining is already held
several times over by the request itself.
Compare the strings instead, keeping the untruncated cause as the key so
failures differing past the display limit still stay apart. Labels are metric
label values and always static, so the entry borrows them.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
|
Weny Xu
|
ed4271af40 |
feat(procedure): record event actor (#8849)
* feat(event): record procedure actor Signed-off-by: WenyXu <wenymedia@gmail.com> * test: handle streamed region migration output Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover procedure actors across SQL protocols Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
76924c2d36 |
feat(mito2): introduce two-phase metric series scans (#8826)
* feat(mito2): add two-phase series scan Signed-off-by: evenyag <realevenyag@gmail.com> * docs: regenerate configuration reference Signed-off-by: evenyag <realevenyag@gmail.com> * test(sqlness): update series scan explain results Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config API expectation Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): bound two-phase series discovery Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): avoid candidate distribution deadlock Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): remove obsolete dead code allowances Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): share series scan memory pool Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
546625c45a |
feat: embedded convention pack for the entity graph (prom/k8s, gen_ai naming) (#8854)
* feat: embed the derivation conventions as data and adopt gen_ai entity naming Move the co-declared edge vocabulary, the agent-edge vocabulary and the virtual-destination candidates from Rust consts into an embedded conventions.yaml (include_str!), parsed once behind a LazyLock and validated against the entity-type grammar and the closed rel_type set; a broken file propagates as a plan error instead of panicking. The agent vocabulary entity types follow the GenAI semantic-convention namespace as written: gen_ai.agent / gen_ai.model / gen_ai.tool. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: drop the tag requirement for entity identity columns Entity declarations no longer require id columns to be tag/primary-key columns; only column existence is validated. Trace pipelines flatten the identifying attributes (span_attributes.gen_ai.agent.id, ...) into field columns, so the tag rule locked real trace tables out of declaring entities while buying no correctness — the read-time derivation works on any column. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: implicit declarations for well-known prometheus info metrics Tables stamped signal_type=metric + source=prometheus whose name matches the conventions.yaml whitelist (kube_pod_info, kube_node_info, kube_pod_owner, target_info) get implicit entity declarations: k8s.pod / k8s.node / k8s.workload with name-based identity and target_info's service / service.instance with the remaining tags as the descriptive snapshot. The existing co-declared vocabulary then derives runs_on and part_of from the same rows, so no new edge branch is needed. Explicit declarations of a type always suppress the implicit one, and the metric engine's physical table is excluded (it aggregates every logical table's columns). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover the prometheus conventions in sqlness and compact the graph cases Add the whitelisted-info-metric scenario (kube_pod_info, kube_pod_owner, target_info deriving runs_on / part_of, a non-whitelisted metric contributing nothing), fold the single-table calls, cross-table pairing and virtual-node cases into one trace scenario (they exercise the same union-before-join path), merge the two declaring-metric-table cases, and reuse one rename probe for both reserved names. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject entity id columns without a stable string form Review follow-ups: the DDL check now validates against the schema and rejects binary-backed and nested types for identity columns (the derivation renders ids via CAST to Utf8, so the failure used to surface only when the graph was scanned); the agent sqlness case keeps its identity columns as fields to cover the relaxed tag rule end to end; stale tag-rule comments and a dangling const reference are cleaned up. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: type-check every entity column role, not only ids The registry renders scope and descriptive values through the same CAST-to-string path as ids, so a binary-backed column in any role fails at scan time; the DDL check is now role-independent (and simpler). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: name the code-anchored vocabulary constants Entity types and edge attributes the derivation code itself anchors on (service, gen_ai.agent, calls, trace/attribute provenance) become constants in the conventions module; the rest of the vocabulary stays YAML-only data. ImplicitEntity is renamed PromImplicitEntity, and the implicit-declaration path logs each skip of a whitelisted info metric (wrong stamps, suppressed by an explicit declaration, missing id column) so a missing graph entity is diagnosable. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: single-source the graph constants The graph tables' column names move to common-catalog (the schemas catalog exposes and the plans operator builds must match column by column), and the conventions module now carries the complete built-in vocabulary — entity types, rel_types, provenances and connection types — with the embedded YAML validated by membership against it, so an edit drifting outside the vocabulary fails the conventions test instead of deriving nothing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: treat empty identity components as absent kube-state-metrics emits empty-string labels an entity id must not be built from: an unscheduled pod's node and an owner-less pod's owner_kind / owner_name. Standard Prometheus drops empty labels (they arrive as NULL and the existing predicate handles them), but other remote-write agents may keep them, which produced ghost entities with empty ids and false runs_on / part_of edges. Every identity predicate (registry, co-declared edges, span endpoints) now requires non-NULL and non-empty components through one shared helper. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: tighten the conventions DSL semantics Rename the co-declaration rule lists to what they are (co_declared_edges / trace_co_declared_edges — derivation rules, not a relation vocabulary), stop overstating the GenAI entity types (Greptime types derived from GenAI attributes; OTel defines no model/tool entities), move target_info's descriptive snapshot to service.instance (the remaining labels are the target's resource attributes, and instances would write conflicting snapshots onto the logical service), and extend the descriptor whitelist with the stable KSM sources: container info metrics (closing the k8s.pod contains k8s.container rule), kube_service_info (new k8s.service entity type) and the fuller descriptive label sets. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: guard entity column types on ALTER as well ALTER MODIFY COLUMN could change a declared entity column to a type without a stable string form, deferring the failure to graph scan time; verify_alter now checks the post-alter schema. Dropping a declared column stays allowed — the read-time derivation skips the stale declaration, and semantic options cannot be altered off yet. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: bridge traces and kube-state-metrics on the pod UID Trace-v1 tables now get implicit declarations from their flattened resource attributes (otlp_trace_entities in conventions.yaml): the service identity — replacing the hardcoded fallback — plus service.instance and k8s.pod, each applied only when its columns exist. A new co-declared rule derives service.instance runs_on k8s.pod, and the whitelisted kube-state-metrics pod identity switches from namespace+pod names to the UID, so the trace-side pod and every KSM descriptor land on one entity while names stay descriptive. This also removes pod identity from the multi-cluster same-name collision. The conventions rejection tests were passing for the wrong reason (a half-renamed fixture key failed deserialization before reaching any validation rule); they now assert the specific error each case targets. Sqlness covers the UID merge across descriptor tables, pod-contains- container, the k8s.service node, and the empty-uid/empty-node rows deriving nothing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover the OTLP-to-graph chain end to end One real OTLP export must come out of semantic_relationships as the zero-configuration chain: service calls service, instance part_of service, instance runs_on pod (bridged by k8s.pod.uid). Resources without service.instance.id or k8s.pod.uid derive nothing extra. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: identify k8s.service by UID Same reasoning as pods: a recreated same-name service must not merge into the old entity and same-named services across clusters must not collide; kube_service_info carries a stable uid and nothing joins on the service's name. Also drop a stale tag-rule mention from the option validation docs. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: cut duplicated test coverage and redundant comments The trace service-fallback test collapsed into the resource-entities test (same synthesis path since the fallback moved to YAML; only the invalid-explicit-no-fallback case was distinct), role-duplicate and subsumed DDL cases are gone, the embedded-conventions test is just the parse (its assertions were decorative), and the YAML section comments no longer restate the struct docs. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
shuiyisong
|
3b3a9032e0 |
feat(servers): expose native histograms over Prometheus HTTP (#8850)
* feat(servers): expose native histograms over Prometheus HTTP Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(servers): refine Prometheus HTTP metadata handling Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(servers): expand Prometheus metadata coverage Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(servers): return OpenMetrics units from metadata API Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Ning Sun
|
3510ef7d4c |
feat!: update native histogram unsigned int types (#8824)
* feat(native-histogram): store counts and span lengths as signed integers Native histograms are unreleased, so the on-disk integer payload columns are switched from unsigned to signed types without backward-compat: - count_u64 / zero_count_u64: uint64 -> int64 - positive_span_lengths / negative_span_lengths: list(uint32) -> list(int32) - Span.length (query-time model): u32 -> i32 The Prometheus remote-write v2 source carries these as uint64/uint32, so the unsigned->signed conversion at the ingestion boundary is overflow checked: an integer count >= 2^63 or a span length >= 2^31 is rejected with an explicit error rather than silently wrapping to a negative value. read_spans additionally rejects negative stored lengths to keep the non-negative invariant sound for downstream `as usize` casts. The UDAF accumulator's own observation counter (transient aggregation state, not part of the persisted histogram value) is intentionally left as uint64. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(native-histogram): rename count/zero_count fields to _i64 Now that the integer payload columns are stored as int64, rename the field constants and persisted names to match: COUNT_U64_FIELD ("count_u64") -> COUNT_I64_FIELD ("count_i64") ZERO_COUNT_U64_FIELD ("zero_count_u64") -> ZERO_COUNT_I64_FIELD ("zero_count_i64") The local builder variables and the docs/JSON snapshot are updated to match. No backward-compat (unreleased feature). Signed-off-by: Ning Sun <sunning@greptime.com> * test(native-histogram): refresh planner plan snapshot for signed types The mixed native-histogram range test embeds the full histogram Struct type in its expected plan string, which still carried the pre-rename unsigned fields. Update the snapshot to match the signed schema: positive/negative_span_lengths: List(UInt32) -> List(Int32) count_u64/zero_count_u64: UInt64 -> count_i64/zero_count_i64: Int64 Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
dennis zhuang
|
4eeb4052d6 |
feat(servers): stamp prometheus remote write v2 metadata as semantic table options (#8797)
* feat(servers): stamp prometheus remote write v2 metadata as semantic table options
Remote write 2.0 carries per-series metadata (type, unit, help) that the
v2 ingest decoded and dropped; tables kept the name-based 'inferred'
quality. Wire it into the semantic layer:
- generalize the OTLP per-table semantic index into a shared, schema-
aware servers::semantic module: v2 lets each series override its
target schema, so the index is keyed {schema -> table -> options} and
the same metric name in two schemas no longer collapses;
- into_write_requests records metric type and unit per written table;
an explicit type upgrades the table's metadata quality to declared,
UNSPECIFIED series keep the request-level inferred stamp, and units
are canonicalised from OpenMetrics words to the UCUM codes the
vocabulary is defined in (unknown units are dropped, help text is not
persisted);
- the consumer folds the index in on both auto-create paths: the
operator row-insert path and the pending-rows batched create, which
bypasses the former.
Table options are stamped at auto-create only; updating existing tables
from later metadata is future work (a metadata registry, see the native
histograms RFC).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim over-commenting in the remote write metadata path
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: parse the per-table semantic index once per create round
The index was re-parsed from JSON for every table being created — a
first write creating N tables (a fleet's first scrape) paid
O(N x index size). Parsing now happens lazily once per create-planning
round, on both consumers: the operator row-insert auto-create (also
serving OTLP metrics) and the pending-rows batched create.
Also validate every non-zero metadata symbol reference up front, as the
remote write 2.0 spec requires: help_ref was never checked, and
unit_ref escaped checking when the metric type was UNSPECIFIED.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): stamp remote-write v2 unit independently of metric type
OpenMetrics models TYPE and UNIT as independent MetricFamily metadata, and
the Prometheus v2 sender emits UNSPECIFIED-type series that still carry a
unit. The early return on UNSPECIFIED dropped that unit, which is
unrecoverable after table auto-create (units are not stored in rows).
Stamp the mapped unit whenever present; the type and the
metadata_quality=declared upgrade still require an explicit type.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim restating comments in the v2 metadata path
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
|
Weny Xu
|
1f1c9270a8 |
feat(event): add event context to procedure events (#8734)
* feat(event): add trigger context to procedure events Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix trigger context event contracts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve trigger context origins Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve lifecycle trigger contexts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): gate enterprise trigger imports Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): assert trigger contexts exactly Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(test): restore migration test literals Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(event): add trigger context to non-table ddl Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): simplify trigger context encoding Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve auto alter trigger reason Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): require explicit trigger context Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): derive trigger context at ddl boundary Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): propagate DDL trigger reasons Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): resolve alter table rebase conflict Signed-off-by: WenyXu <wenymedia@gmail.com> * test(meta): fix alter table trigger context setup Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): pass trigger context to region migration Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix unknown trigger context assertions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): centralize trigger context protocol mapping Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix migration trigger context assertion Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(procedure): rename event runtime context Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event): rename trigger context Signed-off-by: WenyXu <wenymedia@gmail.com> * test(event): fix migration event context assertion Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
57b8239ff8 |
test: rename internal bug numbers in tests to semantic names (#8779)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
3d12273c84 |
feat: read-time entity relationships graph over telemetry (M0+M1) (#8614)
* feat(table): add entity semantic declarations Define open-ended greptime.semantic.entity.* options, validate entity columns at DDL time, and stamp OTLP trace tables with the service entity declaration. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: add read-time entity relationships graph Add computed semantic graph tables, typed DataFusion derivation plans for entity registry and trace calls edges, and streaming read-time execution. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: exclude semantic graph tables from table constraints Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(operator): name the plan-builder source groupings Review feedback: build_registry_plan / build_calls_plan took anonymous (declarations, DataFrame) tuples while the caller already grouped the same fields. Introduce RegistrySource { declarations, scan } and CallsSource { service, scan } next to the builders and flow them through the frontend caller and tests. The frontend-side EntitySource keeps holding a TableRef (the operator builders stay pure over already-built scans), so the named structs live in operator rather than reusing that type. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
f25836a6ed |
fix: support Utf8View labels in Prometheus response (#8754)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
shuiyisong
|
aa72563783 |
refactor!: move native histogram config and prom_validation_mode to prom_store (#8744)
* chore: adjust the position of experimental_enable_prometheus_native_histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: move prom_validation_mode as well Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 Whis Liao
|
e58f21ed6d |
feat(logging): add enable_file_logging option to disable file logging (#8721)
Signed-off-by: xhwhis <hi@whis.me> |
||
|
Ning Sun
|
deb688f572 |
feat: add a dedicated http api server port (#8657)
* feat: add a dedicated http api server port * fix: integration test * refactor: make http-api-port opt-in * refactor: rename attribute to http-api-server * feat: use middleware to check different http server port * refactor: rename config option |
||
|
Weny Xu
|
7344d47756 |
feat(event-recorder): configure lifecycle event recording (#8648)
* refactor(event-recorder): centralize event table helpers Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(procedure): wire lifecycle event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(event-recorder): filter events by type Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): derive event type filter default Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): decouple frontend filtering Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove docs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): complete configuration support Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event-recorder): centralize filter ownership Signed-off-by: WenyXu <wenymedia@gmail.com> * test(config): update event recorder snapshot Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(frontend): decouple slow query event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
shuiyisong
|
b27a97ab2e |
fix(servers): validate remote write native histograms (#8654)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 Lei, HUANG
|
c8f65c7b99 |
feat: update flow windows after metric batch flush (#8544)
* feat: update flow windows after metric batch flush Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: move batch rows into flow notifier Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: preserve timestamp index in metric batches Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: avoid blocking flow notification lookups Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: add context to timestamp extraction logs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: include peer in flow notification errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: compact flow notifications with time ranges Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: bound pending flow notification queue Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: send raw timestamps in flow notifications Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: update config API snapshot Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9andLei, HUANG
|
0564269426 |
fix: bind Prom remote read schema per query (#8591)
Signed-off-by: discord9 <discord9@163.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> |
||
|
discord9
|
56addd0623 |
fix: stream remote analyze metrics while pending (#8405)
* fix: stream remote analyze metrics while pending Signed-off-by: discord9 <discord9@163.com> * test: verify flight metrics preserve pending batch Signed-off-by: discord9 <discord9@163.com> * fix: preserve direct SST perf queries in plans Signed-off-by: discord9 <discord9@163.com> * fix: bind flight metrics capability to query Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 Yash Agrawal
|
026cfbd122 |
feat: support splunk HEC raw endpoint (#8491)
* feat: add /raw handler + unit tests Signed-off-by: yash <yasha4658@gmail.com> * feat: add integration test for splunk /raw handler Signed-off-by: yash <yasha4658@gmail.com> * fix: added logs and captured payload unit test for splunk /raw endpoint Signed-off-by: yash <yasha4658@gmail.com> * fix: apply cargo fmt and optimize raw_line_to_map Signed-off-by: yash <yasha4658@gmail.com> * fix: preserve entire body as a event with opt-in linebreaker query param Signed-off-by: yash <yasha4658@gmail.com> --------- Signed-off-by: yash <yasha4658@gmail.com> |
||
|
shuiyisong
|
3ae8d7d46a |
refactor: reconcile OTLP trace schemas request-wide (#8485)
* refactor: global trace schema Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: split files Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments and doc Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
0a002f4d65 |
feat: support per-region write buffer limits (#8473)
* feat(mito): add per-region write buffer limit Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): add default region write buffer size Signed-off-by: evenyag <realevenyag@gmail.com> * docs: expand configuration change checklist Signed-off-by: evenyag <realevenyag@gmail.com> * test: cover table write buffer size option Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): refine region write buffer checks Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): keep region-stalled writes queued Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): allow zero region write buffer size Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): validate region state before write stall Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): reject writes beyond region hard limit Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update example Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
1122d1e7d7 |
fix: apply log query limit after expressions (#8496)
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
99235e4b2d |
fix: honor exclusive end in log queries (#8495)
* fix: exclude logs end boundary Signed-off-by: discord9 <discord9@163.com> * fix: normalize log query date bounds Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
e6472fd12a |
fix: pause GC during maintenance mode (#8450)
Skip scheduled meta GC while cluster maintenance mode is enabled and reject manual GC requests explicitly instead of returning an empty success report. Also increase mito GC's default lingering time to 1h and update generated config docs and config API expectations. Signed-off-by: discord9 <discord9@163.com> |
||
|
shuiyisong
|
90752f5649 |
feat: persist Prometheus remote write v2 native histograms (#8382)
* chore: add histogram decode Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add convert func Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: implement native histogram persistency Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: refactor Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add README for v2 Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: implement metric engine Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: store nh in one struct Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues and add feature gate Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: add config option Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
33902eef3e |
fix(mito): honor unknown file lingering time (#8365)
* fix(mito): honor unknown file lingering time Signed-off-by: discord9 <discord9@163.com> * fix(mito): allow gc test helper arguments Signed-off-by: discord9 <discord9@163.com> * test(mito): cover unknown file ttl boundaries Signed-off-by: discord9 <discord9@163.com> * test: update unknown file ttl config snapshot Signed-off-by: discord9 <discord9@163.com> * chore: fix typo in logical table alter Signed-off-by: discord9 <discord9@163.com> * refactor(mito): make gc delete helper sync Signed-off-by: discord9 <discord9@163.com> * refactor(mito): call gc delete helper directly Signed-off-by: discord9 <discord9@163.com> * refactor(mito): minimize unknown file ttl changes Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
55852a05b8 |
feat: stream explain analyze metrics over http (#8380)
* feat: stream explain analyze metrics over http Signed-off-by: discord9 <discord9@163.com> * fix: address analyze stream review comments Signed-off-by: discord9 <discord9@163.com> * test: document analyze stream protocol Signed-off-by: discord9 <discord9@163.com> * test: update config api expectation Signed-off-by: discord9 <discord9@163.com> * fix: track slow queries for analyze stream Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
  
|
7c1009c00f |
feat: accept x-greptime-pipeline-name header on /events/logs (#8371)
* feat: accept x-greptime-pipeline-name header on /events/logs The /events/logs (and /logs/ingest) endpoint previously only read the pipeline name from the `pipeline_name` query parameter, while the OTLP/Elasticsearch/Splunk log ingestion endpoints already accept it via the `x-greptime-pipeline-name` header. This inconsistency is unfriendly to users. Make `log_ingester` resolve the pipeline name from the `x-greptime-pipeline-name` header (and the deprecated `x-greptime-log-pipeline-name`), falling back to the query parameter. The header takes precedence, consistent with how other pipeline options (e.g. `x-greptime-pipeline-params`) outrank their query-parameter counterparts. Closes #6095 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com> * address review: prefer non-deprecated pipeline-name header When both pipeline-name headers are present, resolve the non-deprecated `x-greptime-pipeline-name` before the deprecated `x-greptime-log-pipeline-name`, and cover the precedence with a test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com> --------- Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com> Co-authored-by: BootstrapperSBL <yvanwww01@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
shuiyisong
|
59d40c39b1 |
feat: prw_v2 initial commit with sample ingestion (#8361)
* chore: add v2 entrance Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: decode request Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: implement remote write v2 for samples Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove hand-written proto Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: refactor Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: merge tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add source version field Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update proto Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update proto Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yash Agrawal
|
daed55e836 |
feat: Add support for splunk HEC endpoints (#8321)
* feat: add health endpoint for splunk HEC Signed-off-by: yash <yasha4658@gmail.com> * feat: add event endpoint and handler + tests Signed-off-by: yash <yasha4658@gmail.com> * feat: add pipeline override test Signed-off-by: yash <yasha4658@gmail.com> * feat: add error response for auth errors Signed-off-by: yash <yasha4658@gmail.com> * fix: cargo fmt + doc example test Signed-off-by: yash <yasha4658@gmail.com> * fix: clippy warnings Signed-off-by: yash <yasha4658@gmail.com> * fix: address comments Signed-off-by: yash <yasha4658@gmail.com> * fix: address copilot comments and default table name inlining Signed-off-by: yash <yasha4658@gmail.com> * fix: cargo fmt Signed-off-by: yash <yasha4658@gmail.com> * fix: add error code 12, 13 and fix the timestamp override bug Signed-off-by: yash <yasha4658@gmail.com> * fix: add host/source/sourcetype ordering Signed-off-by: yash <yasha4658@gmail.com> * fix: fix typos Signed-off-by: yash <yasha4658@gmail.com> --------- Signed-off-by: yash <yasha4658@gmail.com> |
||
|
Lei, HUANG
|
a3461caf9d |
feat: expose region read load metrics (#8316)
* feat: expose region read load through Prometheus metrics and heartbeat Introduce region-level query load tracking (CPU time and scanned bytes) collected by `RegionScanExec`, exposed via Prometheus metrics and optionally reported through heartbeat region stats. - **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add `greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`, and `greptime_mito_region_written_bytes_since_open` gauge metrics. - **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`, `src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with `IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle management (init, reset, remove-on-drop). - **RegionStatistic** (`src/store-api/src/region_engine.rs`, `src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and `query_scanned_bytes` fields. - **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from metadata and data regions. - **Heartbeat** (`src/datanode/src/heartbeat.rs`, `src/common/meta/src/datanode.rs`): Relay region query load via heartbeat `RegionStat`; add test. - **Query engine** (`src/query/src/options.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dist_plan/merge_scan.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config; wire `RegionScanExec` to accumulate CPU time and scanned bytes. - **Table scan** (`src/table/src/table/scan.rs`, `src/table/src/table/metrics.rs`): Wire table scan metrics. - **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/config.md`): Add example config and documentation for `enable_region_query_load_report`. - **Tests** (`src/mito2/src/engine/basic_test.rs`, `src/mito2/src/engine/close_test.rs`, `src/cmd/tests/load_config_test.rs`, `src/flow/src/adapter.rs`): Add unit tests for region query load reporting and metric cleanup on region close; set default config values. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region read load report config from query layer to mito engine Move the `enable_region_query_load_report` setting from query-level config (`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config (`MitoConfig`), and expose it through the `RegionScanner` trait instead of `ScanRequest`/`PrepareRequest`. - Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs` - Scan region plumbing: `src/mito2/src/read/scan_region.rs` - RegionScanner trait: `src/store-api/src/region_engine.rs` - Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs` - RegionScanExec: `src/table/src/table/scan.rs` - Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs` - Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs` - Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region query load report config from MitoConfig to LoggingOptions Relocate the `enable_region_query_load_report` setting from `MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`), and thread it into `MitoEngineBuilder` instead of reading from the engine config directly. This makes the region read-load reporting a per-node logging/observability concern rather than a per-engine storage setting. - `config/config.md` - `config/datanode.example.toml` - `config/standalone.example.toml` - `src/common/telemetry/src/logging.rs` - `src/datanode/src/datanode.rs` - `src/mito2/src/config.rs` - `src/mito2/src/engine.rs` - `src/mito2/src/region.rs` Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: report region query load on stream drop instead of stream end Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()` to `Drop::drop()` so that region query load is reported even when the stream is dropped prematurely (not just when fully consumed). Affected files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, Huang <huanglei@qiyi.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: make region query load reporting configurable Introduce `enable_region_query_load_report` flag to optionally report per-region `query_cpu_time` and `query_scanned_bytes` metrics instead of always creating them. When disabled, the Prometheus gauges are not created (`None`), avoiding metric churn for workloads that do not need query-level load tracking. - `src/common/meta/src/datanode.rs` — Placeholder fields for query load - `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset - `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener` - `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop` - `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open - `src/mito2/src/worker/handle_create.rs` — Pass flag on region create - `src/mito2/src/worker/handle_open.rs` — Pass flag on region open - `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder` - `src/mito2/src/test_util.rs` — Test helpers for both modes - `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases - `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove elapsed_compute metric from scan stream The elapsed_compute metric conflated poll-wait time with actual CPU computation, making it misleading. Removed the metric and its recording path from StreamMetrics and StreamWithMetricWrapper. Added a test asserting that poll duration is not reported as elapsed_compute. - `src/table/src/table/metrics.rs` — removed elapsed_compute field, builder, and record_elapsed_compute method - `src/table/src/table/scan.rs` — removed record_elapsed_compute call; added SlowRecordBatchStream test helper and wrapper_poll_time_is_not_elapsed_compute test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: disable region query load report for compaction scans Compaction scans are internal operations initiated by the engine, not user queries. Disable region query load reporting when the scan input is marked as compaction to avoid misleading load metrics. - `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report` to `false` when compaction is enabled; add unit test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test: add `enable_per_region_metrics` config to HTTP integration test - Enable per-region metrics config in HTTP test setup \`tests-integration/tests/http.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove region query load reporting tests and helpers Remove the region query load reporting feature from the codebase, including tests, test utilities, and helper infrastructure that were part of this now-deprecated functionality. Specifically: - Remove region query load reporting tests from `src/mito2/src/engine/basic_test.rs` and `src/table/src/table/scan.rs`, and the region close metrics test from `src/mito2/src/engine/close_test.rs` - Remove region query load report test utilities and simplify engine construction helpers in `src/mito2/src/test_util.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf: avoid disabled region query load timing Summary: - Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled. - Keep region query-load CPU accounting active only when reporting is enabled. Files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move per-region query load reporting from storage to query engine Move `enable_per_region_metrics` from datanode to frontend config and migrate query load tracking (CPU time, scanned bytes) from mito2 storage engine to the query engine's distributed scan planner. The storage-level metrics plumbing and `enable_region_query_load_report` flag are removed from mito2, `ScanInput`, `ScanRegion`, and `RegionScanner`. Query-level metrics are now collected in `merge_scan.rs` via `scan_region_load`. - `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes` metrics, `enable_region_query_load_report` plumbing from engine, region, opener, scanner types, workers - `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes` from `RegionStatistic` - `src/metric-engine/` -- Remove query load fields from `get_region_statistic` - `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`; wire through planner, optimizer, merge scan with `scan_region_load` metrics - `src/frontend/` -- Pass `enable_per_region_metrics` into `QueryOptions` - `src/common/meta/` -- Remove TODO for query load fields - `config/` -- Move `enable_per_region_metrics` from datanode to frontend and standalone example configs - `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode config test - `src/flow/` -- Add `enable_per_region_metrics` default to flownode options - `src/table/` -- Remove unused query load fields from scan - `src/datanode/` -- Remove `with_enable_region_query_load_report` calls Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove obsolete mito write load metric Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path. Related files: - \`src/mito2/src/metrics.rs\` - \`src/mito2/src/region.rs\` - \`src/mito2/src/region/opener.rs\` - \`src/mito2/src/region_write_ctx.rs\` - \`src/mito2/src/engine/basic_test.rs\` - \`src/mito2/src/worker.rs\` - \`src/mito2/src/config.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: change region query load metrics from gauge to counter Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from `IntGaugeVec` to `IntCounterVec` since these values are monotonically increasing and do not need gauge semantics. Update corresponding `add` calls to `inc_by` in merge scan reporting. Files: - `src/store-api/src/metrics.rs` — metric type and label changes - `src/query/src/dist_plan/merge_scan.rs` — caller adaptation Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: pass ReadItem directly to report_region_query_load Move `region_scan_load` call to the caller, so `report_region_query_load` accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`. - `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call, remove stale test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: ensure region query load is reported on MergeScanExec drop Remove the `enable_per_region_metrics` parameter from `report_region_query_load` so region load metrics are always emitted. Add a `Drop` impl for `MergeScanExec` that reports sub-stage metrics when the executor is dropped, covering edge cases where per-region metric emission was missed. Add a unit test verifying CPU time and scanned bytes are recorded on drop. Affected file: `src/query/src/dist_plan/merge_scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: gate region query load reporting Guard drop-time region query load reporting with the configured per-region metrics flag. Related files: - \`src/query/src/dist_plan/merge_scan.rs\` Symbols: - \`MergeScanExec::drop\` - \`enable_per_region_metrics\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clean region query load metrics on drop Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry. Related files: - \`src/mito2/src/region.rs\` Symbols: - \`MitoRegion::drop\` - \`REGION_QUERY_CPU_TIME\` - \`REGION_QUERY_SCANNED_BYTES\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, Huang <huanglei@qiyi.com> |
||
 LFC
|
270dce5ed7 |
feat: decouple region edit and compaction (#8272)
* feat: decouple region edit and compaction Signed-off-by: luofucong <luofc@foxmail.com> * make `schedule_compaction_after_edit` default to `true` Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Ning Sun
|
fd64ced4da |
feat: introduce plugin setup functions with richer context (#8256)
feat: enrich plugin setup context |
||
|
Lei, HUANG
|
e7ce3ac0c7 |
fix: accept JSONB OTLP logs against existing schema (#8250)
* fix: accept JSONB OTLP logs against existing schema Handle existing OTLP log JSONB columns that round-trip as Json while direct log rows carry pre-encoded binary JSONB payloads. Add integration coverage for repeated OTLP log batches after table auto-creation. Files: src/servers/src/otlp/logs.rs, tests-integration/tests/http.rs Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf: cache OTLP log JSONB schema check Cache whether an existing OTLP log column is legacy JSONB once per column before coercing rows. Files: src/servers/src/otlp/logs.rs Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: narrow down schema coercion Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
shuiyisong
|
d26a80855b |
chore: align OTLP logs with existing table schemas (#8229)
* chore: otlp logs with schema align Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
dennis zhuang
|
31c2c1f6db |
feat: table semantic layer per-table enrichment (Phase 2) (#8218)
* feat: table semantic layer per-table enrichment (Phase 2)
Phase 2 of the table semantic layer, plus a vocabulary trim so the layer only
records what a machine consumer cannot cheaply recover on its own.
Per-table metric enrichment (OTLP), via an internal per-table channel:
- A `SemanticIndex` accumulator records, per emitted table, the declared metric
keys: type / unit / temporality / metadata_quality=declared / original_name.
Conflicting single-valued keys collapse to `mixed`/`unknown`.
- Recording happens at the `encode_metrics` level where the base name, metric
type, and proto fields are all in scope, so histogram/summary fan-out gets the
correct per-subtable type (`_bucket`=histogram, `_sum`/`_count`=counter)
without threading state through every encoder.
- The index is serialized onto the `greptime.internal.semantic.per_table_index`
context extension; `apply_per_table_semantic_options` folds each table's keys
into its options at auto-create time.
- `trace.conventions` is refined from the request's resource/scope `schema_url`s
(concrete when uniform, else `mixed`/`unknown`).
Vocabulary trimmed to only meaningful keys. Kept: signal_type, source, pipeline,
trace.conventions, metric.{type,unit,temporality,metadata_quality,original_name}.
Dropped: metric.monotonic (a function of type), trace.has_events/has_links
(constant + derivable from columns), log.severity_scheme/body_format (constant /
derivable, and body_format cost an O(rows) scan), resource/scope lineage
(restates columns / collector-config concern), source_version (no cheap
non-constant value today). Prometheus carries type/unit in the metric name by
convention, so it gets identity only — no inferred enrichment.
Identity (signal_type + source) extended to the remaining ingest protocols so
the discovery view is complete: InfluxDB and OpenTSDB (metric), Loki and
Elasticsearch (log). These protocols carry no type/unit metadata, so identity is
all that applies.
Tests: unit coverage for the accumulator, per-metric-type fan-out, and trace
conventions; integration goldens updated for the OTLP metric/trace SHOW CREATE
output and the new Loki identity.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: validate the option value
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
|
dennis zhuang
|
f25ab67af1 |
feat: table semantic layer identity (Phase 1) (#8210)
* feat: table semantic layer identity (Phase 1) Attach a thin layer of semantic metadata to ingested tables via the existing `table_options` slot, so machine consumers (LLM agents, alert/dashboard builders, MCP servers, ETL) can align a table with the observability concept it stands for without guessing from column names. See docs/rfcs/2026-05-28-table-semantic-layer.md. Phase 1 (identity) only: - New `table::requests::semantic` module: the `greptime.semantic.*` vocabulary (signal/source/source_version/pipeline + trace/metric/log/resource-scope keys, defined now, populated by later phases), value constants, the internal `greptime.internal.semantic.per_table_index` transport key (reserved for Phase 2, deliberately outside the public namespace), and `is_semantic_option_key`. - `validate_table_option` accepts the `greptime.semantic.*` prefix, so the keys are valid both on the auto-create path and on explicit `CREATE TABLE ... WITH (...)`. - `fill_table_options_for_create` copies every semantic ctx extension into the new table's options (prefix passthrough alongside the fixed allowlist). - Frontend stamps identity on the context at each ingest entry: OTLP metrics (metric/opentelemetry), traces (+pipeline, has_events/has_links/conventions for the v1 model), logs (log/opentelemetry), and Prometheus remote write (metric/prometheus, metadata_quality=inferred). OTLP metric metadata_quality is left for Phase 2 (declared). - Trace identity is stamped only on the main span table; the derived `_services` / `_operations` lookup tables keep the unstamped context and carry no semantic identity (cross-table relationships are out of scope). Semantic options appear in SHOW CREATE TABLE (like table_data_model / otlp_metric_compat) and in information_schema, so an LLM inspecting a table sees its semantics directly. Tests: unit (validation prefix + internal-key rejection, ctx passthrough) and integration assertions that the common keys land for OTLP metrics (metric-engine logical table), traces, logs, and Prometheus remote write; SHOW CREATE goldens updated. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: prom batcher not cover and white list for semantic keys/values Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: typo Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
28fd796f4e |
fix(flow): harden incremental read correctness (#8196)
* fix(flow): harden incremental read correctness Signed-off-by: discord9 <discord9@163.com> * fix(flow): propagate dirty window options Signed-off-by: discord9 <discord9@163.com> * test: more Signed-off-by: discord9 <discord9@163.com> * chore: test config api Signed-off-by: discord9 <discord9@163.com> * refactor: split gen Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * fix: allowlist key Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
dennis zhuang
|
ed9312f8e3 |
feat: global switch for creating tables automatically (#8203)
* feat: global switch for creating table automatically Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: make auto_create_table as comment by default Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: respect gloabl switch for metric engine Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
e1e75b3ffe |
feat: implement a cache for the prefilter (#8102)
* feat: cache parquet prefilter results Signed-off-by: evenyag <realevenyag@gmail.com> * chore: set result cache size Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename is_stable to is_immutable and reject ScalarVariable Signed-off-by: evenyag <realevenyag@gmail.com> * chore: typo Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use capacity() for prefilter key memory accounting Signed-off-by: evenyag <realevenyag@gmail.com> * feat: per filter cache Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: support other variants in MaybeFilter Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: split compute_projection_mask Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: build_prefilter_masks takes PrefilterEntry Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
f8df016623 |
feat: add InfluxDB default merge mode config (#8134)
* feat/influxdb-default-merge-mode: add InfluxDB merge mode config - `influxdb` config: add `default_merge_mode` parsing and defaults in `src/frontend/src/service_config/influxdb.rs` and `src/frontend/src/service_config.rs` - auto-create behavior: apply configured `merge_mode` for InfluxDB ingestion in `src/frontend/src/instance.rs`, `src/frontend/src/instance/builder.rs`, `src/frontend/src/instance/influxdb.rs`, and `src/operator/src/insert.rs` - config docs: document `influxdb.default_merge_mode` in `config/frontend.example.toml`, `config/standalone.example.toml`, and `config/config.md` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/influxdb-default-merge-mode: derive merge mode default - `influxdb` config: derive `Default` for `InfluxdbMergeMode` in `src/frontend/src/service_config/influxdb.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/influxdb-default-merge-mode: update config API snapshot - `config API`: include `default_merge_mode` in `tests-integration/tests/http.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/influxdb-default-merge-mode: avoid default context clone - `InfluxDB merge mode`: avoid cloning `QueryContext` for default `last_non_null` in `src/frontend/src/instance/influxdb.rs` - `InfluxDB merge mode`: cover default, configured, and explicit `MERGE_MODE_KEY` paths in `src/frontend/src/instance/influxdb.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
shuiyisong
|
d4cfbd3400 |
feat: add otlp to prometheus naming translation options (#8113)
* chore: extract otlp translator Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: add header parsing Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: simplify and merge tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
7840aa1bb4 |
refactor(mito2)!: remove PartitionTreeMemtable (#8080)
* feat: switch partition tree to bulk Signed-off-by: evenyag <realevenyag@gmail.com> * chore: keep partition tree memtable for migration test Restore PartitionTreeMemtable construction when memtable.type=partition_tree is explicit, and move the sparse-encoding bulk override into the default (no explicit memtable.type) arm so phase 2's memtable.type=bulk wins on reopen. Rewrite test_reopen_time_series_sparse_memtable_with_bulk to use a metric-engine-shaped schema and sparse-encoded rows with WriteHint::Sparse, so the test actually exercises a PartitionTreeMemtable in phase 1 and verifies WAL replay into the new BulkMemtable on reopen without flushing. Signed-off-by: evenyag <realevenyag@gmail.com> * chore: drop partition tree memtable from runtime Re-apply the unconditional sparse-encoding override in `MemtableBuilderProvider::builder_for_options` and route the `MemtableOptions::PartitionTree` arm to `BulkMemtable` with a deprecation warning. After this change, `PartitionTreeMemtableBuilder` is no longer reachable from the engine runtime; benchmarks still reference the type. Remove `test_reopen_time_series_sparse_memtable_with_bulk` and the `put_sparse_rows` helper added in the previous commit — that test only existed to validate the PartitionTree -> Bulk reopen migration and is unnecessary now that the override is in place. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): move timestamp_array_to_i64_slice into read module Relocate the timestamp_array_to_i64_slice helper from memtable/partition_tree/data.rs to the read module so that the read path no longer depends on the partition_tree internals. All call sites (both inside and outside the partition_tree module) now import from crate::read. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): use TimeSeriesMemtableBuilder in time_partition tests The time_partition tests use the memtable builder purely as a generic backend for the TimePartitions write/scan paths; nothing in them is specific to the partition-tree memtable. Switch the seven affected tests to TimeSeriesMemtableBuilder so the tests no longer depend on PartitionTreeMemtableBuilder. Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): delete PartitionTreeMemtable implementation The runtime already falls back to BulkMemtable for the PartitionTree variant. Drop the now-unreachable implementation, its metrics, the partition_tree benchmarks, the metric-engine Unsupported fallback in bulk_insert.rs, and the test helpers that only existed for the deleted module. MemtableOptions::PartitionTree, its parsing, the runtime fallback, the store-api MEMTABLE_PARTITION_TREE_* constants, and the SQL fixtures remain so existing region options keep round-tripping. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito-codec): drop skip_partition_column parameter PartitionTreeMemtable was the only caller passing skip_partition_column=true; every other caller passes false. Now that the partition_tree module is gone, the parameter is uniformly false and the guard branch is dead. Drop the parameter from the trait method and both impls, remove the guard and the is_partition_column helper, and update the four remaining call sites in mito2 plus the bench. Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): remove unused MemtableConfig enum Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove unused variant Signed-off-by: evenyag <realevenyag@gmail.com> * test: update test_config_api Signed-off-by: evenyag <realevenyag@gmail.com> * fix: remove unused memtable test helpers Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address review comment Signed-off-by: evenyag <realevenyag@gmail.com> * fix: support bulk memtable options Signed-off-by: evenyag <realevenyag@gmail.com> * fix: sanitize config Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove partition tree options from region options Move primary_key_encoding to the top level Signed-off-by: evenyag <realevenyag@gmail.com> * test: make ssts test datetime replaced text stable Signed-off-by: evenyag <realevenyag@gmail.com> * test: update sqlness result Signed-off-by: evenyag <realevenyag@gmail.com> * chore: validate_enum_options consider bulk memtable Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: pass region id when parsing region options Replace the `TryFrom<&HashMap>` impl for `RegionOptions` with `try_from_options(region_id, options_map)` so the legacy partition_tree fallback can log the affected region. The fallback now also overrides the SST format to flat in addition to clearing the memtable type. Signed-off-by: evenyag <realevenyag@gmail.com> * fix: align sst_format with bulk memtable on parse and open Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |