mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-09-06 13:38:59 +00:00
1409e66837d89ff7f9d7d0604dfdbba3ca9fc7a7
1073
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Weny Xu
|
1409e66837 |
refactor(flight): add request builder and defer DoGet execution (#8953)
* refactor: add flight request builder Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use flight request builder in flow Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: defer frontend flight query execution Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grpc): avoid cloning requests during auth Signed-off-by: WenyXu <wenymedia@gmail.com> * test(grpc): cover flight request timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(client): share Flight message reader Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(flow): add Flight DoGet timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(client): gate Flight DDL helpers for testing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): restore Flight stream error semantics Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(grpc): document Flight stream input constructors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(flight): preserve deferred stream context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): use Flight stream SNAFU context Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 dennis zhuang
|
6d86e6ff06 |
feat: synthesize OTLP resource descriptor for the semantic entity graph (#8904)
* fix(servers): compose OTLP metrics job from service.namespace/service.name
The OTel Prometheus compatibility spec defines job as
"<service.namespace>/<service.name>" when the namespace is present.
The OTLP metrics path only used the bare service.name, so the job tag
diverged from target_info produced by Prometheus-side exporters for the
same resource. Compose the namespace form, and keep not fabricating a
job when service.name is absent.
Behavior change: resources carrying service.namespace now get
"namespace/name" as their job tag value.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(otlp): synthesize otel_resource_info at OTLP metrics ingestion
Ordinary OTLP metrics scatter filtered resource attributes as tags over
every logical metric table, so metrics-only services contribute nothing
to the semantic entity graph. Each request now also projects its
distinct resources into one info-metric-shaped mito table,
otel_resource_info: a fixed allowlist of identity-relevant attributes
under their raw OTel keys (independent of the label translation
strategy and the promote/ignore headers) plus derived job/instance
compatibility columns, value 1.0, and the newest data-point timestamp.
The descriptor is written after the main insert is committed; a failure
there (conflicting pre-existing table, auto-create disabled) degrades
to an OTLP partial_success warning with rejected_data_points = 0
instead of failing the request and triggering client retries of
already-accepted data. A request writing a metric named
otel_resource_info suppresses synthesis. Legacy mode is unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): otel info-metric conventions with host/container entities
Whitelist the ingestion-synthesized otel_resource_info descriptor via a
new otel_info_metrics conventions map, gated on source=opentelemetry
(the existing gate hardcoded source=prometheus). Its declarations use
explicit descriptive lists instead of descriptive_rest so identifying
attributes of other entities do not leak into service.instance.
Conventions tightened per the Astronomy Shop findings: host identity is
host.id with host.name descriptive only (host.name is not stable across
SDKs and resource detectors), a generic container entity (new entity
type) is declared only when container.id is present, and trace-v1
tables now synthesize host/container from their flattened resource
attributes too. New co-declared edges: service.instance runs_on
container, container runs_on host.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): cover the resource descriptor in integration tests
Covers the descriptor's raw-key columns and info-metric options through
the HTTP path, the namespace/name job composition end-to-end, column
names being independent of the translation strategy, the allowlist
excluding unlisted resource attributes, auto-create after a drop, the
metric-name collision suppressing synthesis, and the partial-success
warning (rejected_data_points = 0) when a pre-existing incompatible
table fails the descriptor write while metric data is accepted.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: cargo fmt
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(frontend): degrade descriptor permission denial to a warning
A table-level permission policy denying otel_resource_info would have
failed the whole OTLP metrics request because the descriptor's
permission check ran before the main insert. The descriptor is derived
enrichment: check its permission in the degrade path so a denial skips
the write and surfaces as the partial-success warning, like any other
descriptor write failure.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): guard descriptor writes with semantic ownership markers
A pre-existing schema-compatible table named otel_resource_info would
silently receive descriptor rows while its missing semantic stamps kept
it out of the entity graph. The descriptor write now requires the
auto-created table's ownership markers (mito engine + signal_type +
source + metric.type=info + metadata_quality=declared) and otherwise
degrades to the partial-success warning; the entity-graph gate for the
otel whitelist likewise requires metric.type=info, so a user table
stamped with only signal/source no longer picks up implicit
declarations.
Also fold the descriptor write cost into the response and surface the
degrade warning through the otel-arrow BatchStatus status_message.
Integration tests pin the full marker set on auto-create and that an
existing owned descriptor keeps accepting writes without degrading —
a missing marker would otherwise silently stop every descriptor write
after the first request.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(otlp): build descriptor rows without the per-resource BTreeMap
Projecting a resource allocated a BTreeMap and then collected it into the
row key, and every attribute was matched against the allowlist by linear
scan. Collect the tags into a Vec and sort once, and match the allowlist
instead of scanning it. Measured on the conversion path: descriptor work
drops 16-18%, from 10.6% to 8.9% of conversion CPU on the worst shape
(1000 resources with 4 data points each), where the cost tracks resource
count rather than data-point count.
Also trims the comments and tests added with the descriptor to what
carries information.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): pin the descriptor permission-denial degrade path
A policy denying the descriptor table must not fail the metrics request,
which the fix in
|
||
|
Weny Xu
|
28398138ec |
fix(flight): bound DoGet response wait (#8943)
* fix(flight): defer datanode query initialization Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): retain Flight stream peer context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): improve Flight stream diagnostics Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 shuiyisong
|
04614175fe |
refactor: centralize native histogram encoding in common-query (#8945)
* refactor: centralize native histogram encoding in common-query Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 discord9
|
3493d2d0fb |
perf(servers)!: speed up Prometheus JSON response building with ryu and per-series entry reuse (#8815)
* fix(perf): align direct-SST CREATE TABLE with baked index metadata
The offline fixture generator (query_perf_fixture::direct_sst::
build_region_metadata) bakes greptime:inverted_index /
greptime:skipping_index field metadata into the region manifest for
tag/field columns, but create_table_sql emitted a bare CREATE TABLE
without those declarations. MergeScan's remote-schema validation then
failed on any tag/field projection (HTTP 500 'advertised remote stream
schema field mismatch'), breaking direct_readable_sst perf cases.
CREATE TABLE now declares the matching SKIPPING INDEX WITH
(granularity='1') / INVERTED INDEX column options. A round-trip test
proves the emitted SQL is parser-valid and yields the exact catalog
metadata.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): speed up Prometheus JSON response building with ryu and per-series entry reuse
PrometheusJsonResponse::record_batches_to_data spends ~47% of its CPU
in f64::to_string() per sample and ~32% in IndexMap::entry() per row
(60s profile of concurrent query_range workloads, ~800k series).
- Replace f64::to_string() with ryu::Buffer::format_finite for finite
values (shortest round-trip, 3-5x faster); NaN/+Inf/-Inf keep the
previous std formatting so wire output is unchanged.
- Remember the previous row's label vector and entry index; query output
is clustered by series, so consecutive rows reuse the same IndexMap
entry via get_index_mut instead of rebuilding and hashing the label
vector (worst case adds one Vec comparison per series transition).
Also adds a query-regression case (prom_json_response) that measures the
real Prometheus HTTP range API path (/v1/prometheus/api/v1/query_range),
which is the only frontend path that builds the Prometheus JSON response
(TQL ANALYZE formats the SQL JSON shape instead), plus a prom_http query
kind in the regression runner.
Perf (aligned base
|
||
|
shuiyisong
|
1c5eabcbbf |
feat(otlp): support cumulative exponential histograms (#8900)
* feat: implement exponential histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove duplicate tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): enforce exponential histogram ingestion safety Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update rfc Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): remove protocol-coupled histogram checks Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): reuse native histogram schema across data points Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: merge repeated OTLP histogram fragments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): build rejection messages lazily Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: add doc Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 Ning Sun
|
174577164a |
feat: otlp duration_nano and trace_flag signed integer coercion (#8816)
* feat(servers): add signed→unsigned int coercion to OTLP ingest path Phase 0 of transitioning built-in data models from unsigned to signed integers (#8793): add lossless Int64→UInt64 and Int32→UInt32 coercion arms so existing UInt64/UInt32 columns (e.g. trace `duration_nano`, log `trace_flags`) keep accepting new signed ingest without an ALTER TABLE. The OTLP ingest path already reconciles every incoming column against the existing table schema and treats it as authoritative. With these arms, `choose_trace_reconcile_decision` returns `UseExisting(UInt64/Uint32)` for an existing unsigned column receiving signed data: the table keeps its type byte-for-byte and the request value is coerced. No persisted format is mutated; existing data stays readable as-is. This is the safety net that makes the actual schema flip (Phase 1) safe. Only the signed→unsigned direction is supported — the reverse would be lossy for values above the signed range and is intentionally rejected. Tests cover both new arms plus an end-to-end log test proving an existing UInt64 column coerces an incoming Int64 value while keeping its type. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: add compatibility layer for uint trace/log fields Signed-off-by: Ning Sun <sunning@greptime.com> * fix: jaeger test Signed-off-by: Ning Sun <sunning@greptime.com> * fix: keep trace v0 unsigned, reject negative span durations - Keep the frozen v0 data model on UInt64 duration_nano: the signed ingest compatibility layer only runs on the v1 path, so flipping v0 would break writes into every pre-existing v0 table at mito's schema check. Pin the schema with unit and integration tests. - Reject spans whose end precedes their start (or whose duration does not fit i64) on the v1 path instead of wrapping: new Int64 tables and existing UInt64 tables now fail identically, rather than storing negative durations that break the Jaeger query API. - Extract is_supported_signed_to_unsigned_coercion so the trace and log ingest paths share one supported-pair predicate and cannot drift. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: clamp negative span durations to zero, revert semantic_graph comment - Record duration 0 for spans whose end precedes their start instead of erroring: a malformed span no longer fails the request, and the value written is always a non-negative, in-range i64 so new Int64 tables and existing UInt64 tables (via the checked coercion) behave identically. Durations above i64::MAX saturate rather than wrap. - Revert the doc-comment tweak on the semantic_graph test fixture; the file is untouched by this PR again. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Ning Sun
|
8dbfbee611 | refactor: remove open metrics parser (#8905) | ||
 sun
|
b882e393df | feat: update dashboard to v0.13.13 (#8898) | ||
|
shuiyisong
|
072810159a |
chore: add v2 version label to prom metrics (#8885)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
sun
|
2eb5f593f8 | feat: update dashboard to v0.13.12 (#8882) | ||
|
shuiyisong
|
b1263fc65f |
perf: optimize Prometheus remote write v2 decoding (#8873)
* test: add bench for prom decode Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: merge v2 decode manually and add to bench Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: update v2 path Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: update v2 path Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: use constant Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Ning Sun
|
354921c80e |
chore: update mysql test drivers and lru (#8868)
* chore: update mysql test drivers and lru * fix: test |
||
|
discord9
|
9241e96fa2 |
fix(metric-engine): handle Utf8View tag/label columns without panicking (#8772)
* fix(metric-engine): handle Utf8View tag/label columns without panicking
label_replace (planned as DataFusion regexp_replace) coerces to Utf8View,
so label columns materialize as StringViewArray; build_tag_arrays'
StringArray downcast then panicked ('tag column must be utf8') — e.g. for
OTLP/json2 ingest. TSID computation, sparse-PK encoding and tag
extraction now accept generic ArrayRef tag columns (Utf8/LargeUtf8/
Utf8View/Dictionary) via string_array_value_at_index, and build_tag_arrays
errors instead of panicking on non-string columns. The mito2 time-series
memtable string-field paths are hardened the same way.
Adds label_replace_with_utf8view_labels_does_not_panic (issue #8732).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(metric-engine): add is_string_null_at helper for tag null checks
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(datatypes): use is_none_or to satisfy clippy unnecessary-map-or
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix: reject oversized string batches before memtable append
Distinguish a full active string builder from a batch that cannot fit an
empty Arrow string builder at all. Scan every string field so a later
intrinsically oversized field cannot be skipped after an earlier field
requests a freeze. Return InvalidBatch instead of reaching Arrow's offset
overflow panic.
Also cover Utf8View tags with nulls through the metric-engine tag, TSID,
and sparse-primary-key path.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
|
||
|
shuiyisong
|
3b3a9032e0 |
feat(servers): expose native histograms over Prometheus HTTP (#8850)
* feat(servers): expose native histograms over Prometheus HTTP Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(servers): refine Prometheus HTTP metadata handling Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(servers): expand Prometheus metadata coverage Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(servers): return OpenMetrics units from metadata API Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Ning Sun
|
3510ef7d4c |
feat!: update native histogram unsigned int types (#8824)
* feat(native-histogram): store counts and span lengths as signed integers Native histograms are unreleased, so the on-disk integer payload columns are switched from unsigned to signed types without backward-compat: - count_u64 / zero_count_u64: uint64 -> int64 - positive_span_lengths / negative_span_lengths: list(uint32) -> list(int32) - Span.length (query-time model): u32 -> i32 The Prometheus remote-write v2 source carries these as uint64/uint32, so the unsigned->signed conversion at the ingestion boundary is overflow checked: an integer count >= 2^63 or a span length >= 2^31 is rejected with an explicit error rather than silently wrapping to a negative value. read_spans additionally rejects negative stored lengths to keep the non-negative invariant sound for downstream `as usize` casts. The UDAF accumulator's own observation counter (transient aggregation state, not part of the persisted histogram value) is intentionally left as uint64. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(native-histogram): rename count/zero_count fields to _i64 Now that the integer payload columns are stored as int64, rename the field constants and persisted names to match: COUNT_U64_FIELD ("count_u64") -> COUNT_I64_FIELD ("count_i64") ZERO_COUNT_U64_FIELD ("zero_count_u64") -> ZERO_COUNT_I64_FIELD ("zero_count_i64") The local builder variables and the docs/JSON snapshot are updated to match. No backward-compat (unreleased feature). Signed-off-by: Ning Sun <sunning@greptime.com> * test(native-histogram): refresh planner plan snapshot for signed types The mixed native-histogram range test embeds the full histogram Struct type in its expected plan string, which still carried the pre-rename unsigned fields. Update the snapshot to match the signed schema: positive/negative_span_lengths: List(UInt32) -> List(Int32) count_u64/zero_count_u64: UInt64 -> count_i64/zero_count_i64: Int64 Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
dennis zhuang
|
4eeb4052d6 |
feat(servers): stamp prometheus remote write v2 metadata as semantic table options (#8797)
* feat(servers): stamp prometheus remote write v2 metadata as semantic table options
Remote write 2.0 carries per-series metadata (type, unit, help) that the
v2 ingest decoded and dropped; tables kept the name-based 'inferred'
quality. Wire it into the semantic layer:
- generalize the OTLP per-table semantic index into a shared, schema-
aware servers::semantic module: v2 lets each series override its
target schema, so the index is keyed {schema -> table -> options} and
the same metric name in two schemas no longer collapses;
- into_write_requests records metric type and unit per written table;
an explicit type upgrades the table's metadata quality to declared,
UNSPECIFIED series keep the request-level inferred stamp, and units
are canonicalised from OpenMetrics words to the UCUM codes the
vocabulary is defined in (unknown units are dropped, help text is not
persisted);
- the consumer folds the index in on both auto-create paths: the
operator row-insert path and the pending-rows batched create, which
bypasses the former.
Table options are stamped at auto-create only; updating existing tables
from later metadata is future work (a metadata registry, see the native
histograms RFC).
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim over-commenting in the remote write metadata path
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: parse the per-table semantic index once per create round
The index was re-parsed from JSON for every table being created — a
first write creating N tables (a fleet's first scrape) paid
O(N x index size). Parsing now happens lazily once per create-planning
round, on both consumers: the operator row-insert auto-create (also
serving OTLP metrics) and the pending-rows batched create.
Also validate every non-zero metadata symbol reference up front, as the
remote write 2.0 spec requires: help_ref was never checked, and
unit_ref escaped checking when the metric type was UNSPECIFIED.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): stamp remote-write v2 unit independently of metric type
OpenMetrics models TYPE and UNIT as independent MetricFamily metadata, and
the Prometheus v2 sender emits UNSPECIFIED-type series that still carry a
unit. The early return on UNSPECIFIED dropped that unit, which is
unrecoverable after table auto-create (units are not stored in rows).
Stamp the mapped unit whenever present; the type and the
metadata_quality=declared upgrade still require an explicit type.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: trim restating comments in the v2 metadata path
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
|
Ning Sun
|
9fcfeff9ce |
feat(auth): support HTTP bearer-token authentication (#8719)
* feat(auth): support HTTP bearer-token authentication (#8718) Adds an opt-in bearer-token (JWT / OAuth2) authentication path to the HTTP layer, so clients can authenticate with `Authorization: Bearer <token>` against any `/v1/` interface. Today such requests are rejected with `UnsupportedAuthScheme("bearer")` -> 401 before any handler runs. The token is treated as opaque by the server; validation and identity derivation stay in the UserProvider, so JWT/JWKS/OIDC policy remains pluggable and out of core. Changes: - `auth::UserProvider` gains `auth_token(token, catalog, schema) -> Result<UserInfoRef>` with a default that rejects (`Error::UnsupportedAuthMethod`), so password-only providers keep today's behavior. A provider that supports token auth overrides it to validate the token, resolve it to a user, and authorize the connection. - `auth::Error::UnsupportedAuthMethod` for the default-reject case. - `servers::http::authorize::inner_auth` extracts a bearer token (`extract_bearer_token`) and, when present, authenticates via `UserProvider::auth_token`; otherwise it falls through unchanged to the username/password path (Basic / influxdb / splunk). Basic and bearer coexist on the same server. Backward compatible: the default impl preserves existing behavior, and non-bearer requests take the exact same path as before. Tests: - `extract_bearer_token` recognizes `Bearer` (either header) and ignores Basic/Token/Splunk/empty. - `inner_auth` dispatches a bearer token to `auth_token` and populates the QueryContext user on success; rejects on failure. - A password-only provider (default `auth_token`) rejects bearer tokens. Refs: #8718 * chore: fmt * refactor(auth): address bearer-auth review feedback (#8719) Address the review comments on the HTTP bearer-token authentication PR: - Match the `Bearer` scheme case-insensitively (RFC 9110 §11.1) via `eq_ignore_ascii_case`. `extract_bearer_token` previously only accepted `Bearer`/`bearer`, so a valid `BEARER <token>` fell through to `UnsupportedAuthScheme` and never reached the provider. The opaque token itself is deliberately not lowercased. - Return `Option<&str>` (borrowing the request headers) instead of `Option<String>`, avoiding an allocation per bearer request. - Rename `UserProvider::auth_token` -> `auth_bearer_token` for clarity. - Route bearer-auth failures on Splunk HEC requests through `splunk_hec_err` (FORBIDDEN, code 4) instead of the generic 401 `ErrorResponse`, so HEC clients retain their `{"text":"Invalid token","code":4}` endpoint contract. Adds test coverage for case-insensitive scheme parsing (token preserved verbatim) and a regression test for the bearer/splunk routing path. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Ning Sun
|
606e5fd6a0 |
feat: update opentelemetry family to 0.32 series (#8776)
* feat: update openetelemtry family to 0.32 series * chore: resolve warning * fix: update tests |
||
|
dennis zhuang
|
5d4699db1c |
docs: refine coding agent maps (#8790)
* docs: refine coding agent maps Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: trim license header guidance Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: update README links and project status Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
  Gaurav ShokeenandLei, HUANG
|
d90cca4b75 |
fix(prometheus): custom column remote reads (#8659)
* fix(prometheus): custom column remote reads Resolve timestamp and value column names from the table schema and carry them through query planning and result conversion. Add a remote-read regression test covering custom_ts and custom_value. Signed-off-by: grezzko <me@gauravshokeen.com> * fix: resolve remote-read value columns safely Prefer the sole field for custom schemas and greptime_value for multi-field tables. Reject ambiguous schemas and add regression tests. Signed-off-by: grezzko <me@gauravshokeen.com> --------- Signed-off-by: grezzko <me@gauravshokeen.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Co-authored-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
shuiyisong
|
ec51113ab6 |
feat(protocol): validate native histogram ingestion (#8775)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
shuiyisong
|
9fab11271b |
perf: reduce ingestion and flat-merge overhead (#8778)
chore: minor performance fix Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
dennis zhuang
|
3d12273c84 |
feat: read-time entity relationships graph over telemetry (M0+M1) (#8614)
* feat(table): add entity semantic declarations Define open-ended greptime.semantic.entity.* options, validate entity columns at DDL time, and stamp OTLP trace tables with the service entity declaration. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: add read-time entity relationships graph Add computed semantic graph tables, typed DataFusion derivation plans for entity registry and trace calls edges, and streaming read-time execution. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: exclude semantic graph tables from table constraints Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(operator): name the plan-builder source groupings Review feedback: build_registry_plan / build_calls_plan took anonymous (declarations, DataFrame) tuples while the caller already grouped the same fields. Introduce RegistrySource { declarations, scan } and CallsSource { service, scan } next to the builders and flow them through the frontend caller and tests. The frontend-side EntitySource keeps holding a TableRef (the operator builders stay pure over already-built scans), so the named structs live in operator rather than reusing that type. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
+4   
|
cabc2f6cc6 |
feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (#7987) (#8392)
* feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987) Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987) Signed-off-by: onepizzateam <palakjha916@gmail.com> * test(flow): add sqlness golden result for flow_status Signed-off-by: Palak Jha <palakjha916@gmail.com> * fix(catalog): remove unused OptionExt import in flow_statistics Signed-off-by: onepizzateam <palakjha916@gmail.com> * docs(flow): fix stale 'recent errors' comment on QueryFlowExecStats Signed-off-by: onepizzateam <palakjha916@gmail.com> * test: regenerate golden results for flow_statistics table Signed-off-by: onepizzateam <palakjha916@gmail.com> * style: apply rustfmt to flow_statistics changes Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor(catalog): hoist current_time_millis out of flow loop and clamp uptime Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore: remove accidentally committed fmt_check.log Signed-off-by: onepizzateam <palakjha916@gmail.com> * Update flow_status.result del eof trailing blank line as per review Signed-off-by: onepizzateam <palakjha916@gmail.com> * test(flow): restore runner-generated trailing blank line for sqlness Signed-off-by: onepizzateam <palakjha916@gmail.com> * postgres: include SHOW FLOW STATUS in extended-query describe (return flow_statistics fields) Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: address reviewer feedback on flow_statistics PR Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): resolve merge conflicts with main Signed-off-by: polar <palakjha916@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * sqlness check post gen (information_schema.result) Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): record start_time after req/snapshot_seqs built, before dispatch Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(sql): handle ShowFlowStatus in match statement at util.rs Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: review patch implementation Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore: remove accidentally committed local tool output files and fix fmt Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix worker.rs return type formatting Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): apply rustfmt to get_full_flow_stat return type Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix worker.rs return type formatting Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(flow): re-apply rustfmt to get_full_flow_stat return type after merge Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: merge conflicts Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(auth): warn when credential load disables Postgres SCRAM or drops a line (#8652) * fix(auth): warn when credential load disables Postgres SCRAM or drops a line Static and watch user providers degraded silently in two ways: - A single non-SCRAM verifier (mysql_native_password, or a legacy pbkdf2_sha256 hash that predates SCRAM) disables Postgres SCRAM for every user and falls back to cleartext, with no signal to the operator. - A malformed credential line (commonly a plaintext password containing '=', which splits into more than two parts) was dropped without a trace. Emit a warning at each credential load for both cases so operators don't unknowingly serve cleartext passwords over Postgres or lose a user. This is logging only; authentication behavior is unchanged. The SCRAM check never logs secrets, and the malformed-line warning logs the line number and file, never the line content. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): warn on credential file read error before truncating A read error from lines() (I/O failure or invalid UTF-8) ends the iterator via map_while, silently dropping every remaining credential. Warn with the line number and file before truncating, matching the malformed-line handling, so the drop is observable. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * test(object-store): fix racy SecureFs abort test (#8720) test_writer_abort_is_unsupported_without_atomic_write asserted the file content immediately after abort() returned Unsupported. SecureFsWriter writes through tokio::fs::File, whose write_all() only enqueues a blocking write task (tokio's poll_write returns Ready before the write completes), so the data may not be visible yet when the test reads the file. Drop the race-prone content assertion and only verify the Unsupported contract. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(query): plan native histogram functions (#8705) * feat(query): plan native histogram functions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue & add tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * perf(promql): avoid repeated scans in sliding range evaluation (#8646) * perf(promql): use two pointers for sliding range boundaries Replace the stale cursor heuristic in RangeManipulateStream::calculate_range with monotonic left/right cursors. The old path rescanned each evaluation window (O(E x samples-per-window)) and could lose valid samples after sparse gaps or trailing empty windows. The two pointers keep strict monotonic progress, reducing boundary generation to O(N + E) while preserving (curr-range, curr] semantics, start/end shortening, and empty-window output. Controlled release benchmarks (fixed CPU, ABBA): - Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s, ~96% at 1h/15s. - Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end; shorter windows stayed within run-order noise. Signed-off-by: discord9 <discord9@163.com> * perf(promql): specialize changes/resets with adaptive edge counting The generic range_fn macro slices, downcasts, and rescans every overlapping window for changes() and resets(). Replace the macro path for these two functions with hand-written UDF wrappers backed by a shared private edge-count kernel: direct raw-offset scans when requested edges are few, otherwise one global u64 edge prefix so each window is answered by a prefix difference. Behavior is preserved bit-for-bit, including raw null-buffer values, NaN semantics, signed zero, infinities, empty/singleton windows, independent timestamp/value offsets, arbitrary window layouts, and exact DataFusion error messages. The shared proc macro, planner, serializer, and other range functions are untouched. Controlled release benchmarks (fixed CPU, ABBA): - Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time. - Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster. - Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client and 12.0-20.9% server latency improvement; controls stayed within drift. Signed-off-by: discord9 <discord9@163.com> * ci(query-regression): include PromQL range boundary case in defaults An audit of historical query-regression runs found zero range-query coverage: all 208 PromQL ANALYZE samples were bare selectors, so range evaluation could regress without CI noticing. Wire the promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so label-triggered runs measure the range path. The case is cheap: a ~0.3s synthetic fixture and about a minute of query execution per base/candidate pass. Signed-off-by: discord9 <discord9@163.com> * chore(promql): address sliding range review nits Move test-only imports into their test modules and remove the unused pre-specialization changes and resets helpers. Signed-off-by: discord9 <discord9@163.com> * style(promql): apply pinned rustfmt Signed-off-by: discord9 <discord9@163.com> * test(promql): cover sparse range results Share the changes and resets test scaffolding while keeping their behavior oracles independent. Add an end-to-end sqlness regression for sparse samples, empty intermediate windows, and a valid trailing sample. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * ci: optimize fuzz and split workflows (#8710) * ci: batch fuzz targets in GitHub Actions Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: improve fuzz test observability Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): preserve fuzz setup failure artifacts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(ci): keep fuzz mock output in logs Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: optimize fuzz worker cache Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: warm fuzz target binaries Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: isolate fuzz workflow Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: centralize fuzz target preparation Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: split general workflows Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: streamline docs required checks Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: transfer fuzz targets as artifacts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve fuzz binary permissions Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: streamline fuzz workers Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: cache PR build dependencies Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: retain main build cache policy Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: address fuzz review feedback Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: add public constructor for compactor (#8724) Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(logging): add enable_file_logging option to disable file logging (#8721) Signed-off-by: xhwhis <hi@whis.me> Signed-off-by: onepizzateam <palakjha916@gmail.com> * avoid cloning final Prometheus remote write row (#8733) perf: avoid cloning final Prometheus remote write row Signed-off-by: lyang24 <lanqingy93@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(function): add json_object_keys scalar function (#8722) Expose JSON object key listing for outermost objects, with sqlness coverage. Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor(mito2): revise compaction trigger behavior (#8706) * refactor(mito2): revise compaction trigger behavior Distinguish automatic and manual triggers, coalesce explicit automatic follow-ups, and reject concurrent manual compactions. Remove implicit post-execution continuation and transient idle statuses so scheduler entries always represent an active lifecycle. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): track automatic compaction follow-ups Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): fix compaction transition rustdoc Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark manual compaction conflict retryable Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(mito2): drop unused RequestCancelResult::NotRunning variant request_cancel is only called in tests where the region is guaranteed to be running, so the NotRunning case was dead code. Simplify to unwrap() and remove the variant. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): gate test-only cancellation import Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize DDL after compaction planning Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: update dashboard to v0.13.11 (#8737) Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(object-store): skip removed-entry lister test on Windows (#8735) DirEntry on Windows is a snapshot from FindFirstFileW: file_type() and metadata() keep returning cached data after the file is removed, so read_list_entry() cannot observe the deletion. The test asserts the Unix behavior (lstat returns ENOENT) and fails deterministically on Windows nightly CI (4/4 tries). Gate it with #[cfg(not(windows))]. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(query): preserve remote dynamic filter target (#8615) * fix(query): preserve remote dynamic filter target Signed-off-by: discord9 <discord9@163.com> * fix(query): check RDF subscriber registration Signed-off-by: discord9 <discord9@163.com> * fix(query): refresh initial dyn filter snapshot before dispatch and handle RDF unregister The remote dynamic filter dispatch ordering regression: freeze the target, pre-register subscribers, refresh the initial snapshot, then dispatch. Also implement handle_remote_dyn_filter_unregister to keep unregister targets consistent with do_get/update. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): update test-only RegionQueryHandler impl to new trait signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * docs: rework README release badges, drop star history, fix grpc flag (#8743) * docs: show stable, latest and nightly version badges in README The single release badge rendered whatever GitHub considered newest, so a pre-release such as v1.2.0-beta.1 looked like the recommended version. Split it into three self-updating badges using the shields.io `filter` parameter, keyed off the existing tag naming: - stable: `!*-*` matches tags without a hyphen (v1.1.4) - latest: `!*-*-*` excludes nightly and dev builds (v1.2.0-beta.1) - nightly: `*-nightly-*` matches the weekly build (v1.2.0-nightly-20260706) No workflow changes are needed; the badges track new releases on their own. A one-line caption below them says which channel to pick. The release-date badge is dropped as the three version badges already carry that signal. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: remove star history chart from README The chart carried a sealed_token in three URLs and added a large third-party image to the Project Status section without saying anything the badges and case studies do not already cover. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: use --grpc-bind-addr in README quickstart --rpc-bind-addr is now only a hidden alias of --grpc-bind-addr and no longer shows up in --help. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * Update README.md Co-authored-by: Ning Sun <classicning@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Ning Sun <classicning@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore: check enterprise-gated files are listed in both license configs (#8750) * chore: check enterprise-gated files are listed in both license configs A file reachable only through `#[cfg(feature = "enterprise")] mod ...;` is governed by the GreptimeDB Enterprise License, so it must appear in the `includes` of licenserc-enterprise.toml and the `excludes` of licenserc.toml. hawkeye stays silent when it does not: the file keeps its Apache-2.0 header and passes the default check precisely because it was never excluded from it. scripts/check-enterprise-license.py walks enterprise-gated `mod` declarations, resolves them to files (submodules included) and diffs that set against both configs, also reporting stale entries. It runs in the license job in CI and as `make check-enterprise-license`. Documents the split it cannot decide for you — whole enterprise features get their own file, a gated match arm stays inline — in .agents/architecture-invariants.md. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: tighten enterprise license checks Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(operator): invalidate local cache after dropping view (#8748) Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * chore!: gate soft-drop table behind the enterprise feature (#8747) * chore: gate soft-drop table behind the enterprise feature Soft-drop table becomes an enterprise-only feature: - metasrv rejects gc.experimental_soft_drop.enable=true at startup in non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled without the enterprise feature as a second line of defense - the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table() registration, and information_schema.recycle_bin registration are compiled out unless the enterprise feature is enabled - common-meta procedures, tombstone keys, and DdlTask serde stay unconditional for persisted-procedure recovery and wire compatibility - the [gc.experimental_soft_drop] section is removed from the OSS example config and generated docs (moving to the enterprise repo) - the soft-drop sqlness cases and their CI job are removed from OSS (moving to the enterprise repo); affected information_schema .result files are regenerated Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: limit unused_variables allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch accidental unused variables in register_admin_only. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: include the config key in the soft-drop enterprise gate error Addresses review comment: name gc.experimental_soft_drop.enable in the startup validation error so users can locate the setting quickly when it is set via env vars or layered config. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: limit unused_mut allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch unused mut in the table_ddl_event test setup. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: reject soft-drop DDL submissions in non-enterprise builds Addresses review comment: clients could bypass the SQL-level gates by submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly to the procedure service. Reject fresh submissions at the DdlManager boundary in non-enterprise builds while keeping the procedure loaders registered for crash recovery and wire compatibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: stop --enable-gc from enabling soft drop in the sqlness template Addresses review comment: the metasrv test template rendered [gc.experimental_soft_drop] enable = true under the generic --enable-gc flag, which non-enterprise metasrv now rejects at startup, making the documented --enable-gc mode unusable in OSS. Keep the flag scoped to plain GC; enterprise soft-drop coverage moves to the enterprise repo. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate fresh soft-drop procedures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: gate soft-drop fallback coverage Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate soft-drop procedure implementation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate drop table soft-drop behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate expired soft-drop gc behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: test enterprise table ddl lifecycle Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark purge_table as enterprise licensed The purge_table module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark recycle_bin as enterprise licensed The recycle_bin module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark soft-drop procedure sources as enterprise licensed The purge and undrop procedure implementations plus the recycle-bin test module compile only with the enterprise feature. Apply the Enterprise License header and register them with both license configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: make frontend heartbeat extensible and lifecycle-safe (#8726) * feat: make frontend heartbeat extensible and lifecycle-safe Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: isolate heartbeat extension response handlers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: cancel in-flight heartbeat response handling Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover heartbeat wire compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up failed heartbeat startup Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address frontend heartbeat review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(meta): release region guards after drop rollback (#8751) Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor!: move native histogram config and `prom_validation_mode` to prom_store (#8744) * chore: adjust the position of experimental_enable_prometheus_native_histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: move prom_validation_mode as well Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: add health-aware gRPC client routing (#8684) * feat: add gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: harden gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: defer gRPC client health checks until first use Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(mito2): discard unflushed region data safely (#8600) * feat: support discarding unflushed region data Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): wake stalled writers after discard Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): drop redundant manifest check for discarding unflushed data Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * docs: align wal.sync_period documented default with actual fallback (5s) (#8753) The example TOMLs and generated config.md documented the default of wal.sync_period as "10s", but since #5677 moved the WAL sync task to a background RepeatedTask, an unset sync_period falls back to 5s in RaftEngineLogStore. The two paths therefore had different fsync periods: deployments based on the example configs used 10s while bare configs used 5s. Align the documentation with the actual code behavior (5s) instead of changing the code fallback to 10s, so that no existing deployment silently gets a larger data-loss window on host power loss. - config/datanode.example.toml, config/standalone.example.toml: 10s -> 5s - config/config.md: regenerated via make config-docs - src/cmd/tests/load_config_test.rs: update assertions accordingly Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: support Utf8View labels in Prometheus response (#8754) Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix(query): validate merge scan remote schema (#8579) * fix(query): validate merge scan remote schema Signed-off-by: discord9 <discord9@163.com> * fix(query): treat JSON columns as schema-compatible across wire/decode forms CI (Sqlness json2_limit standalone + distributed) failed on the new remote-schema validation: a JSON column is Binary + extension metadata (ARROW:extension:name=greptime.json, greptime:type=Json) on the wire but decodes to Struct(...) with the extension metadata — validate_remote_schema compared raw arrow data_type and rejected it as a mismatch. Adds json_fields_compatible(): JSON fields are equal when name and nullability match, greptime:type matches, and the JSON2 settings (ARROW:extension:metadata type hints) match, ignoring the physical arrow type. Only JSON fields may bypass the raw-type comparison; non-JSON validation stays strict. Adds 4 regression tests mirroring the CI failure (wire-binary vs decoded-struct accepted both directions; different JSON2 settings rejected; JSON vs plain Binary rejected). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat(grafana): add events dashboard (#8725) * feat(grafana): add events dashboard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): tolerate evolving event schemas Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): address events dashboard review Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): restore events dashboard panels Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): bound events dashboard queries Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): include historical event catalogs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): preserve events drill-down context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): correct events lifecycle outcomes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): handle empty event type ranges Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): scope event catalogs to submissions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): handle empty events dashboard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grafana): refresh event schema variables Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * refactor: separate a json2 extension type (#8745) Signed-off-by: luofucong <luofc@foxmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * feat: add admin function registrar (#8762) * feat: add admin function registrar Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject admin function name collisions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: fix typo in admin function test Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: onepizzateam <palakjha916@gmail.com> * fix: information_schema.rs table initialization issue Signed-off-by: onepizzateam <palakjha916@gmail.com> * rustfmt fix Signed-off-by: onepizzateam <palakjha916@gmail.com> --------- Signed-off-by: onepizzateam <palakjha916@gmail.com> Signed-off-by: Palak Jha <palakjha916@gmail.com> Signed-off-by: polar <palakjha916@gmail.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: discord9 <discord9@163.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: xhwhis <hi@whis.me> Signed-off-by: lyang24 <lanqingy93@gmail.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: luofucong <luofc@foxmail.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com> Co-authored-by: discord9 <discord9@163.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> Co-authored-by: Ning Sun <sunng@protonmail.com> Co-authored-by: Whis Liao <xhwhis@gmail.com> Co-authored-by: Lanqing Yang <lanqingy93@gmail.com> Co-authored-by: sun <sunchang_long@163.com> Co-authored-by: Ning Sun <classicning@gmail.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me> Co-authored-by: Yingwen <realevenyag@gmail.com> Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com> |
||
 Yingwen
|
f25836a6ed |
fix: support Utf8View labels in Prometheus response (#8754)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
shuiyisong
|
aa72563783 |
refactor!: move native histogram config and prom_validation_mode to prom_store (#8744)
* chore: adjust the position of experimental_enable_prometheus_native_histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: move prom_validation_mode as well Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
sun
|
63664bf1aa | feat: update dashboard to v0.13.11 (#8737) | ||
|
Lanqing Yang
|
4159e1b55b |
avoid cloning final Prometheus remote write row (#8733)
perf: avoid cloning final Prometheus remote write row Signed-off-by: lyang24 <lanqingy93@gmail.com> |
||
|
shuiyisong
|
c3d8c976a2 |
feat(query): add native histogram result plumbing (#8693)
* feat(query): add native histogram result plumbing Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
sun
|
2ed3c3c43f | feat: update dashboard to v0.13.10 (#8687) | ||
|
shuiyisong
|
2afcd4a558 |
fix: enforce permissions for restricted HTTP endpoints (#8672)
* chore: add perm check Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
sun
|
1a24a6e782 | feat: update dashboard to v0.13.9 (#8674) | ||
|
discord9
|
26b08f815f |
fix(mysql): fail closed on unrepresentable timestamps (#8580)
* fix(mysql): reject unrepresentable timestamps Signed-off-by: discord9 <discord9@163.com> * fix(mysql): validate timestamp protocol boundaries Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Ning Sun
|
deb688f572 |
feat: add a dedicated http api server port (#8657)
* feat: add a dedicated http api server port * fix: integration test * refactor: make http-api-port opt-in * refactor: rename attribute to http-api-server * feat: use middleware to check different http server port * refactor: rename config option |
||
 LFC
|
4acd2340c7 |
feat: add extra http router provider in metasrv plugin (#8662)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
sun
|
cfa9e95c1d | feat: update dashboard to v0.13.8 (#8666) | ||
|
discord9
|
255572020a |
fix: scope live analyze metrics to streaming requests (#8644)
Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
5ad4e71007 |
fix(prometheus): make remote write timeout retryable (#8639)
* fix(prometheus): make remote write timeout retryable Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(prometheus): enforce pending row timeout budget Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(prometheus): skip pending-row timeout fallback when batcher is disabled PendingRowsBatcher::try_new returns None when max_batch_rows, max_concurrent_flushes, worker_channel_capacity or max_inflight_requests is zero, meaning remote writes bypass batching entirely. The timeout fallback predicate now mirrors these enablement conditions so the HTTP timeout is not raised when no request can wait for a pending-row flush. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(prometheus): skip pending-row timeout fallback in async batch mode With PENDING_ROWS_BATCH_SYNC=false, pending-row submissions return right after enqueue and no request waits for a flush, so raising the global HTTP timeout only delays unrelated routes. Export the batch sync mode predicate from the servers crate and consult it in the frontend's effective_http_options so the fallback is skipped in asynchronous mode. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
shuiyisong
|
b27a97ab2e |
fix(servers): validate remote write native histograms (#8654)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
shuiyisong
|
b462d5d19e |
fix: honor default prefix for all metric columns (#8640)
* fix: honor default prefix for metric columns Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Lanqing Yang
|
32a6fc0915 |
perf(servers): optimize PromQL read conversion (#8587)
* perf(servers): optimize Prometheus remote read conversion Signed-off-by: lyang24 <lanqingy93@gmail.com> * perf(servers): borrow dictionary labels in remote read Signed-off-by: lyang24 <lanqingy93@gmail.com> --------- Signed-off-by: lyang24 <lanqingy93@gmail.com> |
||
|
shuiyisong
|
d9122ece3c |
perf: optimize OTLP trace ingestion (#8604)
perf: optimize trace ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Lei, HUANG
|
c8f65c7b99 |
feat: update flow windows after metric batch flush (#8544)
* feat: update flow windows after metric batch flush Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: move batch rows into flow notifier Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: preserve timestamp index in metric batches Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: avoid blocking flow notification lookups Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: add context to timestamp extraction logs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: include peer in flow notification errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: compact flow notifications with time ranges Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: bound pending flow notification queue Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: send raw timestamps in flow notifications Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: update config API snapshot Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9andLei, HUANG
|
0564269426 |
fix: bind Prom remote read schema per query (#8591)
Signed-off-by: discord9 <discord9@163.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> |
||
|
shuiyisong
|
d4d9e199e7 |
fix: enforce table-aware permissions across query and ingest protocols (#8552)
* feat: extend permission req check Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: preserve Prometheus query and write semantics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
56addd0623 |
fix: stream remote analyze metrics while pending (#8405)
* fix: stream remote analyze metrics while pending Signed-off-by: discord9 <discord9@163.com> * test: verify flight metrics preserve pending batch Signed-off-by: discord9 <discord9@163.com> * fix: preserve direct SST perf queries in plans Signed-off-by: discord9 <discord9@163.com> * fix: bind flight metrics capability to query Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
shuiyisong
|
abcbd4f934 |
chore: ignore mysql connection reset error (#8556)
* chore: ignore mysql connection reset error Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: cr issue and add broken pipe Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 Ruihang Xia
|
0479c05092 |
perf: preserve dictionary-encoded query labels (#8541)
* perf: preserve dictionary-encoded query labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(client): skip dictionary Flight batches Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(servers): decode dictionary labels in HTTP output Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * test(mito2): support dictionary tags in series scans Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(servers): preserve dictionary child nulls in SQL Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(promql): compare dictionary tags by logical nulls Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): keep dictionary tags within query paths Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refactor(query): scope PK dictionary encoding to reads Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): preserve pushdown for dictionary labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): handle dictionary labels in query operators Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): complete label type matching Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): align dictionary query schemas Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): decode incompatible OR labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): preserve dictionary partition pruning Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: handle dictionary query edge cases Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: import dictionary downcast macro Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
 Yash Agrawal
|
026cfbd122 |
feat: support splunk HEC raw endpoint (#8491)
* feat: add /raw handler + unit tests Signed-off-by: yash <yasha4658@gmail.com> * feat: add integration test for splunk /raw handler Signed-off-by: yash <yasha4658@gmail.com> * fix: added logs and captured payload unit test for splunk /raw endpoint Signed-off-by: yash <yasha4658@gmail.com> * fix: apply cargo fmt and optimize raw_line_to_map Signed-off-by: yash <yasha4658@gmail.com> * fix: preserve entire body as a event with opt-in linebreaker query param Signed-off-by: yash <yasha4658@gmail.com> --------- Signed-off-by: yash <yasha4658@gmail.com> |
