mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-18 00:34:11 +00:00
1fd9069418aa7a69ab59a71e212d03ab4b0bbc19
7314
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1fd9069418 |
perf(promql): stop building label-set tables where they cannot collide (#10412)
* perf(promql): stop building label-set tables where they cannot collide A `Window` node used to hold an Emitted table of the label sets it had already published, so that a statement dropping the metric name could still keep two series apart when their remaining labels were identical. Every window paid for that table whether or not it could ever collide: a plain `rate()` over a table whose tag columns are nullable had to collect and merge for its whole life, because the table could not prove two series would not fold into one. The table is gone. The proof moved to plan time: a drop that cannot collide - one metric, and no `__type__`/`__unit__` tag to lose - needs no `Merge`, and any other shape gets a canonical `Merge` inserted where the drop happens. The canonical merge kernel already answers the collision, so the runtime no longer carries an occupancy set to detect one. A decline now records why it declined, which is what tells the two paths apart in metrics. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin that a drop above a name-dropping kernel needs no merge `rate(x[5m]) * 2` plans one window and one broadcast, and `abs(abs(rate(x[5m])))` one window and two maps; neither plans a merge, because the metadata labels are all the drop can lose and the labels proof covers them. The review asked for these two shapes by name: with the labels proof forced to answer `false` the test reports `left: 0, right: 2` for the first, which is the count it planned before. Signed-off-by: discord9 <discord9@163.com> * fix(promql): make the metadata drop's proof ask the storage keys `drop_metric_name` also drops an empty label value (`map.rs:367-368`), so two stored series that differ only in `{zone=""}` versus no `zone` become one label set after a name-dropping kernel. The labels alone cannot see that pair, so the proof was incomplete: a nullable key column let the pass insert no `Merge`, and the runtime then emitted two series of one label set instead of reporting `vector cannot contain metrics with the same labelset`. The proof now asks `keys_survive_metadata_drop` - the rule the physical scan already reads - and a `Merge` at or below the drop answers its empty-value half, so a chain of drops over a nullable table pays one collection at the bottom rather than one per drop. Every GreptimeDB tag column is nullable today, so that boundary is what a real table gets. A canonical merge carrying histogram frames was the one pairing nothing exercised, and the extra boundary made it reachable: the gate that declined it is gone. A histogram statement now crosses the boundary - a producer per partition, a consumer above the router - on the merge kernel's own `SampleRef::Histogram` arm, which is where the duplicate this fix is about is reported. That crossing is covered by an HTTP case over a nullable tag, and by `Merge` coverage for complementary positions, overlapping positions, missing samples and sliced payloads, the evaluated frame's histogram and mixed encodings through producer, router and consumer, a float-producing histogram function below a nested boundary, and the segmented plan. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
b26c0fcff7 |
test(promql): pin the existing route when a fixture defaults to native (#10484)
The three native-route tests turn `experimental_native_promql` on in the standalone's query options, so the "legacy" request that follows the native one also takes the native route. The comparison then checks the native answer against itself, and the observer collects both runs: `sum_and_avg` expects the split's two nodes and saw four. Ask for the existing route explicitly with `x-greptime-hints: query.experimental_native_promql=false`. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
25263af9d0 |
feat(promql): fold and average native histograms in the native aggregate (#10199)
* feat(promql): carry a native histogram in aggregate state `sum` and `avg` over native histograms need a partial state whose payload is variable width, which the numeric state family cannot express. Add two kinds, `histogram_sum` and `histogram_avg`, whose components are a live histogram, a layout-carrying compensation, and the evidence a merge records. A position's payload is one `StatePayload`, either the numeric components a float kind carries or a boxed histogram state, so absence stays a single `None` and a field never holds both families. The numeric encoding is byte for byte what it was, and a histogram component nests the same struct a histogram sample frame already uses. The compensation must carry the live value's bucket layout: the kernel indexes it by the value's positions, so a placeholder without bucket indices would be an out-of-bounds read rather than a loss of precision. Both the producer and the decoder check the schema, the zero threshold, the custom bounds and the expanded logical bucket indices, which accepts two spellings of the same buckets and refuses two different layouts. State accounting charges the payload's real width instead of a fixed slot, so a wide bucket layout is paid for. `HistogramSum` cannot declare `incremental_mean`, which is checked structurally: a sum has no mean mode. No admission, split or runtime wiring yet; nothing produces these states until the aggregate kernels land. Signed-off-by: discord9 <discord9@163.com> * feat(promql): fold native histograms into a `sum` aggregate `sum` over a native histogram now folds in the native aggregate: the first sample at a grid position establishes the accumulator from a layout-carrying zero compensation, and every later one is added with a compensated add whose receiver is the live value. The evidence the final answer needs is recorded as it appears: set when a merge reconciled custom bucket bounds, and both reset-hint bits when samples reported a reset and reported that none happened. Two layouts that cannot be added - exponential and custom buckets in one position - poison the point, which the final answer drops instead of answering a partial sum. A numeric overflow is not a fault and does not poison anything: `sum` answers `Inf` when its cumulative sum saturates, exactly as the kernel's own sum does. `avg` keeps refusing a histogram: its mean mode needs the merge rules, which are not part of this change. A group that kept histogram payloads now refuses to export a partial numeric state rather than exporting an empty one. Signed-off-by: discord9 <discord9@163.com> * feat(promql): average native histograms in the native aggregate `avg` over a native histogram folds the same way the v3.15.0 kernel averages a range: sum with compensation while the sum fits, then switch to a running mean once the next add would overflow. The switch keeps the answer finite where the sum would not be: two histograms whose scalar sum is `1e308` each average to `1e308`, not to `Inf`. It starts the mean from the sum accumulated so far divided by the samples that produced it, and weights every later sample by the fraction of samples it represents, so the compensation is scaled by the same weight as the value it belongs to. The complete answer divides both the sum and its compensation by the sample count, or answers the mean with its compensation. `start_timestamp` is cleared and the payload compacted, as every other derived histogram is. No operation refuses a histogram sample any more, so the refusal variant is gone rather than left unused: what a fold accepts is decided by its arithmetic, not by the sample's type. Signed-off-by: discord9 <discord9@163.com> * feat(promql): merge native histogram state across a split A split aggregate exports its partial state and folds it somewhere else, so a histogram fold has to travel. The binding now says which layout a node carries: `aggregate_state_binding(kind, field_kind)` picks the histogram family for a histogram input and the numeric family otherwise, and `BoundAggregate` records the kind its direct input answers with, decided where the node is bound. The partial half exports the live histogram, the compensation that carries its bucket layout, and the evidence the merge needs; a sum cannot claim a mean mode because its kind has no such component. The final half merges two partials the way the v3.15.0 kernel merges contributions: two compensated adds for a sum, and a mean path that converts each side, turns its count into a weight, and scales before adding so that a pair of `1e308` histograms still averages to `1e308`. A sum/sum pair first runs the same trial the complete fold runs, and only keeps it when it does not overflow. An unreconcilable layout poisons the point in either partial: the merge keeps it poisoned and never revives it, and the answer drops that point. Evidence is merged bit by bit, including from a poisoned side, so which annotation fires does not depend on the merge order. The node that finishes the answer is the only one that reports: `Complete` and `Final` speak after their output is queued, `Partial` stays silent because its evidence travels in the state. A poisoned point suppresses its reconciliation info, which is what makes the message set order-independent. Signed-off-by: discord9 <discord9@163.com> * feat(promql): admit a native `sum` and `avg` over histograms A payload-folding aggregate is now admitted exactly when it binds histogram state: `sum` and `avg` over a native histogram fold and merge their own state, so the gate that refused every such input now asks what the node will build rather than what its input carries. Their answer column is the canonical histogram struct; `count`, `group`, `min`, `max`, `stddev` and `stdvar` keep answering floats whatever they read. The split may carry a histogram answer when the cut aggregate exports histogram state. That is also what makes the state schema's value column count follow the kind, which a histogram split otherwise failed against. A histogram fold the split declines is not segmented. Segmentation answers a group by folding each segment and appending the pieces, so a point whose contributions are spread over partitions would be folded once per segment - a sum would merge histograms that never met, an average would average averages. The whole statement therefore keeps the plan it had, which also covers an aggregate above an aggregate and a whole-vector ranking above a fold: neither can cross the split boundary, and cutting them would answer a piece no segment owns. Multi-input histograms and a histogram behind a canonical merge keep their refusals: the state crossing a canonical boundary has no execution coverage. Signed-off-by: discord9 <discord9@163.com> * test(promql): read a native histogram sum and average over HTTP Three cases write native histograms through Remote Write 2.0 and read the answer back through the Prometheus query API. Each one asserts the native route really executed: the observer records the split's two fused producers, and the one publishing a histogram column is the final consumer, so a statement that fell back to the existing plan cannot satisfy the test. `sum` and `avg` answer the values the v3.15.0 kernels produce, and both are compared against the same query without the native hint. That comparison uses a relative tolerance for `avg` and an exact one for `sum`, because the existing plan averages by weighting each sample (`12 * (1/3)` is one ulp below `12 / 3`) while the native fold divides at the end - the difference this change documents rather than hides. The partitioned case flushes after every write so the scan really splits, then asserts the partial producer reported output in more than one partition and that the same query at parallelism one answers the same JSON. Two layouts in one metric remove the group and report it: the answer is empty and the response's warnings name the incompatible layouts, asserted after the native nodes are known to have run. Signed-off-by: discord9 <discord9@163.com> * fix(promql): decide a reset collision per point and pin the avg merge weights The contradictory-reset warning was decided from the two evidence bits OR-ed across every group and every point, so sum by (job) (h) with CounterReset only in job a and NotCounterReset only in job b warned, as did one series crossing its reset over a range. Prometheus checks counterResetSeen && notCounterResetSeen per group per step, so the bit is now decided per accumulator and only the decision is OR-ed across the answer. The state layout is untouched: it still carries the two raw bits, and the collision is settled when the answer is. Nothing caught a swapped pair of avg merge weights: the existing overflowing-pair test gives both partitions one sample and the same mean, so both weightings answer 1e308. The new case merges a two-sample partition that switched to a mean with a six-sample one that still sums, weights 2/8 and 6/8, and asserts 7 * 2^1019; swapping the weights answers 13 * 2^1019 and fails. Signed-off-by: discord9 <discord9@163.com> * test(promql): merge histogram partials through the accumulator, not a row encoder The eight histogram merge tests drove their partials through a test-side encoder that writes numeric state rows, so they panicked the moment a state carried a histogram payload. That encoder never produced a histogram row for anyone: the merge they want to check is the one HistogramAccumulator::merge_state performs, and the row path is the executor's. They now merge the payloads directly -- from_payload for the first, merge_state for the rest -- and read the answer through finished_histogram and the evidence through the production summary. Every assertion keeps its value and strength: the unequal-count case still asserts 7 * 2^1019, so swapping the merge weights answers 13 * 2^1019 and fails, and the evidence case still fails when the merge order decides the result. Signed-off-by: discord9 <discord9@163.com> * test(promql): drive histogram state rows through merge_row The cross-partition merge reads each partition's state frame as a StateRow, and merge_histogram_row is the code that decodes a histogram payload out of one. No test reached it: the numeric row encoder panics on a histogram, and the histogram merge cases go through the accumulator instead, so a bug in that decode -- a swapped component, a poison bit lost, a field index off by one -- would have left every test green. These construct StateFieldsBatch rows directly and merge them. Sum and the average's mean path assert the same bits the accumulator answers for the same payloads, so the two paths cannot drift; a poisoned row is asserted to poison the point; a row missing a component and a row whose compensation carries another bucket layout are asserted to be refused rather than answer something. Dropping the poison bit, not decoding the mean mode, and reversing the field index each fail one of them. Signed-off-by: discord9 <discord9@163.com> * fix(promql): bind an aggregate's state from the statement's own field kind The rebase replayed the compiler call with a literal Some(FieldKind::Float) for the statement's source kind, so every aggregate bound float state: sum over a native-histogram column asked the gate whether it folds a payload, answered yes, and the statement fell back to the legacy plan with prom_native_histogram_agg_sum in it. The call now passes statement_kind, the kind the prepared scans report for the statement's fields. The duplicate guard on the segmentation path -- which refused every histogram fold rather than only the ones the split declined -- is removed, and the mutation scripts the earlier commits picked up are dropped. Signed-off-by: discord9 <discord9@163.com> * test(http): register the three native histogram aggregate cases Splicing the range test and this branch's cases together during the rebase kept the function bodies but dropped their http_tests! entries, so the three cases existed without ever running. Signed-off-by: discord9 <discord9@163.com> * test(promql): identify the histogram column by type, not by name Native result columns are named after their expressions now, so the three admission tests that looked for a column called greptime_native_histogram found none in a plan that answered exactly the histogram they meant to check. They look for the histogram value type instead, which is what the assertions were about. Signed-off-by: discord9 <discord9@163.com> * test(http): find the histogram answer by type and allow an empty scan partition `answer_consumer` looked for a column named `greptime_native_histogram`, which the answer is no longer called now that result columns take their expression's name (#10231). It reads the histogram value type instead, and only the schema's own fields: a state frame keeps its histogram inside `fields`, so a top-level type check still finds the consumer and not the producer. The partition case also required every scan partition to hold rows. Routing hashes each series' label set, so eight series over four partitions can leave one empty; #10469 stopped asking for non-empty partitions elsewhere and `test_remote_write_v2_histogram_native_range_function` already reads the same fixture this way. The producer still has to run on more than one partition, and the answer still has to count every series once. Signed-off-by: discord9 <discord9@163.com> * fix(promql): carry the histogram fold onto #10433's bound aggregate Rebasing onto #10433 left three things behind. `BoundAggregate` gained `field_kind`, which the quantile tests construct without; the value is `Float` because those fixtures hand the fold float samples, and the one call to `aggregate_state_binding` now names the kind it asks about. `native_composed_binary_admits_complex_operands` also expected `hist * on(tag_0) sum by (tag_0) (hist)` to fall back, for a reason this branch made false: it said `sum` has no histogram state to fold into. It has one now, so the pair composes. `histogram * histogram` is not an error in Prometheus - only `+` and `-` apply and any other operator removes the element with an info annotation - so the composed pair is admitted and keeps the legacy float layout for the answer no sample can fill. The test asserts that shape instead, and the refusal it still checks (`sum(sum(hist))`) is named for what refuses it: the split cannot carry an aggregate above a split aggregate. Signed-off-by: discord9 <discord9@163.com> * fix(promql): carry the histogram fold onto #10202's plan-line display `BoundAggregate.field_kind` again: the display test's `sum(rate(...))` builds one without the field, and its input is a histogram, so the field says so. `native_fused_quantile_is_never_folded_per_partition` looked for the statement's text in the display to show the fold is still named by the quantile it wrote. #10202 renders the tree as plan lines instead, so the quantile is named by its own line - `Aggregate: quantile 0.5` - and that is what the assertion reads now. Signed-off-by: discord9 <discord9@163.com> * style(http): keep the answer-column doc off a list item The first line of the histogram sentence ended on a hyphen, so the next line began with one and clippy read it as an unindented list item. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
89450b1469 |
fix(test): render the schema view in the quantile fold test (#10479)
#10202 changed what `display_indent` shows: plan lines with per-node metrics, without the statement. `native_fused_quantile_is_never_folded_per_partition` asserts the statement is still named by the quantile it wrote, so it now reads `display_indent_schema`, which keeps rendering it. This is the only assertion in the file that depends on the statement text; the other `display_indent` call sites assert plan structure. Every PR based on `narthex` currently fails this test. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
9e8e2f6342 |
feat(promql): take the native route's switch from the query options (#10262)
The unified query option mechanism is already in this branch's base, so what is left is the native route reading it: the switch resolves once in apply_query_options as request over session SET over the engine default, and both the planner and SHOW read that resolved value instead of combining the engine option with the request extension themselves. The engine-wide option therefore stops being a gate a request also has to pass and becomes the default a request or a session states over, which is what the other query options in this crate already do. What that costs is stated rather than implied: an operator can no longer switch the experimental route off for clients that set query options, and none of the entry points (HTTP hints, gRPC context, SET) carries an authorization check of its own. Data permissions are unaffected - the statement still plans through the same scanners and the same per-table checks. #10459 asserted the old gate; its test is rewritten here to pin the new contract in both directions instead of being dropped. Metric discovery stops keying off the request extension for the same reason: a session SET or the engine default can turn the route on without one. It reads the switch through the same resolver planning reads, so a query that leaves the route off is not bounded for discovery and pays no per-table grant for it. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
bd1ad73682 |
perf(promql): cut per-series fixed costs in decoding, packing and accounting (#10416)
Signed-off-by: discord9 <discord9@163.com> |
||
|
|
7ba85f6369 |
feat(promql): show the native tree as plan lines with per-node metrics (#10202)
feat(promql): draw the native tree as plan lines with per-node metrics The fused node prints its own compiled tree instead of echoing the PromQL text it was built from: one node per line inside a `native_tree=[ ... ]` block, with each node's figures on its own line when the statement ran. A split aggregate's two halves therefore show the trees they compute rather than the same statement, and EXPLAIN/ANALYZE of a native statement reads like the plan it runs. The statement text stays on the node as data: it is what names the value columns (`expression_field_names`), so it has to survive to execution and over the wire where a node is rebuilt from its envelope. What changes is that it is no longer what the node prints. The tree's registrations sit in the node's own metric set, so a reader that keys by name reads that set; `ExecutionPlan::metrics` leaves them out because a flat set cannot say which node recorded what. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
7860e4d6cd |
feat(promql): fold pi() and answer quantile as a complete aggregate (#10433)
* feat(promql): fold pi() and answer quantile as a complete aggregate pi() folds through constant_value like any other constant, so pi() * x, clamp_max(x, pi()) and quantile(pi()/4, x) all compile natively. The value is std::f64::consts::PI on both routes. Note this is wider than the legacy planner, which does not treat a call as a literal and so rejects those parameter positions; it is an extension of constant support, not a narrowing. quantile now binds a Complete fold that retains the group's input and answers with the existing aggregate_quantile kernel. A quantile has no partial kernel, so split.rs and the state binding are untouched and a Partial/Final quantile cannot exist: validate refuses it and check_cut declines the cut. A partitioned input therefore gathers above per-series prefixes instead of folding per partition, which is the only correct distributed shape for a non-associative aggregate -- the raw-source boundary keeps fragments from being read as whole series, and the router merges a raw identity once every input reaches EOF. The retained input is not a fixed-width state: a quantile keeps every group until EOF and cannot spill, so it is bounded by pool admission, which the module doc now states. Its parameter is sparse, covering the grid positions the input actually holds rather than the window they span. Two points 100k positions apart used to allocate an 800KB dense parameter before any admission; now the parameter and the kernel's position/value copy are charged by row bound before allocation, and retaining a series reserves the Vec capacity it is about to grow to rather than one slot. A test runs two distant points under a 64KiB pool and finishes with nothing left reserved. quantile over a histogram no longer falls back: the legacy planner groups it with min, max, stddev and stdvar, skipping histogram samples and reporting an info, so the admission check now refuses only sum and avg, which fold the payload. The native runtime asserts the same info as min over a histogram-only chunk. Signed-off-by: discord9 <discord9@163.com> * fix(promql): bill a quantile's retained chunks the way a binary operand is billed series_bytes charges the whole Arrow parent buffer a slice points into, and every histogram table it references, so a series handed over as many slices of one buffer was charged once per chunk: two thousand one-row slices of a 16 KB array came to ~32 MB against a 4 MiB pool and were refused around the 260th, while min over the same input answered. Retained chunks are now billed as a binary operand bills its operand: operand_chunk_bytes measures what each chunk holds on its own, with the parent buffer counted once across its slices, and unbilled_tables bills each histogram table once, remembered by allocation address. QuantileInput::bytes() reports real residency -- its own size, the vector's capacity and the folded held bytes -- instead of a per-chunk sum, so every resize settles on what the node actually keeps. The capacity pre-booking stays and is kept out of held, so a vector slot is not charged twice. The regression hands one array over as 2000 single-row slices under a 4 MiB pool and asserts min and quantile answer the same and both return their reservation. Restoring the old billing makes it fail with ResourcesExhausted, which is the failure it exists to catch. Signed-off-by: discord9 <discord9@163.com> * fix(promql): bill a quantile's retained input once per allocation The previous change stopped charging every chunk for the whole parent buffer, but the figure was still not self-consistent in three ways. operand_chunk_bytes subtracted the whole of fields_bytes() and added back only the row bytes of the slice, so a small slice of a big buffer was billed as small while the whole allocation stayed resident, and a validity bitmap was not counted at all. Retained chunks are now billed per Arrow allocation, keyed on the buffer's data pointer and charged at capacity, the same way unbilled_tables bills histogram tables. The no-identity alternative -- bill every chunk for the whole buffer -- over-estimates and would refuse a legal query, which is the reported bug mirrored. series_bytes includes size_of::<Series>(), and QuantileInput::bytes() also counted every vector slot, so each used slot paid for its Series twice. The chunk figure now carries only what sits beside the samples and the struct is left to the vector. pending() returned the incoming chunk's figure for every fold, but the quantile retainer books its own figure before the push, so that backing was reserved twice and could refuse a query before any memory was actually held. finish_quantile also settled on the output while its input was still alive by value; it now drops the input first. Billing only; no answer changes. A 2000-chunk input goes from 1,757,960 to 1,181,968 bytes (the 2000 x 288 of double-counted structs plus one 16 KB allocation billed once), and every quantile billing test ends with the pool empty. Five mutations -- rows-only figure, no buffer dedup, per-chunk series_bytes, grid-wide parameter, struct added back per chunk -- each fail the test that covers them. Signed-off-by: discord9 <discord9@163.com> * test(promql): cover a quantile over no input and drop its redundant guards A quantile whose input holds no sample in range has no positions to build a parameter from, so it hands the kernel an empty scalar and answers nothing. No test reached that: replacing the fallback with an error left every quantile and aggregate test green. The new one runs the node without pushing a chunk at all and asserts an empty answer with the pool back to zero. new_quantile also re-checked the stage, the kind and the grid. validate already refuses a Partial or Final quantile -- aggregate_state_binding has no state for one -- and the node's schema is the tree's validated output, so those checks could not fire. The quantile is now passed in from the arm that destructured it. The billing test's comment described the operand_chunk_bytes model; the quantile bills chunk_bytes_beside_samples plus the Arrow allocations its slices point into. Signed-off-by: discord9 <discord9@163.com> * test(promql): answer a quantile over a multi-input tree instead of declining it quantile stopped being a kind the runtime refuses, so the query that pinned that refusal now answers natively. It moved out of the declines test into its own case, which asserts the existing plan answers 3/16, that the route plans a fused node for it, and that the native rows equal the legacy ones. The grouping-by-a-value-field case stays where it was. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
037ed1f2df |
refactor(promql): decide native admission with an enumerable predicate (#10424)
Admission to the native PromQL route was decided by trial-building the whole runtime tree and discarding it, at five call sites of NativeExecutionState::try_new(...).is_err(). That is where the refusal lived, not anywhere enumerable. Replace those calls with native_tree_supported(&tree), a predicate on the compiled tree's shapes that builds no runtime: it declines the nine shape classes RuntimeNode::build rejects (raw-sample roots, interleaved inputs, scalar() over a multi-field vector, non-performable broadcasts, quantile aggregates, value-field groupings, vector ops over multi-field vectors, top-k variants, and unsupported window modifiers). try_new keeps its construction-time validation for the refusals that are not shapes. Which queries reach the native route is unchanged; the new tests assert both the predicate's verdict and that the runtime agrees. Verified: promql 1211/1211, query 1130/1130, native_promql 93/93, clippy -D warnings clean. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
48bb6beadd |
feat(promql): name native result columns from expressions (#10231)
Use the statement expression for native value column names, resolve legacy value columns by type rather than by name, and reject label-function destination collisions instead of dropping a value. Keep matrix roots named after the whole statement, preserve an Int64 __tsid value, and propagate schema-collision errors from canonical boundaries. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
3bdde3d339 |
test(query): prove a default engine option keeps native requests on the legacy route (#10459)
The engine-wide `experimental_native_promql` option is the outer gate. #10371 pinned its default, but nothing showed what that default does to a request that explicitly asks for the native route — which is the case that actually matters when the branch is merged back: a client can always send the per-request extension, and the engine option is the only thing left deciding. `plan_native_promql_with_query_options` always sets `query.experimental_native_promql=true` on the request, so passing `QueryOptions::default()` covers exactly that combination. The assertion looks at the rendered plan for both `FusedSeries` and `RawSeriesSource`, so a future change that lets the request alone admit the route fails here. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
6796dfeaee |
test(promql): cover unequal-width owner routing and scalar broadcasts (#10039)
Signed-off-by: discord9 <discord9@163.com> |
||
|
|
32777e458f |
chore(merge): merge main into narthex (#10464)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Same shape as #10362: merge `main` into `narthex` on a branch, so the next `narthex` → `main` PR is a mechanical replay instead of a conflict-resolution session. ## What's changed and what's your intention? Merges `main` (`dd941a8e334`) into `narthex` (`996b7e9e8d0`). Since #10362, `main` moved 42 commits and `narthex` moved ahead independently. This brings the branch back up to date so the eventual merge back carries only narthex's own work. ### Conflicts **Four files.** `src/common/query/Cargo.toml` (two hunks): - `[dependencies] futures-util`: main made it `optional = true` (only `testing` needs it, for the hash-join fixtures); narthex declares it as a regular dependency. - `[dev-dependencies]`: main added `datafusion-datasource` and `futures-util`. Resolution: `futures-util` **stays non-optional**, because narthex's `series_frame` API exposes `futures_util::Stream` in its public signatures (`src/common/query/src/series_frame.rs:55`), so making it optional would gate a production API behind the `testing` feature. `datafusion-datasource` stays optional in `[dependencies]` and non-optional in `[dev-dependencies]`, as main has it. The `testing` feature becomes `["dep:datafusion-datasource"]` — the `dep:futures-util` arm cannot stay, since Cargo rejects `dep:` on a non-optional dependency. `src/frontend/src/instance.rs`: import union — main added `ServerMemoryLimiter`, narthex has `MetricDiscoveryScope`. Both kept. `Cargo.toml` + `Cargo.lock`: follow main's newer `greptime-proto` rev (`c46f7ee9c`, which already contains the `RawSeriesSource` that #355 put on `main`), instead of the rev this branch merged a week ago. `tests-integration/src/standalone.rs`: field union — `query_options` (narthex) and `auto_create_physical_table_timestamp_unit` (main). ### What else comes along - **DataFusion fork bump** `70f84c5b` → `638a77fb` (main moved to a newer fork rev); the branch is rebuilt against it here rather than at merge-back time. - `feat: persist bounded WAL releases on region close (#10329)`, `feat(mito2): replay pending WAL tails in the background (#10342)`, `feat: coordinate region edits with repartition lifecycle requests (#10354)`, `feat(query): serialize remote dynamic membership (#10341)`, `feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN (#10092)`, plus the auth/security fixes (`#10374`, `#10378`, `#10382`, `#10381`) and the mito2 SST metadata work. ### Verification - `cargo check -p common-query --all-targets --features testing` passes (the crate one conflict was in). - `cargo check --workspace --all-targets` passes. - `cargo nextest run -p tests-integration -E 'test(native_aggregate)'` → 6/6 passed. The first push of this branch had a genuine red: `native_aggregate_complete_over_four_nonempty_mito_partitions` failed because four tests still asserted the pre-#10418 topology. That is fixed by #10466 (merged into `narthex`); this branch is rebased onto it, and the same local run is green. - `Cargo.lock` regenerated with `cargo metadata`; no manual edits. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [ ] I have written the necessary rustdoc comments. - [ ] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). ## Merge timing - [ ] Merge during the nightly window <!-- nightly-merge --> |
||
|
|
089121c613 |
Merge main into narthex
Conflicts: - `src/common/query/Cargo.toml`: `futures-util` stays non-optional because the series-frame API exposes `futures_util::Stream`; `testing` only pulls `datafusion-datasource`. - `src/frontend/src/instance.rs`: import union (`ServerMemoryLimiter` from main, `MetricDiscoveryScope` from narthex). - `Cargo.toml` / `Cargo.lock`: follow main's newer greptime-proto rev (`c46f7ee9c`), which contains the `RawSeriesSource` this branch needs. - `tests-integration/src/standalone.rs`: keep both `query_options` and `auto_create_physical_table_timestamp_unit`. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
aba720c341 |
fix(test): stop requiring every scan partition to be non-empty (#10469)
`SeriesRepartitionExec` routes a series by hashing its label set, so the eight
series this test writes over four partitions can legitimately leave one empty:
the fixture happened to hash evenly before, and it no longer does
(`{0: 0, 1: 3, 2: 3, 3: 2}` on `narthex`).
Requiring every partition to be non-empty asserts a property of the hash, not of
the topology. What the test actually needs is that the series are spread over
more than one partition - otherwise the merge it exists for is never exercised -
and that routing loses none of them. Assert those instead, and keep the existing
"all four partitions reported" check.
Signed-off-by: discord9 <discord9@163.com>
|
||
|
|
453c35e9d3 |
feat: support trigger cache invalidation through frontend plugins (#10460)
Signed-off-by: fys <fengys1996@gmail.com> |
||
|
|
c6962802a9 |
feat(meta-srv): bind candidate region leases to the migration's registration (#10440)
* feat(meta-srv): bind candidate region leases to the migration's registration A region migration now gives its candidate region registration a random non-zero candidate lease id. Leases granted because of that registration carry it, but only while the table route exists, and the upgrade instruction carries the same id. The datanode passes the id of each lease to the engine, and Mito records the lease of the latest heartbeat per region outside the region instance, so it survives a reopen during catchup. The migration also drops the candidate registration before it restores the route on rollback, and before the procedure leaves the migration tracker when it ends without a rollback. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): record a lease only while the region stays open Record a granted lease under the same lock that removes it after the region closes, so a lease recorded during a close does not outlive the region. The metric engine records the data region's lease when the metadata region ignores it. Remove the roles-only extraction of operating regions, and cover a present candidate lease id in the upgrade JSON. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * build: point greptime-proto at the candidate lease id change Use the greptime-proto commit that adds GrantedRegion.candidate_lease_id (greptime-proto#357), which also brings in the RawSeriesSource Substrait payload message. Rely on the declaration order of RegionMigrationProcedure to drop the candidate registration before the tracker entry, and document it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drop lease records when workers stop Remove the lease records of the regions a stopping worker closes. Check under the removal lock deterministically in tests, cover a lease that arrives after a region is removed, assert the candidate lease id on the migration's real registration and its retry, and match ids per region in a batched upgrade. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): drop a redundant helper comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(store-api): pin the candidate lease id wire bytes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: document candidate registration and lease record lifecycles Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: keep candidate lease ids through customized lease renewers The heartbeat handler now sets the candidate lease id of each lease a customized renewer returns from the renewed region with the same id. The metric engine records a lease for the data region only when the metadata region ignored it, since the data region records its own leases. The countdown reset command carries a named struct instead of a tuple. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(metric-engine): cover a lease the data region ignores Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: tighten lease id lookups and lease test assertions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: align lease naming and share the txn-observing kv backend Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: tidy lease naming, docs and shared test helpers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: drop field comments that restate their names Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: drop method comments that restate their names Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: say which heartbeat a lease's send time refers to Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: drop the reset deadline field comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: state when an operating region's role and id reach a lease Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: pin greptime-proto to the merged candidate lease id change Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: add the candidate lease id argument to the direct demotion test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: order new keeper helpers after public items and test equal send times Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
c679ca5fdc |
feat(ingestion): configurable time index unit for auto-created physical metric tables (#10189)
* feat(ingestion): configurable time index unit for auto-created physical metric tables Auto-created physical metric tables hardcoded a millisecond time index. Add an `auto_create_physical_table_timestamp_unit` option (frontend and standalone configs, default "Millisecond") that selects the time index unit for physical metric tables created by auto table creation. New logical tables created under such a physical table inherit its unit, so Prometheus remote write, OTLP metrics, and other metric-engine ingestion paths write timestamps in the configured unit. The option threads through all auto-create sites: the operator inserter's `create_physical_table_on_demand`, the frontend instance's batched logical-table flush path (`create_metric_physical_table_if_missing`), the logical-table batcher's unit alignment fallback, and the OTLP metric preflight validation unit. Millisecond remains the default, so existing behavior is unchanged unless configured. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(otlp): match default physical schema to configured timestamp unit Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
7e76d28908 |
fix(postgres): account for buffered InfluxDB and OTLP COPY bodies (#10450)
Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
5b26929a5b |
test(query): pin the plan shapes the uniform topology builds (#10466)
* test(query): pin the aggregate plan shapes the uniform topology builds Four native-aggregate integration tests have failed on `narthex` since the uniform native PromQL topology landed (#10418): their plan-shape helpers demanded the pre-topology chains, panicked before the native-vs-legacy comparison ran, and nextest's fail-fast then left most of the suite unexecuted. Both shapes the helpers now see are the topology's own: - a Partial producer the analyzer keeps above the regions reads the `MergeScanExec` through one `SeriesRepartitionExec` that widens the region stream to `target_partitions` with `RawPrimaryKey`, so each stored key still reaches exactly one owner; - a single-owner Final reads the Partial producer itself through its state router, instead of a region merge scan, because both halves are planned in the frontend tree now. Pin each chain exactly rather than accepting any exchange, and run the legacy comparison before the shape assertions so a shape regression can no longer hide the result check. Verified: native_aggregate 6/6, native_promql 90/94 (the four `integration_http_s3*` cases abort for environment reasons), clippy and fmt clean. Signed-off-by: discord9 <discord9@163.com> * test(query): read one raw source per selector occurrence in the CTE case `cte_native_statement_runs_the_fused_plan` asserted that one logical table keeps both occurrences of a repeated selector region-local, so no raw source appears in the plan. The uniform topology reads every selector through its own raw source, so the statement now names two. Pin that instead: both occurrences still read their own region scan, and each does it through a raw source of its own. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
610794e447 |
perf: batch logical table cleanup when dropping databases (#10461)
* perf: batch logical table cleanup when dropping databases Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: refine drop database batching and recovery Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: simplify drop database traversal and batch sizing Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: tighten drop database batch size estimate Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: narrow logical table metadata cleanup API Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
0df6e25f3e |
feat(mito2): bound tail replay resources and recover from failed rounds (#10444)
* feat(mito2): bound tail replay resources and recover from failed rounds Tail replays now write SSTs through a MemtableSstWriter split out of RegionFlushTask, with their own WriteType and metrics, so replay chunks no longer count as flushes. A node runs at most 16 replays at once, gives their SST writes their own permits, and bounds their chunk memory by a budget of 1/8 of the global write buffer: each replay keeps a floor of budget/32 and the chunks share a pool of budget/2 above it. After a failed round, a writable region whose storage holds a newer manifest is fenced and reloaded once the replay ends, so tails that another writer already committed stop being retried. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): make the replay update check cancellable and keep flush logs The manifest check after a failed replay round listed storage under the manifest read lock and could not be cancelled, so closing a region whose listing stalled waited on it. The listing now runs under the replay's cancellation, and only the fence takes the write lock. Flush completion logs keep their reason, replay metrics record memtable compaction, and tests cover a stalled check on close, flushes running while replay writes are saturated, and the no-index case of async index builds. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep the wait for the replay fence lock cancellable The replay now waits for the manifest write lock under its cancellation and only rechecks and fences once it holds the lock, so a demoted, closed or replaced region ends a replay queued behind another manifest operation. It logs when it waits for a replay permit, and the replay metrics are crate-private. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): wait for the replay to reach the fence before demoting Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): address review of the tail replay limits The update check after a failed round lists the manifest directory without holding the manifest lock; the write-lock recheck catches any version installed meanwhile. A replay snapshots its tails once it holds its permit, logs only when it waits for one, and logs when it fences or declines to. The writer takes the region hook instead of the manifest context, and names, docs and tests follow the review. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): cancel the tail replay on a direct demotion A direct role change that leaves a region unable to replay its tails, such as a demotion when its lease expires, now asks the worker to cancel the replay, as a graceful downgrade already does. A replay waiting on storage I/O then ends, and a later promotion starts another. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): cancel the tail replay when a lease demotes the region A lease that grants a non-writable role now cancels the region's tail replay like a direct demotion does; both share one helper, which sends nothing when the region has no pending tails. Docs, logs, metric observation and tests follow the second review. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): ignore demotion notices of replaced region instances A demotion notice now carries the region instance it applied to, and the worker cancels the replay only while that instance is still registered, so a late notice of a replaced instance leaves the new instance's replay running. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): narrow the tail replay byte counter doc Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): align tail replay names and docs, test reservation bookkeeping Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): derive the replay log reason from its write type The SST writer now logs the flush reason for flushes and the write type otherwise, so the replay no longer passes a bare string. Drops an engine test that also passed without the replay memory pool; the reservation unit tests cover the pool's accounting. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): close the demotion and manifest version races of tail replay A role change now cancels the tail replay whenever it leaves the region unable to replay, instead of comparing with a role read before the change, which a concurrent promotion could make stale. The manifest check after a failed round lists from, and rechecks, the installed manifest version rather than the last version number, which a failed commit leaves ahead of it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): pin the demotion notice to the state after the role change Adds a case where the region is already a follower when the engine sets its role, and words the unwritable notice docs by the state after the change. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): rewrap the unwritable notice doc Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): notify only when a role change makes the region unable to replay Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): assert every set_role return value Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): drop transition headings that repeat the code Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): restrict memtable writer construction and simplify write metrics Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): tie the flush reason to the writer kind and share replay test helpers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): correct the replay histogram range and listener doc Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): keep tail replay doc links within the documented items' visibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
0a93810c23 |
fix: improve exemplar handling and batch memory admission (#10363)
* chore: update README Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(ingest): reserve memory for table batches Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): prefix the labels column Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): hide exemplar table from metadata Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): reconcile source tags before writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(exemplar): prune tags and retain source IDs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): record slow queries Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): skip exemplars when source writes fail Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): skip exemplars when source writes fail Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: account for held memory in batch admission Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: config comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: simplify exemplar and memory tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: deduplicate exemplar setup Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>v1.4.0-alpha.0-fcc1891ba-20261116-1794831195 |
||
|
|
fcc1891ba6 |
feat(mito2): support index plugins in region creation and rebuilds (#10298)
* feat(mito2): validate index plugin options before region creation Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito2): support index plugins in rebuilds Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): skip unnecessary SST reads during index rebuilds Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): adapt index rebuilds to shared file metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): keep async flush indexes independent of sketch providers Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): finish flush setup before seeding historical indexes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): preserve index rebuild scope and published extensions Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): remove speculative flush index guard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): preserve existing indexes on incomplete rebuilds Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): separate background indexing from sketch maintenance Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
dd941a8e33 |
fix(mito2): wait for a stopped region's checkpoint before reopening it (#10376)
* fix(mito2): wait for a stopped region's checkpoint before reopening it A region's manifest checkpoint runs as a background task that writes the checkpoint, then `_last_checkpoint`, then deletes older manifest files: the deltas up to the previous checkpoint, or every older delta at the region's first checkpoint. Stopping the manifest manager does not wait for it, so a new instance of the region could read the manifest while the old task still wrote it. On a store that writes files in place the open could read an empty `_last_checkpoint`. On any store, an open that found no checkpoint could have deltas it had just listed deleted by the first checkpoint's cleanup. Wait for the stopped instance's checkpoint at engine stop, after the worker loop ends, and before a fenced region is reloaded. The wait does not hold the manifest lock, which the worker may take meanwhile: the checkpoint task owns the sender of a watch channel, which closes however the task ends. Fixes the flaky test_change_region_compaction_window (#7135). Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover the checkpoint wait through cleanup, panic and abort Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): align checkpoint wait names and log the wait Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover concurrent and late waits on the checkpoint Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): keep two checkpoint waiters through the end of the task Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that a wait after the checkpoint ended does not start waiting Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): address review on the stopped checkpoint wait Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): state the stop contract and assert it before the checkpoint wait Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): name the checkpoint completion future and keep its task state together Rename `pending_checkpoint_done` to `pending_checkpoint_completion` on `Checkpointer` and `RegionManifestManager`, and state how it differs from `wait_for_pending_checkpoint`. Keep the task handle and its completion receiver in one `CheckpointTask`, so reaping the handle also drops the receiver. Use plain backticks in the `stop` doc, which is public and cannot link to a crate-private method. Narrow the comment in `RegionWorkerLoop::clean`: a datanode or standalone shutdown closes every region before it stops the engines, so there the wait only covers regions whose close failed. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
996b7e9e8d |
docs(promql): trim design-narration comments in native planner and runtime (#10421)
Comment-only cleanup on top of the segmentation move (#10418) and the topology it carries (#10321). |
||
|
|
9948a163c1 |
refactor(mito2): drop the unreachable samples builder and the unused prepared-series contract (#10430)
* refactor(mito2): drop the unreachable per-SST samples builder Production only ever constructs the columnar samples format: SeriesFormats::from_config is reached from with_format with experimental_series_index_format, where Legacy yields no samples backend and Columnar yields ColumnarSamplesFormat. The other SamplesFormat impl is cfg(test). So the per-SST branch of the maintenance task could never run, and build_samples had no caller but its own test. The daily columnar body is now unconditional, and the former branch position holds an explicit UnsupportedOperation error so an unknown backend fails loudly instead of building nothing. The two fake-format lifecycle tests were duplicates of real_samples_reconcile_reopen_and_retire, which drives the same build/reuse/reopen/ catalog-failure/purge scenario against the real format; the abort test moved to build_sample_windows, and a new test pins the refusal. The compatibility fixtures are untouched and still pass. Signed-off-by: discord9 <discord9@163.com> * test(mito2): cover retirement after a successful samples rebuild format_skeleton_build_reopen_and_retire was the only test of the retirement that follows a successful rebuild; real_samples_reconcile_reopen_and_retire covered the failed-publish path only. Removing the samples loop's mark_deleted() from publish_index_version left every other series_index and series_scan test green, which is what made the gap invisible. The real-format test now also rebuilds a day by deleting one of its objects, checks that the superseded generation is no longer in the published snapshot, that nothing is purged while the old snapshot and its handle are alive, and that its objects are purged once they are dropped. With mark_deleted() commented out, all four cases fail on that last assertion. Signed-off-by: discord9 <discord9@163.com> * test(mito2): pin the purge to the last reader handle, not the snapshot Dropping the published snapshot was allowed to release a superseded generation as long as one of its handles was still held elsewhere, which no test covered: the retirement assertions only checked that nothing was purged while the snapshot lived, then dropped every handle at once. The test now holds a reader handle on every generation the superseded snapshot published before releasing it, so that release must purge nothing, and asserts that between the two drops. A rebuild replaces several generations of a source group at once, so holding only the one the test tracks would let its siblings be released legitimately and the assertion would fail for a reason unrelated to the lifecycle under test. A mutation that purges on snapshot drop -- ignoring live reader handles -- fails this new assertion in all four cases and none of the later ones. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
a1627ee1c4 |
refactor(promql): share the extrapolation factor between float and histogram windows (#10425)
* refactor(promql): share the extrapolation factor between float and histogram windows A native histogram counter can carry a start timestamp inside its window, which the histogram range path turned into a synthetic zero and extrapolated from with its own inline millisecond-domain formula. The float path already computed the same factor in extrapolation_factor, so the two routes could drift apart: the histogram branch counted the synthetic zero as an extra sample interval and extrapolated towards zero from a negative first count. The factor now takes an ExtrapolationSpan with two anchors -- sampled_from, where the increase was sampled, and average_from, where the average sample distance is measured. A float window sets both to its first sample; a histogram window starts the sampled interval at the synthetic zero and keeps the average on the real samples. A synthetic zero is the counter's zero point, so it is expressed as zero_point = Some(0.0), which reproduces the removed duration_to_start = 0.0 exactly. The sampled_interval == 0 divergence is deliberate and documented: a float window keeps its increase with a factor of 1, while the histogram window answers no value, since its buckets are all zero and an Inf or NaN rate would only poison its consumer. Signed-off-by: discord9 <discord9@163.com> * test(promql): make the two-anchor extrapolation tests able to fail Each of these could only pass: the route-equality table gave both fillings the same anchors, and the synthetic-zero geometry kept its ends under the threshold with the average anchor taken either way, so an anchor swap answered the same 1.375. The table now also carries a window whose anchors differ, pinned to 1.5 for the histogram filling and 1.75 for the float one, so collapsing the sampled anchor onto the average fails. The synthetic-zero test uses zero 1000, samples 2000/3000, eval 4500, range 4000: correct is 1.25, and measuring the average from the zero gives 1.75. Both numbers are asserted, and a temporary anchor swap was confirmed to fail the test before it was reverted. Signed-off-by: discord9 <discord9@163.com> * test(promql): give the synthetic-zero range test anchors it can tell apart The 1.375 geometry answered the same with the average anchor measured from the synthetic zero, so the end-to-end test agreed with itself either way. It now uses zero 1000, samples 2000/3000, eval 4500, range 4000: the count is (4 - 2) + 2 = 4, the factor 1.25, so an increase answers 5.0 and a rate 1.25, while a swapped anchor answers 7.0 and 1.75. The swap was applied to the production call site temporarily and failed the test with "Increase over two samples: 7 vs 5" before it was reverted. Signed-off-by: discord9 <discord9@163.com> * test(promql): give the two-anchor window geometry that can fail both ways The previous geometry had its end inside both thresholds, so an average interval read from the sampled anchor answered the same 1.5 as the correct one; only a sampled interval read from the average anchor changed the answer. With zero point 2000, average 2500 and last 4500 the end is past the correct threshold (2.2s) and inside the wrong one (2.75s), so it truncates one way and not the other: the correct answer is 1.4, an average anchor read from sampled gives 2.0, and a sampled anchor read from average gives 1.5. Both mutants are pinned as assertions, and both were applied to the production formula to confirm this test fails on them. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
2e4141e77d |
refactor(promql): move the segmentation engine into the promql crate (#10418)
Lands the segmentation-engine move together with the uniform native-PromQL topology it builds on (#10321), which this branch carries as its base. |
||
|
|
94ec1a5a17 |
chore(promql): drop production-dead state and sample helpers (#10408)
- native/state.rs: remove ensure_bound_state and its only call site, plus the now-dead state_kind/state_components helpers (about 60 lines). - instant_eval.rs: remove the strictly-increasing re-check in select_instants; RawTimestamps::try_new already enforces the order. - native/split.rs: the Window arm declines on input_grid.is_none(); the declared-vs-input grid equality is already enforced when the window is bound (native/plan.rs). - datatypes/series/samples.rs: remove Samples::into_dense and its arrow_result helper; only datatypes tests called it. The eight densification tests existed only to exercise it and go with it. Verified: promql 1209/1209, datatypes 402/402, clippy -D warnings clean. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
92cf9b0edb |
chore: drop a PR description that was committed by accident (#10463)
`pr-histshared-body.md` is the body draft for #10420 and was committed along with that change. The repository root only carries the shared docs (AGENTS, CONTRIBUTING, README, SECURITY, AUTHOR), so this file has no reason to be here. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
b136da871f |
test(promql): stop deriving budget expectations from the estimator (#10409)
About fourteen accounting/reservation tests computed their expected figure, pool limit, or 'exactly fits' budget from operand_chunk_bytes / evaluated_bytes or a copied row-size constant, so a wrong formula stayed green. They now assert an independent behavioural property instead: a charge appears and is released to zero, an oversized operand is refused by the pool, a refused chunk leaves nothing reserved, or a bound stated in the bytes the fixture's own data holds. Ablation proof: temporarily doubling operand_chunk_bytes now fails 9 tests instead of 3, so these tests can actually catch a wrong formula. The ablation was reverted; promql 1209/1209 green, clippy clean. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
89e14ab936 |
fix(mito2): cancel compaction merges at batch boundaries (#10411)
* fix(mito2): cancel compaction merges at batch boundaries Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): preserve upload errors and expose merge cancellation Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): normalize encoder cancellation at SST writer boundary Signed-off-by: WenyXu <wenymedia@gmail.com> * style(mito2): use imports for SST write types Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): share cooperative upload cancellation with tokens Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): classify nested upload cancellation errors Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
5517c4b76f |
refactor(promql): let the legacy histogram path read the native kernel (#10420)
* refactor(promql): read the legacy histogram estimators from the native kernel `prom_native_histogram_quantile` and `prom_native_histogram_fraction` estimated from `common_query`'s `NativeHistogram` bucket walk: exponential bucket bounds computed as `2^x`, and a forward walk from the lowest bucket only. The native kernel in `histogram_eval` walks the same histogram the way Prometheus does -- the bound table in `bounds`, and a walk from the highest bucket down once a finite-sum quantile reaches the median -- so one histogram had two answers that differ in their last places. Read the `NativeHistogram` through the borrowed `HistogramView` both representations share and return the kernel's answer, mapping its outcome back onto the NaN-observation notes the legacy UDFs record. Their signatures and the notes are unchanged; only the arithmetic behind them moves. The `2^x` estimator in `common_query::native_histogram` stays for that crate's public API and is annotated as such: no PromQL path reaches it any more. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): delegate the legacy histogram arithmetic to the native kernel `prom_native_histogram_add`, `_sub`, `_mul_scalar`, `_scalar_mul`, `_div_scalar`, `_neg`, the sum/avg accumulators, `delta`/`idelta`/`rate`/`increase`, and the weighted mean that merges partial states all scaled and combined `NativeHistogram`s with that type's own reconciliation: its own zero-bucket widening, schema reduction and custom-bound intersection, over `2^x` bounds. `FloatHistogram::add`/`sub`/`mul`/`div` already implement Prometheus's rules (`addBuckets`, `reconcileZeroBuckets`, `CopyToSchema`, `intersectCustomBucketBounds`) and the native kernel's golden tests cover them, so this keeps one implementation of the semantics. `AddOutcome` folds into the existing semantics rather than being dropped: `custom_bounds_reconciled` is the intersection the kernel keeps in `custom_values`, which the callers already announce from the operands, and `counter_reset_collision` is the contradiction the kernel answers with the unknown hint the legacy arithmetic derived from two hints that disagree; both are asserted in debug builds. The public `NativeHistogram` signatures are unchanged, the sums are compacted as before so `promql_eq` and the serialized spans keep their answers, and unary minus goes through the kernel's multiplication like the other scalings. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the legacy delegation's observable answers Comparing the bridge against the kernel it calls only shows they agree today; it cannot show what either answer means. These tests pin the answers themselves. - quantile with a NaN sum over several buckets answers 32.0 with NaNSkew, because the NaN walk leaves the last bucket for the interpolation; below the rank it answers NaN with NaNResult. - an exact tie answers 1.0 through the legacy walk and 2.0 through the kernel, on a fixture with an empty bucket between the two, and neither records a note. - assert_same_payload now compares every field, including the start timestamp policy, custom values, the complete span segmentation and the bucket bits. - round-tripping preserves reset hints, custom values, zero-length spans and gaps. - add/sub reconcile the reset hint and the custom bounds, compact cancelled buckets, and keep promql_eq's expanded-index equality. - scale/divide_by keep the field-by-field fallback for an unreadable layout -- mismatched counts, a backwards span, an overflowing index, a negative length -- and preserve its start timestamp; add/sub answer None for the same layouts. Zero, negative and NaN factors are pinned. - rate/increase/delta/idelta/irate clear the start timestamp of a derived histogram, while a single-sample sum aggregate keeps its stored one. A layout whose later span steps backwards is refused before the kernel sees it: the legacy bucket_indices rejected it, while side_from_view would have accepted repeated indices. Signed-off-by: discord9 <discord9@163.com> * test(http): pin the legacy histogram route's rate, NaN quantile and derived windows The delegation changes what three kinds of well-formed data answer, so assert it where a user sees it. Every request runs without the native hint and asserts !executes_fused_series on the observed plan, which proves the legacy route rather than assuming it from the missing hint. - rate and increase over a raw sample with a start timestamp inside the window still extrapolate from the synthetic zero: count 8/3 and sum 10/3 for a rate, 8 and 10 for an increase, with no infos or warnings. - histogram_quantile(0.5) over schema 0, buckets [8, 1], count 10 and a NaN sum answers 32.0 and reports the NaN-observation info, matching Prometheus's walk that leaves the last bucket for the interpolation. The previous estimator answered 2^-0.375. - rate over a raw window answers one series while the same window over h*1 answers none: the derived sample carries no start timestamp, so it has no synthetic zero to extrapolate from. Both phases are covered, memtable and flushed SST. The native-histogram test now installs the route interceptor the way its sibling tests do, so the plan can be observed; all of its original assertions are unchanged. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
50b6b5bf82 |
feat(promql): answer max_of and min_of over two scalars natively (#10431)
* feat(promql): answer max_of and min_of over two scalars natively Prometheus defines both as two scalars in, one scalar out (experimental), and the pinned promql-parser carries the same signature, so an instant-vector argument is already a parse error. The ScalarFunction kernel implementing Go's math.Max/math.Min -- an infinity of the function's own sign beats NaN, NaN beats any other value, +0 beats -0 -- existed with its golden cases but was never reachable from a query. The call now binds through the existing parameterized-map tower: compile_call -> scalar_function_call -> ParameterizedOp::MaxOf/MinOf -> ParameterizedMath -> PointMath::Scalar -> ScalarFunction::apply, executed by map_math_with_parameters. Only MaxOf/MinOf take the scalar-in/scalar-out shape; round and clamp* keep their vector contract, and clamp's max < min emptiness rule moved into PointMath verbatim. A constant first argument is folded and materialized as the one chunk it stands for, so max_of(1, 2) answers 2 without reading input, and two scalars on different grids are refused rather than answered wrongly. The 64-case golden now runs through NativeExecutor for both functions instead of the kernel alone, and the by-position pairing test was un-ignored and pointed at the parameterized path. map_scalar_function, whose only caller was that test, is removed. The legacy planner is untouched: neither name was implemented there. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): keep the call dispatch's comment beside the check it describes Signed-off-by: discord9 <discord9@163.com> * chore(compat): move the from-versions window to v1.3.1 v1.3.1 has published release assets, so the compatibility check asks for it and fails on every branch that still names v1.3.0. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
9b627fd861 |
refactor(promql): drop four kernel entries the native runtime replaced (#10429)
* refactor(promql): drop four kernel entries the native runtime replaced Each of these had no caller left outside tests, and each has a live replacement the runtime already uses. The golden cases move to that replacement rather than being deleted with the entry, so the inputs they covered are still covered. FloatMerger::merge/merge_budgeted/merge_series -> merge_row, which reads a StateRow instead of a PartialAggregate. The merge tests now encode partials into rows and drive merge_row, keeping the fold order, the shuffled partition orders, the NaN and signed-zero cases and the admission-before-insert assertions. One test goes away: the spare capacity of an input partial's nested Vec has no counterpart in a row, whose memory is charged as Arrow buffers by the caller and whose shared parent buffer the router charges separately. map_negate -> -1 * x, which is how the compiler already lowers unary minus. NaN bit patterns differ between negation and multiplication (the sign and quiet bits are not specified by IEEE), so the assertion is that the answer is still a NaN carrying its payload; staleness is decided at instant evaluation and a Map's holes are validity bits, so neither reads that payload. Prometheus's neg cases pass unchanged. map_compare -> map_compare_literal/map_compare_series, which is what production already called. The histogram cases now assert that kernel's existing drop-and-report behaviour instead of the dead entry's whole-query error; production behaviour is unchanged. scalar_arithmetic/scalar_compare -> the compile-time constant folding the compiler performs, asserted through constant_series. ScalarFunction and map_scalar_function stay: removing them depends on the parameterized math variants another change adds. Signed-off-by: discord9 <discord9@163.com> * chore(compat): move the from-versions window to v1.3.1 v1.3.1 has published release assets, so the compatibility check asks for it and fails on every branch that still names v1.3.0. Signed-off-by: discord9 <discord9@163.com> * test(promql): spell the state kind out in the merge-row encoder The encoder stands in for the executor that hands merge_row its input, and it was calling native::state::state_kind to name the kind. #10408 deletes that helper along with the other production-dead items in its scope, so depending on it makes these two changes fail to compile when they land together. The mapping is eight lines and belongs to the encoder anyway: it is the one place that has to agree with what the runtime would derive. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
44973dba08 |
refactor: drop two helpers whose production callers are gone (#10427)
* refactor(promql): drop the uncalled merge_exact_series helper The helper carried #[allow(dead_code)] and a comment saying the fused series stack would call it once that caller landed on this branch. No caller exists: no references in src, tests-integration, tests-fuzz or tests, and no re-export. Its only dependency, merge_same_labelset, is a shared grid_series helper with other users and stays. Signed-off-by: discord9 <discord9@163.com> * refactor(datatypes): drop StateFrameRow::to_state_series Only three promql tests called it, and each asserted frame content -- labels, grid length, per-point state values and presence -- after the conversion rather than the conversion itself. They now read the row's fields() and time axis directly, so the unpack and declaration-validation path is still exercised and no assertion was dropped. The private state_float helper went with it; StateSeries itself stays, still produced by pack_aggregate_state and the repartition fixtures. Signed-off-by: discord9 <discord9@163.com> * chore(compat): move the from-versions window to v1.3.1 v1.3.1 has published release assets, so the compatibility check asks for it and fails on every branch that still names v1.3.0. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
a1be266c3c |
fix(promql): ignore histogram samples in scalar() and share one kernel (#10426)
* fix(promql): ignore histogram samples in scalar() and share one kernel Prometheus' funcScalar counts only float samples: histogram samples are ignored silently, so they neither supply a value nor make a point hold more than one element. The select kernel rejected them instead, while the native ScalarOf runtime kept its own inline loop that ignored them, so scalar(histogram_metric) answered differently depending on the route. The kernel now ignores histogram samples and the runtime calls it, deleting the inline copy. A point with one float answers that float, with only histograms or with several floats answers the Go NaN. The kernel builds its answer as one dense run over the grid: a Vec<f64> seeded with the Go NaN, where the first float at a point writes its value and a second writes NaN back, plus the counts the caller already reserves. That buffer becomes the result's Float64Array by move, so the per-point peak is the 8 bytes of values plus 1 of counts the reservation already counts. Staging through Vec<Option<f64>> and GridSeriesBuilder would have held about 33 bytes per point against a 9-byte reservation. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): resolve the evaluation grid once in scalar() grid_points already resolves and validates the grid, so asking for the schema's grid again right after could only disagree with what was just checked. Signed-off-by: discord9 <discord9@163.com> * chore(compat): move the from-versions window to v1.3.1 v1.3.1 has published release assets, so the compatibility check asks for it and fails on every branch that still names v1.3.0. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
aa9c616fe5 |
perf(promql): route proven per-series chains disjointly and forward explicit child frames (#10413)
perf(promql): keep per-series provenance and let identity-preserving windows route disjointly RawSeriesSourceExec re-derived per_series_scan by downcasting the child directly, so EnsureCooperative's wrapper silently dropped the proof and the root router fell back to MergeInputs. Peel CooperativeExec before identifying SeriesScanExec, and check the child's bound tags cover the primary key the same way a bare scan is checked. FusedSeriesExec now accepts a RawSeriesSourceExec input, and a Window whose function keeps the metric name and whose child preserves partition identity, so last_over_time/first_over_time over a per-series scan can publish each identity as soon as its chain closes. When the child already publishes this node's frame, RawSeriesSourceExec forwards it 1:1 instead of unpack→relabel→pack. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
3a032fbbc6 |
test(query): pin the native PromQL option default to off (#10371)
The native route is gated by two halves: the engine-wide `experimental_native_promql` option and the per-request `query.experimental_native_promql` extension. The extension half is already covered by the `native_promql_requested_*` tests in `query::promql::planner::native`, and the plan-shape no-op is covered by `SeriesFrameOutput`'s `a_plan_without_a_frame_declaration_is_untouched`. What was not pinned is the default of the engine option itself. Flipping that `false` to `true` would expose the native route to every query that sets the extension, and no test would fail. Assert it directly so the opt-in property cannot regress silently. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
d0324193fe |
perf(promql): avoid two copies on the native binary path (#10338)
* perf(promql): write single-field binary results in one pass A merge of one field has nothing to merge: the field filed every label set's rows already sorted by point, and already refused two rows at one point. Writing them straight into the builder drops the per-label-set map, the triples table, its sort and the row buffer that the multi-field merge needs, and bills the same columns that merge bills. Profile shares of one 20s workload on a six-thousand-series shape put merge_fields at 6.1% of samples before and 3.2% after; the query result is byte-identical. Wall-clock comparison on that host could not resolve the difference: session drift there is larger than the effect. Signed-off-by: discord9 <discord9@163.com> * perf(promql): move a chunk into its only runtime input One source feeding one runtime input is the common shape, and the chunk was cloned for it and then dropped. Moving it removes that copy. Several inputs still clone per input, and a producer state batch stays refused either way: it carries the charge for its whole payload and belongs to a different handover. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
21ada9de09 |
chore: remove stray merge-main.log (#10445)
merge-main.log at the repository root holds the output of a `git merge`
that stopped on conflicts in src/flow, followed by an `exit=1` line. It
looks like it was committed by accident in
|
||
|
|
4d4278883b |
chore(deps): point greptime-proto back at the upstream repository (#10370)
`narthex` was pinned to a personal fork branch because the `RawSeriesSource` Substrait extension only existed there: GreptimeTeam/greptime-proto#355 was still open when the back-merge landed. #355 has since been merged as `5fc04135b`, so point the pin at that upstream commit. The generated bindings are identical; only the source repository changes, and the branch no longer depends on a fork staying available. `Cargo.lock` regenerated with `cargo metadata`. Verified: `cargo check --workspace` passes. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
b49f542b25 |
fix(promql): declare the per-row name column on every native rows answer
A native rows result now always declares `__name__`, a nullable `Utf8` column appended after the labels and marked as the metric name, and each row carries the name of the series it was read from. A row whose value is computed rather than read (a math kernel, a folding aggregate, a range function) carries null there. The name is part of what the statement answers, not of how a response is displayed: the walk that derives a node's tags reports the name of a discovered source and of an `or` unconditionally, so `sum by (__name__)` groups on the identity of the series rather than collapsing every series into one group. `with_row_name_column` appends the column when the root is a samples root. It moves an existing name column to the end rather than adding a second one, because neither a set operation nor a discovered source leaves an existing name last: `or` keeps the order of the left operand and appends only the tags of the right side it has not seen, while a discovered source sorts its tags, which puts the name wherever its bytes fall. Every rows layout then answers one order - the time index, the value fields, the labels, then the name. A `label_replace(..., "__name__", ...)` result is therefore not declared twice, and a scalar root keeps its two columns. The collision check runs after the column is appended, so a name that would clash with a label still declines the layout, and a split's terminal schema declares the column while the frame producers keep the tags they were compiled with. A CTE answered natively publishes the column too, after the existing planner's columns and only when the answer declares one: `publish_legacy_schema` keeps the metadata-carrying name column out of its type-based pairing, so a label cannot claim it, and validates the published schema column by column. `abs(a or b)` now reports no `__name__` at all instead of the statement's name on every row: the operator drops the name, so the declared column's values are null. Shapes that keep the name (plain selection, filtering comparison, `or`, `topk`/`sort`, `first/last_over_time`, `sum by (__name__)`) carry it per row. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
b30e002cef |
feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN (#10092)
* feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN COPY FROM STDIN WITH (FORMAT promwrite [, version 2] [, <options>]) ingests one Prometheus Remote Write request body over the postgres wire protocol — a GreptimeDB-specific, binary COPY format (wire format code 1). The payload is byte-for-byte the body a remote-write client POSTs to /v1/prometheus/write, compression included: buffered until CopyDone and completed through the exact ingestion core that endpoint runs. To share that core, the HTTP handlers' ingestion bodies are extracted into axum-free functions (ingest_remote_write_v1/v2 plus the context extension stamping), and the HTTP routes become thin wrappers — their behavior is unchanged, and the copy layer runs the identical decode, validation, conversion, batched/prepared write paths, per-batch write-progress accounting and context extensions, under the connection's regular postgres context. The WITH options carry what the HTTP endpoint takes as its content-type/-encoding and query parameters; session-provided values (database, user) are not repeated: - version 1|2 (default 1), matching the content-type's proto - zstd, matching Content-Encoding: zstd (snappy by default) - physical_table, matching the ?physical_table= parameter - pipeline_name/pipeline_version, matching the pipeline headers (v1 only; for v2 they are accepted and ignored, exactly like the HTTP route ignores them) Table-less like the other ingestion formats; one COPY statement is one bounded request (shared 64 MiB limit); the row count reports written rows (v1) or written samples plus histograms (v2). The flush enforces the frontend's shared request-memory quota (max_in_flight_write_bytes) — the same limiter the HTTP ingestion protocols use, so COPY cannot bypass the aggregate budget. Wiring: PostgresServer::with_prom_remote_write carries the handler, metric-engine flag, validation mode and shared memory limiter. e2e: v1 body chunked across CopyData boundaries; v2 samples with readback; physical_table option routing; garbage-body decode failure with connection reuse; a tiny-quota regression where a compressed body whose decompressed size exceeds the budget fails like the HTTP route. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(servers): propagate COPY body reservations and test write accounting Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): share Prometheus logical batching with COPY ingestion Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): narrow Prometheus COPY scope and share write outcomes Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>v1.4.0-nightly-20261116 |
||
|
|
7e177b06fe |
fix(telemetry): avoid spurious log directory error on first start (#10446)
* fix(telemetry): avoid spurious log directory error on first start Since 0.2.4, tracing-appender prunes old log files when an appender with max_log_files is built, and it lists the log directory before creating it. When the directory does not exist yet, it prints "Error reading the log directory/files" to stderr. GreptimeDB sets max_log_files by default, so every start with a missing log directory printed this line. Create the log directory in build_file_appender before building the appender. Creation errors are left to the appender build, which reports them as before. Add a test that checks the stderr of a child process. Works around https://github.com/tokio-rs/tracing/issues/3612. Signed-off-by: shuxiachai <shuxiachai@163.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(telemetry): drop the first-start log directory test Review feedback: the test is not needed for such a minor change. Restore the unix-only test imports that it had merged. Signed-off-by: shuxiachai <shuxiachai@163.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: shuxiachai <shuxiachai@163.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dbd080b840 |
fix(recordbatch): treat -0.0 and +0.0 as equal in SimpleFilterEvaluator (#10443)
* test(recordbatch): compare float zero predicates with DataFusion `SimpleFilterEvaluator` filters SST rows exactly, so it has to agree with the SQL predicate that DataFusion evaluates above the scan. DataFusion compares -0.0 and +0.0 as equal. Evaluate comparisons against both zeros, NaN and NULL in Float64 and Float32 columns, and check the selected rows against DataFusion's filter. On main this fails: `v = Float64(0)` selects only the +0.0 row, while DataFusion selects both zeros. Signed-off-by: dayan02dev <pawarudayan92@gmail.com> * fix(recordbatch): compare -0.0 and +0.0 as equal in simple filters DataFusion 55 normalizes -0.0 to +0.0 before Arrow's comparison kernels, so SQL compares the two zeros as equal. SimpleFilterEvaluator called the kernels directly, and for floats they compare bit patterns: -0.0 != +0.0 and -0.0 < +0.0. Mito applies the evaluator to SST rows as an exact filter, so after a flush `v = 0.0`, `v >= 0` and `v IN (0.0)` dropped the rows holding -0.0, and `v = -0.0` dropped the rows holding +0.0. Rows still in a memtable matched. Normalize the literal when the predicate is built and the input before it is evaluated, with the helpers DataFusion uses for the same purpose. Signed-off-by: dayan02dev <pawarudayan92@gmail.com> * test(sqlness): cover flushed float rows holding -0.0 Flush rows holding -0.0, 0.0 and 1.0, then filter with `= 0.0`, `>= 0`, `= -0.0` and `IN (0.0)`. Both zeros have to match every one of them, as they do while the rows are still in the memtable. Signed-off-by: dayan02dev <pawarudayan92@gmail.com> --------- Signed-off-by: dayan02dev <pawarudayan92@gmail.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> |
||
|
|
1dbc0cd80f |
fix(mito2): avoid unrelated sparse tag expansion during compaction (#10414)
* fix(mito2): avoid unrelated sparse tag expansion during compaction Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): cover sparse compaction filter and compatibility boundaries Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): handle projected fields in sparse compaction mapper Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): remove redundant compaction projection mapper Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
1e39eab06b |
fix(ci): bound nightly kind deployment and restore inotify limits (#10423)
* fix(ci): bound nightly kind deployment and restore inotify limits Signed-off-by: evenyag <realevenyag@gmail.com> * fix(ci): use shell timeouts in composite actions Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
176bc07b53 |
docs(skills): focus release notes on user impact (#10442)
* docs(skills): focus release notes on user impact Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(skills): simplify compatibility guidance Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(skills): clarify release scope and highlight criteria Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(skills): align release targets and generated change groups Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |