mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-03 02:25:35 +00:00
0686f72dc6ad32a3ad8a684c82b2eda6fee1e5fa
6256
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dennis zhuang
|
0686f72dc6 |
chore: add cargo min-publish-age of 7 days (#9384)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>v1.3.0-alpha.1-3c0e2a8d5-20260928-1790639248 |
||
 Mohd Quamar Tyagi
|
dd2c1d1aca |
fix(meta-srv): use NoTls for disabled and Unix socket Postgres KV backends (#9059)
* fix(meta-srv): use NoTls for disabled and Unix socket Postgres KV backends Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> * fix(meta-srv): treat Postgres config as unix socket only when every host is a socket tokio-postgres dials hostaddr over TCP even when host is a socket path, and a mixed host list with Require/VerifyFull TLS would otherwise end up sending plaintext over TCP. is_unix_socket_url now parses via tokio_postgres::Config and requires no hostaddr and all-Unix hosts. Adds regression cases for the libpq keyword form, the user@ percent encoded socket URL, and the mixed host / hostaddr negatives. Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> --------- Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> |
||
 jeremyhi
|
3c0e2a8d55 |
feat: export Metric snapshots with packed Parquet objects (#9382)
* feat: export Metric snapshots with packed Parquet objects Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject empty packed export time ranges Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover cancellation during packed export I/O Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Ning Sun
|
e2b7e5e0a8 |
chore: disable unstable rustfmt features (#9379)
* chore: disable unstable rustfmt features * typo fix |
||
|
dennis zhuang
|
079ffec116 |
chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: remove unused RouteInfoCorrupted error Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
180221240a |
test: remove unused legacy compatibility test suites (#9378)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
408536af18 |
chore: remove leftover flow worker config docs and unused flow metrics (#9375)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Ning Sun
|
fedce5c5ec |
feat: support non-millisecond time index units in the logical batcher (#9346)
* feat: allow customized time index unit for metric engine table * test: provide query tests * refactor: revert unnecessary change * refactor: share timestamp unit conversions in api helper Address review feedback on the time index unit changeset: - Add shared timestamp_unit/timestamp_datatype helpers to api::helper (the only crate that sees both proto ColumnDataType and TimeUnit due to layering; common-time and datatypes have no greptime-proto dep). This removes the ColumnDataType -> TimeUnit match duplicated between operator's insert path and the OTLP logs path. - Collapse the two TimeUnit <-> ValueData matches in convert_timestamp_value_data by reusing api::helper::to_grpc_value for the construction side. - Note that convert_rows_time_unit rewrites the schema before the values, so an overflow mid-batch leaves the request half-converted; harmless because the error aborts the whole insert request. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: align time units per destination table and floor remote-read timestamps Address review feedback on PR #9236: - Align each metric insert request to the unit of the table it actually targets: an existing logical table keeps its own unit (it may be bound to a different physical table than the one selected by the request), and only new tables use the selected physical table's unit. The previous blanket conversion rewrote valid millisecond samples to the selected physical table's unit and the engine rejected them. Regression test: writing an existing millisecond logical table and a new table in one request that selects a microsecond physical table. - Remote read now floors narrowing timestamp conversions towards negative infinity (div_euclid), consistent with Timestamp::convert_to on the ingestion path; arrow's cast truncates towards zero and returned -1ms for a stored -1001us. Widening (second -> millisecond) keeps the exact arrow cast. Regression test: a negative, non-aligned timestamp round-trips as -2ms. Signed-off-by: Ning Sun <sunning@greptime.com> * perf: fold time unit alignment into existing table lookups Address review feedback on PR #9236: - The per-destination unit alignment no longer runs its own pass of table lookups: create_or_alter_tables_on_demand gains an align_time_index_unit parameter (metric engine path only) and converts each request inside the lookups it already performs — existing tables to their own unit, new tables to the selected physical table's. Default ingest paths now issue zero additional catalog lookups compared to main; the separate alignment pass remains only in the opt-in logical batcher pre-gate, next to the eligibility check that already looks up the same tables. - convert_rows_time_unit indexes the time index position directly (validate_column_count_match guarantees row widths) instead of Optional get_mut; the gate-side alignment validates widths itself. Signed-off-by: Ning Sun <sunning@greptime.com> * perf: resolve the batcher time index guard once per write target All batches of one remote write request share the same write target (catalog, schema, physical table), so the batcher time index guard now resolves each distinct target once instead of once per batch. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: support non-millisecond time index units in the logical batcher Make the logical table batcher's bulk encode path unit-aware so physical metric tables with a non-millisecond time index (e.g. TIMESTAMP(6)) can use logical batching instead of falling back to the ordinary insert path. - rows_to_aligned_record_batch builds the time index column in the TARGET schema's unit, converting any timestamp encoding via Timestamp::convert_to (flooring on narrowing, consistent with the ordinary insert path). - New tables created by the batcher use the selected physical table's time index unit (resolved once per submit; a missing physical table keeps the millisecond auto-create default). - columns_taxonomy and the can_batch_metric_rows schema whitelist accept any timestamp unit; the prometheus remote write v1/v2 batcher gates and the OTLP pre-gate alignment are removed together with Inserter::align_metric_row_inserts_time_unit, as the batcher now converts internally. Closes #9342 Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address review issue * refactor: drop the OTLP pre-gate unit alignment made redundant by the bulk path The main merge of #9236 (squash) resurrected the OTLP pre-gate alignment and Inserter::align_metric_row_inserts_time_unit, which this branch had removed. Drop them again: - The pre-gate existed because the #9236-era bulk eligibility gate only accepted millisecond schemas, so nanosecond-encoded OTLP requests had to be converted before the check. This branch makes the bulk path unit-aware (the gate accepts all time index units and batch alignment converts each request to its destination's unit), so the pre-gate is redundant and only added N+1 catalog lookups per batched request — the very lookup-count overhead raised in the #9236 review. - The per-destination unit semantics it implemented remain enforced in the two paths that need them: the ordinary insert path (create_or_alter_tables_on_demand converts inside its existing table lookups) and the batched path (batch alignment resolves each destination schema and converts to it). test_otlp_logical_batcher_alignment (the test the pre-gate originally fixed) and the mixed-physical-table regression both pass without it. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover OTLP batcher cross-physical fallback and nanosecond physical Extend the logical batcher integration coverage for the cases previously guarded by the removed OTLP pre-gate alignment: - test_otlp_logical_batcher_fallback_for_cross_physical_destination: with the batcher enabled, an OTLP request targeting an existing logical table bound to another physical table must NOT enter the batcher (the bulk eligibility check rejects the destination binding) and the ordinary insert path must convert it to the destination's unit (60s -> 60_000_000us). - test_otlp_logical_batcher_non_millisecond_physical_table now covers both microsecond and nanosecond physical tables (parameterized), asserting batcher submissions and unit-precise stored values. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: validate per-batch physical bindings and avoid intermediate timestamp buffers Address review feedback on PR #9346: - accepts_bulk_destinations dedupes on (schema, table, selected physical) instead of (schema, table): one request can select different physical tables per batch (per-series x_greptime_physical_table labels), and the old key let a second selection skip validation and flush rows through the wrong physical's regions. Missing tables additionally reject conflicting physical selections within the same request. Regression test covers an existing destination, a missing destination, and a consistent selection (which must still batch). - The timestamp column builder appends each value directly into the target-unit Arrow builder; values already in the target unit (the unchanged millisecond fast path) are appended without conversion, so the default millisecond physical pays no Timestamp construction or intermediate Vec allocation. - The non-millisecond batching tests assert the submit_build_and_align counter, which increments on every batcher submission in both acknowledgement modes, so a silent fallback to ordinary insertion fails the tests instead of passing on stored values alone. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 discord9
|
28e415a103 |
feat(query): implement PromQL @ modifier on vector and matrix selectors (#9224)
* feat(query): implement PromQL @ modifier on vector and matrix selectors The planner previously dropped the `at` field of selectors (`at: _`), so `some_metric @ 300` silently returned step-following values instead of the samples anchored at the fixed timestamp. Anchoring follows Prometheus's `setOffsetForAtModifier` + `refetch` semantics: resolve the anchor (`@ <ts>`, `@ start()`, `@ end()`), apply `offset` to the anchor, rewrite the selector offset to `eval_start - anchor`, scan only the anchored window, then report the same window at every evaluation step via a grid-wide-replay `InstantManipulate`. `@ start()` / `@ end()` resolve against the statement's evaluation range (`stmt_start` / `stmt_end`), which subquery planning does not rewrite. Timestamps before the Unix epoch are accepted; unrepresentable ones are rejected with `AtModifierTimestampOutOfRange` instead of wrapping. Selectors without `@` are planned exactly as before. Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-2 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): center predict_linear on the evaluation timestamp predict_linear_impl used the window's last sample time as the evaluation timestamp, and the UDF took only (ts_range, value_range, t) with no channel for the current instant. After a range selector is folded for @ (or with an offset) the same window is replayed at every step, so the prediction stayed constant at the anchor's answer; even a plain window ended before the step produced a stale value. Give prom_predict_linear a 4th argument carrying the step's evaluation instant (ms timestamp), derived in the planner from the row's time index plus the offset the window was folded with (at_offset for @, offset_ms otherwise, recorded on PromPlannerContext). The regression is centered on that instant, matching Prometheus' use of enh.Ts. The mixed float/native-histogram path forwards the same argument. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): tighten @ modifier planner and predict_linear - range_fold_offset is always a concrete millisecond offset, not an Option; the single-step replay helper no longer wraps an infallible plan in Result. - predict_linear's eval timestamp is always cast to Timestamp(ms) at the call site, so the UDF drops its dead Int64 branch and the extra func_name argument. - Drop two redundant comparison cases from the at_modifier sqlness test and fix the lookback window comment to the half-open (0s, 300s]. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): accept parse-time rejection of @ on Windows `@ 1e16` is 10^19 milliseconds, beyond i64::MAX. A Unix SystemTime holds it and the planner rejects the anchor it cannot represent, but a Windows SystemTime tops out near 1.8e12 seconds, so the parser's checked_add fails first and the same literal is rejected while parsing. Accept either rejection path so the test passes on both platforms. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): avoid replaying label_join over rewritten series keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): narrow anchored range call promotion Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): address @ modifier review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): satisfy super import format check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): address at modifier review follow-ups Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
jeremyhi
|
03823a9a01 |
feat(log-store): add the enqueued acknowledgement mode to the object store WAL (#9358)
* feat(log-store): add the enqueued acknowledgement mode to the object store WAL Add `ack_mode` (`durable` by default, or `enqueued`) and the backlog thresholds `max_unpersisted_bytes` and `max_unpersisted_age` to the object store WAL config, validated by the datanode and the store. In the `enqueued` mode `append_batch` returns on admission with the entry ids assigned and the object is created in the background. At a backlog threshold the next append is held back until an upload completes. A transient create failure is repeated under the same sequence with the same bytes; any conflicting object poisons the store. `stop` uploads the backlog, or returns the error that dropped it once stop began. `obsolete` clamps the watermark to the durable entry id, and an id the store handed out needs no sequence floor. Add `LogStore::wait_durable` with a default that returns at once. The object store WAL answers it once the region is durable and indexed through the entry id, and fails it after a backlog was lost. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): poison an enqueued store on permanent create failures Repeat a failed create in the enqueued mode only when the storage error is retryable; any other storage error poisons the store. A conflicting object poisons an enqueued store without reading its epoch, so a failed header read cannot turn the conflict into a retry. A durability wait for an id above the highest id the store handed out now waits for the handed-out ids of the region below it instead of returning at once. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): poison on permanent create failures after stop begins A create that fails with a storage error that is not retryable poisons an enqueued store even after stop began; only a transient failure drops the backlog without poisoning. Durability waiters whose callers stopped waiting are pruned before a new waiter is queued. The backlog age test no longer depends on a follow-up append finishing within the threshold. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): answer a durability wait once no earlier entry is pending A durability wait now returns once the region holds no entry at or below the target that is handed out but not durable, instead of waiting for the largest id handed out to the region. A later object of the region that is still being created no longer holds back a wait whose target it does not cover. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): order the pending durability wait check after the actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(store-api): state that the default wait_durable keeps each store's guarantee The default `LogStore::wait_durable` returns at once, which keeps each log store's own acknowledgement guarantee; Raft Engine with `sync_write = false` acknowledges before its periodic sync, so the documentation no longer claims that every entry id a caller holds is durable. The object store WAL configuration test now also serializes the new options and reads them back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): limit the acknowledgement guarantees to the durable mode Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): repeat enqueued creates that a retry layer marks persistent An object store wrapped in the OpenDAL retry layer reports a temporary error that outlasted its retries as persistent rather than temporary. The enqueued mode now repeats a create after any storage error that is not permanent, so a transient outage behind the retry layer no longer poisons the store and drops the acknowledged backlog; after stop began such a failure still drops the backlog without poisoning. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover a persistent create failure after stop begins Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
  LFCandgreptimedb-ci
|
a310ca2bcf |
ci(query-regression): bump RUNNER_IMAGE_EPOCH to 6 (#9381)
ci(query-regression): bump RUNNER_IMAGE_EPOCH to 6 for image m-0xidkbbavm3suxqd0y1m Co-authored-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com> |
||
  
|
75bd8e9ce6 |
feat: add HDFS object storage backend (#8701)
* feat: add HDFS object storage backend Signed-off-by: Minghan2005 <cambrianocean@gmail.com> * fix: make HDFS storage operations durable Gate the native HDFS backend behind an explicit feature. Publish writes through same-directory temporary files and atomic HDFS Rename2 replacement, and provide streaming copy fallback for COPY_REGION. Add regression coverage for interrupted writes and the region-copy path. Signed-off-by: Minghan2005 <cambrianocean@gmail.com> * ci: run HDFS object store tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: note HDFS temporary file cleanup follow-up Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: enable HDFS object storage by default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: remove redundant HDFS build feature notes Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: Minghan2005 <cambrianocean@gmail.com> Signed-off-by: jeremyhi <fengjiachun@gmail.com> Co-authored-by: Minghan2005 <cambrianocean@gmail.com> Co-authored-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
ffdd6d09a6 |
perf(index): cut allocations when building bloom and inverted indexes (#9359)
* perf(index): build bloom filters from element hashes without per-token allocation Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(index): speed up inverted index building with hashed buffers and sync pushes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(index): use BuildHasher::hash_one for element hashes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(index): require callers to act on spill requests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): insert bloom hashes one by one to keep segment set capacity bounded Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): add index build and bloom search benchmarks Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): keep applier setup out of the bloom search benchmark Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): skip empty spills and keep the old inverted sort memory estimate Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): stream fulltext token hashes and test spill dispatch Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(index): note non-ASCII tokens still allocate in analyze_text_hashes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(index): narrow the allocation note to case-insensitive non-ASCII tokens Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
bc5ad4b416 |
perf(mito2): batch index page loads and share cached bloom metadata (#9360)
* perf(mito2): load missing index pages of all ranges in one read Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(mito2): drop unused IndexCacheMetrics::merge Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(mito2): share cached bloom filter metadata instead of cloning it Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): only merge adjacent ranges when batch reading index pages Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): skip empty ranges when batch loading index pages Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(mito2): fetch index ranges concurrently Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): return Arc<BloomFilterMeta> from RecordingReader Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Weny Xu
|
1d365adb3d |
fix(ci): update the shared Actions runner to v2.337.0 (#9376)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Ning Sun
|
55b7e08a1a |
feat: allow customized time index unit for metric engine table (#9236)
* feat: allow customized time index unit for metric engine table * test: provide query tests * refactor: revert unnecessary change * refactor: share timestamp unit conversions in api helper Address review feedback on the time index unit changeset: - Add shared timestamp_unit/timestamp_datatype helpers to api::helper (the only crate that sees both proto ColumnDataType and TimeUnit due to layering; common-time and datatypes have no greptime-proto dep). This removes the ColumnDataType -> TimeUnit match duplicated between operator's insert path and the OTLP logs path. - Collapse the two TimeUnit <-> ValueData matches in convert_timestamp_value_data by reusing api::helper::to_grpc_value for the construction side. - Note that convert_rows_time_unit rewrites the schema before the values, so an overflow mid-batch leaves the request half-converted; harmless because the error aborts the whole insert request. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: align time units per destination table and floor remote-read timestamps Address review feedback on PR #9236: - Align each metric insert request to the unit of the table it actually targets: an existing logical table keeps its own unit (it may be bound to a different physical table than the one selected by the request), and only new tables use the selected physical table's unit. The previous blanket conversion rewrote valid millisecond samples to the selected physical table's unit and the engine rejected them. Regression test: writing an existing millisecond logical table and a new table in one request that selects a microsecond physical table. - Remote read now floors narrowing timestamp conversions towards negative infinity (div_euclid), consistent with Timestamp::convert_to on the ingestion path; arrow's cast truncates towards zero and returned -1ms for a stored -1001us. Widening (second -> millisecond) keeps the exact arrow cast. Regression test: a negative, non-aligned timestamp round-trips as -2ms. Signed-off-by: Ning Sun <sunning@greptime.com> * perf: fold time unit alignment into existing table lookups Address review feedback on PR #9236: - The per-destination unit alignment no longer runs its own pass of table lookups: create_or_alter_tables_on_demand gains an align_time_index_unit parameter (metric engine path only) and converts each request inside the lookups it already performs — existing tables to their own unit, new tables to the selected physical table's. Default ingest paths now issue zero additional catalog lookups compared to main; the separate alignment pass remains only in the opt-in logical batcher pre-gate, next to the eligibility check that already looks up the same tables. - convert_rows_time_unit indexes the time index position directly (validate_column_count_match guarantees row widths) instead of Optional get_mut; the gate-side alignment validates widths itself. Signed-off-by: Ning Sun <sunning@greptime.com> * perf: resolve the batcher time index guard once per write target All batches of one remote write request share the same write target (catalog, schema, physical table), so the batcher time index guard now resolves each distinct target once instead of once per batch. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address review issue --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
dennis zhuang
|
0a03e64d8c |
perf(index): batch bloom filter searches across row groups (#9361)
* perf(index): search bloom filters of all row groups in one read Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(index): bound the filter bytes of each batched bloom search Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(index): count batch filter bytes exactly as the search reads them Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(index): allow single-range groups in the budget test Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): cover filters shared across segments in the budget test Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Ning Sun
|
43eaea7a9a |
fix(ci): teach check-builder-rust-version.sh to handle stable channels (#9369)
* fix(ci): teach check-builder-rust-version.sh to handle stable channels
The script extracted a YYYY-MM-DD date from rust-toolchain.toml to
compare against the rustc build date inside the dev-builder image —
a nightly-era design. With channel = "1.96.1" there is no date in
the file, so every release build failed with 'Error: No rust toolchain
version found in rust-toolchain.toml'.
Extract the channel token instead and branch on it:
- stable channel (X.Y[.Z]): require the builder image's rustc to
exactly match the pinned version
- nightly-YYYY-MM-DD: keep the legacy date-difference check
Verified against a mocked docker for all four paths (stable match /
mismatch, nightly fresh / stale).
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore(toolchain): finish stable-migration cleanup in docs and query-regression pin
- README/AGENTS: the toolchain is now stable Rust pinned by
rust-toolchain.toml, not nightly
- query-regression: align the benchmark toolchain pin with the
workspace (nightly-2026-03-21 = 1.96.0-nightly -> stable 1.96.1),
including the exact-version assertions (cargo 356927216, rustc
31fca3adb, both 2026-06-26) and the runner image default
The query-regression runner image must be rebuilt and
QUERY_REGRESSION_ECS_IMAGE_ID bumped together with these pins
(per .github/runner-scale-sets/query-regression/README.md) before
the next regression run.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(query-regression): derive the Rust toolchain pin from rust-toolchain.toml
Replace the hard-coded RUSTUP_TOOLCHAIN value and the hard-coded
version strings in the runner Verify assertions with a pin resolved
from rust-toolchain.toml:
- the always-running test-tooling job exports the channel parsed from
rust-toolchain.toml as a job output
- query-regression sets RUSTUP_TOOLCHAIN from that output
- the Verify step escapes the pin into the cargo/rustc/active-toolchain
regexes at runtime; the exact commit hash and date are asserted
generically since a stable version identifies the release
Removing the redundant require_eq (workflow yaml vs runner env) since
both now flow from the single source of truth. When rust-toolchain.toml
is bumped, the run fails with a clear signal until the runner image is
rebuilt with the new toolchain, keeping the existing lockstep contract.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(query-regression): derive the runner image toolchain from rust-toolchain.toml
Remove the hard-coded 'ARG RUST_TOOLCHAIN=1.96.1' from the
query-regression runner Dockerfile. The pin is now parsed from a
COPY'd rust-toolchain.toml at build time (the bootstrap script builds
with the repo root as context, so the file is in the build context):
- rustup-init installs the parsed channel as the default toolchain
- the baked ENV RUSTUP_TOOLCHAIN is dropped: the rustup default makes
bare cargo/rustc resolve correctly without it, and the workflow
supplies RUSTUP_TOOLCHAIN explicitly at run time
- the build-time self-verification asserts the active toolchain
against the same parsed pin
With this, rust-toolchain.toml is the single source of truth for the
benchmark toolchain end to end: the image bakes whatever the toml says
at build time and the workflow asserts against the toml at run time.
A toolchain bump now only requires rebuilding the image.
The changed mechanics were verified natively with the real rustup-init
1.29.0 and the real 1.96.1 toolchain (registry pulls are unavailable
in this sandbox): parsing, default-toolchain installation without
RUSTUP_TOOLCHAIN env, bare cargo/rustc resolution, and the
active-toolchain assertion for both '(default)' and '(overridden)'
output forms.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(query-regression): rebuild the runner image automatically on toolchain changes
Mirror the dev-builder automation for the query-regression ECS runner
image: a new rebuild-query-regression-runner-image.yaml workflow runs
whenever rust-toolchain.toml or the query-regression runner directory
changes on main (or via manual dispatch). It drives the existing
build-ecs-image.py ops tool, then completes the documented lockstep
updates in order: bump RUNNER_IMAGE_EPOCH in query-regression.yml and
push the commit to main, and only then point the
QUERY_REGRESSION_ECS_IMAGE_ID repo variable at the new image, so the
next regression run picks up image and epoch together.
Also fix build-ecs-image.py to stage rust-toolchain.toml into the
temporary docker build context: the AMI path builds the embedded
Dockerfile from an empty /tmp/image-context, which would break on the
Dockerfile's COPY of rust-toolchain.toml introduced earlier. The
user-data now base64-stages the toml next to the Dockerfile before
docker build.
Verified: py_compile, render_user_data round-trip (mkdir -> stage ->
docker build ordering), and the RUNNER_IMAGE_EPOCH bump sed against
the real workflow file. Requires a new ALIYUN_ECS_BASE_IMAGE_ID repo
variable (Ubuntu 24.04 public image id in the region); all other
secrets/vars are shared with the provisioning job.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci: fold the query-regression runner rebuild into release-dev-builder-images.yaml
Merge the standalone rebuild workflow into the existing builder-image
release workflow, as one entry point for all builder artifacts:
- push paths extended with .github/runner-scale-sets/query-regression/**
- new 'release_query_regression_runner_image' dispatch input
- a 'changes' job diffs the pushed range (github.event.before..sha,
with an everything-changed fallback for dispatch or unknown bases)
so each expensive rebuild only fires for its own paths:
rust-toolchain.toml gates both, docker/dev-builder/** gates the
dev-builder images, the query-regression runner directory gates the
ECS image rebuild
- the rebuild job itself is unchanged from the standalone workflow
(build-ecs-image.py, then RUNNER_IMAGE_EPOCH commit to main, then
the QUERY_REGRESSION_ECS_IMAGE_ID variable update)
The dev-builder jobs, their ECR/CN/tag-update dependents, and the
runner rebuild now share one workflow; the changes filter preserves
the previous on-push behavior for the dev-builder images while the
runner rebuild keeps its own trigger. The epoch-bump commit only
touches query-regression.yml, which is outside the trigger paths, so
no re-trigger loop.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(ci): auto-resolve the ECS base image for the runner rebuild
The automated rebuild failed with 'Missing required configuration:
--base-image-id' because the ALIYUN_ECS_BASE_IMAGE_ID repo variable
does not exist yet (it was flagged as a one-time setup item).
Remove the setup dependency instead: build-ecs-image.py now defaults
--base-image-id to the latest public Ubuntu 24.04 x86_64 system image
in the region (DescribeImages with image_owner_alias=system), so no
manual variable is required. The runner Dockerfile pins every tool
version itself, so base-image drift is low-risk; --base-image-id or
the ALIYUN_ECS_BASE_IMAGE_ID variable still pin a specific base image
deterministically, and the workflow only passes the flag when the
variable is set.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* docs(query-regression): clarify what an image rebuild requires
A routine runner-image rebuild needs no manual file updates: the
rebuild job updates QUERY_REGRESSION_ECS_IMAGE_ID and
RUNNER_IMAGE_EPOCH; the toolchain derives from rust-toolchain.toml;
uv, sccache, otelgen, rustup, and the runner base are pinned by
digest/sha/commit in the Dockerfile. Only an apt package revision
bump (mold, protoc, python3) between rebuilds requires bumping the
corresponding Verify pins, and that failure is loud with the observed
version.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(ci): correct SDK field names in the base-image resolver
DescribeImagesRequest takes 'ostype' (not 'os_type') and the image
items expose 'osname'/'osname_en' (not 'os_name') in the pinned
alibabacloud_ecs20140526 SDK range, so the auto-resolution added in
|
||
|
discord9
|
f956f4da30 |
refactor(promql): split planner into focused modules (#9354)
* refactor(promql): move planner tests into separate module Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): move function-specific planner methods into module Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): move OR planner method into module Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): move binary island planner into module Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): keep binary result labels in planner root Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): limit planner helper visibility to parent module Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): group set operator planning and localize imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): use crate-rooted planner imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
825402dc70 |
ci: add manually triggered tracesbench workflow (#9372)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
678aa81cae |
fix(client): complete transport lane isolation (#9030)
* fix(client): complete transport lane isolation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(client): deprecate legacy single-manager constructors Per review: mark the legacy single-manager constructors and helper as deprecated so callers move to the isolated query/control manager pair. Tests intentionally exercising the legacy path are annotated with `#[allow(deprecated)]`. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(client): update deprecated constructor callers for CI Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
  DeviousCardiandClaude Opus 5.5
|
d5a6a9ac05 |
fix(auth): follow symlink chains in watch_file_user_provider (#9365)
* fix(auth): follow symlink chains in watch_file_user_provider The shared file watcher only watched the immediate parent directory of the configured path, so swapping a directory symlink elsewhere in the chain (e.g. `ln -sfn secrets.2 secrets`, as sops-nix does) never produced an event and the new users file was not loaded until restart. The watcher now resolves each path component by component and also watches the directory holding every symlink it meets, plus the directory of the final file. After each relevant event the chain is re-resolved and the watches are re-armed, so later in-place edits of the new target are seen. Only the final file's directory is required at startup; a link directory that cannot be watched (e.g. mode 0711) is logged and retried later. Events are now filtered to the chain's links and final files, so that watching e.g. /tmp does not reload on unrelated files. Names are compared case-insensitively on macOS and Windows. TLS cert reload uses the same helper. Close #9310 Signed-off-by: Aarav <aaravsjadav@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL * fix(common-config): warn when the file watcher watches the root dir On macOS /tmp, /var and /etc are symlinks under /, so following a symlink chain there adds / to the watched directories, and the FSEvents backend then sees every event on the volume. Log a warning when that happens. Also shorten the comment on skipped link directories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL Signed-off-by: Aarav <aaravsjadav@gmail.com> --------- Signed-off-by: Aarav <aaravsjadav@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
discord9
|
4ffd93d8dc |
fix(query): preserve global limits with DataFusion optimizer fixes (#9071)
* fix(query): preserve global fetch through physical-optimizer repartitioning Aggregate soft-limit pushdown inserts a global CoalescePartitionsExec(fetch) below an AggregateExec whose input is HashPartitioned/KeyPartitioned. The DataFusion physical optimizer's EnforceDistribution/EnforceSorting passes then dropped or rewrote that fetched coalesce, losing the global limit. Root-caused and fixed in the DataFusion fork (GreptimeTeam/datafusion PR #35); this repo pins to that fix commit and adds regression coverage. - pin datafusion to GreptimeTeam/datafusion 48510c7f5 (fetch preservation fix) - global_limit.rs: adapt to new DF API (input_distribution_requirements, child_distribution, replace_children+Recompute); accept HashPartitioned and KeyPartitioned as partitioning to restore; add regression unit tests - sqlness: extend standalone common ssts and refresh distributed ssts_limit result for the preserved fetch plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: drop dead HashPartitioned compat arms and test wrapper - remove HashPartitioned compatibility matches (pinned DF only emits KeyPartitioned); drop their #[expect(deprecated)] attributes - remove single-use agg_with_limit helper whose seed limit is always overwritten by the soft-limit transform - restore Cargo.lock dependency edges unrelated to the DataFusion pin bump (cargo update --precise had downgraded 26 unrelated packages) Addresses oracle ablation review. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(deps): pin DataFusion to merged fetch preservation fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
jeremyhi
|
bf67246645 |
perf: bound concurrent Metric export writers (#9296)
* fix: preserve objects after conditional Metric export collisions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: share Metric export writer and payload budgets Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover bounded Metric export conversion and mixed roundtrips Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve export output after ambiguous close Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: align HTTP export timeout with managed cancellation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: trim redundant Metric export roundtrips Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: bound pending Metric export writer handles Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: protect export files on failed writes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve ordinary export cleanup and large rows Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: report Metric export collisions as invalid arguments Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up failed conditional export closes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support view arrays in managed ordinary export Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: expect successful cleanup after failed close Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up unsynced overwrite after close failure Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: scope export JSON expansion estimates to JSON columns Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: rechunk ordinary export batches within the write budget Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
411c0f173e |
fix(mito2): stop parallel flat scan tasks once the receiver is dropped (#9368)
* fix(mito2): stop parallel flat scan tasks once the receiver is dropped spawn_flat_scan_task ignored send errors, so after a query was cancelled the detached task kept reading its source to EOF. Break out of the loop when the receiver is gone. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(mito2): log when a parallel scan task stops on a dropped receiver Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(mito2): include the send error in the parallel scan stop log Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
2b596fd52d |
ci: deploy MinIO chart with Silo images (#9362)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Weny Xu
|
5ffd01a70a |
fix: preserve primary key order when syncing columns (#9189)
* fix: preserve primary key order when syncing columns Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: reject invalid SyncColumns metadata Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
194bc2fb3c |
feat(log-store): add the object store WAL durable write path (#9320)
* feat(log-store): add the object store WAL durable write path ObjectStoreLogStore can now write. append_batch admits entries into one open batch and assigns object-sequence-major entry ids at admission. The batch is sealed by size, by the flush interval, or before a region would run past the position range, and sealed batches are uploaded with at most four conditional creates in flight, started in sequence order. Created objects are indexed in sequence order, and an append is acknowledged only once its object is durable and indexed. A transient create failure rolls back the failed batch and every later batch unless a later object is already durable, in which case the store poisons itself with a history-gap error. A conflicting object, an encoding or catalog error, or taking the last representable sequence poisons the store. obsolete now goes through the actor and raises the sequence floor together with the watermark, refusing with a retryable error while the next sequence is not settled. stop drops the open batch and the batches whose create has not started, and lets creates in flight finish. Only the durable acknowledgement mode exists. A testing feature exposes hooks to wait for admissions, seal the open batch, and hold or fail creates. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): synchronize object store WAL write tests with the actor Tests that assert nothing happened round-trip a command through the actor instead of yielding the test task, the conflict test waits for the object it expects, and the obsolete-behind-stop test holds the command channel itself so the unanswered command is deterministic. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): bound admitted WAL appends and never reuse a failed sequence A create that reports an error may still have written its object, so the sequences of failed batches are no longer handed out again: the next batch keeps the sequence after the last sealed one and retries are assigned new ids. A retry batched differently can no longer conflict with that object. As the next sequence never moves back, the sequence floor of obsolete only waits for an open batch that has handed out ids. Appends now arrive on their own bounded channel, which the actor stops reading while MAX_SEALED_BATCHES batches wait to become durable, so a stalled object store holds callers back instead of growing the backlog, while stop and obsolete are still handled. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): reserve room for two seals before admitting a WAL append One admission can seal the open batch before an append that would exhaust its positions and then the append's own batch, so the actor takes an append only while two more sealed batches fit under MAX_SEALED_BATCHES. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(log-store): chain object store WAL objects and recover the latest chain A conditional create can fail with an unknown outcome while its object is stored, or still lands later, and Mito reuses the row sequences of a failed append. Replaying such an object next to a later acknowledged one can let the unacknowledged rows win after a restart. Format version 2 gives every object header its writer's epoch, a link to the object it extends (sequence and writer instance) and a header CRC32, and allows objects without segments. Recovery replays only the chain ending at the complete object with the largest epoch and sequence: a link holds when its predecessor is present with the recorded writer instance, or is missing below every present object. Objects off the chain are orphans that are never replayed but keep their sequences. Each open writes an empty object that starts an epoch above every present object, linked to the recovered tip, before it accepts writes, so a late object of an earlier instance never ends the chain. A start object that meets an object of an earlier epoch moves to the next sequence; one of an equal or later epoch fails the open. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): break the WAL chain at every missing predecessor Accepting a missing predecessor below every present object lets a late object that lands below the chain change which links hold. Nothing collects objects yet, so a missing predecessor now always breaks the link, and recovery fails when objects are present but none completes a chain. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep the object store WAL format at version 1 The object store WAL has not been enabled anywhere, so no object in the previous layout exists and the chained header can stay version 1. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): link object store WAL objects by epoch Only one store instance writes under an epoch, since every open starts an epoch above every present object and a start object that meets the same or a later epoch fails the open. The epoch therefore identifies the instance, and the random writer instance id is dropped from the header. A link now records the sequence and the epoch of the object it extends, and holds when the predecessor carries that epoch. The header shrinks to 46 bytes, and the store logs its epoch when it opens. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail an open whose start object is already present Without a random writer instance, two opens that recover the same objects encode byte-identical start objects, and a conditional create treats the same bytes as its own retry. A start object that is already present therefore fails the open with a retryable error instead of letting both opens claim the epoch; the next open counts the object and starts a later epoch. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): derive the WAL epoch from the claimed start sequence Two opens can recover different listings when a late object lands across a sequence gap between them, pick the same largest epoch plus one, and both create their start objects under different sequences. The epoch of an instance is now one above the sequence its start object claims, so a successful create decides the epoch and no two instances share one. It stays above every epoch recovery listed, and an object that carries an epoch above the next sequence fails the open. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): never run a held WAL create after the store is dropped The create test hook ignored the closed hold channel, so a create parked when the store was dropped could still run. It now returns without creating. Drop the per-admission bookkeeping of issued entry ids, which nothing reads, and move the parked I/O documentation to its helper. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover a WAL start object stored with an unknown outcome Add a fault-injection test in which the create of the start object stores the object but reports an error: the open fails without moving to another sequence, and the next open counts the stored object and claims a later epoch. Rename the test helper that writes a whole object from a given header to put_object_with_header, and drop a needless clone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): make the held-create drop test deterministic Keep the actor running while the store drops the hold sender, so the parked create always completes on the closed channel instead of racing the actor's exit. Drop a comment that restates epoch_of. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Yingwen
|
3c2aac0a55 |
feat: add manual series index reconciliation (#9323)
* feat(mito): support manual series index reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * feat(storage): route series index build requests Signed-off-by: evenyag <realevenyag@gmail.com> * feat(admin): add BUILD_SERIES_INDEX Signed-off-by: evenyag <realevenyag@gmail.com> * test: specify compaction type in series index fixtures Signed-off-by: evenyag <realevenyag@gmail.com> * test: correct series index SQL fixtures and error assertions Signed-off-by: evenyag <realevenyag@gmail.com> * test(compat): preserve legacy index rebuild across upgrades Signed-off-by: evenyag <realevenyag@gmail.com> * test(sql): cover series index admin validation Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): bound series index maintenance queue Signed-off-by: evenyag <realevenyag@gmail.com> * docs: remove series index how-to guide Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove index build upgrade compatibility case Signed-off-by: evenyag <realevenyag@gmail.com> * fix: address series index reconciliation review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * fix: bound manual series index reconciliation admission Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update greptime-proto to merged index build options Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
  github-actions[bot]andgreptimedb-ci
|
b547a3e369 |
ci: update dev-builder image tag (#9355)
Signed-off-by: greptimedb-ci <greptimedb-ci@greptime.com> Co-authored-by: greptimedb-ci <greptimedb-ci@greptime.com> |
||
 shuiyisong
|
b5c0e24625 |
feat(otlp): add histogram rejection metrics and compact zero buckets (#9321)
* feat(otlp): add histogram rejection metrics and compact zero buckets Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test isolation Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix minor naming Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
dennis zhuang
|
357c935804 |
fix: make vector aggregates work with GROUP BY and partial aggregation (#9338)
* fix: make vector aggregates work with GROUP BY and partial aggregation Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: make partitioned vec_avg case distinguish weighted averages Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
822042c7bb |
feat(log-store): chain object store WAL objects and recover the latest chain (#9334)
* feat(log-store): chain object store WAL objects and recover the latest chain A conditional create can fail with an unknown outcome while its object is stored, or still lands later, and Mito reuses the row sequences of a failed append. Replaying such an object next to a later acknowledged one can let the unacknowledged rows win after a restart. Format version 2 gives every object header its writer's epoch, a link to the object it extends (sequence and writer instance) and a header CRC32, and allows objects without segments. Recovery replays only the chain ending at the complete object with the largest epoch and sequence: a link holds when its predecessor is present with the recorded writer instance, or is missing below every present object. Objects off the chain are orphans that are never replayed but keep their sequences. Each open writes an empty object that starts an epoch above every present object, linked to the recovered tip, before it accepts writes, so a late object of an earlier instance never ends the chain. A start object that meets an object of an earlier epoch moves to the next sequence; one of an equal or later epoch fails the open. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): break the WAL chain at every missing predecessor Accepting a missing predecessor below every present object lets a late object that lands below the chain change which links hold. Nothing collects objects yet, so a missing predecessor now always breaks the link, and recovery fails when objects are present but none completes a chain. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep the object store WAL format at version 1 The object store WAL has not been enabled anywhere, so no object in the previous layout exists and the chained header can stay version 1. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): link object store WAL objects by epoch Only one store instance writes under an epoch, since every open starts an epoch above every present object and a start object that meets the same or a later epoch fails the open. The epoch therefore identifies the instance, and the random writer instance id is dropped from the header. A link now records the sequence and the epoch of the object it extends, and holds when the predecessor carries that epoch. The header shrinks to 46 bytes, and the store logs its epoch when it opens. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail an open whose start object is already present Without a random writer instance, two opens that recover the same objects encode byte-identical start objects, and a conditional create treats the same bytes as its own retry. A start object that is already present therefore fails the open with a retryable error instead of letting both opens claim the epoch; the next open counts the object and starts a later epoch. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): derive the WAL epoch from the claimed start sequence Two opens can recover different listings when a late object lands across a sequence gap between them, pick the same largest epoch plus one, and both create their start objects under different sequences. The epoch of an instance is now one above the sequence its start object claims, so a successful create decides the epoch and no two instances share one. It stays above every epoch recovery listed, and an object that carries an epoch above the next sequence fails the open. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover a WAL start object stored with an unknown outcome Add a fault-injection test in which the create of the start object stores the object but reports an error: the open fails without moving to another sequence, and the next open counts the stored object and claims a later epoch. Rename the test helper that writes a whole object from a given header to put_object_with_header, and drop a needless clone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Ning Sun
|
20619273ee |
chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates (#9303)
* chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates Move the workspace from the pinned nightly-2026-03-21 to stable 1.96.1 and drop all 23 '#![feature]' gates across 13 crates, rewriting the still-unstable API usages with stable equivalents: - try_blocks: closures / an async block (table, query, common-function, servers) - duration_constructors: Duration::from_secs(n * 86400) / (n * 60) - iterator_try_collect: collect::<Result<Vec<_>, _>>() - box_patterns: as_deref() + matches! chains (sql) - error_iter: error_chain_root() source-chain walker (common-error); sources() includes the error itself, so the walker never panics - int_roundings: div_floor -> div_euclid (equal for positive divisors) - iter_partition_in_place: stable sort_by_key partition helper (index) - hash_set_entry: HashSet::insert bool / contains+insert - trait_alias: lifetime-parameterized dyn FnOnce type aliases (puffin) - string_from_utf8_lossy_owned: from_utf8_lossy(&v).into_owned() - never_type: Infallible (common-recordbatch) - debug_closure_helpers: closure-backed DebugFmt newtype (mito2) - binary_heap_pop_if: peek().is_some_and() + pop() - exclusive_wrapper: drop Exclusive; C: Send + Unpin already in bounds - stmt_expr_attributes: stale gate, no usages Also fix release-dev-builder-images.yaml, which parsed rust-toolchain.toml with a date-only regex and would produce empty image versions with a stable channel; it now extracts the full channel token. Dev-builder images verified against stable 1.96.1 (image build, default-toolchain behavior, binstall/nextest, riscv64 and android targets, in-image cargo check). Validated on 1.96.1: cargo check --workspace --all-targets, clippy --workspace --all-targets --all-features -D warnings, cargo fmt --check, and nextest on all 13 affected crates (4586 passed). Part of #9289. Depends on #9298 (fuzz nightly quarantine) merging first. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: update flake checksum * chore: use wild for linker in flake --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
discord9
|
abedeb2bea |
fix(query): keep count_values generated label in enclosing expressions (#9223)
* fix(query): keep count_values generated label in enclosing expressions
`count_values("v", m)` projects the generated label as a real output
column, but did not register it in `ctx.tag_columns`. Enclosing
expressions (abs/round/+1/topk/label_replace/vector join) rebuild their
projection from `ctx.tag_columns` and silently drop the label.
Register the generated label in `ctx.tag_columns` after the projection,
and give it the same qualifier as other tag columns so qualified
references resolve. PromQL overwrites an input label with the same name,
so drop any existing tag with that name first to avoid duplicate column
ambiguity.
Fixes https://github.com/GreptimeTeam/greptimedb/issues/9181
Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-1
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): use total_cmp in count_values test helper
Silence clippy::needless_borrow on partial_cmp(&right.1); f64 sorting
uses total_cmp, matching the other planner test helpers.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): keep count_values generated label as sink table primary key
FindGroupByFinalName::f_up only renamed a group key when the projection
aliased the key column directly. count_values now projects its generated
label as a unary UDF over the sampled column
(prom_float_to_string(value) AS label) while the aggregate still groups
by the raw column, so the name match failed and the sink table lost the
label from its PRIMARY KEY, demoting it to a DOUBLE value column.
Allow a projection above the aggregate to rename a group key by deriving
its output from a single group-key column (unary scalar function or cast
over that column only). Multi-column expressions, case, aggregates,
windows, subqueries and literals are still rejected so a computed column
cannot be mistaken for the group key.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): group count_values by the formatted sample value
The numeric count_values branch grouped by the raw sample column and only
formatted the value into the generated label in the post-aggregate
projection. Two distinct raw values that collapse to one label text
(e.g. BIGINT 9007199254740992 and 9007199254740993, both 9007199254740992
in Float64) were split into two groups, each emitting the same label set
at one timestamp, violating Prometheus' unique-label-set-per-timestamp
invariant.
Return the formatted value expression (prom_float_to_string, with a
CAST to Float64 for non-Float64 inputs) as previous_field_expressions so
the aggregate groups by the same expression that produces the label,
mirroring the existing mixed float/native-histogram precedent.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): drop needless borrow in count_values test helper
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): don't replace group key with derived unary expr
FindGroupByFinalName treated any unary expression of a single group
column (scalar fn, Cast, TryCast) as a rename of that group key and
swapped it in as the sink primary key. A derived expression such as
lower(host) is many-to-one, so distinct groups (HOST_A vs host_a) would
collapse to the same primary key and be silently deduplicated.
Narrow the matching to direct name-matched aliases of the actual group
expression only, and drop the derived-unary path (is_unary_expr_of_column)
and the allow_derived flag. count_values already groups by the formatted
expression name, so it is unaffected.
Add a regression asserting host survives and host_lc does not replace it
under both optimizer settings.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
|
||
|
dennis zhuang
|
d59725b04a |
test(sqlness): run environments concurrently and drop redundant restarts and sleeps (#9333)
* test(sqlness): run environments concurrently with external backends The runner forced both environment and instance parallelism to 1 whenever etcd/PG/MySQL or an external Kafka was set up, so the standalone and distributed environments ran one after the other. Only distributed uses the kv backend, and the two environments use different Kafka topic prefixes, so they can run at the same time. Keep one instance per environment, since instances would share the metadata table and Kafka topics. Runs with a runner-managed Kafka, an external server, a test filter, or `-j 1` stay fully serial. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): drop redundant restarts and sleeps - flow_rebuild: drop the R2 block, which repeats R1 from the same state, and the restart in R5, whose result R1 already asserts. - flow_advance_ttl, flow_view, ttl_instant: drop sleeps that no assertion depends on. - flow_basic, flow_null, flow_call_df_func: drop the bytes_log section (covered by flow_insert), a duplicate state_size query, an unchecked insert, and flushes that run with no new data. - alter_table_options, skip_wal: share restarts between independent tables. - session_skip_wal, copy_skip_wal: check every case after a single restart. Tables copied or inserted with skip_wal = true first get a WAL row that is truncated, so the check that truncated WAL entries are not replayed stays. - region_statistics: wait once for the statistics of all three tables. - build_index_table: fold its index_size checks into build_index_table_restart, which runs the same fixture and waits. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): restore session_skip_wal and copy_skip_wal Checking every case after a single restart dropped two state transitions the original cases cover: recovering a region whose memtable only held skipped rows, and writing to the recovered region before restarting again. Restore the original cases. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): restore the one-restart check in flow_auto_sink_table #5112 checked the auto-created sink before a restart and the flow and sink after it. #5987 moved the restart in front of the first SHOW and added a second one. Flow recovery only creates the sink when it is missing, so the second restart recovers from the same persisted state as the first. Restore the original before/after check with one restart. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): keep one instance per environment with external store addresses Instances of the distributed environment share the etcd behind --store-addrs, so run one instance per environment when it is set, same as --setup-etcd. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): restore the false-branch batch in flow_basic and assert it The kept batch (20, 22) is all above the threshold. The second batch (10, 23) is the only input that hits the false branch of the CASE and makes the flow compute again. Restore it and check the result, which the original case never did. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
affc0a1b1d |
fix: align ordered aggregate state type with the accumulator output (#9340)
* fix: align ordered aggregate state type with the accumulator output Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject mismatched aggregate states and keep hard-ordered aggregates unsplit Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: keep WITHIN GROUP aggregates splittable and relabel state fields by position Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
def0c2ec5c |
fix: only push down aggregates grouped by the partition columns themselves (#9337)
* fix: only push down aggregates grouped by the partition columns themselves Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: keep grouping sets on the frontend for partitioned tables Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
b9f991502c |
refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
cc1be82858 |
fix: merge every state row in geo_path and json_encode_path (#9339)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
c7fa48ef95 |
feat(mito): limit approximate series index disk usage (#9313)
* feat(mito): limit series index disk usage Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): enforce series index quota during reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): remove series index disk budget layer Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): simplify series index limit to estimated usage Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): trust index catalogs when loading snapshots Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): minimize series index disk limit changes Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): keep series index cleanup running at capacity Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): avoid no-op index clones and stabilize capacity tests Signed-off-by: evenyag <realevenyag@gmail.com> * fix: update config API expectation and stabilize index build tests Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
1955955b2d |
fix(ci): grant PR write permission for CI command replies (#9331)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
shuiyisong
|
20dde2601f |
feat: enable native histogram ingestion by default (#9301)
* feat: enable native histogram ingestion by default Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Weny Xu
|
4c98fa5265 |
ci: add optional AWS runners for observability benchmarks (#9322)
* ci: add optional AWS runners for observability benchmarks Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: restore automatic benchmark disk sizing Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
shuiyisong
|
2bc18c483d |
feat: add schema metadata stream to SchemaManager (#9325)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
dennis zhuang
|
88197f4019 |
fix(promql): derive vector matching result labels and reject ambiguous matchings (#9306)
* fix(promql): derive vector matching result labels and reject ambiguous matchings A vector-vector binary operation projected one operand's whole tag set and inner-joined without any cardinality check, so `on()`/`ignoring()` did not reduce the result labels, `group_left`/`group_right` changed nothing, and a non-unique match group produced a cross product that PromQL cannot represent. Result labels now follow Prometheus `resultMetric`: `on(...)` keeps the matching labels, `ignoring(...)` drops them, and a group modifier keeps the many side's labels plus the `group_x(...)` labels taken from the one side. A label the one side does not carry is deleted from the result. The reduced label set no longer identifies the operand series, so `__tsid` is dropped from the context on this path. Cardinality is enforced with a `count(1) OVER (PARTITION BY match keys, ts)` window and a scalar UDF that fails the query on a repeated group: on the one side before the join, and on the result labels after it, matching where Prometheus raises each of its three errors. Series are unique by their whole tag set, so the window is only planted when the match keys drop a tag; plain arithmetic and `on(<all tags>)` plan exactly as before. Closes #9207, closes #9208, closes #9209. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): keep __tsid when the result labels are an operand's whole tag set Deriving the result labels dropped `__tsid` from the context unconditionally, so an enclosing operation fell back to joining on the tag columns even where the column still identified the result series. Keep it when every result label comes from one operand and covers that operand's whole tag set: no other operand value reaches the labels, and the matching gives each of its rows a single partner, so its `__tsid` is still one per result series. That is the common `on(<all tags>)` and bare `group_left` shape; a matching that actually drops a tag still clears it. The column is re-qualified as the result's own, which is how the enclosing expression and the context look it up. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep the match group count column unambiguous The cardinality check aliased its row count to a fixed `__promql_match_group_count`. An operand carrying a label of that name made the window output two fields with the same name, and planning failed with "Schema contains qualified field name collide_right.__promql_match_group_count and unqualified field name __promql_match_group_count which would be ambiguous". Pick a name the operand does not already have, the way the `or` operator allocates its match key columns. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover a match group spread over several regions The cardinality check runs above the merge of the region scans, so it counts a match group globally. Nothing pinned that: every table in these cases holds a single region, and a check evaluated per region would pass them all. Partition the operand on a column outside the match keys, which puts the two series of one match group in different regions, and assert both the pre-join and the post-join check still reject it. The case runs in the distributed environment too, where the regions sit on different datanodes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): group an outer aggregate on the labels the operand kept `by`/`without` planning topped up a missing grouping column by walking down to the table scan and re-projecting it. That is right for a column the scan pruned for efficiency, but the labels a matching modifier deletes are also absent from the operand's output, and they were restored the same way: sum without(host) (a / on(host) b) `on(host)` leaves the operand with `host` alone, so the sum covers everything and Prometheus answers `{} 10`. Instead `device` came back from the scan under `a` and split the result into `{device="d1"} 5` and `{device="d2"} 5`. Same for `sum by(device)` of that operand, which has no `device` to group on at all. Take the grouping labels from the operand's own label set rather than from the row keys of the scan beneath it. A label pruned from the plan is still in that set and still gets restored; a label the operand dropped is not. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): drop the unreachable aggregation tag top-up `by`/`without` planning could restore a grouping column that the plan no longer carried by rewriting the scan underneath it. Once the grouping labels come from the operand's own label set, there is nothing left for it to restore: a scan projects every label of `ctx.tag_columns` (`scan_tag_columns` only ever adds matcher columns to that set), so a label in the set is always in the schema. Stubbing the rewriter to a no-op passed the whole sqlness suite, in both the standalone and the distributed environment. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): drop cases that guarded the removed tag top-up Three plain selector aggregates were there to show that restoring a pruned grouping column still worked. With the restore gone they only repeat what the aggregate cases already cover. Also fix a comment that still said the metric engine scan prunes tag columns: it projects every label of the operand. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(promql): note the duplicate a propagated matcher hides A matcher copied onto the one-side operand removes groups without a partner before the cardinality check sees them, so a duplicate in such a group is not reported. Prometheus checks every group of the one side and fails the query. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Weny Xu
|
37fed9f12b |
fix(ci): repair draft PR command dispatch (#9271)
* fix(ci): repair draft PR command dispatch Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): dispatch command workflows by branch ref Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): admit PR authors and writers for CI commands Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): preserve dispatch guard dependency semantics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): configure slash command permissions individually Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(ci): run command workflows at dispatch branch head Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): rerun fork PR checks and report command failures Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
d910183cc4 |
feat(flow): freeze recovery windows and retention bounds for incremental flows (#9312)
* feat(flow): discover recovery windows from exact sequence reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(flow): freeze explicit recovery windows and retention bounds Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): validate retained source windows before recovery Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): qualify recovery source tables before serialization Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 fys
|
5a78e07be6 |
fix(json2): restrict JSON2 type hints (#9316)
* fix(sql): restrict JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> * fix(sql): allow equivalent 64-bit aliases in JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> * fix(sql): support UInt64 conversion and preserve JSON2 hint aliases Signed-off-by: fys <fengys1996@gmail.com> * refactor(sql): remove redundant JSON2 type hint normalization Signed-off-by: fys <fengys1996@gmail.com> * fix(datatypes): restrict JSON2 hint types in JsonSettings::try_new Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
Yingwen
|
bc80071409 |
feat: expose region open failure metrics (#9283)
Signed-off-by: evenyag <realevenyag@gmail.com> |