mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-09-07 22:18:57 +00:00
bb9b7e8778b8d561b76e6cd020b14803ecb4c9ae
6031
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dennis zhuang
|
bb9b7e8778 |
fix: re-scan stream-backed tables in recursive CTEs (#9039)
* fix: re-scan stream-backed tables in recursive CTEs A recursive CTE re-executes its recursive term on every iteration, but DfTableProviderAdapter hands StreamScanAdapter a single-use stream built at planning time. The second iteration failed with "Stream already exhausted" for every table served through DataSource::get_stream — information_schema, pg_catalog, the computed entity-graph tables and numbers. Keep that stream for the first execution and open a new one over the same scan request for later executions. Closes #9037 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: drop redundant binding in stream factory Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Lei, HUANG
|
765ed7865f |
feat(client): compress insert transport (#9036)
* feat(client): compress bulk insert transport Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(client): compress row insert transport Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: address comments Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 localhost
|
4d65e8984a |
chore(ci): Implement /query-regression command handling and admission workflow (#8975)
* Implement `/query-regression` command handling and admission workflow - Add `query-regression-slash.py` script for processing `/query-regression` commands in PR comments, validating case arguments, and checking permissions. - Update `checks.yml` to include tests for the new slash command functionality. - Modify `query-regression-comment.yml` to trigger on the new `Query Regression Command` workflow. - Create `query-regression-slash.yml` to handle the dispatched command, validate allowlist and permissions, and initiate the regression workflow. - Enhance `query-regression.yml` to support additional inputs for PR admission and SHA verification. - Introduce `slash-command-dispatch.yml` to parse and dispatch commands from PR comments. - Document the new command admission process in `AGENTS.md` and `README.md`. - Add unit tests in `test_query_regression_slash.py` to cover command parsing and admission logic. * refactor: enhance query-regression command handling with comment validation and identity checks * feat: implement admission identity handling for query regression workflows * refactor: update PR admission logic in query regression workflow * refactor: update token usage in slash command dispatch and README for clarity * test: add cases for handling re-run failed jobs and stale runner artifacts * refactor: improve repository metadata handling in query regression scripts * chore: enable overwrite for artifact uploads to handle re-run failed jobs * chore: enable overwrite for query regression admission uploads * feat: enhance query-regression admission with HMAC signing and verification - Introduced HMAC signing for admission markers in query-regression workflows to ensure integrity and authenticity. - Updated `query-regression-comment.test.cjs` to include tests for signing and verifying admission markers. - Modified `query-regression-slash.py` to handle admission marker signing and verification, including checks for dispatch sender and head SHA consistency. - Enhanced workflows to securely manage admission markers and HMAC secrets, ensuring they are not exposed to untrusted contexts. - Improved documentation to clarify the admission process and the role of HMAC in securing the workflow. * test: add case to find newly posted marker among newer comments * test: add case to verify multiline output handling in write_outputs function |
||
 Weny Xu
|
f9df4def74 |
chore(deps): switch rskafka to upstream main (#9047)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
c3ea022de5 |
perf(mito2): blazing-fast tournament tree merger (#8989)
* perf(mito2): optimize flat merge heap and primary-key interleave Replace the per-row BinaryHeap pop/push cycle in FlatMerge with an in-place root mutation plus a single sift-down repair on a custom RootHeap, keeping the cold heap and direct-batch fast path unchanged. Fallible or awaiting batch transitions move the hot node out of the heap first, preserving error and cancellation semantics. Exploit the globally sorted merge output to build the internal Dictionary<UInt32, Binary> primary-key column with a one-pass ordered gather: append a Binary value only when the PK changes and reuse the current key for adjacent equal PKs, bypassing Arrow dictionary masks, hash interning and key remapping. Non-PK columns still use Arrow interleave. Also cache the current primary-key byte range in RowCursor to avoid repeated dictionary range decoding during comparisons, and add a setup-free Criterion benchmark with exact output-row assertions. 32-way/1 row-per-series/40-tag improves 955.79ms -> 562.36ms (-41.2%); 0-tag -39.5%, 64 rows/series -79.8%, 8-way -56.1%, single-iterator control +0.3%. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): add rows-per-series sweep to flat merge bench Add 32-way/40-tag shapes for 1, 10, 100, 1000 and 10000 rows per series, and allow FLAT_MERGE_BENCH_SHAPE to match shape name prefixes so the whole sweep can run in one invocation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): add oracle-based correctness tests for RootHeap Drive RootHeap and a std BinaryHeap oracle with the same seeded op sequence (push / pop / mutate-root + repair) and assert peek, len, best_child and the full drain order after every operation. A second run with a tiny value range makes duplicates dominate, covering the equal-key branches of sift_up/sift_down and best_child. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): replace hot heap with a tournament tree in flat merge Replace the hot RootHeap with a fixed-capacity tournament (winner) tree over per-node slots: every internal node caches the champion of its subtree, so advancing the winner only replays the ~log2(k) nodes on its leaf-to-root path with one compare per level, instead of the heap's two-compares-per-level sift that also re-compares the same node pairs on every row. Two fast paths keep dense shapes at O(1) per row: - champion retention: after mutating the winner in place, skip the replay entirely when it still beats the runner-up (its path caches are unchanged by construction); - a second-best slot cache, invalidated on any structural change, so the retention check costs a single compare without walking the tree. The cold heap, hot/cold overlap window, direct-batch fast path and the remove-before-fallible-fetch batch transition semantics are unchanged. Vs the RootHeap version: 1rps/32way/40tag -19.7%, 0tag -34.4%, 8way -15.9%, 64rps -30.5%, sweep 10/100/1000/10000rps -29~32%; vs the original BinaryHeap baseline the main shape is -52.8%. The single-iterator control is +8% (+50ns one-time construction allocation, no merge work). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): support generic schemas in flat merge Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): satisfy clippy in flat merge benchmark Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): cache flat merge primary key index Compute the internal primary-key column index once when constructing BatchBuilder and reuse it for every output batch. Preserve the column-name gate for generic schemas. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): benchmark high-fan-in flat merges Add sparse 64, 128, 256, and 512-way merge shapes while keeping the total input fixed at 3.2 million rows. Compared with the merge-base heap implementation, median time improves by 56.0%, 60.8%, 55.6%, and 56.1%, respectively. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 discord9
|
54fcf36452 |
fix(frontend): isolate internal Flight authentication (#9045)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
fa794fae7a |
fix: match system schema names case-insensitively (#9040)
* fix: match system schema names case-insensitively Database names that arrive over a protocol (the MySQL handshake and COM_INIT_DB, the Postgres startup parameter, the HTTP `db` parameter, the gRPC dbname header) never reach the SQL parser, which is what lowercases unquoted identifiers. Since #8062 stopped lowercasing them wholesale, connecting to `INFORMATION_SCHEMA` in any spelling but the canonical one fails with "Unknown database" -- including the `USE <db>` that a MySQL client turns into COM_INIT_DB. Fold only system schema names to their canonical spelling, so user schema names keep the case they were created with. `is_reserved_schema_name` uses the same match, otherwise a quoted `CREATE DATABASE "INFORMATION_SCHEMA"` creates a schema shadowed by the system one. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: hoist system schema names into a const Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>v1.3.0-alpha.1-nightly-20260907 |
||
 liyang
|
d9ebe5852c |
ci: skip bumping helm charts and homebrew and downstream repository for pre-releases (#9031)
* ci: skip bumping helm charts and homebrew for pre-releases Signed-off-by: liyang <daviderli614@gmail.com> * add skip downstream-repo Signed-off-by: liyang <daviderli614@gmail.com> --------- Signed-off-by: liyang <daviderli614@gmail.com> |
||
|
Weny Xu
|
a932433d21 |
fix(wal): bound Kafka requests and extend latency buckets (#9026)
* fix(wal): bound Kafka requests and extend latency buckets Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(wal): update rskafka request timeout revision Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(config): document Kafka WAL timeouts in MetaSrv Signed-off-by: WenyXu <wenymedia@gmail.com> * style: sort common-wal dev dependencies Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 LFC
|
d67d3501a9 |
fix(json2): keep empty structs in remainder (#9027)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
 Ning Sun
|
27b0ca6676 |
ci: add workflow to auto-create backport PRs from backport labels (#9028)
* ci: add backport workflow to create backport PRs from backport labels Signed-off-by: Ning Sun <sunning@greptime.com> * ci: document backport labels in PR template and AGENTS.md Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 Yingwen
|
cf9a9639b0 |
perf(mito2): postpone covered time index filters (#8998)
* perf(mito2): postpone covered time index filters Signed-off-by: evenyag <realevenyag@gmail.com> * perf(mito2): reuse implied time range for prefilter Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): build finalized scan inputs Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): reject empty implied time filters Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): guard last row shortcut with remaining filters Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
ed1f2d9f4e |
fix(pipeline): coalesce concurrent pipeline cache misses (#9022)
* fix(pipeline): coalesce concurrent pipeline cache misses The pipeline cache reads with a plain `moka::sync::Cache::get` and falls through to a distributed query on a miss, so when the 10s TTL expires every in-flight write request on a frontend issues its own scan of the single-region `greptime_private.pipelines` table. Concurrent scans per expiry scale with write QPS, and every frontend's burst lands on the same datanode. A user running high-throughput ingestion through a pipeline saw that datanode overloaded. Switch to `moka::future::Cache::try_get_with` so concurrent misses on the same key share one loader. This requires a single-key lookup, so cache entries are now keyed by the requested schema rather than the schema the pipeline is stored under; resolving a request to a stored schema stays in the loader, which is the authoritative path and already handles the empty-schema and multi-schema cases. A lookup for a schema not yet cached costs one extra read, now protected from amplification by the coalescing it enables. `remove_cache` previously only walked the compiled-pipeline cache, so an entry populated by `get_pipeline_str` alone (the pipeline read API) survived deletion until it expired. It now walks all three caches. Also make the TTL configurable as `pipeline.cache_ttl`, default unchanged at 10s. The TTL is what propagates a pipeline change to other frontends, so raising it trades staleness for fewer reads. Refs #9021 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(pipeline): restore cross-schema semantics broken by the new cache key Keying cache entries by the requested schema dropped two behaviours that the previous stored-schema key provided for free. Creating a new version only wrote the creating request's schema, so another schema on the same frontend kept serving its cached `latest` — an older version — until the entry expired. Since the whole point of making the TTL configurable is to let operators raise it, that window is not bounded by anything useful. Creation now invalidates every schema's `latest` alias for that name before priming the cache, leaving the version-pinned keys alone. The failover cache lost its reach across schemas the same way: a global pipeline (stored under the empty schema) loaded by schema A was cached under `A`, so schema B using it for the first time while the pipeline table was down missed and failed ingestion. The failover cache has no loader and so is not subject to the single-key model of `try_get_with`; it keeps the stored-schema key and the empty-schema-first resolution. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(pipeline): drop cache priming on create and fold the sweep helpers Priming the cache on create saved one read on a low-frequency operation and cost a concept: entries were written under the creating request's schema while `PipelineContent.schema` said empty, so the two schemas in play disagreed. Invalidating the `latest` aliases is required regardless — that is what makes a new version visible to other schemas — so dropping the priming loses only the saved read, which coalescing now protects anyway. `insert_and_compile` no longer needs the caller's schema. `remove_cache` and the create-time invalidation collapse into one `invalidate(name, version)`; `None` sweeps only the `latest` aliases, which is exactly what creation wants. That leaves `invalidate_by_suffixes` and `cache_keys` with a single caller each, so both are inlined. Drop the `PipelineOptions` humantime test: `load_config_test` loads both example TOMLs, which now carry `cache_ttl = "10s"`, and would fail the same way if the serde attribute were lost. The `toml` dev-dependency goes with it. The two invalidation tests are now checked to be orthogonal: removing the version suffix fails only the delete test, and sweeping just the compiled cache fails both. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(pipeline): keep failover populated across a create The `latest` sweep on create clears the failover cache along with the loaded ones, and after dropping the priming there was nothing writing it back. An outage between the create and the first read-back left neither `latest` nor the explicit version with anything to fall back on, failing ingestion — worse than before, since the previous version's failover entry was swept too. Creation now goes through `PipelineCache::on_pipeline_created`, which pairs the sweep with a failover write of the new empty-schema definition. The two must happen together, so they live behind one method rather than at the call site. Also commit the Cargo.lock entry for the dropped `toml` dev-dependency, and trim the comments added over the last few commits down to what the code does not already say. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
ad7b0ace64 |
feat(flow): support eval schedule offsets (#8878)
* feat(flow): support eval schedule offsets Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): remove redundant schedule assertion Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(flow): trim eval offset compatibility scope Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): trim eval offset edge coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(flow): trim eval offset comment noise Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): address eval offset review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): cover Flow eval offset persistence Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
LFC
|
84bd993131 |
refactor(json2): concretize JSON2 schemas at merge scan boundaries (#9016)
* refactor(json2): concretize JSON2 schemas at merge scan boundaries Infer concrete JSON2 output types from remote plans and expose them on MergeScanLogicalPlan before physical planning. Recompute affected local schemas and remove the JSON2-specific rewrite from MergeScanExec. Add SQLness coverage for whole JSON2 columns in windows and joins. Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
 shuiyisong
|
b86da3d35f |
feat: support raw OTLP delta metrics (#8970)
* feat: support raw OTLP delta metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(promql): update sqlness results for normalized label matching Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: derive temporality label from default column prefix Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(promql): add analyze coverage for delta temporality Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): scope label alignment to temporality marker Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: handle count-only histograms and vector broadcasts Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: exclude temporality marker from entity descriptions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: use a fixed label for OTLP aggregation temporality Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): preserve mixed-range semantics for raw delta Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
945e53e0a3 |
fix(client): isolate query and control transports (#8990)
* refactor(client): isolate query and control transports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(client): cover retained Flight transport isolation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(client): satisfy retained Flight test lint Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(client): clarify transport lane routing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
d664326b1e |
chore: bump version to 1.3.0-alpha.1 (#9014)
* chore: bump version to 1.3.0-alpha.1 Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update Cargo lockfile Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>v1.3.0-alpha.1 |
||
|
LFC
|
d62a5a990a |
feat(json2): support list indexing for JSON2 columns (#9013)
feat(query): support list indexing for JSON2 columns Extend JSON2 paths through DataFusion field-access planning, including nested list indexes and object fields following an index. Preserve Variant reads for bracket JSONPath expressions and normalize dot accesses after subscripts to work around the current DataFusion planner limitation. Add unit and sqlness coverage for nested indexes, type conflicts, missing paths, flushes, and compacted SSTs. Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Yingwen
|
d7571c1278 |
feat(mito2): add SST range index searcher (#9003)
* feat(mito2): add SST range index searcher Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): reuse parquet index reader Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify range index pruning Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): cover missing range index series Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
d351b7d471 |
feat(mito2): pass operation type to write cache upload hook (#9012)
Extend `WriteCacheUploadStoreWrapper::wrap` with the `OperationType` of the upload so implementations can apply per-operation policies (e.g. throttling compaction uploads but not flush uploads). Flush and compaction paths forward their existing `SstWriteRequest::op_type`; `put_and_upload_sst` is flush-only and index rebuild uploads are reported as compaction uploads. Files: `src/mito2/src/cache/write_cache.rs`, `src/mito2/src/sst/index.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 fys
|
05c65f54a8 |
feat(json2): support empty and null JSON2 value (#9010)
* feat(json2): support empty and null JSON2 value Signed-off-by: fys <fengys1996@gmail.com> * test(json2): cover explicit NULL and omitted-column inserts Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo fmt Signed-off-by: fys <fengys1996@gmail.com> * fix: infer empty JSON object as object type Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
discord9
|
27a7047f31 |
feat: preserve row sequences and support exact sequence-range reads (#8865)
* feat(mito2): support exact sequence range reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): cover preserve row sequence table alter Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): clear preserve_row_sequence marker on copy_region_from copy_region_from copies source FileMeta into the target region, which has an independent sequence domain. The physical per-row sequences in the copied file belong to the source region only; trusting them in the target would let an exact sequence-range request replay source-domain rows as if they were target sequences. Clear the preserve_row_sequence marker on copied files so the target fails closed with SequenceRangeUnsupported until the scan provably cannot intersect the copied rows. Add a regression test: copying from a preserve-enabled source into a preserve-enabled target clears the marker, and an exact (2, 7] request on the target returns SequenceRangeUnsupported instead of replaying source rows. Fixes #8865 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: remove redundant doc comments for exact sequence range options Approved comment-cleanup-only changes for #8865: drop outdated doc summaries duplicated on the exact_sequence_range wrapper and the preserve_row_sequence field, drop pure-restatement doc comments on the SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and remove the four structural SQL comments from the alter_preserve_row_sequence case. No behavior changes; .result regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): fail closed exact reads on copied files and extension ranges Address review feedback on #8865: - copy_region_from: clear the source-domain FileMeta::sequence along with the preserve_row_sequence marker. An unmarked file retaining a stale source-domain max sequence could be silently skipped by files_allow_exact_sequence_range() as 'proven disjoint' in the target's independent sequence domain, dropping rows on exact (C, H] reads. With sequence=None the capability check fails closed (SequenceRangeUnsupported) until the copied rows are provably disjoint. - Engine/reader: reject exact sequence-range reads whenever a follower region has an extension range provider attached. Extension streams are returned without a row-level sequence filter, so exactness cannot be proven; treat the capability as missing (fail closed) instead of emitting out-of-range rows. The reader also fails closed as defense in depth. Tests: extend copy_region_from regression to assert the copied file's sequence hint is cleared; mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): use doc comments for test function descriptions Elevate the block comments describing test functions (in scan_test and copy_region_from_test) to /// doc comments, matching the convention used elsewhere in the exact sequence range change. No logic change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): extract helpers and trim comment noise in exact sequence reads PR finalization for #8865 (zero behavior change, full mito2 suite green): - engine: extract validate_sequence_fences and sequence_range_unsupported_reason, keeping error variants, check order and reason strings identical; OSS binds the extension blocker to false. - handle_copy_region: extract remap_copied_file_meta and file_descriptors_for_meta; rename file_ids -> source_file_ids and files_to_copy -> new_file_metas. - compactor: rename max_input_sequence -> known_max_input_sequence, document the None semantics (empty input vs unknown sequence). - Remove restating/outdated comments (ScanInput::sequence_range doc first line, outdated file-pruning note, options test restatements), compress verbatim comments while keeping why/invariants/contracts. Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt, git diff --check, mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): reject foreign-region SSTs in exact sequence reads Reading an SST whose FileMeta.region_id differs from the scanned region means the region's sequence domain is broken (manifest corruption or a repartition/copy path that leaked a source-domain file). Treat this as an explicit RegionSequenceDomainBroken error instead of silently ignoring the file's sequence or falling back to a full scan: the region is unusable for exact sequence-range reads until the foreign lineage is compacted away or repaired. - files_allow_exact_sequence_range / exact_sequence_range now return Result and propagate the error through engine fence validation and scan construction (StatusCode::Internal, distinct from the fallback-capable SequenceRangeUnsupported). - Row-level flat-batch sequence filtering rejects foreign-region files as defense in depth. - Engine test asserts the broken-domain error rather than Unsupported/fallback. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping An unmarked file's FileMeta.sequence may be synthesized by the region-edit or repartition paths (committed+1 import barrier), not a physical max of its rows. Treating it as a whole-file disjoint proof could permanently skip rows that were never incrementally consumed once the flow checkpoint passes that value. Exact sequence-range capability now requires every SST in the region to carry the preserve_row_sequence marker; any unmarked file disables exactness (fallback), and the (C, H] file-selection skip also only applies to marked files. Foreign-region files still raise RegionSequenceDomainBroken as before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): scope exact sequence-range capability to the time-selected read set The exact capability check used to walk the entire SstVersion, so a single unmarked or foreign-region SST anywhere in the region disabled exact reads or raised RegionSequenceDomainBroken even when the request's time range could never touch that file. Both the engine fence and the scan builder now derive the read set with shared time-pruning + exact-min/sst-min selection and validate capability only over the files actually selected: a time-pruned file cannot contribute a row to (C, H], so it cannot affect exactness. The existing fail-loud semantics are unchanged for every selected file (foreign region id -> RegionSequenceDomainBroken; unmarked -> exact unavailable). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): wash untrusted sequences in compaction and restore barrier skipping Compaction with any non-preserved input now writes a sequence-less output: the physical __sequence column is zeroed (the flat format requires the internal columns) and FileMeta.sequence records the region-local admission barrier committed_sequence + 1 (falling back to the flushed frontier). preserve_row_sequence stays false. Exact sequence-range scans interpret an unmarked file's sequence as an admission barrier: barrier <= C means flow has already consumed the whole file, so it is skipped at file level; a missing or newer barrier fails closed. Foreign-region files stay in the selected read set so the capability fence still raises RegionSequenceDomainBroken. This closes the recovery loop: after a region repartition, one time-scoped fallback consumes the migrated rows, then compaction washes the untrusted per-row sequences away and exact incremental reads resume via file-level barrier skipping. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): drop restating comments in known_max_input_sequence tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): trim SQLness result EOF whitespace Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): reuse exact scan file selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): strengthen sequence scan coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): trim ALTER option comments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): remove no-op bulk compaction check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve trusted row sequences when reading SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve target sequence domain for imported SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): add trailing blank line to SQLness result EOF Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): trim exact sequence scan plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): fold exact SST selection checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make legacy compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make PK compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
25d49ba092 |
ci: trigger downstream updates for prereleases (#9008)
* ci: trigger downstream updates for prereleases Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover docs prerelease dispatch Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
LFC
|
15317a131b |
feat(json2): support JSON2 paths in SQL functions (#9007)
feat(query): support JSON2 paths in SQL functions Update the DataFusion fork to expose scalar function planning hooks. Infer JSON2 path output types from scalar, aggregate, and window function signatures, while preserving the default Utf8View behavior for functions that accept arbitrary inputs. Add unit and sqlness coverage for type conflicts, mixed typed and untyped JSON paths, filters, aggregates, and window functions. Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
fys
|
534ab31297 |
feat(flow): add row inserts to frontend client (#9006)
* feat(flow): add row inserts to frontend client Signed-off-by: fys <fengys1996@gmail.com> * feat(flow): support hints for frontend row inserts Signed-off-by: fys <fengys1996@gmail.com> * fix: handle poisoned frontend handler lock in row inserts Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
discord9
|
9c135ebcb3 |
feat!: stabilize streaming analyze metrics (#8966)
* feat: stabilize streaming analyze metrics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: expose analyze memory usage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: simplify analyze stream handling Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve analyze stream sequence on panic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: log analyze stream worker panic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
localhost
|
0de0c01283 | fix: add disk usage logging to GitHub step summary in query regression workflow (#9005) | ||
 discord9andRuihang Xia
|
43c30d1446 |
feat(runtime): add weighted workload scheduler (#8736)
* feat(runtime): add weighted workload scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics Use the GreptimeTeam/catio fork (pinned c20eafc) which adds ClassStats::total_admission_wait and ClassStats::admitted, recorded at each QUEUED -> ADMITTED transition. This exposes the scheduler's own admission delay (excluding Tokio queueing and poll execution), enabling admission-wait based fairness gates. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to dynamic-config revision Bump the catio scheduler fork to 9f4b028 which adds Scheduler::set_weight and Scheduler::set_max_concurrent_polls for runtime configuration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(perf): runtime-adjustable workload scheduler parameters Expose dynamic adjustment of the experimental workload scheduler at runtime: - common-runtime: set_workload_scheduler_weights and set_workload_scheduler_max_concurrent_polls, which forward to the catio scheduler's set_weight/set_max_concurrent_polls when the scheduler is enabled and reject zero values. - servers: /debug/workload_scheduler/weights and /debug/workload_scheduler/max_concurrent_polls POST handlers, so operators can rebalance query/write shares or admission concurrency without restarting the datanode. Both endpoints return 400 with a clear reason when the scheduler is disabled or the requested value is invalid. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(perf): add GET /debug/workload_scheduler status endpoint Returns the current weights (per class), max_concurrent_polls, active_polls and per-class counters (queued, tasks, wakes, polls, completed, cancelled, admitted, total_admission_wait) as JSON. When the scheduler is disabled, returns enabled=false with the other fields omitted, so operators can distinguish 'disabled' from an error. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to time-accounting revision Bump the catio scheduler fork to 257ba56 which replaces admission-count accounting with real execution-time accounting (pass += exec_time / (weight * concurrency)), so CPU share follows the configured weights regardless of poll length. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to lock-free sampling revision Bump the catio scheduler fork to efdc0a4 which adds an optional downsampled clock sampling mode (SchedulerBuilder::sample_every_polls, default off) with a lock-free per-class atomic counter, so the downsampled path costs one fetch_add per poll instead of a global mutex. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin catio to scheduler PR head Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): add scheduler bypass control Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: advance catio scheduler fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin merged catio scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: regenerate config docs for workload scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin catio scheduler test fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(http): satisfy scheduler lifecycle clippy Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: add distributed scheduler toggle coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: finalize workload scheduler runtime controls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin merged catio atomic weights Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: preserve unrelated lockfile resolution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(runtime): downsample scheduler time accounting Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(runtime): verify cross-runtime scheduler progress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): configure scheduler poll sampling Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(runtime): clarify scheduler activation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(runtime): explain scheduler use case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
dennis zhuang
|
02278d301b |
feat: derive k8s.node from OTLP resource attributes (#9002)
The k8s.pod runs_on k8s.node rule can only fire from kube_pod_info: it is the only table declaring both endpoints, and co-declared edges require both on the same row. A deployment sending only OTLP has no kube-state-metrics tables, so its node layer is invisible and that rule has no source at all, even though the resource attributes carry k8s.node.name. Declare k8s.node in otlp_trace_entities and in the synthesized resource descriptor, and project k8s.node.name in the descriptor writer so the column the declaration needs exists. Identity is the name rather than k8s.node.uid: kube-state-metrics carries no node UID, so the name is the only identity both sources can agree on. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Ning Sun
|
6f1dd0cb76 |
feat: allow widening the time index column's timestamp unit via ALTER TABLE, mito2 table only (#8894)
* feat: allow widening the time index column's timestamp unit via ALTER TABLE ... MODIFY COLUMN Previously MODIFY COLUMN rejected the time index column outright. Now the time index unit can be widened (Second -> Milli -> Micro -> Nano), which is lossless for data that fits the target unit: historical data in old SSTs is cast to the new unit on read by the existing schema-compat layer, and compaction rewrites it lazily. Narrowing and non-timestamp targets remain rejected; tag columns keep being rejected. Widening is rejected if any SST's time range would overflow the target unit's i64 range (e.g. millisecond -> nanosecond beyond year 2262), since the cast would silently null those values. Read-path correctness for old-unit SSTs (verified by new engine e2e tests and sqlness WHERE queries): - row-group min/max pruning: parquet statistics of a timestamp column are raw integers in the file's unit; when the region metadata's type differs (also the case for altered field columns), stats are now interpreted in the file's type and converted to the expected type before pruning. Without this, a new-unit predicate silently pruned whole row groups of old-unit files (wrong results, rows missing). - SST-level simple filters are skipped for columns whose file type differs from the expected type; the predicate is applied by the query layer's residual filter above the region scan. Also drop the stale "timestamp columns cannot change type" debug_assert. - retry idempotency: a same-type ModifyColumnType on the time index validates as a no-op and `need_alter` returns false, so a retried alter procedure (region already altered before the previous attempt failed) converges instead of aborting forever. - add TimeUnit ordering and ConcreteDataType::is_timestamp_unit_widening_to - relax ModifyColumnType validation in store-api and table metadata - tests: unit tests in datatypes/store-api/table; mito2 engine e2e tests (flushed SST + new writes + reopen + retry + overflow + predicate scans, cross-unit dedup, mixed-unit SSTs, compaction); sqlness cases incl. partitioned table and WHERE filters over old-unit data Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve parquet filter issue * test: provide sqlness tests * refactor: drop trivial test cases and shorten comments Review pass over the branch's additions: - datatypes: keep a representative subset of the widening-matrix asserts - table: collapse the three single-branch rejection blocks into one loop - mito2: drop the boundary gt_eq and post-compaction predicate asserts (covered by the exact-filter regression test and sqlness); drop the engine-level gt_eq/lt_eq casts (full operator matrix stays in the cast_timestamp_unit unit tests) - sqlness: drop a bare full scan already covered by the filter above it - shorten function doc comments across datatypes/store-api/table/mito2/ recordbatch to the essential semantics Signed-off-by: Ning Sun <sunning@greptime.com> * fix: drop physical prefilter for columns whose file type differs Follow-up to the review feedback on CompatBatch/prune reader/filter handling for widened time index units. Between/InList/IsNull predicates are prefiltered by PhysicalFilterContext, which builds its physical expression against the FILE's schema while the predicate literals are in the expected (post-alter) unit. Evaluating them against an old-unit SST raised a cross-unit comparison error (Timestamp(ms) >= Timestamp(µs)) that failed the whole scan. Physical prefilter predicates are best-effort pruning hints (the query layer re-applies them above the scan), so drop the prefilter when the column's file type differs from the expected type, mirroring the simple-filter strategy. Verified: Between and a non-rewritten (large) InList on old-unit data no longer error and filter exactly end-to-end (sqlness), and no matching rows are lost at the engine level (engine test). Signed-off-by: Ning Sun <sunning@greptime.com> * test: add direct unit tests for stats cast and prefilter drop The two-step stats cast (reinterpret raw Int64 stats in the file's timestamp type, then rescale to the expected type) and the physical prefilter drop on file/expected type mismatch were only covered end-to-end; add localized unit tests so a regression fails at the exact site: - stats.rs (previously no tests): RowGroupPruningStats min/max over a hand-built RowGroupMetaData — passthrough with no expected metadata, passthrough on same type, and rescale (1000ms -> 1_000_000us, not 1000us) on a widened expected unit - reader.rs: PhysicalFilterContext::new_opt keeps a Between prefilter when file and expected types match and drops it on unit mismatch Signed-off-by: Ning Sun <sunning@greptime.com> * fix: tolerate mixed time units range cache key coverage check * docs: flag mixed-unit hazard in the (unwired) series index The series index stores per-series min/max ts as raw Int64 in the unit of the region metadata at write time, and the searcher builds its range predicates from a single per-region metadata. After a time index unit widen, files of one region would carry mixed units, so a per-file unit (or an index rebuild on such alters) is required before this index is wired into scans. Leave notes at both sites. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover mixed-unit compaction for sparse encoding and strict windows Compaction-path audit follow-up. The compat cast and window math were already covered for dense regions; add the two remaining e2e scenarios: - sparse primary key encoding (used by metric-engine physical regions): widening then compacting mixed-unit files rewrites the old-unit time index correctly through the sparse compaction compat path - strict-window manual compaction: each window output trims rows with a predicate built in the region's new unit against an old-unit file; every instant must survive exactly once (no loss, no cross-window duplication), rescaled Also documents the audit finding that Regular ranged (manual) compaction never trims rows: TwcsPicker sets output_time_range to None and the request time range only selects candidate windows. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover time index unit change in FlatCompatBatch directly The compat layer's rescaling of a widened time index was only verified end-to-end; add direct unit tests for both paths: - dense: identical units skip compat entirely; a widened unit rescales the time index column (1000ms -> 1_000_000us, not reinterpreted) while other columns pass through and the output schema matches the expected metadata - compact sparse (the metric-engine compaction path): same rescaling Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: move timestamp unit division into common-time The exact unit division (UnitQuotient + div_mod_units) is time semantics, not filter logic; move it next to TimeUnit in common-time with a compact test covering representable/non-representable values, negative (floor) instants, and quotient overflow. The ScalarValue helpers stay in filter.rs since common-time has no datafusion dependency. Signed-off-by: Ning Sun <sunning@greptime.com> * test: compat rescales a widened time index and fills an added column together The realistic multi-alter sequence (widen at T1, add column at T2, read a T0 SST) exercises cast and default-fill in the same compute_index_and_fields pass; assert both in one output batch. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: preflight time index widening overflow before any region alters Address review feedback on the overflow guard: - preflight: when the frontend operator receives a widening alter on the time index, run an existence scan (ts outside the target unit's i64 range, LIMIT 1, via the query engine so it covers every region of the table in both standalone and distributed modes) BEFORE any DDL task is submitted. A region that fits can no longer commit the new schema while another region rejects the alter with a non-retryable error. File and row-group pruning keep the scan cheap when nothing overflows. The per-region check in mito2 stays as the final guard for data written after the preflight (the remaining race window); without a validate-only wire field (region.proto lives in the external greptime-proto repo) a fully atomic two-phase validate/commit is out of scope here. - fast path: cast_timestamp_unit returns the filter unchanged when the literal is already in the target unit, skipping the div-mod rebuild. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address auto review comments * fix: remove time index widening overflow preflight Overflow needs timestamps beyond the target unit's i64 range (~year 2262 for nanoseconds), which real workloads never write, so the two existence scans before every widening alter are not worth the cost. Region validation already rejects the alter when an SST's time range overflows the target unit; it now logs the rejection (with the offending file) and returns a deterministic client-facing message. Signed-off-by: Ning Sun <sunning@greptime.com> * test: make sqlness test stable * fix: log instead of rejecting time index widening overflow Overflowing values cast to NULL on read but do not otherwise affect reads or writes, so the alter is allowed; the region-level check now only logs (with the offending file) when an SST's time range exceeds the target unit's i64 range. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
LFC
|
529f046110 |
refactor: json2 v2 storage layout (#8979)
* refactor: json2 v2 storage layout Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> * rethinking when "needs_remainder" Signed-off-by: luofucong <luofc@foxmail.com> * restore "ReadColumns" Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
shuiyisong
|
c4dafb5826 |
fix(promql): resolve derived labels in aggregation arithmetic (#8994)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
7252ceb4bb |
feat(flow): add generic delta merge for incremental aggregates (#8938)
* feat(function): add internal delta merge aggregates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(function): cover delta merge aggregates in SQL Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(function): add Welford delta merge aggregate Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Lei, HUANG
|
f5212d3631 |
feat(mito2): add write cache upload hook (#8992)
Add `WriteCacheUploadStoreWrapper` in `src/mito2/src/cache/write_cache.rs` and wire it through `src/mito2/src/worker.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
00d43b29ad |
feat(query): add experimental DataFusion spill-to-disk controls (#8884)
* feat(query): add experimental DataFusion spill-to-disk controls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(config): regenerate configuration reference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update config API for spill defaults Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): address spill configuration review Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): preserve spill settings with runtime plugins Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Yingwen
|
27165c2fdc |
feat(mito2): add SST range index writer (#8954)
* feat(mito2): add SST range index writer Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): share parquet index writer Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): lazily construct index writer context Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
c6b10bfbb9 |
feat(function): add mergeable stddev_pop state functions (#8972)
* feat(function): add Welford stddev functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(function): cover merged Welford time windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(function): use stddev_pop SQL names Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): make Welford arithmetic partition-stable Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): reject invalid singleton Welford states Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(function): pin Welford state compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(function): remove unreachable variance clamp Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): reject DISTINCT Welford aggregates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(compat): bound Welford downgrade targets Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
109257505e |
feat: report what the graph derives and fix two duplicate-node bugs (#8936)
* feat(semantic-graph): let the generic container yield to k8s.container A container inside a Kubernetes pod reaches the graph twice: as the k8s.container entity kube-state-metrics describes, identified by [pod uid, container name], and as the generic container the OTel resource attributes describe, identified by container.id. One physical container, two nodes. Kubernetes is the primary scenario, so k8s.container keeps its identity and the generic type stands down where it applies. Conventions gain a row-level condition for that: `suppressed_by` withdraws a declaration on rows where any of the named columns has a value. The test has to be per row, not per table — one descriptor table holds both pod rows and bare-runtime rows. Every branch that turns a declaration into rows now shares one guard (`declaration_predicate`), so the condition cannot apply to entities but not to the edges they carry. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): report derived entity declarations in table_semantics information_schema.table_semantics only read table options, so the declarations the built-in conventions derive — for trace tables, for whitelisted Prometheus and OTel descriptor metrics — were invisible. "Why is my table not in the graph?" was answerable only from debug logs, which is not a self-service path. A new `entity_declarations` column reports the entities a table actually contributes: each one's identity, whether it came from an option or from the conventions, and any row-level condition attached to it. An expected entity missing from the list is the answer — the table name is not whitelisted, the source stamp is wrong, an id column is absent. The row filter widens to match: a table that declares nothing by option but derives entities by convention now appears, since it is in the graph and the view has to say so. The provider reaches the derivation through a new metadata-only trait method, keeping catalog below operator. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(semantic-graph): never yield a container to an entity nothing derives The generic container withdrew on any row carrying a pod UID, on the assumption that k8s.container would cover it. Nothing guaranteed that: k8s.container came only from kube-state-metrics, so an OTLP-only deployment — or one whose KSM data had expired or fell outside the query window — lost the container node and its edges entirely instead of gaining a more specific one. The rule now names the superseding type rather than a trigger column, and withdraws only where that type's full identity is on the row. Both OTel sources declare k8s.container themselves, under the identity kube-state-metrics gives it ([pod uid, container name]), so the two sources name one node; `k8s.container.name` joins the descriptor's projected attributes to carry it. Resolution runs once every declaration for the table is known, so a superseding type that ends up undeclared — its columns are gone, or an explicit declaration of it was skipped — leaves the guard empty and the generic container standing. A container can change type; it cannot disappear. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(semantic-graph): build the declaration JSON by serializing a type The column was assembled entry by entry into a serde_json::Map, cloning every value and allocating a String per key. A Serialize struct that consumes the declaration moves the same data instead, and the field order now reads type, origin, identity, then description. The scan path around it was doing the same kind of avoidable work: declarations were derived before the predicate could discard the table, the supersession pass cloned every declaration's identity to look up one, and the option parse claimed the time index for tables that turn out to declare nothing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): keep the structured identity for single-column ids `entity_id_attrs` was NULL whenever the identity came from one column, so a consumer holding `host` = `a3f2...` had no way to tell which column produced it, and no way back to the source table. Most identities are single-column — host, k8s.pod, k8s.node, service — so the common case was the opaque one. Build the JSON object unconditionally. Entity equality still reads `entity_id` alone, so this changes no merging: it records which attributes the id was assembled from, beside an id that deliberately omits them. Also corrects the `entity_id` column doc, which still described the `k=v,k=v` rendering replaced in #8904 by values joined in declared order. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(semantic-graph): keep what a container carried when it yields Yielding to k8s.container cost the row two things it had as a generic container. The runtime container id, which kube_pod_container_info keeps descriptive precisely because it is the handle back to runtime logs and metrics, went missing entirely: neither an id nor an attribute of any node. And the edge vocabulary knew only the generic type, so a row with the more complete labels ended up with fewer connections than one without — the container layer no longer reached its host. Both OTel sources now keep the runtime id and name descriptive on k8s.container, and the vocabulary gains the two edges that mirror the generic type's. Nothing checks a supersession against the edge vocabulary, so that requirement is written down where the rule is. Also: the conventions-failure path now reports the explicit half as its comment always claimed, entity_declarations reports scope columns, and identifies() is private again now that only declaration_predicate calls it. The two RFCs catch up with entity_id_attrs being unconditional, the view listing convention-derived tables, and supersession existing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): report unmatched clients and the longest request `real_wins` pins a `calls` edge's RED metrics to the observed span pairs: when a window's edge key holds any pair, the unmatched client spans are suppressed so one (window, edge) yields one row. That leaves no way to tell a callee that stopped responding from traffic that stopped arriving — both show up as a lower request_count. Add two columns to `semantic_relationships`: - `unmatched_count` — client spans with no server span, counted outside `real_wins` on the same row, so the suppressed population stays visible without splitting the edge into two rows. NULL for agent calls, whose inner join leaves nothing unmatched, and for declared edges. - `duration_max` — the longest single request. It goes through `real_wins`: a pair is timed by the server span while an unmatched client is timed by its own (network wait included), so mixing them would make the max describe a different population than duration_sum and duration_count. Agent calls compute it from the child spans they already aggregate. The projection contract goes from 16 to 18 columns; every branch projects both. Explicit column queries are unaffected, `SELECT *` and ordinal-based readers see the new shape. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(semantic-graph): drop the duplicated source-gate case The whitelist gate on `source=opentelemetry` is already covered by `otel_implicit_declarations_are_gated` and by the wrong-source case in `table_semantics`; here it only paid for another table create, insert and full graph derivation. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * style(semantic-graph): trim the comments back to what the code cannot say Several comments restated the code, repeated a rationale already stated at the type or in the RFC, or explained a test in more words than the test body. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
6ce5747e02 |
fix(mito2): avoid chained L1 rewrites in TWCS (#8981)
* fix(mito2): avoid chained L1 rewrites in TWCS Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): fall back after ineligible L0 pick Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
shuiyisong
|
fb86f6573e |
feat(pipeline): support table-aware JSON2 transforms (#8964)
* feat(pipeline): support table-aware JSON2 transforms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat(pipeline): support JSON2 type hints in transforms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(pipeline): default failed JSON2 transforms to null Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(json2): distinguish invalid settings from layout errors Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
51b94bb73f |
fix(cmd): gate daemon integration test on Unix (#8987)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
localhost
|
6504af641e | fix: increase system disk size to 50 GiB for ECS instances (#8986) | ||
|
Ning Sun
|
d32cd77505 |
fix: postgres describe for more statements (#8974)
* fix: postgres describe for more statements Signed-off-by: Ning Sun <sunning@greptime.com> * fix: cover more show statements Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments - add missing `clippy::too_many_arguments` allow on `query_from_information_schema_dataframe` (CI clippy failure) - take `&ShowKind` in the information-schema dataframe helper so `kind` is no longer cloned at every call site; only the WHERE arm (which needs an owned expression for `sql_to_expr`) clones internally - document why re-applying TQL explain formats never overwrites an existing value (per-query context state) Signed-off-by: Ning Sun <sunning@greptime.com> * chore: trim comments to essentials Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 zhaiAohan
|
de6a903cb1 |
feat: Add built-in daemon mode for standalone service (#8960)
* feat: Add built-in daemon mode for standalone service Add an optional --daemon/-d flag to `greptimedb standalone start` to run the server as a background daemon detached from the shell session, similar to Redis `daemonize yes`. - Use the daemonize crate (Unix only) to fork/setsid and detach from the controlling terminal before the Tokio runtime is built. - stdin/stdout/stderr are redirected to /dev/null; logs continue to go to data_home/logs/ via the existing tracing appender. - On non-Unix platforms, --daemon is a no-op with a warning (runs in foreground). Closes #7314 Signed-off-by: tian1220A <1573612521@qq.com> * test(cmd): add integration test for standalone daemon mode Verify that standalone start --daemon detaches from the shell and brings up the HTTP listener, guarding against regressions where the daemon blocks in the foreground or crashes after forking. Signed-off-by: tian1220A <1573612521@qq.com> * fix(cmd): gate --daemon flag with #[cfg(unix)] and fix rustfmt Address review feedback to not expose --daemon/-d on non-Unix platforms. The daemon field is now gated with #[cfg(unix)], and a cross-platform is_daemon() accessor returns false on non-Unix so maybe_daemonize() stays unconditional. Also fix the rustfmt formatting issues that caused fmt-check to fail. Signed-off-by: tian1220A <1573612521@qq.com> * chore(cmd): declare daemonize dependency directly in cmd daemonize is only used by the cmd crate. Move it out of the workspace dependencies and declare it directly in the unix-only dependencies of src/cmd/Cargo.toml. Signed-off-by: tian1220A <1573612521@qq.com> --------- Signed-off-by: tian1220A <1573612521@qq.com> |
||
|
Lei, HUANG
|
9cfbc42126 |
refactor(mito2): prioritize file count in TWCS picker (#8765)
* refactor(mito2): prioritize file count in TWCS picker Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): migrate TWCS tests to async picker API Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove legacy reduce_runs and merge_seq_files pickers Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): balance TWCS picks by file group Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): treat multi-SST groups as barriers in TWCS pick_count_first Previously pick_count_first filtered out multi-SST file groups and could pick singleton groups across them in one interval. Now multi-SST groups split the candidates into independent segments, so a single pick never crosses a multi-SST group. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove redundant overlaps_files_left_behind check The rebase onto main kept both the upstream fix (#8872) and the branch's selected_overlaps_unselected check for the same deletion-marker problem. The upstream check is fully subsumed: files_to_merge only differs from the window files in append mode, where filter_deleted is already false, and selected_overlaps_unselected treats partially-selected groups as unselected, making it strictly stronger. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): pre-filter by selection time span in selected_overlaps_unselected A window group outside the overall time span of the selected groups cannot overlap any of them, so skip the precise overlap check (and the per-group file id set lookup) for it. In typical TWCS windows the pick is clustered in one segment, so most unselected groups are rejected in O(1). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove FileGroup abstraction from TWCS picker Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): drain compactable backlog after successful compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): score TWCS candidates by predicted progress Redefine what a pick candidate is worth: an interval is only eligible if compacting it makes progress on at least one axis — it reduces the physical file count given the max_output_file_size split threshold (predicted output = ceil(input bytes / threshold)), or it resolves at least one overlap between sorted runs. A pure rewrite that achieves neither (e.g. 32 large balanced files whose output would split back into just as many SSTs) is skipped instead of burning I/O. The candidate metrics are accumulated in a Candidate struct as the interval expands, and the score ranks by predicted file reduction, then overlap participants, then smaller input bytes. With no output size limit the behavior degenerates to the previous count-first rule. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): make TWCS input limit configurable Allow operators to tune the maximum SST inputs through a hidden environment variable while keeping 32 as the validated default. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): apply TWCS trigger at window level Check the total physical SST count before candidate selection so trigger values above the per-task input limit still allow bounded compactions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
c0b8612c41 |
fix(cmd): configure meta client in frontend plugin test (#8977)
Provide valid distributed frontend options in `src/cmd/src/frontend.rs` so enterprise plugin setup can preserve prefilled heartbeat extensions without weakening production validation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Weny Xu
|
c01de4afdc |
fix(operator): whitelist private system table auto create (#8930)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
28e2b5aca2 |
feat: expose missing SST manifest fields (#8965)
* feat: expose missing SST manifest fields Signed-off-by: evenyag <realevenyag@gmail.com> * fix: preserve SST manifest column ordinals Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Dhruv Vaishnav
|
9198462869 |
feat(meta): record physical table reconciliation events (#8935)
* feat(meta): record physical table reconciliation events Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * docs(config): add reconciliation table event Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(meta): address reconciliation event review feedback Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(meta): keep reconciliation event summary volatile Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * refactor(meta): remove unused table state downcasting Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> |