mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-09-05 21:18:57 +00:00
ad7b0ace64af036871fccab15d458cdfc62c19d0
6018
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 discord9
|
ad7b0ace64 |
feat(flow): support eval schedule offsets (#8878)
* feat(flow): support eval schedule offsets Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): remove redundant schedule assertion Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(flow): trim eval offset compatibility scope Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): trim eval offset edge coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(flow): trim eval offset comment noise Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): address eval offset review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): cover Flow eval offset persistence Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 LFC
|
84bd993131 |
refactor(json2): concretize JSON2 schemas at merge scan boundaries (#9016)
* refactor(json2): concretize JSON2 schemas at merge scan boundaries Infer concrete JSON2 output types from remote plans and expose them on MergeScanLogicalPlan before physical planning. Recompute affected local schemas and remove the JSON2-specific rewrite from MergeScanExec. Add SQLness coverage for whole JSON2 columns in windows and joins. Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
 shuiyisong
|
b86da3d35f |
feat: support raw OTLP delta metrics (#8970)
* feat: support raw OTLP delta metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(promql): update sqlness results for normalized label matching Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: derive temporality label from default column prefix Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(promql): add analyze coverage for delta temporality Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): scope label alignment to temporality marker Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: handle count-only histograms and vector broadcasts Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: exclude temporality marker from entity descriptions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: use a fixed label for OTLP aggregation temporality Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): preserve mixed-range semantics for raw delta Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
945e53e0a3 |
fix(client): isolate query and control transports (#8990)
* refactor(client): isolate query and control transports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(client): cover retained Flight transport isolation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(client): satisfy retained Flight test lint Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(client): clarify transport lane routing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 Weny Xu
|
d664326b1e |
chore: bump version to 1.3.0-alpha.1 (#9014)
* chore: bump version to 1.3.0-alpha.1 Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update Cargo lockfile Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>v1.3.0-alpha.1 |
||
|
LFC
|
d62a5a990a |
feat(json2): support list indexing for JSON2 columns (#9013)
feat(query): support list indexing for JSON2 columns Extend JSON2 paths through DataFusion field-access planning, including nested list indexes and object fields following an index. Preserve Variant reads for bracket JSONPath expressions and normalize dot accesses after subscripts to work around the current DataFusion planner limitation. Add unit and sqlness coverage for nested indexes, type conflicts, missing paths, flushes, and compacted SSTs. Signed-off-by: luofucong <luofc@foxmail.com> |
||
 Yingwen
|
d7571c1278 |
feat(mito2): add SST range index searcher (#9003)
* feat(mito2): add SST range index searcher Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): reuse parquet index reader Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify range index pruning Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): cover missing range index series Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Lei, HUANG
|
d351b7d471 |
feat(mito2): pass operation type to write cache upload hook (#9012)
Extend `WriteCacheUploadStoreWrapper::wrap` with the `OperationType` of the upload so implementations can apply per-operation policies (e.g. throttling compaction uploads but not flush uploads). Flush and compaction paths forward their existing `SstWriteRequest::op_type`; `put_and_upload_sst` is flush-only and index rebuild uploads are reported as compaction uploads. Files: `src/mito2/src/cache/write_cache.rs`, `src/mito2/src/sst/index.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 fys
|
05c65f54a8 |
feat(json2): support empty and null JSON2 value (#9010)
* feat(json2): support empty and null JSON2 value Signed-off-by: fys <fengys1996@gmail.com> * test(json2): cover explicit NULL and omitted-column inserts Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo fmt Signed-off-by: fys <fengys1996@gmail.com> * fix: infer empty JSON object as object type Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
discord9
|
27a7047f31 |
feat: preserve row sequences and support exact sequence-range reads (#8865)
* feat(mito2): support exact sequence range reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): cover preserve row sequence table alter Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): clear preserve_row_sequence marker on copy_region_from copy_region_from copies source FileMeta into the target region, which has an independent sequence domain. The physical per-row sequences in the copied file belong to the source region only; trusting them in the target would let an exact sequence-range request replay source-domain rows as if they were target sequences. Clear the preserve_row_sequence marker on copied files so the target fails closed with SequenceRangeUnsupported until the scan provably cannot intersect the copied rows. Add a regression test: copying from a preserve-enabled source into a preserve-enabled target clears the marker, and an exact (2, 7] request on the target returns SequenceRangeUnsupported instead of replaying source rows. Fixes #8865 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: remove redundant doc comments for exact sequence range options Approved comment-cleanup-only changes for #8865: drop outdated doc summaries duplicated on the exact_sequence_range wrapper and the preserve_row_sequence field, drop pure-restatement doc comments on the SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and remove the four structural SQL comments from the alter_preserve_row_sequence case. No behavior changes; .result regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): fail closed exact reads on copied files and extension ranges Address review feedback on #8865: - copy_region_from: clear the source-domain FileMeta::sequence along with the preserve_row_sequence marker. An unmarked file retaining a stale source-domain max sequence could be silently skipped by files_allow_exact_sequence_range() as 'proven disjoint' in the target's independent sequence domain, dropping rows on exact (C, H] reads. With sequence=None the capability check fails closed (SequenceRangeUnsupported) until the copied rows are provably disjoint. - Engine/reader: reject exact sequence-range reads whenever a follower region has an extension range provider attached. Extension streams are returned without a row-level sequence filter, so exactness cannot be proven; treat the capability as missing (fail closed) instead of emitting out-of-range rows. The reader also fails closed as defense in depth. Tests: extend copy_region_from regression to assert the copied file's sequence hint is cleared; mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): use doc comments for test function descriptions Elevate the block comments describing test functions (in scan_test and copy_region_from_test) to /// doc comments, matching the convention used elsewhere in the exact sequence range change. No logic change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): extract helpers and trim comment noise in exact sequence reads PR finalization for #8865 (zero behavior change, full mito2 suite green): - engine: extract validate_sequence_fences and sequence_range_unsupported_reason, keeping error variants, check order and reason strings identical; OSS binds the extension blocker to false. - handle_copy_region: extract remap_copied_file_meta and file_descriptors_for_meta; rename file_ids -> source_file_ids and files_to_copy -> new_file_metas. - compactor: rename max_input_sequence -> known_max_input_sequence, document the None semantics (empty input vs unknown sequence). - Remove restating/outdated comments (ScanInput::sequence_range doc first line, outdated file-pruning note, options test restatements), compress verbatim comments while keeping why/invariants/contracts. Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt, git diff --check, mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): reject foreign-region SSTs in exact sequence reads Reading an SST whose FileMeta.region_id differs from the scanned region means the region's sequence domain is broken (manifest corruption or a repartition/copy path that leaked a source-domain file). Treat this as an explicit RegionSequenceDomainBroken error instead of silently ignoring the file's sequence or falling back to a full scan: the region is unusable for exact sequence-range reads until the foreign lineage is compacted away or repaired. - files_allow_exact_sequence_range / exact_sequence_range now return Result and propagate the error through engine fence validation and scan construction (StatusCode::Internal, distinct from the fallback-capable SequenceRangeUnsupported). - Row-level flat-batch sequence filtering rejects foreign-region files as defense in depth. - Engine test asserts the broken-domain error rather than Unsupported/fallback. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping An unmarked file's FileMeta.sequence may be synthesized by the region-edit or repartition paths (committed+1 import barrier), not a physical max of its rows. Treating it as a whole-file disjoint proof could permanently skip rows that were never incrementally consumed once the flow checkpoint passes that value. Exact sequence-range capability now requires every SST in the region to carry the preserve_row_sequence marker; any unmarked file disables exactness (fallback), and the (C, H] file-selection skip also only applies to marked files. Foreign-region files still raise RegionSequenceDomainBroken as before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): scope exact sequence-range capability to the time-selected read set The exact capability check used to walk the entire SstVersion, so a single unmarked or foreign-region SST anywhere in the region disabled exact reads or raised RegionSequenceDomainBroken even when the request's time range could never touch that file. Both the engine fence and the scan builder now derive the read set with shared time-pruning + exact-min/sst-min selection and validate capability only over the files actually selected: a time-pruned file cannot contribute a row to (C, H], so it cannot affect exactness. The existing fail-loud semantics are unchanged for every selected file (foreign region id -> RegionSequenceDomainBroken; unmarked -> exact unavailable). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): wash untrusted sequences in compaction and restore barrier skipping Compaction with any non-preserved input now writes a sequence-less output: the physical __sequence column is zeroed (the flat format requires the internal columns) and FileMeta.sequence records the region-local admission barrier committed_sequence + 1 (falling back to the flushed frontier). preserve_row_sequence stays false. Exact sequence-range scans interpret an unmarked file's sequence as an admission barrier: barrier <= C means flow has already consumed the whole file, so it is skipped at file level; a missing or newer barrier fails closed. Foreign-region files stay in the selected read set so the capability fence still raises RegionSequenceDomainBroken. This closes the recovery loop: after a region repartition, one time-scoped fallback consumes the migrated rows, then compaction washes the untrusted per-row sequences away and exact incremental reads resume via file-level barrier skipping. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): drop restating comments in known_max_input_sequence tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): trim SQLness result EOF whitespace Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): reuse exact scan file selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): strengthen sequence scan coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): trim ALTER option comments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): remove no-op bulk compaction check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve trusted row sequences when reading SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve target sequence domain for imported SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): add trailing blank line to SQLness result EOF Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): trim exact sequence scan plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): fold exact SST selection checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make legacy compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make PK compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
25d49ba092 |
ci: trigger downstream updates for prereleases (#9008)
* ci: trigger downstream updates for prereleases Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover docs prerelease dispatch Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
LFC
|
15317a131b |
feat(json2): support JSON2 paths in SQL functions (#9007)
feat(query): support JSON2 paths in SQL functions Update the DataFusion fork to expose scalar function planning hooks. Infer JSON2 path output types from scalar, aggregate, and window function signatures, while preserving the default Utf8View behavior for functions that accept arbitrary inputs. Add unit and sqlness coverage for type conflicts, mixed typed and untyped JSON paths, filters, aggregates, and window functions. Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
fys
|
534ab31297 |
feat(flow): add row inserts to frontend client (#9006)
* feat(flow): add row inserts to frontend client Signed-off-by: fys <fengys1996@gmail.com> * feat(flow): support hints for frontend row inserts Signed-off-by: fys <fengys1996@gmail.com> * fix: handle poisoned frontend handler lock in row inserts Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
discord9
|
9c135ebcb3 |
feat!: stabilize streaming analyze metrics (#8966)
* feat: stabilize streaming analyze metrics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: expose analyze memory usage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: simplify analyze stream handling Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve analyze stream sequence on panic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: log analyze stream worker panic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 localhost
|
0de0c01283 | fix: add disk usage logging to GitHub step summary in query regression workflow (#9005) | ||
 discord9andRuihang Xia
|
43c30d1446 |
feat(runtime): add weighted workload scheduler (#8736)
* feat(runtime): add weighted workload scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics Use the GreptimeTeam/catio fork (pinned c20eafc) which adds ClassStats::total_admission_wait and ClassStats::admitted, recorded at each QUEUED -> ADMITTED transition. This exposes the scheduler's own admission delay (excluding Tokio queueing and poll execution), enabling admission-wait based fairness gates. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to dynamic-config revision Bump the catio scheduler fork to 9f4b028 which adds Scheduler::set_weight and Scheduler::set_max_concurrent_polls for runtime configuration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(perf): runtime-adjustable workload scheduler parameters Expose dynamic adjustment of the experimental workload scheduler at runtime: - common-runtime: set_workload_scheduler_weights and set_workload_scheduler_max_concurrent_polls, which forward to the catio scheduler's set_weight/set_max_concurrent_polls when the scheduler is enabled and reject zero values. - servers: /debug/workload_scheduler/weights and /debug/workload_scheduler/max_concurrent_polls POST handlers, so operators can rebalance query/write shares or admission concurrency without restarting the datanode. Both endpoints return 400 with a clear reason when the scheduler is disabled or the requested value is invalid. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(perf): add GET /debug/workload_scheduler status endpoint Returns the current weights (per class), max_concurrent_polls, active_polls and per-class counters (queued, tasks, wakes, polls, completed, cancelled, admitted, total_admission_wait) as JSON. When the scheduler is disabled, returns enabled=false with the other fields omitted, so operators can distinguish 'disabled' from an error. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to time-accounting revision Bump the catio scheduler fork to 257ba56 which replaces admission-count accounting with real execution-time accounting (pass += exec_time / (weight * concurrency)), so CPU share follows the configured weights regardless of poll length. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to lock-free sampling revision Bump the catio scheduler fork to efdc0a4 which adds an optional downsampled clock sampling mode (SchedulerBuilder::sample_every_polls, default off) with a lock-free per-class atomic counter, so the downsampled path costs one fetch_add per poll instead of a global mutex. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin catio to scheduler PR head Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): add scheduler bypass control Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: advance catio scheduler fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin merged catio scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: regenerate config docs for workload scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin catio scheduler test fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(http): satisfy scheduler lifecycle clippy Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: add distributed scheduler toggle coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: finalize workload scheduler runtime controls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin merged catio atomic weights Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: preserve unrelated lockfile resolution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(runtime): downsample scheduler time accounting Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(runtime): verify cross-runtime scheduler progress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): configure scheduler poll sampling Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(runtime): clarify scheduler activation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(runtime): explain scheduler use case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> |
||
 dennis zhuang
|
02278d301b |
feat: derive k8s.node from OTLP resource attributes (#9002)
The k8s.pod runs_on k8s.node rule can only fire from kube_pod_info: it is the only table declaring both endpoints, and co-declared edges require both on the same row. A deployment sending only OTLP has no kube-state-metrics tables, so its node layer is invisible and that rule has no source at all, even though the resource attributes carry k8s.node.name. Declare k8s.node in otlp_trace_entities and in the synthesized resource descriptor, and project k8s.node.name in the descriptor writer so the column the declaration needs exists. Identity is the name rather than k8s.node.uid: kube-state-metrics carries no node UID, so the name is the only identity both sources can agree on. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Ning Sun
|
6f1dd0cb76 |
feat: allow widening the time index column's timestamp unit via ALTER TABLE, mito2 table only (#8894)
* feat: allow widening the time index column's timestamp unit via ALTER TABLE ... MODIFY COLUMN Previously MODIFY COLUMN rejected the time index column outright. Now the time index unit can be widened (Second -> Milli -> Micro -> Nano), which is lossless for data that fits the target unit: historical data in old SSTs is cast to the new unit on read by the existing schema-compat layer, and compaction rewrites it lazily. Narrowing and non-timestamp targets remain rejected; tag columns keep being rejected. Widening is rejected if any SST's time range would overflow the target unit's i64 range (e.g. millisecond -> nanosecond beyond year 2262), since the cast would silently null those values. Read-path correctness for old-unit SSTs (verified by new engine e2e tests and sqlness WHERE queries): - row-group min/max pruning: parquet statistics of a timestamp column are raw integers in the file's unit; when the region metadata's type differs (also the case for altered field columns), stats are now interpreted in the file's type and converted to the expected type before pruning. Without this, a new-unit predicate silently pruned whole row groups of old-unit files (wrong results, rows missing). - SST-level simple filters are skipped for columns whose file type differs from the expected type; the predicate is applied by the query layer's residual filter above the region scan. Also drop the stale "timestamp columns cannot change type" debug_assert. - retry idempotency: a same-type ModifyColumnType on the time index validates as a no-op and `need_alter` returns false, so a retried alter procedure (region already altered before the previous attempt failed) converges instead of aborting forever. - add TimeUnit ordering and ConcreteDataType::is_timestamp_unit_widening_to - relax ModifyColumnType validation in store-api and table metadata - tests: unit tests in datatypes/store-api/table; mito2 engine e2e tests (flushed SST + new writes + reopen + retry + overflow + predicate scans, cross-unit dedup, mixed-unit SSTs, compaction); sqlness cases incl. partitioned table and WHERE filters over old-unit data Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve parquet filter issue * test: provide sqlness tests * refactor: drop trivial test cases and shorten comments Review pass over the branch's additions: - datatypes: keep a representative subset of the widening-matrix asserts - table: collapse the three single-branch rejection blocks into one loop - mito2: drop the boundary gt_eq and post-compaction predicate asserts (covered by the exact-filter regression test and sqlness); drop the engine-level gt_eq/lt_eq casts (full operator matrix stays in the cast_timestamp_unit unit tests) - sqlness: drop a bare full scan already covered by the filter above it - shorten function doc comments across datatypes/store-api/table/mito2/ recordbatch to the essential semantics Signed-off-by: Ning Sun <sunning@greptime.com> * fix: drop physical prefilter for columns whose file type differs Follow-up to the review feedback on CompatBatch/prune reader/filter handling for widened time index units. Between/InList/IsNull predicates are prefiltered by PhysicalFilterContext, which builds its physical expression against the FILE's schema while the predicate literals are in the expected (post-alter) unit. Evaluating them against an old-unit SST raised a cross-unit comparison error (Timestamp(ms) >= Timestamp(µs)) that failed the whole scan. Physical prefilter predicates are best-effort pruning hints (the query layer re-applies them above the scan), so drop the prefilter when the column's file type differs from the expected type, mirroring the simple-filter strategy. Verified: Between and a non-rewritten (large) InList on old-unit data no longer error and filter exactly end-to-end (sqlness), and no matching rows are lost at the engine level (engine test). Signed-off-by: Ning Sun <sunning@greptime.com> * test: add direct unit tests for stats cast and prefilter drop The two-step stats cast (reinterpret raw Int64 stats in the file's timestamp type, then rescale to the expected type) and the physical prefilter drop on file/expected type mismatch were only covered end-to-end; add localized unit tests so a regression fails at the exact site: - stats.rs (previously no tests): RowGroupPruningStats min/max over a hand-built RowGroupMetaData — passthrough with no expected metadata, passthrough on same type, and rescale (1000ms -> 1_000_000us, not 1000us) on a widened expected unit - reader.rs: PhysicalFilterContext::new_opt keeps a Between prefilter when file and expected types match and drops it on unit mismatch Signed-off-by: Ning Sun <sunning@greptime.com> * fix: tolerate mixed time units range cache key coverage check * docs: flag mixed-unit hazard in the (unwired) series index The series index stores per-series min/max ts as raw Int64 in the unit of the region metadata at write time, and the searcher builds its range predicates from a single per-region metadata. After a time index unit widen, files of one region would carry mixed units, so a per-file unit (or an index rebuild on such alters) is required before this index is wired into scans. Leave notes at both sites. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover mixed-unit compaction for sparse encoding and strict windows Compaction-path audit follow-up. The compat cast and window math were already covered for dense regions; add the two remaining e2e scenarios: - sparse primary key encoding (used by metric-engine physical regions): widening then compacting mixed-unit files rewrites the old-unit time index correctly through the sparse compaction compat path - strict-window manual compaction: each window output trims rows with a predicate built in the region's new unit against an old-unit file; every instant must survive exactly once (no loss, no cross-window duplication), rescaled Also documents the audit finding that Regular ranged (manual) compaction never trims rows: TwcsPicker sets output_time_range to None and the request time range only selects candidate windows. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover time index unit change in FlatCompatBatch directly The compat layer's rescaling of a widened time index was only verified end-to-end; add direct unit tests for both paths: - dense: identical units skip compat entirely; a widened unit rescales the time index column (1000ms -> 1_000_000us, not reinterpreted) while other columns pass through and the output schema matches the expected metadata - compact sparse (the metric-engine compaction path): same rescaling Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: move timestamp unit division into common-time The exact unit division (UnitQuotient + div_mod_units) is time semantics, not filter logic; move it next to TimeUnit in common-time with a compact test covering representable/non-representable values, negative (floor) instants, and quotient overflow. The ScalarValue helpers stay in filter.rs since common-time has no datafusion dependency. Signed-off-by: Ning Sun <sunning@greptime.com> * test: compat rescales a widened time index and fills an added column together The realistic multi-alter sequence (widen at T1, add column at T2, read a T0 SST) exercises cast and default-fill in the same compute_index_and_fields pass; assert both in one output batch. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: preflight time index widening overflow before any region alters Address review feedback on the overflow guard: - preflight: when the frontend operator receives a widening alter on the time index, run an existence scan (ts outside the target unit's i64 range, LIMIT 1, via the query engine so it covers every region of the table in both standalone and distributed modes) BEFORE any DDL task is submitted. A region that fits can no longer commit the new schema while another region rejects the alter with a non-retryable error. File and row-group pruning keep the scan cheap when nothing overflows. The per-region check in mito2 stays as the final guard for data written after the preflight (the remaining race window); without a validate-only wire field (region.proto lives in the external greptime-proto repo) a fully atomic two-phase validate/commit is out of scope here. - fast path: cast_timestamp_unit returns the filter unchanged when the literal is already in the target unit, skipping the div-mod rebuild. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address auto review comments * fix: remove time index widening overflow preflight Overflow needs timestamps beyond the target unit's i64 range (~year 2262 for nanoseconds), which real workloads never write, so the two existence scans before every widening alter are not worth the cost. Region validation already rejects the alter when an SST's time range overflows the target unit; it now logs the rejection (with the offending file) and returns a deterministic client-facing message. Signed-off-by: Ning Sun <sunning@greptime.com> * test: make sqlness test stable * fix: log instead of rejecting time index widening overflow Overflowing values cast to NULL on read but do not otherwise affect reads or writes, so the alter is allowed; the region-level check now only logs (with the offending file) when an SST's time range exceeds the target unit's i64 range. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
LFC
|
529f046110 |
refactor: json2 v2 storage layout (#8979)
* refactor: json2 v2 storage layout Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> * rethinking when "needs_remainder" Signed-off-by: luofucong <luofc@foxmail.com> * restore "ReadColumns" Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
shuiyisong
|
c4dafb5826 |
fix(promql): resolve derived labels in aggregation arithmetic (#8994)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
7252ceb4bb |
feat(flow): add generic delta merge for incremental aggregates (#8938)
* feat(function): add internal delta merge aggregates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(function): cover delta merge aggregates in SQL Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(function): add Welford delta merge aggregate Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Lei, HUANG
|
f5212d3631 |
feat(mito2): add write cache upload hook (#8992)
Add `WriteCacheUploadStoreWrapper` in `src/mito2/src/cache/write_cache.rs` and wire it through `src/mito2/src/worker.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
00d43b29ad |
feat(query): add experimental DataFusion spill-to-disk controls (#8884)
* feat(query): add experimental DataFusion spill-to-disk controls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(config): regenerate configuration reference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update config API for spill defaults Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): address spill configuration review Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): preserve spill settings with runtime plugins Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Yingwen
|
27165c2fdc |
feat(mito2): add SST range index writer (#8954)
* feat(mito2): add SST range index writer Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): share parquet index writer Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): lazily construct index writer context Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
c6b10bfbb9 |
feat(function): add mergeable stddev_pop state functions (#8972)
* feat(function): add Welford stddev functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(function): cover merged Welford time windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(function): use stddev_pop SQL names Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): make Welford arithmetic partition-stable Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): reject invalid singleton Welford states Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(function): pin Welford state compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(function): remove unreachable variance clamp Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): reject DISTINCT Welford aggregates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(compat): bound Welford downgrade targets Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
109257505e |
feat: report what the graph derives and fix two duplicate-node bugs (#8936)
* feat(semantic-graph): let the generic container yield to k8s.container A container inside a Kubernetes pod reaches the graph twice: as the k8s.container entity kube-state-metrics describes, identified by [pod uid, container name], and as the generic container the OTel resource attributes describe, identified by container.id. One physical container, two nodes. Kubernetes is the primary scenario, so k8s.container keeps its identity and the generic type stands down where it applies. Conventions gain a row-level condition for that: `suppressed_by` withdraws a declaration on rows where any of the named columns has a value. The test has to be per row, not per table — one descriptor table holds both pod rows and bare-runtime rows. Every branch that turns a declaration into rows now shares one guard (`declaration_predicate`), so the condition cannot apply to entities but not to the edges they carry. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): report derived entity declarations in table_semantics information_schema.table_semantics only read table options, so the declarations the built-in conventions derive — for trace tables, for whitelisted Prometheus and OTel descriptor metrics — were invisible. "Why is my table not in the graph?" was answerable only from debug logs, which is not a self-service path. A new `entity_declarations` column reports the entities a table actually contributes: each one's identity, whether it came from an option or from the conventions, and any row-level condition attached to it. An expected entity missing from the list is the answer — the table name is not whitelisted, the source stamp is wrong, an id column is absent. The row filter widens to match: a table that declares nothing by option but derives entities by convention now appears, since it is in the graph and the view has to say so. The provider reaches the derivation through a new metadata-only trait method, keeping catalog below operator. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(semantic-graph): never yield a container to an entity nothing derives The generic container withdrew on any row carrying a pod UID, on the assumption that k8s.container would cover it. Nothing guaranteed that: k8s.container came only from kube-state-metrics, so an OTLP-only deployment — or one whose KSM data had expired or fell outside the query window — lost the container node and its edges entirely instead of gaining a more specific one. The rule now names the superseding type rather than a trigger column, and withdraws only where that type's full identity is on the row. Both OTel sources declare k8s.container themselves, under the identity kube-state-metrics gives it ([pod uid, container name]), so the two sources name one node; `k8s.container.name` joins the descriptor's projected attributes to carry it. Resolution runs once every declaration for the table is known, so a superseding type that ends up undeclared — its columns are gone, or an explicit declaration of it was skipped — leaves the guard empty and the generic container standing. A container can change type; it cannot disappear. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(semantic-graph): build the declaration JSON by serializing a type The column was assembled entry by entry into a serde_json::Map, cloning every value and allocating a String per key. A Serialize struct that consumes the declaration moves the same data instead, and the field order now reads type, origin, identity, then description. The scan path around it was doing the same kind of avoidable work: declarations were derived before the predicate could discard the table, the supersession pass cloned every declaration's identity to look up one, and the option parse claimed the time index for tables that turn out to declare nothing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): keep the structured identity for single-column ids `entity_id_attrs` was NULL whenever the identity came from one column, so a consumer holding `host` = `a3f2...` had no way to tell which column produced it, and no way back to the source table. Most identities are single-column — host, k8s.pod, k8s.node, service — so the common case was the opaque one. Build the JSON object unconditionally. Entity equality still reads `entity_id` alone, so this changes no merging: it records which attributes the id was assembled from, beside an id that deliberately omits them. Also corrects the `entity_id` column doc, which still described the `k=v,k=v` rendering replaced in #8904 by values joined in declared order. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(semantic-graph): keep what a container carried when it yields Yielding to k8s.container cost the row two things it had as a generic container. The runtime container id, which kube_pod_container_info keeps descriptive precisely because it is the handle back to runtime logs and metrics, went missing entirely: neither an id nor an attribute of any node. And the edge vocabulary knew only the generic type, so a row with the more complete labels ended up with fewer connections than one without — the container layer no longer reached its host. Both OTel sources now keep the runtime id and name descriptive on k8s.container, and the vocabulary gains the two edges that mirror the generic type's. Nothing checks a supersession against the edge vocabulary, so that requirement is written down where the rule is. Also: the conventions-failure path now reports the explicit half as its comment always claimed, entity_declarations reports scope columns, and identifies() is private again now that only declaration_predicate calls it. The two RFCs catch up with entity_id_attrs being unconditional, the view listing convention-derived tables, and supersession existing. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(semantic-graph): report unmatched clients and the longest request `real_wins` pins a `calls` edge's RED metrics to the observed span pairs: when a window's edge key holds any pair, the unmatched client spans are suppressed so one (window, edge) yields one row. That leaves no way to tell a callee that stopped responding from traffic that stopped arriving — both show up as a lower request_count. Add two columns to `semantic_relationships`: - `unmatched_count` — client spans with no server span, counted outside `real_wins` on the same row, so the suppressed population stays visible without splitting the edge into two rows. NULL for agent calls, whose inner join leaves nothing unmatched, and for declared edges. - `duration_max` — the longest single request. It goes through `real_wins`: a pair is timed by the server span while an unmatched client is timed by its own (network wait included), so mixing them would make the max describe a different population than duration_sum and duration_count. Agent calls compute it from the child spans they already aggregate. The projection contract goes from 16 to 18 columns; every branch projects both. Explicit column queries are unaffected, `SELECT *` and ordinal-based readers see the new shape. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(semantic-graph): drop the duplicated source-gate case The whitelist gate on `source=opentelemetry` is already covered by `otel_implicit_declarations_are_gated` and by the wrong-source case in `table_semantics`; here it only paid for another table create, insert and full graph derivation. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * style(semantic-graph): trim the comments back to what the code cannot say Several comments restated the code, repeated a rationale already stated at the type or in the RFC, or explained a test in more words than the test body. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
6ce5747e02 |
fix(mito2): avoid chained L1 rewrites in TWCS (#8981)
* fix(mito2): avoid chained L1 rewrites in TWCS Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): fall back after ineligible L0 pick Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
shuiyisong
|
fb86f6573e |
feat(pipeline): support table-aware JSON2 transforms (#8964)
* feat(pipeline): support table-aware JSON2 transforms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat(pipeline): support JSON2 type hints in transforms Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(pipeline): default failed JSON2 transforms to null Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(json2): distinguish invalid settings from layout errors Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
51b94bb73f |
fix(cmd): gate daemon integration test on Unix (#8987)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
localhost
|
6504af641e | fix: increase system disk size to 50 GiB for ECS instances (#8986) | ||
|
Ning Sun
|
d32cd77505 |
fix: postgres describe for more statements (#8974)
* fix: postgres describe for more statements Signed-off-by: Ning Sun <sunning@greptime.com> * fix: cover more show statements Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments - add missing `clippy::too_many_arguments` allow on `query_from_information_schema_dataframe` (CI clippy failure) - take `&ShowKind` in the information-schema dataframe helper so `kind` is no longer cloned at every call site; only the WHERE arm (which needs an owned expression for `sql_to_expr`) clones internally - document why re-applying TQL explain formats never overwrites an existing value (per-query context state) Signed-off-by: Ning Sun <sunning@greptime.com> * chore: trim comments to essentials Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 zhaiAohan
|
de6a903cb1 |
feat: Add built-in daemon mode for standalone service (#8960)
* feat: Add built-in daemon mode for standalone service Add an optional --daemon/-d flag to `greptimedb standalone start` to run the server as a background daemon detached from the shell session, similar to Redis `daemonize yes`. - Use the daemonize crate (Unix only) to fork/setsid and detach from the controlling terminal before the Tokio runtime is built. - stdin/stdout/stderr are redirected to /dev/null; logs continue to go to data_home/logs/ via the existing tracing appender. - On non-Unix platforms, --daemon is a no-op with a warning (runs in foreground). Closes #7314 Signed-off-by: tian1220A <1573612521@qq.com> * test(cmd): add integration test for standalone daemon mode Verify that standalone start --daemon detaches from the shell and brings up the HTTP listener, guarding against regressions where the daemon blocks in the foreground or crashes after forking. Signed-off-by: tian1220A <1573612521@qq.com> * fix(cmd): gate --daemon flag with #[cfg(unix)] and fix rustfmt Address review feedback to not expose --daemon/-d on non-Unix platforms. The daemon field is now gated with #[cfg(unix)], and a cross-platform is_daemon() accessor returns false on non-Unix so maybe_daemonize() stays unconditional. Also fix the rustfmt formatting issues that caused fmt-check to fail. Signed-off-by: tian1220A <1573612521@qq.com> * chore(cmd): declare daemonize dependency directly in cmd daemonize is only used by the cmd crate. Move it out of the workspace dependencies and declare it directly in the unix-only dependencies of src/cmd/Cargo.toml. Signed-off-by: tian1220A <1573612521@qq.com> --------- Signed-off-by: tian1220A <1573612521@qq.com> |
||
|
Lei, HUANG
|
9cfbc42126 |
refactor(mito2): prioritize file count in TWCS picker (#8765)
* refactor(mito2): prioritize file count in TWCS picker Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): migrate TWCS tests to async picker API Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove legacy reduce_runs and merge_seq_files pickers Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): balance TWCS picks by file group Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): treat multi-SST groups as barriers in TWCS pick_count_first Previously pick_count_first filtered out multi-SST file groups and could pick singleton groups across them in one interval. Now multi-SST groups split the candidates into independent segments, so a single pick never crosses a multi-SST group. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove redundant overlaps_files_left_behind check The rebase onto main kept both the upstream fix (#8872) and the branch's selected_overlaps_unselected check for the same deletion-marker problem. The upstream check is fully subsumed: files_to_merge only differs from the window files in append mode, where filter_deleted is already false, and selected_overlaps_unselected treats partially-selected groups as unselected, making it strictly stronger. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): pre-filter by selection time span in selected_overlaps_unselected A window group outside the overall time span of the selected groups cannot overlap any of them, so skip the precise overlap check (and the per-group file id set lookup) for it. In typical TWCS windows the pick is clustered in one segment, so most unselected groups are rejected in O(1). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove FileGroup abstraction from TWCS picker Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): drain compactable backlog after successful compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): score TWCS candidates by predicted progress Redefine what a pick candidate is worth: an interval is only eligible if compacting it makes progress on at least one axis — it reduces the physical file count given the max_output_file_size split threshold (predicted output = ceil(input bytes / threshold)), or it resolves at least one overlap between sorted runs. A pure rewrite that achieves neither (e.g. 32 large balanced files whose output would split back into just as many SSTs) is skipped instead of burning I/O. The candidate metrics are accumulated in a Candidate struct as the interval expands, and the score ranks by predicted file reduction, then overlap participants, then smaller input bytes. With no output size limit the behavior degenerates to the previous count-first rule. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): make TWCS input limit configurable Allow operators to tune the maximum SST inputs through a hidden environment variable while keeping 32 as the validated default. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): apply TWCS trigger at window level Check the total physical SST count before candidate selection so trigger values above the per-task input limit still allow bounded compactions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
c0b8612c41 |
fix(cmd): configure meta client in frontend plugin test (#8977)
Provide valid distributed frontend options in `src/cmd/src/frontend.rs` so enterprise plugin setup can preserve prefilled heartbeat extensions without weakening production validation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Weny Xu
|
c01de4afdc |
fix(operator): whitelist private system table auto create (#8930)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
28e2b5aca2 |
feat: expose missing SST manifest fields (#8965)
* feat: expose missing SST manifest fields Signed-off-by: evenyag <realevenyag@gmail.com> * fix: preserve SST manifest column ordinals Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Dhruv Vaishnav
|
9198462869 |
feat(meta): record physical table reconciliation events (#8935)
* feat(meta): record physical table reconciliation events Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * docs(config): add reconciliation table event Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(meta): address reconciliation event review feedback Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(meta): keep reconciliation event summary volatile Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * refactor(meta): remove unused table state downcasting Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> |
||
|
Yingwen
|
ba3c5a939e |
chore(mito2): reduce default auto flush interval (#8971)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
bd7d2c1dfa |
fix(mito2): use target sequence for foreign SSTs (#8946)
* fix(mito2): use target sequence for foreign SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): address foreign SST review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 wy471xandNing Sun
|
aaa843104b |
fix(mysql): interpret prepared statement datetime params in session timezone (#8923)
* fix(mysql): interpret prepared statement datetime params in session timezone Binary DATETIME parameters of server-side prepared statements were converted as if UTC, ignoring the session timezone set via SET time_zone. Convert them with the session timezone and add an integration test covering prepared inserts and predicates under Asia/Shanghai. Signed-off-by: wy471x <wy471x@gmail.com> * refactor: share naive datetime timezone policy via common-time Address review feedback on the prepared-statement timezone fix: - Expose Timestamp::from_naive_datetime in common-time so the DST policy (gap -> error, ambiguous -> earlier instant) lives in one place, shared by the text protocol (Timestamp::from_str) and the MySQL binary protocol. - Route the MySQL prepared-statement datetime conversion through it. - Match the target type before converting datetime params so PreparedStmtTypeMismatch fails fast without wasted conversion. - Use the short Timezone import form for consistency with the rest of servers. Signed-off-by: wy471x <wy471x@gmail.com> --------- Signed-off-by: wy471x <wy471x@gmail.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
Lei, HUANG
|
a2f39ecf7b |
feat: harden frontend heartbeat extensions (#8803)
Add a typed `Command::build_with_heartbeat_extensions` seam in `src/cmd/src/frontend.rs`. Freeze and harden `FrontendHeartbeatExtensions` in `src/frontend/src/heartbeat.rs`, with lifecycle and race coverage in `src/frontend/src/heartbeat/tests.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 sun
|
9befb7e64c | feat: update dashboard to v0.13.14 (#8968) | ||
|
discord9
|
beded6e232 |
fix(flow): avoid insert select HTTP/2 stalls (#8962)
* fix(flow): avoid insert select HTTP/2 stalls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): share record batch forwarding for DML Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
a7ba8b01e1 |
fix(flow): restore FrontendClient::sql API (#8963)
fix(flow): restore batching frontend SQL Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Ning Sun
|
b31f05eb59 |
fix: update tokio-postgres and correct explain/fetch cursor output schema (#8955)
* chore(deps): update tokio-postgres * fix: describing fetch cursor and analyze |
||
|
Weny Xu
|
1409e66837 |
refactor(flight): add request builder and defer DoGet execution (#8953)
* refactor: add flight request builder Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use flight request builder in flow Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: defer frontend flight query execution Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grpc): avoid cloning requests during auth Signed-off-by: WenyXu <wenymedia@gmail.com> * test(grpc): cover flight request timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(client): share Flight message reader Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(flow): add Flight DoGet timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(client): gate Flight DDL helpers for testing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): restore Flight stream error semantics Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(grpc): document Flight stream input constructors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(flight): preserve deferred stream context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): use Flight stream SNAFU context Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
localhost
|
35ea88a4ef |
feat(ci): run query regression on ephemeral Aliyun ECS runners (#8937)
* feat(ci): add aliyun ecs ephemeral runner path for query regression Signed-off-by: paomian <xpaomian@gmail.com> * fix: improve condition for query-regression job execution in workflow * feat: update Docker installation to use official repository and add GPG key handling * Refactor query regression runner setup and configuration - Removed deprecated PersistentVolumeClaim for build cache. - Introduced a new bootstrap script for setting up the ECS runner host. - Deleted obsolete Helm values files for runner configuration. - Updated the Aliyun ECS runner provisioning script to reflect new cache paths. - Modified GitHub workflows to use the new Aliyun ECS runner setup. - Adjusted documentation to clarify the new runner lifecycle and provisioning process. * fix: enhance runner service management during bootstrap process * fix: update alibabacloud_tea_openapi dependency version in metadata * feat: enhance ECS runner scripts with region_id and resource_group_id support * fix: move containerd content store to data root for improved storage management * feat: rename query-regression runner to ephemeral-github runner and update related scripts * fix: update sentinel polling method to use serial console output for improved reliability * fix: add environment variable checks for Alibaba Cloud access keys in ECS client * fix: improve error handling in GitHub API requests for better diagnostics * fix: improve cache disk detection logic for Aliyun ECS instances * fix: enhance cache disk waiting logic with detailed output and error handling * fix: update dependency version for alibabacloud_tea_openapi in teardown script * fix: enhance cache disk waiting logic for better compatibility and clarity * fix: enhance console output handling and add incremental logging during instance provisioning * fix: add PATH environment variable for runner jobs in service and provision script * fix: add machine telemetry sampling and logging during query regression jobs * fix: update query regression documentation and provision script for cache disk handling * fix: update SCCACHE_CACHE_SIZE validation to 10G for improved caching efficiency * fix: remove outdated cache size checks and cleanup logic for fresh system disk runs * fix: enhance instance deletion logic with region handling and console output export * fix: add swap file setup and OOM handling for ECS runner to improve stability * fix: update OOM handling and service restart logic for ECS runner to enhance stability * fix: increase system disk size to 100 GiB for cold double nightly builds to prevent ENOSPC errors * fix: increase system disk size to 150 GiB for ECS runner to prevent ENOSPC errors * fix: add keep_instance option to preserve ECS instance for post-mortem debugging * fix: disable unattended upgrades to prevent job cancellations during library updates * fix: reduce system disk size to 40 GiB for ECS runner to prevent ENOSPC errors * feat: Refactor Aliyun ECS runner provisioning and introduce nightly regression comparison - Update `aliyun-ecs-runner-provision.py` to remove cache disk handling, simplifying the provisioning process. - Introduce `query-regression-nightly-refs.py` to resolve and compare SHAs from successful nightly builds. - Create `query-regression-nightly.yml` workflow to trigger nightly comparisons based on successful builds. - Enhance `query-regression.yml` to include a `test-tooling` job for validating Python scripts before provisioning. - Update tests for the new nightly reference selection logic and refactor existing tests to align with the new caching strategy. - Modify documentation to reflect changes in caching and nightly comparison workflows. * fix: enhance runner image tool verification with detailed checks * fix: improve error handling in runner image tool verification * fix: update tool versions in ECS image and workflow for consistency * fix: correct typo in error message for unparseable ECS creation time * fix: update README and workflow files for query regression tests and image hygiene --------- Signed-off-by: paomian <xpaomian@gmail.com> |
||
|
Weny Xu
|
144f83528d |
test: renew etcd TLS certificates (#8956)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
dennis zhuang
|
6d86e6ff06 |
feat: synthesize OTLP resource descriptor for the semantic entity graph (#8904)
* fix(servers): compose OTLP metrics job from service.namespace/service.name
The OTel Prometheus compatibility spec defines job as
"<service.namespace>/<service.name>" when the namespace is present.
The OTLP metrics path only used the bare service.name, so the job tag
diverged from target_info produced by Prometheus-side exporters for the
same resource. Compose the namespace form, and keep not fabricating a
job when service.name is absent.
Behavior change: resources carrying service.namespace now get
"namespace/name" as their job tag value.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(otlp): synthesize otel_resource_info at OTLP metrics ingestion
Ordinary OTLP metrics scatter filtered resource attributes as tags over
every logical metric table, so metrics-only services contribute nothing
to the semantic entity graph. Each request now also projects its
distinct resources into one info-metric-shaped mito table,
otel_resource_info: a fixed allowlist of identity-relevant attributes
under their raw OTel keys (independent of the label translation
strategy and the promote/ignore headers) plus derived job/instance
compatibility columns, value 1.0, and the newest data-point timestamp.
The descriptor is written after the main insert is committed; a failure
there (conflicting pre-existing table, auto-create disabled) degrades
to an OTLP partial_success warning with rejected_data_points = 0
instead of failing the request and triggering client retries of
already-accepted data. A request writing a metric named
otel_resource_info suppresses synthesis. Legacy mode is unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(operator): otel info-metric conventions with host/container entities
Whitelist the ingestion-synthesized otel_resource_info descriptor via a
new otel_info_metrics conventions map, gated on source=opentelemetry
(the existing gate hardcoded source=prometheus). Its declarations use
explicit descriptive lists instead of descriptive_rest so identifying
attributes of other entities do not leak into service.instance.
Conventions tightened per the Astronomy Shop findings: host identity is
host.id with host.name descriptive only (host.name is not stable across
SDKs and resource detectors), a generic container entity (new entity
type) is declared only when container.id is present, and trace-v1
tables now synthesize host/container from their flattened resource
attributes too. New co-declared edges: service.instance runs_on
container, container runs_on host.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): cover the resource descriptor in integration tests
Covers the descriptor's raw-key columns and info-metric options through
the HTTP path, the namespace/name job composition end-to-end, column
names being independent of the translation strategy, the allowlist
excluding unlisted resource attributes, auto-create after a drop, the
metric-name collision suppressing synthesis, and the partial-success
warning (rejected_data_points = 0) when a pre-existing incompatible
table fails the descriptor write while metric data is accepted.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: cargo fmt
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(frontend): degrade descriptor permission denial to a warning
A table-level permission policy denying otel_resource_info would have
failed the whole OTLP metrics request because the descriptor's
permission check ran before the main insert. The descriptor is derived
enrichment: check its permission in the degrade path so a denial skips
the write and surfaces as the partial-success warning, like any other
descriptor write failure.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(otlp): guard descriptor writes with semantic ownership markers
A pre-existing schema-compatible table named otel_resource_info would
silently receive descriptor rows while its missing semantic stamps kept
it out of the entity graph. The descriptor write now requires the
auto-created table's ownership markers (mito engine + signal_type +
source + metric.type=info + metadata_quality=declared) and otherwise
degrades to the partial-success warning; the entity-graph gate for the
otel whitelist likewise requires metric.type=info, so a user table
stamped with only signal/source no longer picks up implicit
declarations.
Also fold the descriptor write cost into the response and surface the
degrade warning through the otel-arrow BatchStatus status_message.
Integration tests pin the full marker set on auto-create and that an
existing owned descriptor keeps accepting writes without degrading —
a missing marker would otherwise silently stop every descriptor write
after the first request.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(otlp): build descriptor rows without the per-resource BTreeMap
Projecting a resource allocated a BTreeMap and then collected it into the
row key, and every attribute was matched against the allowlist by linear
scan. Collect the tags into a Vec and sort once, and match the allowlist
instead of scanning it. Measured on the conversion path: descriptor work
drops 16-18%, from 10.6% to 8.9% of conversion CPU on the worst shape
(1000 resources with 4 data points each), where the cost tracks resource
count rather than data-point count.
Also trims the comments and tests added with the descriptor to what
carries information.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(otlp): pin the descriptor permission-denial degrade path
A policy denying the descriptor table must not fail the metrics request,
which the fix in
|
||
|
dennis zhuang
|
1851f6bf4d |
fix(query): keep INSERT timestamp conversion out of the source query (#8911)
* fix(query): keep INSERT timestamp conversion out of the source query Interpreting an INSERT's string timestamps used to work by pushing the conversion down into the source query, which changed what that query means. Two consequences: - Pushing through a UNION's DISTINCT moved the dedup key from the raw strings to parsed instants, so rows spelling the same instant differently collapsed into one. On an append-only table that is a silently dropped row. - A UNION branch that needed no conversion (a NULL, or an explicit cast) made the whole column give up, leaving sibling branches on UTC while the rest of the row used the session timezone. Convert at the assignment instead, by routing its cast through a timezone-carrying timestamp type and back. Arrow applies the timezone when a cast target carries one, and stripping it afterwards preserves the value. The source query is no longer touched, so both cases go away and the tree-walking rewrite (roughly 160 lines) is deleted. The rewrite reads source types, so it now runs TypeCoercion first: a UNION still carries its loose per-branch schema before coercion, and retargeting a cast whose input later becomes a timestamp would shift the value rather than reinterpret it. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): address review on INSERT assignment rewrite - Clone the input `Arc` instead of the whole subtree, and only rebuild it when a `Values` row actually changes. - Defer cloning the cast source until the literal-folding path has been ruled out. - Move the UTC check onto `Timezone::is_utc`, replacing a bare string compare. - Cover a prepared `INSERT ... VALUES (?)`: an untyped placeholder types as `Null`, so the assignment cast is left for parameter substitution. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |