* ci: add backport workflow to create backport PRs from backport labels
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci: document backport labels in PR template and AGENTS.md
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(pipeline): coalesce concurrent pipeline cache misses
The pipeline cache reads with a plain `moka::sync::Cache::get` and falls
through to a distributed query on a miss, so when the 10s TTL expires every
in-flight write request on a frontend issues its own scan of the single-region
`greptime_private.pipelines` table. Concurrent scans per expiry scale with
write QPS, and every frontend's burst lands on the same datanode. A user
running high-throughput ingestion through a pipeline saw that datanode
overloaded.
Switch to `moka::future::Cache::try_get_with` so concurrent misses on the same
key share one loader. This requires a single-key lookup, so cache entries are
now keyed by the requested schema rather than the schema the pipeline is stored
under; resolving a request to a stored schema stays in the loader, which is the
authoritative path and already handles the empty-schema and multi-schema cases.
A lookup for a schema not yet cached costs one extra read, now protected from
amplification by the coalescing it enables.
`remove_cache` previously only walked the compiled-pipeline cache, so an entry
populated by `get_pipeline_str` alone (the pipeline read API) survived deletion
until it expired. It now walks all three caches.
Also make the TTL configurable as `pipeline.cache_ttl`, default unchanged at
10s. The TTL is what propagates a pipeline change to other frontends, so
raising it trades staleness for fewer reads.
Refs #9021
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(pipeline): restore cross-schema semantics broken by the new cache key
Keying cache entries by the requested schema dropped two behaviours that the
previous stored-schema key provided for free.
Creating a new version only wrote the creating request's schema, so another
schema on the same frontend kept serving its cached `latest` — an older
version — until the entry expired. Since the whole point of making the TTL
configurable is to let operators raise it, that window is not bounded by
anything useful. Creation now invalidates every schema's `latest` alias for
that name before priming the cache, leaving the version-pinned keys alone.
The failover cache lost its reach across schemas the same way: a global
pipeline (stored under the empty schema) loaded by schema A was cached under
`A`, so schema B using it for the first time while the pipeline table was down
missed and failed ingestion. The failover cache has no loader and so is not
subject to the single-key model of `try_get_with`; it keeps the stored-schema
key and the empty-schema-first resolution.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(pipeline): drop cache priming on create and fold the sweep helpers
Priming the cache on create saved one read on a low-frequency operation and
cost a concept: entries were written under the creating request's schema while
`PipelineContent.schema` said empty, so the two schemas in play disagreed.
Invalidating the `latest` aliases is required regardless — that is what makes
a new version visible to other schemas — so dropping the priming loses only
the saved read, which coalescing now protects anyway. `insert_and_compile` no
longer needs the caller's schema.
`remove_cache` and the create-time invalidation collapse into one
`invalidate(name, version)`; `None` sweeps only the `latest` aliases, which is
exactly what creation wants. That leaves `invalidate_by_suffixes` and
`cache_keys` with a single caller each, so both are inlined.
Drop the `PipelineOptions` humantime test: `load_config_test` loads both
example TOMLs, which now carry `cache_ttl = "10s"`, and would fail the same
way if the serde attribute were lost. The `toml` dev-dependency goes with it.
The two invalidation tests are now checked to be orthogonal: removing the
version suffix fails only the delete test, and sweeping just the compiled
cache fails both.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(pipeline): keep failover populated across a create
The `latest` sweep on create clears the failover cache along with the loaded
ones, and after dropping the priming there was nothing writing it back. An
outage between the create and the first read-back left neither `latest` nor the
explicit version with anything to fall back on, failing ingestion — worse than
before, since the previous version's failover entry was swept too.
Creation now goes through `PipelineCache::on_pipeline_created`, which pairs the
sweep with a failover write of the new empty-schema definition. The two must
happen together, so they live behind one method rather than at the call site.
Also commit the Cargo.lock entry for the dropped `toml` dev-dependency, and
trim the comments added over the last few commits down to what the code does
not already say.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(json2): concretize JSON2 schemas at merge scan boundaries
Infer concrete JSON2 output types from remote plans and expose them on MergeScanLogicalPlan before physical planning. Recompute affected local schemas and remove the JSON2-specific rewrite from MergeScanExec.
Add SQLness coverage for whole JSON2 columns in windows and joins.
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
feat(query): support list indexing for JSON2 columns
Extend JSON2 paths through DataFusion field-access planning, including nested list indexes and object fields following an index.
Preserve Variant reads for bracket JSONPath expressions and normalize dot accesses after subscripts to work around the current DataFusion planner limitation.
Add unit and sqlness coverage for nested indexes, type conflicts, missing paths, flushes, and compacted SSTs.
Signed-off-by: luofucong <luofc@foxmail.com>
Extend `WriteCacheUploadStoreWrapper::wrap` with the `OperationType` of the upload so implementations can apply per-operation policies (e.g. throttling compaction uploads but not flush uploads). Flush and compaction paths forward their existing `SstWriteRequest::op_type`; `put_and_upload_sst` is flush-only and index rebuild uploads are reported as compaction uploads.
Files: `src/mito2/src/cache/write_cache.rs`, `src/mito2/src/sst/index.rs`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): support exact sequence range reads
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): cover preserve row sequence table alter
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): clear preserve_row_sequence marker on copy_region_from
copy_region_from copies source FileMeta into the target region, which has an
independent sequence domain. The physical per-row sequences in the copied
file belong to the source region only; trusting them in the target would let
an exact sequence-range request replay source-domain rows as if they were
target sequences. Clear the preserve_row_sequence marker on copied files so
the target fails closed with SequenceRangeUnsupported until the scan provably
cannot intersect the copied rows.
Add a regression test: copying from a preserve-enabled source into a
preserve-enabled target clears the marker, and an exact (2, 7] request on the
target returns SequenceRangeUnsupported instead of replaying source rows.
Fixes#8865
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* style: remove redundant doc comments for exact sequence range options
Approved comment-cleanup-only changes for #8865: drop outdated doc
summaries duplicated on the exact_sequence_range wrapper and the
preserve_row_sequence field, drop pure-restatement doc comments on the
SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and
remove the four structural SQL comments from the alter_preserve_row_sequence
case. No behavior changes; .result regenerated by the sqlness runner.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): fail closed exact reads on copied files and extension ranges
Address review feedback on #8865:
- copy_region_from: clear the source-domain FileMeta::sequence along with
the preserve_row_sequence marker. An unmarked file retaining a stale
source-domain max sequence could be silently skipped by
files_allow_exact_sequence_range() as 'proven disjoint' in the target's
independent sequence domain, dropping rows on exact (C, H] reads. With
sequence=None the capability check fails closed (SequenceRangeUnsupported)
until the copied rows are provably disjoint.
- Engine/reader: reject exact sequence-range reads whenever a follower
region has an extension range provider attached. Extension streams are
returned without a row-level sequence filter, so exactness cannot be
proven; treat the capability as missing (fail closed) instead of emitting
out-of-range rows. The reader also fails closed as defense in depth.
Tests: extend copy_region_from regression to assert the copied file's
sequence hint is cleared; mito2 suite 1148/1148 passing.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* style(mito2): use doc comments for test function descriptions
Elevate the block comments describing test functions (in scan_test and
copy_region_from_test) to /// doc comments, matching the convention used
elsewhere in the exact sequence range change. No logic change.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(mito2): extract helpers and trim comment noise in exact sequence reads
PR finalization for #8865 (zero behavior change, full mito2 suite green):
- engine: extract validate_sequence_fences and
sequence_range_unsupported_reason, keeping error variants, check order
and reason strings identical; OSS binds the extension blocker to false.
- handle_copy_region: extract remap_copied_file_meta and
file_descriptors_for_meta; rename file_ids -> source_file_ids and
files_to_copy -> new_file_metas.
- compactor: rename max_input_sequence -> known_max_input_sequence,
document the None semantics (empty input vs unknown sequence).
- Remove restating/outdated comments (ScanInput::sequence_range doc
first line, outdated file-pruning note, options test restatements),
compress verbatim comments while keeping why/invariants/contracts.
Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt,
git diff --check, mito2 suite 1148/1148 passing.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): reject foreign-region SSTs in exact sequence reads
Reading an SST whose FileMeta.region_id differs from the scanned region
means the region's sequence domain is broken (manifest corruption or a
repartition/copy path that leaked a source-domain file). Treat this as
an explicit RegionSequenceDomainBroken error instead of silently
ignoring the file's sequence or falling back to a full scan: the region
is unusable for exact sequence-range reads until the foreign lineage is
compacted away or repaired.
- files_allow_exact_sequence_range / exact_sequence_range now return
Result and propagate the error through engine fence validation and
scan construction (StatusCode::Internal, distinct from the
fallback-capable SequenceRangeUnsupported).
- Row-level flat-batch sequence filtering rejects foreign-region files
as defense in depth.
- Engine test asserts the broken-domain error rather than
Unsupported/fallback.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping
An unmarked file's FileMeta.sequence may be synthesized by the
region-edit or repartition paths (committed+1 import barrier), not a
physical max of its rows. Treating it as a whole-file disjoint proof
could permanently skip rows that were never incrementally consumed
once the flow checkpoint passes that value.
Exact sequence-range capability now requires every SST in the region to
carry the preserve_row_sequence marker; any unmarked file disables
exactness (fallback), and the (C, H] file-selection skip also only
applies to marked files. Foreign-region files still raise
RegionSequenceDomainBroken as before.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): scope exact sequence-range capability to the time-selected read set
The exact capability check used to walk the entire SstVersion, so a
single unmarked or foreign-region SST anywhere in the region disabled
exact reads or raised RegionSequenceDomainBroken even when the
request's time range could never touch that file.
Both the engine fence and the scan builder now derive the read set with
shared time-pruning + exact-min/sst-min selection and validate
capability only over the files actually selected: a time-pruned file
cannot contribute a row to (C, H], so it cannot affect exactness. The
existing fail-loud semantics are unchanged for every selected file
(foreign region id -> RegionSequenceDomainBroken; unmarked -> exact
unavailable).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): wash untrusted sequences in compaction and restore barrier skipping
Compaction with any non-preserved input now writes a sequence-less
output: the physical __sequence column is zeroed (the flat format
requires the internal columns) and FileMeta.sequence records the
region-local admission barrier committed_sequence + 1 (falling back to
the flushed frontier). preserve_row_sequence stays false.
Exact sequence-range scans interpret an unmarked file's sequence as an
admission barrier: barrier <= C means flow has already consumed the
whole file, so it is skipped at file level; a missing or newer barrier
fails closed. Foreign-region files stay in the selected read set so the
capability fence still raises RegionSequenceDomainBroken.
This closes the recovery loop: after a region repartition, one
time-scoped fallback consumes the migrated rows, then compaction washes
the untrusted per-row sequences away and exact incremental reads resume
via file-level barrier skipping.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore(mito2): drop restating comments in known_max_input_sequence tests
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): trim SQLness result EOF whitespace
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(mito2): reuse exact scan file selection
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): strengthen sequence scan coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* style(mito2): trim ALTER option comments
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): remove no-op bulk compaction check
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): preserve trusted row sequences when reading SSTs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): preserve target sequence domain for imported SSTs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): add trailing blank line to SQLness result EOF
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore(mito2): trim exact sequence scan plumbing
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(mito2): fold exact SST selection checks
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): make legacy compaction rewrite deterministic
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): make PK compaction rewrite deterministic
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
feat(query): support JSON2 paths in SQL functions
Update the DataFusion fork to expose scalar function planning hooks.
Infer JSON2 path output types from scalar, aggregate, and window function signatures, while preserving the default Utf8View behavior for functions that accept arbitrary inputs.
Add unit and sqlness coverage for type conflicts, mixed typed and untyped JSON paths, filters, aggregates, and window functions.
Signed-off-by: luofucong <luofc@foxmail.com>
* feat(runtime): add weighted workload scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics
Use the GreptimeTeam/catio fork (pinned c20eafc) which adds
ClassStats::total_admission_wait and ClassStats::admitted, recorded
at each QUEUED -> ADMITTED transition. This exposes the scheduler's
own admission delay (excluding Tokio queueing and poll execution),
enabling admission-wait based fairness gates.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to dynamic-config revision
Bump the catio scheduler fork to 9f4b028 which adds
Scheduler::set_weight and Scheduler::set_max_concurrent_polls for
runtime configuration.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(perf): runtime-adjustable workload scheduler parameters
Expose dynamic adjustment of the experimental workload scheduler at
runtime:
- common-runtime: set_workload_scheduler_weights and
set_workload_scheduler_max_concurrent_polls, which forward to the
catio scheduler's set_weight/set_max_concurrent_polls when the
scheduler is enabled and reject zero values.
- servers: /debug/workload_scheduler/weights and
/debug/workload_scheduler/max_concurrent_polls POST handlers, so
operators can rebalance query/write shares or admission concurrency
without restarting the datanode.
Both endpoints return 400 with a clear reason when the scheduler is
disabled or the requested value is invalid.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(perf): add GET /debug/workload_scheduler status endpoint
Returns the current weights (per class), max_concurrent_polls,
active_polls and per-class counters (queued, tasks, wakes, polls,
completed, cancelled, admitted, total_admission_wait) as JSON. When the
scheduler is disabled, returns enabled=false with the other fields
omitted, so operators can distinguish 'disabled' from an error.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to time-accounting revision
Bump the catio scheduler fork to 257ba56 which replaces
admission-count accounting with real execution-time accounting
(pass += exec_time / (weight * concurrency)), so CPU share follows the
configured weights regardless of poll length.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: bump catio to lock-free sampling revision
Bump the catio scheduler fork to efdc0a4 which adds an optional
downsampled clock sampling mode (SchedulerBuilder::sample_every_polls,
default off) with a lock-free per-class atomic counter, so the
downsampled path costs one fetch_add per poll instead of a global
mutex.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin catio to scheduler PR head
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): add scheduler bypass control
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: advance catio scheduler fixes
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin merged catio scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: regenerate config docs for workload scheduler
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin catio scheduler test fix
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(http): satisfy scheduler lifecycle clippy
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: add distributed scheduler toggle coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat: finalize workload scheduler runtime controls
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: pin merged catio atomic weights
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore: preserve unrelated lockfile resolution
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(runtime): downsample scheduler time accounting
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(runtime): verify cross-runtime scheduler progress
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* feat(runtime): configure scheduler poll sampling
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(runtime): clarify scheduler activation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(runtime): explain scheduler use case
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
The k8s.pod runs_on k8s.node rule can only fire from kube_pod_info: it is
the only table declaring both endpoints, and co-declared edges require
both on the same row. A deployment sending only OTLP has no
kube-state-metrics tables, so its node layer is invisible and that rule
has no source at all, even though the resource attributes carry
k8s.node.name.
Declare k8s.node in otlp_trace_entities and in the synthesized resource
descriptor, and project k8s.node.name in the descriptor writer so the
column the declaration needs exists. Identity is the name rather than
k8s.node.uid: kube-state-metrics carries no node UID, so the name is the
only identity both sources can agree on.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: allow widening the time index column's timestamp unit via ALTER TABLE ... MODIFY COLUMN
Previously MODIFY COLUMN rejected the time index column outright. Now the
time index unit can be widened (Second -> Milli -> Micro -> Nano), which is
lossless for data that fits the target unit: historical data in old SSTs is
cast to the new unit on read by the existing schema-compat layer, and
compaction rewrites it lazily. Narrowing and non-timestamp targets remain
rejected; tag columns keep being rejected. Widening is rejected if any
SST's time range would overflow the target unit's i64 range (e.g.
millisecond -> nanosecond beyond year 2262), since the cast would silently
null those values.
Read-path correctness for old-unit SSTs (verified by new engine e2e tests
and sqlness WHERE queries):
- row-group min/max pruning: parquet statistics of a timestamp column are
raw integers in the file's unit; when the region metadata's type differs
(also the case for altered field columns), stats are now interpreted in
the file's type and converted to the expected type before pruning.
Without this, a new-unit predicate silently pruned whole row groups of
old-unit files (wrong results, rows missing).
- SST-level simple filters are skipped for columns whose file type differs
from the expected type; the predicate is applied by the query layer's
residual filter above the region scan. Also drop the stale
"timestamp columns cannot change type" debug_assert.
- retry idempotency: a same-type ModifyColumnType on the time index
validates as a no-op and `need_alter` returns false, so a retried alter
procedure (region already altered before the previous attempt failed)
converges instead of aborting forever.
- add TimeUnit ordering and ConcreteDataType::is_timestamp_unit_widening_to
- relax ModifyColumnType validation in store-api and table metadata
- tests: unit tests in datatypes/store-api/table; mito2 engine e2e tests
(flushed SST + new writes + reopen + retry + overflow + predicate scans,
cross-unit dedup, mixed-unit SSTs, compaction); sqlness cases incl.
partitioned table and WHERE filters over old-unit data
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: resolve parquet filter issue
* test: provide sqlness tests
* refactor: drop trivial test cases and shorten comments
Review pass over the branch's additions:
- datatypes: keep a representative subset of the widening-matrix asserts
- table: collapse the three single-branch rejection blocks into one loop
- mito2: drop the boundary gt_eq and post-compaction predicate asserts
(covered by the exact-filter regression test and sqlness); drop the
engine-level gt_eq/lt_eq casts (full operator matrix stays in the
cast_timestamp_unit unit tests)
- sqlness: drop a bare full scan already covered by the filter above it
- shorten function doc comments across datatypes/store-api/table/mito2/
recordbatch to the essential semantics
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: drop physical prefilter for columns whose file type differs
Follow-up to the review feedback on CompatBatch/prune reader/filter
handling for widened time index units.
Between/InList/IsNull predicates are prefiltered by PhysicalFilterContext,
which builds its physical expression against the FILE's schema while the
predicate literals are in the expected (post-alter) unit. Evaluating them
against an old-unit SST raised a cross-unit comparison error (Timestamp(ms)
>= Timestamp(µs)) that failed the whole scan. Physical prefilter predicates
are best-effort pruning hints (the query layer re-applies them above the
scan), so drop the prefilter when the column's file type differs from the
expected type, mirroring the simple-filter strategy.
Verified: Between and a non-rewritten (large) InList on old-unit data no
longer error and filter exactly end-to-end (sqlness), and no matching rows
are lost at the engine level (engine test).
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: add direct unit tests for stats cast and prefilter drop
The two-step stats cast (reinterpret raw Int64 stats in the file's
timestamp type, then rescale to the expected type) and the physical
prefilter drop on file/expected type mismatch were only covered
end-to-end; add localized unit tests so a regression fails at the
exact site:
- stats.rs (previously no tests): RowGroupPruningStats min/max over a
hand-built RowGroupMetaData — passthrough with no expected metadata,
passthrough on same type, and rescale (1000ms -> 1_000_000us, not
1000us) on a widened expected unit
- reader.rs: PhysicalFilterContext::new_opt keeps a Between prefilter
when file and expected types match and drops it on unit mismatch
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: tolerate mixed time units range cache key coverage check
* docs: flag mixed-unit hazard in the (unwired) series index
The series index stores per-series min/max ts as raw Int64 in the unit
of the region metadata at write time, and the searcher builds its range
predicates from a single per-region metadata. After a time index unit
widen, files of one region would carry mixed units, so a per-file unit
(or an index rebuild on such alters) is required before this index is
wired into scans. Leave notes at both sites.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: cover mixed-unit compaction for sparse encoding and strict windows
Compaction-path audit follow-up. The compat cast and window math were
already covered for dense regions; add the two remaining e2e scenarios:
- sparse primary key encoding (used by metric-engine physical regions):
widening then compacting mixed-unit files rewrites the old-unit time
index correctly through the sparse compaction compat path
- strict-window manual compaction: each window output trims rows with a
predicate built in the region's new unit against an old-unit file;
every instant must survive exactly once (no loss, no cross-window
duplication), rescaled
Also documents the audit finding that Regular ranged (manual)
compaction never trims rows: TwcsPicker sets output_time_range to None
and the request time range only selects candidate windows.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: cover time index unit change in FlatCompatBatch directly
The compat layer's rescaling of a widened time index was only verified
end-to-end; add direct unit tests for both paths:
- dense: identical units skip compat entirely; a widened unit rescales
the time index column (1000ms -> 1_000_000us, not reinterpreted) while
other columns pass through and the output schema matches the expected
metadata
- compact sparse (the metric-engine compaction path): same rescaling
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor: move timestamp unit division into common-time
The exact unit division (UnitQuotient + div_mod_units) is time
semantics, not filter logic; move it next to TimeUnit in common-time
with a compact test covering representable/non-representable values,
negative (floor) instants, and quotient overflow. The ScalarValue
helpers stay in filter.rs since common-time has no datafusion
dependency.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: compat rescales a widened time index and fills an added column together
The realistic multi-alter sequence (widen at T1, add column at T2,
read a T0 SST) exercises cast and default-fill in the same
compute_index_and_fields pass; assert both in one output batch.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: preflight time index widening overflow before any region alters
Address review feedback on the overflow guard:
- preflight: when the frontend operator receives a widening alter on the
time index, run an existence scan (ts outside the target unit's i64
range, LIMIT 1, via the query engine so it covers every region of the
table in both standalone and distributed modes) BEFORE any DDL task is
submitted. A region that fits can no longer commit the new schema
while another region rejects the alter with a non-retryable error.
File and row-group pruning keep the scan cheap when nothing overflows.
The per-region check in mito2 stays as the final guard for data
written after the preflight (the remaining race window); without a
validate-only wire field (region.proto lives in the external
greptime-proto repo) a fully atomic two-phase validate/commit is out
of scope here.
- fast path: cast_timestamp_unit returns the filter unchanged when the
literal is already in the target unit, skipping the div-mod rebuild.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
* fix: address auto review comments
* fix: remove time index widening overflow preflight
Overflow needs timestamps beyond the target unit's i64 range (~year
2262 for nanoseconds), which real workloads never write, so the two
existence scans before every widening alter are not worth the cost.
Region validation already rejects the alter when an SST's time range
overflows the target unit; it now logs the rejection (with the
offending file) and returns a deterministic client-facing message.
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: make sqlness test stable
* fix: log instead of rejecting time index widening overflow
Overflowing values cast to NULL on read but do not otherwise affect
reads or writes, so the alter is allowed; the region-level check now
only logs (with the offending file) when an SST's time range exceeds
the target unit's i64 range.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Add `WriteCacheUploadStoreWrapper` in `src/mito2/src/cache/write_cache.rs` and wire it through `src/mito2/src/worker.rs`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(semantic-graph): let the generic container yield to k8s.container
A container inside a Kubernetes pod reaches the graph twice: as the
k8s.container entity kube-state-metrics describes, identified by
[pod uid, container name], and as the generic container the OTel resource
attributes describe, identified by container.id. One physical container,
two nodes.
Kubernetes is the primary scenario, so k8s.container keeps its identity
and the generic type stands down where it applies. Conventions gain a
row-level condition for that: `suppressed_by` withdraws a declaration on
rows where any of the named columns has a value. The test has to be per
row, not per table — one descriptor table holds both pod rows and
bare-runtime rows.
Every branch that turns a declaration into rows now shares one guard
(`declaration_predicate`), so the condition cannot apply to entities but
not to the edges they carry.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(semantic-graph): report derived entity declarations in table_semantics
information_schema.table_semantics only read table options, so the
declarations the built-in conventions derive — for trace tables, for
whitelisted Prometheus and OTel descriptor metrics — were invisible.
"Why is my table not in the graph?" was answerable only from debug logs,
which is not a self-service path.
A new `entity_declarations` column reports the entities a table actually
contributes: each one's identity, whether it came from an option or from
the conventions, and any row-level condition attached to it. An expected
entity missing from the list is the answer — the table name is not
whitelisted, the source stamp is wrong, an id column is absent.
The row filter widens to match: a table that declares nothing by option
but derives entities by convention now appears, since it is in the graph
and the view has to say so. The provider reaches the derivation through a
new metadata-only trait method, keeping catalog below operator.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(semantic-graph): never yield a container to an entity nothing derives
The generic container withdrew on any row carrying a pod UID, on the
assumption that k8s.container would cover it. Nothing guaranteed that:
k8s.container came only from kube-state-metrics, so an OTLP-only
deployment — or one whose KSM data had expired or fell outside the query
window — lost the container node and its edges entirely instead of
gaining a more specific one.
The rule now names the superseding type rather than a trigger column, and
withdraws only where that type's full identity is on the row. Both OTel
sources declare k8s.container themselves, under the identity
kube-state-metrics gives it ([pod uid, container name]), so the two
sources name one node; `k8s.container.name` joins the descriptor's
projected attributes to carry it.
Resolution runs once every declaration for the table is known, so a
superseding type that ends up undeclared — its columns are gone, or an
explicit declaration of it was skipped — leaves the guard empty and the
generic container standing. A container can change type; it cannot
disappear.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(semantic-graph): build the declaration JSON by serializing a type
The column was assembled entry by entry into a serde_json::Map, cloning
every value and allocating a String per key. A Serialize struct that
consumes the declaration moves the same data instead, and the field order
now reads type, origin, identity, then description.
The scan path around it was doing the same kind of avoidable work:
declarations were derived before the predicate could discard the table,
the supersession pass cloned every declaration's identity to look up one,
and the option parse claimed the time index for tables that turn out to
declare nothing.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(semantic-graph): keep the structured identity for single-column ids
`entity_id_attrs` was NULL whenever the identity came from one column, so a
consumer holding `host` = `a3f2...` had no way to tell which column produced
it, and no way back to the source table. Most identities are single-column —
host, k8s.pod, k8s.node, service — so the common case was the opaque one.
Build the JSON object unconditionally. Entity equality still reads `entity_id`
alone, so this changes no merging: it records which attributes the id was
assembled from, beside an id that deliberately omits them.
Also corrects the `entity_id` column doc, which still described the `k=v,k=v`
rendering replaced in #8904 by values joined in declared order.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(semantic-graph): keep what a container carried when it yields
Yielding to k8s.container cost the row two things it had as a generic
container. The runtime container id, which kube_pod_container_info keeps
descriptive precisely because it is the handle back to runtime logs and
metrics, went missing entirely: neither an id nor an attribute of any
node. And the edge vocabulary knew only the generic type, so a row with
the more complete labels ended up with fewer connections than one
without — the container layer no longer reached its host.
Both OTel sources now keep the runtime id and name descriptive on
k8s.container, and the vocabulary gains the two edges that mirror the
generic type's. Nothing checks a supersession against the edge
vocabulary, so that requirement is written down where the rule is.
Also: the conventions-failure path now reports the explicit half as its
comment always claimed, entity_declarations reports scope columns, and
identifies() is private again now that only declaration_predicate calls
it. The two RFCs catch up with entity_id_attrs being unconditional, the
view listing convention-derived tables, and supersession existing.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(semantic-graph): report unmatched clients and the longest request
`real_wins` pins a `calls` edge's RED metrics to the observed span pairs:
when a window's edge key holds any pair, the unmatched client spans are
suppressed so one (window, edge) yields one row. That leaves no way to
tell a callee that stopped responding from traffic that stopped arriving —
both show up as a lower request_count.
Add two columns to `semantic_relationships`:
- `unmatched_count` — client spans with no server span, counted outside
`real_wins` on the same row, so the suppressed population stays visible
without splitting the edge into two rows. NULL for agent calls, whose
inner join leaves nothing unmatched, and for declared edges.
- `duration_max` — the longest single request. It goes through
`real_wins`: a pair is timed by the server span while an unmatched
client is timed by its own (network wait included), so mixing them would
make the max describe a different population than duration_sum and
duration_count. Agent calls compute it from the child spans they already
aggregate.
The projection contract goes from 16 to 18 columns; every branch projects
both. Explicit column queries are unaffected, `SELECT *` and
ordinal-based readers see the new shape.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(semantic-graph): drop the duplicated source-gate case
The whitelist gate on `source=opentelemetry` is already covered by
`otel_implicit_declarations_are_gated` and by the wrong-source case in
`table_semantics`; here it only paid for another table create, insert and
full graph derivation.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* style(semantic-graph): trim the comments back to what the code cannot say
Several comments restated the code, repeated a rationale already stated at
the type or in the RFC, or explained a test in more words than the test
body.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: postgres describe for more statements
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: cover more show statements
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments
- add missing `clippy::too_many_arguments` allow on
`query_from_information_schema_dataframe` (CI clippy failure)
- take `&ShowKind` in the information-schema dataframe helper so `kind`
is no longer cloned at every call site; only the WHERE arm (which needs
an owned expression for `sql_to_expr`) clones internally
- document why re-applying TQL explain formats never overwrites an
existing value (per-query context state)
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: trim comments to essentials
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: Add built-in daemon mode for standalone service
Add an optional --daemon/-d flag to `greptimedb standalone start` to run the server as a background daemon detached from the shell session, similar to Redis `daemonize yes`.
- Use the daemonize crate (Unix only) to fork/setsid and detach from the controlling terminal before the Tokio runtime is built.
- stdin/stdout/stderr are redirected to /dev/null; logs continue to go to data_home/logs/ via the existing tracing appender.
- On non-Unix platforms, --daemon is a no-op with a warning (runs in foreground).
Closes#7314
Signed-off-by: tian1220A <1573612521@qq.com>
* test(cmd): add integration test for standalone daemon mode
Verify that standalone start --daemon detaches from the shell and brings up the HTTP listener, guarding against regressions where the daemon blocks in the foreground or crashes after forking.
Signed-off-by: tian1220A <1573612521@qq.com>
* fix(cmd): gate --daemon flag with #[cfg(unix)] and fix rustfmt
Address review feedback to not expose --daemon/-d on non-Unix platforms. The daemon field is now gated with #[cfg(unix)], and a cross-platform is_daemon() accessor returns false on non-Unix so maybe_daemonize() stays unconditional.
Also fix the rustfmt formatting issues that caused fmt-check to fail.
Signed-off-by: tian1220A <1573612521@qq.com>
* chore(cmd): declare daemonize dependency directly in cmd
daemonize is only used by the cmd crate. Move it out of the workspace
dependencies and declare it directly in the unix-only dependencies of
src/cmd/Cargo.toml.
Signed-off-by: tian1220A <1573612521@qq.com>
---------
Signed-off-by: tian1220A <1573612521@qq.com>
* refactor(mito2): prioritize file count in TWCS picker
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): migrate TWCS tests to async picker API
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): remove legacy reduce_runs and merge_seq_files pickers
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): balance TWCS picks by file group
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): treat multi-SST groups as barriers in TWCS pick_count_first
Previously pick_count_first filtered out multi-SST file groups and could
pick singleton groups across them in one interval. Now multi-SST groups
split the candidates into independent segments, so a single pick never
crosses a multi-SST group.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): remove redundant overlaps_files_left_behind check
The rebase onto main kept both the upstream fix (#8872) and the branch's
selected_overlaps_unselected check for the same deletion-marker problem.
The upstream check is fully subsumed: files_to_merge only differs from
the window files in append mode, where filter_deleted is already false,
and selected_overlaps_unselected treats partially-selected groups as
unselected, making it strictly stronger.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): pre-filter by selection time span in selected_overlaps_unselected
A window group outside the overall time span of the selected groups
cannot overlap any of them, so skip the precise overlap check (and the
per-group file id set lookup) for it. In typical TWCS windows the pick
is clustered in one segment, so most unselected groups are rejected in
O(1).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): remove FileGroup abstraction from TWCS picker
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): drain compactable backlog after successful compaction
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): score TWCS candidates by predicted progress
Redefine what a pick candidate is worth: an interval is only eligible if
compacting it makes progress on at least one axis — it reduces the
physical file count given the max_output_file_size split threshold
(predicted output = ceil(input bytes / threshold)), or it resolves at
least one overlap between sorted runs. A pure rewrite that achieves
neither (e.g. 32 large balanced files whose output would split back into
just as many SSTs) is skipped instead of burning I/O.
The candidate metrics are accumulated in a Candidate struct as the
interval expands, and the score ranks by predicted file reduction, then
overlap participants, then smaller input bytes. With no output size
limit the behavior degenerates to the previous count-first rule.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): make TWCS input limit configurable
Allow operators to tune the maximum SST inputs through a hidden environment variable while keeping 32 as the validated default.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): apply TWCS trigger at window level
Check the total physical SST count before candidate selection so trigger values above the per-task input limit still allow bounded compactions.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Provide valid distributed frontend options in `src/cmd/src/frontend.rs` so enterprise plugin setup can preserve prefilled heartbeat extensions without weakening production validation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mysql): interpret prepared statement datetime params in session timezone
Binary DATETIME parameters of server-side prepared statements were
converted as if UTC, ignoring the session timezone set via SET time_zone.
Convert them with the session timezone and add an integration test
covering prepared inserts and predicates under Asia/Shanghai.
Signed-off-by: wy471x <wy471x@gmail.com>
* refactor: share naive datetime timezone policy via common-time
Address review feedback on the prepared-statement timezone fix:
- Expose Timestamp::from_naive_datetime in common-time so the DST policy
(gap -> error, ambiguous -> earlier instant) lives in one place, shared
by the text protocol (Timestamp::from_str) and the MySQL binary protocol.
- Route the MySQL prepared-statement datetime conversion through it.
- Match the target type before converting datetime params so
PreparedStmtTypeMismatch fails fast without wasted conversion.
- Use the short Timezone import form for consistency with the rest of servers.
Signed-off-by: wy471x <wy471x@gmail.com>
---------
Signed-off-by: wy471x <wy471x@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Add a typed `Command::build_with_heartbeat_extensions` seam in `src/cmd/src/frontend.rs`.
Freeze and harden `FrontendHeartbeatExtensions` in `src/frontend/src/heartbeat.rs`, with lifecycle and race coverage in `src/frontend/src/heartbeat/tests.rs`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
liyang
Weny Xu
LFC
Ning Sun
Yingwen
dennis zhuang
discord9
shuiyisong
Lei, HUANG
fys
localhost
zhaiAohan
Dhruv Vaishnav
wy471xandNing Sun
sun