When a compaction finishes with no pending request/DDL, on_compaction_finished
leaves the region's CompactionStatus in the map with no active compaction. If
the worker then skips schedule_next_compaction because min_compaction_interval
has not passed, the stale status is never cleaned up: all subsequent compaction
requests are swallowed by the "status exists" branch (regular waiters hang,
manual StrictWindow requests pend forever) and the region never compacts again
until restart.
Add CompactionScheduler::remove_inactive_status() and call it from
handle_compaction_finished when the interval gate skips scheduling the next
compaction, restoring the invariant that a status in the map always has an
active compaction.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Parse PostgreSQL DSNs (URL or libpq keyword) with tokio_postgres::Config —
the backend's own parser — and log its Debug, which redacts the password.
This matches the backend grammar exactly (multi-host URIs, backslash
escapes, any Unicode whitespace, percent-encoded query keys, and '&'/';'/
'://' inside values) rather than approximating it by hand. Other URLs are
redacted via the url crate; a best-effort keyword fallback covers inputs
neither parser accepts.
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
* docs: add entity relationships and graph query RFC
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: revise entity-graph RFC after review
- Zero-configuration declarations: the Prometheus-on-Kubernetes convention
pack (job/instance per the OTel-Prometheus compatibility spec, k8s SD
labels, *_info descriptors, target_info enrichment) alongside the OTLP
trace auto-stamp, plus Remote Write 2.0 inline metadata.
- Calls endpoints follow the service entity declaration; self-calls
compare full endpoint ids; no silent identity fallback.
- Strict time-window contract: the source window is never narrower than
the query's observed_at range; unsafe-to-extract predicates error
instead of silently defaulting.
- scope removed from the relationship schema (kept on entities as a
display property); entity row contract restated per projected
observation; endpoint encoding documented as the v1 storage-level key
with its known collision limitation.
- Sampling caveats corrected (ratios are representative only under
unbiased sampling); snapshot relation synthesizes endpoint-only
vertices; shared attributes provide join keys while co-declaration
provides relationship semantics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: align entity-graph RFC contracts and tighten prose
- Metric/trace service unification is promised only when service.namespace
is empty and job is not relabeled (the compatibility spec renders job as
<namespace>/<name>); otherwise alignment needs pipeline normalization or
explicit declarations.
- Entity row contract stated once (per projected observation); the calls
defining SQL is marked as the single-column simplification of the
declaration-derived endpoint ids; Remote Write 2.0 metadata intake and
the Prometheus implicit declarations are listed as M1 work.
- Compress survey/example/reference prose.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: pin down the k8s convention pack rules and edge directions
- kube_pod_owner implicitly declares k8s.workload with id
(namespace, owner_kind, owner_name) and derives pod part_of workload;
target_info's non-job/instance labels are implicit service descriptive
columns — fixed rules, no new declaration syntax.
- One direction for pod placement: k8s.pod runs_on k8s.node (pod added to
runs_on sources; node->pod removed from contains).
- Drop the remaining 'canonical' wording for the v1 storage-level id.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: make contains/part_of a true inverse pair
part_of covers service.instance->service and k8s.pod->k8s.workload with
contains as its inverse; has_instance is dropped.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: rewrite entity-graph RFC as a design narrative
Restructure for human review: mainline first, cases illustrate the
design instead of specifying it.
- New Architecture and Benefits and Drawbacks sections; the calls
derivation stays expanded as the flagship example while schema
enumerations, window-rule listings, and executor edge-case handling
move out of the document.
- The cross-signal promise is stated honestly: neighbours and their
source tables are discovered first, their telemetry is the next
query — one engine, two statements; the worked example shows the
full declaration -> entity -> edge -> telemetry flow.
- Default materialisation added as the most direct alternative to
read-time derivation, with its costs.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: final wording pass on the entity-graph RFC
Mark the property-graph DDL as illustrative rather than settled M2
syntax, credit standards as foundations rather than claiming wholesale
alignment, and clean up punctuation-heavy prose.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: keep the RFC at design altitude
Demote the convention-pack rule details and snapshot property-merge
semantics to the implementing changes; correct the single-trace-table
assumption (traces can be routed to multiple tables); record
attribute-key participation in entity equality as an open question.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: correct service graph terminology
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(flow): handle time_ranges in DirtyWindowRequest
Bump greptime-proto to include the new `time_ranges` field on
DirtyWindowRequest (GreptimeTeam/greptime-proto#330) and mark the
corresponding aligned time windows as dirty in the batching engine,
in addition to the existing per-timestamp dirty marking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): fix doc comment spacing in align_time_window
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time_ranges in handle_mark_dirty_time_window
Verify a valid [start_inclusive, end_exclusive) range is aligned to
time window boundaries and stored with an explicit end, and that empty
or reversed ranges are skipped.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): union merged dirty windows with the larger end
Merging a bounded dirty range with a window contained in it (e.g.
[0s, 15s) with nested [5s, 10s), or an unbounded dirty window inside a
bounded range) previously assigned the contained window's upper bound,
shrinking the merged window and permanently dropping the tail range
from re-computation. Keep max(prev_upper, cur_upper) instead.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): clip bounded dirty ranges at the expire bound
merge_dirty_time_windows dropped every window whose start is before
expire_lower_bound, so a bounded dirty range crossing the bound (e.g.
[0s, 15s) with expire 10s) lost its still-live suffix [10s, 15s). Now
bounded ranges are dropped only when their end is at/before the expire
bound, and crossing ranges are clipped to the bound (which the caller
aligns to the time window boundary). Unbounded windows keep the
existing start-based behavior.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): fall back to full dirty on dirty-window alignment failure
An eval/alignment error previously aborted the per-task dirty-marking
closure, losing every dirty timestamp and range accumulated for that
task, while the RPC still returned Ok so the producer would not retry.
On alignment failure now log a warning and mark the whole task dirty
(set_dirty) instead, so the affected data is conservatively
recomputed.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): apply rustfmt to new dirty-window merge tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time index units in dirty window marking
Document that DirtyWindowRequest timestamps/time_ranges are bare i64s
interpreted in the source table's time index native unit, and add a
test expressing the same [3s, 11s) range in second/millisecond/
microsecond/nanosecond units across four tables, asserting all align
to the same dirty window [0s, 15s).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to 8127f179
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): return dirty-window alignment errors to callers
Do not acknowledge a DirtyWindowRequest when a time-windowed task cannot
align a timestamp or range. The previous conservative fallback used
set_dirty(), but that marker only represents a single epoch-start window
for time-windowed flows, so it could still lose the affected dirty
range. Propagate task errors through the join loop instead so producers
can retry.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: update proto to commits on main
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: add field id and extension type to histogram
* chore: revert histogram check
* chore: lint
* test: add test coverage for maybe_update_schema
* feat: calculate sub field id from parent column id
* refactor(mito2): guard histogram sub-field ids and cover parquet footer
Address review on #8528:
- native_histogram: derive sub-field ids with checked arithmetic. The SST
writer now surfaces a new InvalidNativeHistogramSubfield error when a
sub-field id cannot be resolved (unknown name or i32 overflow) instead of
silently dropping the id or wrapping. maybe_wrap_schema is now fallible.
- sst: add parquet writer/footer round-trip tests asserting the
greptime.histogram extension and nested PARQUET:field_id (incl. list
elements) survive on disk, and a non-canonical struct is left untouched.
- sst: fix a broken rustdoc link to stamp_native_histogram_subfield_ids.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: return error when fail to get i32 column id
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
shuiyisong
yimeng
Lei, HUANG
Weny Xu
discord9
jeremyhi
Lanqing Yang
Ning Sun
fys
Yingwen
raphaelroshan
dennis zhuang
LFC
dependabot[bot]