When a compaction finishes with no pending request/DDL, on_compaction_finished
leaves the region's CompactionStatus in the map with no active compaction. If
the worker then skips schedule_next_compaction because min_compaction_interval
has not passed, the stale status is never cleaned up: all subsequent compaction
requests are swallowed by the "status exists" branch (regular waiters hang,
manual StrictWindow requests pend forever) and the region never compacts again
until restart.
Add CompactionScheduler::remove_inactive_status() and call it from
handle_compaction_finished when the interval gate skips scheduling the next
compaction, restoring the invariant that a status in the map always has an
active compaction.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
When a compaction finishes but its execution no longer matches the
current one, the region may have been reopened or truncated, or the
compaction was superseded. Reporting RegionClosed to waiters is
misleading; introduce a neutral StaleCompactionExecution error (same
Cancelled status code) for this case.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
schedule_compaction, handle_pending_compaction_request and
schedule_next_compaction no longer await anything after compaction
planning became fire-and-forget. Drop the async signature to make the
no-suspension-point invariant explicit: these methods always run to
completion on the worker loop without reentrancy.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
When a compaction finished within min_compaction_interval with no
pending requests, on_compaction_finished left an idle status
(phase = None) behind. The worker then skipped chaining the next
compaction due to the interval gate, and the leftover status made
schedule_compaction swallow all future triggers of the region: regular
waiters were queued but never woken, and manual requests stayed pending
forever. The region stopped compacting until close/drop/truncate.
Add CompactionScheduler::remove_idle_status and call it from
handle_compaction_finished when the interval has not elapsed and no
chained planning is scheduled. The chain-until-no-plan semantics for
compactions that outlast the interval is preserved.
Also drops an unused import left by the previous commit.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
DefaultNotifier::notify decremented INFLIGHT_COMPACTION_COUNT before the
re-entrancy guard, so a duplicate notify (which should never happen, but
the guard exists to defend against it) would decrement the gauge an
extra time and let it drift negative. Move the decrement after the
guard, matching the local compaction path's guard-then-account order.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
The worker only leaves the picking phase after receiving the
CompactionPickFinished notification. Previously the planning task was
spawned fire-and-forget: if it panicked before sending the notification,
the region would be stuck in the picking phase forever, blocking all
future compactions and pending DDLs (e.g. entering staging) of the
region.
Wrap the planning future with catch_unwind so a panic is converted into
a CompactionPlanningResult::Error and the notification is always sent,
letting the worker run the normal error cleanup path.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Parse PostgreSQL DSNs (URL or libpq keyword) with tokio_postgres::Config —
the backend's own parser — and log its Debug, which redacts the password.
This matches the backend grammar exactly (multi-host URIs, backslash
escapes, any Unicode whitespace, percent-encoded query keys, and '&'/';'/
'://' inside values) rather than approximating it by hand. Other URLs are
redacted via the url crate; a best-effort keyword fallback covers inputs
neither parser accepts.
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
* docs: add entity relationships and graph query RFC
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: revise entity-graph RFC after review
- Zero-configuration declarations: the Prometheus-on-Kubernetes convention
pack (job/instance per the OTel-Prometheus compatibility spec, k8s SD
labels, *_info descriptors, target_info enrichment) alongside the OTLP
trace auto-stamp, plus Remote Write 2.0 inline metadata.
- Calls endpoints follow the service entity declaration; self-calls
compare full endpoint ids; no silent identity fallback.
- Strict time-window contract: the source window is never narrower than
the query's observed_at range; unsafe-to-extract predicates error
instead of silently defaulting.
- scope removed from the relationship schema (kept on entities as a
display property); entity row contract restated per projected
observation; endpoint encoding documented as the v1 storage-level key
with its known collision limitation.
- Sampling caveats corrected (ratios are representative only under
unbiased sampling); snapshot relation synthesizes endpoint-only
vertices; shared attributes provide join keys while co-declaration
provides relationship semantics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: align entity-graph RFC contracts and tighten prose
- Metric/trace service unification is promised only when service.namespace
is empty and job is not relabeled (the compatibility spec renders job as
<namespace>/<name>); otherwise alignment needs pipeline normalization or
explicit declarations.
- Entity row contract stated once (per projected observation); the calls
defining SQL is marked as the single-column simplification of the
declaration-derived endpoint ids; Remote Write 2.0 metadata intake and
the Prometheus implicit declarations are listed as M1 work.
- Compress survey/example/reference prose.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: pin down the k8s convention pack rules and edge directions
- kube_pod_owner implicitly declares k8s.workload with id
(namespace, owner_kind, owner_name) and derives pod part_of workload;
target_info's non-job/instance labels are implicit service descriptive
columns — fixed rules, no new declaration syntax.
- One direction for pod placement: k8s.pod runs_on k8s.node (pod added to
runs_on sources; node->pod removed from contains).
- Drop the remaining 'canonical' wording for the v1 storage-level id.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: make contains/part_of a true inverse pair
part_of covers service.instance->service and k8s.pod->k8s.workload with
contains as its inverse; has_instance is dropped.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: rewrite entity-graph RFC as a design narrative
Restructure for human review: mainline first, cases illustrate the
design instead of specifying it.
- New Architecture and Benefits and Drawbacks sections; the calls
derivation stays expanded as the flagship example while schema
enumerations, window-rule listings, and executor edge-case handling
move out of the document.
- The cross-signal promise is stated honestly: neighbours and their
source tables are discovered first, their telemetry is the next
query — one engine, two statements; the worked example shows the
full declaration -> entity -> edge -> telemetry flow.
- Default materialisation added as the most direct alternative to
read-time derivation, with its costs.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: final wording pass on the entity-graph RFC
Mark the property-graph DDL as illustrative rather than settled M2
syntax, credit standards as foundations rather than claiming wholesale
alignment, and clean up punctuation-heavy prose.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: keep the RFC at design altitude
Demote the convention-pack rule details and snapshot property-merge
semantics to the implementing changes; correct the single-trace-table
assumption (traces can be routed to multiple tables); record
attribute-key participation in entity equality as an open question.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: correct service graph terminology
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Move the compaction picker out of the region worker's critical path by
dispatching planning to a background task and reporting the result back
via CompactionPickFinished. CompactionStatus now tracks an explicit
picking phase keyed by a monotonic plan id, so stale planning results
are rejected and duplicate regular triggers coalesce while picking.
Before submitting a prepared compaction, the picker output is refreshed
against the current SST version (file handles are re-resolved and
conflicts roll back reservations), ensuring the plan still matches live
state. CompactionExecution identifies the running task by
(plan id, kind, version control) so finish/cancel/fail notifications
from outdated executions are ignored.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(flow): handle time_ranges in DirtyWindowRequest
Bump greptime-proto to include the new `time_ranges` field on
DirtyWindowRequest (GreptimeTeam/greptime-proto#330) and mark the
corresponding aligned time windows as dirty in the batching engine,
in addition to the existing per-timestamp dirty marking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): fix doc comment spacing in align_time_window
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time_ranges in handle_mark_dirty_time_window
Verify a valid [start_inclusive, end_exclusive) range is aligned to
time window boundaries and stored with an explicit end, and that empty
or reversed ranges are skipped.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): union merged dirty windows with the larger end
Merging a bounded dirty range with a window contained in it (e.g.
[0s, 15s) with nested [5s, 10s), or an unbounded dirty window inside a
bounded range) previously assigned the contained window's upper bound,
shrinking the merged window and permanently dropping the tail range
from re-computation. Keep max(prev_upper, cur_upper) instead.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): clip bounded dirty ranges at the expire bound
merge_dirty_time_windows dropped every window whose start is before
expire_lower_bound, so a bounded dirty range crossing the bound (e.g.
[0s, 15s) with expire 10s) lost its still-live suffix [10s, 15s). Now
bounded ranges are dropped only when their end is at/before the expire
bound, and crossing ranges are clipped to the bound (which the caller
aligns to the time window boundary). Unbounded windows keep the
existing start-based behavior.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): fall back to full dirty on dirty-window alignment failure
An eval/alignment error previously aborted the per-task dirty-marking
closure, losing every dirty timestamp and range accumulated for that
task, while the RPC still returned Ok so the producer would not retry.
On alignment failure now log a warning and mark the whole task dirty
(set_dirty) instead, so the affected data is conservatively
recomputed.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): apply rustfmt to new dirty-window merge tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time index units in dirty window marking
Document that DirtyWindowRequest timestamps/time_ranges are bare i64s
interpreted in the source table's time index native unit, and add a
test expressing the same [3s, 11s) range in second/millisecond/
microsecond/nanosecond units across four tables, asserting all align
to the same dirty window [0s, 15s).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to 8127f179
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): return dirty-window alignment errors to callers
Do not acknowledge a DirtyWindowRequest when a time-windowed task cannot
align a timestamp or range. The previous conservative fallback used
set_dirty(), but that marker only represents a single epoch-start window
for time-windowed flows, so it could still lose the affected dirty
range. Propagate task errors through the join loop instead so producers
can retry.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: update proto to commits on main
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: add field id and extension type to histogram
* chore: revert histogram check
* chore: lint
* test: add test coverage for maybe_update_schema
* feat: calculate sub field id from parent column id
* refactor(mito2): guard histogram sub-field ids and cover parquet footer
Address review on #8528:
- native_histogram: derive sub-field ids with checked arithmetic. The SST
writer now surfaces a new InvalidNativeHistogramSubfield error when a
sub-field id cannot be resolved (unknown name or i32 overflow) instead of
silently dropping the id or wrapping. maybe_wrap_schema is now fallible.
- sst: add parquet writer/footer round-trip tests asserting the
greptime.histogram extension and nested PARQUET:field_id (incl. list
elements) survive on disk, and a non-canonical struct is left untouched.
- sst: fix a broken rustdoc link to stamp_native_histogram_subfield_ids.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: return error when fail to get i32 column id
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Lei, HUANG
Weny Xu
Lei, HUANG
Ning Sun
fys
Yingwen
raphaelroshan
dennis zhuang
discord9
shuiyisong
LFC
dependabot[bot]