I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Fixes#8617
## What's changed and what's your intention?
**Summary:** Fixes a mito2 compaction scheduler bug where a region
permanently stops compacting after a finished compaction when
`min_compaction_interval` is configured to a non-zero value.
**Root cause:**
1. `CompactionScheduler::on_compaction_finished` falls through without
removing the region's `CompactionStatus` when there is no pending manual
request and no pending DDL, leaving a status with `active_compaction =
None` in `region_status`.
2. Back in `RegionWorkerLoop::handle_compaction_finished`,
`is_compacting()` is now `false`, but the follow-up
`schedule_next_compaction` (whose `Ok(None)` branch is the only normal
path that removes the status) is gated by `min_compaction_interval`.
Since the scheduling timestamp was already updated when the task was
scheduled, a compaction that finishes within the interval skips this
cleanup, and the stale status remains forever.
3. From then on, every compaction trigger hits the "status exists"
branch of `CompactionScheduler::schedule_compaction`, which does not
check whether a compaction is actually active: regular requests are
merged into `status.waiters` (never woken until close/drop/failure) and
manual `StrictWindow` requests are parked in `status.pending_request`
(never executed). The region never compacts again until restart and SSTs
keep accumulating.
Note this only manifests with `min_compaction_interval > 0`; with the
default `0` the gate always passes and `schedule_next_compaction` cleans
up the status.
**How this PR works:**
- Adds `CompactionScheduler::remove_inactive_status(region_id)`, which
removes the status only when it has no active compaction. A
`debug_assert!` guards the invariant that such a status has no
waiters/pending requests/pending DDLs (they can only be enqueued while a
compaction is active, and the worker loop is strictly sequential).
- Calls it from `handle_compaction_finished` when the
`min_compaction_interval` gate skips `schedule_next_compaction`,
restoring the invariant that a status in the map always has an active
compaction. The throttling semantics are unchanged: the next trigger
still has to wait for the interval to pass.
**Tests:**
- `test_remove_inactive_status_keeps_running_compaction`: status with an
active compaction (or no status) is never removed.
- `test_compaction_reschedulable_when_next_compaction_throttled`:
reproduces the reported flow — schedule, finish (stale status remains),
remove it as the throttled worker path does, and verify the region can
be scheduled for compaction again.
**Compatibility:** no API, persisted-format, or wire-format changes;
internal scheduler behavior only.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
## What's changed and what's your intention?
This patch adds region hook to soft-drop cleanup phase so we can remove
extension files managed by the hook.
some open questions:
- this cleanup logic sits in `handle_open.rs`, which is a little
confusing
- can we merge this cleanup with our current GC concept to reduce the
complexity?
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [ ] I have written the necessary rustdoc comments.
- [ ] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [ ] API changes are backward compatible.
- [ ] Schema or data changes are backward compatible.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
## What's changed and what's your intention?
- Limit JSON2 structured nesting to 50 levels and encode deeper
objects/arrays as JSONB variants, avoiding Arrow DepthLimitReached.
- Decode nested JSONB variants when projecting JSON2 values to strings.
- Limit type hint path depth.
<!--
__!!! DO NOT LEAVE THIS BLOCK EMPTY !!!__
Please explain IN DETAIL what the changes are in this PR and why they
are needed:
- Summarize your change (**mandatory**)
- How does this PR work? Need a brief introduction for the changed logic
(optional)
- Describe clearly one logical change and avoid lazy messages (optional)
- Describe any limitations of the current code (optional)
- Describe if this PR will break **API or data compatibility**
(optional)
-->
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [ ] API changes are backward compatible.
- [ ] Schema or data changes are backward compatible.
When a compaction finishes with no pending request/DDL, on_compaction_finished
leaves the region's CompactionStatus in the map with no active compaction. If
the worker then skips schedule_next_compaction because min_compaction_interval
has not passed, the stale status is never cleaned up: all subsequent compaction
requests are swallowed by the "status exists" branch (regular waiters hang,
manual StrictWindow requests pend forever) and the region never compacts again
until restart.
Add CompactionScheduler::remove_inactive_status() and call it from
handle_compaction_finished when the interval gate skips scheduling the next
compaction, restoring the invariant that a status in the map always has an
active compaction.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
## What's changed and what's your intention?
This PR adds the candidate-series discovery stage for metric series
scans.
The new scanner reads only encoded primary keys from native memtable and
SST ranges, applies supported tag predicates to those keys, locally
deduplicates them, and uses DataFusion's streaming merge to produce
globally sorted, deduplicated `(table_id, tsid)` candidates in bounded
batches. This avoids reading full field data while discovering which
metric series need to participate in the later data scan.
Supporting changes:
- add a primary-key-only Parquet row-group reader while preserving
binary/dictionary decoding;
- expose reusable encoded-primary-key filters from reader prefilter
plans;
- separate candidate-series entries from normal data entries in the
range-result cache;
- add sparse primary-key decoding helpers and error variants;
- add unit tests and a benchmark for filtering, sorting, deduplication,
decoding, and cache-key isolation;
- reject unsupported metadata and extension-range layouts so callers can
retain the legacy fallback.
The change is internal to Mito's read path and does not alter public
APIs, persisted schemas, or on-disk formats.
Validation:
- `cargo fmt --all -- --check`
- `cargo nextest run --retries 0 -p mito2 series_candidate` was
attempted, but compilation stopped because the local disk ran out of
space while rustc was writing build artifacts.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
Reopens#8486, which was accidentally closed (and whose branch history
I've since repaired, so it couldn't be reopened in place).
## What's changed and why
`sanitize_connection_string` redacts `store_addrs` before they're
logged, but the previous hand-rolled parsing missed several forms and
could leak passwords.
Per the review on #8486 (@killme2008 suggested the `url` crate; @v0y4g3r
listed additional libpq formats):
- **URL-form addresses** (`mysql://`, `postgresql://`, `http://` etcd)
are now parsed with the **`url` crate**; the userinfo password and any
`password` query parameter are redacted.
- **libpq keyword DSNs** (`host=… password=…`) are handled by a small
redactor covering `password = x` (spaces around `=`),
single/double-quoted values with spaces, and `\`-escaped quotes.
I kept a dedicated redactor for the keyword-DSN form rather than
`tokio-postgres`, since `tokio-postgres` is an optional/feature-gated
dependency in `common-meta` (this function runs unconditionally) and
`store_addrs` also covers mysql/etcd, which it can't parse.
## Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests.
By submitting this pull request, I confirm that my contribution is made
under the terms of the GreptimeDB CLA.
Parse PostgreSQL DSNs (URL or libpq keyword) with tokio_postgres::Config —
the backend's own parser — and log its Debug, which redacts the password.
This matches the backend grammar exactly (multi-host URIs, backslash
escapes, any Unicode whitespace, percent-encoded query keys, and '&'/';'/
'://' inside values) rather than approximating it by hand. Other URLs are
redacted via the url crate; a best-effort keyword fallback covers inputs
neither parser accepts.
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
## What's changed and what's your intention?
Remote dynamic filter updates and unregisters can legitimately arrive
after the
datanode registration has already been removed. This happens when
stream-scoped
cleanup wins the race with asynchronous frontend fanout, and when
multiple
subscriber regions on the same datanode fan out unregister requests for
a
query/filter registration that the first unregister already removed.
These expected lifecycle races return `MissingRegistration` as an
idempotent
no-op, but previously emitted both a registry-specific warning and a
generic
outcome warning for every late request. On multi-region queries this
could
produce large warning bursts.
This PR:
- logs missing query/filter registrations at debug level;
- logs the generic `MissingRegistration` outcome at debug level;
- preserves warnings for `AlreadyComplete`, oversized payloads, decode
failures,
and other genuine anomalies;
- leaves outcomes, metrics, and control flow unchanged.
Validation:
- `cargo nextest run -p datanode
test_apply_remote_dyn_filter_update_missing_registration`
- `cargo nextest run -p datanode
test_remote_dyn_filter_unregister_removes_all_region_subscribers_for_filter`
- `rustfmt +nightly --edition 2024 --check
src/datanode/src/region_server/registrations.rs
src/datanode/src/region_server/remote_dyn_filter.rs`
- `git diff --cached --check`
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments. (No public API was
added.)
- [x] I have added the necessary unit tests and integration tests.
(Existing lifecycle tests cover the unchanged outcomes.)
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
## What's changed and what's your intention?
This PR adds a `max_row_group_row_count` region option for the Mito
engine and wires it through region options, alter handling, flush, and
Parquet bulk memtable writing so SST row group sizing can be controlled
per region.
It also updates inverted index row selection logic so selected row
ranges are split correctly across Parquet row group boundaries, and adds
SQL/engine tests for creating and flushing regions with the new option.
Validation: not run in this session; this PR was created from the
existing branch state.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
## What's changed and what's your intention?
This pull request improves the OTLP trace-ingestion hot path by
resolving fixed column indexes once per chunk, writing fixed values
directly by index, and replacing per-byte formatting with `hex::encode`.
It raises the default `otlp.trace_ingest_chunk_size` from 128 to 512 and
updates the example configurations, generated configuration reference,
and ingestion README. Explicit chunk-size settings remain unchanged, but
deployments relying on the released default will batch more spans per
chunk.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [ ] I have written the necessary rustdoc comments.
- [ ] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [ ] API changes are backward compatible.
- [ ] Schema or data changes are backward compatible.
## Summary
- Add post-flush dirty-window notifications for metric writes through
`PendingRowsBatcher`, so batching-mode flows are updated when Prometheus
remote write uses the pending-row batcher path.
- Notifications fire only after the entire physical flush succeeds;
failed flushes (including partial region failures) send no notification.
Notification errors are best-effort and never affect the persisted write
result.
- Optimize timestamp extraction with a null-free fast path
(`extend_from_slice(values())`) and a nullable fallback
(`iter().flatten()`).
- Refactor notifier to take ownership of `Vec<TableBatch>`, eliminating
the per-flush vector clone.
- Key `PendingBatch` by logical `TableId` to prevent same-name
drop/recreate merging within one flush interval.
## Known follow-ups (from self-review)
- Per-peer `Vec<i64>` clone in notifier could use `Arc<Vec<i64>>`;
intentionally deferred as the cost is negligible for typical peer
counts.
- No integration/sqlness test for end-to-end flow notification through
the batcher.
- Cache lookup `Err` path and multi-table flush path lack dedicated unit
tests.
## Test Plan
- [x] `cargo nextest run -p servers --lib --retries 0` — 293 tests pass
- [x] `cargo fmt --all -- --check`
- [x] `cargo check -p servers -p frontend -p operator -p
tests-integration`
- [x] `git diff --check`
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
## What's changed and what's your intention?
as title
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [ ] I have written the necessary rustdoc comments.
- [ ] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [ ] API changes are backward compatible.
- [ ] Schema or data changes are backward compatible.
Lei, HUANG
LFC
Lei, HUANG
fys
Ning Sun
Yingwen
raphaelroshan
discord9
shuiyisong
Ning Sun
dennis zhuang
shuiyisong
fys
luofucong
Ning Sun