schedule_compaction, handle_pending_compaction_request and
schedule_next_compaction no longer await anything after compaction
planning became fire-and-forget. Drop the async signature to make the
no-suspension-point invariant explicit: these methods always run to
completion on the worker loop without reentrancy.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
When a compaction finished within min_compaction_interval with no
pending requests, on_compaction_finished left an idle status
(phase = None) behind. The worker then skipped chaining the next
compaction due to the interval gate, and the leftover status made
schedule_compaction swallow all future triggers of the region: regular
waiters were queued but never woken, and manual requests stayed pending
forever. The region stopped compacting until close/drop/truncate.
Add CompactionScheduler::remove_idle_status and call it from
handle_compaction_finished when the interval has not elapsed and no
chained planning is scheduled. The chain-until-no-plan semantics for
compactions that outlast the interval is preserved.
Also drops an unused import left by the previous commit.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
DefaultNotifier::notify decremented INFLIGHT_COMPACTION_COUNT before the
re-entrancy guard, so a duplicate notify (which should never happen, but
the guard exists to defend against it) would decrement the gauge an
extra time and let it drift negative. Move the decrement after the
guard, matching the local compaction path's guard-then-account order.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
The worker only leaves the picking phase after receiving the
CompactionPickFinished notification. Previously the planning task was
spawned fire-and-forget: if it panicked before sending the notification,
the region would be stuck in the picking phase forever, blocking all
future compactions and pending DDLs (e.g. entering staging) of the
region.
Wrap the planning future with catch_unwind so a panic is converted into
a CompactionPlanningResult::Error and the notification is always sent,
letting the worker run the normal error cleanup path.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Move the compaction picker out of the region worker's critical path by
dispatching planning to a background task and reporting the result back
via CompactionPickFinished. CompactionStatus now tracks an explicit
picking phase keyed by a monotonic plan id, so stale planning results
are rejected and duplicate regular triggers coalesce while picking.
Before submitting a prepared compaction, the picker output is refreshed
against the current SST version (file handles are re-resolved and
conflicts roll back reservations), ensuring the plan still matches live
state. CompactionExecution identifies the running task by
(plan id, kind, version control) so finish/cancel/fail notifications
from outdated executions are ignored.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(flow): handle time_ranges in DirtyWindowRequest
Bump greptime-proto to include the new `time_ranges` field on
DirtyWindowRequest (GreptimeTeam/greptime-proto#330) and mark the
corresponding aligned time windows as dirty in the batching engine,
in addition to the existing per-timestamp dirty marking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): fix doc comment spacing in align_time_window
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time_ranges in handle_mark_dirty_time_window
Verify a valid [start_inclusive, end_exclusive) range is aligned to
time window boundaries and stored with an explicit end, and that empty
or reversed ranges are skipped.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): union merged dirty windows with the larger end
Merging a bounded dirty range with a window contained in it (e.g.
[0s, 15s) with nested [5s, 10s), or an unbounded dirty window inside a
bounded range) previously assigned the contained window's upper bound,
shrinking the merged window and permanently dropping the tail range
from re-computation. Keep max(prev_upper, cur_upper) instead.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): clip bounded dirty ranges at the expire bound
merge_dirty_time_windows dropped every window whose start is before
expire_lower_bound, so a bounded dirty range crossing the bound (e.g.
[0s, 15s) with expire 10s) lost its still-live suffix [10s, 15s). Now
bounded ranges are dropped only when their end is at/before the expire
bound, and crossing ranges are clipped to the bound (which the caller
aligns to the time window boundary). Unbounded windows keep the
existing start-based behavior.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): fall back to full dirty on dirty-window alignment failure
An eval/alignment error previously aborted the per-task dirty-marking
closure, losing every dirty timestamp and range accumulated for that
task, while the RPC still returned Ok so the producer would not retry.
On alignment failure now log a warning and mark the whole task dirty
(set_dirty) instead, so the affected data is conservatively
recomputed.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): apply rustfmt to new dirty-window merge tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time index units in dirty window marking
Document that DirtyWindowRequest timestamps/time_ranges are bare i64s
interpreted in the source table's time index native unit, and add a
test expressing the same [3s, 11s) range in second/millisecond/
microsecond/nanosecond units across four tables, asserting all align
to the same dirty window [0s, 15s).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to 8127f179
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): return dirty-window alignment errors to callers
Do not acknowledge a DirtyWindowRequest when a time-windowed task cannot
align a timestamp or range. The previous conservative fallback used
set_dirty(), but that marker only represents a single epoch-start window
for time-windowed flows, so it could still lose the affected dirty
range. Propagate task errors through the join loop instead so producers
can retry.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: update proto to commits on main
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: add field id and extension type to histogram
* chore: revert histogram check
* chore: lint
* test: add test coverage for maybe_update_schema
* feat: calculate sub field id from parent column id
* refactor(mito2): guard histogram sub-field ids and cover parquet footer
Address review on #8528:
- native_histogram: derive sub-field ids with checked arithmetic. The SST
writer now surfaces a new InvalidNativeHistogramSubfield error when a
sub-field id cannot be resolved (unknown name or i32 overflow) instead of
silently dropping the id or wrapping. maybe_wrap_schema is now fallible.
- sst: add parquet writer/footer round-trip tests asserting the
greptime.histogram extension and nested PARQUET:field_id (incl. list
elements) survive on disk, and a non-canonical struct is left untouched.
- sst: fix a broken rustdoc link to stamp_native_histogram_subfield_ids.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: return error when fail to get i32 column id
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: allow unknown PluginOptions with a warning message
* refactor(plugins): deserialize plugin option from borrowed Value
Address review feedback on #8550: in filter_known_plugin_options,
deserialize each entry from a borrowed serde_json::Value reference
instead of cloning it. This removes the per-entry Value clone on the
happy path while keeping the original value available for the warning
log when an entry is unrecognized.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: address review comments
* fix: verify against downstream project
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: support full WAL retirement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: complete close request migration
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: guard Kafka provider setup behind index collector check
Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.
Affected file:
- `src/log-store/src/kafka/log_store.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: avoid to_vec
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: clean up soft-dropped regions offline
Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.
Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): reject file-engine soft drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: preserve soft-drop cleanup split state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove obsolete CleanUp match arm from RegionRequest
The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): clean every soft-dropped region replica
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): order soft-drop replica cleanup
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): order soft-drop replica cleanup"
This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): clean every soft-dropped region replica"
This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(catalog): expose soft drops in recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(sql): add UNDROP TABLE
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sql): cover UNDROP TABLE execution
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(sql): keep successful UNDROP result
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: reject stale undrop by name-based lookup after tombstone consumed
When a table is dropped, recreated under the same name (consuming the name
tombstone), and then the recreated table is dropped, an undrop procedure
that was built before the first drop and holds a stale original table name
should fail with TableNotFound instead of silently matching a different
table.
Changed `UndropTableProcedure::on_prepare` to perform a name-based lookup
when `table_name` is available and filter by table ID, ensuring that a
dropped table can only be recovered when its name tombstone still maps to
the expected ID.
- `src/common/meta/src/ddl/undrop_table.rs`: name-first lookup in on_prepare
- `src/common/meta/src/ddl/tests/drop_table.rs`: test for the stale-id
rejection case
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(log-store): keep Kafka obsolete_all as no-op
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): hide purging tables from recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): scope recycle bin scans by catalog
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sqlness): update recycle bin expectations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: avoid redundant recycle bin allocations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: timestamp display precision should respect column schema (#8227)
Previously, the MySQL writer converted timestamps to NaiveDateTime via
to_chrono_datetime_with_timezone(), then passed the NaiveDateTime to
write_col(). NaiveDateTime::Display uses a fixed 6-digit fractional-
second format, causing:
- TIMESTAMP(3) to show '.195000' instead of '.195'
- TIMESTAMP(9) to show '.195123' instead of '.195123456'
Fix: use Timestamp::to_timezone_aware_string() directly, which formats
with chrono's '%.f' specifier — it strips trailing zeros and preserves
full nanosecond fidelity based on the actual stored unit.
Also add comprehensive unit tests in mysql_writer_test.rs covering:
- All concrete data type → MySQL column type mappings
- UNSIGNED_FLAG propagation for unsigned integer types
- Timestamp precision for all four units (0/3/6/9 decimal places)
- Edge cases: zero subseconds, trailing-zero stripping, Unix epoch,
negative timestamps, and timezone offset shifts
- Column-def ordering, empty schema, and decimal variants
Signed-off-by: Divyansh <anshmcs@gmail.com>
* perf: reuse format buffer for timestamp serialization to avoid per-row heap allocation
Signed-off-by: Divyansh <anshmcs@gmail.com>
* refactor: rename test variable and update visibility of create_mysql_column function
Signed-off-by: Divyansh <anshmcs@gmail.com>
* test: fix timestamp test constants and add sqlness case for display precision
- Correct the base epoch constant: 1_748_836_200 is 2025-06-02, not
2026-06-02; use 1_780_372_200 to match the documented instant.
- Fix subsecond expectations: chrono's %.f renders fractional digits in
groups of 3 (.100 / .010), it does not strip to .1 / .01.
- Add sqlness case timestamp_precision_display reproducing issue #8227
over the MySQL protocol (TIMESTAMP(0/3/6/9) rendering).
- cargo fmt reflow of create_mysql_column signature.
Signed-off-by: Divyansh <anshmcs@gmail.com>
* fix: support binary protocol for timestamp column
Signed-off-by: Divyansh <anshmcs@gmail.com>
* test: update sqlness results for new timestamp display precision
Signed-off-by: Divyansh <anshmcs@gmail.com>
---------
Signed-off-by: Divyansh <anshmcs@gmail.com>
* fix: count Postgres SCRAM auth failures in the auth failure metric
The SCRAM SASL paths returned `Failed` without touching
`METRIC_AUTH_FAILURE`, so once SCRAM is enabled wrong-password and
unknown-user attempts disappeared from `greptime_servers_auth_failure_count`.
Funnel every SCRAM rejection through `record_scram_failure`, which records
one failure with a uniform `UserPasswordMismatch` label so the counter stays
useful without revealing whether the user exists.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: correct per-protocol auth notes in config examples
The user_provider note no longer matched the implementation: pbkdf2_sha256
is excluded from Postgres SCRAM (so its iteration count is never exposed in a
SCRAM handshake), and the warning that hash-only verifiers cannot use MySQL's
native password handshake had been dropped.
State the actual per-protocol fallbacks and incompatibilities, scope the
iteration/salt enumeration caveat to pg_scram_sha256, and regenerate
config.md.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: avoid double-counting Postgres SCRAM authorization failures
authorize() already increments METRIC_AUTH_FAILURE with its own status
code, so routing the authorization-rejection path through
record_scram_failure counted it twice, mislabeling the second increment
as UserPasswordMismatch. Return early to bypass the recorder.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: support SCRAM auth for Postgres
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add pg_scram_sha256 format to hash-password command
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: harden Postgres SCRAM auth
- Verify the client-final nonce matches the server-issued nonce, per RFC 5802
transcript validation, instead of only checking the channel-binding field.
- Replace the per-connection PBKDF2 over a random password for unknown users
with a deterministic mock verifier keyed by the username and a process-wide
secret. This avoids a CPU-exhaustion DoS on unknown usernames and removes a
username-enumeration oracle: the SCRAM server-first salt and iteration count
are now stable per username and indistinguishable from a real user, with no
PBKDF2 cost and random keys that never accept a proof.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* style: format PG_SCRAM_MOCK_SECRET declaration
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: precompute stable SCRAM verifier for plaintext users
Plaintext-backed credentials derived a Postgres SCRAM verifier on the fly
on every connection, using a fresh random salt and running PBKDF2 each
time. That made a known plaintext user distinguishable from stored-hash
and unknown (mock) users through both the unstable server-first salt and
the per-connection timing, enabling username enumeration.
Precompute the SCRAM verifier once at load time (stable salt, default
iteration count) and reuse it, matching the mock verifier handed to
unknown users. Document that non-default iteration counts remain
observable in the SCRAM handshake and weaken enumeration resistance.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: normalize passwords for Postgres SCRAM
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: docs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Value::try_negative and the temporal negative() helpers negated with raw
unary minus, which panics (debug) or wraps (release) on MIN values such as
-i64::MIN. try_negative already returns None for the unsigned arms; make
the signed and temporal arms honor that contract via checked_neg /
checked_negative so a MIN literal produces a clean error instead.
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
Lei, HUANG
fys
Lei, HUANG
shuiyisong
Yingwen
Weny Xu
discord9
Ning Sun
dependabot[bot]
Ruihang Xia
Yash Agrawal
Div
dennis zhuang
LFC
raphaelroshan