* feat(flow): handle time_ranges in DirtyWindowRequest
Bump greptime-proto to include the new `time_ranges` field on
DirtyWindowRequest (GreptimeTeam/greptime-proto#330) and mark the
corresponding aligned time windows as dirty in the batching engine,
in addition to the existing per-timestamp dirty marking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): fix doc comment spacing in align_time_window
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time_ranges in handle_mark_dirty_time_window
Verify a valid [start_inclusive, end_exclusive) range is aligned to
time window boundaries and stored with an explicit end, and that empty
or reversed ranges are skipped.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): union merged dirty windows with the larger end
Merging a bounded dirty range with a window contained in it (e.g.
[0s, 15s) with nested [5s, 10s), or an unbounded dirty window inside a
bounded range) previously assigned the contained window's upper bound,
shrinking the merged window and permanently dropping the tail range
from re-computation. Keep max(prev_upper, cur_upper) instead.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): clip bounded dirty ranges at the expire bound
merge_dirty_time_windows dropped every window whose start is before
expire_lower_bound, so a bounded dirty range crossing the bound (e.g.
[0s, 15s) with expire 10s) lost its still-live suffix [10s, 15s). Now
bounded ranges are dropped only when their end is at/before the expire
bound, and crossing ranges are clipped to the bound (which the caller
aligns to the time window boundary). Unbounded windows keep the
existing start-based behavior.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): fall back to full dirty on dirty-window alignment failure
An eval/alignment error previously aborted the per-task dirty-marking
closure, losing every dirty timestamp and range accumulated for that
task, while the RPC still returned Ok so the producer would not retry.
On alignment failure now log a warning and mark the whole task dirty
(set_dirty) instead, so the affected data is conservatively
recomputed.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(flow): apply rustfmt to new dirty-window merge tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(flow): cover time index units in dirty window marking
Document that DirtyWindowRequest timestamps/time_ranges are bare i64s
interpreted in the source table's time index native unit, and add a
test expressing the same [3s, 11s) range in second/millisecond/
microsecond/nanosecond units across four tables, asserting all align
to the same dirty window [0s, 15s).
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to 8127f179
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): return dirty-window alignment errors to callers
Do not acknowledge a DirtyWindowRequest when a time-windowed task cannot
align a timestamp or range. The previous conservative fallback used
set_dirty(), but that marker only represents a single epoch-start window
for time-windowed flows, so it could still lose the affected dirty
range. Propagate task errors through the join loop instead so producers
can retry.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: update proto to commits on main
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: add field id and extension type to histogram
* chore: revert histogram check
* chore: lint
* test: add test coverage for maybe_update_schema
* feat: calculate sub field id from parent column id
* refactor(mito2): guard histogram sub-field ids and cover parquet footer
Address review on #8528:
- native_histogram: derive sub-field ids with checked arithmetic. The SST
writer now surfaces a new InvalidNativeHistogramSubfield error when a
sub-field id cannot be resolved (unknown name or i32 overflow) instead of
silently dropping the id or wrapping. maybe_wrap_schema is now fallible.
- sst: add parquet writer/footer round-trip tests asserting the
greptime.histogram extension and nested PARQUET:field_id (incl. list
elements) survive on disk, and a non-canonical struct is left untouched.
- sst: fix a broken rustdoc link to stamp_native_histogram_subfield_ids.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: return error when fail to get i32 column id
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: allow unknown PluginOptions with a warning message
* refactor(plugins): deserialize plugin option from borrowed Value
Address review feedback on #8550: in filter_known_plugin_options,
deserialize each entry from a borrowed serde_json::Value reference
instead of cloning it. This removes the per-entry Value clone on the
happy path while keeping the original value available for the warning
log when an entry is unrecognized.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: address review comments
* fix: verify against downstream project
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: support full WAL retirement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: complete close request migration
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: guard Kafka provider setup behind index collector check
Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.
Affected file:
- `src/log-store/src/kafka/log_store.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: avoid to_vec
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: clean up soft-dropped regions offline
Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.
Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): reject file-engine soft drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: preserve soft-drop cleanup split state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove obsolete CleanUp match arm from RegionRequest
The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): clean every soft-dropped region replica
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): order soft-drop replica cleanup
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): order soft-drop replica cleanup"
This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): clean every soft-dropped region replica"
This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(catalog): expose soft drops in recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(sql): add UNDROP TABLE
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sql): cover UNDROP TABLE execution
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(sql): keep successful UNDROP result
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: reject stale undrop by name-based lookup after tombstone consumed
When a table is dropped, recreated under the same name (consuming the name
tombstone), and then the recreated table is dropped, an undrop procedure
that was built before the first drop and holds a stale original table name
should fail with TableNotFound instead of silently matching a different
table.
Changed `UndropTableProcedure::on_prepare` to perform a name-based lookup
when `table_name` is available and filter by table ID, ensuring that a
dropped table can only be recovered when its name tombstone still maps to
the expected ID.
- `src/common/meta/src/ddl/undrop_table.rs`: name-first lookup in on_prepare
- `src/common/meta/src/ddl/tests/drop_table.rs`: test for the stale-id
rejection case
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(log-store): keep Kafka obsolete_all as no-op
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): hide purging tables from recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): scope recycle bin scans by catalog
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sqlness): update recycle bin expectations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: avoid redundant recycle bin allocations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: timestamp display precision should respect column schema (#8227)
Previously, the MySQL writer converted timestamps to NaiveDateTime via
to_chrono_datetime_with_timezone(), then passed the NaiveDateTime to
write_col(). NaiveDateTime::Display uses a fixed 6-digit fractional-
second format, causing:
- TIMESTAMP(3) to show '.195000' instead of '.195'
- TIMESTAMP(9) to show '.195123' instead of '.195123456'
Fix: use Timestamp::to_timezone_aware_string() directly, which formats
with chrono's '%.f' specifier — it strips trailing zeros and preserves
full nanosecond fidelity based on the actual stored unit.
Also add comprehensive unit tests in mysql_writer_test.rs covering:
- All concrete data type → MySQL column type mappings
- UNSIGNED_FLAG propagation for unsigned integer types
- Timestamp precision for all four units (0/3/6/9 decimal places)
- Edge cases: zero subseconds, trailing-zero stripping, Unix epoch,
negative timestamps, and timezone offset shifts
- Column-def ordering, empty schema, and decimal variants
Signed-off-by: Divyansh <anshmcs@gmail.com>
* perf: reuse format buffer for timestamp serialization to avoid per-row heap allocation
Signed-off-by: Divyansh <anshmcs@gmail.com>
* refactor: rename test variable and update visibility of create_mysql_column function
Signed-off-by: Divyansh <anshmcs@gmail.com>
* test: fix timestamp test constants and add sqlness case for display precision
- Correct the base epoch constant: 1_748_836_200 is 2025-06-02, not
2026-06-02; use 1_780_372_200 to match the documented instant.
- Fix subsecond expectations: chrono's %.f renders fractional digits in
groups of 3 (.100 / .010), it does not strip to .1 / .01.
- Add sqlness case timestamp_precision_display reproducing issue #8227
over the MySQL protocol (TIMESTAMP(0/3/6/9) rendering).
- cargo fmt reflow of create_mysql_column signature.
Signed-off-by: Divyansh <anshmcs@gmail.com>
* fix: support binary protocol for timestamp column
Signed-off-by: Divyansh <anshmcs@gmail.com>
* test: update sqlness results for new timestamp display precision
Signed-off-by: Divyansh <anshmcs@gmail.com>
---------
Signed-off-by: Divyansh <anshmcs@gmail.com>
* fix: count Postgres SCRAM auth failures in the auth failure metric
The SCRAM SASL paths returned `Failed` without touching
`METRIC_AUTH_FAILURE`, so once SCRAM is enabled wrong-password and
unknown-user attempts disappeared from `greptime_servers_auth_failure_count`.
Funnel every SCRAM rejection through `record_scram_failure`, which records
one failure with a uniform `UserPasswordMismatch` label so the counter stays
useful without revealing whether the user exists.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: correct per-protocol auth notes in config examples
The user_provider note no longer matched the implementation: pbkdf2_sha256
is excluded from Postgres SCRAM (so its iteration count is never exposed in a
SCRAM handshake), and the warning that hash-only verifiers cannot use MySQL's
native password handshake had been dropped.
State the actual per-protocol fallbacks and incompatibilities, scope the
iteration/salt enumeration caveat to pg_scram_sha256, and regenerate
config.md.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: avoid double-counting Postgres SCRAM authorization failures
authorize() already increments METRIC_AUTH_FAILURE with its own status
code, so routing the authorization-rejection path through
record_scram_failure counted it twice, mislabeling the second increment
as UserPasswordMismatch. Return early to bypass the recorder.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
luofucong
discord9
Yingwen
shuiyisong
Lei, HUANG
LFC
fys
Ning Sun
Weny Xu
dependabot[bot]
Ruihang Xia
Yash Agrawal
Div
dennis zhuang