* fix(query): validate merge scan remote schema
Signed-off-by: discord9 <discord9@163.com>
* fix(query): treat JSON columns as schema-compatible across wire/decode forms
CI (Sqlness json2_limit standalone + distributed) failed on the new
remote-schema validation: a JSON column is Binary + extension metadata
(ARROW:extension:name=greptime.json, greptime:type=Json) on the wire but
decodes to Struct(...) with the extension metadata — validate_remote_schema
compared raw arrow data_type and rejected it as a mismatch.
Adds json_fields_compatible(): JSON fields are equal when name and
nullability match, greptime:type matches, and the JSON2 settings
(ARROW:extension:metadata type hints) match, ignoring the physical arrow
type. Only JSON fields may bypass the raw-type comparison; non-JSON
validation stays strict.
Adds 4 regression tests mirroring the CI failure (wire-binary vs
decoded-struct accepted both directions; different JSON2 settings
rejected; JSON vs plain Binary rejected).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
The example TOMLs and generated config.md documented the default of
wal.sync_period as "10s", but since #5677 moved the WAL sync task to a
background RepeatedTask, an unset sync_period falls back to 5s in
RaftEngineLogStore. The two paths therefore had different fsync
periods: deployments based on the example configs used 10s while bare
configs used 5s.
Align the documentation with the actual code behavior (5s) instead of
changing the code fallback to 10s, so that no existing deployment
silently gets a larger data-loss window on host power loss.
- config/datanode.example.toml, config/standalone.example.toml: 10s -> 5s
- config/config.md: regenerated via make config-docs
- src/cmd/tests/load_config_test.rs: update assertions accordingly
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat: support discarding unflushed region data
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): wake stalled writers after discard
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): drop redundant manifest check for discarding unflushed data
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: adjust the position of experimental_enable_prometheus_native_histogram
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* chore: move prom_validation_mode as well
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
---------
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
* chore: gate soft-drop table behind the enterprise feature
Soft-drop table becomes an enterprise-only feature:
- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
registration, and information_schema.recycle_bin registration are
compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
(moving to the enterprise repo); affected information_schema .result
files are regenerated
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: limit unused_variables allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: include the config key in the soft-drop enterprise gate error
Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: limit unused_mut allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: reject soft-drop DDL submissions in non-enterprise builds
Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: stop --enable-gc from enabling soft drop in the sqlness template
Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate fresh soft-drop procedures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: gate soft-drop fallback coverage
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate soft-drop procedure implementation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate drop table soft-drop behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate expired soft-drop gc behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: test enterprise table ddl lifecycle
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark purge_table as enterprise licensed
The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark recycle_bin as enterprise licensed
The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark soft-drop procedure sources as enterprise licensed
The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: check enterprise-gated files are listed in both license configs
A file reachable only through `#[cfg(feature = "enterprise")] mod ...;` is
governed by the GreptimeDB Enterprise License, so it must appear in the
`includes` of licenserc-enterprise.toml and the `excludes` of licenserc.toml.
hawkeye stays silent when it does not: the file keeps its Apache-2.0 header and
passes the default check precisely because it was never excluded from it.
scripts/check-enterprise-license.py walks enterprise-gated `mod` declarations,
resolves them to files (submodules included) and diffs that set against both
configs, also reporting stale entries. It runs in the license job in CI and as
`make check-enterprise-license`.
Documents the split it cannot decide for you — whole enterprise features get
their own file, a gated match arm stays inline — in
.agents/architecture-invariants.md.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: tighten enterprise license checks
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: show stable, latest and nightly version badges in README
The single release badge rendered whatever GitHub considered newest, so a
pre-release such as v1.2.0-beta.1 looked like the recommended version.
Split it into three self-updating badges using the shields.io `filter`
parameter, keyed off the existing tag naming:
- stable: `!*-*` matches tags without a hyphen (v1.1.4)
- latest: `!*-*-*` excludes nightly and dev builds (v1.2.0-beta.1)
- nightly: `*-nightly-*` matches the weekly build (v1.2.0-nightly-20260706)
No workflow changes are needed; the badges track new releases on their own.
A one-line caption below them says which channel to pick. The release-date
badge is dropped as the three version badges already carry that signal.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: remove star history chart from README
The chart carried a sealed_token in three URLs and added a large
third-party image to the Project Status section without saying anything
the badges and case studies do not already cover.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs: use --grpc-bind-addr in README quickstart
--rpc-bind-addr is now only a hidden alias of --grpc-bind-addr and no
longer shows up in --help.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* Update README.md
Co-authored-by: Ning Sun <classicning@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Ning Sun <classicning@gmail.com>
DirEntry on Windows is a snapshot from FindFirstFileW: file_type() and
metadata() keep returning cached data after the file is removed, so
read_list_entry() cannot observe the deletion. The test asserts the
Unix behavior (lstat returns ENOENT) and fails deterministically on
Windows nightly CI (4/4 tries). Gate it with #[cfg(not(windows))].
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): use two pointers for sliding range boundaries
Replace the stale cursor heuristic in RangeManipulateStream::calculate_range
with monotonic left/right cursors. The old path rescanned each evaluation
window (O(E x samples-per-window)) and could lose valid samples after sparse
gaps or trailing empty windows. The two pointers keep strict monotonic
progress, reducing boundary generation to O(N + E) while preserving
(curr-range, curr] semantics, start/end shortening, and empty-window output.
Controlled release benchmarks (fixed CPU, ABBA):
- Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s,
~96% at 1h/15s.
- Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end;
shorter windows stayed within run-order noise.
Signed-off-by: discord9 <discord9@163.com>
* perf(promql): specialize changes/resets with adaptive edge counting
The generic range_fn macro slices, downcasts, and rescans every overlapping
window for changes() and resets(). Replace the macro path for these two
functions with hand-written UDF wrappers backed by a shared private
edge-count kernel: direct raw-offset scans when requested edges are few,
otherwise one global u64 edge prefix so each window is answered by a prefix
difference.
Behavior is preserved bit-for-bit, including raw null-buffer values, NaN
semantics, signed zero, infinities, empty/singleton windows, independent
timestamp/value offsets, arbitrary window layouts, and exact DataFusion
error messages. The shared proc macro, planner, serializer, and other range
functions are untouched.
Controlled release benchmarks (fixed CPU, ABBA):
- Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time.
- Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster.
- Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client
and 12.0-20.9% server latency improvement; controls stayed within drift.
Signed-off-by: discord9 <discord9@163.com>
* ci(query-regression): include PromQL range boundary case in defaults
An audit of historical query-regression runs found zero range-query
coverage: all 208 PromQL ANALYZE samples were bare selectors, so range
evaluation could regress without CI noticing. Wire the
promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so
label-triggered runs measure the range path. The case is cheap: a ~0.3s
synthetic fixture and about a minute of query execution per base/candidate
pass.
Signed-off-by: discord9 <discord9@163.com>
* chore(promql): address sliding range review nits
Move test-only imports into their test modules and remove the unused
pre-specialization changes and resets helpers.
Signed-off-by: discord9 <discord9@163.com>
* style(promql): apply pinned rustfmt
Signed-off-by: discord9 <discord9@163.com>
* test(promql): cover sparse range results
Share the changes and resets test scaffolding while keeping their behavior
oracles independent. Add an end-to-end sqlness regression for sparse samples,
empty intermediate windows, and a valid trailing sample.
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
test_writer_abort_is_unsupported_without_atomic_write asserted the file
content immediately after abort() returned Unsupported. SecureFsWriter
writes through tokio::fs::File, whose write_all() only enqueues a blocking
write task (tokio's poll_write returns Ready before the write completes),
so the data may not be visible yet when the test reads the file. Drop the
race-prone content assertion and only verify the Unsupported contract.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(auth): warn when credential load disables Postgres SCRAM or drops a line
Static and watch user providers degraded silently in two ways:
- A single non-SCRAM verifier (mysql_native_password, or a legacy
pbkdf2_sha256 hash that predates SCRAM) disables Postgres SCRAM for
every user and falls back to cleartext, with no signal to the operator.
- A malformed credential line (commonly a plaintext password containing
'=', which splits into more than two parts) was dropped without a trace.
Emit a warning at each credential load for both cases so operators don't
unknowingly serve cleartext passwords over Postgres or lose a user. This
is logging only; authentication behavior is unchanged. The SCRAM check
never logs secrets, and the malformed-line warning logs the line number
and file, never the line content.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(auth): warn on credential file read error before truncating
A read error from lines() (I/O failure or invalid UTF-8) ends the
iterator via map_while, silently dropping every remaining credential.
Warn with the line number and file before truncating, matching the
malformed-line handling, so the drop is observable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): fence async index builds by schema generation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): retry stale index builds after schema changes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito2): split compaction module into scheduler/reader submodules
Extract the compaction scheduler lifecycle (scheduler, status, phases,
execution, SST reservations, pending requests) and its tests out of
compaction.rs into compaction/scheduler.rs and compaction/scheduler_test.rs.
Split the remaining helpers by responsibility:
- estimate_compaction_bytes/refresh_picker_output move to scheduler.rs,
the only call site
- get_expired_ssts moves to picker.rs, shared by the TWCS and window
pickers
- CompactionSstReaderBuilder/time_range_to_predicate/ts_to_lit move to
the new compaction/reader.rs
The root compaction.rs keeps the shared output types and
find_dynamic_options, and re-exports the moved types so existing call
paths stay unchanged. Pure code motion, no behavior change.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): separate compaction scheduling from execution details
Turn compaction/scheduler.rs into a directory module to make the
scheduling flow easier to review:
- scheduler.rs keeps the pure scheduling core: the CompactionScheduler
state machine, scheduling entry points, termination chaining, DDL
coordination and region lifecycle events
- scheduler/planning.rs holds the execution-facing parts: background
planning dispatch, picker invocation, plan acceptance, remote/local
submission and memory estimation
- scheduler/state.rs holds the per-region lifecycle types:
CompactionStatus, ActiveCompaction, CompactionPhase, CompactingFiles,
LocalCompactionState, CompactionExecution and PendingCompaction
Child modules keep access to the scheduler's private methods, so the
split is pure code motion with minimal visibility changes (pub(super)
only where the parent module or tests reach into child items).
No behavior change.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): use absolute scheduler imports
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): document compaction scheduler modules
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): make async index publication conditional
Check the captured SST generation and commit index metadata under the same manifest write lock. Revalidate the committed metadata before applying it to the in-memory version, and clean exact-version artifacts when either publication stage becomes stale.
Add deterministic compaction and overlapping-index tests covering reopen consistency, duplicate rows, cache cleanup, and both file purgers.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito2): centralize manifest update finalization
Share the locked update, lock release, follower check, and hook firing path between regular manifest updates and conditional index publication.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): retain index build leases across reopen
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): avoid retiring scheduler on index failure
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): handle cross-region index publication
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): use physical region for index paths
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(cmd): update noop index builder
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): avoid reusing published index versions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito2): log untracked index build stops
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): set compaction time range in index test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: expose MitoRegion::all_manifest_files for metadata rebuild
Adds a public read-only accessor that returns all live SST file metas
and the current manifest version from the region manifest. Used by the
downstream project admin path to enumerate the authoritative live file
set without going through the worker loop.
* fix(mito2): merge staging manifest files in all_manifest_files
The original implementation only read the normal manifest
(manifest_ctx.manifest()) and skipped staging_manifest(). While the
region is in staging mode (e.g. region copy/migration), the authoritative
live file set lives in the staging manifest, so callers would silently
miss those files.
Now matches the semantics of manifest_sst_entries() (~L771), which
explicitly merges manifest().files with staging_manifest().files via a
HashMap collect (dedup by FileId). The returned manifest version is the
staging version when a staging manifest is present, otherwise the normal
manifest version.
Also removed downstream-specific references from the rustdoc comments.
LFC
Weny Xu
discord9
Yingwen
jeremyhi
shuiyisong
Lei, HUANG
dennis zhuang
sun
Whis Liao
Lanqing Yang
Ning Sun