* feat: support compaction options in HTTP hints
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: share TWCS compaction option prefix
Use the store-api constant for both HTTP hint inference and Mito region option validation so the two paths cannot drift.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add per-column COMPRESSION option for SST columns
CREATE TABLE accepts COMPRESSION WITH (type = 'zstd', level = N,
page_size = '<size>') on a column. The options are stored in the column
metadata, carried through gRPC column options, shown by SHOW CREATE
TABLE, and applied by the mito SST writer and bulk part encoder as
parquet column properties. Unset fields keep the SST defaults. Nested
types are rejected.
A page size below the dictionary page limit also lowers the column's
dictionary page limit: parquet sizes write batches by the dictionary
budget and keeps that size after falling back to plain encoding, which
would otherwise ignore the smaller data page limit.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: validate column compression options before persistence
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: reject compression overrides on metric logical tables
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito): retain region compaction and DDL lifecycle state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): finalize opening batch in the worker on registration
Clean up the `opening_regions` entry and reply to its waiters in the same
serialized worker step that registers an opened region, instead of waiting
for the background opener to resume after the registration acknowledgement.
Previously the entry was only removed by the background opener, so a close
followed by a reopen could join the stale opening batch and receive a false
success after the region no longer existed. Replying from the worker closes
that window because registration and waiter cleanup are now atomic with
respect to the worker loop.
Add a test-only `EventListener::on_region_open_registered` hook and a
regression test that holds the first background opener to verify a reopen
does not wait for the previous open task. On failure the region is still
stopped and waiters receive `RegionClosed`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): unregister drop runtime state before WAL cleanup
The region leaves the region map before its WAL entries are obsoleted. A
failure in `wal.obsolete(...).await?` returned early and skipped
`cleanup_dropped_region_runtime_state`, so the resident compaction, flush,
and index scheduler entries were never unregistered. Since the region is no
longer in the map, a later close cannot clean them up either, leaking the
entries for the rest of the worker's lifetime.
Move the runtime-state cleanup to immediately after the region leaves the
map, before the fallible WAL cleanup. The schedulers only touch their own
state and do not depend on the region map or WAL, so the reordering is safe
and leaves the success path unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): tighten visibility of new compaction state items
Keep `ActiveCompaction::idle` private (it is only used inside `state.rs`)
and expose the other items added by the resident-region refactor as
`pub(crate)` instead of `pub(super)`, per the style guide. Also drop a
stale comment in `on_compaction_finished`: DDL handoff now happens in
`take_ready_ddls` before this block, which only runs when no DDL is
queued.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): skip compaction scheduling while a region is truncating
`handle_flush_finished` dispatches queued DDLs before scheduling
compaction. A truncate queued behind a committing flush marks the region
as `Truncating` synchronously, but `schedule_compaction` only skipped
staging states, so it could schedule a new automatic compaction while the
truncate was still running.
Add `MitoRegion::is_truncating` and skip compaction scheduling for a
truncating region. Add a regression test that queues a full truncate
behind a committing flush and asserts no compaction is scheduled.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito): log DDL-fenced compaction rejection
The refactor dropped the log emitted when a region was skipped because of
pending DDL requests. Automatic triggers carry no waiter, so the rejection
became silent and could not be told apart from a scheduling interval that
had not elapsed. Log the queued and executing DDL state before cancelling.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito): document DDL reply fence invariants
Explain why fence_ddl_replies installs executing_ddl before handing off
the batch and why CompactionDdlComplete carries both a scheduler-wide
generation and a per-slot request_id. The generation fences replies from
superseded batches or region incarnations, while request_id makes
completion idempotent so only the last distinct reply releases the fence.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito): rename DDL completion request_id to request_index
The value is the DDL's index within the handed-off batch, not a globally
stable request identifier; request_id suggested otherwise. Rename it to
request_index across the scheduler, worker dispatch, and tests.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(log-store): collect object store WAL objects below a persisted reclaim boundary
Every object header carries a reclaim boundary: every object below it was
fully flushed when the object was sealed. The writer computes the boundary
from the catalog and the obsolete watermarks after every obsolete and every
indexed object, and stamps it on every sealed object and on the start
object. The boundary of the last indexed object permits deletion: every
present object below it, indexed, listed by recovery off the chain, or
left by a failed batch, is deleted in the background, at most four at a
time.
Recovery judges every candidate with its own boundary: a link to a
predecessor below it holds whether or not the predecessor is present, and
the chain stops there. A read that listed an object before it was
collected skips it instead of failing.
The header grows from 46 to 54 bytes; the format version stays 1.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* perf(log-store): scan the reclaim boundary from the previous boundary
The boundary was recomputed by scanning every region's indexed objects
from the first one, so the flushed objects of busy regions were scanned
again on every indexed object while another region held the boundary.
Scan the objects in sequence order from the previous boundary instead
and stop at the first one holding a live segment. Watermarks never move
down and a region first appears above the boundary, so the result is
the same and an object is not scanned again once the boundary passed it.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(log-store): name the parked operation helpers after both their uses
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(log-store): skip corrupted object store WAL segments per region
A segment that does not decode, because its checksum does not match or
its content disagrees with its footer entry, is fetched once more. If it
still does not decode, the read follows `on_corrupted_segment`: `skip`
records a WAL hole for the region with the object key, sequence and the
footer's entry id range, increments
`greptime_logstore_object_store_wal_skipped_segments_total`, logs a
warning and continues with the next object; `fail` fails the read. A
failed fetch fails the read in both modes. Holes are kept in memory per
region and can be listed from the store.
The standalone test builder allocates WAL options through
`build_standalone_wal_provider`, and integration tests run a standalone
instance on the object store WAL across restarts, on S3 when the
`GT_S3_*` variables are set. The standalone shutdown runs every step and
returns the first error.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(cmd): cover the best-effort standalone stop
Build a standalone instance whose leader services fail to stop and whose
datanode servers fail afterwards, and check that `Instance::stop` still
stops the object store WAL and returns the leader services error.
Rename `skip_segment` to `record_skipped_segment` and the file store
integration test to `test_standalone_object_store_wal_survives_restarts_on_file`.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): truncate parquet column index min/max for SST writes
The SST writer and the bulk part encoder disabled column index truncation
together with statistics truncation in #6977. Mito never reads the column
index, but every page of a large string or binary column still stored its
full min and max values, uncompressed. Truncate the column index to the
parquet default of 64 bytes and keep chunk statistics untruncated, since
primary key pruning decodes them.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(mito2): note that the column index must not be used to decode primary keys
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(mito): wait for WAL durability before publishing a manifest watermark
A log store that acknowledges appends before their entries are durable can
hand out entry ids that a crash loses. Before a flush, a full truncate or a
discard of unflushed data records an entry id in the manifest, Mito now waits
on `LogStore::wait_durable` for that id. The flush waits after its SSTs are
written and observes its cancellation, so a drop or truncate queued behind it
is not held up by the upload.
Add engine tests of the object store WAL across restarts and of the barrier,
and a log store testing hook that stores an object and then reports its create
as failed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito): synchronize the WAL barrier tests on store signals
The barrier tests waited with fixed sleeps, which do not establish that a
flush or truncate has reached its durability wait, and the lost backlog test
could stop the store before the seal was handled. The object store WAL gets
testing hooks that observe parked creates and durability waits, and one that
ends the actor as a crash would; the tests wait on them, crash the store and
the engine before a reopen until nothing holds the old store, and cover a
truncate or discard of unflushed data that completes once its entry is
durable.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito): rename the recovery state and the crash test in the WAL recovery tests
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(mito): drop a comment that restates the empty recovery state
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(datanode): wire the object store WAL into the datanode and standalone
A datanode configured with `provider = "experimental_object_store"` now
builds an `ObjectStoreLogStore` instead of failing with the placeholder
"not supported yet" error. The builder resolves `storage_provider`
against the object store manager (empty selects the default store, an
unknown name is rejected as an invalid `storage_provider`), opens the
store under the node id and the standalone generation, stops it if a
later build step fails, and hands it to `Datanode`, whose shutdown stops
it after the region server. Datanode shutdown is now best-effort: every
step runs and the first error is returned. A datanode with a metasrv
client still rejects the provider.
The standalone bootstrap builds `WalProvider::ObjectStore` with the same
derived node prefix before the metasrv WAL conversion, so the WAL
options it allocates match the prefix the store runs under.
`log-store` exports `ObjectStoreLogStore` and its testing hooks. Mito's
test utilities can build an engine on the object store WAL over an
in-memory object store, and new engine tests cover create, reopen,
replay isolation between regions, batch open, drop and offline cleanup,
the latest entry id after replay, and the rejection of object store WAL
options on a Raft Engine log store.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(datanode): stop every region engine before the object store WAL
The region server returned at the first engine whose stop failed, so the
engines after it in map order, Mito included, were never stopped before
the datanode stopped the object store WAL. It now attempts every engine,
returns the first error and logs the later ones.
`ObjectStoreLogStore` is also exported from the `log-store` crate root,
and its callers import it from there.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* chore: remove dead code left behind by removed features
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: remove unused RouteInfoCorrupted error
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(index): build bloom filters from element hashes without per-token allocation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(index): speed up inverted index building with hashed buffers and sync pushes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(index): use BuildHasher::hash_one for element hashes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(index): require callers to act on spill requests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(index): insert bloom hashes one by one to keep segment set capacity bounded
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(index): add index build and bloom search benchmarks
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(index): keep applier setup out of the bloom search benchmark
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(index): skip empty spills and keep the old inverted sort memory estimate
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(index): stream fulltext token hashes and test spill dispatch
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(index): note non-ASCII tokens still allocate in analyze_text_hashes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(index): narrow the allocation note to case-insensitive non-ASCII tokens
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(mito2): load missing index pages of all ranges in one read
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito2): drop unused IndexCacheMetrics::merge
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(mito2): share cached bloom filter metadata instead of cloning it
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): only merge adjacent ranges when batch reading index pages
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): skip empty ranges when batch loading index pages
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(mito2): fetch index ranges concurrently
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(index): return Arc<BloomFilterMeta> from RecordingReader
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(index): search bloom filters of all row groups in one read
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(index): bound the filter bytes of each batched bloom search
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(index): count batch filter bytes exactly as the search reads them
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(index): allow single-range groups in the budget test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(index): cover filters shared across segments in the budget test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): stop parallel flat scan tasks once the receiver is dropped
spawn_flat_scan_task ignored send errors, so after a query was cancelled the
detached task kept reading its source to EOF. Break out of the loop when the
receiver is gone.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito2): log when a parallel scan task stops on a dropped receiver
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito2): include the send error in the parallel scan stop log
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates
Move the workspace from the pinned nightly-2026-03-21 to stable
1.96.1 and drop all 23 '#![feature]' gates across 13 crates,
rewriting the still-unstable API usages with stable equivalents:
- try_blocks: closures / an async block (table, query, common-function, servers)
- duration_constructors: Duration::from_secs(n * 86400) / (n * 60)
- iterator_try_collect: collect::<Result<Vec<_>, _>>()
- box_patterns: as_deref() + matches! chains (sql)
- error_iter: error_chain_root() source-chain walker (common-error);
sources() includes the error itself, so the walker never panics
- int_roundings: div_floor -> div_euclid (equal for positive divisors)
- iter_partition_in_place: stable sort_by_key partition helper (index)
- hash_set_entry: HashSet::insert bool / contains+insert
- trait_alias: lifetime-parameterized dyn FnOnce type aliases (puffin)
- string_from_utf8_lossy_owned: from_utf8_lossy(&v).into_owned()
- never_type: Infallible (common-recordbatch)
- debug_closure_helpers: closure-backed DebugFmt newtype (mito2)
- binary_heap_pop_if: peek().is_some_and() + pop()
- exclusive_wrapper: drop Exclusive; C: Send + Unpin already in bounds
- stmt_expr_attributes: stale gate, no usages
Also fix release-dev-builder-images.yaml, which parsed
rust-toolchain.toml with a date-only regex and would produce empty
image versions with a stable channel; it now extracts the full
channel token. Dev-builder images verified against stable 1.96.1
(image build, default-toolchain behavior, binstall/nextest, riscv64
and android targets, in-image cargo check).
Validated on 1.96.1: cargo check --workspace --all-targets, clippy
--workspace --all-targets --all-features -D warnings, cargo fmt
--check, and nextest on all 13 affected crates (4586 passed).
Part of #9289. Depends on #9298 (fuzz nightly quarantine) merging
first.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: update flake checksum
* chore: use wild for linker in flake
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat(mito): limit series index disk usage
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): enforce series index quota during reconciliation
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): remove series index disk budget layer
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): simplify series index limit to estimated usage
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): trust index catalogs when loading snapshots
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): minimize series index disk limit changes
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): keep series index cleanup running at capacity
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): avoid no-op index clones and stabilize capacity tests
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: update config API expectation and stabilize index build tests
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: use relative object keys for Windows filesystem access
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: cover Windows path and time limits in full test CI
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: use relative keys in metadata snapshot tests
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks
cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.
- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
misplaced deps to the correct sections, drop orphaned
[workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
for dependencies that are structurally required despite no textual
reference: sqlparser (required by sqlparser_derive expansions in
datatypes, common-query), common-error (required by common-macro's
stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
(feature-pinning for the transitive kube dependency in tests-fuzz),
tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
common-mem-prof), protobuf (required by build.rs-generated bindings in
log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections
Part of #9289
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(meta): populate physical metric table column ids (#9286)
* fix(meta): populate physical metric table column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* test(meta): verify physical metric column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
---------
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* fix(postgres): return empty responses for comment-only SQL (#9295)
fix(postgres): handle parsed empty queries in both protocols
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
* ci: create docs follow-up issue on PR merge instead of on label (#9237)
* ci: create docs follow-up issue on PR merge instead of on label
The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.
Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
carries the docs-required label
This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on docs issue creation timing
- Only touch docs labels when the docs checkbox state actually changed
in an edit. Previously, editing any other part of the PR body while
the checkbox stayed checked removed the docs-required label, silently
dropping the docs follow-up now that issue creation happens at merge.
Unchanged checkbox now leaves labels untouched, which also preserves
manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
re-read the live PR via the API and create the docs issue if the
docs-required label is present OR the checkbox is ticked in the
current body.
- Make the workflow concurrency group action-aware so a merge run does
not cancel an in-flight label update from an edit run.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: make docs-required label the single source of truth at merge
The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.
At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(promql): check matcher safety on the receiving operand
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): spell out the shapes a filter may cross
`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.
Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.
No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): assert the new pushdowns reach the scan
The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.
Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(ci): build tests-integration lib with meta-srv/mock (#9299)
* fix(ci): build tests-integration lib with meta-srv/mock
tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.
Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(repartition): split per-case repartition tests
test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.
Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns
Signed-off-by: fys <fengys1996@gmail.com>
* fix(json2): preserve rows on type hint mismatch during compaction
* refactor(json2): simplify alter settings handling
* fix(json2): preserve coerced values during compaction
* chore: remove unnecessary clone
* chor: reduce memory allocations
* fix: cargo clippy
* chore: update greptime-proto to main branch
* refactor(datatypes): unify string handling with other JSON type hints
* fix: cr
---------
Signed-off-by: fys <fengys1996@gmail.com>
* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)
* fix: run compaction pruner tasks on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep compaction metadata and index work on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev
Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add Jev choice and score functions
Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: use generic AI SQL function names
Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: share constant AI criteria within each batch
Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: preserve AI score uncertainty in JSONB results
Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: explain reuse of volatile AI evaluations
Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: share logical table batching with OTLP metrics (#9288)
* feat: share logical table batching with OTLP metrics
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: unify pending rows batch acknowledgement policy
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align logical batcher example configuration expectations
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align batcher worker channel defaults to 65536
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(mito2): lazily decode dense primary key columns (#9226)
* perf(mito2): lazily decode dense primary key columns
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): bypass lazy decoding for full primary keys
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito-codec): preserve prefix decoding errors
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): align encoded length helper naming
Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify conditional dense key decoding
Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): share string framing in row converter
Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: sync lock
* fix: shear and check issues
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* perf(mito2): lazily decode dense primary key columns
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): bypass lazy decoding for full primary keys
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito-codec): preserve prefix decoding errors
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): align encoded length helper naming
Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify conditional dense key decoding
Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): share string framing in row converter
Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): compare primary key ranges across schema versions
Bind FileHandle ranges to the pinned region schema and append cached constant defaults to historical Dense keys. Preserve raw SST statistics, reject inexact bounds, and avoid invalidating views for unrelated metadata changes.
Cover schema evolution, default changes, and tombstone retention through real compaction and reopen regressions.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): report invalid primary key ranges
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): scope primary key ranges to comparisons
Keep raw PK bounds only in FileHandleInner and move schema-aware mapping and caching into task-local comparison contexts.
Use explicit contexts for compaction overlap checks, window aggregation, and series scans. Preserve pinned-schema isolation, late statistics, and shared file lifecycle state without rebinding every handle.
Cover cache isolation across region owners and adapt range fixtures to real Dense encodings. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): cache aligned primary key ranges per file
Replace task-local range maps with a single-slot cache in FileHandleInner, keyed by the target schema version. Preserve raw bounds for realignment across snapshots and default changes.
Share schema mappers across comparison paths and use copy-on-write SST lists for metadata updates. Simplify range mapping to accept encoded bounds and assert the same-table contract at the file accessor.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): clarify primary key mapper schema snapshot
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): align primary key range fixtures with table contract
Remove obsolete cross-table fallback expectations after region validation became a caller contract. Give compaction fixtures matching table identities, including the active-window L1 scenario.
Clarify the mapper precondition and format the simplified alignment call. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): enable filesystem GC in release unit tests
Let unit tests use the filesystem-backed object-store GC path regardless of optimization profile. Keep the production release GC selection unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito-codec): skip release value decoding in PK prefix counts
Validate field values only in debug builds and unit tests while keeping boundary, truncation, and trailing-byte checks in every build.
Cover the linked library in debug and release integration tests, and verify that release unit tests still perform value validation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito-codec): remove redundant prefix integration tests
Retain the codec unit tests and cross-schema compaction regressions while dropping the standalone build-profile test file and its release-only invalid-value expectation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: wait for MySQL to accept authenticated TCP queries
Add a healthcheck using the configured test account and database. Docker Compose --wait previously only observed container startup because the fixture image had no healthcheck, allowing metasrv to connect before MySQL initialization completed.
Verify readiness with SELECT 1 over TCP rather than the initialization socket.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): split SWCS output files by size threshold
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): use resolved SWCS output size options
Read the SWCS output file size threshold from the resolved region options so database-level compaction settings are honored. Align the existing picker test with this configuration source.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): cancel cache construction for incomplete scans
CacheBatchBuffer spawned the background concat task and dropped its join
handle. A scan that was cancelled or failed therefore left the task alive:
it kept compacting already queued batches, and while waiting for a range
result memory permit it held them, even though without a finish command
the result can never be put into the cache.
Keep the handle and abort it when the buffer is dropped. The handle is
cleared once the task owns the finish command, so a completed scan still
populates the cache after its stream is dropped. Abort does not preempt a
concat that is already running; it takes effect the next time the task is
polled.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(mito2): wait for the permit park before cancelling the buffer
An empty buffered_batches only proves the batches were enqueued, so the
cancellation test could abort a concat task that had never been polled.
Count acquisitions that find too few permits, a test-only signal, and drop
the buffer once the task has reached that wait. Nothing awaits between the
check and the parking, so an observed increment means the caller is about
to wait for permits the test holds for the rest of the case.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add config flag to control range index reads
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: disable range index builds when configured and default to off
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
CompactDispatcher acquired a borrowed permit inside the async wrapper and
dropped it as soon as the blocking task was submitted, so the semaphore
never bounded the compactions that actually ran. Acquire an owned permit
and move it into the blocking closure so it covers the queue wait and the
merge itself.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: bump memcomparable to d8fb3558 for bytes I/O optimizations
The new revision (v0y4g3r/memcomparable main, PR #1) includes:
- Deserializer::read_bytes_into for reusable-buffer decoding
- read_bytes 32B pre-allocation (no more 0→8→16→32 realloc chain)
- bulk put_slice in both normal and reverse modes
- Error::Eof instead of panic on truncated input
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): extract sparse tag values lazily without full PK decode
Replace the full primary-key decode in two hot consumers of
SortField::deserialize with column-lazy extraction over
SparsePrimaryKeyView, eliminating the per-label owned values
(Vec<Value>, one heap allocation per string) for every distinct key:
- flat_format: DecodedPrimaryKeys keeps sparse keys encoded and
extracts each tag column directly from the raw dictionary bytes into
a reusable buffer. FlatConvertFormat (flush/compaction write path)
builds all projected tag columns in one pass per key, sharing offset
discovery between columns; the read path (file_range filters) keeps
per-column lazy extraction.
- series_index writer: builds rows from the reserved prefix and only
the indexed tags via a shared offsets cache and buffer, instead of
decoding all labels and copying out a few strings.
- mito-codec: add SparsePrimaryKeyView::reserved_value for typed
table_id/tsid extraction.
Persisted formats are unchanged; values pushed to column builders are
identical (NULL vs empty string, reserved column types, duplicate
dictionary entries and consecutive-run order are preserved).
Benchmark (bench_pk_tag_column, 4096 rows, 40 labels per key, Criterion
30 samples): old = full decode + per-column builds, new = lazy
extraction (+ one-pass for multi-column projection).
| Workload | Before | After | Change |
| --- | ---: | ---: | ---: |
| 1 of 40 tags, 4096 distinct keys | 6.95 ms | 803 us | -88% |
| 1 of 40 tags, 128 distinct keys | 206.8 us | 29.2 us | -86% |
| 10 of 40 tags, 128 distinct keys | 236.2 us | 75.5 us | -68% |
| 40 of 40 tags, 128 distinct keys | 314 us | 215.2 us | -31% |
| 1 of 10 tags, 4096 distinct keys | 1.75 ms | 207.6 us | -88% |
Also adds a decode_sparse benchmark group to mito-codec's
bench_sparse_encoding covering 10/40 labels x 10/24/64-byte values.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): share sparse tag scans across filters
Batch missing tag columns used by simple filters and partition pruning, while retaining single-column extraction and skip-mode semantics.
Replace the optional reserved-byte accessor with typed table_id/tsid reads from the validated prefix. Add full-decode equivalence tests for sliced dictionaries, NULL/empty labels and cross-key scratch reuse, plus series-index roundtrips across batches.
Benchmark real precise filtering with 4096 rows: 10 tag predicates drop from 307.85 to 162.49 us; 40 predicates drop from 894.03 to 550.55 us. All 1459 mito2/mito-codec tests pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): benchmark sparse tag predicate count scaling
Add a dedicated precise-filter benchmark group for 1, 2, 4, 8, 16, and 32 predicates with both 1 and 32 rows per primary key. Keep 4096 rows, 40 labels, and full selectivity constant across cases, and validate the workload outside the timed section.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: use series indexes for SeriesScan candidate discovery
Signed-off-by: evenyag <realevenyag@gmail.com>
* perf: use range indexes in two-phase series reader
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): update series index test fixtures after rebase
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): move lazy range index searchers into file contexts
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: pin series index snapshot before data snapshot
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: preserve builder caching for index-covered SSTs
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat(wal): add the object store WAL provider identity and configuration
Add the identity and configuration of the experimental object store WAL
without the log store implementation:
- store-api: `Provider::ObjectStore` scoped by region id and prefix; it is a
remote WAL.
- common-wal: `DatanodeWalConfig::ObjectStore` (`experimental_object_store`)
with `storage_provider`, `prefix`, `flush_interval`, `max_batch_bytes` and
`on_corrupted_segment`, and `WalOptions::ObjectStore` persisted as
`object_store` with the key `wal.object_store.prefix`. The metasrv config
conversion rejects the new provider.
- common-meta, meta-srv, mito2: handle the new variants, map the region WAL
options to the provider and reject them on a Raft Engine or Kafka log store.
- datanode: validate the configuration and fail with a "not supported yet"
error until the log store lands.
- Example configs and the generated config docs.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(wal): accept object store WAL options when re-enabling WAL
Count object store WAL options as an existing WAL provider when setting skip_wal to false, drop an inaccurate replay note on Provider::is_remote_wal and fix the new rustdoc link.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): preserve last-non-null values during compaction
Use a dedicated LastNonNullPicker to close TWCS seeds over overlapping SSTs, merge connected seed groups, and defer closures with busy inputs. Keep LastRow and strict-window selection unchanged.
Index overlap candidates on the blocking compaction runtime and cover field promotion, transitive dependencies, expiration, and conservative key-range pruning with regression tests.
Refs #9146
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): guard last-non-null compaction against pending memtables
Track conservative memtable sequence lower bounds and capture them from the same version as compaction inputs. Defer unsafe SST closures and strict-window plans until pending writes no longer intersect their sequence span.
Cache minimum sequences during bulk conversion and expose a lightweight memtable getter to avoid rescanning rows or computing full statistics.
Fixes#9157
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): limit eligible last-non-null compaction outputs
Enumerate all TWCS seeds for LastNonNull while retaining bounded window planning concurrency. Apply the output budget after closure expansion and busy/memtable safety checks, preserving execution priority and ordinary TWCS behavior.
Cover rejected and absorbed seeds, priority ordering, and bounded and unlimited output counts with real-picker regression tests.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): include all candidates before last-non-null closure
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify last-non-null compaction closure
Extract transitive closure expansion from output scheduling and clarify seed handling. Document the dedicated picker rationale and schema-evolution risks in PK overlap checks.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover deferred non-seed compaction bridges
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): defer strict-window compaction with busy dependencies
Keep busy SSTs when building the LastNonNull strict-window dependency closure, then defer the entire rewrite plan if any selected input is busy. Preserve LastRow filtering and expired-file cleanup.\n\nCover requested, transitive and unrelated busy windows, including retry after the dependency is released.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): document compaction overlap index fields
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Weny Xu
Lei, HUANG
dennis zhuang
Yingwen
jeremyhi
fys
Ning Sun
shuiyisong
Sainath Singineedi
discord9