mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-28 06:36:18 +00:00
60655a933bb786984afaf24904bf6e60f8e4cbf3
1127
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jeremyhi
|
60655a933b |
fix(log-store): persist object store WAL region releases (#9579)
* fix(log-store): persist object store WAL region releases A region's release lived only in memory: a restart after DROP TABLE and before the next object never re-established the dropped region's watermark, so its segments counted as live and pinned garbage collection of the whole prefix. A release now writes a release record (region id and the entry id the region is released through) into an object sealed at once, and returns once that object is durable and indexed, in both acknowledgement modes. Recovery seeds the obsolete watermarks from the release records of the indexed objects before it computes any reclaim boundary. obsolete_all releases in full; the new LogStore::release_region, a no-op by default, releases a dropped region through the largest entry id the store handed out for it, and Mito calls it after obsolete when it drops a region. Objects with release records use format version 2, the version 1 layout with the records at the end of the footer; every other object is still written as version 1. A counter of release records made durable is added. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe persisted object store WAL region releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): carry a failed region release into the next object A release whose batch failed left its record only in an orphan object, so a drop that failed there could still leave the region unreleased after a restart. The store now keeps the release records of failed batches and seals them with its next batch, on a timer tick when no entry is admitted, so the release becomes durable once a create succeeds again. The version 1 and version 2 object fixtures are frozen under tests/compatibility/wal/object_store and loaded by the format tests, and the RFC no longer says that state is persisted only in headers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): complete a region drop whose WAL release fails A drop request is not sent again once the region was removed, so a drop whose WAL release failed used to leave the region half dropped. The drop now completes, cleaning up the region's runtime state and scheduling its files for deletion, and reports the release failure. In the durable mode the object store WAL now creates one more object at stop with the release records of failed and dropped batches, so a graceful stop does not lose a release that failed. The release record counter is described as counting the records of indexed objects. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe the stop-time release create of the WAL actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): state the accepted window of unreleased dropped regions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe release waiters in the WAL actor's outcome table Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail region releases while the object store WAL is failing A failing store keeps its backlog and rejects appends until a create succeeds, so a release would otherwise wait for an upload that is not coming. A release that arrives in that state now fails at once with the retryable error, and one whose batch is still pending when the store enters it fails as well. Neither poisons the store: the record of the first waits for the next sealed batch, like that of a batch that failed, and the record of the second stays in the backlog, so both become durable once a create succeeds. Rejected releases count in rejected_appends_total. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe region releases while the object store WAL is failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the drop crash window in the drop procedure The crash window between the deletion of a table's metadata and a durable release record is a gap of the drop procedure, not of the WAL. Describe it as far as the code shows, name the remedy that belongs in the procedure, cleaning up offline a region a resumed region drop finds not open, and list it under future work. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep rejected release records coalesced while failing A release rejected while the store is failing kept its record for the next sealed batch, but every timer tick then sealed that record alone behind the refused backlog, so a release repeated across ticks grew the backlog without bound. A failing store now keeps such records coalesced and seals them on the first tick after a create succeeds. The fault-matrix case of a release while the backlog is refused now runs with a 10 ms flush interval, repeats the release, and checks that the backlog does not grow and that recovery reads the record back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): keep rejected release records unsealed while failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): cite the drop procedure issue for the crash window Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): let a stop seal rejected release records of a failing store Keeping the records of rejected releases out of sealed batches while the store is failing also kept them out of the seal a stop requests, so a stop whose last attempt of the backlog succeeded dropped them and the released region pinned collection after the restart. A stop now seals them with the backlog. A fault-matrix case rejects a release while the store is failing, stops, lets both creates succeed, and checks the record after a reopen. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the stop of a failing store with rejected releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep release records readable by older object store WAL readers Release records were written as a version 2 footer, which a build without it refuses to read, so rolling a node back after a drop failed to open the whole WAL prefix. They are now the single entry of a segment of the reserved region u64::MAX at the first position of a version 1 object: an older reader indexes that segment as a region it never opens, and recovery reads the segment back with one range read per object that holds one. The segment never holds the reclaim boundary and names no namespace. A durable stop whose last create of release records fails now reports the error instead of returning Ok and losing the records silently. The tests are trimmed: the release unit tests and fault-matrix cases that repeated each other are merged, and the Mito drop tests share one parameterized case. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe release records as a segment of a reserved region Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): report release records a poisoning stop drops A durable stop whose last create of release records met a conflict poisoned the store and returned Ok, losing the records silently. Stop now reports the error whenever release records are dropped after it began, by a rollback or by poisoning. The test of a failed release covers that conflict, and seals its record on a tick after a one-shot create failure, so the next tick cannot fail too. The documentation now says that recovery reads the segments of release records, and that one that does not decode fails recovery. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how recovery reads segments of release records Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): say that stop reports release records a conflict drops Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): restore the transiently failing release create in enqueued mode The fault matrix again covers a release whose create fails transiently in the enqueued mode: the release waits for the repeat under the same sequence and returns once it succeeds. The encode_batch doc now says the release segment is the last segment of the object. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the release segment and count the restored case Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL release Returning the release error from the drop left a metric physical region half dropped: the metric engine stops after the data region and never drops the metadata region or forgets the physical region, and a retried drop finds the data region gone and counts it as done. The release error gave the caller nothing to retry, since the region is already removed and the log store keeps the record for its next object. The drop now logs the error and succeeds. A metric engine test drops a physical region while the WAL releases of both mito regions fail, and checks that both regions and the physical region state are gone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed release no longer fails the drop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL obsolete A store that the release of another region poisoned fails obsolete as well, which left the metric engine with its metadata region undropped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(metric-engine): drop a physical region whose WAL obsolete or release fails Cover a failed data release, two failed releases and a conflicting release that poisons the store, after the regions wrote to the WAL, and check that a store opened later keeps no segment of either region. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed obsolete does not fail the drop either Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): fail the drop on an obsolete error of other WAL providers Only the object store WAL is shared by the regions of a store that the release of one can poison; Raft Engine and Kafka keep failing the drop when obsolete fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a stopped or poisoned store keeps no failed release Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Lei, HUANG
|
3b442adba2 |
feat(mito): decouple compaction unit execution and publication (#9131)
* feat(mito): decouple compaction unit execution and publication Execute shared-input dependency groups as independent local compaction units. Publish and acknowledge each unit separately, retain successful sibling results, and release each unit's input leases after worker apply. Preserve picker priority, panic notifications, snapshot-free execution context, and indexed input leases. Include regression coverage for independent visibility, purge, cancellation, partial failure, and restart recovery. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): use crate visibility for compaction unit items Replace the newly introduced pub(super) declarations with pub(crate), and make CompactingFiles::try_reserve private since it is only used within its module, per docs/style-guide.md. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): simplify compaction unit state and test fixtures Keep local input leases in the task and derive running units from tracked and pending work. Replace the single-implementation task trait with an inherent async method and align exposed state type visibility. Use snapshot-free region fixtures and named test scenarios, and cover independent input purging across three units with bounded parallelism. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): align compaction task and test module names Rename CompactionTaskImpl to CompactionTask now that the CompactionTask trait has been removed and the Impl suffix no longer marks a trait implementation. Rename the state test module from lease_tests to tests to follow the test module naming convention. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): align compaction task debug label The Debug impl reported "TwcsCompactionTask" while the type is now CompactionTask. Report the type name for consistency. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): label compaction elapsed metric per published unit Record greptime_mito_compaction_total_elapsed once per published unit and label it by type: a local dependency unit or a remote job. Update the metric docs and group the Grafana compaction OPS/latency panels by type. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito): clarify compacting file lease sharing `CompactingFiles::for_inputs` clones the shared per-file leases instead of transferring them. Document that the original collection keeps its leases, so an input is released only after every owner drops its lease. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito): document local compaction unit map key Document that `LocalCompaction::units` is keyed by each unit's execution plan ID, which matches `CompactionExecution::plan_id()` and the `plan_id` carried by unit notifications. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): rename submit_prepared_compaction The function submits remote jobs but only prepares local unit tasks; the caller dispatches them after installing the returned phase. Rename it to `prepare_or_submit_compaction` and document that ordering. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 dennis zhuang
|
5c4d9eb4f8 |
fix: align fulltext bloom tokens with matches_term and use them for LIKE (#9470)
* fix: align fulltext bloom tokens with matches_term and use them for LIKE The bloom fulltext index stored tokens that did not follow matches_term's word boundaries (`_`, ASCII next to Han, Han substrings), so pruning dropped rows matches_term accepts. Tokens and query probes now share the character classes of matches_term, Han runs are indexed as bigrams, and LIKE patterns use the same probes. The bloom blob is written under a new key; blobs from older versions are no longer probed. Tantivy no longer serves matches_term. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: drop unused lazy_static from index and avoid typo-flagged test tokens Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: link like_probes to tokenize instead of the downstream ScriptTokenizer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: name probe extraction with verbs and document CharClass Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: rebuild legacy fulltext bloom indexes and probe lower(col) The manifest records the fulltext bloom version per column, so ADMIN BUILD_INDEX now rebuilds SSTs whose bloom index predates the current tokens, and ssts_index_meta shows the version. Case-insensitive bloom indexes also store the tokens of the whole lowercased text for rows where it can differ (U+0130, U+212A, sigma), so matches_term(lower(col), ..) and lower(col) LIKE .. use the index. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover legacy bloom index rewrite by compaction in the compat case Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: log outdated fulltext bloom indexes in the consistency check Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: explain the unrecorded fulltext bloom version Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
a08e5bcceb |
fix(log-store): keep the enqueued backlog on permanent object store errors (#9558)
* fix(log-store): keep the enqueued backlog on permanent object store errors On object storage the WAL's object store sits behind OpenDAL's retry layer, which marks every error it returns as persistent. The enqueued mode classified create failures by that status, so a 403 or a missing bucket was repeated every 100 ms on every create in flight, logged nothing, and made appends hang once the sealed batches were full. Object store errors are now classified by kind: denied access, an invalid configuration (how OpenDAL reports a missing S3 bucket) and a write to a missing location are non-retriable. In the enqueued mode such an error makes the store failing instead of poisoning it: the acknowledged backlog is kept under its sequences and bytes, its lowest pending batch is attempted alone with a delay that doubles from 100 ms to 30 s, appends are rejected at once with a retryable error that carries the last object store error, and the first create that succeeds ends the state. Stop uploads the backlog at once and reports a failure as before. The state is logged on entry, at most once a minute while it lasts and on exit, and exported as the `failing` gauge and the `rejected_appends_total` counter. Conflicts, encoding and catalog errors still poison the store; the durable mode is unchanged. An append held back because the sealed batches are full is now counted and timed by the stall metrics in both modes. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how the enqueued object store WAL handles refused creates Describe the enqueued failure contract as the store now implements it: non-retriable object store errors are told apart by kind because OpenDAL's retry layer under the store marks every error persistent, a failing store keeps its backlog and attempts it with backoff while it rejects appends, and stop uploads the backlog at once. Add the failure matrix row, the failing gauge and the rejected appends counter, the sealed-batch admission limit and the stall metrics that now count it, and the fault matrix case counts. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): time the failing attempts where they start and fix two statements The test of the attempts of a failing store now records when each refused write starts and checks the gaps between them, instead of sampling a counter after a sleep of the test task. The delay comment and the RFC describe 30 seconds as the longest delay between two attempts, not as a bound on when appends resume, and the RFC states that the next append is held once seven sealed batches wait. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): drop the wait_until helper that main now defines Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): name errors that need an operator and trim the RFC contract Rename is_non_retriable to requires_intervention: such a create is still repeated, only slowly, until someone fixes the credentials, configuration or bucket. Name the backoff test after what it asserts, cut the test of the lowest pending batch to two failed rounds since the cadence is covered by the backoff test, and shorten the RFC failure contract to the contract and its rationale, leaving error mappings, request counts and logging cadence to the code comments. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): tie the slow repeat to errors requiring intervention Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Weny Xu
|
f690bbb204 |
refactor!: remove datanode remote WAL index (#9563)
* refactor: remove datanode remote WAL index Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): update new WAL replay tests after index removal Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
4bd36cfd20 |
feat: support compaction options in HTTP hints (#9560)
* feat: support compaction options in HTTP hints Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: share TWCS compaction option prefix Use the store-api constant for both HTTP hint inference and Mito region option validation so the two paths cannot drift. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
2a70194930 |
feat: add per-column COMPRESSION option for SST columns (#9426)
* feat: add per-column COMPRESSION option for SST columns CREATE TABLE accepts COMPRESSION WITH (type = 'zstd', level = N, page_size = '<size>') on a column. The options are stored in the column metadata, carried through gRPC column options, shown by SHOW CREATE TABLE, and applied by the mito SST writer and bulk part encoder as parquet column properties. Unset fields keep the SST defaults. Nested types are rejected. A page size below the dictionary page limit also lowers the column's dictionary page limit: parquet sizes write batches by the dictionary budget and keeps that size after falling back to plain encoding, which would otherwise ignore the smaller data page limit. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: validate column compression options before persistence Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject compression overrides on metric logical tables Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
18634e17bb |
refactor(mito): retain region compaction and DDL lifecycle state (#9498)
* refactor(mito): retain region compaction and DDL lifecycle state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): finalize opening batch in the worker on registration Clean up the `opening_regions` entry and reply to its waiters in the same serialized worker step that registers an opened region, instead of waiting for the background opener to resume after the registration acknowledgement. Previously the entry was only removed by the background opener, so a close followed by a reopen could join the stale opening batch and receive a false success after the region no longer existed. Replying from the worker closes that window because registration and waiter cleanup are now atomic with respect to the worker loop. Add a test-only `EventListener::on_region_open_registered` hook and a regression test that holds the first background opener to verify a reopen does not wait for the previous open task. On failure the region is still stopped and waiters receive `RegionClosed`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): unregister drop runtime state before WAL cleanup The region leaves the region map before its WAL entries are obsoleted. A failure in `wal.obsolete(...).await?` returned early and skipped `cleanup_dropped_region_runtime_state`, so the resident compaction, flush, and index scheduler entries were never unregistered. Since the region is no longer in the map, a later close cannot clean them up either, leaking the entries for the rest of the worker's lifetime. Move the runtime-state cleanup to immediately after the region leaves the map, before the fallible WAL cleanup. The schedulers only touch their own state and do not depend on the region map or WAL, so the reordering is safe and leaves the success path unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): tighten visibility of new compaction state items Keep `ActiveCompaction::idle` private (it is only used inside `state.rs`) and expose the other items added by the resident-region refactor as `pub(crate)` instead of `pub(super)`, per the style guide. Also drop a stale comment in `on_compaction_finished`: DDL handoff now happens in `take_ready_ddls` before this block, which only runs when no DDL is queued. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): skip compaction scheduling while a region is truncating `handle_flush_finished` dispatches queued DDLs before scheduling compaction. A truncate queued behind a committing flush marks the region as `Truncating` synchronously, but `schedule_compaction` only skipped staging states, so it could schedule a new automatic compaction while the truncate was still running. Add `MitoRegion::is_truncating` and skip compaction scheduling for a truncating region. Add a regression test that queues a full truncate behind a committing flush and asserts no compaction is scheduled. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): log DDL-fenced compaction rejection The refactor dropped the log emitted when a region was skipped because of pending DDL requests. Automatic triggers carry no waiter, so the rejection became silent and could not be told apart from a scheduling interval that had not elapsed. Log the queued and executing DDL state before cancelling. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito): document DDL reply fence invariants Explain why fence_ddl_replies installs executing_ddl before handing off the batch and why CompactionDdlComplete carries both a scheduler-wide generation and a per-slot request_id. The generation fences replies from superseded batches or region incarnations, while request_id makes completion idempotent so only the last distinct reply releases the fence. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): rename DDL completion request_id to request_index The value is the DDL's index within the handed-off batch, not a globally stable request identifier; request_id suggested otherwise. Rename it to request_index across the scheduler, worker dispatch, and tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Weny Xu
|
287ced2de2 |
fix(mito2): ignore missing inverted indexes in old SSTs (#9545)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
684c8ff291 |
feat(client): support region request reuse and route refresh hints (#9540)
* feat(client): support shared region requests without deep cloning Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate route refresh retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * style: preserve local error variant imports in retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): distinguish staging rejection from write backpressure Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): preserve legacy retry hints in instruction replies Signed-off-by: WenyXu <wenymedia@gmail.com> * test(client): inline retry hint compatibility cases Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): align batch-open retry hint assertion Signed-off-by: WenyXu <wenymedia@gmail.com> * style: use imported types in region request tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Yingwen
|
674ba72669 |
fix(mito): preserve version ordering in bulk merges and flushes (#9517)
* fix(mito): preserve delete barriers in flat last-non-null dedup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): respect sequence ranges in bulk merges and flushes Signed-off-by: evenyag <realevenyag@gmail.com> * perf(mito): avoid merge selection on the bulk write path Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
jeremyhi
|
18726c7602 |
fix: bound metric snapshot memory and reduce export overhead (#9505)
* fix(mito): bound retained scan memory Compact unused tag dictionary values before merge retention and avoid splitting a single scan source into redundant batches. Include regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(snapshot): bound writes and reduce logical export overhead Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible string dictionaries, bound conversion work, combine Parquet row groups and filter single-table exports before routing. Include resource-boundary and packed-restore regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: clarify metric snapshot acceptance criteria Specify populated scale fixtures, controlled S3 request delay, native macOS validation, paging exclusions and alternating comparison rounds. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): split batches only for merged readers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(snapshot): cover supported types in both layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: consolidate metric snapshot experiment conclusions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): honor series statistics when splitting L0 scans Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics. Cover short-series overlapping L0 files with an ordered latest-value regression. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito): explain memory risk of splitting short L0 series Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: address snapshot export review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(cli): accept source errors in lazy snapshot contexts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
a026b3b280 |
fix(mito): flush memtables containing skip-WAL writes on region close (#9440)
* fix(mito): flush unlogged writes when closing regions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): align skip-WAL naming and metric recovery tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
e4872ec1a2 |
feat(log-store): collect object store WAL objects below a persisted reclaim boundary (#9471)
* feat(log-store): collect object store WAL objects below a persisted reclaim boundary Every object header carries a reclaim boundary: every object below it was fully flushed when the object was sealed. The writer computes the boundary from the catalog and the obsolete watermarks after every obsolete and every indexed object, and stamps it on every sealed object and on the start object. The boundary of the last indexed object permits deletion: every present object below it, indexed, listed by recovery off the chain, or left by a failed batch, is deleted in the background, at most four at a time. Recovery judges every candidate with its own boundary: a link to a predecessor below it holds whether or not the predecessor is present, and the chain stops there. A read that listed an object before it was collected skips it instead of failing. The header grows from 46 to 54 bytes; the format version stays 1. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): scan the reclaim boundary from the previous boundary The boundary was recomputed by scanning every region's indexed objects from the first one, so the flushed objects of busy regions were scanned again on every indexed object while another region held the boundary. Scan the objects in sequence order from the previous boundary instead and stop at the first one holding a live segment. Watermarks never move down and a region first appears above the boundary, so the result is the same and an object is not scanned again once the boundary passed it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): name the parked operation helpers after both their uses Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Lei, HUANG
|
5126623e63 |
fix: align TWCS assignment with half-open time windows (#9464)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 fys
|
c46fbde9ac |
fix(json2): preserve metadata when adding columns via alter table (#8696)
* fix(json2): support adding columns via alter table Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): scope missing-column null handling to JSON2 --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
 Ning Sun
|
7f0f592b6b | chore: update rust toolchain to 1.98.1 (#9429) | ||
|
dennis zhuang
|
526dd48bda |
fix(promql): restore exact filtering for nullable label matchers (#9418)
* fix(promql): restore exact filtering for nullable label matchers Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: verify compound filter semantics and last-row selection Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: consolidate label filter regression coverage Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: separate filter evaluation and timestamp adaptation Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: inline predicate and timestamp tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: keep direct IN predicates outside simple filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: coerce partition bounds before simple filtering Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover partition bound overflow after timestamp widening Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
f9ab3c296b |
feat(log-store): skip corrupted object store WAL segments per region (#9425)
* feat(log-store): skip corrupted object store WAL segments per region A segment that does not decode, because its checksum does not match or its content disagrees with its footer entry, is fetched once more. If it still does not decode, the read follows `on_corrupted_segment`: `skip` records a WAL hole for the region with the object key, sequence and the footer's entry id range, increments `greptime_logstore_object_store_wal_skipped_segments_total`, logs a warning and continues with the next object; `fail` fails the read. A failed fetch fails the read in both modes. Holes are kept in memory per region and can be listed from the store. The standalone test builder allocates WAL options through `build_standalone_wal_provider`, and integration tests run a standalone instance on the object store WAL across restarts, on S3 when the `GT_S3_*` variables are set. The standalone shutdown runs every step and returns the first error. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cmd): cover the best-effort standalone stop Build a standalone instance whose leader services fail to stop and whose datanode servers fail afterwards, and check that `Instance::stop` still stops the object store WAL and returns the leader services error. Rename `skip_segment` to `record_skipped_segment` and the file store integration test to `test_standalone_object_store_wal_survives_restarts_on_file`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Yingwen
|
3a4c24a863 |
fix: temporarily disable v2 series scan by default (#9436)
* fix: disable v2 series scan by default temporarily Signed-off-by: evenyag <realevenyag@gmail.com> * test: explicitly enable v2 series scan in sqlness configs Signed-off-by: evenyag <realevenyag@gmail.com> * test: regenerate sqlness results for legacy series scan default Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
a314ac4281 |
fix(mito2): truncate parquet column index min/max for SST writes (#9420)
* fix(mito2): truncate parquet column index min/max for SST writes The SST writer and the bulk part encoder disabled column index truncation together with statistics truncation in #6977. Mito never reads the column index, but every page of a large string or binary column still stored its full min and max values, uncompressed. Truncate the column index to the parquet default of 64 bytes and keep chunk statistics untruncated, since primary key pruning decodes them. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(mito2): note that the column index must not be used to decode primary keys Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
8553db3f27 |
feat(mito): wait for WAL durability before publishing a manifest watermark (#9410)
* feat(mito): wait for WAL durability before publishing a manifest watermark A log store that acknowledges appends before their entries are durable can hand out entry ids that a crash loses. Before a flush, a full truncate or a discard of unflushed data records an entry id in the manifest, Mito now waits on `LogStore::wait_durable` for that id. The flush waits after its SSTs are written and observes its cancellation, so a drop or truncate queued behind it is not held up by the upload. Add engine tests of the object store WAL across restarts and of the barrier, and a log store testing hook that stores an object and then reports its create as failed. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito): synchronize the WAL barrier tests on store signals The barrier tests waited with fixed sleeps, which do not establish that a flush or truncate has reached its durability wait, and the lost backlog test could stop the store before the seal was handled. The object store WAL gets testing hooks that observe parked creates and durability waits, and one that ends the actor as a crash would; the tests wait on them, crash the store and the engine before a reopen until nothing holds the old store, and cover a truncate or discard of unflushed data that completes once its entry is durable. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito): rename the recovery state and the crash test in the WAL recovery tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito): drop a comment that restates the empty recovery state Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
a99cb56d71 |
feat(datanode): wire the object store WAL into the datanode and standalone (#9386)
* feat(datanode): wire the object store WAL into the datanode and standalone A datanode configured with `provider = "experimental_object_store"` now builds an `ObjectStoreLogStore` instead of failing with the placeholder "not supported yet" error. The builder resolves `storage_provider` against the object store manager (empty selects the default store, an unknown name is rejected as an invalid `storage_provider`), opens the store under the node id and the standalone generation, stops it if a later build step fails, and hands it to `Datanode`, whose shutdown stops it after the region server. Datanode shutdown is now best-effort: every step runs and the first error is returned. A datanode with a metasrv client still rejects the provider. The standalone bootstrap builds `WalProvider::ObjectStore` with the same derived node prefix before the metasrv WAL conversion, so the WAL options it allocates match the prefix the store runs under. `log-store` exports `ObjectStoreLogStore` and its testing hooks. Mito's test utilities can build an engine on the object store WAL over an in-memory object store, and new engine tests cover create, reopen, replay isolation between regions, batch open, drop and offline cleanup, the latest entry id after replay, and the rejection of object store WAL options on a Raft Engine log store. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(datanode): stop every region engine before the object store WAL The region server returned at the first engine whose stop failed, so the engines after it in map order, Mito included, were never stopped before the datanode stopped the object store WAL. It now attempts every engine, returns the first error and logs the later ones. `ObjectStoreLogStore` is also exported from the `log-store` crate root, and its callers import it from there. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
079ffec116 |
chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: remove unused RouteInfoCorrupted error Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
  
|
75bd8e9ce6 |
feat: add HDFS object storage backend (#8701)
* feat: add HDFS object storage backend Signed-off-by: Minghan2005 <cambrianocean@gmail.com> * fix: make HDFS storage operations durable Gate the native HDFS backend behind an explicit feature. Publish writes through same-directory temporary files and atomic HDFS Rename2 replacement, and provide streaming copy fallback for COPY_REGION. Add regression coverage for interrupted writes and the region-copy path. Signed-off-by: Minghan2005 <cambrianocean@gmail.com> * ci: run HDFS object store tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: note HDFS temporary file cleanup follow-up Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: enable HDFS object storage by default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: remove redundant HDFS build feature notes Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: Minghan2005 <cambrianocean@gmail.com> Signed-off-by: jeremyhi <fengjiachun@gmail.com> Co-authored-by: Minghan2005 <cambrianocean@gmail.com> Co-authored-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
ffdd6d09a6 |
perf(index): cut allocations when building bloom and inverted indexes (#9359)
* perf(index): build bloom filters from element hashes without per-token allocation Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(index): speed up inverted index building with hashed buffers and sync pushes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(index): use BuildHasher::hash_one for element hashes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(index): require callers to act on spill requests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): insert bloom hashes one by one to keep segment set capacity bounded Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): add index build and bloom search benchmarks Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): keep applier setup out of the bloom search benchmark Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): skip empty spills and keep the old inverted sort memory estimate Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(index): stream fulltext token hashes and test spill dispatch Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(index): note non-ASCII tokens still allocate in analyze_text_hashes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(index): narrow the allocation note to case-insensitive non-ASCII tokens Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
bc5ad4b416 |
perf(mito2): batch index page loads and share cached bloom metadata (#9360)
* perf(mito2): load missing index pages of all ranges in one read Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(mito2): drop unused IndexCacheMetrics::merge Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(mito2): share cached bloom filter metadata instead of cloning it Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): only merge adjacent ranges when batch reading index pages Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): skip empty ranges when batch loading index pages Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(mito2): fetch index ranges concurrently Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): return Arc<BloomFilterMeta> from RecordingReader Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
0a03e64d8c |
perf(index): batch bloom filter searches across row groups (#9361)
* perf(index): search bloom filters of all row groups in one read Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(index): bound the filter bytes of each batched bloom search Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(index): count batch filter bytes exactly as the search reads them Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(index): allow single-range groups in the budget test Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(index): cover filters shared across segments in the budget test Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
411c0f173e |
fix(mito2): stop parallel flat scan tasks once the receiver is dropped (#9368)
* fix(mito2): stop parallel flat scan tasks once the receiver is dropped spawn_flat_scan_task ignored send errors, so after a query was cancelled the detached task kept reading its source to EOF. Break out of the loop when the receiver is gone. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(mito2): log when a parallel scan task stops on a dropped receiver Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(mito2): include the send error in the parallel scan stop log Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Weny Xu
|
5ffd01a70a |
fix: preserve primary key order when syncing columns (#9189)
* fix: preserve primary key order when syncing columns Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: reject invalid SyncColumns metadata Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
3c2aac0a55 |
feat: add manual series index reconciliation (#9323)
* feat(mito): support manual series index reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * feat(storage): route series index build requests Signed-off-by: evenyag <realevenyag@gmail.com> * feat(admin): add BUILD_SERIES_INDEX Signed-off-by: evenyag <realevenyag@gmail.com> * test: specify compaction type in series index fixtures Signed-off-by: evenyag <realevenyag@gmail.com> * test: correct series index SQL fixtures and error assertions Signed-off-by: evenyag <realevenyag@gmail.com> * test(compat): preserve legacy index rebuild across upgrades Signed-off-by: evenyag <realevenyag@gmail.com> * test(sql): cover series index admin validation Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): bound series index maintenance queue Signed-off-by: evenyag <realevenyag@gmail.com> * docs: remove series index how-to guide Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove index build upgrade compatibility case Signed-off-by: evenyag <realevenyag@gmail.com> * fix: address series index reconciliation review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * fix: bound manual series index reconciliation admission Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update greptime-proto to merged index build options Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Ning Sun
|
20619273ee |
chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates (#9303)
* chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates Move the workspace from the pinned nightly-2026-03-21 to stable 1.96.1 and drop all 23 '#![feature]' gates across 13 crates, rewriting the still-unstable API usages with stable equivalents: - try_blocks: closures / an async block (table, query, common-function, servers) - duration_constructors: Duration::from_secs(n * 86400) / (n * 60) - iterator_try_collect: collect::<Result<Vec<_>, _>>() - box_patterns: as_deref() + matches! chains (sql) - error_iter: error_chain_root() source-chain walker (common-error); sources() includes the error itself, so the walker never panics - int_roundings: div_floor -> div_euclid (equal for positive divisors) - iter_partition_in_place: stable sort_by_key partition helper (index) - hash_set_entry: HashSet::insert bool / contains+insert - trait_alias: lifetime-parameterized dyn FnOnce type aliases (puffin) - string_from_utf8_lossy_owned: from_utf8_lossy(&v).into_owned() - never_type: Infallible (common-recordbatch) - debug_closure_helpers: closure-backed DebugFmt newtype (mito2) - binary_heap_pop_if: peek().is_some_and() + pop() - exclusive_wrapper: drop Exclusive; C: Send + Unpin already in bounds - stmt_expr_attributes: stale gate, no usages Also fix release-dev-builder-images.yaml, which parsed rust-toolchain.toml with a date-only regex and would produce empty image versions with a stable channel; it now extracts the full channel token. Dev-builder images verified against stable 1.96.1 (image build, default-toolchain behavior, binstall/nextest, riscv64 and android targets, in-image cargo check). Validated on 1.96.1: cargo check --workspace --all-targets, clippy --workspace --all-targets --all-features -D warnings, cargo fmt --check, and nextest on all 13 affected crates (4586 passed). Part of #9289. Depends on #9298 (fuzz nightly quarantine) merging first. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: update flake checksum * chore: use wild for linker in flake --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
dennis zhuang
|
b9f991502c |
refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
c7fa48ef95 |
feat(mito): limit approximate series index disk usage (#9313)
* feat(mito): limit series index disk usage Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): enforce series index quota during reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): remove series index disk budget layer Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): simplify series index limit to estimated usage Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): trust index catalogs when loading snapshots Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): minimize series index disk limit changes Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): keep series index cleanup running at capacity Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): avoid no-op index clones and stabilize capacity tests Signed-off-by: evenyag <realevenyag@gmail.com> * fix: update config API expectation and stabilize index build tests Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 shuiyisong
|
20dde2601f |
feat: enable native histogram ingestion by default (#9301)
* feat: enable native histogram ingestion by default Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Weny Xu
|
a35f9d5c78 |
fix: address Windows test failures and run full Windows CI (#9305)
* fix: use relative object keys for Windows filesystem access Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: cover Windows path and time limits in full test CI Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: use relative keys in metadata snapshot tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 
|
1d8d95c12a |
chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks cargo-udeps requires a nightly toolchain and its pinned version (0.1.61) no longer detects unused dependencies against current cargo internals — unused deps have landed on main undetected (e.g. humantime in common-frontend since #6689). cargo-shear is a standalone static analyzer that runs on any toolchain. - Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' / 'cargo shear --fix' and retire scripts/fix-udeps.py - CI: install cargo-shear in the check-udeps job; drop the build cache and protoc steps (cargo-shear never compiles) - Remove ~150 unused dependency declarations found by cargo-shear, move misplaced deps to the correct sections, drop orphaned [workspace.dependencies] entries (arrow-cast, rustc-hash) - Add [package.metadata.cargo-shear] ignored entries with explanations for dependencies that are structurally required despite no textual reference: sqlparser (required by sqlparser_derive expansions in datatypes, common-query), common-error (required by common-macro's stack_trace_debug expansions in session, tests-fuzz), k8s-openapi (feature-pinning for the transitive kube dependency in tests-fuzz), tikv-jemalloc-sys (link-only, enables jemalloc profiling features in common-mem-prof), protobuf (required by build.rs-generated bindings in log-store) - Drop the obsolete [package.metadata.cargo-udeps.ignore] sections Part of #9289 Signed-off-by: Ning Sun <sunning@greptime.com> * fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(postgres): return empty responses for comment-only SQL (#9295) fix(postgres): handle parsed empty queries in both protocols Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> * ci: create docs follow-up issue on PR merge instead of on label (#9237) * ci: create docs follow-up issue on PR merge instead of on label The docbot workflow previously created a docs-repo issue as soon as the 'docs-required' condition was detected (PR opened/edited with the docs checkbox ticked), even if the PR was never merged. Now the workflow also triggers on PR 'closed': - opened/edited: only manage the docs-required/docs-not-required labels - closed: create the docs issue only when the PR was actually merged and carries the docs-required label This also lets maintainers control issue creation by manually adding or removing the docs-required label before merging. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on docs issue creation timing - Only touch docs labels when the docs checkbox state actually changed in an edit. Previously, editing any other part of the PR body while the checkbox stayed checked removed the docs-required label, silently dropping the docs follow-up now that issue creation happens at merge. Unchanged checkbox now leaves labels untouched, which also preserves manual label overrides. - Do not trust the closed event's stale label snapshot at merge time: re-read the live PR via the API and create the docs issue if the docs-required label is present OR the checkbox is ticked in the current body. - Make the workflow concurrency group action-aware so a merge run does not cancel an in-flight label update from an edit run. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: make docs-required label the single source of truth at merge The label-OR-checkbox merge condition could not distinguish an intentional opt-out from an unfinished label update: removing docs-required while the checkbox stayed checked still produced an issue, and unchecking the box could still produce one if the merge read the stale label before the edit run removed it. At merge time, wait for any pending docbot runs on the PR head SHA to finish their label updates (bounded to 5 minutes), then decide solely by the live docs-required label. Adds actions: read permission for listing workflow runs. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> * perf(promql): push label filters into grouped join inputs (#9280) * perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(ci): build tests-integration lib with meta-srv/mock (#9299) * fix(ci): build tests-integration lib with meta-srv/mock tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but the dependency carrying the mock feature sits in [dev-dependencies]. Builds that only touch the lib, such as the apidoc job's cargo doc --workspace, resolve meta-srv without mock and fail with E0432. --all-targets builds unify dev-dependency features, which is why check, clippy and nextest stayed green. Move the mock-enabled meta-srv entry back to [dependencies]. The other testing features moved out in #9072 are not needed by the lib and stay in [dev-dependencies]. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(repartition): split per-case repartition tests test_repartition_metric ran four format/primary-key-encoding cases in a single test function, and test_repartition_mito ran two format cases. Each case builds its own 3-datanode cluster and runs a full repartition plus GC cycle, so on S3 the metric test took 165-178s against the 180s nextest terminate-after. Merge queue runs failed on it at random. Split each case into its own test. Cases were already independent, so they now run in parallel and each stays far inside the timeout, and a failure points at one encoding instead of four. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(json2): support altering JSON2 settings (#9029) * feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> * fix: keep compaction pruning, metadata, and index work on compact runtime (#9304) * fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add AI matching, classification, and scoring functions (#9300) * feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: share logical table batching with OTLP metrics (#9288) * feat: share logical table batching with OTLP metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: unify pending rows batch acknowledgement policy Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align logical batcher example configuration expectations Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker channel defaults to 65536 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): lazily decode dense primary key columns (#9226) * perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: sync lock * fix: shear and check issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
e91faa9df8 |
perf(mito2): lazily decode dense primary key columns (#9226)
* perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
a9e2a89b7b |
fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)
* fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
fys
|
045441e3cc |
feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
Lei, HUANG
|
6a084d9a4f |
fix: release completed SST write buffers during compaction (#9243)
* fix: upgrade OpenDAL and honor SST write buffer size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: track removal of the OpenDAL bridge fork Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): honor SST write buffers across upload paths Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
e9dd79d1d2 |
fix(mito2): compare primary key ranges across schema versions (#9205)
* fix(mito2): compare primary key ranges across schema versions Bind FileHandle ranges to the pinned region schema and append cached constant defaults to historical Dense keys. Preserve raw SST statistics, reject inexact bounds, and avoid invalidating views for unrelated metadata changes. Cover schema evolution, default changes, and tombstone retention through real compaction and reopen regressions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): report invalid primary key ranges Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): scope primary key ranges to comparisons Keep raw PK bounds only in FileHandleInner and move schema-aware mapping and caching into task-local comparison contexts. Use explicit contexts for compaction overlap checks, window aggregation, and series scans. Preserve pinned-schema isolation, late statistics, and shared file lifecycle state without rebinding every handle. Cover cache isolation across region owners and adapt range fixtures to real Dense encodings. All 1530 mito2 tests and Clippy for all targets with the testing feature pass. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): cache aligned primary key ranges per file Replace task-local range maps with a single-slot cache in FileHandleInner, keyed by the target schema version. Preserve raw bounds for realignment across snapshots and default changes. Share schema mappers across comparison paths and use copy-on-write SST lists for metadata updates. Simplify range mapping to accept encoded bounds and assert the same-table contract at the file accessor. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): clarify primary key mapper schema snapshot Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): align primary key range fixtures with table contract Remove obsolete cross-table fallback expectations after region validation became a caller contract. Give compaction fixtures matching table identities, including the active-window L1 scenario. Clarify the mapper precondition and format the simplified alignment call. All 1530 mito2 tests and Clippy for all targets with the testing feature pass. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): enable filesystem GC in release unit tests Let unit tests use the filesystem-backed object-store GC path regardless of optimization profile. Keep the production release GC selection unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito-codec): skip release value decoding in PK prefix counts Validate field values only in debug builds and unit tests while keeping boundary, truncation, and trailing-byte checks in every build. Cover the linked library in debug and release integration tests, and verify that release unit tests still perform value validation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito-codec): remove redundant prefix integration tests Retain the codec unit tests and cross-schema compaction regressions while dropping the standalone build-profile test file and its release-only invalid-value expectation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: wait for MySQL to accept authenticated TCP queries Add a healthcheck using the configured test account and database. Docker Compose --wait previously only observed container startup because the fixture image had no healthcheck, allowing metasrv to connect before MySQL initialization completed. Verify readiness with SELECT 1 over TCP rather than the initialization socket. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
f40d20110c |
feat(mito2): split SWCS output files by size threshold (#9259)
* feat(mito2): split SWCS output files by size threshold Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): use resolved SWCS output size options Read the SWCS output file size threshold from the resolved region options so database-level compaction settings are honored. Align the existing picker test with this configuration source. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
43ce07fa69 |
fix(mito2): cancel cache construction for incomplete scans (#9254)
* fix(mito2): cancel cache construction for incomplete scans CacheBatchBuffer spawned the background concat task and dropped its join handle. A scan that was cancelled or failed therefore left the task alive: it kept compacting already queued batches, and while waiting for a range result memory permit it held them, even though without a finish command the result can never be put into the cache. Keep the handle and abort it when the buffer is dropped. The handle is cleared once the task owns the finish command, so a completed scan still populates the cache after its stream is dropped. Abort does not preempt a concat that is already running; it takes effect the next time the task is polled. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(mito2): wait for the permit park before cancelling the buffer An empty buffered_batches only proves the batches were enqueued, so the cancellation test could abort a concat task that had never been polled. Count acquisitions that find too few permits, a test-only signal, and drop the buffer once the task has reached that wait. Nothing awaits between the check and the parking, so an observed increment means the caller is about to wait for permits the test holds for the rest of the case. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
798adb64e7 |
feat(mito2): make range index reads and builds opt-in (#9219)
* feat: add config flag to control range index reads Signed-off-by: evenyag <realevenyag@gmail.com> * fix: disable range index builds when configured and default to off Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
e99f75f9b3 |
fix(mito2): hold the bulk compact permit for the whole blocking task (#9252)
CompactDispatcher acquired a borrowed permit inside the async wrapper and dropped it as soon as the blocking task was submitted, so the semaphore never bounded the compactions that actually ran. Acquire an owned permit and move it into the blocking closure so it covers the queue wait and the merge itself. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Sainath Singineedi
|
fbbc017be4 |
feat: expose region min/max timestamp in region_statistics (#9060)
* feat: expose region min/max timestamp in region_statistics Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> * test: cover region_statistic time range assembly and projected values Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> --------- Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> |
||
|
Lei, HUANG
|
cb6c7a5c6d |
perf(mito-codec): streamline sparse label extraction and buffer sizing (#9217)
* perf(mito-codec): validate sparse labels once Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito-codec): reserve complete sparse label chunks Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
fys
|
5b631fd8eb |
feat(json2)!: remove nullable and default type hint options (#9213)
feat(json2): remove nullable and default type hint options |
||
|
Lei, HUANG
|
ab21f684bf |
perf(mito2): lazily materialize sparse primary key tags (#9199)
* chore: bump memcomparable to d8fb3558 for bytes I/O optimizations The new revision (v0y4g3r/memcomparable main, PR #1) includes: - Deserializer::read_bytes_into for reusable-buffer decoding - read_bytes 32B pre-allocation (no more 0→8→16→32 realloc chain) - bulk put_slice in both normal and reverse modes - Error::Eof instead of panic on truncated input Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): extract sparse tag values lazily without full PK decode Replace the full primary-key decode in two hot consumers of SortField::deserialize with column-lazy extraction over SparsePrimaryKeyView, eliminating the per-label owned values (Vec<Value>, one heap allocation per string) for every distinct key: - flat_format: DecodedPrimaryKeys keeps sparse keys encoded and extracts each tag column directly from the raw dictionary bytes into a reusable buffer. FlatConvertFormat (flush/compaction write path) builds all projected tag columns in one pass per key, sharing offset discovery between columns; the read path (file_range filters) keeps per-column lazy extraction. - series_index writer: builds rows from the reserved prefix and only the indexed tags via a shared offsets cache and buffer, instead of decoding all labels and copying out a few strings. - mito-codec: add SparsePrimaryKeyView::reserved_value for typed table_id/tsid extraction. Persisted formats are unchanged; values pushed to column builders are identical (NULL vs empty string, reserved column types, duplicate dictionary entries and consecutive-run order are preserved). Benchmark (bench_pk_tag_column, 4096 rows, 40 labels per key, Criterion 30 samples): old = full decode + per-column builds, new = lazy extraction (+ one-pass for multi-column projection). | Workload | Before | After | Change | | --- | ---: | ---: | ---: | | 1 of 40 tags, 4096 distinct keys | 6.95 ms | 803 us | -88% | | 1 of 40 tags, 128 distinct keys | 206.8 us | 29.2 us | -86% | | 10 of 40 tags, 128 distinct keys | 236.2 us | 75.5 us | -68% | | 40 of 40 tags, 128 distinct keys | 314 us | 215.2 us | -31% | | 1 of 10 tags, 4096 distinct keys | 1.75 ms | 207.6 us | -88% | Also adds a decode_sparse benchmark group to mito-codec's bench_sparse_encoding covering 10/40 labels x 10/24/64-byte values. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): share sparse tag scans across filters Batch missing tag columns used by simple filters and partition pruning, while retaining single-column extraction and skip-mode semantics. Replace the optional reserved-byte accessor with typed table_id/tsid reads from the validated prefix. Add full-decode equivalence tests for sliced dictionaries, NULL/empty labels and cross-key scratch reuse, plus series-index roundtrips across batches. Benchmark real precise filtering with 4096 rows: 10 tag predicates drop from 307.85 to 162.49 us; 40 predicates drop from 894.03 to 550.55 us. All 1459 mito2/mito-codec tests pass. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): benchmark sparse tag predicate count scaling Add a dedicated precise-filter benchmark group for 1, 2, 4, 8, 16, and 32 predicates with both 1 and 32 rows per primary key. Keep 4096 rows, 40 labels, and full selectivity constant across cases, and validate the workload outside the timed section. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |