mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-29 23:16:57 +00:00
1e97a0e7c3e9c2e6023c12bbfe4b04e283f0aaa5
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dragon Roar
|
88cb2a986e |
fix: reclaim HDFS temporary files left behind by a datanode crash (#9616)
* fix: reclaim HDFS temporary files left behind by a datanode crash Signed-off-by: lczllx <2181719471@qq.com> * fix(mito2): stop the temporary file sweep from starving GC jobs Signed-off-by: lczllx <2181719471@qq.com> * feat(mito2): add metrics for the temporary file sweep Signed-off-by: lczllx <2181719471@qq.com> * refactor(object-store): share one definition of the temporary file name Signed-off-by: lczllx <2181719471@qq.com> * fix(mito2): skip the temporary file sweep when no GC job is allowed Signed-off-by: lczllx <2181719471@qq.com> * refactor: gate the temporary file sweep behind the hdfs-object-store feature Signed-off-by: lczllx <2181719471@qq.com> * fix(mito2): sweep temporary files when a create opens an existing region Signed-off-by: lczllx <2181719471@qq.com> * docs(config): document the temporary file sweep in the GC settings Signed-off-by: lczllx <2181719471@qq.com> * refactor: align the temporary file sweep code with the surrounding style Signed-off-by: lczllx <2181719471@qq.com> --------- Signed-off-by: lczllx <2181719471@qq.com> |
||
 dennis zhuang
|
4645ab9ed7 |
perf(mito2): gather last rows with one take per input batch (#9737)
* perf(mito2): gather last rows with one take per input batch FlatLastTimestampSelector buffered a single-row slice per series and concatenated up to 8192 of them. Concatenating slices with dictionary columns merges the dictionaries once per slice, which made high-cardinality instant PromQL queries using the LastRow scan path (#9034) slower. Select the rows of series that end inside a batch with one take instead. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(mito2): cover duplicate last timestamps in the take path Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): group last rows by primary key bytes Flat batches may assign different dictionary keys to the same primary key, for example when binary keys are wrapped with identity keys. Merge adjacent ranges with equal key bytes before selecting last rows. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 jeremyhi
|
7397ab6b23 |
perf(datanode): send the mito puts of one request to each worker at once (#9692)
* perf(datanode): send the mito puts of one request to each worker at once The region server handled the puts of a request to several regions as separate requests. A Mito worker woke on the first of them and wrote it alone; in the durable object store WAL mode its append then blocked the worker until the next seal, so the rest of the request went into the next object, and clients writing a partitioned table locked into one phase that wastes a tick. When every put of a request goes to a ready mito region, the region server now hands them to `MitoEngine::put_regions_batch`, which submits the puts of each worker in one message, so the worker writes its regions of the request in one batch and one WAL append. Requests to one region and to other engines are handled as before. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): await each region of a batched mito put on its own task The batched path waited for the regions of a request in order and sent the worker messages one after another, so an error or a full worker queue of one region held back the others, and its metrics counted the request as a whole. `MitoEngine::put_regions_batch` now sends the worker messages concurrently on their own task and returns each region's result. The region server awaits and accounts every region on its own ingest task, as the per-region path does: the first error returns at once, and each region records its latency, changed rows or failure and is named in its error. A request with several puts to one region keeps the per-region path. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): spawn the accounting of batched mito puts from the batch task The request spawned the per-region accounting tasks only after the batch task returned, so a request dropped in between left accepted writes unaccounted. The batch task now spawns them and the request only joins their handles. Also state in the architecture invariants that the region server's batched puts to the metric and mito engines are the one request dispatch that does not go through the RegionEngine trait. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): batch mito puts through the RegionEngine trait Add `RegionEngine::handle_batch_put_requests`, which returns each region's response of the puts of one request. The default handles each put with `handle_request`, so other engines keep their behaviour; Mito overrides it with the one message per worker submission. The region server calls the trait method instead of downcasting to `MitoEngine`, and the architecture invariants no longer need an exception. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): trace each region of a batched mito put Attach the per-region `RegionServer::handle_region_request` span of the per-region path to each region's task of a batched mito put, so the region ids stay in distributed traces. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): let a batch put response answer for several regions `RegionEngine::handle_batch_put_requests` now returns each response with the regions it answers for. The default and Mito answer each region on its own; an engine that writes several puts together, such as the metric engine's batched puts, can answer them with one response. The region server accounts a response for each of its regions. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): batch any request with several mito puts Take the batch path for a request with at least two puts to ready mito regions, also when they all target one region: a request of many puts to one region otherwise still sends one worker message per put. Each put keeps its own response, affected rows and error. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): account batched mito puts in one task Await the responses of a batched mito put in the batch task with a FuturesUnordered instead of one spawned task per put. The task accounts each response as it completes, hands the first error back to the caller at once and keeps accounting the rest, so a request of many puts no longer spawns a task per put. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(mito2): name the batched worker message BatchWrite Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(mito2): submit batched writes on the ingest runtime Grouping and submitting the writes of a request is part of ingestion, so the task runs on the ingest runtime and takes the write task class when the workload scheduler is enabled. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(datanode): share ingest result accounting across put paths The per-region path, the metric batch path and the mito batch path each recorded the changed rows or the failures of an ingest response. They now call one helper, so a metric or label change happens in one place. A batch put response lists the region of each put it answers for, so the failure count stays per put when two puts go to one region; the trait documentation now says so. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * fix(mito2): give batched puts of a stopped worker the worker's error A batched write message that could not reach a stopped worker was dropped, so its puts failed with "Failed to recv result" while a put submitted on its own fails with "Worker <id> is stopped". The worker now returns the message it could not take, and each of its puts receives the WorkerStopped error. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(datanode): look up the batch latency histogram once The label of the request latency histogram is the same for every put of a batch, so the batch task looks it up once instead of per observation. Signed-off-by: jeremyhi <jiachun_feng@proton.me> --------- Signed-off-by: jeremyhi <jiachun_feng@proton.me> |
||
 Yingwen
|
b3dc5674e9 |
feat(mito2)!: account SST decoding against shared scan memory limits (#9101)
* feat(mito2): add shared estimated SST scan memory budget Signed-off-by: evenyag <realevenyag@gmail.com> * feat: unify scan memory limits with configurable accounting modes Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): reuse query runtime memory pool for series merges Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): wait for cancelled scan reservations before exhaustion check Signed-off-by: evenyag <realevenyag@gmail.com> * docs(mito2): explain parallel scan cancellation cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): deprecate waiting for scan memory quota Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(recordbatch): remove scan memory wait handling Signed-off-by: evenyag <realevenyag@gmail.com> * fix(table): initialize memory pool in scan test fixtures Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Lei, HUANG
|
3b442adba2 |
feat(mito): decouple compaction unit execution and publication (#9131)
* feat(mito): decouple compaction unit execution and publication Execute shared-input dependency groups as independent local compaction units. Publish and acknowledge each unit separately, retain successful sibling results, and release each unit's input leases after worker apply. Preserve picker priority, panic notifications, snapshot-free execution context, and indexed input leases. Include regression coverage for independent visibility, purge, cancellation, partial failure, and restart recovery. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): use crate visibility for compaction unit items Replace the newly introduced pub(super) declarations with pub(crate), and make CompactingFiles::try_reserve private since it is only used within its module, per docs/style-guide.md. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): simplify compaction unit state and test fixtures Keep local input leases in the task and derive running units from tracked and pending work. Replace the single-implementation task trait with an inherent async method and align exposed state type visibility. Use snapshot-free region fixtures and named test scenarios, and cover independent input purging across three units with bounded parallelism. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): align compaction task and test module names Rename CompactionTaskImpl to CompactionTask now that the CompactionTask trait has been removed and the Impl suffix no longer marks a trait implementation. Rename the state test module from lease_tests to tests to follow the test module naming convention. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): align compaction task debug label The Debug impl reported "TwcsCompactionTask" while the type is now CompactionTask. Report the type name for consistency. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): label compaction elapsed metric per published unit Record greptime_mito_compaction_total_elapsed once per published unit and label it by type: a local dependency unit or a remote job. Update the metric docs and group the Grafana compaction OPS/latency panels by type. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito): clarify compacting file lease sharing `CompactingFiles::for_inputs` clones the shared per-file leases instead of transferring them. Document that the original collection keeps its leases, so an input is released only after every owner drops its lease. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito): document local compaction unit map key Document that `LocalCompaction::units` is keyed by each unit's execution plan ID, which matches `CompactionExecution::plan_id()` and the `plan_id` carried by unit notifications. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): rename submit_prepared_compaction The function submits remote jobs but only prepares local unit tasks; the caller dispatches them after installing the returned phase. Rename it to `prepare_or_submit_compaction` and document that ordering. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
18634e17bb |
refactor(mito): retain region compaction and DDL lifecycle state (#9498)
* refactor(mito): retain region compaction and DDL lifecycle state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): finalize opening batch in the worker on registration Clean up the `opening_regions` entry and reply to its waiters in the same serialized worker step that registers an opened region, instead of waiting for the background opener to resume after the registration acknowledgement. Previously the entry was only removed by the background opener, so a close followed by a reopen could join the stale opening batch and receive a false success after the region no longer existed. Replying from the worker closes that window because registration and waiter cleanup are now atomic with respect to the worker loop. Add a test-only `EventListener::on_region_open_registered` hook and a regression test that holds the first background opener to verify a reopen does not wait for the previous open task. On failure the region is still stopped and waiters receive `RegionClosed`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): unregister drop runtime state before WAL cleanup The region leaves the region map before its WAL entries are obsoleted. A failure in `wal.obsolete(...).await?` returned early and skipped `cleanup_dropped_region_runtime_state`, so the resident compaction, flush, and index scheduler entries were never unregistered. Since the region is no longer in the map, a later close cannot clean them up either, leaking the entries for the rest of the worker's lifetime. Move the runtime-state cleanup to immediately after the region leaves the map, before the fallible WAL cleanup. The schedulers only touch their own state and do not depend on the region map or WAL, so the reordering is safe and leaves the success path unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): tighten visibility of new compaction state items Keep `ActiveCompaction::idle` private (it is only used inside `state.rs`) and expose the other items added by the resident-region refactor as `pub(crate)` instead of `pub(super)`, per the style guide. Also drop a stale comment in `on_compaction_finished`: DDL handoff now happens in `take_ready_ddls` before this block, which only runs when no DDL is queued. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): skip compaction scheduling while a region is truncating `handle_flush_finished` dispatches queued DDLs before scheduling compaction. A truncate queued behind a committing flush marks the region as `Truncating` synchronously, but `schedule_compaction` only skipped staging states, so it could schedule a new automatic compaction while the truncate was still running. Add `MitoRegion::is_truncating` and skip compaction scheduling for a truncating region. Add a regression test that queues a full truncate behind a committing flush and asserts no compaction is scheduled. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): log DDL-fenced compaction rejection The refactor dropped the log emitted when a region was skipped because of pending DDL requests. Automatic triggers carry no waiter, so the rejection became silent and could not be told apart from a scheduling interval that had not elapsed. Log the queued and executing DDL state before cancelling. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito): document DDL reply fence invariants Explain why fence_ddl_replies installs executing_ddl before handing off the batch and why CompactionDdlComplete carries both a scheduler-wide generation and a per-slot request_id. The generation fences replies from superseded batches or region incarnations, while request_id makes completion idempotent so only the last distinct reply releases the fence. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): rename DDL completion request_id to request_index The value is the DDL's index within the handed-off batch, not a globally stable request identifier; request_id suggested otherwise. Rename it to request_index across the scheduler, worker dispatch, and tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Yingwen
|
c7fa48ef95 |
feat(mito): limit approximate series index disk usage (#9313)
* feat(mito): limit series index disk usage Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): enforce series index quota during reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): remove series index disk budget layer Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): simplify series index limit to estimated usage Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): trust index catalogs when loading snapshots Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): minimize series index disk limit changes Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): keep series index cleanup running at capacity Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): avoid no-op index clones and stabilize capacity tests Signed-off-by: evenyag <realevenyag@gmail.com> * fix: update config API expectation and stabilize index build tests Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
7c85798ad0 |
feat(mito2): reconcile series indexes in background (#9086)
* feat(mito2): reconcile series indexes in background Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): clean up series indexes published during region drop Signed-off-by: evenyag <realevenyag@gmail.com> * fix(config): align series index examples with upstream enable flag Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): run series index tasks on compaction runtime Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
b8e0f8e62e |
feat(mito2): add series index planning and builders (#9085)
* feat(mito2): add series index planning and builders Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): track window coverage and separate index builds Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): normalize index inputs and bound SST window expansion Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): simplify series index source summaries Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): assign series index deduplication TODO Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): initialize skip_wal in series index tests Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
6fb5d2ebad |
feat(mito2): add series index catalog and lifecycle foundation (#9053)
* refactor(mito2): add series index catalog and lifecycle components Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito2): restore series index catalogs on region open Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify series index foundation and maintenance Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): separate series index purge task and simplify tests Signed-off-by: evenyag <realevenyag@gmail.com> * docs: defer experimental series index configuration examples Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): make series index maintenance interval configurable Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): correct series index cleanup on close and drop Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): revert drop test changes Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config API expectation for series index settings Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use tokio unbounded channel for series index purger Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): simplify review test scope and clarify index config Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
a3a0db63b8 |
feat(mito2): add series index searcher (#8926)
* feat(mito2): add series index searcher Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): use parquet push decoder for series index Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): handle evolved series index schemas Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
4126cf99b6 |
refactor(mito2): rename pk index to series index (#8893)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
cb30837cd5 |
feat: add incremental primary key index writer (#8788)
* feat: initial implementation of the pk index writer Signed-off-by: evenyag <realevenyag@gmail.com> * perf: optimize primary key index writer Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add todo Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito2): track series count in pk index metrics Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify pk index writer cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): clean up aborted pk index writers Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
eb8ae171cc |
refactor(mito2): split compaction scheduler modules (#8698)
* refactor(mito2): split compaction module into scheduler/reader submodules Extract the compaction scheduler lifecycle (scheduler, status, phases, execution, SST reservations, pending requests) and its tests out of compaction.rs into compaction/scheduler.rs and compaction/scheduler_test.rs. Split the remaining helpers by responsibility: - estimate_compaction_bytes/refresh_picker_output move to scheduler.rs, the only call site - get_expired_ssts moves to picker.rs, shared by the TWCS and window pickers - CompactionSstReaderBuilder/time_range_to_predicate/ts_to_lit move to the new compaction/reader.rs The root compaction.rs keeps the shared output types and find_dynamic_options, and re-exports the moved types so existing call paths stay unchanged. Pure code motion, no behavior change. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): separate compaction scheduling from execution details Turn compaction/scheduler.rs into a directory module to make the scheduling flow easier to review: - scheduler.rs keeps the pure scheduling core: the CompactionScheduler state machine, scheduling entry points, termination chaining, DDL coordination and region lifecycle events - scheduler/planning.rs holds the execution-facing parts: background planning dispatch, picker invocation, plan acceptance, remote/local submission and memory estimation - scheduler/state.rs holds the per-region lifecycle types: CompactionStatus, ActiveCompaction, CompactionPhase, CompactingFiles, LocalCompactionState, CompactionExecution and PendingCompaction Child modules keep access to the scheduler's private methods, so the split is pure code motion with minimal visibility changes (pub(super) only where the parent module or tests reach into child items). No behavior change. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): use absolute scheduler imports Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document compaction scheduler modules Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
3b8f55e490 |
docs(agents): add per-crate guides, architecture invariants, and generated-files list (#8346)
* docs(agents): add per-crate guides, architecture invariants, and generated-files list Add agent/contributor navigation docs modeled on the AGENTS.md convention: - Per-crate AGENTS.md for hot crates (mito2, metric-engine, flow, frontend, meta-srv): module map, read/write paths, change-coupling points, test commands, and gotchas. - .agents/architecture-invariants.md: repo-wide rules that clippy and the style guide do not cover (format compatibility, crate layering, async runtimes, error handling, experimental gating, the DataFusion fork). - .agents/generated-files.md: tool-generated artifacts that must not be hand-edited (sqlness .result, config.md, dashboards, build.rs output, proto). - Anchor the .gitignore CLAUDE.md/AGENTS.md rules to the repo root so per-crate AGENTS.md files are tracked while root-level personal config stays ignored. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: update crate AGENTS.md and fix config.md path Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(agents): fix DataFusion patch layout and SQL query lifecycle order Address review feedback on #8346: - architecture-invariants: the DataFusion sub-crates pin an exact crates.io version in [workspace.dependencies] and are redirected to the fork rev in [patch.crates-io]; the two sections hold different forms, not the same rev. - frontend: the SQL query lifecycle runs the pre_parsing/post_parsing interceptors around parsing, before the per-statement permission check. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |