mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-16 08:52:30 +00:00
ef1298ce026924bbd8f80cf279eedfed06718635
6370
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dennis zhuang
|
ef1298ce02 |
fix(query): accept narrowed nullability in MergeScan remote schema check (#9592)
On append-mode tables a region reports an exact row count. When it is zero, DataFusion's AggregateStatistics replaces the region-side __min_state/__max_state aggregate with a non-null struct literal, so the remote stream advertises the state column as non-nullable. The MergeScan schema check required equal nullability and failed the query. Accept a remote field that is non-nullable where the expected field is nullable, matching DataFusion's own physical-plan schema invariant. Batches are already rebuilt with the expected schema. Widening is still rejected, and nested field nullability is still compared exactly as part of the data type. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 jeremyhi
|
60655a933b |
fix(log-store): persist object store WAL region releases (#9579)
* fix(log-store): persist object store WAL region releases A region's release lived only in memory: a restart after DROP TABLE and before the next object never re-established the dropped region's watermark, so its segments counted as live and pinned garbage collection of the whole prefix. A release now writes a release record (region id and the entry id the region is released through) into an object sealed at once, and returns once that object is durable and indexed, in both acknowledgement modes. Recovery seeds the obsolete watermarks from the release records of the indexed objects before it computes any reclaim boundary. obsolete_all releases in full; the new LogStore::release_region, a no-op by default, releases a dropped region through the largest entry id the store handed out for it, and Mito calls it after obsolete when it drops a region. Objects with release records use format version 2, the version 1 layout with the records at the end of the footer; every other object is still written as version 1. A counter of release records made durable is added. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe persisted object store WAL region releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): carry a failed region release into the next object A release whose batch failed left its record only in an orphan object, so a drop that failed there could still leave the region unreleased after a restart. The store now keeps the release records of failed batches and seals them with its next batch, on a timer tick when no entry is admitted, so the release becomes durable once a create succeeds again. The version 1 and version 2 object fixtures are frozen under tests/compatibility/wal/object_store and loaded by the format tests, and the RFC no longer says that state is persisted only in headers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): complete a region drop whose WAL release fails A drop request is not sent again once the region was removed, so a drop whose WAL release failed used to leave the region half dropped. The drop now completes, cleaning up the region's runtime state and scheduling its files for deletion, and reports the release failure. In the durable mode the object store WAL now creates one more object at stop with the release records of failed and dropped batches, so a graceful stop does not lose a release that failed. The release record counter is described as counting the records of indexed objects. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe the stop-time release create of the WAL actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): state the accepted window of unreleased dropped regions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe release waiters in the WAL actor's outcome table Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail region releases while the object store WAL is failing A failing store keeps its backlog and rejects appends until a create succeeds, so a release would otherwise wait for an upload that is not coming. A release that arrives in that state now fails at once with the retryable error, and one whose batch is still pending when the store enters it fails as well. Neither poisons the store: the record of the first waits for the next sealed batch, like that of a batch that failed, and the record of the second stays in the backlog, so both become durable once a create succeeds. Rejected releases count in rejected_appends_total. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe region releases while the object store WAL is failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the drop crash window in the drop procedure The crash window between the deletion of a table's metadata and a durable release record is a gap of the drop procedure, not of the WAL. Describe it as far as the code shows, name the remedy that belongs in the procedure, cleaning up offline a region a resumed region drop finds not open, and list it under future work. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep rejected release records coalesced while failing A release rejected while the store is failing kept its record for the next sealed batch, but every timer tick then sealed that record alone behind the refused backlog, so a release repeated across ticks grew the backlog without bound. A failing store now keeps such records coalesced and seals them on the first tick after a create succeeds. The fault-matrix case of a release while the backlog is refused now runs with a 10 ms flush interval, repeats the release, and checks that the backlog does not grow and that recovery reads the record back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): keep rejected release records unsealed while failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): cite the drop procedure issue for the crash window Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): let a stop seal rejected release records of a failing store Keeping the records of rejected releases out of sealed batches while the store is failing also kept them out of the seal a stop requests, so a stop whose last attempt of the backlog succeeded dropped them and the released region pinned collection after the restart. A stop now seals them with the backlog. A fault-matrix case rejects a release while the store is failing, stops, lets both creates succeed, and checks the record after a reopen. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the stop of a failing store with rejected releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep release records readable by older object store WAL readers Release records were written as a version 2 footer, which a build without it refuses to read, so rolling a node back after a drop failed to open the whole WAL prefix. They are now the single entry of a segment of the reserved region u64::MAX at the first position of a version 1 object: an older reader indexes that segment as a region it never opens, and recovery reads the segment back with one range read per object that holds one. The segment never holds the reclaim boundary and names no namespace. A durable stop whose last create of release records fails now reports the error instead of returning Ok and losing the records silently. The tests are trimmed: the release unit tests and fault-matrix cases that repeated each other are merged, and the Mito drop tests share one parameterized case. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe release records as a segment of a reserved region Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): report release records a poisoning stop drops A durable stop whose last create of release records met a conflict poisoned the store and returned Ok, losing the records silently. Stop now reports the error whenever release records are dropped after it began, by a rollback or by poisoning. The test of a failed release covers that conflict, and seals its record on a tick after a one-shot create failure, so the next tick cannot fail too. The documentation now says that recovery reads the segments of release records, and that one that does not decode fails recovery. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how recovery reads segments of release records Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): say that stop reports release records a conflict drops Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): restore the transiently failing release create in enqueued mode The fault matrix again covers a release whose create fails transiently in the enqueued mode: the release waits for the repeat under the same sequence and returns once it succeeds. The encode_batch doc now says the release segment is the last segment of the object. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the release segment and count the restored case Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL release Returning the release error from the drop left a metric physical region half dropped: the metric engine stops after the data region and never drops the metadata region or forgets the physical region, and a retried drop finds the data region gone and counts it as done. The release error gave the caller nothing to retry, since the region is already removed and the log store keeps the record for its next object. The drop now logs the error and succeeds. A metric engine test drops a physical region while the WAL releases of both mito regions fail, and checks that both regions and the physical region state are gone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed release no longer fails the drop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL obsolete A store that the release of another region poisoned fails obsolete as well, which left the metric engine with its metadata region undropped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(metric-engine): drop a physical region whose WAL obsolete or release fails Cover a failed data release, two failed releases and a conflicting release that poisons the store, after the regions wrote to the WAL, and check that a store opened later keeps no segment of either region. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed obsolete does not fail the drop either Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): fail the drop on an obsolete error of other WAL providers Only the object store WAL is shared by the regions of a store that the release of one can poison; Raft Engine and Kafka keep failing the drop when obsolete fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a stopped or poisoned store keeps no failed release Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Ning Sun
|
40770ddb72 |
test: only assert genuinely invalid read-preference values at request ingresses (#9576)
The flow and nested-MergeScan ingress tests hardcoded "follower" among the read-preference values that must be rejected, coupling them to the current leader-only ReadPreference enum. Assert only genuinely invalid values there, matching the wire-values test which already limits itself to ["", "unknown"]. This keeps the ingress tests valid for builds that extend ReadPreference with more variants. Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 Lei, HUANG
|
3b442adba2 |
feat(mito): decouple compaction unit execution and publication (#9131)
* feat(mito): decouple compaction unit execution and publication Execute shared-input dependency groups as independent local compaction units. Publish and acknowledge each unit separately, retain successful sibling results, and release each unit's input leases after worker apply. Preserve picker priority, panic notifications, snapshot-free execution context, and indexed input leases. Include regression coverage for independent visibility, purge, cancellation, partial failure, and restart recovery. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): use crate visibility for compaction unit items Replace the newly introduced pub(super) declarations with pub(crate), and make CompactingFiles::try_reserve private since it is only used within its module, per docs/style-guide.md. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): simplify compaction unit state and test fixtures Keep local input leases in the task and derive running units from tracked and pending work. Replace the single-implementation task trait with an inherent async method and align exposed state type visibility. Use snapshot-free region fixtures and named test scenarios, and cover independent input purging across three units with bounded parallelism. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): align compaction task and test module names Rename CompactionTaskImpl to CompactionTask now that the CompactionTask trait has been removed and the Impl suffix no longer marks a trait implementation. Rename the state test module from lease_tests to tests to follow the test module naming convention. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): align compaction task debug label The Debug impl reported "TwcsCompactionTask" while the type is now CompactionTask. Report the type name for consistency. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): label compaction elapsed metric per published unit Record greptime_mito_compaction_total_elapsed once per published unit and label it by type: a local dependency unit or a remote job. Update the metric docs and group the Grafana compaction OPS/latency panels by type. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito): clarify compacting file lease sharing `CompactingFiles::for_inputs` clones the shared per-file leases instead of transferring them. Document that the original collection keeps its leases, so an input is released only after every owner drops its lease. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito): document local compaction unit map key Document that `LocalCompaction::units` is keyed by each unit's execution plan ID, which matches `CompactionExecution::plan_id()` and the `plan_id` carried by unit notifications. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): rename submit_prepared_compaction The function submits remote jobs but only prepares local unit tasks; the caller dispatches them after installing the returned phase. Rename it to `prepare_or_submit_compaction` and document that ordering. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 fengyue-xve
|
a35661d69c |
fix(hdfs): clean up temporary file when an atomic write fails (#9483)
* fix(hdfs): clean up temporary file when an atomic write fails Signed-off-by: fengyue-xve <1759519137@qq.com> * style: apply rustfmt to hdfs layer test Signed-off-by: fengyue-xve <1759519137@qq.com> * fix(hdfs): report earlier write failures and log cleanup failures - Later `write`/`close` calls on a writer whose failed write was cleaned up now mention the original failure instead of the bare "unavailable after close or abort" message. - `abort_and_delete` logs abort and deletion failures with the temporary path instead of discarding them; the original write error is still returned to the caller. - The temporary-file cleanup test now lists the directory holding the temporary file, so it fails if the cleanup is reverted. Signed-off-by: 风月 <1759519137@qq.com> Signed-off-by: fengyue-xve <1759519137@qq.com> --------- Signed-off-by: fengyue-xve <1759519137@qq.com> Signed-off-by: 风月 <1759519137@qq.com> |
||
 Weny Xu
|
36423a95c6 |
fix(operator): reuse catalog table resolution for write retries (#9574)
* fix(operator): inherit logical table partitions when retrying writes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(operator): resolve retried writes through catalog manager Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
dennis zhuang
|
5c4d9eb4f8 |
fix: align fulltext bloom tokens with matches_term and use them for LIKE (#9470)
* fix: align fulltext bloom tokens with matches_term and use them for LIKE The bloom fulltext index stored tokens that did not follow matches_term's word boundaries (`_`, ASCII next to Han, Han substrings), so pruning dropped rows matches_term accepts. Tokens and query probes now share the character classes of matches_term, Han runs are indexed as bigrams, and LIKE patterns use the same probes. The bloom blob is written under a new key; blobs from older versions are no longer probed. Tantivy no longer serves matches_term. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: drop unused lazy_static from index and avoid typo-flagged test tokens Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: link like_probes to tokenize instead of the downstream ScriptTokenizer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: name probe extraction with verbs and document CharClass Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: rebuild legacy fulltext bloom indexes and probe lower(col) The manifest records the fulltext bloom version per column, so ADMIN BUILD_INDEX now rebuilds SSTs whose bloom index predates the current tokens, and ssts_index_meta shows the version. Case-insensitive bloom indexes also store the tokens of the whole lowercased text for rows where it can differ (U+0130, U+212A, sigma), so matches_term(lower(col), ..) and lower(col) LIKE .. use the index. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover legacy bloom index rewrite by compaction in the compat case Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: log outdated fulltext bloom indexes in the consistency check Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: explain the unrecorded fulltext bloom version Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
fengyue-xve
|
09bbe1cf9f |
refactor(promql): centralize PromQL function registration in promql::functions (#9500)
* refactor(promql): centralize PromQL function registration in promql::functions Move the PromQL scalar UDF and UDAF registrations out of the query plan decoder into a single `promql::functions::register_promql_functions` entry point, so plan decoding no longer has to enumerate every PromQL function implementation. The registered function set, aliases (e.g. `prom_holt_winters`) and registration order are unchanged. Add a registration test pinning the full set of `prom_*` scalar functions and the `prom_holt_winters` alias, plus a decode regression test covering the centralized registration path. Fixes #9493 Signed-off-by: fengyue-xve <1759519137@qq.com> * refactor(promql): register Ieee UDFs centrally and address review Rebase onto main picked up #9479, which registered the Ieee comparison and sqrt UDFs from the decoder directly. Register them from `register_promql_functions` instead so the single entry point stays complete, and add the seven `prom_eq`..`prom_sqrt` names to the registration test. Also per review: drop the doc paragraph that explains why the function was moved, drop the redundant `test_serializer_decode_centralized_promql_registration` test and the registration comment in the decoder, move `use std::sync::Arc` up with the imports, and simplify the registration test assertions. Signed-off-by: 风月 <1759519137@qq.com> Signed-off-by: fengyue-xve <1759519137@qq.com> --------- Signed-off-by: fengyue-xve <1759519137@qq.com> Signed-off-by: 风月 <1759519137@qq.com> |
||
 Yingwen
|
00aa65de7a |
fix(query): guard deep expressions during query planning (#9564)
* fix(query): avoid speculative clones in constant normalization Signed-off-by: evenyag <realevenyag@gmail.com> * fix(query): reject deeply nested expressions before optimization Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 sun
|
4ff3296f97 | feat: update dashboard to v0.14.2 (#9570) | ||
|
jeremyhi
|
a08e5bcceb |
fix(log-store): keep the enqueued backlog on permanent object store errors (#9558)
* fix(log-store): keep the enqueued backlog on permanent object store errors On object storage the WAL's object store sits behind OpenDAL's retry layer, which marks every error it returns as persistent. The enqueued mode classified create failures by that status, so a 403 or a missing bucket was repeated every 100 ms on every create in flight, logged nothing, and made appends hang once the sealed batches were full. Object store errors are now classified by kind: denied access, an invalid configuration (how OpenDAL reports a missing S3 bucket) and a write to a missing location are non-retriable. In the enqueued mode such an error makes the store failing instead of poisoning it: the acknowledged backlog is kept under its sequences and bytes, its lowest pending batch is attempted alone with a delay that doubles from 100 ms to 30 s, appends are rejected at once with a retryable error that carries the last object store error, and the first create that succeeds ends the state. Stop uploads the backlog at once and reports a failure as before. The state is logged on entry, at most once a minute while it lasts and on exit, and exported as the `failing` gauge and the `rejected_appends_total` counter. Conflicts, encoding and catalog errors still poison the store; the durable mode is unchanged. An append held back because the sealed batches are full is now counted and timed by the stall metrics in both modes. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how the enqueued object store WAL handles refused creates Describe the enqueued failure contract as the store now implements it: non-retriable object store errors are told apart by kind because OpenDAL's retry layer under the store marks every error persistent, a failing store keeps its backlog and attempts it with backoff while it rejects appends, and stop uploads the backlog at once. Add the failure matrix row, the failing gauge and the rejected appends counter, the sealed-batch admission limit and the stall metrics that now count it, and the fault matrix case counts. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): time the failing attempts where they start and fix two statements The test of the attempts of a failing store now records when each refused write starts and checks the gaps between them, instead of sampling a counter after a sleep of the test task. The delay comment and the RFC describe 30 seconds as the longest delay between two attempts, not as a bound on when appends resume, and the RFC states that the next append is held once seven sealed batches wait. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): drop the wait_until helper that main now defines Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): name errors that need an operator and trim the RFC contract Rename is_non_retriable to requires_intervention: such a create is still repeated, only slowly, until someone fixes the credentials, configuration or bucket. Name the backoff test after what it asserts, cut the test of the lowest pending batch to two failed rounds since the cadence is covered by the backoff test, and shorten the RFC failure contract to the contract and its rationale, leaving error mappings, request counts and logging cadence to the code comments. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): tie the slow repeat to errors requiring intervention Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 discord9
|
0e40281262 |
fix(promql): stop claiming scalar calculate preserves order (#9523)
* test: cover scalar calculate order contract Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): stop claiming scalar calculate preserves order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(test): describe scalar ordering regression as pre-fix behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Ning Sun
|
f41e86f43a |
feat(meta-client): forward actor and event context for region follower RPCs (#9568)
The manage-region-follower chain submitted its procedures with an empty executor context: `ProcedureServiceHandler::manage_region_follower` accepted no query context, the operator built an `ExecutorContext::default()`, and `MetaClient` dropped the context before delegating to `RegionFollowerClient`. The submitting user (actor) and the procedure event metadata therefore never reached the submitted procedures. - `ProcedureServiceHandler::manage_region_follower` now takes a `QueryContextRef`, consistent with `migrate_region`, `gc_regions` and `gc_table`. - `ProcedureServiceOperator` derives the `ExecutorContext` (actor and `TriggerReason::Manual` event input) from the query context. - The `RegionFollowerClient` trait methods now accept `&ExecutorContext` and `MetaClient` forwards it, so client implementations can transport the actor and event context on the wire. - `procedure_event_context`/`procedure_actor` are now public so client implementations can reuse the trusted protocol derivation. |
||
|
discord9
|
75bc4ce46d |
fix(table): stop claiming tag order for logical metric scans (#9522)
* test: cover metric logical scan tag ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): only claim projected primary key prefix ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): keep physical key ordering for metric projections Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(table): compare unknown partition counts explicitly Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(table): model scoped internal TSID scan properties Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
  NightG1owZandNightG1owZ
|
caefcfd012 |
test: add inverted index compatibility case (#9519)
* test: add inverted index compatibility case (#5188) Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * test(compat): assert the newly indexed column stays queryable (#9535) Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * test(compat): narrow to_range to versions carrying #9545 The filtered read added in the previous commit only returns rows once #9545 is present. That fix is on main only (no backport-* labels), so it ships in the v1.4.0 line. Narrow to_range to >=v1.4.0-alpha.0 so the documented downgrade invocation (--to-version v1.1.4) is not broken. Also add the missing trailing newline to the three case files. Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> --------- Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> Co-authored-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> |
||
|
Weny Xu
|
f690bbb204 |
refactor!: remove datanode remote WAL index (#9563)
* refactor: remove datanode remote WAL index Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): update new WAL replay tests after index removal Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
863eebdecb |
feat(frontend): retry transient distributed write failures (#9543)
* refactor(operator): route shared requests through NodeManager Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): preserve retry policy through error hints Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(operator): restore sparse partition tags when rebuilding writes Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(operator): stop detached write retries when callers cancel Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(frontend): configure retries across distributed write paths Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage. Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
4bd36cfd20 |
feat: support compaction options in HTTP hints (#9560)
* feat: support compaction options in HTTP hints Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: share TWCS compaction option prefix Use the store-api constant for both HTTP hint inference and Mito region option validation so the two paths cannot drift. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
55d753ca58 |
perf(log-store): fetch object store WAL replay segments concurrently (#9555)
* perf(log-store): fetch object store WAL replay segments concurrently A region read fetched each segment only after the previous object's entries were yielded, so a replay paid the full request latency once per object. Fetch up to READ_PREFETCH_DEPTH (8) segments ahead, each on its own task, and handle their outcomes in object sequence order: entries, collected-object skips, corrupted-segment holes and errors surface exactly as before, and dropping the stream aborts the fetches in flight. A read holds at most eight decoded segments besides the one it yields. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): raise object store WAL recovery concurrency to 64 On S3, recovery reads the header and tail of every object, about 57 to 91 ms per object per lane on 4, 8 and 16 vCPUs alike, and its time scales with objects divided by concurrency up to 128 lanes without throttling: about 9,500 objects recovered in 99 s at 8 and 10.5 s at 64. At 64 recovery issues about 1,800 reads per second, a third of S3's request rate per prefix, and holds one 64 KiB tail window per lane, 4 MiB in all. The recovery concurrency tests now scale with the constant. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): share one prefetch byte budget among object store WAL reads Each read fetched up to eight segments ahead on its own, so regions opened in parallel (16 by default) could together hold 128 segments of up to max_batch_bytes each. A read now always fetches the segment it yields next, as before, and fetches a later one only while its length fits in a 64 MiB budget shared by every read of the store. Regions replayed in parallel thus hold one segment each plus at most 64 MiB, however many open at once. Nothing waits on the budget: a read that cannot reserve stops fetching ahead until its next poll. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail an object store WAL read whose fetch task panics A fetch task that panicked was re-raised in the reader with resume_unwind, which unwinds through the caller, such as Mito's region opener task, past its cleanup. Return it as a WalSegmentFetch error instead, so the read ends at that object like any other failed fetch. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): bound replay prefetch by decoded size held by fetch tasks The prefetch reservation sat in the read's queue, so dropping a read released it at once while an aborted fetch could still hold its buffers until it next yielded. The fetch task now holds the reservation and returns it with its outcome, so it is released with the buffers. A reservation also counted only the encoded segment length, while a fetched segment is held decoded. It now counts the length plus one Record per entry, and the comment states the whole bound. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): round the measured figures in the recovery concurrency comment Keep the magnitudes in the comment; the exact S3 figures belong to the measurement run rather than the code. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): abort replay fetch tasks with tokio-util's AbortOnDropHandle Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): rename the replay prefetch reservation to try_reserve_prefetch_bytes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): share the bounded polling helper across the object store WAL tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
2a70194930 |
feat: add per-column COMPRESSION option for SST columns (#9426)
* feat: add per-column COMPRESSION option for SST columns CREATE TABLE accepts COMPRESSION WITH (type = 'zstd', level = N, page_size = '<size>') on a column. The options are stored in the column metadata, carried through gRPC column options, shown by SHOW CREATE TABLE, and applied by the mito SST writer and bulk part encoder as parquet column properties. Unset fields keep the SST defaults. Nested types are rejected. A page size below the dictionary page limit also lowers the column's dictionary page limit: parquet sizes write batches by the dictionary budget and keeps that size after falling back to plain encoding, which would otherwise ignore the smaller data page limit. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: validate column compression options before persistence Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject compression overrides on metric logical tables Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
2039f508fc |
fix(promql): stop fabricating histogram fold hash partitioning (#9509)
* fix(promql): stop fabricating histogram fold hash partitioning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover histogram fraction binary join missing rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): normalize dictionary labels in histogram regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover histogram joins on logical metric tables Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: specify insert columns for logical metric fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use supported histogram child replacement API Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): keep histogram fold fixtures truthful Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
d86741d0ad |
feat(operator): retry failed writes with refreshed partitions (#9542)
* feat(operator): retry failed writes with refreshed partitions Signed-off-by: WenyXu <wenymedia@gmail.com> * style(operator): simplify retry test type references Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(operator): clarify write request rebuilding names Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(operator): encapsulate write retry attempts Signed-off-by: WenyXu <wenymedia@gmail.com> * style(operator): group inserter configuration methods Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(operator): share routed peers and separate retry decisions Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(operator): avoid temporary write routing allocations Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
18634e17bb |
refactor(mito): retain region compaction and DDL lifecycle state (#9498)
* refactor(mito): retain region compaction and DDL lifecycle state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): finalize opening batch in the worker on registration Clean up the `opening_regions` entry and reply to its waiters in the same serialized worker step that registers an opened region, instead of waiting for the background opener to resume after the registration acknowledgement. Previously the entry was only removed by the background opener, so a close followed by a reopen could join the stale opening batch and receive a false success after the region no longer existed. Replying from the worker closes that window because registration and waiter cleanup are now atomic with respect to the worker loop. Add a test-only `EventListener::on_region_open_registered` hook and a regression test that holds the first background opener to verify a reopen does not wait for the previous open task. On failure the region is still stopped and waiters receive `RegionClosed`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): unregister drop runtime state before WAL cleanup The region leaves the region map before its WAL entries are obsoleted. A failure in `wal.obsolete(...).await?` returned early and skipped `cleanup_dropped_region_runtime_state`, so the resident compaction, flush, and index scheduler entries were never unregistered. Since the region is no longer in the map, a later close cannot clean them up either, leaking the entries for the rest of the worker's lifetime. Move the runtime-state cleanup to immediately after the region leaves the map, before the fallible WAL cleanup. The schedulers only touch their own state and do not depend on the region map or WAL, so the reordering is safe and leaves the success path unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): tighten visibility of new compaction state items Keep `ActiveCompaction::idle` private (it is only used inside `state.rs`) and expose the other items added by the resident-region refactor as `pub(crate)` instead of `pub(super)`, per the style guide. Also drop a stale comment in `on_compaction_finished`: DDL handoff now happens in `take_ready_ddls` before this block, which only runs when no DDL is queued. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): skip compaction scheduling while a region is truncating `handle_flush_finished` dispatches queued DDLs before scheduling compaction. A truncate queued behind a committing flush marks the region as `Truncating` synchronously, but `schedule_compaction` only skipped staging states, so it could schedule a new automatic compaction while the truncate was still running. Add `MitoRegion::is_truncating` and skip compaction scheduling for a truncating region. Add a regression test that queues a full truncate behind a committing flush and asserts no compaction is scheduled. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito): log DDL-fenced compaction rejection The refactor dropped the log emitted when a region was skipped because of pending DDL requests. Automatic triggers carry no waiter, so the rejection became silent and could not be told apart from a scheduling interval that had not elapsed. Log the queued and executing DDL state before cancelling. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito): document DDL reply fence invariants Explain why fence_ddl_replies installs executing_ddl before handing off the batch and why CompactionDdlComplete carries both a scheduler-wide generation and a per-slot request_id. The generation fences replies from superseded batches or region incarnations, while request_id makes completion idempotent so only the last distinct reply releases the fence. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito): rename DDL completion request_id to request_index The value is the DDL's index within the handed-off batch, not a globally stable request identifier; request_id suggested otherwise. Rename it to request_index across the scheduler, worker dispatch, and tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
394562aa9e |
fix: prevent stale leader metadata after concurrent writes (#9549)
fix: fence leader cache fills around store mutations Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
19a128f8d3 |
fix(session): respect supported read preferences in wire tests (#9550)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
287ced2de2 |
fix(mito2): ignore missing inverted indexes in old SSTs (#9545)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
83e62a7725 |
feat: remove the server-side Metric export gate (#9547)
* feat: remove the server-side Metric export gate Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: remove unused export capability import Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: remove unused CLI capability import Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: import the packed export error selector Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
a2d28e6060 |
refactor(operator): centralize write partitioning and dispatch (#9541)
* refactor(operator): centralize write partitioning and dispatch Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(operator): preserve write dispatch behavior during refactor Signed-off-by: WenyXu <wenymedia@gmail.com> * style(operator): import write dispatch error paths Signed-off-by: WenyXu <wenymedia@gmail.com> * style(operator): align dispatch imports and test utilities Signed-off-by: WenyXu <wenymedia@gmail.com> * style(operator): use imported types in partition tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
684c8ff291 |
feat(client): support region request reuse and route refresh hints (#9540)
* feat(client): support shared region requests without deep cloning Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate route refresh retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * style: preserve local error variant imports in retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): distinguish staging rejection from write backpressure Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): preserve legacy retry hints in instruction replies Signed-off-by: WenyXu <wenymedia@gmail.com> * test(client): inline retry hint compatibility cases Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): align batch-open retry hint assertion Signed-off-by: WenyXu <wenymedia@gmail.com> * style: use imported types in region request tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
c1a0ca9595 |
ci: benchmark the object store WAL against S3 from an EC2 runner (#9536)
* ci: benchmark the object store WAL against S3 from an EC2 runner Add a workflow_dispatch workflow that loads the o11ybench agent observability dataset into a fresh GreptimeDB container per cell and compares three WAL configurations that differ only in [wal]: Raft Engine with sync_write on the instance store, and the object store WAL in durable and enqueued mode, with the data files on S3 in the runner's region for all three. Rounds alternate the configuration order. A summary reports rows per second, D/R and E/R as the geometric mean over rounds, append acknowledgement p50 and p99, and the object store WAL counters. A smoke mode runs the same chain on ubuntu-latest against the MinIO fixture. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: scrape WAL metrics on failed loads and keep logic out of the benchmark YAML Scrape the object store WAL metrics before stopping a cell's server whether the load succeeded or failed, so failed cells keep their stall, poison and create failure counters. Build the summary from the plan so that cells which never ran are listed and leave their ratios undefined. Move budget, region, runtime image, manifest and credential scan logic from the workflow into the helper scripts, and drive the cell lifecycle in the tests through fake docker and curl binaries. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: drop redundant required flags from benchmark inputs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: fail a benchmark cell whose metrics scrape fails and zero untouched WAL counters A load that succeeds without its metrics scrape now fails the cell with exit code 4 and records metrics_scraped in cell.json. The summary reports an event counter that never registered as zero when the cell is an object store WAL cell whose scrape succeeded and created objects, and as unknown otherwise. Planned cells that never ran are read through the same path as recorded ones. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: register the S3 benchmark workflow for dispatch from its branch A workflow_dispatch workflow can only be dispatched once GitHub has registered it, which happens on the default branch or when one of its triggers fires. A push trigger limited to this branch registers it; validate runs only for workflow_dispatch, so a push run skips every job. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: delete the benchmark S3 prefix one object at a time boto3 now sends a CRC checksum instead of Content-MD5 with DeleteObjects, which the MinIO fixture of the smoke run rejects. DeleteObject needs neither and the prefix of one run holds at most a few thousand objects. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: default the S3 benchmark to c6id.2xlarge c7id is not offered in the runner region ap-southeast-1, where DescribeInstanceTypes rejects it. c6id is the compute optimized x86 family with a local NVMe instance store that the region offers, with the same 8 vCPU and 16 GiB at 2xlarge. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: drop the registration trigger of the S3 benchmark workflow The workflow is registered now, so dispatches no longer need a push trigger on its branch. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: fail an S3 benchmark run on EC2 without a local instance store Without an instance store the Raft Engine WAL of the R cells silently stayed on the root EBS volume, so the run measured a different baseline. Only a smoke run may keep it on the root volume now. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: let the instance store check fail its workflow step The step piped wal-disk into tee without pipefail, so a rejected host still passed. wal-disk now writes its record itself and the step runs it alone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Yingwen
|
674ba72669 |
fix(mito): preserve version ordering in bulk merges and flushes (#9517)
* fix(mito): preserve delete barriers in flat last-non-null dedup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): respect sequence ranges in bulk merges and flushes Signed-off-by: evenyag <realevenyag@gmail.com> * perf(mito): avoid merge selection on the bulk write path Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
jeremyhi
|
4ef312ecf2 |
docs: add the object store WAL RFC (#9195)
* docs: add the object store WAL RFC Propose an object store backed write-ahead log as a WAL provider next to Raft Engine and Kafka: a datanode batches the entries of its regions into immutable objects under one prefix, creates each object conditionally under a monotonically increasing sequence, acknowledges a write once its object is durable by default, and replays each region from its own segments after listing the prefix. A reference implementation exists and follows this RFC in a series of pull requests; the document describes it and marks what is proposed on top of it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: link the object store WAL RFC to its tracking issue Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: describe the predecessor chain and instance epoch of the object store WAL Recovery replays only the chain of objects that ends at the latest complete object, and every open starts a new epoch with an empty start object, so an object whose create was reported as failed is never replayed once a later append has been acknowledged. Update the object format, recovery, writing, the failure matrix, the garbage collection constraints, compatibility, evidence, drawbacks and the cluster mode alignment accordingly. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: identify object store WAL objects by sequence and epoch Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: derive the object store WAL epoch from the start object sequence Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: separate the retryable and the conflicting start object outcomes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: describe garbage collection of the object store WAL through the reclaim boundary Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: judge vanished objects after chain selection and fix the evidence references Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: record the stop-time conflict and the completed fault matrix in the object store WAL RFC Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: limit the stop-time conflict to objects of the same or a later epoch Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: keep the conflicting-content condition in the stop-time conflict sentence Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: report the S3 write path measurements of the object store WAL Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: qualify the seal timer findings and link the S3 runs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: state the serial durable cycle behind the one-batch-per-interval observation Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
70a0c826ca |
docs(config): qualify when the object store WAL batch seals on size (#9538)
The `wal.flush_interval` comment claimed that under sustained load the batch always seals on `max_batch_bytes` and the interval no longer matters. Sustained-load runs on S3 sealed on the timer in every case because the admitted bytes did not reach `max_batch_bytes` within one interval. State the actual condition: the batch seals on size only when the admitted bytes reach `max_batch_bytes` before the next timer tick; otherwise it seals on the timer, whatever the load. Regenerated `config/config.md` from the example TOMLs. Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 liyang
|
7c32c38ba3 |
feat(cmd): add --http-api-server-addr CLI flags (#9512)
* feat(cmd): add --enable-api-server and --api-server-addr CLI flags Signed-off-by: liyang <daviderli614@gmail.com> * apply suggestion Signed-off-by: liyang <daviderli614@gmail.com> --------- Signed-off-by: liyang <daviderli614@gmail.com> |
||
|
jeremyhi
|
fc5831043f |
feat(cli): simplify packed metric exports with a boolean flag (#9531)
* feat(cli): select packed metric export with the layout option Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cli): inspect the packed capability error cause Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cli): follow the standard capability error source Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(cli): clarify the metric export gate helper name Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(cli): use a boolean flag for packed metric exports Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 shuiyisong
|
2df33380a7 |
feat(catalog): declare information-schema table scopes (#9504)
* feat(catalog): declare information-schema table scopes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: rename cluster to global Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: update sqlness result Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(catalog): require information-schema table scopes Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
f50f42140a |
build: reduce checkout-specific inputs for cache reuse (#9198)
* test(mito2): cover regex inverted index pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * build: keep build inputs checkout-relative for cross-worktree cache reuse Two small changes so compilations are byte-identical across worktrees of the same commit and can share a build cache (mbx, sccache): - src/common/version/build.rs: derive the workspace root relative to CARGO_MANIFEST_DIR instead of the CARGO_WORKSPACE_DIR injected by .cargo/config.toml (an absolute path). The env var remains as an explicit override for the submodule case; the [env] injection in .cargo/config.toml is removed so the relative derivation is used by default. - Replace three env!("CARGO_MANIFEST_DIR") uses in test/bench code (servers prom_decode bench + decode test, common-meta test_util) with a runtime std::env::var lookup, so the compiled artifact no longer embeds the checkout path. No behavior change: same files are read, same version info emitted. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * build: address cross-worktree cache review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
jeremyhi
|
29d5854e6a |
test(log-store): drive recovery, read, delete and stall failures through the fault matrix (#9488)
* test(log-store): drive recovery, read, delete and stall failures through the fault matrix Add the second half of the object store WAL fault matrix: the failures of the object store access and of the store state that the conditional-create rows did not drive. Each case asserts the failed waiters and their error, whether the store is poisoned, the sequence, both indexed gauges at the failure, the exact movement of every counter and what a read sees after. Recovery: a listing, a trailer fetch and a footer fetch that fail, the catalog rejecting an object after every footer was read, and an object gone by the time it is fetched, below the boundary of the tip (ignored) and at it (construction fails); each leaves the gauges and the recovery metrics untouched. Reads: a segment fetch that fails, without a hole or a read timer sample, and a later read of the region that succeeds. Collection: a delete that fails and is attempted again at the next collection, and a stop that waits for a delete in flight, with the catalog agreeing with the listing in both outcomes. Store state: per-region entry id exhaustion refusing the append without touching the batch it sealed, and the durable stop paths with several creates in flight that succeed, fail transiently or conflict after stop began. Stalled appends released by admission, by a stop, by a terminal error and by the teardown of the actor, the first two timed as a pair. The recovery and read timers observed around parked fetches, twice each, with holds that cannot overlap. The cases use the existing doubles, extended with a failing listing and a failing range read at an offset on RecordingIo and parked range reads on ParkedIo. Case::row is now optional, and the assertion that every row of the failure matrix has a case is unchanged. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): lose a wide object at its header and footer reads in the fault matrix The vanished-object case now runs three ways: a short object read whole, and a wide object deleted before its header and tail window reads or before its separate footer read, each below and at the boundary of the tip. The entry id exhaustion case reads the other region of the sealed batch back, the teardown case asserts the indexed gauges before and after the actor is gone, and two comments that restated their names are gone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): prove the stop-path ordering and share the wide-footer fixture in the fault matrix The case of creates that succeed after stop began keeps object 1 parked while objects 4 to 2 complete, round-trips a command through the actor and asserts that no append is acknowledged and nothing is indexed before object 1 is released. The stop-with-delete case replaces its yields with the same command barrier, queued behind the stop command. The wide-footer fixture moves to the tests module and is shared with the recovery test that fetches a footer longer than the tail window. The gauge helper is named for its side effect and the vanished-object stage enum for what it names. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): keep the four-create scenario doc on its fault matrix case Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
18726c7602 |
fix: bound metric snapshot memory and reduce export overhead (#9505)
* fix(mito): bound retained scan memory Compact unused tag dictionary values before merge retention and avoid splitting a single scan source into redundant batches. Include regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(snapshot): bound writes and reduce logical export overhead Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible string dictionaries, bound conversion work, combine Parquet row groups and filter single-table exports before routing. Include resource-boundary and packed-restore regression coverage. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: clarify metric snapshot acceptance criteria Specify populated scale fixtures, controlled S3 request delay, native macOS validation, paging exclusions and alternating comparison rounds. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): split batches only for merged readers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(snapshot): cover supported types in both layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: consolidate metric snapshot experiment conclusions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito): honor series statistics when splitting L0 scans Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics. Cover short-series overlapping L0 files with an ordered latest-value regression. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito): explain memory risk of splitting short L0 series Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: address snapshot export review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(cli): accept source errors in lazy snapshot contexts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9
|
483d1e49a3 |
fix(query): keep filter fetch when rewriting matches term (#9521)
* test: cover matches term limit dropped by constant rewrite Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep filter fetch when rewriting matches term Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
2d99a5cf69 |
feat(jaeger): add query endpoints for Jaeger UI v2 (#9409)
* feat(jaeger): support v3 HTTP query APIs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(jaeger): preserve full traces with v3 time hints Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(jaeger): clarify validation of v3 lookup hints Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(jaeger): bound v3 searches and preserve v1 binary attributes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(jaeger): match resource attributes in v0 tag filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
f3a0841f4f |
fix(promql): follow IEEE 754 float semantics in comparisons and aggregations (#9479)
* fix(promql): follow IEEE 754 float semantics in operators and aggregations - Comparisons: Arrow orders floats totally (NaN == NaN, NaN > +Inf); PromQL comparisons with NaN are false except !=. - min/max: skip NaN unless the whole group is NaN. DataFusion's grouped min/max let NaN replace and be replaced by neighbours, so the result depended on input order. - topk/bottomk rank NaN last. - round() rounds ties up and returns NaN for a zero step, like Prometheus. - sqrt() of a negative number is NaN instead of an error. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): round() without a step rounds to integers The planner passed a zero step for single-argument round(), which now yields NaN like Prometheus; pass 1 instead. Found by the compatibility harness. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): apply the sqrt guard to builtin functions and register min/max for plan decoding sqrt resolves through the session's builtin functions, not the UDF branch. Distributed plans are decoded with the serializer's session, which did not know prom_min/prom_max. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover NaN and rounding semantics Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep the nested count rewrite for PromQL min/max count(min(x) by (l)) was rewritten by count_nest_aggr only for the builtin min/max names. prom_min/prom_max return NULL for the same groups, so the rewrite stays valid. Update plan snapshots for the IEEE comparison, sqrt, topk and min/max changes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): register prom_min/prom_max in the function registry Distributed plans split an aggregation into state and merge steps only for functions whose state function exists in FUNCTION_REGISTRY; prom_min and prom_max lived in the promql crate, so min/max stopped being pushed down to datanodes. Move them to common-function and register them before the state functions are derived. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: update sqlness results for PromQL float semantics Plans and column names now show prom_min/prom_max, the NaN guards of comparisons and sqrt, and the NaN sort key of topk. round() results follow Prometheus' formula. Distributed min/max keep the state/merge split. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep column names and NULLs in IEEE comparisons and sqrt Rewriting comparisons as `a op b AND NOT (isnan(a) OR isnan(b))` changed TQL output column names and turned NULL operands into true for `!=` and false for the bool forms. Implement comparisons and sqrt as PromQL UDFs that propagate NULL and report the column name of the expression they replace; prom_min/prom_max report min/max column names the same way. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: restore column names in sqlness results for PromQL float semantics Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): share the prom_min/prom_max groups accumulator with tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep distinct names for prom_min/prom_max Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
845fc0af9c |
fix(promql): default a stepless subquery to a one-minute step (#9481)
* fix(promql): default a stepless subquery to a one-minute step `x[5m:]` evaluated the inner expression at the query step, which is 1s for instant queries. Prometheus uses the evaluation interval, one minute by default. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover the default subquery step Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover the default subquery step in a range query Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Weny Xu
|
e6f3ca10a5 |
fix(deps): bump OpenDAL to 0.59.4 for OSS RRSA credentials (#9520)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
28ec3ff5aa |
fix(table): require complete primary keys for hash partitioning (#9507)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>v1.4.0-nightly-20261012 |
||
|
discord9
|
32e8467cda |
fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): use current DataFusion plan downcasts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): match null rendering in join regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh union plan for unknown scan partitioning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh shared plans after distribution rule removal Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): cover semi joins with remote dynamic filtering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): strengthen partitioned merge scan regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove unused merge scan test helpers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): measure merge scan hash repartition cost Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt merge scan deserialization to two-arg constructor Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): adapt nested merge scan integration constructor Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
sun
|
ad83232ad1 | feat: update dashboard to v0.14.0 (#9510) | ||
 fys
|
9a57e1195d |
feat(json2): json_get respect type hints (#9349)
Signed-off-by: fys <fengys1996@gmail.com> |
||
|
dennis zhuang
|
b913be2af7 |
fix(promql): return range-vector results for instant queries (#9477)
* fix(promql): return range-vector results for instant queries An instant query of a range selector or subquery (e.g. `up[5m]`) returned the internal range arrays, which panicked when the frontend decoded them as dictionaries. Plan such queries as the selected samples with their own timestamps, and reject range-vector expressions in range queries like Prometheus does. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): keep encode_substrait on a valid range query A range query of a range-vector expression now fails like in Prometheus. Wrap the subquery in max_over_time so the case still encodes both RangeManipulate nodes through substrait. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): plan wrapped range-vector roots and reject a zero subquery step TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the range-vector path did not unwrap. A zero subquery step divided by zero. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover wrapped range-vector roots and a zero subquery step Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): default subquery step and reject range vectors in range queries A root subquery without a step uses the one-minute default instead of the query step. The HTTP range query endpoint rejects range-vector expressions even when start equals end. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover root subquery default step Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): align root subqueries with subquery_child_window Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): return millisecond timestamps for range-vector results Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): keep the time index timezone and name its table in errors Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
28697c9e55 |
feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode Give a datanode the ability to decode and execute a plan whose MergeScan carries a nested plan (e.g. an inner MergeScan over a build table), so a distributed sub-plan pushed to a datanode can itself fan out to other datanodes. This is the execution/codec capability only; it is not wired to any user-facing switch and produces no plan by itself. Codec (src/query/src/query_engine/default_serializer.rs): - MergeScanAwareSerializer keeps the request's SessionState and the engine catalog manager, and decodes a MergeScan payload's tables by (catalog, schema, table) name through the engine catalog rather than the region-aware request catalog list (which binds every table to the request's region and breaks on mismatched columns). Datanode (src/datanode): - New DatanodeRegionQueryHandler (region_query.rs) serves region queries for the MergeScan nodes of a received plan, resolving region leaders via PartitionRuleManager and dispatching to peer datanodes. - Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into the datanode's catalog/cache registry (with_dist_planner stays false: a datanode never rewrites its own plan). Bug fixes uncovered while wiring this up: - cache registry: register the table/table_info/table_name/table_route/ partition_info caches the datanode's catalog manager and partition rule manager require (previously missing -> panic on lookup). - default_serializer: decode_sub_plan bridge no longer drives a future on a current-thread runtime from within another executor (spawn a dedicated thread + runtime when not on a multi-thread runtime). Tested by query_engine::default_serializer unit tests: nested MergeScan round-trip and engine-catalog resolution with mismatched column names. No user-facing behavior change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep GreptimeDB function bindings after rebuilding decode state SessionStateBuilder::new_from_existing collects the function registry (HashMap) into a Vec and build() re-registers each entry; register_udf registers aliases before the function's own name and the Vec order is unstable. Rebuilding the session state to install MergeScanAwareSerializer after the GreptimeDB functions were registered could therefore rebind a name/alias (e.g. date_format, an alias of DataFusion's to_char) back to the built-in function. This path runs for every decoded plan, not only nested MergeScan ones. Extract the GreptimeDB function registration into register_greptime_functions and re-apply it after the final build of both the top-level decode state and the MergeScan payload state. Add regression tests asserting date_format stays bound to the GreptimeDB implementation on both paths. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): address review on nested MergeScan capability Three follow-ups from review of the nested MergeScan execution capability: 1. Layer the datanode caches (P2b). table_cache derives from table_info_cache + table_name_cache, and partition_info_cache derives from table_route_cache, but they were all registered in a single flat CacheRegistry whose invalidate() runs concurrently with no ordering. A derived cache could be invalidated and refilled from a base cache that had not been invalidated yet, leaving stale metadata. Build the base caches and the derived caches as separate layers of a LayeredCacheRegistry (reusing the existing layering mechanism) so base caches finish invalidating before derived ones. Both the production path (cmd) and the test scaffolding now build the same layered registry; the flat build_datanode_cache_registry is removed. 2. Don't let a nested MergeScan's internal region query compete for the concurrency permit (P2a). An outer query holds its permit until its stream is fully consumed, and a nested MergeScan makes the datanode issue region queries to peer datanodes as execution stages of that same query. If such an inner stage also took a permit, the outer query waiting on it would deadlock against its own held permit (or time out when max_concurrent_queries > 0). Mark inner DN->DN region queries as internal stages: QueryRequest gains an flag carried over the wire in the request header's query_context extensions (reserved key, not injectable via hints), and the region server skips acquiring a concurrency permit for internal-stage requests on both acquire sites. Outer queries still acquire normally. 3. Add an end-to-end capability test (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build Join(<local probe region>, MergeScan(build regions on both datanodes)) with mismatched column names and duplicate build keys, ship it to a datanode over a real TCP Flight endpoint, consume the stream fully and compare the row multiset against the frontend's reference join result, assert a real cross-datanode DoGet via an RPC counter, and assert an inner region failure fails the whole query. Also migrate the with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt date_format regression test to the updated DataFusion fork ScalarUDFImpl's trait bounds changed so its impl can no longer be downcast via as_any(); assert the function equals the GreptimeDB implementation instead, which to_char can never satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover the internal stage under the concurrency limit Add an end-to-end regression test for the nested MergeScan capability under max_concurrent_queries = 1: both datanodes' only permit is held by the outer queries' streams while their inner MergeScans fetch the build regions from the same datanodes over real Flight connections. The result consumption is bounded by a timeout, so a broken link anywhere in the chain (marking the inner request internal, carrying the marker in the request header, honoring it at the receiving acquire sites) fails the test instead of deadlocking. Also add a minimal with_datanode_options_override hook to the cluster builder, and fix the test module doc: the outer request is handed to RegionServer::handle_remote_read in process; only the inner region queries go over the wire. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover cold route loads under unrelated invalidations The table route cache uses InitStrategy::VersionChecked, whose version counter belongs to the whole CacheContainer rather than to a single table: any TableId invalidation bumps the shared version, so a cold load of one table's route retries when an unrelated table is invalidated concurrently. Document this as a known limitation on the region query handler, and pin the behavior with an integration test: run the nested plans while a background task invalidates an unrelated table id on every datanode, assert the results still match the expected multiset (retries only add work, they never return a stale route), and report the observed load amplification via the cache miss counter (2 loads without invalidations, 8 over three runs with 24 unrelated invalidations in this run). Also expose the datanode cache registries on the test cluster so a test can drive the local invalidation path that the heartbeat handler drives after a metasrv broadcast. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): trim nested plan decoder commentary Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: remove redundant nested scan support plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): decode nested MergeScan payloads asynchronously Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): avoid unnecessary extension name allocation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain extension decoding for manual session states Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datanode): keep nested scan query admission unchanged Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): honor query handler factories and trim nested scan fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): retain required cache error import Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): reuse registered state for nested scan decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): verify payload information schema table functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): retain nested scan runtime and cold-route guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): address nested scan review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(session): preserve read preference across region query hops Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(session): verify strict context rejection at Flow ingress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove redundant serializer error case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |