mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-21 19:32:34 +00:00
c51293836e46d92bd362b844f293240d8b459b68
6417
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dennis zhuang
|
c51293836e |
fix(query): push LIKE filters down into MergeScan remote plans (#9648)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Yingwen
|
3e2751d71d |
feat: improve SQL analyze scan diagnostics (#9565)
* feat: improve SQL analyze scan diagnostics Signed-off-by: evenyag <realevenyag@gmail.com> * test: redact dynamic filters in sqlness results Signed-off-by: evenyag <realevenyag@gmail.com> * test: stabilize analyze snapshots for merge queue Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove obsolete scan_details assertion Signed-off-by: evenyag <realevenyag@gmail.com> * test: refresh explain snapshots after partition filter changes Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
4645ab9ed7 |
perf(mito2): gather last rows with one take per input batch (#9737)
* perf(mito2): gather last rows with one take per input batch FlatLastTimestampSelector buffered a single-row slice per series and concatenated up to 8192 of them. Concatenating slices with dictionary columns merges the dictionaries once per slice, which made high-cardinality instant PromQL queries using the LastRow scan path (#9034) slower. Select the rows of series that end inside a batch with one take instead. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(mito2): cover duplicate last timestamps in the take path Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): group last rows by primary key bytes Flat batches may assign different dictionary keys to the same primary key, for example when binary keys are wrapped with identity keys. Merge adjacent ranges with equal key bytes before selecting last rows. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 shuiyisong
|
9c7e1e1605 |
fix: record slow INSERT SELECT queries (#9733)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 jeremyhi
|
a436213bf3 |
ci: compare several images in one object store WAL S3 benchmark run (#9691)
* ci: compare several images in one object store WAL S3 benchmark run
A new `images` input takes a JSON list of 2..4 {label, image}. Every
cell then runs once per image, back to back, and the image order
reverses with the configuration order every other round. The cell id
gains the label. The single-image inputs keep their meaning and cell ids.
The validate job pulls every image before an instance is allocated. It
pins each image to its digest and records the version and commit that
the image reports. Each cell starts the image of its label. An optional
`warmup` runs the first load cell once per image as round 0, and no
statistic uses it.
The plan step estimates the benchmark job's minutes on c6id.2xlarge
from the cell times of earlier runs. It refuses a plan that does not
fit the job timeout.
The summary keeps the Raft Engine ratios within each image. With
several images it also reports, per group, concurrency and config:
- every round's rows/s and client p50/p99;
- each image's ratio to the first, with the geometric mean, minimum
and maximum over the rounds.
Every cell records the host load averages at its start and end.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scale the S3 benchmark time estimate with the batch size
The per-cell rates were measured with 1000-row batches. A smaller batch
keeps about the request rate, not the row rate, so the estimate now
shrinks rows/s with the batch. A one-row durable plan is refused instead
of passing a timeout it cannot fit.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin the S3 benchmark runtime image without pulling it in validate
An anonymous pull from ECR Public on a hosted runner can exceed the
registry's data limit, and it failed the validate job of several
dispatches with "toomanyrequests: Data limit exceeded". Validate now
reads only the registry manifest to pin the runtime image's digest. A
reference that names a digest must be served under that digest.
The check that the image holds the loader and the generator moves to
the benchmark host. It runs right after the runtime pull and before any
GreptimeDB image is pulled or any cell starts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: run the S3 benchmark runtime content check against both files
The fake docker now runs the pull step's own check in a directory that stands in for the runtime image. It covers both files present, the loader missing and the generator missing, so dropping either file check fails the test.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin a digest-named S3 benchmark runtime image without the registry
ECR Public's anonymous data limit now refuses hosted runners even the
manifest read, and validate failed in "Pin the runtime image digest".
A runtime reference that names its sha256 digest, like the default one,
is already pinned, so validate takes that digest with no network call.
The format checks stay. Only a tag-only reference is resolved through
the registry. The content check on the benchmark host is unchanged.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check the full S3 benchmark runtime reference before pinning it offline
A named digest no longer reaches the registry, so nothing parsed the rest of the reference. Validate now checks the whole reference against Docker's reference grammar first. It refuses an empty repository, a doubled '@' or a bad tag before any pin is emitted or any instance is allocated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
|
||
|
jeremyhi
|
93c3b116da |
fix(mito2): re-check a filled write as sent after a schema change (#9701)
* fix(mito2): re-check a filled write as sent after a schema change The engine fills the columns a write leaves out with their defaults when it accepts the write, appending them to the rows without adding them to `name_to_index` and `has_null`. If the region schema changes before the worker handles the write, the worker checks the request again and `check_schema` looks up a filled column in `name_to_index`, which panics and stops the region worker. If the filled column was dropped, the write fails with an unknown column that the client never sent. Remove the filled columns before checking, so the request is checked as sent and filled again for the current schema. Nothing is filled yet when the engine checks the request, so that check does no extra work. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): rename put_request helper to build_put_request Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
da88ff8653 |
fix(partition): route NULLs like row inserts and accept signed-minimum bounds (#9697)
* fix(partition): route NULLs like row inserts and accept signed-minimum bounds Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): keep equality two-valued for NULL rows and accept unsigned -0 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): treat NULL masks as false in batch routing and keep comparisons bare Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(compat): update partition filter in analyze verbose expected output Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
7397ab6b23 |
perf(datanode): send the mito puts of one request to each worker at once (#9692)
* perf(datanode): send the mito puts of one request to each worker at once The region server handled the puts of a request to several regions as separate requests. A Mito worker woke on the first of them and wrote it alone; in the durable object store WAL mode its append then blocked the worker until the next seal, so the rest of the request went into the next object, and clients writing a partitioned table locked into one phase that wastes a tick. When every put of a request goes to a ready mito region, the region server now hands them to `MitoEngine::put_regions_batch`, which submits the puts of each worker in one message, so the worker writes its regions of the request in one batch and one WAL append. Requests to one region and to other engines are handled as before. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): await each region of a batched mito put on its own task The batched path waited for the regions of a request in order and sent the worker messages one after another, so an error or a full worker queue of one region held back the others, and its metrics counted the request as a whole. `MitoEngine::put_regions_batch` now sends the worker messages concurrently on their own task and returns each region's result. The region server awaits and accounts every region on its own ingest task, as the per-region path does: the first error returns at once, and each region records its latency, changed rows or failure and is named in its error. A request with several puts to one region keeps the per-region path. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): spawn the accounting of batched mito puts from the batch task The request spawned the per-region accounting tasks only after the batch task returned, so a request dropped in between left accepted writes unaccounted. The batch task now spawns them and the request only joins their handles. Also state in the architecture invariants that the region server's batched puts to the metric and mito engines are the one request dispatch that does not go through the RegionEngine trait. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): batch mito puts through the RegionEngine trait Add `RegionEngine::handle_batch_put_requests`, which returns each region's response of the puts of one request. The default handles each put with `handle_request`, so other engines keep their behaviour; Mito overrides it with the one message per worker submission. The region server calls the trait method instead of downcasting to `MitoEngine`, and the architecture invariants no longer need an exception. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): trace each region of a batched mito put Attach the per-region `RegionServer::handle_region_request` span of the per-region path to each region's task of a batched mito put, so the region ids stay in distributed traces. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): let a batch put response answer for several regions `RegionEngine::handle_batch_put_requests` now returns each response with the regions it answers for. The default and Mito answer each region on its own; an engine that writes several puts together, such as the metric engine's batched puts, can answer them with one response. The region server accounts a response for each of its regions. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): batch any request with several mito puts Take the batch path for a request with at least two puts to ready mito regions, also when they all target one region: a request of many puts to one region otherwise still sends one worker message per put. Each put keeps its own response, affected rows and error. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): account batched mito puts in one task Await the responses of a batched mito put in the batch task with a FuturesUnordered instead of one spawned task per put. The task accounts each response as it completes, hands the first error back to the caller at once and keeps accounting the rest, so a request of many puts no longer spawns a task per put. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(mito2): name the batched worker message BatchWrite Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(mito2): submit batched writes on the ingest runtime Grouping and submitting the writes of a request is part of ingestion, so the task runs on the ingest runtime and takes the write task class when the workload scheduler is enabled. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(datanode): share ingest result accounting across put paths The per-region path, the metric batch path and the mito batch path each recorded the changed rows or the failures of an ingest response. They now call one helper, so a metric or label change happens in one place. A batch put response lists the region of each put it answers for, so the failure count stays per put when two puts go to one region; the trait documentation now says so. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * fix(mito2): give batched puts of a stopped worker the worker's error A batched write message that could not reach a stopped worker was dropped, so its puts failed with "Failed to recv result" while a put submitted on its own fails with "Worker <id> is stopped". The worker now returns the message it could not take, and each of its puts receives the WorkerStopped error. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(datanode): look up the batch latency histogram once The label of the request latency histogram is the same for every put of a batch, so the batch task looks it up once instead of per observation. Signed-off-by: jeremyhi <jiachun_feng@proton.me> --------- Signed-off-by: jeremyhi <jiachun_feng@proton.me> |
||
 fengyue-xve
|
1e29e8d4b6 |
fix(meta): validate persisted partition bounds before altering regions (#9487)
* fix(meta): validate persisted partition bounds before altering regions
A partition bound is persisted with the type of the column at the time
it was written. When an ALTER TABLE changes the type of a column
referenced by a partition expression (widening the time index unit is
the common case), each region rewrites the bound literal to the new
type while building its scan predicate. A millisecond bound like year
3000 fits milliseconds but overflows nanoseconds, so every scan of the
region fails once the table schema is updated -- and some regions may
already be altered when the failure surfaces.
Validate every region's persisted expression on prepare, before any
region is modified: parse the persisted JSON (mirrored types, since
common-meta cannot depend on the partition crate which depends on it),
replay the same value -> ScalarValue conversion and checked cast the
region performs, and reject the ALTER naming the offending region when
a bound cannot be represented with the new column type. Incompatible
changes are rejected atomically; compatible widening proceeds unchanged.
Adds meta-level tests (multi-region rejection naming the second region,
compatible widening completing) and an end-to-end sqlness case.
Fixes #9454
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix: access ColumnSchema fields directly and use plain format args
ColumnSchema exposes name and data_type as public fields, not methods,
and the telemetry macros only accept a format string after the error.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: access ColumnSchema data_type field in alter table tests
Same fix as the previous commit: data_type is a public field on
ColumnSchema, not a method.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop borrows now that data_type is read as a field
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop useless format! in nested expression test
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: cover the partition tail in change_col_type sqlness case
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: drop spurious blank line in change_col_type result
Signed-off-by: fengyue-xve <1759519137@qq.com>
* fix(meta): compare partition bounds with the scan comparison coercion
The preflight validator cast the persisted bound directly to the altered
column type, which rejected ALTERs the scan itself accepts: the region
coerces both the column and the bound to the common comparison type
(ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field
column with an out-of-range bound still scans. Compare through
comparison_coercion(target_type, bound_type) instead and only require the
bound to be castable to the common type.
Also add a round-trip test in the partition crate that serializes real
PartitionExprs and runs them through the validator, so drift between the
Persisted* mirror and the partition types fails CI, and fix the error
message to match the comparison semantics.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* test: update the sqlness fixture for the new incompatibility message
The error message now reflects the comparison semantics: the persisted
bound can no longer be compared with the new column type.
Signed-off-by: fengyue-xve <1759519137@qq.com>
* build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency
common-meta gained a datafusion-expr dependency in
|
||
|
dennis zhuang
|
e0032534d2 |
fix(mito2): build indexes for encoded bulk parts flushed in sync mode (#9722)
* fix(mito2): build indexes for encoded bulk parts flushed in sync mode Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): skip reading the SST when an index build has nothing to create Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
660c4efe1f |
fix(mito2): fence writes before a graceful downgrade (#9716)
* test(mito2): cover graceful role changes racing with writes A graceful role change flips the region state in a background task while the worker keeps handling writes. Add bulk and row write tests that check a write handled after the role change request is either rejected or covered by the returned last_entry_id. Add tests for DowngradingLeader while an ALTER keeps the region in Altering, alone and in the downgrade sequence the datanode runs (DowngradingLeader, Flush, Follower) with a migration candidate on its own local WAL. Add a test that the downgrade sequence covers a write in flight when a lease renewal sets Downgrading. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): fence writes before a graceful downgrade set_region_role_state_gracefully hands the role change to a background task that waits for the manifest lock. The worker keeps handling writes until the task flips the state, so a write could be acknowledged with an entry id above the returned last_entry_id. Set Downgrading in the worker before spawning the task for Follower and DowngradingLeader requests, as lease renewal already does, so every write the worker handles afterwards is rejected. DowngradingLeader also did nothing and returned success when the region was in another leader state such as Altering. The region became writable again when the operation finished, and the datanode then flushed and turned it into a follower in one step. Return a retryable error instead, so metasrv retries the downgrade. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): assert the retryable downgrade error during ALTER Check that DowngradingLeader during an ALTER returns a retryable error, and release the paused ALTER on that path. The local WAL test now retries the downgrade after the ALTER, as metasrv does. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): write before the retried downgrade in the local WAL test Rows written before an ALTER were already flushed, so the candidate check did not depend on the downgrade's final flush. Write rows after the ALTER and before the downgrade is retried, which only that flush can carry to the candidate. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Yingwen
|
b3dc5674e9 |
feat(mito2)!: account SST decoding against shared scan memory limits (#9101)
* feat(mito2): add shared estimated SST scan memory budget Signed-off-by: evenyag <realevenyag@gmail.com> * feat: unify scan memory limits with configurable accounting modes Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): reuse query runtime memory pool for series merges Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): wait for cancelled scan reservations before exhaustion check Signed-off-by: evenyag <realevenyag@gmail.com> * docs(mito2): explain parallel scan cancellation cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): deprecate waiting for scan memory quota Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(recordbatch): remove scan memory wait handling Signed-off-by: evenyag <realevenyag@gmail.com> * fix(table): initialize memory pool in scan test fixtures Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Lei, HUANG
|
52554db52f |
perf(mito2): keep short series unsplit in compaction merges (#9706)
* perf(mito2): keep short series unsplit in compaction merges Compaction writes every merged batch to the output SST. Splitting short series before the merge made it emit one small batch per series and input file, which the SST writer and indexer then handled one by one. Compaction now splits only when series are long enough to fill most of a batch. With 16 L0 files of 10k series x 30 rows, one output went from 160,560 merged batches to 592, and merging plus writing took 1.66s instead of 2.67s. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): benchmark compacting L0 files into one output The benchmark writes 16 L0 files through the engine, each holding every series over its own time span as consecutive flushes do, and times a strict window compaction that merges them into one output. Every iteration compacts a fresh copy of the same files. With 10k series of 30 rows per file, keeping short series unsplit in compaction merges cut the time from 830ms to 520ms. With 60 series of 5,000 rows per file, which compaction still splits, both took 266ms. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
791f429246 |
fix(mito2): validate bulk insert batches against the region schema (#9698)
* fix(mito2): validate bulk insert batches against the region schema Bulk inserts were only checked for the time index column, unlike row writes, which WriteRequest::check_schema checks against the region schema. A batch with columns of another type, nulls in the time index or in non-nullable columns, duplicate or unknown columns reached the region worker: a time index that is not a timestamp or holds only nulls made the worker fail, and other mismatches failed after the WAL append or were stored as written, depending on the SST format. Move the column checks of check_schema into a core shared by row writes and bulk inserts, and check bulk insert batches at the same points as rows: when the engine builds the worker request, and again in the worker if the region schema changed since. check_schema keeps its behavior. Bulk insert columns must have the column's Arrow type, except that tags may use any string encoding, as the memtables already convert them. Sparse batches must carry one encoded primary key column of binary without nulls. A batch that misses columns gets them filled by the worker even if it claims to match the current schema. The metric engine checks logical bulk inserts against the logical region before rewriting them into the physical region, as the rewrite copies columns without checking them and would encode a duplicated tag into the primary key twice. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): report an error when a write is interrupted by a panic WriteNotify answers its waiter when it is dropped and reported success unless an error had been set, also when it was dropped while unwinding from a panic in the middle of the write. Report an error in that case. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): tolerate a poisoned series lock when dropping a series map SeriesMap::drop reads every series to update the memtable metrics and unwrapped the lock, so dropping the map while unwinding from a panic that poisoned a series lock panicked again. Read through the poisoned lock instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 yihong
|
4684b95f72 |
ci: do not run these actions on fork (#9721)
Signed-off-by: yihong0618 <zouzou0208@gmail.com> |
||
 fys
|
4e8bc6df86 |
fix(prom): align remote read label matcher semantics (#9633)
* fix(prom): align remote read label matcher semantics Signed-off-by: fys <fengys1996@gmail.com> * chore: optimize comment of validate_matchers * fix(prom): preserve existing binary label regex handling Signed-off-by: fys <fengys1996@gmail.com> * refactor: condition * test(prom): simplify remote read matcher tests and clarify comments * fix(prom): return binary remote read labels as UTF-8 * fix(prom): reject binary labels in remote read * refactor(prom): align remote read null label handling with PromQL Signed-off-by: fys <fengys1996@gmail.com> * refactor(prom): simplify null handling in label matcher branches Signed-off-by: fys <fengys1996@gmail.com> * fix(prom): restrict remote read labels to string types Signed-off-by: fys <fengys1996@gmail.com> * test(prom): use string label in default value column test Signed-off-by: fys <fengys1996@gmail.com> * fix: code review --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
jeremyhi
|
cca95b2637 |
fix(datanode): stop the object store WAL before the region server (#9639)
* fix(mito2): cancel flushes waiting for the WAL when the engine stops A flush waits for the WAL to be durable through the entry id it records before it writes its manifest edit. With the object store WAL in the `enqueued` mode that wait lasts until an upload succeeds, while the log store is stopped only after the engines. A graceful shutdown during an outage of the WAL's object store therefore never finished: closing a region does not cancel its running flush, stopping the worker group waits for running flushes and runs the queued ones, and a region holding writes that skipped the WAL is flushed on close. The worker group now owns a cancellation token, raised before its schedulers stop and kept raised. `RegionEngine::begin_stop`, which the region server calls before it closes regions, raises it earlier, so that a close-time flush cannot wait either. A flush polls its durability barrier first and gives up at the barrier once the token is raised: it writes no manifest edit, deletes its SSTs and fails as cancelled. A flush whose entries are already durable commits as before. The region server reports a failed close, which loses the region's writes that skipped the WAL, in its stop result. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): wait for the outage on the store under test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): give a healthy WAL a grace period before a stopping engine cancels flushes A flush of a stopping engine no longer gives up its wait for the WAL as soon as the stop begins. It gives up at once while the log store reports that it fails to make entries durable, through the new `LogStore::is_failing`, and otherwise 5 s after the engine began to stop, so a healthy object store WAL uploads its last entries first and a close-time flush keeps the writes that skipped the WAL. The object store WAL reports that it fails while the last create of its backlog failed, which covers a refused and an unreachable object store, and once it is poisoned or stopped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drain the WAL for a stopping flush and cancel its failed waits A flush of a stopping engine now asks the log store to drain before it waits out the grace period, through a new `LogStore::begin_drain` that does nothing by default. The object store WAL then seals its open batch at once and, until it stops, as soon as a durability wait needs a region it holds, so a healthy store makes the entries durable after about one upload whatever its `flush_interval`, which has no upper bound. A barrier wait that fails while the engine stops, for instance on a poisoned or stopped log store, now fails the flush as cancelled like a wait that gives up. The healthy-store test no longer depends on the seal timer: its store never seals on its own, and the close must finish well within the grace period. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drain the WAL when the engine begins to stop The drain started only when a stopping flush reached its durability barrier, so a close behind slow flushes that reached it after the grace period had no time left to upload its entries. The worker group now asks the log store to drain as soon as it begins to stop, and the object store WAL seals its open batch then and every later admission right away, so its entries become durable while the flushes still write their SSTs. `LogStore::begin_drain` becomes synchronous for that. The healthy-store test now writes after the stop began and closes the region only after the grace period. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): refuse appends once the object store WAL drains A WAL write admitted late in a stop could still be uploading when the close-time flush of its region reached an expired grace period, so the writes that skipped the WAL in that region were lost although the store was healthy. A draining object store WAL now refuses every later append, also one held back, with the stopped error, so the entries a stopping engine waits for are fixed when the stop begins and become durable while the regions still close; writes that skip the WAL are still taken. Tests cover a close-time flush whose wait completes within the grace period once a parked create is released, a close that reaches its wait after the grace period, and a drain refusing a held-back append. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(datanode): word a failed region close neutrally in the shutdown log A close can fail for reasons other than its flush, such as a stopped worker, and for regions without writes that skipped the WAL, so the error line no longer states that such writes were lost: it says they may be. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(datanode): stop the object store WAL before the region server The datanode stopped the object store WAL only after the region server, whose stop waits for flushes that wait for the WAL to become durable, so a shutdown never finished while the WAL's uploads kept failing. The WAL now stops first: its stop uploads the backlog or drops it and answers every wait for its entries, so a running, queued or close-time flush either commits or fails with the WAL-stopped error without writing a manifest edit. This replaces the stop signal and grace period the engine used to give up those waits, and the log store and engine hooks they needed. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(datanode): check rows after a healthy shutdown and wait for the WAL stop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: keep only the shutdown tests that prove the new stop order The engine-level tests pinned the log store's stop contract, which the log store's own tests already cover, and the stalled-create test asserted a limitation rather than a guarantee. Drop them with the testing hooks only they used; the datanode and region server tests cover the change end to end. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
baa5630119 |
perf(log-store): keep the object store WAL reclaim boundary independent of the region count (#9644)
* perf(log-store): keep the object store WAL reclaim boundary independent of the region count The WAL actor recomputed the reclaim boundary after every watermark update, release and indexed object by scanning the footers of the objects from the previous boundary, checking each segment against the obsolete watermarks. Each scan started at the first segment of the boundary object, so when one object holds segments of many regions whose watermarks advance one region at a time, every update inspected the segments already known to be obsolete again, about R^2/2 footer entries in total for R regions. The catalog now keeps the obsolete watermarks itself and counts the live segments of every object. A watermark move visits only the segments of its region that it newly reaches, through the per-region index, and the boundary is the first object at or above the floor that still holds a live segment. The boundary is unchanged: segments of release records never hold it, a region without a watermark does, a watermark of EntryId::MAX releases every segment of its region, and it never moves below the floor. The store records each watermark in the catalog together with the one reads use. The per-region index is now keyed by the largest entry id of each segment, which is also object sequence order because the entry ranges of a region increase strictly with the sequence. Indexing an object already enforced that; it now checks the neighbours by entry id instead of by sequence, which accepts and rejects the same objects. A property test checks the boundary against a scan of the footers over random appends, watermark moves, releases, drops, collections and recoveries, and ignored measurement tests time watermark updates and tail reads through the store. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): seek the object store WAL objects of a region by entry id A read found the objects of its region by filtering every object of the region that the catalog retains, so a read of the last few objects cost time in proportion to the whole retained history of the region, under the catalog read lock. The per-region index is keyed by the largest entry id of each segment, and the entry ranges of a region increase strictly with the object sequence, which the catalog enforces when it indexes an object, whatever entry ids the objects carry. So the lookup seeks to the first segment that ends at or after the start of the range and stops at the first that starts after its end, and the result is still in sequence order. The property test of the catalog now also checks range lookups against a scan of the footers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 DeviousCardi
|
095c288f16 |
fix(promql): honor @ modifier on subqueries (#9446)
* fix(promql): honor @ modifier on subqueries `prom_subquery_expr_to_plan` never read the `at` field of a subquery, so `sum_over_time(m[20s:10s] @ 60)` was evaluated like `m[20s:10s]` and followed the outer evaluation grid. Handle `@` the way Prometheus does: fold the anchor into the subquery offset (`offset + (eval_start - anchor)`, as `setOffsetForAtModifier` does), evaluate the subquery once at the start of the evaluation, and replay the folded window over the outer grid, reusing the anchored range selector machinery. A call whose range argument is an anchored subquery is promoted as step-invariant, like one over an anchored range selector. `@ start()`/`@ end()` and `offset` combine as for selectors, and an unrepresentable anchor is rejected with the existing error. Closes #9411 Signed-off-by: Aarav <aaravsjadav@gmail.com> * test(promql): add sqlness cases for @ on subqueries Cover the #9411 reproducer end to end: `@ <ts>`, `@ start()`/`@ end()`, `@` combined with `offset` in either order, an instant query away from the anchor, `*_over_time` functions and `rate` over an anchored subquery, a computed inner expression, and an aggregation above the anchored call. Two series per table so a dropped or merged series shows up as a wrong row. Refs #9411 Signed-off-by: Aarav <aaravsjadav@gmail.com> --------- Signed-off-by: Aarav <aaravsjadav@gmail.com> |
||
|
jeremyhi
|
e68e8c46d8 |
perf(log-store): reuse the release records the object store WAL recovery already read (#9645)
* perf(log-store): reuse the release records the object store WAL recovery already read Recovery reads every object it lists to verify its header and footer: an object of at most 64 KiB whole, a larger one as its header and the last 64 KiB. It then read the segment of release records of every indexed object that holds one again with a range read, although the segment, the last one before the footer, is almost always among the bytes already read. A prefix of many dropped regions, each released in an object of its own, paid two reads per such object. The footer read now keeps a copy of that segment when the bytes it read hold it, and recovery decodes the copy, or reads the segment as before when the copy is missing, only for the objects on the selected chain. The segment is copied so that the read window is not held until the releases are decoded. Chain selection, the merge of the largest released entry id per region, and the failure of recovery on a release segment that cannot be fetched or decoded, whatever on_corrupted_segment says, are unchanged. Tests cover the reads of a recovery whose release segments are all in the read bytes, a release segment that starts before the tail window, and that a damaged release segment fails recovery on the chain and is never decoded off it. An ignored measurement test counts the reads and times recovery over a prefix of dropped regions. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): bound the release segments object store WAL recovery keeps Recovery keeps the segment of release records of every object it fetches before it selects the chain, so an object off the chain could hold up to the tail window until recovery ends. Keep a segment only up to 4 KiB, far above the records of the few regions a release seals at once, and read a larger one again only when its object is on the chain, as before. Tests cover a release segment inside the tail window of an object read as its header and its tail, which exercises the offset into the window, and a segment above the kept size in an object read whole. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
822c90c15f |
fix(test): download darwin release assets for compat runs on macOS (#9630)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Weny Xu
|
98f79d0cf9 |
fix(meta): allow affinity fallback during failover (#9641)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
0e82b29b23 |
ci: add crash, restart, multi-region and drop cells to the S3 WAL benchmark (#9632)
* ci: add crash, restart, multi-region and drop cells to the S3 WAL benchmark Extend the object store WAL S3 benchmark beyond one load per configuration: - Inputs for a full image reference, the configurations to run (R0 adds Raft Engine with the standalone default sync_write = false), extra environment for every server container, a restart after each load, and a JSON list of named cell groups that override those inputs. - Restart cells SIGKILL the loaded server, restart it on the same prefix and data home, and time /health, the first correct count and the log store recovery; min_objects fails a cell that left too few objects indexed. - Crash groups write striped 1000-row batches from several writers, kill the server at a random point, restart it and check every acknowledged batch against the table (durable: present exactly once; enqueued: no duplicate, no partial batch, losses counted). A pinned variant keeps one row of a second table unflushed and samples the indexed objects while it runs. - Idle windows scrape the server before and after a quiet period. - A load group may split the loaded table into equal event_time ranges, so every batch writes to every region, and checks that each region took rows. - Drop groups write a table, drop it, kill and restart the server, then write and flush another table and wait for the dropped table's objects to be collected. - The summary adds WAL traffic, objects per second, mean object size, entries and appends per object, S3 requests per operation, server and host CPU, the loader's own request latency, rows per region, crash cycles, and each restart's replay per region from the server log. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: report the object sequences a replay spans instead of objects replayed The width of a region's replayed entry range counts every object sequence in it, including objects that hold no entry of the region and are skipped, so the summary labels it as a span. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: fail a crash cycle that loses a batch an earlier restart recovered The oracle accepted every missing acknowledged batch in the enqueued mode, so a batch that one restart recovered and a later restart no longer found passed. Every cycle keeps what its restart found, and a cycle now fails in either mode unless every batch an earlier restart recovered is still present, so only the current cycle's backlog that was not durable at its kill may be missing. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: keep the pinned WAL pinned and label OpenDAL operations as such The pin table inherited the engine's 10-minute auto flush, so a longer pinned cycle could flush its row and release the WAL before the kill. It now sets its own auto_flush_interval of one day, longer than any crash group can run, and a pinned cycle fails unless its indexed objects never fall while it runs. The summary's request counts come from opendal_operation_duration_seconds, which counts OpenDAL calls rather than HTTP requests (a paginated list or a multipart write is one operation), so the fields and columns now say S3 operations. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: check a pinned cycle's retention by the collected-object counter Net growth of the indexed objects can hide collection, and short cycles take no periodic sample, so a pinned cycle now fails unless deleted_objects_total is the same in the scrape before its writers start and in the one before the kill, both of the same server process. The operation columns name OpenDAL. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: let a pinned cell collect only the objects below its pin The first write after a start lets the store collect the empty start object asynchronously, so a deletion after a cycle's baseline does not mean the pin let go. A pinned cell now scrapes the server before it writes the pin row and fails only once its servers together have collected more objects than were indexed then, counting the first server from that scrape and every restarted one from its start. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * ci: test that a pinned cell's allowance spans its server restarts Two servers that each collect fewer objects than lie below the pin but more together now fail the cell in the second cycle. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
8b0a155702 |
perf(otlp): chunk trace spans across resource groups (#9621)
* perf(otlp): chunk trace spans across resource groups Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(otlp): plan spans of a mixed binary/JSON chunk individually Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(otlp): describe cross-group trace chunking Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
672e94f61d |
feat: add column encoding option and use it for trace span ids (#9629)
* feat: add column encoding option and set it for trace span ids COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'. Either disables the dictionary for the column; delta_binary_packed is limited to integer and timestamp columns. OTLP trace v1/v2 tables now create span_id and parent_span_id with plain encoding and 64KiB pages, and timestamp_end with delta_binary_packed. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: add compatibility cases for column encoding options Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: move column encoding downgrade checks into downgrade_compatibility Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
88bdab0f66 |
fix(mito2): keep processing bulk requests after one fails to encode (#9638)
* fix(mito2): keep processing bulk requests after one fails to encode When `push_bulk` fails to build the WAL entry of a bulk request, it answers that request with the error. `process_bulk_requests` then returned from the loop, so the remaining requests of the drained batch were dropped without an answer and their callers only saw a receive error. Skip the failed request and continue with the rest of the batch instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): keep bulk install barriers independent per target The bulk install test barrier was a single process-wide slot, so tests that arm it while running in the same process replaced each other's barrier. Keep one barrier per armed target instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Lei, HUANG
|
9089ea7a4b |
fix(mito): avoid pinning region version when rebuilding indexes (#9597)
refactor(mito): avoid pinning region version when rebuilding indexes handle_rebuild_index kept the whole region version alive across the manifest read and schedule_build awaits, which pins memtables and the SST version even though index build tasks only need the region metadata, index options and the candidate SST file handles. Extract those inputs into a small IndexBuildSnapshot and drop the version before awaiting. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
a0188c8c41 |
docs: cover pre-releases and upgrade notes in the release-note skill (#9635)
* docs: cover pre-releases and upgrade notes in the release-note skill Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: add style and length rules to the release-note skill Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: drop machine-specific notes from the release-note skill Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: base the first pre-release on the previous release branch point Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: subtract the whole previous line when building a minor release note Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 localhost
|
3de8610e2e |
fix(ci): compare release validation against the previous stable version (#9636)
Ancestry-based tag selection skips stable tags that live only on older release branches. Signed-off-by: paomian <xpaomian@gmail.com> |
||
 discord9
|
778f4556a8 |
feat: unify query options across SQL, HTTP and gRPC (#9494)
Isolated rebase verification of PR #9494. Preserve the reviewed net change; original PR branch remains untouched. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Lei, HUANG
|
f663569cf0 |
feat(auth): expose public authentication config (#9626)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 YingNing
|
ce428bb74d |
feat(servers): accept application/x-json-stream for log ingestion (#9581)
* feat(servers): accept application/x-json-stream for log ingestion
Support fluentbit's json_stream output format on the HTTP log ingestion
endpoints by accepting `application/x-json-stream` as a payload content
type.
The payload is parsed as concatenated JSON -- multiple JSON values placed
back-to-back without a separator, e.g. `{"a":1}{"b":2}` -- reusing the
streaming value iterator already used for `application/json`, so the
behavior for concatenated or mixed payloads is identical.
Aimed at #5404 (msgpack support is left as a possible follow-up).
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
* chore(servers): remove redundant comments per review feedback
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
---------
Signed-off-by: dxqlovestudy <59612604+dxqlovestudy@users.noreply.github.com>
|
||
|
Weny Xu
|
4859e57e0b |
feat(meta): apply affinity to failover and repartition (#9604)
* feat(meta): apply affinity to failover and repartition Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): share repartition affinity until group allocation Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
deae75ffee |
fix(log-store): refuse to open object store WAL regions whose history is missing (#9561)
* fix(log-store): refuse to open object store WAL regions whose history is missing A region persists only its WAL prefix, not the object store that holds it. After `[wal].storage_provider`, or the bucket or root of the provider it names, changes, or after the objects under the prefix are deleted, every region reopened against a prefix without its history, replayed nothing and silently lost its acknowledged writes that were not flushed. Add `LogStore::check_region_history` and `LogStore::start_region_history`, whose defaults accept and do nothing, so Raft Engine and Kafka are unchanged. The object store WAL reports the history of a region as missing when recovery found no object under the prefix, or when the region's watermark names an object above the newest object holding entries that recovery indexed, unless the catalog holds the region's entries through the watermark or the region's history starts in the store. Both rest on invariants of the prefix: an opened prefix is never empty, and a persisted watermark names an object at or below the newest object holding entries, which collection never deletes. The segment of release records counts as entries, since an object holding only release records can be the one collection keeps; start objects hold no segment and do not move the bound. When a store that found its prefix empty refuses a region, the actor gives up its start object, so the next start refuses the region again. It agrees only while nothing was handed an id, no batch, append, release or create waits, no failed release is kept, no other object was listed, the catalog holds only the start object and no region's history started in the store, deciding under the lock region registration takes. It then records a terminal error, deletes the object as one of its tracked deletes, which `stop` waits for, and answers every refusal once that delete settled. Mito runs the check before it replays an existing region on open, on catch-up and when a create finds the region present, with the larger of the flushed and truncated entry ids, and records the history of a region it creates; regions that skip the WAL and noop regions are not checked. A missing history fails the open with an error naming the region, the prefix, the configured storage and how to move the WAL by copying its objects. The new `[wal].accept_missing_history` option, off by default, opens such a region without its unflushed writes instead, with a warning and the counter `greptime_logstore_object_store_wal_missing_history_accepted_total`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the object store WAL history check Describe the open-time history check, the prefix invariants it relies on, how release records bound the history, the start object withdrawal of a refused open and the window that remains, the accept_missing_history option and how to move a WAL by copying its objects, and update the metrics, Mito integration, evidence, compatibility and cluster alignment sections. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): unindex the start object a refused open withdrew A withdrawal that deleted the start object left it in the catalog and in the indexed object and byte gauges for the rest of the process. Remove it from both once the delete succeeded, and keep it when the delete failed. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): separate the withdrawal windows of start objects and entries An append or a release admitted before a refusal leaves an object holding entries, which may have collected the start object, so the next start also passes watermarks at or below that object. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep a successor's chain complete across a late withdrawal A start object withdrawn after another instance recovered it was linked by that instance's start object, which carried reclaim boundary 0, so once the delete landed the next open found no complete chain. A start object now reclaims every object below it when the recovered chain holds no segment, so such a delete stays below its boundary. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): record a created region's WAL history before its manifest A create recorded the region's WAL history after it persisted the manifest, so a withdrawal in between failed the create but left a manifest that the next start refused as a region without its history. Record the history first, so such a create fails before it writes anything. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the start boundary of a chain without segments and the create order Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
fys
|
cf0f6c308a |
fix(prom): return empty remote read results for missing metrics (#9613)
* fix: return empty remote read results for missing metrics Signed-off-by: fys <fengys1996@gmail.com> * test(prom): cover empty reads for missing databases Signed-off-by: fys <fengys1996@gmail.com> * test(prom): simplify remote read regression cases Signed-off-by: fys <fengys1996@gmail.com> * fix(prom): log missing tables during remote read Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
discord9
|
60097a0628 |
ci(coverage): set RUST_MIN_STACK=8MB in merge_group coverage job (#9615)
The merge_group coverage job did not set RUST_MIN_STACK, so test threads ran with the default 2MiB stack while the regular test job (and the production runtime in debug) use 8MiB. The deeper PromQL planning chain from #9494 overflows 2MiB under llvm-cov instrumentation, aborting tests::promql_test::anon_promql_ratio_repro::case_1_test_with_standalone with SIGABRT (12/12 failures across three merge-queue candidates). Align the coverage job with the existing 8MiB test policy. Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
c3469fad51 |
fix(flow): retain tracked write completion across transient failures (#9525)
* fix(flow): retain tracked write completion across transient failures Use receiver-issued single-write capabilities at the frontend and datanode. Rejoin exact admitted writes across unknown transport outcomes, then close and drain before allowing subsequent rounds. Keep ordinary clients and reads on their existing paths. Separate schema map equality from encoded row identity so fresh protobuf decodes replay consistently even with NaN values. Cover receiver admission, owner cancellation, all-child drain, and retired capabilities. Local feature-worktree checkpoint only. Main verified core enterprise check and 484 tests, plus DN enterprise check and all four tracked owner tests. Corrected seven-case timer acceptance remains to be rerun; this checkpoint does not authorize publishing or integration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): preserve upstream dependencies after rebase Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): fail fast on definitive tracked-write open rejection A tracked write treats every application-level error from the frontend as an ambiguous transport failure and retries the same pinned frontend forever. During a rolling upgrade an older frontend does not recognize the tracked-write command extension and processes Open as an ordinary empty SQL query, rejecting it with InvalidSql (InvalidArguments); a non-internal endpoint rejects the extension outright. Both are definitive rejections that no same-frontend retry can ever resolve, so the flow stalls. Classify the Open-phase errors by status code: a definitive rejection (Unsupported / InvalidArguments / PermissionDenied) fails fast with a terminal pre-write error, while genuinely ambiguous transport failures (Unavailable, timeout, connection reset, ...) keep the same-capability retry. Write/Close rejections are deliberately NOT failed fast: once a write may have been admitted, a same-code rejection (e.g. WriterLost, or UnknownCapability after a VIP moved Open->A/Write->B) leaves the outcome ambiguous, and failing fast would let the caller replay and duplicate rows. Add a regression test that a legacy empty-SQL Open rejection fails fast with a single attempt and surfaces as Error::External carrying a TerminalError. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): preflight tracked writes without submitting data Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): retain tracked command extensions while decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Lei, HUANG
|
33879b07c8 |
chore: bump rustls to 0.23.46 (#9614)
rustls 0.23.46 fixes rustls/rustls#3301: on TLS 1.2 with AES-GCM, reaching the 2^24 record limit dropped a fragment while reporting the write as complete, then kept sending records after close_notify. This stalled Kafka WAL produce requests until the request timeout. |
||
|
shuiyisong
|
ad823db020 |
fix(pipeline)!: disable VRL host access by default (#9610)
fix(pipeline): disable VRL host access by default Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Lei, HUANG
|
0dc81777e3 |
fix(pprof): initialize framehop before signal sampling (#9612)
fix(pprof): pin upstream framehop initialization fix Pin pprof workspace-wide to 01cff82d containing tikv/pprof-rs#282. Initialize framehop before sampling instead of allocating and discovering shared libraries in the first SIGPROF handler. Regenerate the lockfile without unrelated dependency changes. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
7df99f946e |
test: add manual S3 metric snapshot benchmark (#9573)
* test: add repeatable metric snapshot S3 benchmark Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use only existing CI S3 permissions in the metric snapshot benchmark The CI S3 user is not allowed GetBucketLocation or ListBucketMultipartUploads, so the bucket probe and the prefix cleanup failed with AccessDenied. Use only list, put, get and delete under the run prefix, as the object store WAL benchmark does, and probe S3 before allocating the EC2 runner. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: scope metric snapshot inventory to the current round The full round's S3 inventory listed the whole run prefix, so it also counted the smoke snapshot. List only the round's snapshot prefix while cleanup still covers the run, and assert that the teardown's S3 cleanup step runs always() with the validated run prefix. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: record the AWS validation run of the metric snapshot benchmark Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
cbfbad469f |
feat(meta): add affinity hooks (#9596)
* feat(meta): add affinity hooks Signed-off-by: WenyXu <wenymedia@gmail.com> * test(meta): cover load-based affinity fallback queries Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): borrow affinity requirements during selection Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): remove unused affinity error variant Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
dennis zhuang
|
f7a3a7033c |
fix(query): keep state column names unique when splitting aggregates (#9594)
* fix(query): keep state column names unique when splitting aggregates Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): probe allocated names when renaming colliding state columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
cf16c7313b |
fix(jaeger): return the most recent traces when a search hits its limit (#9595)
* fix(jaeger): return the most recent traces when a search hits its limit Trace search applied LIMIT to an unordered DISTINCT trace_id, so a search matching more traces than the limit returned an arbitrary subset that changed between identical requests. Order traces by their latest matching span before applying the limit. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(jaeger): name the trace dataframe builder with a verb and document execution Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
af396e60a3 |
perf(query): infer safe aggregate input bounds from HAVING (#9582)
* perf(query): infer safe aggregate input bounds from HAVING Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(query): refresh explain snapshots for aggregate bound inference Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(query): avoid reinferring bounds across MergeScan Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(query): validate aggregate bound column provenance Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(query): retain zero-match groups for nonpositive HAVING thresholds Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(query): assert inferred bounds in remote scan plans Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Yingwen
|
ce66d8d4ac |
fix: avoid pinning memtables during series-index reconciliation (#9588)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
5cfdc30a69 |
fix(table): stop claiming exact null and distinct counts for append scans (#9593)
RegionScanExec reported null_count = 0 and distinct_count = total_rows for every column of an append-only table. DataFusion answers count(col) and count(DISTINCT col) from these statistics, so both returned the row count. Keep the exact row count and an exact zero null count for non-nullable columns, so count(*) is still answered from statistics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
ef1298ce02 |
fix(query): accept narrowed nullability in MergeScan remote schema check (#9592)
On append-mode tables a region reports an exact row count. When it is zero, DataFusion's AggregateStatistics replaces the region-side __min_state/__max_state aggregate with a non-null struct literal, so the remote stream advertises the state column as non-nullable. The MergeScan schema check required equal nullability and failed the query. Accept a remote field that is non-nullable where the expected field is nullable, matching DataFusion's own physical-plan schema invariant. Batches are already rebuilt with the expected schema. Widening is still rejected, and nested field nullability is still compared exactly as part of the data type. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
60655a933b |
fix(log-store): persist object store WAL region releases (#9579)
* fix(log-store): persist object store WAL region releases A region's release lived only in memory: a restart after DROP TABLE and before the next object never re-established the dropped region's watermark, so its segments counted as live and pinned garbage collection of the whole prefix. A release now writes a release record (region id and the entry id the region is released through) into an object sealed at once, and returns once that object is durable and indexed, in both acknowledgement modes. Recovery seeds the obsolete watermarks from the release records of the indexed objects before it computes any reclaim boundary. obsolete_all releases in full; the new LogStore::release_region, a no-op by default, releases a dropped region through the largest entry id the store handed out for it, and Mito calls it after obsolete when it drops a region. Objects with release records use format version 2, the version 1 layout with the records at the end of the footer; every other object is still written as version 1. A counter of release records made durable is added. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe persisted object store WAL region releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): carry a failed region release into the next object A release whose batch failed left its record only in an orphan object, so a drop that failed there could still leave the region unreleased after a restart. The store now keeps the release records of failed batches and seals them with its next batch, on a timer tick when no entry is admitted, so the release becomes durable once a create succeeds again. The version 1 and version 2 object fixtures are frozen under tests/compatibility/wal/object_store and loaded by the format tests, and the RFC no longer says that state is persisted only in headers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): complete a region drop whose WAL release fails A drop request is not sent again once the region was removed, so a drop whose WAL release failed used to leave the region half dropped. The drop now completes, cleaning up the region's runtime state and scheduling its files for deletion, and reports the release failure. In the durable mode the object store WAL now creates one more object at stop with the release records of failed and dropped batches, so a graceful stop does not lose a release that failed. The release record counter is described as counting the records of indexed objects. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe the stop-time release create of the WAL actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): state the accepted window of unreleased dropped regions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe release waiters in the WAL actor's outcome table Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail region releases while the object store WAL is failing A failing store keeps its backlog and rejects appends until a create succeeds, so a release would otherwise wait for an upload that is not coming. A release that arrives in that state now fails at once with the retryable error, and one whose batch is still pending when the store enters it fails as well. Neither poisons the store: the record of the first waits for the next sealed batch, like that of a batch that failed, and the record of the second stays in the backlog, so both become durable once a create succeeds. Rejected releases count in rejected_appends_total. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe region releases while the object store WAL is failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the drop crash window in the drop procedure The crash window between the deletion of a table's metadata and a durable release record is a gap of the drop procedure, not of the WAL. Describe it as far as the code shows, name the remedy that belongs in the procedure, cleaning up offline a region a resumed region drop finds not open, and list it under future work. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep rejected release records coalesced while failing A release rejected while the store is failing kept its record for the next sealed batch, but every timer tick then sealed that record alone behind the refused backlog, so a release repeated across ticks grew the backlog without bound. A failing store now keeps such records coalesced and seals them on the first tick after a create succeeds. The fault-matrix case of a release while the backlog is refused now runs with a 10 ms flush interval, repeats the release, and checks that the backlog does not grow and that recovery reads the record back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): keep rejected release records unsealed while failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): cite the drop procedure issue for the crash window Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): let a stop seal rejected release records of a failing store Keeping the records of rejected releases out of sealed batches while the store is failing also kept them out of the seal a stop requests, so a stop whose last attempt of the backlog succeeded dropped them and the released region pinned collection after the restart. A stop now seals them with the backlog. A fault-matrix case rejects a release while the store is failing, stops, lets both creates succeed, and checks the record after a reopen. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the stop of a failing store with rejected releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep release records readable by older object store WAL readers Release records were written as a version 2 footer, which a build without it refuses to read, so rolling a node back after a drop failed to open the whole WAL prefix. They are now the single entry of a segment of the reserved region u64::MAX at the first position of a version 1 object: an older reader indexes that segment as a region it never opens, and recovery reads the segment back with one range read per object that holds one. The segment never holds the reclaim boundary and names no namespace. A durable stop whose last create of release records fails now reports the error instead of returning Ok and losing the records silently. The tests are trimmed: the release unit tests and fault-matrix cases that repeated each other are merged, and the Mito drop tests share one parameterized case. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe release records as a segment of a reserved region Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): report release records a poisoning stop drops A durable stop whose last create of release records met a conflict poisoned the store and returned Ok, losing the records silently. Stop now reports the error whenever release records are dropped after it began, by a rollback or by poisoning. The test of a failed release covers that conflict, and seals its record on a tick after a one-shot create failure, so the next tick cannot fail too. The documentation now says that recovery reads the segments of release records, and that one that does not decode fails recovery. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how recovery reads segments of release records Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): say that stop reports release records a conflict drops Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): restore the transiently failing release create in enqueued mode The fault matrix again covers a release whose create fails transiently in the enqueued mode: the release waits for the repeat under the same sequence and returns once it succeeds. The encode_batch doc now says the release segment is the last segment of the object. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the release segment and count the restored case Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL release Returning the release error from the drop left a metric physical region half dropped: the metric engine stops after the data region and never drops the metadata region or forgets the physical region, and a retried drop finds the data region gone and counts it as done. The release error gave the caller nothing to retry, since the region is already removed and the log store keeps the record for its next object. The drop now logs the error and succeeds. A metric engine test drops a physical region while the WAL releases of both mito regions fail, and checks that both regions and the physical region state are gone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed release no longer fails the drop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL obsolete A store that the release of another region poisoned fails obsolete as well, which left the metric engine with its metadata region undropped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(metric-engine): drop a physical region whose WAL obsolete or release fails Cover a failed data release, two failed releases and a conflicting release that poisons the store, after the regions wrote to the WAL, and check that a store opened later keeps no segment of either region. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed obsolete does not fail the drop either Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): fail the drop on an obsolete error of other WAL providers Only the object store WAL is shared by the regions of a store that the release of one can poison; Raft Engine and Kafka keep failing the drop when obsolete fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a stopped or poisoned store keeps no failed release Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Ning Sun
|
40770ddb72 |
test: only assert genuinely invalid read-preference values at request ingresses (#9576)
The flow and nested-MergeScan ingress tests hardcoded "follower" among the read-preference values that must be rejected, coupling them to the current leader-only ReadPreference enum. Assert only genuinely invalid values there, matching the wire-values test which already limits itself to ["", "unknown"]. This keeps the ingress tests valid for builds that extend ReadPreference with more variants. Signed-off-by: Ning Sun <sunning@greptime.com> |