mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-18 16:54:09 +00:00
643bdd72db9f7972f7a2368746467cfa8fd50801
5479
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
643bdd72db |
fix(metric-engine): reuse existing column ids when adding columns is retried (#10372)
The metric engine derives the columns to add from the cached physical schema, and `DataRegion::assemble_alter_request` assigned every given column `max(non-reserved column id) + 1` without checking the existing names. That cache can lag behind: Mito reads its in-memory metadata without going through the region worker, and a caller future dropped on a client timeout does not withdraw the request that is already queued. A retry then treats an installed column as new and Mito rejects it, because the name exists with a different id. Recheck the real metadata in `assemble_alter_request`, validate an existing column with `validate_shared_column` and drop it from the payload, assign ids only to genuinely new columns, and skip the alter when nothing is new. Serialize the existence check, the id allocation and the physical alter per physical data region: the mutex lives in `PhysicalRegionState` and is held by the metric engine orchestration, so concurrent requests on logical regions sharing the region cannot allocate one id to different columns. `DataRegion` stays a stateless adapter. Build the logical mapping and the cached physical columns from the metadata read after the alter, and report a missing column as an error instead of panicking. Signed-off-by: Wenyu Xu <wenymedia@gmail.com> |
||
|
|
6e025eef61 |
fix(auth): secure MySQL and PostgreSQL database selection (#10382)
* fix(mysql): authorize the default database at login Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): validate MySQL handshake scope and SQL USE Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): authorize PostgreSQL schema selection Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(servers): defer PostgreSQL database lookup until authorized Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(servers): propagate catalog lookup errors in protocol tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(postgres): skip unusable search path candidates Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): log database authorization and selection outcomes Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): preserve search path responses and schema boundaries Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): clarify database selection diagnostics Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mysql): format handshake username in structured logs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(auth): secure database-less sessions and failed startup Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mysql): retain authorized default database selection Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(auth): separate database selection from request dispatch Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(mysql): align schema validation with database-less login Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(auth): explain database-less session restrictions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
f88af153ec |
feat(query): serialize remote dynamic membership (#10341)
* feat(query): serialize remote dynamic membership Recover single-column UTF-8 build-key membership from InList and HashTableLookupExpr producers and send it to remote datanodes using the existing PhysicalInListNode. Non-monotone or unsupported subtrees widen to true; the receiver keeps the live filter authoritative. Signed-off-by: discord9 <discord9@163.com> * refactor(query): send lookup membership as a key list Serialize a HashTableLookupExpr as the existing PhysicalInListNode and preserve the rest of the expression tree. Remove per-subtree monotone and true rewrites; only an unrepresentable lookup or an over-budget payload falls back at the whole-payload level. Signed-off-by: discord9 <discord9@163.com> * docs: tighten narrow membership contract Clarify that the key-list transport is a positive pruning hint, not universal Boolean equivalence, and test that a NULL probe survives the IS NULL wrapper. Signed-off-by: discord9 <discord9@163.com> * fix(query): retain safe bounds on membership fallback When a key-list predicate cannot be shipped, keep top-level conjuncts that contain no membership and re-encode their conjunction. Drop membership-bearing conjuncts whole; fall back to true only when no safe bounds remain. Signed-off-by: discord9 <discord9@163.com> * fix(query): skip unrepresentable dynamic filter snapshots Do not synthesize a true predicate when a runtime hash-table lookup has no wire representation. Capture and fanout skip the affected snapshot, keep the query running, and report PushedDown::No so the parent filter is retained. Safe membership-free conjuncts still travel, and a producer-supplied true remains encodable. Cover restrictive-to-true updates, unsupported initial registrations with later supported updates, final unsupported cleanup, and NULL build keys. Signed-off-by: discord9 <discord9@163.com> * docs(query): clarify 642-key payload budget comment Signed-off-by: discord9 <discord9@163.com> * refactor(query): tighten dynamic membership encoding Preserve static IN conjuncts when only a runtime lookup is unrepresentable, apply the IN-list exclusion only on the over-budget path, and derive the membership count guard from the byte budget before materializing literals. Document that decoded dynamic filters are pruning-only, fix the NOT/CASE contract wording, deduplicate test helpers, and justify the schema-less InList proto encoding. Signed-off-by: discord9 <discord9@163.com> * fix(query): reset stale dynamic filters on unsupported updates Publish an explicit true reset when a producer generation cannot be carried, instead of leaving an older narrower predicate that could prune rows after widening. Keep the unsupported expression unrewritten; the reset is a transport-built invalidation. Type over-budget build-key memberships as DynFilterPayloadTooLarge, accept scalar-subquery IN lists under the conjunct-retention contract, and clarify the no-initial-predicate wording. Signed-off-by: discord9 <discord9@163.com> * docs(query): scope no-true rule to the encoder Clarify that the encoder never substitutes true for a producer node, while the long-lived fanout path publishes an explicit true reset to invalidate a stale region predicate. Point at the fanout caller instead of claiming producers always skip refused generations. Signed-off-by: discord9 <discord9@163.com> * docs(query): correct reset wording Fix the inverted conjunct-retention sentence and describe initial capture versus ongoing fanout resets consistently in the error docs. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
7eb766c9e4 |
feat(common-meta): make RDS metadata backends closable (#10417)
* feat(common-meta): make RDS metadata backends closable Standalone closes the metadata backend on shutdown, and `MetadataKvBackendCreator` now returns a `ClosableKvBackendRef`. Only the raft-engine backend implemented `ClosableKvBackend`, so external metadata backends could not be wired into the standalone creator. Implement `ClosableKvBackend` for `RdsStore` and return `ClosableKvBackendRef` from `with_mysql_pool`/`with_pg_pool` and the metasrv RDS builders. The stores only hold reference-counted connection pools that are released with the last backend handle, so `close()` has nothing to release explicitly. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(common-meta): close RDS connection pools on shutdown Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(common-meta): preserve closable RDS constructors and test shutdown semantics Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
180dfc68f7 |
perf(mito): reduce SST metadata memory overhead (#10356)
* perf(mito): reduce SST metadata memory overhead Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(cache): add a sharded weak interner Share equal live allocations without exposing lock guards or retaining payloads. Sweep candidate buckets incrementally and cover concurrent publication and weak-reference churn. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): share SST metadata through the sharded interner Require shared metadata wiring for manifests, canonicalize checkpoint entries without cloning their payloads, and centralize runtime registration. Keep live content variants distinct and verify runtime and recovered allocation identity. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: compile SST metadata consumers across feature sets Wrap fixture manifest metadata in Arc explicitly and gate the test-only HashMap import with its consuming feature. Validate default mito tests and all cmd targets and features with Clippy warnings denied. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): reclaim retired metadata while idle Run bounded interner maintenance on the shared blocking runtime without retaining its owner between passes. Shrink sparse indexes and release empty backing storage, preserving live readers and reclaiming their metadata after delayed release. Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito): use intern-arc for shared SST metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): expose interner counts and decouple manifest setup Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(partition): support hashing partition expressions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): hash intern SST metadata without ordering values Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
6faa9ec862 |
feat(mito2): replay pending WAL tails in the background (#10342)
* feat(log-store): read the WAL of another node through the log store `LogStore::read_node` streams one region's entries in `(after, upper]` from the WAL of another node that shares the log store's storage, without writing to it. The default returns `None`. The object store WAL implements it with a registry of read-only readers over its own object store. The node prefix comes from the configured root and the store's own generation, and the returned stream holds the reader, so reads of the same node share its cached object indexes. `WalReadUpperBound` moves to store-api so the trait can name it; log-store re-exports it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(mito2): replay pending WAL tails in the background A leader that is writable, altering or editing and still has pending tails replays them in one background task per region and publishes the result in one manifest edit, which opens the read gate and resumes compaction once no tail remains. - The worker starts the task when a lease, a catchup or a role change makes the region writable, and cancels it on close, drop, downgrade and when it replaces the instance. The task always reports its end, so the worker can start another replay or reload a region a conflict fenced. - Each tail is read from this node's WAL, or through `LogStore::read_node` for another node, and replayed with the per-entry code of open-time replay into memtables outside the region's version. Chunks of about 64 MiB are written with the SST half of the flush task, never across two tails, since an SST gives its rows its largest sequence. - The edit adds the SSTs, clears the tails, and advances the flushed entry id to the last entry read and the flushed sequence to the largest chunk sequence. It commits outside the edit queue, under the manifest lock, only while the region is a writable leader recording exactly the replayed tails. A busy region or an unknown result retries the same bytes; a conflict keeps the chunks; a certain failure deletes them. - Another node's listed tail is only a bound. After a catchup made the region writable, the edit keeps its unread part until the region receives a Leader lease, then a second edit clears it. - After the commit the task releases the WAL up to the new replay start, kept below this node's remaining tails, and the worker raises the region's last entry id to the new flushed entry id, so a later truncate cannot pull it back. - WAL pressure no longer flushes a region whose WAL release one of this node's tails caps. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): harden tail replay across leases, cancellation and uncertain commits - A Leader lease counts as received after a catchup only if its heartbeat was sent after the catchup promotion: the writable fast path does not check the start of a downgrade, so a lease granted before an earlier downgrade could otherwise clear the mark. The mark is the promotion time, so the fast path still reads one atomic. - A direct promotion counts as a lease. - The tail commit also raises the committed sequence, so later writes of the region take sequences above the replayed rows. - Registering a writable region starts a replay, which covers a create that reopens an existing region. - Once an attempt may have persisted the edit, later certain failures retry the same content instead of building a new edit. - WAL reads stop when the task is cancelled, and a cancelled task does not start another commit attempt. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that closing a region cancels a stalled tail read Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): hold tail commit attempts by number instead of sleeping Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): cancel tail replays when the worker stops The worker's cleanup closes its remaining regions without the close handler, so it now cancels every running tail replay first; otherwise a replay stalled in a WAL read or waiting for a lease outlives the engine. The stalled-read test covers an engine stop as well as a close. The docs no longer claim that every SST gives all its rows its largest sequence. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): shorten two tail replay comments Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): stop tail reads at their upper bound A tail read over this node's WAL kept reading past the tail's upto until the log ended, so an unreadable later entry failed a replay that already had every entry it needed. Stop the read once it reaches upto. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): inline the tail replay notify helper Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that writes during a tail replay win over replayed rows Whatever records pending tails must raise the region's committed sequence to the tails' largest sequence, so writes accepted while the replay runs get larger sequences than the replayed rows. Record this in AGENTS.md and test it with a write that lands between two commit attempts of the replay. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): bound reads of this node's listed tails A listed bound may name no entry, so a read of this node's WAL never saw an entry equal to it and kept polling later objects; one that failed to read failed the tail. Read every listed tail through `read_node`, which stops at the bound, and keep reported tails on the node's own WAL. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that rows around a tail replay survive a reopen The tail commit raises the flushed entry id to the largest entry read, which can be another node's. Whatever records tails keeps this node's later entry ids above every tail and leaves none of its unflushed entries below one outside its own tails. Record this in AGENTS.md, and make the replay test meet it and reopen the region without a flush afterwards. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): reject graceful staging entry of a region with pending tails Entering staging through `set_region_role_state_gracefully` skipped the pending-tail check that `EnterStaging` runs, so a region with pending tails could enter staging, end its replay, and return to Leader with nothing replaying its tails. Run the same check before entering staging there. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): replay tails of a region altered right after promotion A promotion checked whether it made the region writable after setting the role. If the worker started an allowed ALTER or edit in between, the check saw Altering or Editing and asked for no replay, and nothing asked for one later. Check for a leader that a replay runs on instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover the WAL release and locked checks of tail commits Check that a tail commit releases the replayed WAL, that it is refused under the manifest lock once the region was demoted or its tails changed, that a commit retried while altering keeps its chunks, and that a replay waiting for a lease does not end and start again. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): note that read_node also reads this node's WAL Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep no region version during a tail replay round A replay round held the region's version until it ended, and with it the memtables the region had when the round started. A flush removed them from newer versions, but their write buffer memory was only freed once the round dropped them, so a slow read, a commit retry or a lease wait could keep flushed memory charged and stall writes. Copy the tails and the flushed entry id instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): count a direct Leader role set as a lease only when it promotes A direct role set to Leader counted as a Leader lease even when the region already was a leader. On a region a catchup promoted, it cleared the mark that keeps another node's unread listed tail until a lease arrives, so the next commit could drop that tail. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): move the tail replay leader check onto MitoRegion The engine and the replay task both use it, next to the other role checks. Its doc now names the Writable, Altering and Editing states and why staging and downgrading regions replay nothing. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log when a tail replay waits for a Leader lease Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): pin the pressure skip, staging rejection and remote read error The WAL pressure test gets a case whose own tail starts above the flushed entry id and still flushes; the graceful staging case checks the rejection; and a remote tail on a log store that reads no other node's WAL fails as unsupported and keeps its tails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): list every tail replay trigger and the missing admission limit Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): rename tail replay helpers and notifications The names now say what each one checks, records or announces: the replay predicate is a capability rather than a running replay, the lease hook records a lease, the writable notice is an event like its siblings, and the commit notice pairs with the end notice. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): ask whether a Leader lease came after a catchup `is_writable_by_catchup` was negated at both of its call sites. Its inverse, `leader_lease_received`, is what the replay reads. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): plan the read of a pending tail in the tail replay `Wal` no longer interprets `PendingTail` or `UptoKind`. The replay task decides whether a tail comes from this node's own WAL or up to its bound through `Wal::read_node`, which now takes plain arguments. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log when a tail replay result goes stale Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): send the writable notice through notify_worker The tail replay already has one helper that sends a background notice to a region's worker; the writable notice now uses it too, and so logs a worker that has stopped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): note that tail replays skip staging regions and index like flushes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): say why each tail gets its own SSTs and what chunks are for Ordering between tails needs a tail per SST: an SST can give all its rows its largest sequence, so an earlier tail sharing an SST with a later one would take the later tail's sequence and outrank rows that a leader between the two tails flushed. Within a tail, the 64 MiB chunks only bound memory: the sequence floor already puts writes accepted during the replay above every tail. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): name the tail replay end guard like other drop guards Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): skip a tail replay edit that a sync already superseded A commit notice that the worker handles after a sync rebuilt the region's version from a later manifest no longer brings back tails or files that manifest removed. The notice carries the tails the replay read, and the worker applies the edit only while the version still records them. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log when a tail replay stops before publishing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover lock waits, busy edits and repeated triggers in tail replays Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log every exit that drops a tail replay Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): tell the event listener when a tail replay checks for a lease Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover uncertain commits, the SST hook and lease waits of tail replays Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log each tail replay exit once, where it happens Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): reuse the mock region hook and wait for role change tasks in tail replay tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover failed-round cleanup, instance replacement and leased first rounds of tail replays Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): make the replacement helper a TestRegion method and fix a lease comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
ed62f8ff54 |
feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions (#10361)
* feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions `iceberg_hash` partition rules could not be evaluated from SQL, so users had no way to measure a hash distribution or check which bucket a value falls into. Expose the same UDF that partition rules plan, plus `iceberg_hash_bounds(n)` which prints the equal-width breakpoints `bi = i32::MIN + floor(i * 2^32 / n)` as text for copy-paste into `PARTITION ON COLUMNS`. Both functions are immutable and read-only, and no persisted format changes. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): handle untyped NULL and small integer counts `iceberg_hash(NULL)` arrived as ScalarValue::Null, which becomes a NullArray that the shared array form had no arm for, so it failed with an unsupported-type error instead of hashing to NULL. Add a DataType::Null arm in the shared implementation, so both the SQL function and any future rule evaluation keep the advertised NULL-to-NULL behavior. `iceberg_hash_bounds` rejected Int8/Int16/UInt8/UInt16 constants even when the value was a valid bucket count, because Signature::any leaves the argument width unchanged. Accept every integer width like the existing Int32/Int64/UInt32/UInt64 arms, and cover them plus the untyped NULL case in unit and SQL tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(function): add iceberg_hash_breakpoints SQL function Equal-width `iceberg_hash_bounds(n)` buckets cannot follow a key distribution: a handful of hot keys can still land in one bucket, and a uniform split keeps hashing them together. Add `iceberg_hash_breakpoints(v1, v2, ...)`, which takes any number of constant values in any order (one shared hashable type, exactly the values a partition rule compares the hashed column against), hashes them through the shared implementation, and prints them sorted and deduplicated: SELECT iceberg_hash_breakpoints('iceberg', 34); -- 1210000089, 2017239379 The output drops straight into the breakpoint list of a `PARTITION ON COLUMNS` rule, so buckets can be placed around known key values instead of equal widths. Columns and NULLs are rejected with explicit errors: a column cannot be hashed at planning time, and a NULL hash never matches a comparison bound, so a NULL value cannot be a breakpoint (NULL rows always fall in the lowest bucket). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(function): add iceberg_hash_ranges_sql SQL function The helpers so far print bare breakpoint numbers, so a user still had to expand them into comparison rules by hand: n buckets mean n rules with up to 2n - 1 `iceberg_hash` call sites. `iceberg_hash_ranges_sql(column, n)` renders the whole list from the same equal-width breakpoints: SELECT iceberg_hash_ranges_sql('host', 2); -- iceberg_hash(host) < 0, -- iceberg_hash(host) >= 0 The output drops straight into the expression list of `PARTITION ON COLUMNS` (rule per line). It only renders text; it does not create or change partitions. The breakpoints come from the shared `IcebergHashBoundsFunction` logic, so the rendered rules and `iceberg_hash_bounds(n)` always agree. NULL name or bucket count propagates as NULL; columns, non-string names, empty names, out-of-range counts and wrong arity are rejected with explicit errors. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): include function identity in ScalarUdf equality The ScalarUdf wrapper compared and hashed only the signature, so any two functions sharing a signature were the same ScalarUDF: isnull, iceberg_hash and iceberg_hash_bounds all have Signature::any(1, Immutable) and compared equal. DataFusion compares expressions structurally, and its SQL planner rewrites SELECT expressions against GROUP BY keys with base_exprs.contains (rebase_expr); the collision made SELECT isnull(x) FROM t GROUP BY x, iceberg_hash(x) fail planning, because isnull(x) matched the grouped iceberg_hash(x) and was rewritten into an aggregate column that does not exist. Compare and hash the function name alongside the signature, so functions are distinguishable while two instances of the same function still compare (and hash) equal. Add regressions: same-signature functions stay distinct ScalarUDFs and stay separable in hash-based containers, and grouping by one while selecting the other plans and executes in both directions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): quote rendered column names and reject mixed hash domains `iceberg_hash_ranges_sql` interpolated the column argument verbatim, so a legal name such as `host-name` produced `iceberg_hash(host-name)`, which parses as a subtraction; names with spaces, dots, quotes or upper case could not be used either. Quote the name as a SQL identifier and double embedded quotes, so the rendered rules stay one identifier and survive parsing. Cover the round trip by feeding the output back into `CREATE TABLE ... PARTITION ON COLUMNS` for plain, dashed, upper-case, quoted and spaced names. `iceberg_hash_breakpoints` accepted values of different types, but strings and integers hash through different encodings (UTF-8 bytes versus Iceberg longs), so mixing them yields breakpoints that describe no real column. Require one hash domain (all strings or all integers, integer widths may mix) and name the offending argument otherwise. The existing SQL case that mixed `('iceberg', 34)` now uses two strings. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(function): register the raw hash UDF and tidy the helpers Registering `iceberg_hash` through a `Function` wrapper built a different `ScalarUDFImpl` type from the one partition rules plan, so DataFusion compared the SQL expression and the rule expression as unequal; the wrapper also rebuilt a `ScalarUDF` per batch. `ScalarFunctionFactory` already has `From<ScalarUDF>`, so register `iceberg_hash_udf()` directly and add a test that the registered function and `IcebergHash::to_logical_expr` produce equal expressions. Also: collapse the arity check and slice destructuring into one `let ... else`; make the bucket-count helpers associated functions instead of constructing a `default()` instance; pass the caller name through so an error raised from `iceberg_hash_ranges_sql` names that function rather than `iceberg_hash_bounds`; and fetch the SQL test's UDFs from `FUNCTION_REGISTRY` instead of rebuilding and wrapping each one. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): accept every string column name and cover the rules end to end `iceberg_hash_ranges_sql` matched only `Utf8` and `LargeUtf8` for its column-name argument, so a valid constant string that arrives as `Utf8View` (for example `arrow_cast('host', 'Utf8View')`) was rejected as non-string. `Signature::any` leaves argument types unchanged, so the function has to accept all string representations itself; use `ScalarValue::try_as_str` and keep the NULL and empty-name checks. A unit case and a SQL-level case pin that `Utf8View` renders the same rules as a plain literal. Add a sqlness case that renders the four buckets for the quoted column `"host-name"`, creates the table from that exact text, inserts one value per bucket plus NULL, and checks the row counts implied by the rendered predicates against the rows each region stored. This closes the gap between rendering the rules and routing rows with them. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(partition): hash constants through IcebergHash::evaluate_array `iceberg_hash_values` duplicated `IcebergHash::evaluate_array` line for line. Call the existing method from `into_hashed_scalar` and drop the extra public entry point, so the SQL helpers still hash through the code a partition rule evaluates without a second identical API. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
ca1b8c7c98 |
fix(config)!: redact credentials from diagnostic TOML (#10381)
* fix(config): redact credentials from diagnostic TOML Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(config): redact credentials at their owning fields Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(config): cover skipped SAS tokens and export logs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(cli): mask SAS tokens before overlapping account names Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(cli): redact SAS tokens before overlapping account fields Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(cli): mask containing credentials before shorter values Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
7b455f2b81 |
docs: record authorization invariants in servers and frontend guides (#10384)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
e7f75a73dd |
fix(frontend): classify physical entity graph sources (#10383)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
56cb03f4fb |
fix(function): handle unaligned binary vector literals (#10380)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
7d0186457b |
fix(mito2): notify flush waiters after failure cleanup (#10398)
* fix(mito2): notify flush waiters after failure cleanup Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): simplify flush failure replies and test notification ordering Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
4bda315fde |
fix(frontend): check table permissions for COPY FROM STDIN (#10374)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
5d88de5290 |
fix(servers): authorize HTTP database overrides (#10378)
* fix(servers): authorize HTTP database overrides Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(servers): update the HTTP output benchmark handler call Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(servers): authorize Prometheus database overrides Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(servers): retain authorized label lookup scope Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
2e1ce9808a | feat: update dashboard to v0.14.6 (#10365) | ||
|
|
9f6c957e2b |
fix(mito2): commit staging edits once when staging ends (#10322)
* fix(mito2): commit staging edits once when staging ends Staging edits (the new partition expression, the imported files) were saved as deltas in the staging directory, each taking a manifest version. The end of staging listed that directory and merged every delta it found, including deltas a stale writer wrote late. Staged actions now stay in memory and take no version. The end of staging commits them once, as the next version, in a new EndingStaging state that stalls writes and holds role changes back until the commit returns. If the commit may have been written and the region stays staging, the encoded commit is kept: a retried request stages nothing new and commits the same content, and other staged edits are refused with a retryable error meanwhile. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): settle an uncertain end of staging before later commits Under overwrite publication a save that may have landed must not be overwritten, so every commit takes its version before the save and a failed save leaves that version unused. For the end of staging, the version is now taken only once the result is known: a failed save is read back at once, and if the read-back cannot tell, the commit stays unsettled. Every later commit first reads it back again, applies it to the manifest and the region version if it landed, and is refused with a retryable error while it still cannot tell. A follower sync past that version forgets it. Committed staging actions are applied to the region version through the manifest context, so the end of staging and a later commit that settles it share one path. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): settle before checking a commit and keep the lock into the end of staging - Settle an unknown end of staging when taking the manifest lock, before the write is checked against the manifest, and fire the hook of a landed one with the lock released, so a failing commit cannot drop it. Entering staging settles too; a staging region settles only when staging ends. - Pass the held manifest lock from the graceful Leader path into the end of staging, so a queued close cannot abort staging in between. - Wait for the worker to drain the stalled writes before clearing the staging directory. - Let an empty import retry a frozen end commit. - Count the size of an end delta found by reading it back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep settling out of staging and before a new staging - Do not settle while the region is staging or a staged edit is running; only the end of staging settles there, as it also leaves staging. - Settle before entering staging through a graceful role change, and fire the hook of a landed result even if entering staging then fails. - List the end-of-staging outcomes in the order they are decided. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): settle before the metadata backfill and fence the late staging clear - Settle an unknown end of staging before a graceful Leader on a writable region checks whether to backfill metadata; a landed result is merged into the backfill notification. - Skip the staging directory clear after the end of staging if the region was closed meanwhile, as it may be open and staging again. - Clear the staging directory after an end of staging with nothing to commit. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): take the overwrite version before the end of staging is saved Under overwrite publication the end of staging takes its version before the save again, as before this change, so a later commit never overwrites a save that may have landed. After an error that may have written it, the region goes back to staging with its staged actions and encoded bytes kept, and the retry writes the same bytes at the next version; a reopen replays both as one commit. This removes the immediate read-back of a failed end commit, the unsettled mark and the settling before later commits, the ManifestUnsettled error, and the region version handle on the manifest context. The end of staging applies the committed actions to the region version itself again. A graceful Leader from staging takes the manifest lock again after the end of staging and runs the metadata backfill, firing one notification for both, as before this change. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): check the lease and fence conflicts in the end of staging commit The end of staging commits its staged actions on its own path, so it skipped the checks other manifest commits make: in conditional publication it saved after the Leader deadline had passed, and a version conflict left the region unfenced, so the worker never reloaded it. Run both commits through one core that makes those checks. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): let the worker pass the drain to the end of staging The end of staging built and sent the worker's StagingEnded notification itself, so the region module depended on the worker's dispatch types. The worker now passes the drain as a future that the end of staging awaits after it drops the manifest lock. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep the write order across the end of staging The end of staging stores the region's next state under the manifest lock, and the worker handles the writes it stalled meanwhile later, on StagingEnded. A write that the worker took in between was applied before the earlier stalled ones. The worker now stalls a region's later writes behind those until StagingEnded drains them. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(mito2): read and encode the staged actions without copying them The end of staging copied the staged actions to read them, to encode them and to commit them. Read and encode them in place, and copy them for the region hook only when one is registered. The commit keeps its own copy, so the staged actions survive a failed commit for the retry. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover a flush while staging ends and the staging directory clear Check that a migration flush gets a retryable error while the region ends staging and succeeds after it. Plant a delta from an older binary in the staging directory before applying the staging manifest, so the check that the directory is cleared looks at the target region and fails if the end of staging leaves it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): tidy the end of staging wording and a test helper Nothing is merged at the end of staging any more, so its log, error reasons and bindings say staged or committed. Fix the region hook doc sentence and table cells for actions staged in memory, and move the import helper of the conditional manifest tests next to the other helpers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drop the staged files of a region that left staging An end of staging that leaves for Follower or Downgrading kept the staged actions in the manager, and the SST listing and all_manifest_files merged them into the files of a region that never committed them. The end of staging now drops them when it leaves staging, and both readers merge the staging manifest only while the region is staging. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(mito2): check a resent import against the staged files in linear time The check that a resent import only adds files that are already staged walked every staged edit for every file, quadratic in the file count. It now collects the staged file ids into a set once. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(mito2): apply the committed staged actions without another copy The end of staging cloned the staged actions to split them for the region version before the commit, on top of the copy the commit applies. The commit now hands the staged list back once it succeeds, and the end of staging checks the staged Change by reference before the commit and splits the returned list after it. One copy stays, because a successful commit uses the actions twice: a copy builds the new manifest, and the returned list updates the region version. The region hook gets its copy only when one is registered, as before. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): return the normal version from all_manifest_files and drop a dead fallback all_manifest_files preferred the staging manifest's version. It now returns the normal version, as its doc says. A test helper's unreachable `unwrap_or(path)` after `rsplit` becomes `unwrap_or_default()`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): name the staging helpers after what they do - Rename is_staging_or_staged_edit to in_staging_session. The Staging and EndingStaging pair lives in RegionRoleState::in_staging_session, which the compaction scheduler, region_role, should_abort_index and the region-level helper use. - Rename finish_ending_staging to leave_ending_staging, since it can also land on Follower, Downgrading or Staging, and tie end_staging, EndingStaging and StagingEnded together in the EndingStaging doc. - Rename contains_import to only_adds_staged_files and the staged list field to actions. - Move the staging paragraph to the public update(), state on end_staging that the drain is never awaited under the manifest lock, say when all_manifest_files merges staged files, say that the listings do not rely on the clear at the end of staging, and count the end of staging in the flush rule of the mito2 AGENTS.md, naming the predicates the flush paths test. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): keep the end-of-staging drain next to the stalled requests Move drain_ended_staging into worker.rs, and the set of regions whose writes wait for it into StalledRequests, which adds, checks and removes them, so the stall bookkeeping and its release live together. Rejecting all stalled writes now also forgets these regions. Their stalled writes are rejected with them, so a later write has nothing to keep order behind and is handled under the region's current state. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): keep the staged view with the staged actions The manifest with the staged actions applied was a separate field of the manager, set and cleared next to the actions by hand. It is now a field of StagedActions, so clearing the staged actions clears it as well, and its accessor says what it returns. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): split EndStagingPending out of RegionBusy A staged append that arrives while a possibly written end commit waits for its retry was refused with RegionBusy, which therefore had to be in the not-written arm of may_have_persisted_manifest_update for every producer. It now gets EndStagingPending (status RegionBusy, retryable), which alone takes that place, and RegionBusy keeps its earlier classification. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover regions that are ending staging Check that a region ending staging reports the StagingLeader role and skips compaction scheduling, and that its bulk writes stall, as do later ones behind them after staging ended, until the worker drains them. Share the stall wait and the StagingEnded drain between the tests. The tests wait for these stalls through a listener on their own engine, so the end-of-staging stalls notify the test listener as the write buffer stalls do, and CountingStallListener moves to the shared test listeners. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
6420082685 |
fix(meta): skip metric engine identity columns when merging partition columns (#10368)
CreateLogicalTables copies every physical partition column into the logical schema as a nullable String tag. After a physical metric table is repartitioned by the internal `__table_id` column, that column reached the region create request and the metric engine rejected it with "Internal column __table_id is reserved", so no logical table could be created. Filter engine-owned identity columns while merging, reusing `is_metric_engine_internal_column`. Filtering at the merge point also protects procedures that persisted an unfiltered `physical_partition_columns` list. User partition columns keep the previous behavior. Write routing for a physical table partitioned by `__table_id` is still tracked by #10352; this change only unblocks logical table creation. Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
c937e3997b |
perf(mito2): prefilter single-column dynamic filters (#10340)
* perf(mito2): prefilter dynamic join membership Read the narrow dynamic-membership column before the full projection and use a stable per-row-group mask to select rows. The live filter above the scan stays authoritative; bounds-only and unsupported shapes keep all rows. Signed-off-by: discord9 <discord9@163.com> * test: cover dynamic prefilter cache and generations Prove the prefilter-result cache keeps the static mask but never reuses a dynamic mask across producer generations. Document why a superseded dynamic mask is discarded rather than assumed monotone. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
b5cdcba82a |
fix(datanode): keep flush task error status code and retry hint (#10360)
The synchronous FlushRegions handler wrapped the error returned by the flush task into Unexpected before building the reply. InstructionError then only copied the Unexpected status code and the default NonRetryable hint, so metasrv could not tell a transient flush failure from a non-retryable one. Return the task error unchanged so its status code and retry hint reach FlushRegionReply, and rebuild the reply message with the same debug chain the previous wrapper produced to keep the diagnostics unchanged. Add handler-level tests covering RegionBusy, RegionNotFound and Unexpected task errors under both FailFast and TryAll. Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
6a202adbe7 |
fix(log-store): skip objects collected below the boundary in WAL reads (#10343)
* fix(log-store): skip objects collected below the boundary in WAL reads The read-only WAL reader lists the source prefix and then reads the header and footer of every listed object. While the store writing the prefix is still collecting objects, a listed object can be deleted before its footer is read. The NotFound failed the whole read, even when the object lay below the reclaim boundary and was off the chain the read replays, so reads of a large prefix under steady writes failed often. Apply the rule startup recovery already uses. An object gone below the reclaim boundary at which the chain of the read stops is skipped. Any other vanished object, on the chain or off it, still fails the read: it may be the object that completes a later chain or carries the largest epoch. Startup recovery and the reader now share one NotFound split and one boundary check. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover unreadable objects in listed WAL reads A NotFound below the boundary is skipped, but any other error reading a header or footer must still fail the read. Cover it with a corrupted header below the boundary in both modes. Describe the boundary in `ensure_vanished_below` and `read` in terms that also fit the `LastEntry` mode, where the chain can stop at the object of `after` instead of a reclaim boundary. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe where WAL reads stop following links In the `LastEntry` mode a later object can carry a reclaim boundary above the object of the reported entry. The read then keeps that object as its tip below the boundary and follows no link from it. Say that the boundary stops the links to predecessors, and that the tip is the one object below it a read can need. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): pin the outcomes of vanished-object reader tests The unreadable-object test now asserts the InvalidWalObject error for object 1 inside the index cache's wrapper, instead of any error other than NotFound. The vanished chain-object test states the outcome it guards against: without object 2 a listed read could select the chain ending at object 1 and succeed with no entries. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
62dad907d0 |
fix: redact credential table options in user-visible output (#10346)
information_schema.tables.create_options (and SHOW TABLE STATUS) printed object store credentials of external tables verbatim, and SHOW CREATE TABLE masked only access_key_id and secret_access_key. Mask all four credential keys in both paths, and sort extra options in create_options by key. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
f8d8fe1071 |
fix: honor the pipeline version header and URL parameter in Elasticsearch _bulk (#9452)
* fix: honor the pipeline version header and URL parameter in Elasticsearch _bulk The Elasticsearch _bulk handlers read only pipeline_name and msg_field and hardcoded the pipeline version to None, so the x-greptime-pipeline-version header and the version URL parameter were silently ignored and the latest version was always used. Resolve the version from the version query parameter, falling back to the x-greptime-pipeline-version header, and pass it to PipelineDefinition::from_name; an unparsable version now fails the request instead of being ignored. Without either source the latest version is used, as before. Signed-off-by: fengyue-xve <1759519137@qq.com> * fix(elasticsearch): reject invalid pipeline version headers and use HTTP item status Signed-off-by: fengyue-xve <1759519137@qq.com> --------- Signed-off-by: fengyue-xve <1759519137@qq.com> |
||
|
|
6dbff8a081 |
fix(log-store): close raft-engine engines on shutdown (#9518)
* fix(log-store): close raft-engine engines on shutdown raft-engine preallocates the active log file and only trims that tail when the Engine is dropped. GreptimeDB never dropped its engines before exiting, so every start after a clean shutdown truncated the tail as a broken batch and logged "Truncating log file due to broken batch". RaftEngineBackend::close and RaftEngineLogStore::stop now release the engine; standalone closes the metadata backend last, and the datanode stops the raft-engine WAL after the region server. Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * refactor: address review on raft-engine shutdown - Add ClosableKvBackend to common-meta; the standalone metadata backend creator returns one and Instance closes it explicitly instead of downcasting to RaftEngineBackend, so wrappers forward the close. - Record a failed metadata backend close as a shutdown error. - Sync and drop the engine on a blocking thread. - Only check tail trimming when raft-engine preallocated, and check reopening after close/stop on every platform. Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix: release raft-engine engines on every shutdown path - Stop the raft-engine WAL when the datanode build fails after opening it. - Purge and sync tasks hold the engine weakly, so dropping the last backend or log store reference (as the CLI tools do) closes the engine. - Fix needless borrows reported by clippy. Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix: drop unused ClosableKvBackend import Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix: cancel log store readers on stop; address review - RaftEngineLogStore tracks the tasks feeding read streams and cancels and joins them on stop, so a retained, unconsumed stream no longer keeps the engine open past shutdown. - Tests: stop with a retained read stream; datanode shutdown with a failing region engine and after a normal shutdown reopens the raft-engine WAL; standalone stop with failing steps reopens the metadata directory. - entries_to_batch no longer takes self; rename opened to get_open_engine, log_files_len to log_files_size and PREALLOCATED_LEN to PREALLOCATION_THRESHOLD; mod declarations before imports; rename the standalone shutdown closure to record_shutdown_error. Model: claude-opus-5-5 Tokens: 15.8k input, 148k output, 2.9M cache write, 103M cache read Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix: update Cargo.lock for tokio-util's rt feature The rt feature makes tokio-util depend on futures-util, which the lock file has to record for `cargo check --locked`. Model: claude-opus-5-5 Tokens: 15.9k input, 162k output, 3.0M cache write, 124M cache read Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * refactor: close the raft-engine WAL by ownership instead of a lock Per review, RaftEngineLogStore holds the engine as a plain Arc<Engine> again: - MitoEngine drops its Wal raw entry reader once its workers are stopped, so it no longer holds the log store after stop. - RaftEngineLogStore::close(self) stops the tasks and readers and closes the engine. Datanode::shutdown takes the log store after stopping the region server and closes it if it holds the last reference; otherwise it logs a warning and only stops it. - A failed build stops the region server it created before closing the log store, so the engines release it there too. Model: claude-opus-5-5 Effort: medium Tokens: 78, 19161, 107293, 4916570 del commit Cost: 2.23 USD del commit Session: 82696960-1984-4432-bcf1-33bfece8b526 Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * test: drop the index argument from LogStore::read in the raft-engine tests #9563 removed it from the trait. Model: claude-opus-5-5 Effort: medium Tokens: 16, 3715, 6600, 1706794 del commit Cost: 0.47 USD del commit Session: 82696960-1984-4432-bcf1-33bfece8b526 Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix: end a cancelled WAL read with an error; test a written region - A raft-engine read cancelled by stop now ends its stream with an IllegalState error instead of an end of stream, so a replay cannot take a truncated WAL as fully read. The reader flags the cancel instead of sending the error, since it is usually blocked on a full channel at that point. - Datanode test: a raft-engine datanode with a created region and writes shuts down cleanly and its WAL reopens, which fails if anything still holds the log store. - Move the doc comment of is_stopped back onto it. Model: claude-opus-5-5 Effort: medium Tokens: 42, 10316, 407027, 4108529 del commit Cost: 4.28 USD del commit Session: 82696960-1984-4432-bcf1-33bfece8b526 Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix(log-store): drop the no-op sync before closing the raft-engine engine Per review: on raft-engine 0.4.2 `Engine::sync` writes an empty batch, and `Engine::write` returns before the fsync, so the sync in `close_engine` read as a final flush that never ran. The active log file is trimmed and synced by `LogFileWriter::close` when the last reference is dropped, so `close_engine` now only drops it on a blocking thread. Model: claude-opus-5-5 Effort: xhigh Tokens: 120, 33613, 182918, 7511982 del commit Cost: 3.64 USD del commit Session: 57ca77a8-f6fe-4fa2-a55a-dda31e5c541f Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> --------- Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> |
||
|
|
15fd97e369 |
fix(meta-srv): stop failing over regions that are still opening (#10290)
* chore: update proto for opening region heartbeats Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: reset failure detectors during background region initialization Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve detector history for opening regions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update opening region proto revision Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update opening region proto to merged main Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
90d0c00a82 |
fix(promql): resolve the OR right operand's time-index column from its schema (#10335)
`or` inferred the qualifier of an operand plan from its first field (`schema.qualified_field(0)`). A binary operation with an explicit matching modifier derives its result labels from the left operand when the match reduces the label set, so its projected plan carries two qualifiers: the labels are `lhs.<label>` while the time index keeps the context table's qualifier (`rhs.<greptime_timestamp>`). Prefixing the time index with the first field's qualifier then referenced a column that does not exist, and planning failed with Internal error during building DataFusion plan: No field named lhs.greptime_timestamp. Did you mean 'rhs.greptime_timestamp'? for e.g. `sum by (l) (f(a) / ignoring(x) group_left() f(b))` used as the right operand of `or on (l)`. `f(a) / ignoring(x) f(b)` triggers it as well, so this is not specific to a group modifier. Without a matching modifier every projected column takes the right operand's qualifier, which is why plain `a / b` never hit this. Resolve the time index against the actual operand schema: keep the preferred qualifier when it really holds the column (uniform schemas, so no plan changes), otherwise take the qualifier of the column found by name and let DataFusion report ambiguity instead of guessing. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
d1e5a2df8f |
fix(mito2): check the lease before manifest commits and reload regions after conflicts (#10311)
* fix(mito2): check the lease before manifest commits and reload regions after conflicts In conditional manifest mode, refuse a commit whose Leader deadline has passed before its PUT, and fence a region whose commit hits a manifest version conflict, then reload it in the background and swap the new instance in. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): release the flush status when the WAL release after a flush fails handle_flush_finished failed the flush's waiters and returned when the WAL release after a committed flush failed, but left the flush registered in the flush scheduler, so every later flush of the region, and the reload of a fenced region, waited behind it forever. Handle the failure as a failed flush, which removes the registration and answers the pending requests. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover staging saves after the deadline and a fence during catchup Add a listener hook before a catchup reopens the region, and test that a fence landing there makes the catchup answer RegionBusy without replacing the instance. Also test that entering staging after the Leader deadline saves no staging manifest. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): note that set_region_role does not check the reload mark Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
c8e0d29df2 |
feat(mito2): publish table compaction results conditionally (#10310)
* feat(mito2): publish table compaction results conditionally Publish one region from a live input snapshot using an ordinary file-only edit. Permit repeated submissions and partial success without manifest task state, while preserving sequence, GC protection and uncertain-write recovery. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): write table compaction outputs to level 0 Route split outputs into the ordinary L0 pipeline so window merges and semantic folding stay with region compaction. Update publication validation and tests accordingly. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): release verified footers before manifest publication Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): compare typed table compaction options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover table compaction output validation failures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): clarify series count verification limits Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): reject index declarations without output index objects Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document table compaction outputs as L0 Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): map the table compaction publication module Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
ddc16bafe8 |
fix(datatypes): render jsonb stored as JSON text without panics or wrong values (#10197)
* fix(datatypes): decode jsonb before rendering it as text jsonb_to_string serialized the bytes through RawJsonb's Serialize impl, which panics with an out-of-range slice index on JSON text such as "hello world" or " null" (stored by COPY FROM and MySQL text parameters) and on truncated jsonb. Its text fallback was also stricter than the extended parser reads used before. Decode with jsonb::from_slice, as reads did before #10112, then write the value out with serde_json escaping for keys and strings. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(datatypes): keep streaming jsonb serialization in jsonb_to_string Decoding into a jsonb::Value tree allocates per array, object and key on every JSON cell returned over PostgreSQL and MySQL. Go back to serializing the raw bytes with RawJsonb's Serialize impl, and bump jsonb to a rev whose Serialize returns an error instead of panicking on out-of-range payloads (GreptimeTeam/jsonb#2). Bytes that fail to serialize are parsed as JSON text with the extended parser, as jsonb::from_slice does. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(datatypes): pin text values that used to read back as false Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: bump jsonb to reject scalar headers with non-zero length bits JSON text over ~2 MB could still pass the payload bounds check and read back as a wrong value (GreptimeTeam/jsonb#2). Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(datatypes): report the text parse error when JSON bytes are neither jsonb nor text Also pin jsonb_to_serde_json on JSON text scalars that used to decode as false. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: pin jsonb to the merged commit on main GreptimeTeam/jsonb#2 reached main through GreptimeTeam/jsonb#3. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
97c8136c3c |
chore(deps): bump datafusion to ae04160b (#10320)
* chore(deps): bump datafusion to ae04160b Move all 16 `[patch.crates-io]` DataFusion pins from `cc5534492` to `ae04160b`, the merge commit of GreptimeTeam/datafusion#42. That fork commit backports six upstream apache/datafusion fixes: - #26452 timestamp-to-time casts no longer drive statistics pruning - #24675 memory-limited NLJ fallback refuses left-emission joins with a multi-partition probe side instead of emitting wrong unmatched rows - #24891 RepartitionExec spill-pool deadlock with two open files - #25076 null-aware anti join no longer emits rows after a sibling probe partition saw NULL - #25339 correlated NOT IN with a non-equality correlation - #26578 null-aware anti join drops build rows with a dictionary NULL key The first three are faithful backports. The last three are re-implementations: upstream restructured `hash_join/stream.rs` before writing them, so the patches do not apply to this fork's older copy of that file. `#25339` covers the single-equi-key plus residual-filter shape; multi-key correlated `NOT IN` still fails at planning as before. `Cargo.lock` is regenerated (35 `datafusion*` packages). No runtime acceptance has been run against this pin; see the PR description for the three checks worth doing before shipping. Signed-off-by: discord9 <discord9@163.com> * fix(common-query): pass the pruning bitmap to HashTableLookupExpr::new #39 added a fifth parameter to `HashTableLookupExpr::new`; the dynamic-filter payload round-trip test still constructed it with four arguments, which breaks `--all-targets` builds even though the library itself compiles. Signed-off-by: discord9 <discord9@163.com> * chore(deps): bump datafusion to 70f84c5b Move the pins from `ae04160b` to `70f84c5b`, the head of `greptimedb-55.1-thin-fork` after GreptimeTeam/datafusion#37 was merged on top of #39/#41/#42. `ae04160b` did not contain #37, so the reverse-cast constraint propagation guard was lost and `tests/cases/distributed/optimizer/partition_pruning_cast` failed: the filter `CAST(k AS BIGINT) = 1` made the planner treat the float column `k` as a singleton and drop the requested `ORDER BY k DESC`. The new pin carries both the reverse-cast guard and the six release backports, so the regression is covered again. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
d061e08717 |
feat: flush regions under object store WAL retention pressure (#10309)
* feat(store-api): expose advisory WAL flush candidates Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(log-store): select regions blocking WAL object reclamation Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): schedule region flushes under WAL retention pressure Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito): cover WAL pressure reclamation and recovery Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): prioritize oldest WAL reclamation blockers Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito): deterministically exhaust range cache admission budget Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
54b6bd5b44 |
feat(mito2): support extension index providers and input reuse (#10297)
* feat(mito2): register extension index providers Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito2): support index input reuse and extension budgets Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
77cb39b5d1 |
feat(mito2): record pending WAL tails in region manifests (#10308)
* feat(mito2): record pending WAL tails in the region manifest Adds `pending_tails` and `handoff_floors` to the region manifest and its checkpoint, and an optional `pending_tails` / `handoff_floor` pair to `RegionEdit`. An edit without them leaves the records unchanged. A floor is dropped once the flushed entry id reaches it; a tail stays until a commit removes it. Neither is written when empty, so manifests without them encode as before. Opening a region replays its WAL after this node's tails and floor, and WAL releases at open and after a flush stay below this node's tails. Nothing writes these records yet, so every region behaves as before. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(mito2): hold back reads and region changes while WAL tails are pending A region with pending WAL tails is missing data that only those tails hold. Until they are recovered, scans and memtable and SST stats fail with the retryable `RegionPendingTails` error. Truncate, entering staging, a repartition flush, a close-time flush on request, and ALTERs other than adding nullable columns without a default or changing options are rejected before they change any region state. Compaction pauses. A full-listing GC pass skips the region while it has pending tails, or while a newer manifest may exist or the manifest listing fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): name the WAL releases that keep pending tails Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): reject no-op ALTERs of restricted kinds while WAL tails are pending Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that reopening restores the pending-tail read gate Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): pin what the pending-tail fixtures and rejections promise Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): name the replay start in the open log and share the pending-tail check Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): state the per-node order of pending tails and who keeps later writes out of them Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that a cancelled compaction starts nothing while WAL tails are pending When a compaction cancelled by task id finishes, the queued manual request goes through handle_pending_compaction_request, whose pending-tails check replies Ok(0). The reply is read with try_recv so that a dispatched request fails the test at once instead of hanging it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drop the pending compaction task state when WAL tails skip it Skipping a pending manual compaction while WAL tails are pending replied Ok(0) but left its task state in place, so task inspection kept listing it as queued and cancelling it by id still succeeded. Clear the pending task state and publish before replying. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
e2e980fe39 |
feat(mito2): add column sketch options and index metadata (#10295)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
ee1fc7782f |
feat(prometheus): query persisted metric exemplars (#9924)
* feat(prometheus): add exemplar queries and lifecycle coverage Expose /v1/prometheus/api/v1/query_exemplars with exact time bounds and series selectors. Resolve and authorize source selectors before scanning exemplar storage, enforce query limits and cancellation, preserve exact observation identity, and return warnings for malformed stored rows. Includes HTTP and authorization tests, pre-created-table and concurrent-DDL coverage, OTLP Arrow ingestion checks, and cross-protocol lifecycle tests across memory, WAL, SST, compaction and reopen. The query-dependent ingestion tests live here so every lower layer builds independently. Add metric-exemplar usage and retention guidance. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): preserve query labels with source identities Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): floor bounds and add audit hooks Signed-off-by: shuiyisong <xixing.sys@gmail.com> * • fix(exemplar): include aggregation parameter selectors Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): avoid v1 exemplar quota stalls Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: docs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: align exemplar query timestamps with Prometheus Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove unnecessory doc Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
09c0291575 |
fix: restrict JSON retries to invalid UTF-8 errors (#10302)
* fix: restrict JSON retries to invalid UTF-8 errors Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: isolate UTF-8 retry counter assertions Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
65dc999180 |
feat: support cancelling compaction tasks by task ID (#10229)
* feat: add ADMIN compaction task cancellation Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: reject metadata region compaction cancellation before routing Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin greptime-proto to merged cancellation commit Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
77a7d998bb |
perf(puffin): reuse reader metadata and scoped cache keys (#10294)
perf(puffin): reuse reader metadata and blob lookup Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
d2be2980c2 |
refactor(mito2): unify builtin index planning and execution (#10293)
* refactor(mito2): separate index target resolution from creation
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit
|
||
|
|
f5c1020f5f |
feat(mito2): add shared SST storage and reclamation foundations (#10256)
* feat(mito2): add shared SST storage and reclamation foundations Add shared-level isolation, receipt replay and recovery, migration support, and fail-closed file protection. Keep coordinator submission and verified publication in the next stack layer. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): isolate table compaction manifest semantics Keep atomic manifest extension envelopes and lifecycle safety in the common engine. Gate table-specific receipts and remap admission with enterprise, preserving their wire format and rejecting unsupported state before replay or GC. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove manifest-backed compaction task state Keep file-only edits, shared-reference admission and GC safety. Task retries no longer require persisted receipts or task epochs; reject unpublished prototype task-state records rather than silently dropping them. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): guard staging manifests and reclaim shared range indexes Reject prototype task state when decoding staging blobs, matching checkpoint and delta decoding. Delete the region-scoped range index when a shared SST reference is retired, while leaving the shared object to global GC. Document the level-2 merge-split rationale and bind the delta guard's action tag with a test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): simplify GC deletion and lingering rules Require unknown files in a directory without a manifest to pass the lingering window before deletion, so an unpublished table-compaction output cannot be reclaimed out from under an active attempt. Replace the per-batch deletion loop with a single publication recheck taken before remote deletion, and drop the batching constant, the deferred-deletion accounting, and their test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): keep regions open after manifest validation rejection Classify InvalidRequest from pre-write applicability checks as definitely not persisted. Cover both shared-removal and unsupported-level rejections without stopping the manifest manager, while keeping ambiguous storage errors conservative. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): reject ambiguous shared SST reclassification Reject edits that both add a shared SST and remove the same file ID before admission and under the manifest lock. Keep ordinary edit semantics and verify add-only L0 reclassification survives reopen without diverging from the live Version. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): define truncate semantics for shared SST references Keep explicit truncation applicable to shared references and document the distinction from physical reclamation. Exercise full and time-range truncation with real shared SSTs, an old query, another referencing region, and reopen. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): define shared copy results and reject metadata imports Document copied_file_ids as installed logical references, including no-op retries. Reject shared SSTs in metadata-region copies before copying any objects or changing the target, and cover shared, mixed and repeated copy behavior. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(mito2): unify shared SST level references and bounds docs Use module-qualified shared-level constants consistently and document MAX_LEVEL as an exclusive bound for SST version operations. No runtime behavior changes. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): move the shared SST level to a max sentinel Use the u8 maximum as the shared classification so future ordinary levels can never collide with it, and map wire levels to fixed storage slots instead of indexing by value. Shared metadata, admission and filters keep their semantics; unallocated levels stay rejected. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): keep dropped-region tombstones while files are deferred Young unknown objects and hook-protected candidates on a dropped region must report needs_retry so the meta tombstone survives until they are actually reclaimed; reporting success would let the procedure drop the tombstone and orphan the retained objects. Cover both deferral paths plus the eventual clear. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): reject adopting an existing L1 SST as shared Adopting a shared reference is L0-only: the manifest would replace an existing L1 entry while the live version keeps both handles, duplicating rows until reopen. Reject the edit at admission before any state change and cover the rejection with a real SST. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
7cd2a3050e | fix: pgwire update 0.42.2 (#10300) | ||
|
|
16123d806b |
perf(query): push column projections through MergeScan (#10221)
* perf(query): push column projections through MergeScan Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): retain empty projections and refresh sqlness plans Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(query): update TSID join plans after projection pushdown Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
8da533c35f |
fix(mito2): keep rows written during staging in the WAL (#10253)
* fix(mito2): keep rows written during staging in the WAL A flush during staging committed its SSTs to the staging manifest, then dropped the memtable and released the WAL. If staging did not end successfully those rows were gone. A flush that committed after staging had ended lost its rows even though staging succeeded. Staging regions no longer flush. They are skipped by memory-pressure, periodic and region-limit flushes, and their writes are rejected instead of queued at twice the region limit or when the node's staging memtables reach half of the engine's flush limit. A manual or repartition flush on a staging region with data gets a retryable error; the migration pre-flush and the WAL prune flush reply success without flushing. Closing a staging region leaves staging first, so skip-WAL rows are flushed to the normal manifest. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep accounting and closing staging regions during staged edits Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): track staged edits without relying on the partition info Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): synchronize staging close tests with worker admission Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): observe the manifest lock wait when closing staging regions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): detect staged edits through the staging partition info Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): record that staging regions must not flush Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not restore staging after a close abandons the edit Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): reject region edits during a close-time flush Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style(mito2): format the staging tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): reject staging edits once staging is abandoned Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): bind apply-staging edits to their partition expression Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): note that staging regions pin their topic's prune point Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
4c4c8a58fd |
feat: update dashboard to v0.14.5 (#10270)
Co-authored-by: dennis zhuang <killme2008@gmail.com> |
||
|
|
402e4a97fc |
fix(time): avoid i32 overflow when converting an out-of-range date (#10299)
Signed-off-by: Arpit Jain <arpitjain099@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
73698114de |
feat(log-store): add a read-only object store WAL reader (#10269)
* refactor(log-store): extract shared WAL reading helpers Move object indexing, chain traversal, history checks, segment prefetching and object I/O out of the writable store. Preserve local recovery and read policies. Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(log-store): separate read-only WAL object access Introduce WalObjectRead as the read capability shared by indexing and prefetching. Keep writes and startup listing on WalObjectIo, and adapt writable-store test doubles without changing their behavior. Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(log-store): add a read-only object store WAL reader Validate source identity with NodeWalPrefix and share readers by source. Read known-last-entry or listed ranges using fresh listings and bounded footer concurrency, without holding the cache lock across I/O. Evict indexes outside the current listing, keep release extraction in startup recovery, and cover bounds, history, cache lifetime, prefix validation and concurrent reads. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): coalesce remote WAL footer loads with shared cache Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): report missing source WAL history separately Signed-off-by: WenyXu <wenymedia@gmail.com> * test(log-store): cover remote WAL chain and listing boundaries Signed-off-by: WenyXu <wenymedia@gmail.com> * test(log-store): wait for asynchronous WAL index release Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): align WAL recovery assertions with shared history messages Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
50a2c4d378 |
fix(mito2): publish manifest deltas by conditional create under a lease (#10234)
* fix(mito2): publish manifest deltas by conditional create under a lease
A late old leader and the region's new holder can commit the same manifest
version. For a region whose writes require a lease and whose WAL is the object
store WAL, the manifest now behaves as follows. Every other region keeps its
behaviour and issues no extra request.
- Each delta has one fixed name, `{version:020}.json.gz`, written by a
conditional create. Without `compress_manifest` the delta is written at gzip
level 0, so that all writers share the name.
- On an existing object, the writer reads it from storage: identical content is
a retry of a write that landed, anything else is a retryable
`ManifestVersionConflict` that keeps the output files.
- The version advances only after a successful save.
- Recovery loads the highest checkpoint found by LIST and uses the checkpoint
pointer only for its checksum, because a stale writer's checkpoint task can
leave the pointer at an older version.
- Checkpoint cleanup frees names, so a late create can still succeed on a
deleted name. A commit whose save returns after the region's Leader deadline
lists the checkpoints once: a checkpoint at or above its version, or a LIST
error, is a conflict; otherwise it applies but starts no checkpoint. A commit
within the deadline does one conditional PUT and no LIST.
`RegionManifestManager::update` becomes the one commit entry and takes
`allow_checkpoint` and `must_verify` as required parameters. A backend that does
not declare conditional create is refused at open.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): treat every LIST error as a conflict when verifying a commit
The listing helper turned a NotFound from lister creation into an empty result,
so the check after a late commit could read a failed LIST as proof that no
checkpoint covers the version. List the checkpoints directly, so that any LIST
error reaches the caller. Recovery still treats a missing directory as having no
checkpoint.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): keep the LIST failure in manifest conflicts and follow the style guide
A failed LIST during commit verification was reported as "OpenDAL operator
failed", because the conflict reason used the display of the wrapper error.
Report the root cause through `output_msg` and assert it in the test.
Align the change with docs/style-guide.md: name boolean methods `is_*`, name
the listing helper `list_highest_checkpoint` and move it after the public
methods, qualify `encode_gzip_stored` through its module, and name the mode
setter after the field it sets, which is now `pub(crate)`. Move the shared test
helpers to `crate::test_util` and `manifest/tests/utils.rs` so that the manifest
test modules stay private.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): order the manifest test helpers public before private
Move the private `NotFoundReader` and `CheckpointTaskBlocker::block_write`
below the public items next to them, and document
`fail_manifest_delta_reads_layer`.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): fail recovery when a listed checkpoint disappears
Recovery treated a checkpoint that LIST found but that cleanup removed before
the read as no checkpoint at all. It then replayed from the first version over
deltas that were already deleted, and could report an empty region or build a
manifest from the surviving tail. Return a retryable version conflict instead.
Record the size of the stored object when a retry finds an identical delta, as
the other writer may have encoded it differently. Reopen the region in the test
of a commit that crosses its deadline, and reduce the AGENTS.md entry to the
invariant.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): verify conditional commits from callers without a deadline
Restore `RegionManifestManager::update(action_list, is_staging)`. Without a
Leader deadline a caller cannot show that its commit returned in time, so a
normal commit under conditional publication is now always verified on this
path and starts no checkpoint.
Only `ManifestContext::update_locked` has a deadline, so the variant that takes
`allow_checkpoint` and `must_verify` becomes `update_with_verification` and is
crate private. `sst-replace` and the existing tests return to the two-argument
call.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): name the verifying manifest update in the hook docs
`ManifestContext::update_locked` now calls `update_with_verification`, and the
public `update` also calls it, so neither doc can call `update_locked` the sole
caller of `RegionManifestManager::update`.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): keep the conditional publication rule in region options
Move the rule that decides which regions publish manifests conditionally
to RegionOptions, so the manifest options only hold the result. Report a
checkpoint that vanished after LIST with its own error instead of a
version conflict, and keep the stored gzip encoder with the other
CompressionType helpers.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
|
||
|
|
297a343fd4 |
fix(query): use current partition columns in prepared plans (#10244)
A prepared statement keeps its logical plan, and that plan holds the table as it was at prepare time. The analyzer reads the partition columns from that table while the regions to scan come from the current route. After a table's partition columns change, the analyzer can consider a query safe to push down, so each region runs the whole query and the frontend concatenates their results. The same happens to the source of a prepared INSERT ... SELECT, which then stores wrong values. Before executing a kept plan, compare each scanned table's partition columns with the current table found by id. Scans of a table whose columns changed get the current table when the schema is unchanged. Otherwise they keep their schema, because a scan's projection holds indices into it, and report the current columns as absent from it, so the query keeps merging on the frontend. Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
361f3595db |
fix(promql): bound scatter time filter expression depth (#10272)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
0cb70d4f84 |
fix: tolerate invalid UTF-8 in JSON log ingestion (#10245)
* fix: tolerate invalid UTF-8 in JSON ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: tolerate invalid UTF-8 in NDJSON ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: upgrade simd json Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: add server default for ignore_errors Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
c844d44721 |
perf(mito2): reuse static manifest checks across row groups (#10220)
* perf(mito2): reuse static manifest checks for cached readers Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(mito2): drop redundant pruner test and release entry lock early Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(mito2): cache static manifest keep verdicts directly Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |