mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-21 10:14:06 +00:00
3a032fbbc6f1f100b79b9b736908b07b8a251efb
74
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ee9ad87fa1 |
Merge upstream/main into narthex (pre-merge rehearsal, not for push)
Rehearsal merge of main ( |
||
|
|
115d85b669 |
fix(mito2): identify series by their exact sparse key, not by tsid (#10330)
* fix(mito2): identify series by their exact sparse key, not by tsid `(table_id, tsid)` is a hash, so distinct series can share it: 2000 label values already produce 20 colliding pairs and a 36k-series SST keeps only 27856 distinct tsids. The columnar series writer silently merged such pairs (first label set kept, row counts added) and the samples writer either reported time_regressed or, with increasing timestamps, attached one series' samples to another's labels. Every one of those decisions used the hash as identity. The SST already stores the exact sparse primary key, so nothing on the storage side changes: `tsid`, the hash function and the key layout are untouched. What changes is that the artifacts, their readers and the prepared read path now decide identity on the canonical sparse key bytes, and `(table_id, tsid)` is only the coarse, order-preserving prefix used for ordering, pruning and bucketing. - `ExactSeriesKey` is the new identity type: equality, ordering and hashing are defined on the key bytes alone, `id()` is documented as a derived prefix that is never compared. `from_sparse_key` validates the entire key, not just the prefix, and bounds it at 8 KiB. - Series format v2 keeps one row per exact key; the descriptor layout and the existing per-row tags already express this once the writer aggregates by key. Coarse page/block/part bounds may now touch, so readers prune conservatively. - Samples format v2 records the owning key per block, appends a `__primary_key` column, and lets a key change restart timestamps and ordinals while the same key keeps the monotonic rules. Exact selection travels in `SamplesReadRequest.keys` next to the coarse `ids`. - The prepared path keys candidates by `ExactSeriesKey`, re-encodes identity from the artifact's own tags, associates samples through `__primary_key`, and uses the exact key for the merge key and the direct-label path. - Both manifests carry a mandatory `identity_encoding` discriminator. Artifacts without it are structurally valid and perfectly ordered yet may contain an already merged row, so they are rejected and rebuilt from SSTs; maintenance rotates them out instead of reusing them, and the local cache version drops the rebuildable store. There is no dual reader and no migration. Verified: `cargo nextest run -p mito2` 1824 passed, 2 skipped (three times, identical); `cargo nextest run -p mito-codec` 57 passed; clippy `-D warnings` and rustfmt clean. Tests pin the real captured pair (`host-020078` / `host-029838`, both retained with their own labels), the silent-loss variant with globally increasing timestamps, sixteen distinct keys under one forced tsid in both PK encodings, candidate isolation end to end, and rejection of a frozen v1 fixture. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): close the exact-identity review findings - The two-phase series scan assumed candidate discovery had already enforced tag predicates through an "exact assigned-series set". An assignment is a `(table_id, tsid)` hash, so a colliding partner shares it without matching the predicate, and skipping tag filters returned its rows. Apply the predicates the precise filter already holds instead of assuming them. - `MAX_KEY_BYTES` was checked when reconstructing a key from an artifact but not when a writer built one, so the writer could publish a key the reader refused; a single 8 KiB label already encodes past the cap. Length is bounded by the containers that store a key (one key per block, descriptor, page and column limits), and long labels are real data, so drop the cap instead of enforcing it on both sides. - The `__primary_key` column copies one key per output row, but the read reservation budgeted a fixed 32 bytes per column and metadata pages did not count the per-block key heap: 8192 rows of 8 KiB keys allocated ~64 MiB against a ~1 MiB estimate. Budget the real bytes and count the heap. - `SamplesReadRequest` carried both `keys` and `ids`, which the only caller derived from the same candidates; selection is key-only now and the reader derives its coarse pruning set, so the two selectors cannot disagree. - Restore the unknown-field-codec projection coverage that the frozen v1 fixture used to carry, against a generated v2 artifact, and assert v1 rejection through the compatibility property rather than a field name. - Drop the redundant columnar increasing-timestamp test (that writer never reads timestamps) and correct the samples format doc: the five internal columns and `__primary_key` are non-null, value fields stay nullable. Verified: `cargo nextest run -p mito2` 1830 passed, 2 skipped; clippy `-D warnings` and rustfmt clean. Each new test was run with its fix reverted first: the two-phase test returned the colliding partner's rows, the long-key tests failed with "series key exceeds 8192 bytes", and the 4 MiB pool test read 8192 rows of 1 KiB keys while allocating ~8 MiB. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): budget the exact key column exactly and drop stale skip-tag claims The read reservation charged the exact-key column its logical bytes, but the column was built with `BinaryArray::from_iter_values`, whose value buffer grows geometrically: measured on a 2135-row fixture it retained 4,989,432 bytes for 2,551,325 bytes of key data (1.956x), so a pool sized from the reservation was oversubscribed. `BinaryArray::new_repeated` allocates the offsets once and reserves the key bytes once (1.003x on the same fixture), which is what the reservation already assumes. The new test asserts the pool covers what the yielded batch retains, that the column keeps no growth slack, and that the reservation is released on drop; 2135 rows sits just past a doubling step so a growing builder fails it. Also make the documentation match the mechanism and the encoder: - `read/pruner.rs` still told readers the data phase skips tag filtering because candidate discovery had applied the predicates exactly once. Assignments are `(table_id, tsid)` hashes, so a colliding partner shares the identifier without matching the predicate, and the data phase does apply tag predicates. - The initial-window fallback test anchored on `reads[5]`, which is the second payload window; it now identifies the initial window by its payload-header offset and asserts the window spans 128 blocks, so a traversal change cannot quietly point it at another read. - The samples format descriptor sizes are recomputed from the encoder (`48 + 44 * C + key_length`): `224`, `45,236`, and with the leaf prefix `45,248`, each `+ key_length`; the page-fit sentence now states the enforced constraint (descriptor and leaf prefix inside one page) instead of claiming the widest descriptor always fits. Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy `-D warnings` and rustfmt clean. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): budget each projected sample array's fixed overhead The read reservation charged per-row bytes plus the payload sizes, but not the fixed cost of the arrays it builds: Arrow's `get_array_memory_size` is the array struct plus its buffer capacities, so a one-row, key-only projection reserved `key_length + 72` for the 1344 bytes it actually retained. Add `ARRAY_HEADER_BYTES` (`size_of::<BinaryArray>()`) plus `BUFFER_ALIGNMENT` per projected column, as their own term, and cover the one-row key-only case in `exact_key_output_reservation_covers_the_retained_array`: the pool must cover the batch's reported memory size while it is alive and be empty after drop. Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy `-D warnings` and rustfmt clean. Signed-off-by: discord9 <discord9@163.com> * chore(mito2): pin the series-index cache token to 1 The branch inherits `2` from an earlier layout change. This branch is experimental and has no deployments, so it pins the token at `1`: whatever a different build wrote into the rebuildable local store is discarded on startup and rebuilt from the SSTs. The token is not the compatibility boundary on its own. Artifacts are rejected on open when they lack the mandatory `identity_encoding` discriminator, so a hash-only artifact is never served even when the reset has not run yet (a read that precedes it, or an artifact outside this cache). Signed-off-by: discord9 <discord9@163.com> * feat(mito2): keep exact series keys out of the samples descriptor page A block descriptor used to embed the whole exact key, and a descriptor shares a 65,536-byte metadata page with its leaf prefix. That put an accidental ceiling on key length: 65,344 bytes with no value fields, 65,300 with one, and 20,288 with 1,024 fields. Valid SST primary keys can be longer, so their samples builds failed with "sample descriptor exceeds leaf limit" even though the key itself was accepted. The key now lives in a per-part key area and the descriptor carries a reference: part = [header 16][column payloads][key area][leaf pages / root][trailer 120] trailer = root ref 72 | depth 4 | version 4 | file_length 8 | payload_end 8 | key_end 8 (body 104), magic 8, XXH64 over the preceding 112 bytes 8 block = table_id u32 | tsid u64 | key offset u64 | key length u32 | min i64 | max i64 | ordinal u64 | rows u32 | columns u32 (56 bytes, fixed) Descriptor size no longer depends on key length (232 bytes for one present field, 45,244 for 1,024), so the ceilings are gone. Keys are written once per series, back to back in block order and without padding, so an equal reference means "same exact series" and a key may exceed any page limit; a reference is validated against `key_end` with a checked add, an overlap between consecutive new series is rejected, and a resolved key whose `(table_id, tsid)` disagrees with its descriptor is refused before any row is emitted. Reads stay lazy: only blocks that survive coarse selection fetch a key, at most once per series, merged per page and bounded by `MAX_READ_BYTES`, reused from the part's aligned cache when one is configured. Measured on the 260-series fixture: 260 per-series key reads before, 1 merged read after, with the fetched byte count asserted to be exactly the key bytes. The read contract also stops materializing one key per row: `SamplesBatch { batch, key }` carries the block's key once, the exact-key column is refused in projections, and `prepared` attributes a whole batch with a single binary search. For an 8,192-row block with a 1 KiB key the reservation drops from 8,388,608 B to 1,120 B (~7,490x), which is what lets a 4 MiB pool read it; `Candidate` keeps one owned copy of the key instead of two, and `retained_rows` is gone in favour of the `prepare_retained_rows` metric. No version bump: the layout changes in place. Artifacts written by an earlier shape of this branch (112-byte trailer, or keys inside descriptors) are unreadable and must be rebuilt, which the cache token and the required `identity_encoding` discriminator already bring about. One user-visible cost: keys retained by a scan are charged to its memory pool, so a scan that used to fit in 16 KiB needs ~14 KB more for 128 series. The fixture that pinned 16 KiB is raised to 64 KiB with the arithmetic in a comment. Verified: `cargo test -p mito2 --lib -- series_index::samples series_scan::prepared series_index::identity` 72 passed / 0 failed / 2 ignored on six consecutive runs; clippy `-D warnings` and rustfmt clean. Failing-before evidence, both produced by reverting the relevant hunk: - restoring `48 + 44 * C + key_len` fails the >64 KiB round-trip with "sample descriptor exceeds leaf limit"; - the 260-series coalescing assertion fails against per-series reads. Signed-off-by: discord9 <discord9@163.com> * test(mito2): assert the duplicate-drop on the preparation's own rows `overlapping_buckets_retain_only_new_label_rows` compared the deltas of two process-global counters (`prepare_unique_series`, `prepare_retained_rows`). Every preparation in the process increments both, so under parallel test execution the two deltas do not pair up and the test failed intermittently - it always passed in isolation. Assert the property the pairing was standing in for, on this preparation's own rows: read every series of the prepared source and require that no series keeps two rows at the same timestamp. A surviving duplicate from the overlapping bucket would repeat a timestamp; the counters can no longer make the result depend on what else the test binary happens to be running. Signed-off-by: discord9 <discord9@163.com> * test(mito2): drop the now-unused preparation counter helper The duplicate-drop assertion no longer reads the process-global counters, so `prepared_count` has no callers and `-D warnings` rejects the dead function in `cargo check` and `cargo clippy` CI. Remove it. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): scope key references per part, bound key reads, checksum keys Review of the key-area change found three defects that ordinary data can hit and three smaller ones. All six are fixed with a test that fails beforehand. Key references are file-local offsets, and the ordering state kept them across parts: a series continuing into another part looked like an overlapping reference (or a new series with a non-zero ordinal), and two different series in structurally identical parts could be mistaken for one because their offsets and lengths coincided. Reference ordering is now scoped to a part; continuation across a boundary is decided on the exact key bytes plus time/ordinal, and the loaded keys must be nondecreasing, a repeated key must continue its series. A key prefetch fetched the whole window and only then grew the reservation, and the per-key fallback charged after its read, so a small pool could read and assemble a page of long keys before reporting exhaustion. The peak (fetched entry + owned key + alignment/assembly) is now reserved before the read; on refusal the reader drops alignment first, then halves the window, and a returned key is cloned only after the block reserved for it. A test asserts that an unpayable prefetch issues no key read at all. The key area had no integrity check, so flipping one byte of a label - leaving framing and `(table_id, tsid)` intact - could silently change which series a block belongs to. Every stored key now carries its XXH64, excluded from the reference length and verified when fetched. The manifest discriminator now names the layout (`exact-sparse-key-area` for samples; the series index keeps `exact-sparse-key`), so an artifact with inline keys is refused at open instead of being marked reusable by maintenance and failing later at a part read. No version number changes anywhere. A key copy is also counted correctly when a series spans a metadata rollover: the "first copy of this part" decision is taken after the rollover, not before. Verified: `series_index::samples` 51 passed, `series_scan::prepared` 27 passed, `series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): charge the key peak on every path and chain the order check Four follow-ups from the confirmation review of the key-area change. The per-key fallback reserved only the retained and stored bytes and then fetched through the aligned payload cache, whose aligned/assembly peak it never charged, so a leftover key could read a whole aligned block the pool had not paid for. It now reserves the same peak as the page prefetch: aligned budget first, and on refusal an explicitly cache-less reader with the unaligned peak, before any read. The key carried across a part boundary was a deep copy taken while the per-part reservation was being released, so it lived on uncharged until the next part. It is now held by its own pool consumer, charged before the copy and released when it is replaced or when the boundary falls behind the scan. The content-order check was chained over the last *traversed* block, which an unloaded block overwrote, so a block pruned by selection could hide reversed or restarted keys - and could drop the boundary key entirely when a part ended on a pruned block. It is now chained over the last *loaded* key, with adjacency taken from "the last traversed block is the last loaded one", so a gap still requires increasing order and a strictly greater ordinal but not exactly +1. Finally, the key-area read counter in the tests only saw reads fully inside the key area, missing aligned reads that start before it; it now counts reads that intersect, and the cached-read test is documented as a cache-behaviour guard rather than a failing-before proof for the budget work. Verified: `series_index::samples` 55 passed, `series_scan::prepared` 27 passed, `series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean. Failing-before evidence: a 16384-byte key-area read issued with only 15080 reserved; a boundary key with 0 of its 1195 bytes charged; reversed keys hidden behind a skipped block; and a pruned block hiding the boundary key. Signed-off-by: discord9 <discord9@163.com> * docs(mito2): qualify the ordinal rule to one exact key The samples format said a loaded key's ordinal must still grow, which reads as if ordinals had to increase across different series too. They do not: only an equal exact key continues, an adjacent block of that key requires +1, and a different key starts at zero. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
c8e0d29df2 |
feat(mito2): publish table compaction results conditionally (#10310)
* feat(mito2): publish table compaction results conditionally Publish one region from a live input snapshot using an ordinary file-only edit. Permit repeated submissions and partial success without manifest task state, while preserving sequence, GC protection and uncertain-write recovery. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): write table compaction outputs to level 0 Route split outputs into the ordinary L0 pipeline so window merges and semantic folding stay with region compaction. Update publication validation and tests accordingly. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): release verified footers before manifest publication Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): compare typed table compaction options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover table compaction output validation failures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): clarify series count verification limits Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): reject index declarations without output index objects Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document table compaction outputs as L0 Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): map the table compaction publication module Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
77cb39b5d1 |
feat(mito2): record pending WAL tails in region manifests (#10308)
* feat(mito2): record pending WAL tails in the region manifest Adds `pending_tails` and `handoff_floors` to the region manifest and its checkpoint, and an optional `pending_tails` / `handoff_floor` pair to `RegionEdit`. An edit without them leaves the records unchanged. A floor is dropped once the flushed entry id reaches it; a tail stays until a commit removes it. Neither is written when empty, so manifests without them encode as before. Opening a region replays its WAL after this node's tails and floor, and WAL releases at open and after a flush stay below this node's tails. Nothing writes these records yet, so every region behaves as before. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(mito2): hold back reads and region changes while WAL tails are pending A region with pending WAL tails is missing data that only those tails hold. Until they are recovered, scans and memtable and SST stats fail with the retryable `RegionPendingTails` error. Truncate, entering staging, a repartition flush, a close-time flush on request, and ALTERs other than adding nullable columns without a default or changing options are rejected before they change any region state. Compaction pauses. A full-listing GC pass skips the region while it has pending tails, or while a newer manifest may exist or the manifest listing fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): name the WAL releases that keep pending tails Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): reject no-op ALTERs of restricted kinds while WAL tails are pending Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that reopening restores the pending-tail read gate Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): pin what the pending-tail fixtures and rejections promise Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): name the replay start in the open log and share the pending-tail check Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): state the per-node order of pending tails and who keeps later writes out of them Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that a cancelled compaction starts nothing while WAL tails are pending When a compaction cancelled by task id finishes, the queued manual request goes through handle_pending_compaction_request, whose pending-tails check replies Ok(0). The reply is read with try_recv so that a dispatched request fails the test at once instead of hanging it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drop the pending compaction task state when WAL tails skip it Skipping a pending manual compaction while WAL tails are pending replied Ok(0) but left its task state in place, so task inspection kept listing it as queued and cancelling it by id still succeeded. Clear the pending task state and publish before replying. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
e2e980fe39 |
feat(mito2): add column sketch options and index metadata (#10295)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
ffa0815285 |
refactor(series): hold the frame layout marker at one value
The series frame is the native PromQL path's internal transport, and that path lives on `narthex`, an experimental branch: it may break its own formats between revisions, so a frame revision number has no compatibility promise to protect. The marker becomes movable once the path reaches a release branch, where a producer and a consumer can legitimately be different revisions. The reader still refuses a frame that declares any other value, which turns a mismatched producer into an error instead of values decoded under the wrong columns. The two wire fixtures are renamed to `v1_metadata.json`, and the drift cases now use the value that is not current. |
||
|
|
f5c1020f5f |
feat(mito2): add shared SST storage and reclamation foundations (#10256)
* feat(mito2): add shared SST storage and reclamation foundations Add shared-level isolation, receipt replay and recovery, migration support, and fail-closed file protection. Keep coordinator submission and verified publication in the next stack layer. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): isolate table compaction manifest semantics Keep atomic manifest extension envelopes and lifecycle safety in the common engine. Gate table-specific receipts and remap admission with enterprise, preserving their wire format and rejecting unsupported state before replay or GC. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove manifest-backed compaction task state Keep file-only edits, shared-reference admission and GC safety. Task retries no longer require persisted receipts or task epochs; reject unpublished prototype task-state records rather than silently dropping them. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): guard staging manifests and reclaim shared range indexes Reject prototype task state when decoding staging blobs, matching checkpoint and delta decoding. Delete the region-scoped range index when a shared SST reference is retired, while leaving the shared object to global GC. Document the level-2 merge-split rationale and bind the delta guard's action tag with a test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): simplify GC deletion and lingering rules Require unknown files in a directory without a manifest to pass the lingering window before deletion, so an unpublished table-compaction output cannot be reclaimed out from under an active attempt. Replace the per-batch deletion loop with a single publication recheck taken before remote deletion, and drop the batching constant, the deferred-deletion accounting, and their test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): keep regions open after manifest validation rejection Classify InvalidRequest from pre-write applicability checks as definitely not persisted. Cover both shared-removal and unsupported-level rejections without stopping the manifest manager, while keeping ambiguous storage errors conservative. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): reject ambiguous shared SST reclassification Reject edits that both add a shared SST and remove the same file ID before admission and under the manifest lock. Keep ordinary edit semantics and verify add-only L0 reclassification survives reopen without diverging from the live Version. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): define truncate semantics for shared SST references Keep explicit truncation applicable to shared references and document the distinction from physical reclamation. Exercise full and time-range truncation with real shared SSTs, an old query, another referencing region, and reopen. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): define shared copy results and reject metadata imports Document copied_file_ids as installed logical references, including no-op retries. Reject shared SSTs in metadata-region copies before copying any objects or changing the target, and cover shared, mixed and repeated copy behavior. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(mito2): unify shared SST level references and bounds docs Use module-qualified shared-level constants consistently and document MAX_LEVEL as an exclusive bound for SST version operations. No runtime behavior changes. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): move the shared SST level to a max sentinel Use the u8 maximum as the shared classification so future ordinary levels can never collide with it, and map wire levels to fixed storage slots instead of indexing by value. Shared metadata, admission and filters keep their semantics; unallocated levels stay rejected. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): keep dropped-region tombstones while files are deferred Young unknown objects and hook-protected candidates on a dropped region must report needs_retry so the meta tombstone survives until they are actually reclaimed; reporting success would let the procedure drop the tombstone and orphan the retained objects. Cover both deferral paths plus the eventual clear. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): reject adopting an existing L1 SST as shared Adopting a shared reference is L0-only: the manifest would replace an existing L1 entry while the live version keeps both handles, duplicating rows until reopen. Reject the edit at admission before any state change and cover the rejection with a real SST. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
3051cb41db |
feat(json2): support fulltext indexes on JSON paths (#10064)
* feat(json2): support fulltext indexes on JSON paths Signed-off-by: fys <fengys1996@gmail.com> * chore: use rebased proto for JSON path fulltext Signed-off-by: fys <fengys1996@gmail.com> * refactor: reuse index target memory estimate for fulltext cache Signed-off-by: fys <fengys1996@gmail.com> * fix: filter disabled fulltext options for column targets Signed-off-by: fys <fengys1996@gmail.com> * docs: clarify fulltext intermediate path uniqueness Signed-off-by: fys <fengys1996@gmail.com> * refactor: name index targets consistently Signed-off-by: fys <fengys1996@gmail.com> * fix(mito2): apply JSON fulltext indexes per compatible target Signed-off-by: fys <fengys1996@gmail.com> * fix(mito2): skip invalid JSON fulltext index targets Signed-off-by: fys <fengys1996@gmail.com> * fix(mito2): require enabled fulltext indexes for column targets Signed-off-by: fys <fengys1996@gmail.com> * refactor: simplify JSON fulltext index construction and pruning Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): unify fulltext target compatibility and creator APIs Signed-off-by: fys <fengys1996@gmail.com> * perf(function): preserve string layouts when evaluating matches Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): centralize fulltext predicate extraction errors Signed-off-by: fys <fengys1996@gmail.com> * refactor: remove enable flag from JSON fulltext proto options Signed-off-by: fys <fengys1996@gmail.com> * refactor: clarify JSON fulltext index helpers and imports Signed-off-by: fys <fengys1996@gmail.com> * fix: allow rebuilt fulltext indexes on ordinary columns Signed-off-by: fys <fengys1996@gmail.com> * test: spell out false positive rate in fulltext tests Signed-off-by: fys <fengys1996@gmail.com> * chore: update greptime-proto to upstream merged revision Signed-off-by: fys <fengys1996@gmail.com> * fix(mito2): initialize fulltext option in multi-region tests Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
|
735401adc0 |
feat(mito2): consolidate columnar samples into bounded daily generations (#10129)
* perf(mito2): consolidate columnar samples into daily windows Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): isolate daily source coverage and version the local cache Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): use the exported index purge channel Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): exercise daily source groups across the merge limit Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): build stable daily groups from populated source windows Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): anchor daily coverage once per source group Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): validate partial-day reads and grouped rebuild cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): skip daily sample readers outside query bounds Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
58a67a47c3 |
feat(query): execute native raw promql over real datanodes (#10150)
* feat(promql): serialize raw series source bindings through Substrait Signed-off-by: discord9 <discord9@163.com> * feat(query): transport interleaved raw series fragments Signed-off-by: discord9 <discord9@163.com> * feat(promql): reconstruct interleaved raw series at repartition Signed-off-by: discord9 <discord9@163.com> * feat(query): plan distributed native raw series Signed-off-by: discord9 <discord9@163.com> * chore: satisfy clippy for the raw series transport The workspace clippy run rejects a redundant rebinding, a constant-size `chunks_exact`, and an implicit `String` clone in the tests these commits carry. No behavior changes. Signed-off-by: discord9 <discord9@163.com> * feat(query): execute native raw promql over real datanodes Plan a whole native statement as one fused node over raw series sources when the frontend reaches regions over the wire, including single-selector statements, while keeping the in-process split and segment shapes unchanged. Validate the whole tree with the runtime constructor before committing to that plan, so unsupported shapes still fall back to the legacy route. Bound the region reads of that topology with a query-wide permit budget of target_partitions, held until each region body is drained; other plan shapes keep unbounded reads rather than risk a body budget they could deadlock. Acceptance on a real two-datanode cluster with TCP frontend clients: time-partitioned series are reconciled before the fused node and match the unpartitioned control, with the DoGet peak at the target bound; series that no region shares keep contiguous per-series order and reach the fused node without a reconciling router. Signed-off-by: discord9 <discord9@163.com> * chore(deps): pin greptime-proto to the raw series transport PR The raw series Substrait message lives in greptime-proto PR #349, which is not merged, so pin its head commit instead of carrying a local path patch. Move the rev to the merge commit once #349 lands. https://github.com/GreptimeTeam/greptime-proto/pull/349 Signed-off-by: discord9 <discord9@163.com> * fix(promql): reconcile a discovered union of raw series A nameless selector reads its matches as one `Union`, and `UnionExec` reports the frame metadata of its last child. A time-partitioned metric sorted before a host-partitioned one therefore looked contiguous, and the fused node skipped the reconciliation of the fragmented member's series, silently folding region fragments as separate series. Lower the declaration on the fused edge to the weakest ordering any member declares, before the fused node reads it. The union itself is unchanged; the fused node's existing interleaved-input path inserts the router. A projection carries the lowered declaration because the pinned physical planner ignores a union's own schema. Cover it with a planner regression for both member orders and an all-complete control, and with a real two-datanode acceptance case that mixes a time-partitioned and a host-partitioned metric and checks the extrapolated increase per series. Signed-off-by: discord9 <discord9@163.com> * test(promql): cover aggregates over discovered metrics A nameless selector's metric comes from its own frames, so a folding aggregate is a multi-input tree even with one selector: apply the same preflight and order-dependent refusals on the remote route. Cover `sum(increase(...))` and grouping by the metric name over the mixed discovery fixture. A grouping that names only `__name__` stays declined, since the output schema cannot name that group: here the metrics also differ by `site`, so grouping by the name and `site` exercises the two groups. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep remote multi-source refusals the split made Skipping the aggregate split and `segment_tree` over the wire must not drop their admission refusals. Two sources that fold groups by metric name would report groups that differ only by name under one label set, and an order-dependent operation over a discovered selector's union would answer in arrival order. Scope both refusals to the actual multi-source inputs rather than to the logical-table-id heuristic: a discovered selector reads several metrics through one selector, and a Mito table beside a metric one carries no logical id. Cover the Mito-plus-metric grouping refusal on a real two-datanode cluster, and cover a discovered `sum by (__name__)` over a raw selector (the name survives it) and a folding `sum(increase(...))`. Signed-off-by: discord9 <discord9@163.com> * docs: drop the local proto patch prerequisite The dependency is pinned to greptime-proto PR #349's commit, so the wire fixture no longer needs a local path patch. Signed-off-by: discord9 <discord9@163.com> * chore: finish the narthex rebase resolution Keep the upstream aggregate split (`FusedSeries::split_aggregate`) and pass the engine state through the native planner's callers, while the remote route still declines the split it cannot carry. Signed-off-by: discord9 <discord9@163.com> * test: pass the engine state to the native operand test The native operand planner now takes the engine state, which decides whether a statement is planned for remote regions. Keep the histogram operand contract test on that signature. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep the split and segment plans over the wire Remote planning disabled the aggregate split and segmentation because the split's producer read a bare scan. The producer reads whatever the statement prepared, and over the wire that is the raw series source: the partial half runs per partition on the frontend, the final half folds the exported states, and no aggregate state crosses the wire. Keep the split and let segmentation build evaluated segments from the raw sources instead of bare scans, so per-series window work keeps the scan width. Admit the resulting fused chain to the query-wide read budget, since each fused node polls its input while it is drained. Remove the recovery-projection wrapper: the analyzer no longer inserts that projection, and the physical planner consumes the frame boundary as it is. Serialization is now only the node's encoding: the codec no longer decides placement, and the metric-name binding is checked against the declared frames rather than the node types below. Signed-off-by: discord9 <discord9@163.com> * refactor(query): drop the in-process encodability exception A split aggregate encodes now, so the analyzer no longer needs a special case that skipped the encoding gate for it. The gate itself stays: every plan a region receives still has to encode. Signed-off-by: discord9 <discord9@163.com> * test: cover a repeated selector over one partitioned table Two occurrences of one time-partitioned metric keep their own raw source, time bounds and identity over the wire, so a ratio of two windows and a binary expression over two identical ones stay native and reconcile each occurrence's fragments before the fused node reads them. Compare both against the unpartitioned control. Signed-off-by: discord9 <discord9@163.com> * fix(table): refuse a scan that plans interleaved ordering A scan's planned ordering is a guarantee its own partitions keep. Interleaved is the transport annotation for fragments several producers contribute, so a scanner that planned it has no raw-series read whose frame would honour that declaration. Reject it in the scan conversion, while the frame codec keeps accepting interleaved raw frames from the router that reconciles them. Signed-off-by: discord9 <discord9@163.com> * test(promql): assert the metric name beside the value `sum by (__name__)` over a discovered selector was asserted as a bare sorted list of values, so a plan that grouped by the wrong metric name - or dropped the name and folded everything into one series - could produce those same numbers. Read `__name__` and `val` by column name (the result puts `__name__` last, and a fold that drops the name drops the column rather than emptying it) and compare the `(name, value)` pairs against the fixture's own metric names. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep the hybrid route for a partitioned pair A multi-selector tree over a partitioned table was admitted as one fused node for any pair of tables, not only for scans of one table. That preempted the frontend hybrid route, which copies the matcher's labels into each operand's scan and checks uniqueness above the join. `native_hybrid_binary_05_06_08` depends on exactly that plan: its partitioned RHS holds a duplicated `job="other"` group, so the whole-native binary reports the duplicate instead of the value the hybrid route returns. Refuse the whole-fused plan for that shape in process again; the remote raw-source admission is unchanged, since a remote multi-selector tree over a partitioned table is the point of this branch and the distributed acceptance tests read it. The hybrid route then has to behave the same way it does without native operands: `matching_filters::propagate` now runs for them too, and a rewritten operand is re-planned through `try_plan_native_operand` with its original `selector_matcher` preserved, falling back to the ordinary plan when the native route declines. The unpartitioned case, whose duplicate group the whole-native binary is expected to report, is unchanged. The two new re-plan call sites put the native route's large future inside the recursive planner frame at every nesting level (~178 KB became ~209 KB), which overflowed the 2 MiB test-thread stack in `test_parse_or_operator` even where the new branch never ran. Box the future in a non-async helper and move the re-plan into `replan_rewritten_operand`, which brings the frame back to the old size without changing the behaviour. The partitioned-pair unit test asserted the removed shape. It now asks for the remote engine state, where a cross-table pair really is one fused node over two raw sources, and asserts that the same pair in process keeps the hybrid route. Signed-off-by: discord9 <discord9@163.com> * chore(deps): pin greptime-proto to the merged raw series transport GreptimeTeam/greptime-proto#349 has been merged into `narthex-proto` as `f51cb5312bf7`. Move the rev from that PR's head commit to its merge commit, in `Cargo.toml` and `Cargo.lock` together, so the pin no longer points at an unmerged ref. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
a995ad0951 |
test(compat): exercise the bloom-token case only below v1.3.0 (#10170)
`fulltext_bloom_tokens` reads a fulltext bloom SST in the first format and
asserts that rows written by the older binary stay visible next to a current
one. Its `from_range` started at v1.0.0, but the released v1.3.0 binary
already writes `bloom_version: 2`, so the setup produces two current-format
SSTs and the expectation ("one old, one new") can never hold:
expected: | false | 1 | actual: | true | 2 |
expected: | true | 1 |
Narrow the range to the releases that still write the first format. The
constraint language has `>=`, `<=`, `==` and `=` only, and several entries
are alternatives rather than a conjunction, so `<=v1.2.99` is the bound
that excludes v1.3.0 while keeping v1.2.x and older - including the
nightly window.
Verified with the compat runner: `--from-version v1.2.1 --dry-run` selects
the case, `--from-version v1.3.0 --dry-run` skips it with
`from_range ["<=v1.2.99"] does not match version 'v1.3.0'`, and
`update-compat-versions.py --check --published-only --check-anchors`
passes.
Signed-off-by: discord9 <discord9@163.com>
|
||
|
|
5fe9bdc814 |
feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision. Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat(exemplar): persist source metric identities Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): bound exemplar write memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): correct batching with exemplars Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): accept mixed-case sparse encoding Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(exemplar): reduce write preparation overhead Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(prom): reuse exemplar source targets Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: clippy Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): defer exemplar series expansion Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
7d42feb023 |
chore(ci): update compatibility versions (#10051)
* chore(ci): update compatibility versions
Signed-off-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com>
* test(compat): bound the fulltext bloom case below v1.3.0
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit
|
||
|
|
ec240ad7e6 |
feat(mito): preserve ordinary sparse keys across writes and schema changes (#9920)
* feat(mito): support ordinary sparse append tables Ordinary append tables need sparse keys for optional series tags, including JSON2 values written through row and bulk paths. This change validates sparse tag schemas and defaults, keeps key identity stable when nullable tags are added, selects flat SST storage, and preserves append-mode semantics when reopening regions. Includes storage, bulk-write, schema-evolution and repartition tests, plus SQL and mixed-version fixtures. Existing metric-engine sparse key handling remains supported. This is the storage prerequisite for exemplar ingestion; it introduces no exemplar protocol endpoint. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito): preserve reserved sparse key semantics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito): clarify sparse errors and avoid panics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(mito): reduce sparse key validation overhead Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(mito-codec): reduce sparse validation cost Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(mito): remove ordinary sparse support Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
eb8e4316ec |
ci: backport scoped heavy CI to narthex (#10053)
* ci: backport scoped heavy CI to narthex Backport #9927 and the CI-only changes from #10043. Adapt script references to narthex, preserve manual platform checks, and document the default-branch schedule limitation. Upstream-Commit: |
||
|
|
045cf74c3f |
perf(mito2): pipeline compaction and parallelize decoding and encoding (#9945)
* perf(mito2): merge compaction inputs on a separate task Compaction read, merged and encoded each output on a single task, so parquet encoding waited for the merge and the merge waited for encoding. The merge now runs on its own compaction runtime task and hands merged batches to the SST writer through a bounded channel. The channel also closes when that task panics, so the reading side awaits the task before reporting the end of the input. A panic fails the compaction instead of producing a truncated SST. One output of 4.8M rows took 1.28s instead of 1.65s. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): parallelize decoding and encoding of compaction outputs Parquet encoding took about two thirds of a compaction output's time and ran on a single thread. When `experimental_compaction_output_parallelism` is above 1, the SST writer encodes the columns of each row group on up to that many blocking threads of the compaction runtime while the next row group is buffered, and the merge decodes up to that many input files ahead of it on compaction runtime tasks. The parallel writer feeds every column writer the same input as `ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group reach the object store writer one row group later, so `max_file_size` may split a file one row group later. The option defaults to 1, which keeps today's CPU usage. One output of 4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): cancel compaction producers before consumer polling Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): support per-table compaction output parallelism Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain encoding tasks before returning failures Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime. Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain encoders on SST write errors Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error. Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain compaction outputs on cancellation Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures. Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
2e84833ac8 |
feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): reject partition column type changes and quote rule identifiers Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(table): keep time index widening for partition columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(partition): replace PartitionFunction with IcebergHash Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(partition): describe iceberg_hash in the function module Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: quote partition rule identifiers in SHOW CREATE TABLE expectations Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(partition): check hash rules in information_schema and repartition target columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): coerce hash partition rules against the evaluation schema Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): keep quotes in partition column names when building filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
5e9bb3c689 |
feat: support native histograms in sampled remote read (#9971)
* feat: support native histograms in sampled remote read Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: align native histogram ingestion with remote read validation Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
6f5c064f54 |
fix(meta): fence datanode processes replaced by a newer process (#9878)
* fix(meta): fence datanode processes replaced by a newer process
Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.
The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.
Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache load and pusher updates coherent
A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.
A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.
The epoch check moves into the heartbeat service.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache coherent with writes while it is disabled
A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(meta): pin the epoch 0 exemption and pusher order, and update docs
The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): register a datanode pusher only after its first request passes
A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.
Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): check a datanode epoch again before replying
A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): handle a datanode's requests one at a time
A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.
The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
|
||
|
|
2b67d0d60c |
test(series): pin literal ordering declarations and decoder validation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
62652ec62e |
fix(series): validate v2 ordering and repair scan binding fixtures
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
b20e22b426 |
refactor(series): unify raw and evaluated SeriesFrame codec
SeriesFrame is the internal transport of the Narthex native PromQL path, and
it could only carry raw samples: an evaluated result had to fall back to a
plain row table. This makes one self-describing Arrow format carry both, so a
receiver decodes a frame from the frame alone, without the scanner, region, or
planning context its producer had.
What is unified is the physical representation and its codec, not the runtime
data structures. RawSeriesChunk, Series, and EvaluatedChunk stay as they are;
only the adapter boundaries branch on the sample shape.
Three combinations are legal and every other one is refused in both
directions:
raw + key + timestamps a storage raw-series scan
raw + explicit + timestamps a raw input whose labels are decoded
evaluated + explicit + grid_spans a compute node's output
An evaluated frame cannot claim a storage key, because the computation may
rewrite its labels; raw samples have no grid to be addressed on. The schema
builder and the declaration parser both enforce that, so a frame cannot be
written or read in a shape no producer could mean.
An evaluated frame declares its GridSpec once in schema metadata and carries
absolute grid positions per row as spans, so a sparse grid costs its runs and
not its holes: nothing is expanded, and no position a chunk does not hold is
filled in. A chunk with no position is legal exactly when it closes its chain,
and it is kept as one frame row, because dropping it would drop the finality
it exists to carry.
Layout is v2 and the writer and reader of the reserved greptime.series.*
metadata live in one place each, so a producer and a consumer cannot spell one
frame two ways. The unreleased v1 format is not read: the in-tree producers,
consumers, tests, and the two frozen wire fixtures all move to v2 together.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit
|
||
|
|
6a40c0b811 |
feat(mito): implement columnar samples format (#9887)
* docs(mito): specify columnar samples format Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): implement columnar samples format Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): address columnar samples review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * style(mito): use crate-rooted samples imports Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
5af1e3301e |
fix(mysql): cap timestamp output at microsecond precision (#9831)
* fix(mysql): cap timestamp output at microsecond precision Truncate MySQL text timestamp results to microseconds while preserving the existing second and millisecond formatting. Keep stored nanoseconds and explicit integer/string representations unchanged. Fixes #9828 Signed-off-by: luofucong <luofc@foxmail.com> * chore: ignore local output directory Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
7309800aef |
feat(datatypes): describe label decoding in the raw series frame schema (#9786)
* feat(datatypes): describe label decoding in the raw series frame schema A raw series frame written with key-encoded labels carries the primary key as opaque bytes; only the binding held by the scanner says how to turn those bytes back into a metric name and labels. A consumer that receives the frame without that binding cannot decode it, so the schema now describes its own label layout. The description is compact JSON under `greptime.series.label_decode`, carrying a version, the metric name, the primary-key encoding, the ordered key columns with their physical types, and the logical tags to expose -- exactly what decoding reads. It is derived from the binding, so there is one writer, and a reader that does not implement the version it finds rejects the frame instead of guessing. `unpack_raw_series` keeps its binding argument and now requires it to agree with the description the schema declares, so the two cannot drift. `PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key layout, so the metadata-backed and description-backed paths share their validation instead of growing a second decoder. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datatypes): compare label descriptions by what their order means The frame's declared label-decode description was compared to the binding's with plain equality, but the description has one part whose order carries no meaning: the decoder sorts logical tags by name when it binds them, so the same tags listed in another order denote the same labels and were being refused. `same_label_layout` canonicalises exactly that part and leaves `primary_key` in place, because for a dense key the array position is the encoding position: the same columns in another order are a different key, not the same key reordered. Both fields now say which order they mean. Three checks go with it. The binding stays for a stated reason -- it establishes identity, the physical region key layout a decoded identity came from -- rather than because a count of call sites kept it. The post-pack schema comparison in the series scan could not fire: the leaf already re-derives that schema and compares it, and every `SeriesSchema` field is covered by the checks between them. And two test groups re-asserted what the tracked v1_metadata fixture pins case by case. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): decode a raw frame from the frame alone `unpack_raw_series` took the label mode, the ordering and the caller's binding along with the stream, so a consumer that received a frame from a peer still needed a local copy of the binding to read it — the thing the frame's own description exists to avoid, and the reason the description was freezing a wire format for nobody. The stream is now the only input: the field schema, the label mode, the label decode description and the producer ordering all come from the frame's schema, and every frame is validated whether or not a caller asked for it. `expected` stays as an optional check on the schema that is declared — field names and their order, and no evaluation grid — and supplies nothing to the decoder, so a caller cannot smuggle in a layout the frame does not declare. Three call sites compared the caller's binding against the frame and all three were load-bearing: a peer's response is not trusted to be the region that was planned, and a native plan's child is an arbitrary execution plan. Those became `ensure_frame_matches_label_source`, which states what it compares: the label layout, not the region. Two regions with the same tag ids and names compare equal, and the doc now says so instead of claiming to establish identity. A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers `label_decode_spec()` instead; every consumer that decides whether to decode reads the description, so the placeholder never stands in for "these labels are complete". The merge contract is the description, not the binding: inputs are compared by the layout they declare, and a binding that could not be expressed on the wire — a `RegionMetadata` snapshot — no longer makes the merge stricter than the frames it merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a copy, because the caller's binding keeps its own order and the table scan compares producer tag lists positionally, and readers compare by equality. The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to cover. Rather than mirror all 25 datatype variants into a type this module owns, a frozen fixture records the text of every variant and an exhaustive match makes a new variant fail to compile until it is recorded. The version policy is written next to the constant: exact-text comparison across nodes, unknown versions rejected, any same-version change breaking — with the planning-time fallback stated as intended and marked not implemented. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(store-api): make every key-encoded raw stream declare its layout A raw series stream whose chunks carry key-encoded labels had to be decoded against a scan-bound `RawLabelSource`, and two consumers compensated for producers that stayed silent by deriving a description from that binding: `pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels` did not compensate, so an unframed producer stream reached it with no description and a chunk carrying no labels, and failed. The stream itself is the authority for its chunks. A key-encoded producer knows its own region metadata and can state the layout it writes, exactly as a frame states the layout it carries, so the split contract — "a producer declares its identity, a frame declares its layout" — is replaced by one rule: a stream whose chunks carry key-encoded labels declares the description, and a stream whose chunks carry complete labels answers `None`. mito2's `RawStream` caches the derivation at construction, so the declaration exists before the first poll and for an empty stream. The trait documentation states the rule and no longer exempts unframed producers. Both fallbacks are gone: `pack_raw_series` reads the stream's declaration only, and rejects a key-encoded stream that declares none rather than packing it as an explicit-label frame, which would describe chunks carrying no labels as if they carried all of them. The merge reads the declaration only. Test fixtures that model key-encoded producers now declare their layout too; fixtures that carry complete labels keep answering `None`. The new mito2 test decodes a real produced key through the declared description for both dense and sparse encodings, and checks the declaration is present before polling even for an empty stream. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(common-query): let the declaration decide the frame mode alone `pack_raw_series` rejected a key-encoded stream that declared no description outright, so a stream that stays silent could not be packed at all. That check was redundant: the declaration already decides the frame's label mode, and the first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor` with "raw chunk label mode does not match frame mode", so no such stream can pack data either way. What the check added was an early error for a producer that already breaks the contract, in exchange for reading the stream's scan binding here — in the one function whose rule is that the declaration is the only decode input. Drop it, so pack keeps deciding the mode from the declaration and nothing else. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(store-api): assert the fixture's declared label decode description The mock prepared source derives a `LabelDecodeSpec` from its label source and reports it through `RawSeriesStream::label_decode_spec`, but no assertion read it, so the derivation could drift from the chunks the fixture actually builds. Check it against the description the fixture's own key columns imply: an explicit source declares nothing, a primary-key source declares the metric, the dense encoding, the one key column and its tag. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): let the frame's declaration be the only label authority Two things still treated a caller-supplied binding as authoritative after the declaration became the decode input: - A merged stream forwarded its first input's scan-bound identity, though a merge is not a region scan and decodes from the declaration alone. It now reports no bound identity. - `KeyColumn.tag` repeated an invariant every key column is a tag. The check moves to the one place a decoder is bound, so a snapshot built without validation is still rejected instead of silently decoded. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
c879564671 |
chore(common-query): drop the raw-series request extension transport (#9815)
* chore(query): drop the prepared raw-series RPC skeleton The raw-series preparation RPC was merged into `narthex` for a plan that never landed: no server or frontend dispatches `greptime.prepare_raw_series.v1`, and the three `Datanode` methods, `bind_raw_series_query` and the `QueryEngine` trio have no call sites and no implementors. `PreparedNamedRawSeries` had no user outside them. The wire envelope never shipped -- no tag contains the commit that added it -- so the compatibility fixtures go with it. The harness drives `wire/` fixtures from the Rust tests that own their context conversion rather than from SQLness discovery, and has no retirement mechanism to record instead. Left in place: `RawSeriesRequest` and its extension key, `raw_series_frame_ordering`, `unpack_raw_series` and `RawLabelSource`, all of which the surviving series-frame path still uses. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(common-query): drop the raw-series request extension transport The prepared raw-series RPC carried a `RawSeriesRequest` through a query-context extension, and with that RPC gone (`607de914464`) nothing in `src/` writes or reads the extension any more. What remains is the binding the table scan actually uses: `RawSeriesRequest`, `RawSeriesTag`, `bind`, `validate` and `same_encoding_identity`, unchanged. Deleted: the extension key, `take_raw_series_request`, and the two serialization methods that only existed to encode and decode that transport, plus the wire fixture whose only reader was the test for them. The fixture never shipped — the commit that introduced it is in no tag. The tests that covered `validate` are kept, with their rejections now asserted directly instead of through a JSON round-trip: an unsupported version, an empty field list, a duplicate field, and a non-field column id are all still refused. The binding test is untouched, including its order, identity and `Arc` assertions. `pub` items in a library crate never trip `dead_code`, so the compiler could not have found this; the greps above are the evidence. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
1156cfbe0c |
chore(query): drop the prepared raw-series RPC skeleton (#9797)
The raw-series preparation RPC was merged into `narthex` for a plan that never landed: no server or frontend dispatches `greptime.prepare_raw_series.v1`, and the three `Datanode` methods, `bind_raw_series_query` and the `QueryEngine` trio have no call sites and no implementors. `PreparedNamedRawSeries` had no user outside them. The wire envelope never shipped -- no tag contains the commit that added it -- so the compatibility fixtures go with it. The harness drives `wire/` fixtures from the Rust tests that own their context conversion rather than from SQLness discovery, and has no retirement mechanism to record instead. Left in place: `RawSeriesRequest` and its extension key, `raw_series_frame_ordering`, `unpack_raw_series` and `RawLabelSource`, all of which the surviving series-frame path still uses. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
8543cbf401 |
feat(mito): scaffold series format writers and artifact lifecycle (#9788)
* feat(mito): add pluggable series artifact reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): prepare independent series reads from SST artifacts Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): prepare mixed sidecar SST and memtable series reads Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): make prepared series sources replaceable Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): narrow series formats to artifact lifecycle integration Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito): populate artifact regression buckets above build threshold Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
3e2751d71d |
feat: improve SQL analyze scan diagnostics (#9565)
* feat: improve SQL analyze scan diagnostics Signed-off-by: evenyag <realevenyag@gmail.com> * test: redact dynamic filters in sqlness results Signed-off-by: evenyag <realevenyag@gmail.com> * test: stabilize analyze snapshots for merge queue Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove obsolete scan_details assertion Signed-off-by: evenyag <realevenyag@gmail.com> * test: refresh explain snapshots after partition filter changes Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
da88ff8653 |
fix(partition): route NULLs like row inserts and accept signed-minimum bounds (#9697)
* fix(partition): route NULLs like row inserts and accept signed-minimum bounds Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): keep equality two-valued for NULL rows and accept unsigned -0 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): treat NULL masks as false in batch routing and keep comparisons bare Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(compat): update partition filter in analyze verbose expected output Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
3ef71b5702 |
feat(common-query): add raw series request binding and preflight types
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
997cbf87f7 |
feat(common-query): declare raw series frame ordering in schema metadata
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
672e94f61d |
feat: add column encoding option and use it for trace span ids (#9629)
* feat: add column encoding option and set it for trace span ids COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'. Either disables the dictionary for the column; delta_binary_packed is limited to integer and timestamp columns. OTLP trace v1/v2 tables now create span_id and parent_span_id with plain encoding and 64KiB pages, and timestamp_end with delta_binary_packed. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: add compatibility cases for column encoding options Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: move column encoding downgrade checks into downgrade_compatibility Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
60655a933b |
fix(log-store): persist object store WAL region releases (#9579)
* fix(log-store): persist object store WAL region releases A region's release lived only in memory: a restart after DROP TABLE and before the next object never re-established the dropped region's watermark, so its segments counted as live and pinned garbage collection of the whole prefix. A release now writes a release record (region id and the entry id the region is released through) into an object sealed at once, and returns once that object is durable and indexed, in both acknowledgement modes. Recovery seeds the obsolete watermarks from the release records of the indexed objects before it computes any reclaim boundary. obsolete_all releases in full; the new LogStore::release_region, a no-op by default, releases a dropped region through the largest entry id the store handed out for it, and Mito calls it after obsolete when it drops a region. Objects with release records use format version 2, the version 1 layout with the records at the end of the footer; every other object is still written as version 1. A counter of release records made durable is added. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe persisted object store WAL region releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): carry a failed region release into the next object A release whose batch failed left its record only in an orphan object, so a drop that failed there could still leave the region unreleased after a restart. The store now keeps the release records of failed batches and seals them with its next batch, on a timer tick when no entry is admitted, so the release becomes durable once a create succeeds again. The version 1 and version 2 object fixtures are frozen under tests/compatibility/wal/object_store and loaded by the format tests, and the RFC no longer says that state is persisted only in headers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): complete a region drop whose WAL release fails A drop request is not sent again once the region was removed, so a drop whose WAL release failed used to leave the region half dropped. The drop now completes, cleaning up the region's runtime state and scheduling its files for deletion, and reports the release failure. In the durable mode the object store WAL now creates one more object at stop with the release records of failed and dropped batches, so a graceful stop does not lose a release that failed. The release record counter is described as counting the records of indexed objects. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe the stop-time release create of the WAL actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): state the accepted window of unreleased dropped regions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe release waiters in the WAL actor's outcome table Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail region releases while the object store WAL is failing A failing store keeps its backlog and rejects appends until a create succeeds, so a release would otherwise wait for an upload that is not coming. A release that arrives in that state now fails at once with the retryable error, and one whose batch is still pending when the store enters it fails as well. Neither poisons the store: the record of the first waits for the next sealed batch, like that of a batch that failed, and the record of the second stays in the backlog, so both become durable once a create succeeds. Rejected releases count in rejected_appends_total. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe region releases while the object store WAL is failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the drop crash window in the drop procedure The crash window between the deletion of a table's metadata and a durable release record is a gap of the drop procedure, not of the WAL. Describe it as far as the code shows, name the remedy that belongs in the procedure, cleaning up offline a region a resumed region drop finds not open, and list it under future work. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep rejected release records coalesced while failing A release rejected while the store is failing kept its record for the next sealed batch, but every timer tick then sealed that record alone behind the refused backlog, so a release repeated across ticks grew the backlog without bound. A failing store now keeps such records coalesced and seals them on the first tick after a create succeeds. The fault-matrix case of a release while the backlog is refused now runs with a 10 ms flush interval, repeats the release, and checks that the backlog does not grow and that recovery reads the record back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): keep rejected release records unsealed while failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): cite the drop procedure issue for the crash window Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): let a stop seal rejected release records of a failing store Keeping the records of rejected releases out of sealed batches while the store is failing also kept them out of the seal a stop requests, so a stop whose last attempt of the backlog succeeded dropped them and the released region pinned collection after the restart. A stop now seals them with the backlog. A fault-matrix case rejects a release while the store is failing, stops, lets both creates succeed, and checks the record after a reopen. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the stop of a failing store with rejected releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep release records readable by older object store WAL readers Release records were written as a version 2 footer, which a build without it refuses to read, so rolling a node back after a drop failed to open the whole WAL prefix. They are now the single entry of a segment of the reserved region u64::MAX at the first position of a version 1 object: an older reader indexes that segment as a region it never opens, and recovery reads the segment back with one range read per object that holds one. The segment never holds the reclaim boundary and names no namespace. A durable stop whose last create of release records fails now reports the error instead of returning Ok and losing the records silently. The tests are trimmed: the release unit tests and fault-matrix cases that repeated each other are merged, and the Mito drop tests share one parameterized case. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe release records as a segment of a reserved region Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): report release records a poisoning stop drops A durable stop whose last create of release records met a conflict poisoned the store and returned Ok, losing the records silently. Stop now reports the error whenever release records are dropped after it began, by a rollback or by poisoning. The test of a failed release covers that conflict, and seals its record on a tick after a one-shot create failure, so the next tick cannot fail too. The documentation now says that recovery reads the segments of release records, and that one that does not decode fails recovery. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how recovery reads segments of release records Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): say that stop reports release records a conflict drops Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): restore the transiently failing release create in enqueued mode The fault matrix again covers a release whose create fails transiently in the enqueued mode: the release waits for the repeat under the same sequence and returns once it succeeds. The encode_batch doc now says the release segment is the last segment of the object. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the release segment and count the restored case Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL release Returning the release error from the drop left a metric physical region half dropped: the metric engine stops after the data region and never drops the metadata region or forgets the physical region, and a retried drop finds the data region gone and counts it as done. The release error gave the caller nothing to retry, since the region is already removed and the log store keeps the record for its next object. The drop now logs the error and succeeds. A metric engine test drops a physical region while the WAL releases of both mito regions fail, and checks that both regions and the physical region state are gone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed release no longer fails the drop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL obsolete A store that the release of another region poisoned fails obsolete as well, which left the metric engine with its metadata region undropped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(metric-engine): drop a physical region whose WAL obsolete or release fails Cover a failed data release, two failed releases and a conflicting release that poisons the store, after the regions wrote to the WAL, and check that a store opened later keeps no segment of either region. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed obsolete does not fail the drop either Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): fail the drop on an obsolete error of other WAL providers Only the object store WAL is shared by the regions of a store that the release of one can poison; Raft Engine and Kafka keep failing the drop when obsolete fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a stopped or poisoned store keeps no failed release Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
5c4d9eb4f8 |
fix: align fulltext bloom tokens with matches_term and use them for LIKE (#9470)
* fix: align fulltext bloom tokens with matches_term and use them for LIKE The bloom fulltext index stored tokens that did not follow matches_term's word boundaries (`_`, ASCII next to Han, Han substrings), so pruning dropped rows matches_term accepts. Tokens and query probes now share the character classes of matches_term, Han runs are indexed as bigrams, and LIKE patterns use the same probes. The bloom blob is written under a new key; blobs from older versions are no longer probed. Tantivy no longer serves matches_term. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: drop unused lazy_static from index and avoid typo-flagged test tokens Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: link like_probes to tokenize instead of the downstream ScriptTokenizer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: name probe extraction with verbs and document CharClass Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: rebuild legacy fulltext bloom indexes and probe lower(col) The manifest records the fulltext bloom version per column, so ADMIN BUILD_INDEX now rebuilds SSTs whose bloom index predates the current tokens, and ssts_index_meta shows the version. Case-insensitive bloom indexes also store the tokens of the whole lowercased text for rows where it can differ (U+0130, U+212A, sigma), so matches_term(lower(col), ..) and lower(col) LIKE .. use the index. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover legacy bloom index rewrite by compaction in the compat case Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: log outdated fulltext bloom indexes in the consistency check Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs: explain the unrecorded fulltext bloom version Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
caefcfd012 |
test: add inverted index compatibility case (#9519)
* test: add inverted index compatibility case (#5188) Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * test(compat): assert the newly indexed column stays queryable (#9535) Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * test(compat): narrow to_range to versions carrying #9545 The filtered read added in the previous commit only returns rows once #9545 is present. That fix is on main only (no backport-* labels), so it ships in the v1.4.0 line. Narrow to_range to >=v1.4.0-alpha.0 so the documented downgrade invocation (--to-version v1.1.4) is not broken. Also add the missing trailing newline to the three case files. Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> --------- Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> Co-authored-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> |
||
|
|
2a70194930 |
feat: add per-column COMPRESSION option for SST columns (#9426)
* feat: add per-column COMPRESSION option for SST columns CREATE TABLE accepts COMPRESSION WITH (type = 'zstd', level = N, page_size = '<size>') on a column. The options are stored in the column metadata, carried through gRPC column options, shown by SHOW CREATE TABLE, and applied by the mito SST writer and bulk part encoder as parquet column properties. Unset fields keep the SST defaults. Nested types are rejected. A page size below the dictionary page limit also lowers the column's dictionary page limit: parquet sizes write batches by the dictionary budget and keeps that size after falling back to plain encoding, which would otherwise ignore the smaller data page limit. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: validate column compression options before persistence Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject compression overrides on metric logical tables Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
28697c9e55 |
feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode Give a datanode the ability to decode and execute a plan whose MergeScan carries a nested plan (e.g. an inner MergeScan over a build table), so a distributed sub-plan pushed to a datanode can itself fan out to other datanodes. This is the execution/codec capability only; it is not wired to any user-facing switch and produces no plan by itself. Codec (src/query/src/query_engine/default_serializer.rs): - MergeScanAwareSerializer keeps the request's SessionState and the engine catalog manager, and decodes a MergeScan payload's tables by (catalog, schema, table) name through the engine catalog rather than the region-aware request catalog list (which binds every table to the request's region and breaks on mismatched columns). Datanode (src/datanode): - New DatanodeRegionQueryHandler (region_query.rs) serves region queries for the MergeScan nodes of a received plan, resolving region leaders via PartitionRuleManager and dispatching to peer datanodes. - Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into the datanode's catalog/cache registry (with_dist_planner stays false: a datanode never rewrites its own plan). Bug fixes uncovered while wiring this up: - cache registry: register the table/table_info/table_name/table_route/ partition_info caches the datanode's catalog manager and partition rule manager require (previously missing -> panic on lookup). - default_serializer: decode_sub_plan bridge no longer drives a future on a current-thread runtime from within another executor (spawn a dedicated thread + runtime when not on a multi-thread runtime). Tested by query_engine::default_serializer unit tests: nested MergeScan round-trip and engine-catalog resolution with mismatched column names. No user-facing behavior change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep GreptimeDB function bindings after rebuilding decode state SessionStateBuilder::new_from_existing collects the function registry (HashMap) into a Vec and build() re-registers each entry; register_udf registers aliases before the function's own name and the Vec order is unstable. Rebuilding the session state to install MergeScanAwareSerializer after the GreptimeDB functions were registered could therefore rebind a name/alias (e.g. date_format, an alias of DataFusion's to_char) back to the built-in function. This path runs for every decoded plan, not only nested MergeScan ones. Extract the GreptimeDB function registration into register_greptime_functions and re-apply it after the final build of both the top-level decode state and the MergeScan payload state. Add regression tests asserting date_format stays bound to the GreptimeDB implementation on both paths. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): address review on nested MergeScan capability Three follow-ups from review of the nested MergeScan execution capability: 1. Layer the datanode caches (P2b). table_cache derives from table_info_cache + table_name_cache, and partition_info_cache derives from table_route_cache, but they were all registered in a single flat CacheRegistry whose invalidate() runs concurrently with no ordering. A derived cache could be invalidated and refilled from a base cache that had not been invalidated yet, leaving stale metadata. Build the base caches and the derived caches as separate layers of a LayeredCacheRegistry (reusing the existing layering mechanism) so base caches finish invalidating before derived ones. Both the production path (cmd) and the test scaffolding now build the same layered registry; the flat build_datanode_cache_registry is removed. 2. Don't let a nested MergeScan's internal region query compete for the concurrency permit (P2a). An outer query holds its permit until its stream is fully consumed, and a nested MergeScan makes the datanode issue region queries to peer datanodes as execution stages of that same query. If such an inner stage also took a permit, the outer query waiting on it would deadlock against its own held permit (or time out when max_concurrent_queries > 0). Mark inner DN->DN region queries as internal stages: QueryRequest gains an flag carried over the wire in the request header's query_context extensions (reserved key, not injectable via hints), and the region server skips acquiring a concurrency permit for internal-stage requests on both acquire sites. Outer queries still acquire normally. 3. Add an end-to-end capability test (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build Join(<local probe region>, MergeScan(build regions on both datanodes)) with mismatched column names and duplicate build keys, ship it to a datanode over a real TCP Flight endpoint, consume the stream fully and compare the row multiset against the frontend's reference join result, assert a real cross-datanode DoGet via an RPC counter, and assert an inner region failure fails the whole query. Also migrate the with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt date_format regression test to the updated DataFusion fork ScalarUDFImpl's trait bounds changed so its impl can no longer be downcast via as_any(); assert the function equals the GreptimeDB implementation instead, which to_char can never satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover the internal stage under the concurrency limit Add an end-to-end regression test for the nested MergeScan capability under max_concurrent_queries = 1: both datanodes' only permit is held by the outer queries' streams while their inner MergeScans fetch the build regions from the same datanodes over real Flight connections. The result consumption is bounded by a timeout, so a broken link anywhere in the chain (marking the inner request internal, carrying the marker in the request header, honoring it at the receiving acquire sites) fails the test instead of deadlocking. Also add a minimal with_datanode_options_override hook to the cluster builder, and fix the test module doc: the outer request is handed to RegionServer::handle_remote_read in process; only the inner region queries go over the wire. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover cold route loads under unrelated invalidations The table route cache uses InitStrategy::VersionChecked, whose version counter belongs to the whole CacheContainer rather than to a single table: any TableId invalidation bumps the shared version, so a cold load of one table's route retries when an unrelated table is invalidated concurrently. Document this as a known limitation on the region query handler, and pin the behavior with an integration test: run the nested plans while a background task invalidates an unrelated table id on every datanode, assert the results still match the expected multiset (retries only add work, they never return a stale route), and report the observed load amplification via the cache miss counter (2 loads without invalidations, 8 over three runs with 24 unrelated invalidations in this run). Also expose the datanode cache registries on the test cluster so a test can drive the local invalidation path that the heartbeat handler drives after a metasrv broadcast. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): trim nested plan decoder commentary Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: remove redundant nested scan support plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): decode nested MergeScan payloads asynchronously Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): avoid unnecessary extension name allocation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain extension decoding for manual session states Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datanode): keep nested scan query admission unchanged Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): honor query handler factories and trim nested scan fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): retain required cache error import Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): reuse registered state for nested scan decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): verify payload information schema table functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): retain nested scan runtime and cold-route guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): address nested scan review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(session): preserve read preference across region query hops Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(session): verify strict context rejection at Flow ingress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove redundant serializer error case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
3c0e2a8d55 |
feat: export Metric snapshots with packed Parquet objects (#9382)
* feat: export Metric snapshots with packed Parquet objects Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject empty packed export time ranges Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover cancellation during packed export I/O Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
19c127d9c3 |
feat: restore packed metric snapshots (#9250)
* test: cover snapshot parquet restore compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: restore packed metric snapshots Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: stream large packed parquet entries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: allow default S3 endpoint in packed restore test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: skip unconfigured S3 in packed restore test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use portable file URLs in packed restore fixtures Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate packed snapshot structure before restore Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve strict manifest decoding and verify fixture Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject packed layout on both database export paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: implement file size in coordinator test storage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use expect_err for rejected export layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
09a9d9088d |
feat(otlp): preserve trace v2 events and links as JSON (follow-up to #9192) (#9232)
feat(otlp): preserve trace v2 events and links as JSON Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
5ef8a46e3f |
feat(function): add mergeable binary average states (#9062)
* feat(function): add mergeable binary average states Signed-off-by: discord9 <discord9@163.com> * feat(function): expose avg_calc as an OSS scalar Signed-off-by: discord9 <discord9@163.com> * fix(function): borrow invalid AVG scalar argument type Signed-off-by: discord9 <discord9@163.com> * test(function): verify public AVG state SQL finalization Signed-off-by: discord9 <discord9@163.com> * fix(function): return canonical AVG1 state for empty window frames DataFusion's plain-aggregate window executor bypasses the accumulator and calls default_value() directly when a window frame contains no rows. create_udaf's SimpleAggregateUDF derives default_value from the return type, which yields SQL NULL for Binary output instead of the canonical AVG1 empty state, so empty frames and frames over only NULL inputs became observable differently (IS NULL, direct state saves, state comparisons). Register avg_state/avg_merge through a small AggregateUDFImpl (AvgUdaf) that keeps the existing accumulator and declares the canonical AVG1 empty state as default_value, restoring the documented contract. Add a unit test asserting default_value equals the empty accumulator's evaluate() and an sqlness case covering the empty-frame window. Signed-off-by: discord9 <discord9@outlook.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <discord9@outlook.com> Co-authored-by: discord9 <discord9@outlook.com> |
||
|
|
697cc5fa29 |
fix(json): fix JSONPath panic with jsonb 0.5.6 (follow-up to #9192) (#9228)
* fix(json): upgrade jsonb to fix unterminated JSONPath panic Signed-off-by: luofucong <luofc@foxmail.com> * fix(json): align integer extraction with JSON2 conversions Signed-off-by: luofucong <luofc@foxmail.com> * style: format jsonb dependency declaration Signed-off-by: luofucong <luofc@foxmail.com> * fix(otlp): report concrete unsupported JSONB type names Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
aacf04cf6e |
feat(otlp): add trace v2 ingestion with JSON2 attributes (#9192)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
f5428f8a6a |
test: regenerate expired TLS certificates for integration fixtures (#9196)
* test: regenerate expired TLS certificates for integration fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore root.srl serial file for TLS fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(ci): update compatibility test window to v1.2.1 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add proper TLS extensions to regenerated integration certificates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
|
9f6a79da30 |
fix: support native JSON2 row inserts over gRPC (#9145)
* fix: support native JSON2 row inserts over gRPC Signed-off-by: luofucong <luofc@foxmail.com> * test: cover unknown JSON2 schema compatibility Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
ba0f7acd93 |
feat(mito2): add opt-in byte-stream-split encoding for float SST fields (#9069)
* feat(mito2): add opt-in byte stream split encoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): correct float encoding checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): cover float SST encoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): compile float encoding tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): release parquet test writer Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): register float test primary key Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): verify BSS write lifecycles Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(metric-engine): verify BSS physical SST Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): verify bulk BSS lifecycle Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): compile bulk BSS test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): narrow bulk encoding constructors Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): accept generated float upgrade output Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): accept generated float downgrade output Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): narrow bulk encoding builder Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): add default versus BSS storage comparison Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): align BSS reader benchmarks with prior study Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): parse current read benchmark averages Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): retain default float encoding in direct SST fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): isolate BSS user SSTs and benchmark every file Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): record measured BSS storage and reader tradeoffs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): expose warm scan variability and evidence limits Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): clarify BSS baseline and storage measurement scope Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): model bounded mixed integer and fractional metric series Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): report bounded mixed BSS measurements and query regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): qualify timings affected by concurrent host builds Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): include float BSS comparison in default regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): omit unsupported float encoding option from baseline setup Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
6fa1023b7f |
feat(mito2): introduce TWCS active window compaction (#9011)
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows Split the single TWCS trigger_file_num into per-window-state thresholds: the active window keeps the existing trigger (default 4, legacy compaction.twcs.trigger_file_num stays a compatible alias), while inactive windows use a new trigger (default 2). Inactive windows additionally fall back from balanced L0-only/L1-only candidates to a progress-making unbalanced mixed candidate so historical windows can converge; the active window retains the row/byte balance guards. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): bound inactive TWCS window convergence by rewrite budget Inactive windows that cannot compact within one level previously either stayed stuck (a threshold-qualified but unbalanced level returned no candidate without trying any fallback) or fell back to a mixed merge with no balance checks at all, which could rewrite a huge compacted file to absorb tiny fresh files. Inactive windows now converge progressively: threshold-qualified balanced picks, sub-threshold balanced single-level picks, a mixed merge whose total rewrite must fit in the output file budget, and finally an L0-only merge without balance checks. Windows that qualify for none of these are left uncompacted, bounding write amplification. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover TWCS window trigger options in alter_table_options sqlness case Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and compaction.twcs.inactive_window.trigger_file_num end to end, including that setting the canonical active key removes the legacy compaction.twcs.trigger_file_num alias from the table options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): derive TWCS active window from the max-sequence file The active window was determined by the max event-time window among level-0 files. Between an L0 compaction removing its inputs and the next flush landing, level 0 is empty, so the active window transiently became None and every window fell back to the inactive rules - triggering full-window convergence merges during ongoing ingestion whose outputs are then superseded by new data. Flush and compaction outputs both inherit the max input sequence, so the file with the highest sequence across all levels always tracks the most recent write. Use its window as the active window, falling back to the previous L0-based rule when no file carries a sequence (legacy files). The new helper deliberately computes window keys with the assign_to_windows convention (truncate to seconds, then align up), because the result is compared against window keys produced there; the older ceil-based helper is kept unchanged for the legacy fallback path. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): add active-window L1 compaction safety trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(compat): cover TWCS active window options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): align TWCS window trigger validation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve database TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize newer compaction windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): repick serial compaction outputs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): configure inactive-window L1 trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): prioritize TWCS compaction candidates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): preserve TWCS trigger downgrade compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(sql): distinguish invalid database option values Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(meta): explain TWCS legacy key compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify active window trigger field Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): normalize TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): ignore ineligible files for active window Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark explicit TWCS options as overrides Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): normalize zero compaction output size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): validate database TWCS trigger values Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(store-api): collapse TWCS alias conflict condition Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
0d5d97e7cb |
chore(ci): update compatibility versions (#9075)
Signed-off-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com> Co-authored-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com> |