diff --git a/backport-review/v1.2-beta2/README.md b/backport-review/v1.2-beta2/README.md index 1277200a521..f471df36359 100644 --- a/backport-review/v1.2-beta2/README.md +++ b/backport-review/v1.2-beta2/README.md @@ -12,6 +12,8 @@ ## How to read the artifacts - **[Start here: canonical manual decision index](decisions/README.md)**: priority-ordered semantic concerns, exact differences, exclusions, compatibility impact, and reviewer file pointers. +- PR #8895 mapping: upstream merge `d7f1233f775d54380318d6ad9ff62504a7cbcff1`, head `6b8645bf584a04b45828c7e8d0fb19418db70d2e`; carrier `fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d`; see [#8895 raw evidence](range-diffs/pr-8895.txt) and [JSON2 decision coverage](decisions/json2-mergescan.md). +- PR #8901 mapping: upstream merge `b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73`, head `b450abd46e473a07cb1b42a9c7c62d0b0d551e17`; carrier `5a642091cd2b6185fbe6a08d6f76d8295955152f`; see [#8901 raw evidence](range-diffs/pr-8901.txt) and [JSON2 decision coverage](decisions/json2-mergescan.md). - PR #8825 mapping: upstream head `bf947141753a9ea554aa1ca31637fdd3ad1429f3`, merge `60c7bbcaf30a0e65ad7fa2b2b8ad0c1199a662b1`; carrier `d18430b874f9c8054747dbf688585cd7d2988e3e`; final head `40e02ff6b62ad3445f7035e437bd9e322c8af31b`. Requested by WenyXu in [Issue #8892 comment 5353700059](https://github.com/GreptimeTeam/greptimedb/issues/8892#issuecomment-5353700059). See [#8825 raw evidence](range-diffs/pr-8825.txt) and [event-context compatibility](decisions/event-context-compatibility.md). - `range-diffs/pr-.txt`: normally a final one-commit `git range-diff --creation-factor=100` from the upstream merged PR commit to the final aggregate carrier commit. These raw `.txt` files are **unabridged differences-only evidence**, not the complete shipping patch; `pr-8825.txt` is explicitly raw metadata/command evidence because its upstream PR-head object is unavailable locally. - `patch-id-verdict.tsv`: stable patch IDs and `exact`/`adapted` verdicts. Patch-ID equality means patch equivalence; it does not claim the cherry-pick had no conflict. @@ -23,7 +25,7 @@ ## Scope -The review scope has 33 explicit PRs: the prior 32 plus #8902. Official PR #8921 is now in the release base `2f5e97850e55d86c0ed9eff1719994c88b4450a0`, so it is base-present and is not represented as an aggregate carrier. #8672 and #8699 remain base-present as well. The other explicit PRs are represented by backport mappings; #8825 remains carried by `d18430b874f9c8054747dbf688585cd7d2988e3e`, and #8902 is appended at final head `40e02ff6b62ad3445f7035e437bd9e322c8af31b`. The dependency closure also includes full #8734 and a release-compatible slice of #8392. +The review scope has 36 explicit PRs: the original 31 plus Issue-requested #8825, #8921, #8895, #8901, and #8902. Official PR #8921 is now in the release base `2f5e97850e55d86c0ed9eff1719994c88b4450a0`, so it is base-present and is not represented as an aggregate carrier. #8672 and #8699 remain base-present as well. The other explicit PRs are represented by backport mappings; #8825 remains carried by `d18430b874f9c8054747dbf688585cd7d2988e3e`, and #8902 is appended at final head `40e02ff6b62ad3445f7035e437bd9e322c8af31b`. The dependency closure adds exactly two provenance relationships: full #8734 and a release-compatible slice of #8392, for 38 total provenance relationships. ## Mapping classes @@ -42,4 +44,6 @@ PR #8824 is accepted by explicit release-owner direction in this backport sessio The final review range is `2f5e97850e55d86c0ed9eff1719994c88b4450a0..40e02ff6b62ad3445f7035e437bd9e322c8af31b`, a linear 46-commit range. The superseded beta2 base `8eeff8b4417e6fb675a66cecbdba40060322785c` and endpoint `b7fdcb89568dc707c323f8c0a3b9413ec4e889ab` are historical provenance only; all active aggregate carrier fields in the TSV indexes are remapped to the rebased commits. Official #8921 is the release-base commit `2f5e97850e55d86c0ed9eff1719994c88b4450a0`, not a duplicate aggregate carrier. +The three explicit PRs #8672, #8699, and #8921 are base-present; the other explicit PRs have aggregate provenance. Thus the machine-readable indexes contain exactly 36 explicit PR rows plus exactly the two closure rows (#8734 and #8392): 38 unique provenance relationships. + PR #8902 is appended as final carrier `40e02ff6b62ad3445f7035e437bd9e322c8af31b`; its readable decision is [Dictionary group-by fast path](decisions/dictionary-groupby-fast-path.md), and its raw patch comparison is [#8902](range-diffs/pr-8902.txt). diff --git a/backport-review/v1.2-beta2/aggregate-commit-provenance.tsv b/backport-review/v1.2-beta2/aggregate-commit-provenance.tsv index 4d05ec30315..58170a84a5e 100644 --- a/backport-review/v1.2-beta2/aggregate-commit-provenance.tsv +++ b/backport-review/v1.2-beta2/aggregate-commit-provenance.tsv @@ -41,6 +41,6 @@ e61a6df6b2b110badd5755f68010b6439d69ac84 3351de9102b6418207e49c090054b71cc63205 9bf878f562518d30a412138ad4244d7d8e7867e6 668df91d76785b93a00f279916d2584646a56080 fix: adapt event context tests to release channel API 6595b5d7bf1f59587bf89676d420db7371cdc459 922a4f6141ea7ed051b9c891cb05bbcee0b51e69 fix: adapt frontend event context tests d18430b874f9c8054747dbf688585cd7d2988e3e 60c7bbcaf30a0e65ad7fa2b2b8ad0c1199a662b1 unavailable refactor: share query channel definition -fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d d18430b874f9c8054747dbf688585cd7d2988e3e (unclassified) feat: add JSON2 storage layout settings -5a642091cd2b6185fbe6a08d6f76d8295955152f fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d (unclassified) feat: add JSON2 physical layout primitives +fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d d7f1233f775d54380318d6ad9ff62504a7cbcff1 ba5de162f5dd269aa0081d8d97492c578fc37d05 feat: add JSON2 storage layout settings +5a642091cd2b6185fbe6a08d6f76d8295955152f b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73 a1011136c44b4f6f2520683739fbd845257c0a0e feat: add JSON2 physical layout primitives 40e02ff6b62ad3445f7035e437bd9e322c8af31b d99b0df374eec0329ac3f430e40b80a1e6c6963c; 1a91c76c615afec71239b84af7787a0505bef914 (absorbed) 8c3fca20ce058fd569d436c38bc96e41fc1adbca fix(query): restore columnar group-by for dictionary-encoded tags diff --git a/backport-review/v1.2-beta2/decisions/README.md b/backport-review/v1.2-beta2/decisions/README.md index feae8e5c13f..ce1a9facdcd 100644 --- a/backport-review/v1.2-beta2/decisions/README.md +++ b/backport-review/v1.2-beta2/decisions/README.md @@ -52,7 +52,11 @@ These entries are intentionally not given separate concern pages: stable patch-I | **Exact stable patch-ID** | #8777, #8832, #8787, #8849, #8859, #8889, #8833, #8600, #8808, #8709, #8522, #8739, #8772 | [1dd27e9](https://github.com/GreptimeTeam/greptimedb/commit/1dd27e9162ef5438e9f44b5a48a040f5c399227a), [e80f539](https://github.com/GreptimeTeam/greptimedb/commit/e80f539077616507b04df68a85a67f5da1c8f93e), [0b49947](https://github.com/GreptimeTeam/greptimedb/commit/0b499475dedfa462e9f951e0bb7b516011a195a0), [ab80858](https://github.com/GreptimeTeam/greptimedb/commit/ab808586525877230b9774f2ce29aca590152642), [824c5ac](https://github.com/GreptimeTeam/greptimedb/commit/824c5acfa7cf1865d2c20b77281c6b12e8e73de7), [c894a24](https://github.com/GreptimeTeam/greptimedb/commit/c894a2468c2bbb621bc26bf242046f1c60493460), [4ab29b2](https://github.com/GreptimeTeam/greptimedb/commit/4ab29b22f381156225cf9a50f73e5acc1ba8bbe8), [9c683fc](https://github.com/GreptimeTeam/greptimedb/commit/9c683fccfc11cd5db720b14c7adb22b7d734defd), [1a2e7b8](https://github.com/GreptimeTeam/greptimedb/commit/1a2e7b81679c602ae76f868b66974af9397d4341), [0b6c40e](https://github.com/GreptimeTeam/greptimedb/commit/0b6c40e7f465273ac424beca3475e82f4def7991), [7d68d8d](https://github.com/GreptimeTeam/greptimedb/commit/7d68d8dbe0600b86535293210ec70417e4b2b3d5), [be0985c](https://github.com/GreptimeTeam/greptimedb/commit/be0985cc39c648a35328bf9c340a8d32b814ddc6), [5e0a643](https://github.com/GreptimeTeam/greptimedb/commit/5e0a6436ee91fb4cb48a1a63564e019be02b3e22) (in the same order) | Patch-equivalent to each upstream PR; any cherry-pick conflict is a replay detail, not a semantic delta. | No separate page; see audit TSVs and raw range-diffs | | **Base-present** | #8672, #8699, #8921 | #8921 official release-base commit `2f5e97850e55d86c0ed9eff1719994c88b4450a0`; no aggregate carrier | Already present in beta2 before this aggregate; no backport patch ships for any of these PRs. | No separate page; see [`README.md`](../README.md) scope | -Exact entries with nearby context can be reviewed from [dependency and baseline adaptations](dependency-and-baseline-adaptations.md) and the [event context page](event-context-compatibility.md). The complete 33-explicit-PR mapping, plus dependency #8734 and slice #8392, is indexed in [`upstream-pr-map.tsv`](../upstream-pr-map.tsv), [`patch-id-verdict.tsv`](../patch-id-verdict.tsv), and [`aggregate-commit-provenance.tsv`](../aggregate-commit-provenance.tsv). +Exact entries with nearby context can be reviewed from [dependency and baseline adaptations](dependency-and-baseline-adaptations.md) and the [event context page](event-context-compatibility.md). The complete 36-explicit-PR mapping (including #8895 and #8901), plus dependency #8734 and slice #8392, is indexed in [`upstream-pr-map.tsv`](../upstream-pr-map.tsv), [`patch-id-verdict.tsv`](../patch-id-verdict.tsv), and [`aggregate-commit-provenance.tsv`](../aggregate-commit-provenance.tsv). + +## Final provenance scope + +The final review records 36 explicit Issue-requested PR relationships: the original 31 plus #8825, #8921, #8895, #8901, and #8902. Three explicit PRs—#8672, #8699, and #8921—are base-present. The dependency closure adds exactly two relationships: full #8734 and the release-compatible slice of #8392. The indexes therefore contain 38 unique provenance relationships total. The #8895 and #8901 mappings and complete upstream ranges are covered in [JSON2 merge/layout coverage](json2-mergescan.md) and [`manual-deltas.md`](../manual-deltas.md). ## Recorded verification context diff --git a/backport-review/v1.2-beta2/decisions/json2-mergescan.md b/backport-review/v1.2-beta2/decisions/json2-mergescan.md index e09f6180f10..bbe9afbaed8 100644 --- a/backport-review/v1.2-beta2/decisions/json2-mergescan.md +++ b/backport-review/v1.2-beta2/decisions/json2-mergescan.md @@ -39,3 +39,12 @@ Recorded CI status is documented in the [canonical CI status](README.md#canonica ## Raw audit evidence — unabridged differences only [`range-diffs/pr-8745.txt`](../range-diffs/pr-8745.txt) is an unabridged patch-to-patch range-diff, not the full shipping patch. + +## Additional JSON2 layout provenance + +The explicit Issue-requested JSON2 layout PRs are also represented in the machine-readable indexes and have complete raw range evidence: + +- **#8895**: upstream merge `d7f1233f775d54380318d6ad9ff62504a7cbcff1`, head `6b8645bf584a04b45828c7e8d0fb19418db70d2e`, complete upstream range `0a6a22033e3ebb4d6c7243874d840558a9eb7216^..6b8645bf584a04b45828c7e8d0fb19418db70d2e`, aggregate carrier `fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d`. The carrier folds the two upstream commits, retains JSON2 DDL/layout settings, and omits the upstream-only >=v1.3.0 compatibility case; it is `adapted`. +- **#8901**: upstream merge `b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73`, head `b450abd46e473a07cb1b42a9c7c62d0b0d551e17`, complete upstream range `6e637fbe58f36643fea40f38d5ff352fe00254f9^..b450abd46e473a07cb1b42a9c7c62d0b0d551e17`, aggregate carrier `5a642091cd2b6185fbe6a08d6f76d8295955152f`. The carrier folds the three upstream commits and is `exact` by stable patch ID. + +These are part of the 36 explicit Issue-requested PR relationships. Together with base-present #8672/#8699/#8921 and the two closure relationships (#8734 full, #8392 release-compatible slice), the review contains 38 provenance relationships. diff --git a/backport-review/v1.2-beta2/manual-deltas.md b/backport-review/v1.2-beta2/manual-deltas.md index 596dddfa06d..8b7129b33cd 100644 --- a/backport-review/v1.2-beta2/manual-deltas.md +++ b/backport-review/v1.2-beta2/manual-deltas.md @@ -137,3 +137,19 @@ The canonical reviewer index is [`decisions/README.md`](decisions/README.md). Th - DataFusion `452cb4b` advances to linear successor `4c8a6bf`, preserving #8842 and adding `DictionaryGroupValuesColumn`. - GreptimeDB adds `tests-integration/tests/dict_groupby_sst.rs` plus module registration. The test writes 1,200 rows, flushes append-mode flat SST, and exactly verifies six ordered tuples. - This integration test is not direct fast-path proof: AggregateExec EXPLAIN/metrics do not expose GroupValues identity. Direct `DictionaryGroupValuesColumn` coverage is in the pinned DataFusion fork tests. + +### #8895 — adapted, two-commit upstream range folded into one carrier +- Upstream PR range: `0a6a22033e3ebb4d6c7243874d840558a9eb7216` followed by `6b8645bf584a04b45828c7e8d0fb19418db70d2e`; PR head: `6b8645bf584a04b45828c7e8d0fb19418db70d2e`; merge: `d7f1233f775d54380318d6ad9ff62504a7cbcff1`. +- Aggregate carrier: `fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d`. +- The final carrier folds the complete upstream PR range, retains the JSON2 DDL/layout-settings behavior, and omits the upstream-only >=v1.3.0 compatibility-test case. Stable patch IDs therefore differ and the verdict is `adapted`. +- Complete range-diff evidence, including endpoint metadata and the exact generation command, is [`range-diffs/pr-8895.txt`](range-diffs/pr-8895.txt). + +### #8901 — exact patch-id, three-commit upstream range folded into one carrier +- Upstream PR range: `6e637fbe58f36643fea40f38d5ff352fe00254f9`, `f2e3d6d6e16c24fdede441cf54ff1c04a3effd91`, and `b450abd46e473a07cb1b42a9c7c62d0b0d551e17`; PR head: `b450abd46e473a07cb1b42a9c7c62d0b0d551e17`; merge: `b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73`. +- Aggregate carrier: `5a642091cd2b6185fbe6a08d6f76d8295955152f`. +- The final carrier folds the complete upstream PR range and retains the JSON2 v2 physical-layout primitives and dependency updates. Its stable patch ID equals the upstream merge patch ID; the changed subject is an aggregate adaptation, so the verdict records `exact` patch equivalence. +- Complete range-diff evidence, including endpoint metadata and the exact generation command, is [`range-diffs/pr-8901.txt`](range-diffs/pr-8901.txt). + +## Final provenance scope + +The final review records 36 explicit Issue-requested PR relationships: the original 31 plus #8825, #8921, #8895, #8901, and #8902. Of those, #8672, #8699, and #8921 are base-present. The dependency closure contributes exactly two additional relationships—full #8734 and the release-compatible slice of #8392—for 38 unique provenance relationships total. diff --git a/backport-review/v1.2-beta2/patch-id-verdict.tsv b/backport-review/v1.2-beta2/patch-id-verdict.tsv index 0babf23eefe..59cc954951c 100644 --- a/backport-review/v1.2-beta2/patch-id-verdict.tsv +++ b/backport-review/v1.2-beta2/patch-id-verdict.tsv @@ -30,3 +30,5 @@ 8392 cabc2f6cc667291d7bc3dcbb8f9cc82e27bd93a0 f9135380b5591b82c9ec5bbf1706459672bd72db adapted 288b8990db827c0d0f4e11a0f20e0f77fdbf4991 49fecedeb0ec2f57c348307f5fdcfc558de42c59 8825 bf947141753a9ea554aa1ca31637fdd3ad1429f3 d18430b874f9c8054747dbf688585cd7d2988e3e adapted unavailable unavailable 8902 d99b0df374eec0329ac3f430e40b80a1e6c6963c 40e02ff6b62ad3445f7035e437bd9e322c8af31b adapted bdc71e83b0147f1d94f1db4cbe80d0dbf8a6aa58 8c3fca20ce058fd569d436c38bc96e41fc1adbca +8895 d7f1233f775d54380318d6ad9ff62504a7cbcff1 fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d adapted e26d83fb1aab2ee25d7e9d74ba1c42a163507c0d ba5de162f5dd269aa0081d8d97492c578fc37d05 +8901 b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73 5a642091cd2b6185fbe6a08d6f76d8295955152f exact a1011136c44b4f6f2520683739fbd845257c0a0e a1011136c44b4f6f2520683739fbd845257c0a0e diff --git a/backport-review/v1.2-beta2/range-diffs/pr-8895.txt b/backport-review/v1.2-beta2/range-diffs/pr-8895.txt new file mode 100644 index 00000000000..e9a97e112af --- /dev/null +++ b/backport-review/v1.2-beta2/range-diffs/pr-8895.txt @@ -0,0 +1,330 @@ +PR #8895 +Upstream PR range: 0a6a22033e3ebb4d6c7243874d840558a9eb7216^..6b8645bf584a04b45828c7e8d0fb19418db70d2e +Aggregate range: fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d^..fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d +Upstream PR merge commit: d7f1233f775d54380318d6ad9ff62504a7cbcff1 +Upstream PR head: 6b8645bf584a04b45828c7e8d0fb19418db70d2e +Aggregate carrier: fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d +Endpoint metadata: the upstream PR contains 0a6a22033e3ebb4d6c7243874d840558a9eb7216 and 6b8645bf584a04b45828c7e8d0fb19418db70d2e; GitHub records d7f1233f775d54380318d6ad9ff62504a7cbcff1 as its merge commit. Beta2 carries the requested behavior in one final carrier. +Adaptation/folding notes: the carrier folds both upstream PR commits into one commit, changes the subject to the exact aggregate subject, and omits the upstream-only compatibility-test case (which targets >=v1.3.0). The JSON2 layout-settings implementation is retained; this is an adapted backport, not a claim of exact commit identity. +Command: git range-diff --creation-factor=100 0a6a22033e3ebb4d6c7243874d840558a9eb7216^..6b8645bf584a04b45828c7e8d0fb19418db70d2e fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d^..fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d + +1: 0a6a22033 ! 1: fed5c80d7 refactor(json2): add JSON2 storage layout settings + @@ + ## Metadata ## + -Author: luofucong + +Author: discord9 + + ## Commit message ## + - refactor(json2): add JSON2 storage layout settings + + feat: add JSON2 storage layout settings + + - Signed-off-by: luofucong + + Signed-off-by: discord9 + + ## src/datatypes/src/error.rs ## + @@ src/datatypes/src/error.rs: pub enum Error { + @@ src/datatypes/src/json.rs: pub struct JsonContext<'a> { + + }) + + } + + + -+ #[cfg(test)] + -+ fn new(type_hints: Vec) -> Self { + -+ Self { + -+ type_hints, + -+ max_auto_expanded_paths: None, + -+ } + -+ } + -+ + + pub fn type_hints(&self) -> &[JsonTypeHint] { + + &self.type_hints + + } + @@ src/datatypes/src/json.rs: impl JsonSettings { + + .fail(); + + } + + }; + -+ insert_type_hint(&mut object, &hint.path, data_type)?; + ++ validate_type_hint(&mut object, &hint.path, data_type)?; + + } + + Ok(()) + +} + + + -+fn insert_type_hint( + ++fn validate_type_hint( + + object: &mut JsonObjectType, + + path: &[String], + + data_type: JsonNativeType, + @@ src/datatypes/src/json.rs: impl JsonSettings { + + } + + .fail(); + + }; + -+ insert_type_hint(child, path, data_type) + ++ validate_type_hint(child, path, data_type) + +} + + + impl<'a> JsonContext<'a> { + @@ src/datatypes/src/json.rs: impl JsonSettings { + @@ src/datatypes/src/json.rs: mod tests { + } + + + #[test] + +- fn test_json_settings_forward_compatibility() { + ++ fn test_json_settings_forward_compatibility() + ++ -> std::result::Result<(), Box> { + + let json_str = r#"{ + + "type_hints": [ + + { + +@@ src/datatypes/src/json.rs: mod tests { + + ] + + }"#; + + + +- let deserialized = serde_json::from_str::(json_str).unwrap(); + ++ let deserialized = serde_json::from_str::(json_str)?; + + + +- assert_eq!( + +- deserialized, + +- JsonSettings::new(vec![ + ++ let expected = JsonSettings::try_new( + ++ vec![ + + JsonTypeHint { + + path: vec!["user".to_string(), "age".to_string()], + + data_type: ConcreteDataType::int64_datatype(), + +@@ src/datatypes/src/json.rs: mod tests { + + default_constraint: None, + + inverted_index: false, + + }, + +- ]) + +- ); + ++ ], + ++ None, + ++ )?; + ++ assert_eq!(deserialized, expected); + ++ Ok(()) + + } + + + #[test] + - fn test_json_settings_ser_de() { + - let settings = JsonSettings::new(vec![ + @@ src/datatypes/src/json.rs: mod tests { + + ); + } + + + #[test] + +@@ src/datatypes/src/json.rs: mod tests { + + } + + + + #[test] + +- fn test_encode_json_respects_type_hint() { + +- let settings = JsonSettings::new(vec![JsonTypeHint { + ++ fn test_encode_json_respects_type_hint() -> Result<()> { + ++ let type_hints = vec![JsonTypeHint { + + path: vec!["age".to_string()], + + data_type: ConcreteDataType::int64_datatype(), + + nullable: false, + + default_constraint: None, + + inverted_index: false, + +- }]); + ++ }]; + ++ let settings = JsonSettings::try_new(type_hints, None)?; + + + + let result = settings + + .encode(json!({ + +@@ src/datatypes/src/json.rs: mod tests { + + })) + + .unwrap_err(); + + assert!(err.to_string().contains("does not match JSON2 type hint")); + ++ Ok(()) + + } + + + + #[test] + +- fn test_encode_json_respects_unsigned_type_hint() { + +- let settings = JsonSettings::new(vec![JsonTypeHint { + ++ fn test_encode_json_respects_unsigned_type_hint() -> Result<()> { + ++ let type_hints = vec![JsonTypeHint { + + path: vec!["count".to_string()], + + data_type: ConcreteDataType::uint64_datatype(), + + nullable: false, + + default_constraint: None, + + inverted_index: false, + +- }]); + ++ }]; + ++ let settings = JsonSettings::try_new(type_hints, None)?; + + + + let result = settings + + .encode(json!({ + +@@ src/datatypes/src/json.rs: mod tests { + + })) + + .unwrap_err(); + + assert!(err.to_string().contains("does not match JSON2 type hint")); + ++ Ok(()) + + } + + + + #[test] + +- fn test_encode_json_fills_missing_type_hint_with_default() { + +- let settings = JsonSettings::new(vec![JsonTypeHint { + ++ fn test_encode_json_fills_missing_type_hint_with_default() -> Result<()> { + ++ let type_hints = vec![JsonTypeHint { + + path: vec!["user".to_string(), "age".to_string()], + + data_type: ConcreteDataType::int64_datatype(), + + nullable: false, + + default_constraint: Some(ColumnDefaultConstraint::Value(Value::Int64(7))), + + inverted_index: false, + +- }]); + ++ }]; + ++ let settings = JsonSettings::try_new(type_hints, None)?; + + + + let result = settings + + .encode(json!({})) + +@@ src/datatypes/src/json.rs: mod tests { + + panic!("Expected user Struct value"); + + }; + + assert_eq!(struct_field_value(user, "age"), &Value::Int64(7)); + ++ Ok(()) + + } + + + + #[test] + +- fn test_encode_json_fills_missing_nullable_type_hint_with_null() { + +- let settings = JsonSettings::new(vec![JsonTypeHint { + ++ fn test_encode_json_fills_missing_nullable_type_hint_with_null() -> Result<()> { + ++ let type_hints = vec![JsonTypeHint { + + path: vec!["user".to_string(), "name".to_string()], + + data_type: ConcreteDataType::string_datatype(), + + nullable: true, + + default_constraint: None, + + inverted_index: false, + +- }]); + ++ }]; + ++ let settings = JsonSettings::try_new(type_hints, None)?; + + + + let result = settings + + .encode(json!({ "user": {} })) + +@@ src/datatypes/src/json.rs: mod tests { + + panic!("Expected user Struct value"); + + }; + + assert_eq!(struct_field_value(user, "name"), &Value::Null); + ++ Ok(()) + + } + + + + #[test] + +- fn test_encode_json_rejects_missing_non_null_type_hint() { + +- let settings = JsonSettings::new(vec![JsonTypeHint { + ++ fn test_encode_json_rejects_missing_non_null_type_hint() -> Result<()> { + ++ let type_hints = vec![JsonTypeHint { + + path: vec!["user".to_string(), "age".to_string()], + + data_type: ConcreteDataType::int64_datatype(), + + nullable: false, + + default_constraint: None, + + inverted_index: false, + +- }]); + ++ }]; + ++ let settings = JsonSettings::try_new(type_hints, None)?; + + + + let err = settings.encode(json!({})).unwrap_err(); + + assert!( + + err.to_string() + + .contains("missing non-null JSON2 type hint path user.age") + + ); + ++ Ok(()) + + } + + + + #[test] + +- fn test_encode_json_merges_missing_type_hint_prefix() { + +- let settings = JsonSettings::new(vec![ + ++ fn test_encode_json_merges_missing_type_hint_prefix() -> Result<()> { + ++ let type_hints = vec![ + + JsonTypeHint { + + path: vec!["user".to_string(), "age".to_string()], + + data_type: ConcreteDataType::int64_datatype(), + +@@ src/datatypes/src/json.rs: mod tests { + + ))), + + inverted_index: false, + + }, + +- ]); + ++ ]; + ++ let settings = JsonSettings::try_new(type_hints, None)?; + + + +- let result = settings + +- .encode(json!({})) + +- .unwrap() + +- .into_json_inner() + +- .unwrap(); + ++ let result = settings.encode(json!({}))?.into_json_inner().unwrap(); + + + + let Value::Struct(root) = result else { + + panic!("Expected Struct value"); + +@@ src/datatypes/src/json.rs: mod tests { + + struct_field_value(user, "name"), + + &Value::String("unknown".into()) + + ); + ++ Ok(()) + + } + + + #[test] + + ## src/sql/src/error.rs ## + @@ src/sql/src/statements/create.rs: AS SELECT number FROM numbers_input where numb + }; + let result = extensions.build_vector_index_options(); + assert!(result.is_ok()); + - + - ## tests/compatibility/cases/json2_storage_layout_settings/case.toml (new) ## + -@@ + -+name = "json2_storage_layout_settings" + -+reason = "Verify JSON2 storage layout settings in Arrow extension metadata survive restarts and upgrades." + -+introduced_by = "PR #8895" + -+topologies = ["distributed", "standalone"] + -+from_range = [">=v1.3.0"] + -+to_range = [">=v1.3.0"] + -+features = ["json2", "table", "metadata", "read"] + -+owner = "query" + - + - ## tests/compatibility/cases/json2_storage_layout_settings/setup.sql (new) ## + -@@ + -+CREATE TABLE t_json2_storage_layout_settings ( + -+ ts TIMESTAMP TIME INDEX, + -+ j JSON2(max_auto_expanded_paths = 0) + -+) WITH ( + -+ 'append_mode' = 'true' + -+); + -+ + -+INSERT INTO t_json2_storage_layout_settings VALUES + -+ ('2026-08-17 00:00:00+0000', '{"a": 1}'); + -+ + -+ADMIN FLUSH_TABLE('t_json2_storage_layout_settings'); + - + - ## tests/compatibility/cases/json2_storage_layout_settings/verify.result (new) ## + -@@ + -+SELECT j.a AS a + -+FROM t_json2_storage_layout_settings; + -+ + -++---+ + -+| a | + -++---+ + -+| 1 | + -++---+ + -+ + -+SHOW CREATE TABLE t_json2_storage_layout_settings; + -+ + -++---------------------------------+----------------------------------------------------------------+ + -+| Table | Create Table | + -++---------------------------------+----------------------------------------------------------------+ + -+| t_json2_storage_layout_settings | CREATE TABLE IF NOT EXISTS "t_json2_storage_layout_settings" ( | + -+| | "ts" TIMESTAMP(3) NOT NULL, | + -+| | "j" JSON2( | + -+| | max_auto_expanded_paths = 0 | + -+| | ) NULL, | + -+| | TIME INDEX ("ts") | + -+| | ) | + -+| | | + -+| | ENGINE=mito | + -+| | WITH( | + -+| | append_mode = 'true' | + -+| | ) | + -++---------------------------------+----------------------------------------------------------------+ + - + - ## tests/compatibility/cases/json2_storage_layout_settings/verify.sql (new) ## + -@@ + -+SELECT j.a AS a + -+FROM t_json2_storage_layout_settings; + -+ + -+SHOW CREATE TABLE t_json2_storage_layout_settings; +2: 6b8645bf5 < -: --------- resolve PR comments diff --git a/backport-review/v1.2-beta2/range-diffs/pr-8901.txt b/backport-review/v1.2-beta2/range-diffs/pr-8901.txt new file mode 100644 index 00000000000..64e5369f048 --- /dev/null +++ b/backport-review/v1.2-beta2/range-diffs/pr-8901.txt @@ -0,0 +1,215 @@ +PR #8901 +Upstream PR range: 6e637fbe58f36643fea40f38d5ff352fe00254f9^..b450abd46e473a07cb1b42a9c7c62d0b0d551e17 +Aggregate range: 5a642091cd2b6185fbe6a08d6f76d8295955152f^..5a642091cd2b6185fbe6a08d6f76d8295955152f +Upstream PR merge commit: b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73 +Upstream PR head: b450abd46e473a07cb1b42a9c7c62d0b0d551e17 +Aggregate carrier: 5a642091cd2b6185fbe6a08d6f76d8295955152f +Endpoint metadata: the upstream PR contains 6e637fbe58f36643fea40f38d5ff352fe00254f9, f2e3d6d6e16c24fdede441cf54ff1c04a3effd91, and b450abd46e473a07cb1b42a9c7c62d0b0d551e17; GitHub records b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73 as its merge commit. Beta2 carries the requested behavior in one final carrier. +Adaptation/folding notes: the carrier folds all three upstream PR commits into one commit and changes the subject to the exact aggregate subject. The JSON2 v2 physical-layout primitives and dependency updates are retained; release lock/context resolution is represented by the final carrier. This is an exact stable patch-ID replay with a folded commit message. +Command: git range-diff --creation-factor=100 6e637fbe58f36643fea40f38d5ff352fe00254f9^..b450abd46e473a07cb1b42a9c7c62d0b0d551e17 5a642091cd2b6185fbe6a08d6f76d8295955152f^..5a642091cd2b6185fbe6a08d6f76d8295955152f + +1: 6e637fbe5 ! 1: 5a642091c refactor(json2): add JSON2 v2 physical layout primitives + @@ + ## Metadata ## + -Author: luofucong + +Author: discord9 + + ## Commit message ## + - refactor(json2): add JSON2 v2 physical layout primitives + + feat: add JSON2 physical layout primitives + + - Signed-off-by: luofucong + + Signed-off-by: discord9 + + ## Cargo.lock ## + @@ Cargo.lock: dependencies = [ + @@ src/datatypes/Cargo.toml: jsonb.workspace = true + serde.workspace = true + + ## src/datatypes/src/extension/json.rs ## + -@@ + - use std::collections::HashMap; + - use std::sync::Arc; + - + -+#[cfg(test)] + -+use arrow_schema::FieldRef; + +@@ src/datatypes/src/extension/json.rs: use std::sync::Arc; + use arrow_schema::extension::{ + EXTENSION_TYPE_METADATA_KEY, EXTENSION_TYPE_NAME_KEY, ExtensionType, + }; + - use arrow_schema::{ArrowError, DataType, Field}; + -+#[cfg(test)] + +-use arrow_schema::{ArrowError, DataType, Field}; + ++use arrow_schema::{ArrowError, DataType, Field, FieldRef}; + +use parquet_variant_compute::VariantType; + use serde::{Deserialize, Serialize}; + -use snafu::ResultExt; + -+#[cfg(test)] + -+use snafu::OptionExt; + +use snafu::{ResultExt, ensure}; + + +use crate::error::InvalidJson2LayoutSnafu; + @@ src/datatypes/src/extension/json.rs + const LEGACY_JSON_STRUCTURE_SETTINGS_KEY: &str = "json_structure_settings"; + +const JSON2_LAYOUT_V1: u8 = 1; + +const JSON2_LAYOUT_V2: u8 = 2; + -+ + + + +-#[derive(Debug, Clone, Serialize, Deserialize, Default)] + +/// Parsed physical layout of a JSON2 Arrow root field. + +#[derive(Debug, Clone, PartialEq, Eq)] + +pub struct Json2PhysicalLayout { + @@ src/datatypes/src/extension/json.rs + +} + + + +impl Json2PhysicalLayout { + -+ /// Parses and validates the JSON2 layout version from a root field. + ++ /// Parses the JSON2 layout version from a root field. + ++ /// + ++ /// The version is read from the extension metadata; missing metadata + ++ /// defaults to V1. + ++ /// + ++ /// Supported versions: + ++ /// - V1 + ++ /// - V2 + ++ /// + ++ /// Errors when the field is not a JSON2 extension or the version is not + ++ /// one of the supported versions. + + pub fn try_from_root(field: &Field) -> crate::error::Result { + + ensure!( + + is_json2_extension_type(field), + @@ src/datatypes/src/extension/json.rs + +} + + + +/// Returns the remainder field of a JSON2 v2 root. + -+#[cfg(test)] + -+fn json2_remainder_field(field: &Field) -> crate::error::Result<&FieldRef> { + -+ if !Json2PhysicalLayout::try_from_root(field)?.is_version_2() { + -+ return InvalidJson2LayoutSnafu { + -+ reason: format!("JSON2 root '{}' is not layout v2", field.name()), + -+ } + -+ .fail(); + ++pub(crate) fn json2_remainder_field(field: &Field) -> crate::error::Result> { + ++ let layout = Json2PhysicalLayout::try_from_root(field)?; + ++ if !layout.is_version_2() { + ++ return Ok(None); + + } + - + --#[derive(Debug, Clone, Serialize, Deserialize, Default)] + -+ let DataType::Struct(fields) = field.data_type() else { + -+ return InvalidJson2LayoutSnafu { + -+ reason: format!( + -+ "expecting the Struct datatype, actual: '{}'", + -+ field.data_type(), + -+ ), + -+ } + -+ .fail(); + ++ + ++ let remainder = if let DataType::Struct(fields) = field.data_type() { + ++ fields + ++ .iter() + ++ .find(|x| x.name() == JSON2_REMAINDER_FIELD_NAME) + ++ } else { + ++ None + + }; + -+ let remainder = fields + -+ .iter() + -+ .find(|x| x.name() == JSON2_REMAINDER_FIELD_NAME) + -+ .context(InvalidJson2LayoutSnafu { + -+ reason: "remainder field not found", + ++ + ++ if let Some(remainder) = remainder { + ++ let _ = remainder.try_extension_type::().map_err(|e| { + ++ InvalidJson2LayoutSnafu { + ++ reason: e.to_string(), + ++ } + ++ .build() + + })?; + -+ let _ = remainder.try_extension_type::().map_err(|e| { + -+ InvalidJson2LayoutSnafu { + -+ reason: e.to_string(), + -+ } + -+ .build() + -+ })?; + ++ } + + Ok(remainder) + +} + + + @@ src/datatypes/src/extension/json.rs: mod tests { + + JsonSettings::default(), + + )))); + + assert!(Json2PhysicalLayout::try_from_root(&v2)?.is_version_2()); + -+ let remainder = json2_remainder_field(&v2)?; + ++ let remainder = json2_remainder_field(&v2)?.unwrap(); + + assert_eq!(JSON2_REMAINDER_FIELD_NAME, remainder.name()); + + Ok(()) + + } + @@ src/datatypes/src/extension/json.rs: mod tests { + + Json2ExtensionType::new(Arc::new(JsonMetadata::new_v2(JsonSettings::default()))); + + let missing = Field::new("data", DataType::Struct(Fields::empty()), true) + + .with_extension_type(metadata.clone()); + -+ assert!(json2_remainder_field(&missing).is_err()); + ++ assert!(json2_remainder_field(&missing).is_ok_and(|x| x.is_none())); + + + + let invalid = Field::new( + + "data", + @@ src/datatypes/src/vectors/json/array.rs: use crate::data_type::ConcreteDataType; + use crate::error::{ + AlignJsonArraySnafu, ArrowComputeSnafu, InvalidJsonSnafu, InvalidJsonbSnafu, Result, + }; + -+use crate::extension::json::{JSON2_REMAINDER_FIELD_NAME, Json2PhysicalLayout}; + ++use crate::extension::json::{JSON2_REMAINDER_FIELD_NAME, json2_remainder_field}; + use crate::json::value::{decode_json_variant, encode_serde_json_as_jsonb}; + use crate::prelude::{DataType as _, Value as GreptimeValue}; + use crate::value::{ListValue, StructValue}; + @@ src/datatypes/src/vectors/json/array.rs: impl JsonArray<'_> { + } + + + /// Projects a physical JSON2 array to a logical query type. + -+ pub fn project_json2(&self, field: &Field, target: &DataType) -> Result { + -+ let layout = Json2PhysicalLayout::try_from_root(field)?; + -+ if !layout.is_version_2() { + -+ return self.project_to(target); + -+ } + -+ + -+ let has_remainder = match field.data_type() { + -+ DataType::Struct(fields) => fields + -+ .iter() + -+ .any(|x| x.name() == JSON2_REMAINDER_FIELD_NAME), + -+ _ => false, + -+ }; + -+ if !has_remainder { + -+ return self.project_to(target); + ++ /// + ++ /// TODO(LFC) Supersede `project_to_v2` to `project_to`. + ++ pub fn project_to_v2(&self, field: &Field, target: &DataType) -> Result { + ++ if json2_remainder_field(field)?.is_some() { + ++ project_json_values(self.json2_values()?, target) + ++ } else { + ++ self.project_to(target) + + } + -+ + -+ project_json_values(self.json2_values()?, target) + + } + + + + fn json2_values(&self) -> Result> { + @@ src/datatypes/src/vectors/json/array.rs: mod test { + + ] + + .into(), + + ); + -+ let projected = JsonArray::from(&array).project_json2(&field, &target)?; + ++ let projected = JsonArray::from(&array).project_to_v2(&field, &target)?; + + assert_eq!( + + json!({"cold": 1, "count": 42}), + + JsonArray::from(&projected).try_get_value(0)? + @@ src/datatypes/src/vectors/json/array.rs: mod test { + + Json2ExtensionType::new(Arc::new(JsonMetadata::new_v2(JsonSettings::default()))), + + ); + + + -+ let projected = JsonArray::from(&array).project_json2(&field, field.data_type())?; + ++ let projected = JsonArray::from(&array).project_to_v2(&field, field.data_type())?; + + assert!(Arc::ptr_eq(&array, &projected)); + + Ok(()) + + } +2: f2e3d6d6e < -: --------- resolve PR comments +3: b450abd46 < -: --------- fix ci diff --git a/backport-review/v1.2-beta2/upstream-pr-map.tsv b/backport-review/v1.2-beta2/upstream-pr-map.tsv index 2027a4c3cf1..8e25d5d4f4d 100644 --- a/backport-review/v1.2-beta2/upstream-pr-map.tsv +++ b/backport-review/v1.2-beta2/upstream-pr-map.tsv @@ -34,3 +34,5 @@ 8825 bf947141753a9ea554aa1ca31637fdd3ad1429f3 MERGED https://github.com/GreptimeTeam/greptimedb/pull/8825 refactor: share query channel definition 8902 8d887ddd00d67699d0fb873cda11eb50b5771555 MERGED https://github.com/GreptimeTeam/greptimedb/pull/8902 fix(query): restore columnar group-by for dictionary-encoded tags (PR head 1a91c76c615afec71239b84af7787a0505bef914; range starts d99b0df374eec0329ac3f430e40b80a1e6c6963c) 8921 2f5e97850e55d86c0ed9eff1719994c88b4450a0 BASE-PRESENT (official release base) https://github.com/GreptimeTeam/greptimedb/pull/8921 fix: clear pooled Prometheus Remote Write decoder state +8895 d7f1233f775d54380318d6ad9ff62504a7cbcff1 MERGED https://github.com/GreptimeTeam/greptimedb/pull/8895 refactor(json2): support JSON2 storage layout settings in DDL +8901 b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73 MERGED https://github.com/GreptimeTeam/greptimedb/pull/8901 refactor(json2): add JSON2 v2 physical layout primitives