mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-15 00:12:29 +00:00
docs: complete JSON2 provenance in beta2 review artifacts
Signed-off-by: discord9 <discord9@163.com>
This commit is contained in:
discord9
@@ -12,6 +12,8 @@
|
||||
## How to read the artifacts
|
||||
|
||||
- **[Start here: canonical manual decision index](decisions/README.md)**: priority-ordered semantic concerns, exact differences, exclusions, compatibility impact, and reviewer file pointers.
|
||||
- PR #8895 mapping: upstream merge `d7f1233f775d54380318d6ad9ff62504a7cbcff1`, head `6b8645bf584a04b45828c7e8d0fb19418db70d2e`; carrier `fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d`; see [#8895 raw evidence](range-diffs/pr-8895.txt) and [JSON2 decision coverage](decisions/json2-mergescan.md).
|
||||
- PR #8901 mapping: upstream merge `b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73`, head `b450abd46e473a07cb1b42a9c7c62d0b0d551e17`; carrier `5a642091cd2b6185fbe6a08d6f76d8295955152f`; see [#8901 raw evidence](range-diffs/pr-8901.txt) and [JSON2 decision coverage](decisions/json2-mergescan.md).
|
||||
- PR #8825 mapping: upstream head `bf947141753a9ea554aa1ca31637fdd3ad1429f3`, merge `60c7bbcaf30a0e65ad7fa2b2b8ad0c1199a662b1`; carrier `d18430b874f9c8054747dbf688585cd7d2988e3e`; final head `40e02ff6b62ad3445f7035e437bd9e322c8af31b`. Requested by WenyXu in [Issue #8892 comment 5353700059](https://github.com/GreptimeTeam/greptimedb/issues/8892#issuecomment-5353700059). See [#8825 raw evidence](range-diffs/pr-8825.txt) and [event-context compatibility](decisions/event-context-compatibility.md).
|
||||
- `range-diffs/pr-<number>.txt`: normally a final one-commit `git range-diff --creation-factor=100` from the upstream merged PR commit to the final aggregate carrier commit. These raw `.txt` files are **unabridged differences-only evidence**, not the complete shipping patch; `pr-8825.txt` is explicitly raw metadata/command evidence because its upstream PR-head object is unavailable locally.
|
||||
- `patch-id-verdict.tsv`: stable patch IDs and `exact`/`adapted` verdicts. Patch-ID equality means patch equivalence; it does not claim the cherry-pick had no conflict.
|
||||
@@ -23,7 +25,7 @@
|
||||
|
||||
## Scope
|
||||
|
||||
The review scope has 33 explicit PRs: the prior 32 plus #8902. Official PR #8921 is now in the release base `2f5e97850e55d86c0ed9eff1719994c88b4450a0`, so it is base-present and is not represented as an aggregate carrier. #8672 and #8699 remain base-present as well. The other explicit PRs are represented by backport mappings; #8825 remains carried by `d18430b874f9c8054747dbf688585cd7d2988e3e`, and #8902 is appended at final head `40e02ff6b62ad3445f7035e437bd9e322c8af31b`. The dependency closure also includes full #8734 and a release-compatible slice of #8392.
|
||||
The review scope has 36 explicit PRs: the original 31 plus Issue-requested #8825, #8921, #8895, #8901, and #8902. Official PR #8921 is now in the release base `2f5e97850e55d86c0ed9eff1719994c88b4450a0`, so it is base-present and is not represented as an aggregate carrier. #8672 and #8699 remain base-present as well. The other explicit PRs are represented by backport mappings; #8825 remains carried by `d18430b874f9c8054747dbf688585cd7d2988e3e`, and #8902 is appended at final head `40e02ff6b62ad3445f7035e437bd9e322c8af31b`. The dependency closure adds exactly two provenance relationships: full #8734 and a release-compatible slice of #8392, for 38 total provenance relationships.
|
||||
|
||||
## Mapping classes
|
||||
|
||||
@@ -42,4 +44,6 @@ PR #8824 is accepted by explicit release-owner direction in this backport sessio
|
||||
|
||||
The final review range is `2f5e97850e55d86c0ed9eff1719994c88b4450a0..40e02ff6b62ad3445f7035e437bd9e322c8af31b`, a linear 46-commit range. The superseded beta2 base `8eeff8b4417e6fb675a66cecbdba40060322785c` and endpoint `b7fdcb89568dc707c323f8c0a3b9413ec4e889ab` are historical provenance only; all active aggregate carrier fields in the TSV indexes are remapped to the rebased commits. Official #8921 is the release-base commit `2f5e97850e55d86c0ed9eff1719994c88b4450a0`, not a duplicate aggregate carrier.
|
||||
|
||||
The three explicit PRs #8672, #8699, and #8921 are base-present; the other explicit PRs have aggregate provenance. Thus the machine-readable indexes contain exactly 36 explicit PR rows plus exactly the two closure rows (#8734 and #8392): 38 unique provenance relationships.
|
||||
|
||||
PR #8902 is appended as final carrier `40e02ff6b62ad3445f7035e437bd9e322c8af31b`; its readable decision is [Dictionary group-by fast path](decisions/dictionary-groupby-fast-path.md), and its raw patch comparison is [#8902](range-diffs/pr-8902.txt).
|
||||
|
||||
@@ -41,6 +41,6 @@ e61a6df6b2b110badd5755f68010b6439d69ac84 3351de9102b6418207e49c090054b71cc63205
|
||||
9bf878f562518d30a412138ad4244d7d8e7867e6 668df91d76785b93a00f279916d2584646a56080 fix: adapt event context tests to release channel API
|
||||
6595b5d7bf1f59587bf89676d420db7371cdc459 922a4f6141ea7ed051b9c891cb05bbcee0b51e69 fix: adapt frontend event context tests
|
||||
d18430b874f9c8054747dbf688585cd7d2988e3e 60c7bbcaf30a0e65ad7fa2b2b8ad0c1199a662b1 unavailable refactor: share query channel definition
|
||||
fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d d18430b874f9c8054747dbf688585cd7d2988e3e (unclassified) feat: add JSON2 storage layout settings
|
||||
5a642091cd2b6185fbe6a08d6f76d8295955152f fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d (unclassified) feat: add JSON2 physical layout primitives
|
||||
fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d d7f1233f775d54380318d6ad9ff62504a7cbcff1 ba5de162f5dd269aa0081d8d97492c578fc37d05 feat: add JSON2 storage layout settings
|
||||
5a642091cd2b6185fbe6a08d6f76d8295955152f b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73 a1011136c44b4f6f2520683739fbd845257c0a0e feat: add JSON2 physical layout primitives
|
||||
40e02ff6b62ad3445f7035e437bd9e322c8af31b d99b0df374eec0329ac3f430e40b80a1e6c6963c; 1a91c76c615afec71239b84af7787a0505bef914 (absorbed) 8c3fca20ce058fd569d436c38bc96e41fc1adbca fix(query): restore columnar group-by for dictionary-encoded tags
|
||||
|
||||
|
@@ -52,7 +52,11 @@ These entries are intentionally not given separate concern pages: stable patch-I
|
||||
| **Exact stable patch-ID** | #8777, #8832, #8787, #8849, #8859, #8889, #8833, #8600, #8808, #8709, #8522, #8739, #8772 | [1dd27e9](https://github.com/GreptimeTeam/greptimedb/commit/1dd27e9162ef5438e9f44b5a48a040f5c399227a), [e80f539](https://github.com/GreptimeTeam/greptimedb/commit/e80f539077616507b04df68a85a67f5da1c8f93e), [0b49947](https://github.com/GreptimeTeam/greptimedb/commit/0b499475dedfa462e9f951e0bb7b516011a195a0), [ab80858](https://github.com/GreptimeTeam/greptimedb/commit/ab808586525877230b9774f2ce29aca590152642), [824c5ac](https://github.com/GreptimeTeam/greptimedb/commit/824c5acfa7cf1865d2c20b77281c6b12e8e73de7), [c894a24](https://github.com/GreptimeTeam/greptimedb/commit/c894a2468c2bbb621bc26bf242046f1c60493460), [4ab29b2](https://github.com/GreptimeTeam/greptimedb/commit/4ab29b22f381156225cf9a50f73e5acc1ba8bbe8), [9c683fc](https://github.com/GreptimeTeam/greptimedb/commit/9c683fccfc11cd5db720b14c7adb22b7d734defd), [1a2e7b8](https://github.com/GreptimeTeam/greptimedb/commit/1a2e7b81679c602ae76f868b66974af9397d4341), [0b6c40e](https://github.com/GreptimeTeam/greptimedb/commit/0b6c40e7f465273ac424beca3475e82f4def7991), [7d68d8d](https://github.com/GreptimeTeam/greptimedb/commit/7d68d8dbe0600b86535293210ec70417e4b2b3d5), [be0985c](https://github.com/GreptimeTeam/greptimedb/commit/be0985cc39c648a35328bf9c340a8d32b814ddc6), [5e0a643](https://github.com/GreptimeTeam/greptimedb/commit/5e0a6436ee91fb4cb48a1a63564e019be02b3e22) (in the same order) | Patch-equivalent to each upstream PR; any cherry-pick conflict is a replay detail, not a semantic delta. | No separate page; see audit TSVs and raw range-diffs |
|
||||
| **Base-present** | #8672, #8699, #8921 | #8921 official release-base commit `2f5e97850e55d86c0ed9eff1719994c88b4450a0`; no aggregate carrier | Already present in beta2 before this aggregate; no backport patch ships for any of these PRs. | No separate page; see [`README.md`](../README.md) scope |
|
||||
|
||||
Exact entries with nearby context can be reviewed from [dependency and baseline adaptations](dependency-and-baseline-adaptations.md) and the [event context page](event-context-compatibility.md). The complete 33-explicit-PR mapping, plus dependency #8734 and slice #8392, is indexed in [`upstream-pr-map.tsv`](../upstream-pr-map.tsv), [`patch-id-verdict.tsv`](../patch-id-verdict.tsv), and [`aggregate-commit-provenance.tsv`](../aggregate-commit-provenance.tsv).
|
||||
Exact entries with nearby context can be reviewed from [dependency and baseline adaptations](dependency-and-baseline-adaptations.md) and the [event context page](event-context-compatibility.md). The complete 36-explicit-PR mapping (including #8895 and #8901), plus dependency #8734 and slice #8392, is indexed in [`upstream-pr-map.tsv`](../upstream-pr-map.tsv), [`patch-id-verdict.tsv`](../patch-id-verdict.tsv), and [`aggregate-commit-provenance.tsv`](../aggregate-commit-provenance.tsv).
|
||||
|
||||
## Final provenance scope
|
||||
|
||||
The final review records 36 explicit Issue-requested PR relationships: the original 31 plus #8825, #8921, #8895, #8901, and #8902. Three explicit PRs—#8672, #8699, and #8921—are base-present. The dependency closure adds exactly two relationships: full #8734 and the release-compatible slice of #8392. The indexes therefore contain 38 unique provenance relationships total. The #8895 and #8901 mappings and complete upstream ranges are covered in [JSON2 merge/layout coverage](json2-mergescan.md) and [`manual-deltas.md`](../manual-deltas.md).
|
||||
|
||||
## Recorded verification context
|
||||
|
||||
|
||||
@@ -39,3 +39,12 @@ Recorded CI status is documented in the [canonical CI status](README.md#canonica
|
||||
## Raw audit evidence — unabridged differences only
|
||||
|
||||
[`range-diffs/pr-8745.txt`](../range-diffs/pr-8745.txt) is an unabridged patch-to-patch range-diff, not the full shipping patch.
|
||||
|
||||
## Additional JSON2 layout provenance
|
||||
|
||||
The explicit Issue-requested JSON2 layout PRs are also represented in the machine-readable indexes and have complete raw range evidence:
|
||||
|
||||
- **#8895**: upstream merge `d7f1233f775d54380318d6ad9ff62504a7cbcff1`, head `6b8645bf584a04b45828c7e8d0fb19418db70d2e`, complete upstream range `0a6a22033e3ebb4d6c7243874d840558a9eb7216^..6b8645bf584a04b45828c7e8d0fb19418db70d2e`, aggregate carrier `fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d`. The carrier folds the two upstream commits, retains JSON2 DDL/layout settings, and omits the upstream-only >=v1.3.0 compatibility case; it is `adapted`.
|
||||
- **#8901**: upstream merge `b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73`, head `b450abd46e473a07cb1b42a9c7c62d0b0d551e17`, complete upstream range `6e637fbe58f36643fea40f38d5ff352fe00254f9^..b450abd46e473a07cb1b42a9c7c62d0b0d551e17`, aggregate carrier `5a642091cd2b6185fbe6a08d6f76d8295955152f`. The carrier folds the three upstream commits and is `exact` by stable patch ID.
|
||||
|
||||
These are part of the 36 explicit Issue-requested PR relationships. Together with base-present #8672/#8699/#8921 and the two closure relationships (#8734 full, #8392 release-compatible slice), the review contains 38 provenance relationships.
|
||||
|
||||
@@ -137,3 +137,19 @@ The canonical reviewer index is [`decisions/README.md`](decisions/README.md). Th
|
||||
- DataFusion `452cb4b` advances to linear successor `4c8a6bf`, preserving #8842 and adding `DictionaryGroupValuesColumn`.
|
||||
- GreptimeDB adds `tests-integration/tests/dict_groupby_sst.rs` plus module registration. The test writes 1,200 rows, flushes append-mode flat SST, and exactly verifies six ordered tuples.
|
||||
- This integration test is not direct fast-path proof: AggregateExec EXPLAIN/metrics do not expose GroupValues identity. Direct `DictionaryGroupValuesColumn` coverage is in the pinned DataFusion fork tests.
|
||||
|
||||
### #8895 — adapted, two-commit upstream range folded into one carrier
|
||||
- Upstream PR range: `0a6a22033e3ebb4d6c7243874d840558a9eb7216` followed by `6b8645bf584a04b45828c7e8d0fb19418db70d2e`; PR head: `6b8645bf584a04b45828c7e8d0fb19418db70d2e`; merge: `d7f1233f775d54380318d6ad9ff62504a7cbcff1`.
|
||||
- Aggregate carrier: `fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d`.
|
||||
- The final carrier folds the complete upstream PR range, retains the JSON2 DDL/layout-settings behavior, and omits the upstream-only >=v1.3.0 compatibility-test case. Stable patch IDs therefore differ and the verdict is `adapted`.
|
||||
- Complete range-diff evidence, including endpoint metadata and the exact generation command, is [`range-diffs/pr-8895.txt`](range-diffs/pr-8895.txt).
|
||||
|
||||
### #8901 — exact patch-id, three-commit upstream range folded into one carrier
|
||||
- Upstream PR range: `6e637fbe58f36643fea40f38d5ff352fe00254f9`, `f2e3d6d6e16c24fdede441cf54ff1c04a3effd91`, and `b450abd46e473a07cb1b42a9c7c62d0b0d551e17`; PR head: `b450abd46e473a07cb1b42a9c7c62d0b0d551e17`; merge: `b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73`.
|
||||
- Aggregate carrier: `5a642091cd2b6185fbe6a08d6f76d8295955152f`.
|
||||
- The final carrier folds the complete upstream PR range and retains the JSON2 v2 physical-layout primitives and dependency updates. Its stable patch ID equals the upstream merge patch ID; the changed subject is an aggregate adaptation, so the verdict records `exact` patch equivalence.
|
||||
- Complete range-diff evidence, including endpoint metadata and the exact generation command, is [`range-diffs/pr-8901.txt`](range-diffs/pr-8901.txt).
|
||||
|
||||
## Final provenance scope
|
||||
|
||||
The final review records 36 explicit Issue-requested PR relationships: the original 31 plus #8825, #8921, #8895, #8901, and #8902. Of those, #8672, #8699, and #8921 are base-present. The dependency closure contributes exactly two additional relationships—full #8734 and the release-compatible slice of #8392—for 38 unique provenance relationships total.
|
||||
|
||||
@@ -30,3 +30,5 @@
|
||||
8392 cabc2f6cc667291d7bc3dcbb8f9cc82e27bd93a0 f9135380b5591b82c9ec5bbf1706459672bd72db adapted 288b8990db827c0d0f4e11a0f20e0f77fdbf4991 49fecedeb0ec2f57c348307f5fdcfc558de42c59
|
||||
8825 bf947141753a9ea554aa1ca31637fdd3ad1429f3 d18430b874f9c8054747dbf688585cd7d2988e3e adapted unavailable unavailable
|
||||
8902 d99b0df374eec0329ac3f430e40b80a1e6c6963c 40e02ff6b62ad3445f7035e437bd9e322c8af31b adapted bdc71e83b0147f1d94f1db4cbe80d0dbf8a6aa58 8c3fca20ce058fd569d436c38bc96e41fc1adbca
|
||||
8895 d7f1233f775d54380318d6ad9ff62504a7cbcff1 fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d adapted e26d83fb1aab2ee25d7e9d74ba1c42a163507c0d ba5de162f5dd269aa0081d8d97492c578fc37d05
|
||||
8901 b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73 5a642091cd2b6185fbe6a08d6f76d8295955152f exact a1011136c44b4f6f2520683739fbd845257c0a0e a1011136c44b4f6f2520683739fbd845257c0a0e
|
||||
|
||||
|
@@ -0,0 +1,330 @@
|
||||
PR #8895
|
||||
Upstream PR range: 0a6a22033e3ebb4d6c7243874d840558a9eb7216^..6b8645bf584a04b45828c7e8d0fb19418db70d2e
|
||||
Aggregate range: fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d^..fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d
|
||||
Upstream PR merge commit: d7f1233f775d54380318d6ad9ff62504a7cbcff1
|
||||
Upstream PR head: 6b8645bf584a04b45828c7e8d0fb19418db70d2e
|
||||
Aggregate carrier: fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d
|
||||
Endpoint metadata: the upstream PR contains 0a6a22033e3ebb4d6c7243874d840558a9eb7216 and 6b8645bf584a04b45828c7e8d0fb19418db70d2e; GitHub records d7f1233f775d54380318d6ad9ff62504a7cbcff1 as its merge commit. Beta2 carries the requested behavior in one final carrier.
|
||||
Adaptation/folding notes: the carrier folds both upstream PR commits into one commit, changes the subject to the exact aggregate subject, and omits the upstream-only compatibility-test case (which targets >=v1.3.0). The JSON2 layout-settings implementation is retained; this is an adapted backport, not a claim of exact commit identity.
|
||||
Command: git range-diff --creation-factor=100 0a6a22033e3ebb4d6c7243874d840558a9eb7216^..6b8645bf584a04b45828c7e8d0fb19418db70d2e fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d^..fed5c80d7e94bd2c8ab1f7ddac8ab205d993156d
|
||||
|
||||
1: 0a6a22033 ! 1: fed5c80d7 refactor(json2): add JSON2 storage layout settings
|
||||
@@
|
||||
## Metadata ##
|
||||
-Author: luofucong <luofc@foxmail.com>
|
||||
+Author: discord9 <discord9@163.com>
|
||||
|
||||
## Commit message ##
|
||||
- refactor(json2): add JSON2 storage layout settings
|
||||
+ feat: add JSON2 storage layout settings
|
||||
|
||||
- Signed-off-by: luofucong <luofc@foxmail.com>
|
||||
+ Signed-off-by: discord9 <discord9@163.com>
|
||||
|
||||
## src/datatypes/src/error.rs ##
|
||||
@@ src/datatypes/src/error.rs: pub enum Error {
|
||||
@@ src/datatypes/src/json.rs: pub struct JsonContext<'a> {
|
||||
+ })
|
||||
+ }
|
||||
+
|
||||
-+ #[cfg(test)]
|
||||
-+ fn new(type_hints: Vec<JsonTypeHint>) -> Self {
|
||||
-+ Self {
|
||||
-+ type_hints,
|
||||
-+ max_auto_expanded_paths: None,
|
||||
-+ }
|
||||
-+ }
|
||||
-+
|
||||
+ pub fn type_hints(&self) -> &[JsonTypeHint] {
|
||||
+ &self.type_hints
|
||||
+ }
|
||||
@@ src/datatypes/src/json.rs: impl JsonSettings {
|
||||
+ .fail();
|
||||
+ }
|
||||
+ };
|
||||
-+ insert_type_hint(&mut object, &hint.path, data_type)?;
|
||||
++ validate_type_hint(&mut object, &hint.path, data_type)?;
|
||||
+ }
|
||||
+ Ok(())
|
||||
+}
|
||||
+
|
||||
-+fn insert_type_hint(
|
||||
++fn validate_type_hint(
|
||||
+ object: &mut JsonObjectType,
|
||||
+ path: &[String],
|
||||
+ data_type: JsonNativeType,
|
||||
@@ src/datatypes/src/json.rs: impl JsonSettings {
|
||||
+ }
|
||||
+ .fail();
|
||||
+ };
|
||||
-+ insert_type_hint(child, path, data_type)
|
||||
++ validate_type_hint(child, path, data_type)
|
||||
+}
|
||||
+
|
||||
impl<'a> JsonContext<'a> {
|
||||
@@ src/datatypes/src/json.rs: impl JsonSettings {
|
||||
@@ src/datatypes/src/json.rs: mod tests {
|
||||
}
|
||||
|
||||
+ #[test]
|
||||
+- fn test_json_settings_forward_compatibility() {
|
||||
++ fn test_json_settings_forward_compatibility()
|
||||
++ -> std::result::Result<(), Box<dyn std::error::Error>> {
|
||||
+ let json_str = r#"{
|
||||
+ "type_hints": [
|
||||
+ {
|
||||
+@@ src/datatypes/src/json.rs: mod tests {
|
||||
+ ]
|
||||
+ }"#;
|
||||
+
|
||||
+- let deserialized = serde_json::from_str::<JsonSettings>(json_str).unwrap();
|
||||
++ let deserialized = serde_json::from_str::<JsonSettings>(json_str)?;
|
||||
+
|
||||
+- assert_eq!(
|
||||
+- deserialized,
|
||||
+- JsonSettings::new(vec![
|
||||
++ let expected = JsonSettings::try_new(
|
||||
++ vec![
|
||||
+ JsonTypeHint {
|
||||
+ path: vec!["user".to_string(), "age".to_string()],
|
||||
+ data_type: ConcreteDataType::int64_datatype(),
|
||||
+@@ src/datatypes/src/json.rs: mod tests {
|
||||
+ default_constraint: None,
|
||||
+ inverted_index: false,
|
||||
+ },
|
||||
+- ])
|
||||
+- );
|
||||
++ ],
|
||||
++ None,
|
||||
++ )?;
|
||||
++ assert_eq!(deserialized, expected);
|
||||
++ Ok(())
|
||||
+ }
|
||||
+
|
||||
#[test]
|
||||
- fn test_json_settings_ser_de() {
|
||||
- let settings = JsonSettings::new(vec![
|
||||
@@ src/datatypes/src/json.rs: mod tests {
|
||||
+ );
|
||||
}
|
||||
|
||||
+ #[test]
|
||||
+@@ src/datatypes/src/json.rs: mod tests {
|
||||
+ }
|
||||
+
|
||||
+ #[test]
|
||||
+- fn test_encode_json_respects_type_hint() {
|
||||
+- let settings = JsonSettings::new(vec![JsonTypeHint {
|
||||
++ fn test_encode_json_respects_type_hint() -> Result<()> {
|
||||
++ let type_hints = vec![JsonTypeHint {
|
||||
+ path: vec!["age".to_string()],
|
||||
+ data_type: ConcreteDataType::int64_datatype(),
|
||||
+ nullable: false,
|
||||
+ default_constraint: None,
|
||||
+ inverted_index: false,
|
||||
+- }]);
|
||||
++ }];
|
||||
++ let settings = JsonSettings::try_new(type_hints, None)?;
|
||||
+
|
||||
+ let result = settings
|
||||
+ .encode(json!({
|
||||
+@@ src/datatypes/src/json.rs: mod tests {
|
||||
+ }))
|
||||
+ .unwrap_err();
|
||||
+ assert!(err.to_string().contains("does not match JSON2 type hint"));
|
||||
++ Ok(())
|
||||
+ }
|
||||
+
|
||||
+ #[test]
|
||||
+- fn test_encode_json_respects_unsigned_type_hint() {
|
||||
+- let settings = JsonSettings::new(vec![JsonTypeHint {
|
||||
++ fn test_encode_json_respects_unsigned_type_hint() -> Result<()> {
|
||||
++ let type_hints = vec![JsonTypeHint {
|
||||
+ path: vec!["count".to_string()],
|
||||
+ data_type: ConcreteDataType::uint64_datatype(),
|
||||
+ nullable: false,
|
||||
+ default_constraint: None,
|
||||
+ inverted_index: false,
|
||||
+- }]);
|
||||
++ }];
|
||||
++ let settings = JsonSettings::try_new(type_hints, None)?;
|
||||
+
|
||||
+ let result = settings
|
||||
+ .encode(json!({
|
||||
+@@ src/datatypes/src/json.rs: mod tests {
|
||||
+ }))
|
||||
+ .unwrap_err();
|
||||
+ assert!(err.to_string().contains("does not match JSON2 type hint"));
|
||||
++ Ok(())
|
||||
+ }
|
||||
+
|
||||
+ #[test]
|
||||
+- fn test_encode_json_fills_missing_type_hint_with_default() {
|
||||
+- let settings = JsonSettings::new(vec![JsonTypeHint {
|
||||
++ fn test_encode_json_fills_missing_type_hint_with_default() -> Result<()> {
|
||||
++ let type_hints = vec![JsonTypeHint {
|
||||
+ path: vec!["user".to_string(), "age".to_string()],
|
||||
+ data_type: ConcreteDataType::int64_datatype(),
|
||||
+ nullable: false,
|
||||
+ default_constraint: Some(ColumnDefaultConstraint::Value(Value::Int64(7))),
|
||||
+ inverted_index: false,
|
||||
+- }]);
|
||||
++ }];
|
||||
++ let settings = JsonSettings::try_new(type_hints, None)?;
|
||||
+
|
||||
+ let result = settings
|
||||
+ .encode(json!({}))
|
||||
+@@ src/datatypes/src/json.rs: mod tests {
|
||||
+ panic!("Expected user Struct value");
|
||||
+ };
|
||||
+ assert_eq!(struct_field_value(user, "age"), &Value::Int64(7));
|
||||
++ Ok(())
|
||||
+ }
|
||||
+
|
||||
+ #[test]
|
||||
+- fn test_encode_json_fills_missing_nullable_type_hint_with_null() {
|
||||
+- let settings = JsonSettings::new(vec![JsonTypeHint {
|
||||
++ fn test_encode_json_fills_missing_nullable_type_hint_with_null() -> Result<()> {
|
||||
++ let type_hints = vec![JsonTypeHint {
|
||||
+ path: vec!["user".to_string(), "name".to_string()],
|
||||
+ data_type: ConcreteDataType::string_datatype(),
|
||||
+ nullable: true,
|
||||
+ default_constraint: None,
|
||||
+ inverted_index: false,
|
||||
+- }]);
|
||||
++ }];
|
||||
++ let settings = JsonSettings::try_new(type_hints, None)?;
|
||||
+
|
||||
+ let result = settings
|
||||
+ .encode(json!({ "user": {} }))
|
||||
+@@ src/datatypes/src/json.rs: mod tests {
|
||||
+ panic!("Expected user Struct value");
|
||||
+ };
|
||||
+ assert_eq!(struct_field_value(user, "name"), &Value::Null);
|
||||
++ Ok(())
|
||||
+ }
|
||||
+
|
||||
+ #[test]
|
||||
+- fn test_encode_json_rejects_missing_non_null_type_hint() {
|
||||
+- let settings = JsonSettings::new(vec![JsonTypeHint {
|
||||
++ fn test_encode_json_rejects_missing_non_null_type_hint() -> Result<()> {
|
||||
++ let type_hints = vec![JsonTypeHint {
|
||||
+ path: vec!["user".to_string(), "age".to_string()],
|
||||
+ data_type: ConcreteDataType::int64_datatype(),
|
||||
+ nullable: false,
|
||||
+ default_constraint: None,
|
||||
+ inverted_index: false,
|
||||
+- }]);
|
||||
++ }];
|
||||
++ let settings = JsonSettings::try_new(type_hints, None)?;
|
||||
+
|
||||
+ let err = settings.encode(json!({})).unwrap_err();
|
||||
+ assert!(
|
||||
+ err.to_string()
|
||||
+ .contains("missing non-null JSON2 type hint path user.age")
|
||||
+ );
|
||||
++ Ok(())
|
||||
+ }
|
||||
+
|
||||
+ #[test]
|
||||
+- fn test_encode_json_merges_missing_type_hint_prefix() {
|
||||
+- let settings = JsonSettings::new(vec![
|
||||
++ fn test_encode_json_merges_missing_type_hint_prefix() -> Result<()> {
|
||||
++ let type_hints = vec![
|
||||
+ JsonTypeHint {
|
||||
+ path: vec!["user".to_string(), "age".to_string()],
|
||||
+ data_type: ConcreteDataType::int64_datatype(),
|
||||
+@@ src/datatypes/src/json.rs: mod tests {
|
||||
+ ))),
|
||||
+ inverted_index: false,
|
||||
+ },
|
||||
+- ]);
|
||||
++ ];
|
||||
++ let settings = JsonSettings::try_new(type_hints, None)?;
|
||||
+
|
||||
+- let result = settings
|
||||
+- .encode(json!({}))
|
||||
+- .unwrap()
|
||||
+- .into_json_inner()
|
||||
+- .unwrap();
|
||||
++ let result = settings.encode(json!({}))?.into_json_inner().unwrap();
|
||||
+
|
||||
+ let Value::Struct(root) = result else {
|
||||
+ panic!("Expected Struct value");
|
||||
+@@ src/datatypes/src/json.rs: mod tests {
|
||||
+ struct_field_value(user, "name"),
|
||||
+ &Value::String("unknown".into())
|
||||
+ );
|
||||
++ Ok(())
|
||||
+ }
|
||||
+
|
||||
#[test]
|
||||
|
||||
## src/sql/src/error.rs ##
|
||||
@@ src/sql/src/statements/create.rs: AS SELECT number FROM numbers_input where numb
|
||||
};
|
||||
let result = extensions.build_vector_index_options();
|
||||
assert!(result.is_ok());
|
||||
-
|
||||
- ## tests/compatibility/cases/json2_storage_layout_settings/case.toml (new) ##
|
||||
-@@
|
||||
-+name = "json2_storage_layout_settings"
|
||||
-+reason = "Verify JSON2 storage layout settings in Arrow extension metadata survive restarts and upgrades."
|
||||
-+introduced_by = "PR #8895"
|
||||
-+topologies = ["distributed", "standalone"]
|
||||
-+from_range = [">=v1.3.0"]
|
||||
-+to_range = [">=v1.3.0"]
|
||||
-+features = ["json2", "table", "metadata", "read"]
|
||||
-+owner = "query"
|
||||
-
|
||||
- ## tests/compatibility/cases/json2_storage_layout_settings/setup.sql (new) ##
|
||||
-@@
|
||||
-+CREATE TABLE t_json2_storage_layout_settings (
|
||||
-+ ts TIMESTAMP TIME INDEX,
|
||||
-+ j JSON2(max_auto_expanded_paths = 0)
|
||||
-+) WITH (
|
||||
-+ 'append_mode' = 'true'
|
||||
-+);
|
||||
-+
|
||||
-+INSERT INTO t_json2_storage_layout_settings VALUES
|
||||
-+ ('2026-08-17 00:00:00+0000', '{"a": 1}');
|
||||
-+
|
||||
-+ADMIN FLUSH_TABLE('t_json2_storage_layout_settings');
|
||||
-
|
||||
- ## tests/compatibility/cases/json2_storage_layout_settings/verify.result (new) ##
|
||||
-@@
|
||||
-+SELECT j.a AS a
|
||||
-+FROM t_json2_storage_layout_settings;
|
||||
-+
|
||||
-++---+
|
||||
-+| a |
|
||||
-++---+
|
||||
-+| 1 |
|
||||
-++---+
|
||||
-+
|
||||
-+SHOW CREATE TABLE t_json2_storage_layout_settings;
|
||||
-+
|
||||
-++---------------------------------+----------------------------------------------------------------+
|
||||
-+| Table | Create Table |
|
||||
-++---------------------------------+----------------------------------------------------------------+
|
||||
-+| t_json2_storage_layout_settings | CREATE TABLE IF NOT EXISTS "t_json2_storage_layout_settings" ( |
|
||||
-+| | "ts" TIMESTAMP(3) NOT NULL, |
|
||||
-+| | "j" JSON2( |
|
||||
-+| | max_auto_expanded_paths = 0 |
|
||||
-+| | ) NULL, |
|
||||
-+| | TIME INDEX ("ts") |
|
||||
-+| | ) |
|
||||
-+| | |
|
||||
-+| | ENGINE=mito |
|
||||
-+| | WITH( |
|
||||
-+| | append_mode = 'true' |
|
||||
-+| | ) |
|
||||
-++---------------------------------+----------------------------------------------------------------+
|
||||
-
|
||||
- ## tests/compatibility/cases/json2_storage_layout_settings/verify.sql (new) ##
|
||||
-@@
|
||||
-+SELECT j.a AS a
|
||||
-+FROM t_json2_storage_layout_settings;
|
||||
-+
|
||||
-+SHOW CREATE TABLE t_json2_storage_layout_settings;
|
||||
2: 6b8645bf5 < -: --------- resolve PR comments
|
||||
@@ -0,0 +1,215 @@
|
||||
PR #8901
|
||||
Upstream PR range: 6e637fbe58f36643fea40f38d5ff352fe00254f9^..b450abd46e473a07cb1b42a9c7c62d0b0d551e17
|
||||
Aggregate range: 5a642091cd2b6185fbe6a08d6f76d8295955152f^..5a642091cd2b6185fbe6a08d6f76d8295955152f
|
||||
Upstream PR merge commit: b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73
|
||||
Upstream PR head: b450abd46e473a07cb1b42a9c7c62d0b0d551e17
|
||||
Aggregate carrier: 5a642091cd2b6185fbe6a08d6f76d8295955152f
|
||||
Endpoint metadata: the upstream PR contains 6e637fbe58f36643fea40f38d5ff352fe00254f9, f2e3d6d6e16c24fdede441cf54ff1c04a3effd91, and b450abd46e473a07cb1b42a9c7c62d0b0d551e17; GitHub records b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73 as its merge commit. Beta2 carries the requested behavior in one final carrier.
|
||||
Adaptation/folding notes: the carrier folds all three upstream PR commits into one commit and changes the subject to the exact aggregate subject. The JSON2 v2 physical-layout primitives and dependency updates are retained; release lock/context resolution is represented by the final carrier. This is an exact stable patch-ID replay with a folded commit message.
|
||||
Command: git range-diff --creation-factor=100 6e637fbe58f36643fea40f38d5ff352fe00254f9^..b450abd46e473a07cb1b42a9c7c62d0b0d551e17 5a642091cd2b6185fbe6a08d6f76d8295955152f^..5a642091cd2b6185fbe6a08d6f76d8295955152f
|
||||
|
||||
1: 6e637fbe5 ! 1: 5a642091c refactor(json2): add JSON2 v2 physical layout primitives
|
||||
@@
|
||||
## Metadata ##
|
||||
-Author: luofucong <luofc@foxmail.com>
|
||||
+Author: discord9 <discord9@163.com>
|
||||
|
||||
## Commit message ##
|
||||
- refactor(json2): add JSON2 v2 physical layout primitives
|
||||
+ feat: add JSON2 physical layout primitives
|
||||
|
||||
- Signed-off-by: luofucong <luofc@foxmail.com>
|
||||
+ Signed-off-by: discord9 <discord9@163.com>
|
||||
|
||||
## Cargo.lock ##
|
||||
@@ Cargo.lock: dependencies = [
|
||||
@@ src/datatypes/Cargo.toml: jsonb.workspace = true
|
||||
serde.workspace = true
|
||||
|
||||
## src/datatypes/src/extension/json.rs ##
|
||||
-@@
|
||||
- use std::collections::HashMap;
|
||||
- use std::sync::Arc;
|
||||
-
|
||||
-+#[cfg(test)]
|
||||
-+use arrow_schema::FieldRef;
|
||||
+@@ src/datatypes/src/extension/json.rs: use std::sync::Arc;
|
||||
use arrow_schema::extension::{
|
||||
EXTENSION_TYPE_METADATA_KEY, EXTENSION_TYPE_NAME_KEY, ExtensionType,
|
||||
};
|
||||
- use arrow_schema::{ArrowError, DataType, Field};
|
||||
-+#[cfg(test)]
|
||||
+-use arrow_schema::{ArrowError, DataType, Field};
|
||||
++use arrow_schema::{ArrowError, DataType, Field, FieldRef};
|
||||
+use parquet_variant_compute::VariantType;
|
||||
use serde::{Deserialize, Serialize};
|
||||
-use snafu::ResultExt;
|
||||
-+#[cfg(test)]
|
||||
-+use snafu::OptionExt;
|
||||
+use snafu::{ResultExt, ensure};
|
||||
|
||||
+use crate::error::InvalidJson2LayoutSnafu;
|
||||
@@ src/datatypes/src/extension/json.rs
|
||||
const LEGACY_JSON_STRUCTURE_SETTINGS_KEY: &str = "json_structure_settings";
|
||||
+const JSON2_LAYOUT_V1: u8 = 1;
|
||||
+const JSON2_LAYOUT_V2: u8 = 2;
|
||||
-+
|
||||
+
|
||||
+-#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
+/// Parsed physical layout of a JSON2 Arrow root field.
|
||||
+#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
+pub struct Json2PhysicalLayout {
|
||||
@@ src/datatypes/src/extension/json.rs
|
||||
+}
|
||||
+
|
||||
+impl Json2PhysicalLayout {
|
||||
-+ /// Parses and validates the JSON2 layout version from a root field.
|
||||
++ /// Parses the JSON2 layout version from a root field.
|
||||
++ ///
|
||||
++ /// The version is read from the extension metadata; missing metadata
|
||||
++ /// defaults to V1.
|
||||
++ ///
|
||||
++ /// Supported versions:
|
||||
++ /// - V1
|
||||
++ /// - V2
|
||||
++ ///
|
||||
++ /// Errors when the field is not a JSON2 extension or the version is not
|
||||
++ /// one of the supported versions.
|
||||
+ pub fn try_from_root(field: &Field) -> crate::error::Result<Self> {
|
||||
+ ensure!(
|
||||
+ is_json2_extension_type(field),
|
||||
@@ src/datatypes/src/extension/json.rs
|
||||
+}
|
||||
+
|
||||
+/// Returns the remainder field of a JSON2 v2 root.
|
||||
-+#[cfg(test)]
|
||||
-+fn json2_remainder_field(field: &Field) -> crate::error::Result<&FieldRef> {
|
||||
-+ if !Json2PhysicalLayout::try_from_root(field)?.is_version_2() {
|
||||
-+ return InvalidJson2LayoutSnafu {
|
||||
-+ reason: format!("JSON2 root '{}' is not layout v2", field.name()),
|
||||
-+ }
|
||||
-+ .fail();
|
||||
++pub(crate) fn json2_remainder_field(field: &Field) -> crate::error::Result<Option<&FieldRef>> {
|
||||
++ let layout = Json2PhysicalLayout::try_from_root(field)?;
|
||||
++ if !layout.is_version_2() {
|
||||
++ return Ok(None);
|
||||
+ }
|
||||
-
|
||||
--#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
-+ let DataType::Struct(fields) = field.data_type() else {
|
||||
-+ return InvalidJson2LayoutSnafu {
|
||||
-+ reason: format!(
|
||||
-+ "expecting the Struct datatype, actual: '{}'",
|
||||
-+ field.data_type(),
|
||||
-+ ),
|
||||
-+ }
|
||||
-+ .fail();
|
||||
++
|
||||
++ let remainder = if let DataType::Struct(fields) = field.data_type() {
|
||||
++ fields
|
||||
++ .iter()
|
||||
++ .find(|x| x.name() == JSON2_REMAINDER_FIELD_NAME)
|
||||
++ } else {
|
||||
++ None
|
||||
+ };
|
||||
-+ let remainder = fields
|
||||
-+ .iter()
|
||||
-+ .find(|x| x.name() == JSON2_REMAINDER_FIELD_NAME)
|
||||
-+ .context(InvalidJson2LayoutSnafu {
|
||||
-+ reason: "remainder field not found",
|
||||
++
|
||||
++ if let Some(remainder) = remainder {
|
||||
++ let _ = remainder.try_extension_type::<VariantType>().map_err(|e| {
|
||||
++ InvalidJson2LayoutSnafu {
|
||||
++ reason: e.to_string(),
|
||||
++ }
|
||||
++ .build()
|
||||
+ })?;
|
||||
-+ let _ = remainder.try_extension_type::<VariantType>().map_err(|e| {
|
||||
-+ InvalidJson2LayoutSnafu {
|
||||
-+ reason: e.to_string(),
|
||||
-+ }
|
||||
-+ .build()
|
||||
-+ })?;
|
||||
++ }
|
||||
+ Ok(remainder)
|
||||
+}
|
||||
+
|
||||
@@ src/datatypes/src/extension/json.rs: mod tests {
|
||||
+ JsonSettings::default(),
|
||||
+ ))));
|
||||
+ assert!(Json2PhysicalLayout::try_from_root(&v2)?.is_version_2());
|
||||
-+ let remainder = json2_remainder_field(&v2)?;
|
||||
++ let remainder = json2_remainder_field(&v2)?.unwrap();
|
||||
+ assert_eq!(JSON2_REMAINDER_FIELD_NAME, remainder.name());
|
||||
+ Ok(())
|
||||
+ }
|
||||
@@ src/datatypes/src/extension/json.rs: mod tests {
|
||||
+ Json2ExtensionType::new(Arc::new(JsonMetadata::new_v2(JsonSettings::default())));
|
||||
+ let missing = Field::new("data", DataType::Struct(Fields::empty()), true)
|
||||
+ .with_extension_type(metadata.clone());
|
||||
-+ assert!(json2_remainder_field(&missing).is_err());
|
||||
++ assert!(json2_remainder_field(&missing).is_ok_and(|x| x.is_none()));
|
||||
+
|
||||
+ let invalid = Field::new(
|
||||
+ "data",
|
||||
@@ src/datatypes/src/vectors/json/array.rs: use crate::data_type::ConcreteDataType;
|
||||
use crate::error::{
|
||||
AlignJsonArraySnafu, ArrowComputeSnafu, InvalidJsonSnafu, InvalidJsonbSnafu, Result,
|
||||
};
|
||||
-+use crate::extension::json::{JSON2_REMAINDER_FIELD_NAME, Json2PhysicalLayout};
|
||||
++use crate::extension::json::{JSON2_REMAINDER_FIELD_NAME, json2_remainder_field};
|
||||
use crate::json::value::{decode_json_variant, encode_serde_json_as_jsonb};
|
||||
use crate::prelude::{DataType as _, Value as GreptimeValue};
|
||||
use crate::value::{ListValue, StructValue};
|
||||
@@ src/datatypes/src/vectors/json/array.rs: impl JsonArray<'_> {
|
||||
}
|
||||
|
||||
+ /// Projects a physical JSON2 array to a logical query type.
|
||||
-+ pub fn project_json2(&self, field: &Field, target: &DataType) -> Result<ArrayRef> {
|
||||
-+ let layout = Json2PhysicalLayout::try_from_root(field)?;
|
||||
-+ if !layout.is_version_2() {
|
||||
-+ return self.project_to(target);
|
||||
-+ }
|
||||
-+
|
||||
-+ let has_remainder = match field.data_type() {
|
||||
-+ DataType::Struct(fields) => fields
|
||||
-+ .iter()
|
||||
-+ .any(|x| x.name() == JSON2_REMAINDER_FIELD_NAME),
|
||||
-+ _ => false,
|
||||
-+ };
|
||||
-+ if !has_remainder {
|
||||
-+ return self.project_to(target);
|
||||
++ ///
|
||||
++ /// TODO(LFC) Supersede `project_to_v2` to `project_to`.
|
||||
++ pub fn project_to_v2(&self, field: &Field, target: &DataType) -> Result<ArrayRef> {
|
||||
++ if json2_remainder_field(field)?.is_some() {
|
||||
++ project_json_values(self.json2_values()?, target)
|
||||
++ } else {
|
||||
++ self.project_to(target)
|
||||
+ }
|
||||
-+
|
||||
-+ project_json_values(self.json2_values()?, target)
|
||||
+ }
|
||||
+
|
||||
+ fn json2_values(&self) -> Result<Vec<Value>> {
|
||||
@@ src/datatypes/src/vectors/json/array.rs: mod test {
|
||||
+ ]
|
||||
+ .into(),
|
||||
+ );
|
||||
-+ let projected = JsonArray::from(&array).project_json2(&field, &target)?;
|
||||
++ let projected = JsonArray::from(&array).project_to_v2(&field, &target)?;
|
||||
+ assert_eq!(
|
||||
+ json!({"cold": 1, "count": 42}),
|
||||
+ JsonArray::from(&projected).try_get_value(0)?
|
||||
@@ src/datatypes/src/vectors/json/array.rs: mod test {
|
||||
+ Json2ExtensionType::new(Arc::new(JsonMetadata::new_v2(JsonSettings::default()))),
|
||||
+ );
|
||||
+
|
||||
-+ let projected = JsonArray::from(&array).project_json2(&field, field.data_type())?;
|
||||
++ let projected = JsonArray::from(&array).project_to_v2(&field, field.data_type())?;
|
||||
+ assert!(Arc::ptr_eq(&array, &projected));
|
||||
+ Ok(())
|
||||
+ }
|
||||
2: f2e3d6d6e < -: --------- resolve PR comments
|
||||
3: b450abd46 < -: --------- fix ci
|
||||
@@ -34,3 +34,5 @@
|
||||
8825 bf947141753a9ea554aa1ca31637fdd3ad1429f3 MERGED https://github.com/GreptimeTeam/greptimedb/pull/8825 refactor: share query channel definition
|
||||
8902 8d887ddd00d67699d0fb873cda11eb50b5771555 MERGED https://github.com/GreptimeTeam/greptimedb/pull/8902 fix(query): restore columnar group-by for dictionary-encoded tags (PR head 1a91c76c615afec71239b84af7787a0505bef914; range starts d99b0df374eec0329ac3f430e40b80a1e6c6963c)
|
||||
8921 2f5e97850e55d86c0ed9eff1719994c88b4450a0 BASE-PRESENT (official release base) https://github.com/GreptimeTeam/greptimedb/pull/8921 fix: clear pooled Prometheus Remote Write decoder state
|
||||
8895 d7f1233f775d54380318d6ad9ff62504a7cbcff1 MERGED https://github.com/GreptimeTeam/greptimedb/pull/8895 refactor(json2): support JSON2 storage layout settings in DDL
|
||||
8901 b96ea86a621e7283e77e4e37dd4e3ec8dfc44f73 MERGED https://github.com/GreptimeTeam/greptimedb/pull/8901 refactor(json2): add JSON2 v2 physical layout primitives
|
||||
|
||||
|
Reference in New Issue
Block a user