mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
discord9
e7044931fb
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) This is the wire-format half of the closed #9993 line. The raw fan-in, aggregate split and remote-source work that #10002 also carried moves to a separate PR on top of the public execution framework. ## What's changed and what's your intention? ### The problem A native PromQL plan could not leave the frontend. `FusedSeries` carries a `NativeTree`, and that tree had no wire representation, so a datanode could only ever be asked for raw samples. ### The wire format `src/promql/src/native/wire.rs` defines protocol version 1 for the prefix shapes the remote route can actually produce — `Input → Instant|Window → Map(Math*)+`, i.e. a selection under any number of math maps — as an explicit byte layout documented in the module: - every node kind and every operation has a **fixed tag**; the tags are declared wire-stable and deliberately do not follow Rust discriminants, so reordering an enum cannot change the protocol; - floats travel as `f64::to_bits()`, so NaN payloads, signed zero, infinities and subnormals round-trip exactly; - the input ordering is carried as data (0 Interleaved, 1 Contiguous, 2 KeyThenTime) rather than filtered by the encoder: legality belongs to `NativeTree::try_new` on decode, not to the transport; - decoding rebuilds grids through their validating constructor and ends with `NativeTree::try_new`, so a structurally invalid tree is refused at decode; compatibility between a window modifier and its function, or a negative `lookback_ms`, is refused later, when the datanode builds the physical plan; - the payload is bounded (1 MiB) and rejects unknown versions, unknown tags, truncation and trailing bytes instead of guessing. `Instant`'s value selector is carried as a tag (0 sample, 1 timestamp, 2 info timestamp) rather than a bool, because `InstantValue` has three cases since `info()` became native. Anything outside the supported shapes fails to encode *and* to decode with an error naming the unsupported operation. `label_replace`/`label_join` are not encoded because admission already declines them. ### Carrying it `FusedSeries` gains `serialize`/`deserialize` (a small versioned envelope around the tree plus its binding) and registers arms in the existing Substrait extension serializer, so a datanode's `DefaultPlanDecoder` already understands the node. No new transport, no protocol change, no `greptime-proto` bump: extension payloads are opaque bytes. Serialization is restricted to the `SeriesFrame` output with no `partial_fold`, because only that shape derives its output schema from the tree and can therefore rebuild itself after decoding. ### Executing it on a datanode The existing `PromExtensionPlanner::plan_fused_series` already knows how to build a decoded `FusedSeries` over a `RegionScanExec`. The integration test `native_prefix_executes_over_real_mito_region` uses a real standalone Mito region to prove the whole path: a flushed table, an encoded `Input → Window(rate)` prefix, `encode` → `handle_remote_read` on the real region server, decode and execute, then unpack the returned evaluated frame and check that the tag `host` survives, `__name__` is dropped as `rate` requires, the grid timestamps are `3000`/`4000`, and the rates are `20/3` and `10`. Those two numbers are not interchangeable: a counter whose first sample is `0` cannot be extrapolated backwards at all, while the second window's sample pattern allows the full 1 s gap. The test also tampers with the embedded version byte and asserts the datanode fails rather than returning a plausible frame. ### Encode/decode agreement A payload the encoder accepts must be one this version can decode. Three places disagreed and are fixed here: - **Field count.** The decoder limited an `Input`'s field count to 128 while the encoder wrote any count, so a 129-field `Input → Instant` encoded to 983 bytes and failed to decode. Both sides now use `MAX_FIELD_COUNT`. - **Tag count.** `serialize` only checked that the tag count fit in a `u32` while `deserialize` limited it to 1024. Both sides now use `MAX_TAG_COUNT`. - **Frame schema.** `deserialize` passed an empty schema to `try_new_with_format`, which since #9968 *checks* a frame node's declared schema against the layout derived from its tree instead of overwriting it. Every serialized frame node was therefore impossible to rebuild. `deserialize` now derives the schema from the decoded tree, which is where it comes from in the first place. Counts were left at their previously published values (depth 128, operations 128, fields 128, tags 1024); only the missing encoder-side checks and the wrong schema were added. `MAX_COUNT` is split into `MAX_DEPTH` / `MAX_FIELD_COUNT` / `MAX_OPERATION_COUNT` so each limit names what it bounds, and a field-count overflow no longer reports "operation count exceeds maximum". ### Limitations - **The frontend does not route queries through this yet.** It is a plan-serialization and datanode-execution boundary, not a query-plan change. Wiring the frontend to push the prefix and consume evaluated frames is the next increment. - Only one input source and the prefix shapes listed above are transported; binary, vector-wide, aggregate-state and subquery trees are not. - No performance claim: a datanode that executes the prefix returns evaluated frames instead of raw samples, but the frontend still gathers every selected region into one partition. - A selection may sit under any number of math maps, because a map is transparent to shape: it keeps its input's samples type and grid, and only the operation kinds are restricted. Non-math operations are still refused at every level. The payload stays bounded by `MAX_DEPTH` (128 node levels, so a selection plus 126 maps), and both the encoder and the decoder enforce it. ### Known limitation: the datanode boundary is not chain-atomic The region-read merge point wraps a non-streaming plan in `CoalescePartitionsExec`, which coalesces **per batch**. A series frame declares `Contiguous` and one series chain spans several batches (the packer bounds rows and flushes on `Pending`), so on a multi-partition datanode the batches of one chain are interleaved with another's and decoding fails with `evaluated series identity changed before its chain closed`. This is reproduced by `native_prefix_multi_partition_series_chains` (16 hosts × 20k samples, `rate[10s]`, 16 partitions). That test asserts the failure, so it passes today and will turn red when the merge is fixed — which is the signal to change it. The `parallelism: 1` path is covered by `native_prefix_executes_over_real_mito_region`. The fix does not belong here: the merge point should become chain-atomic in the frontend-integration change, for example by reusing the public `SeriesRepartitionExec`. Until then, **a multi-partition datanode read of a series frame is not correct** — this PR is not evidence that the datanode path is usable. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`).
Setup tests for multiple storage backend
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
Take s3 for example. You need to set your S3 bucket, access key id and secret key:
# Settings for s3 test
GT_S3_BUCKET=S3 bucket
GT_S3_REGION=S3 region
GT_S3_ACCESS_KEY_ID=S3 access key id
GT_S3_ACCESS_KEY=S3 secret access key
Run
Execute the following command in the project root folder:
cargo test integration
Test s3 storage:
cargo test s3
Test oss storage:
cargo test oss
Test azblob storage:
cargo test azblob
Setup tests with Kafka wal
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
GT_KAFKA_ENDPOINTS = localhost:9092
Setup kafka standalone
cd tests-integration/fixtures
docker compose -f docker-compose.yml up kafka
Setup tests with etcd TLS
This guide explains how to set up and test TLS-enabled etcd connections in GreptimeDB integration tests.
Quick Start
TLS certificates are already at tests-integration/fixtures/etcd-tls-certs/.
-
Start TLS-enabled etcd:
cd tests-integration/fixtures docker compose up etcd-tls -d -
Start all services (including etcd-tls):
cd tests-integration/fixtures docker compose up -d --wait
Certificate Details
The checked-in certificates include:
ca.crt- Certificate Authority certificateserver.crt/server-key.pem- Server certificate for etcd-tls serviceclient.crt/client-key.pem- Client certificate for connecting to etcd-tls
The server certificate includes SANs for localhost, etcd-tls, 127.0.0.1, and ::1.
Regenerating Certificates (Optional)
If you need to regenerate the etcd certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate-etcd-tls-certs.sh
# Or generate in custom location
./scripts/generate-etcd-tls-certs.sh /path/to/cert/directory
If you need to regenerate the mysql and postgres certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate_certs.sh
# Or generate in custom location
./scripts/generate_certs.sh /path/to/cert/directory
Note: The checked-in certificates are for testing purposes only and should never be used in production.