mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
discord9
fe2e364d23
* feat(datatypes): add the shared aggregate-state representation A frame that carries partial aggregate state has to say which aggregate it is state of and which accumulator components it carries, or a consumer can read a sum state as an average one. These types are that statement: the kind, the ordered component list, one position's values, and the schema that validates the two against each other. Nothing uses them yet; the frame content that carries them follows. Signed-off-by: discord9 <discord9@163.com> * feat(datatypes): carry aggregate state as a third frame content A state frame keeps the series-major columns a frame already has — the key, finality, grid spans and labels — and puts each aggregate field's accumulators in one list of component structs under `fields`, the shape a classic histogram payload already uses. An absent list item is that field having no contribution at that position, so a hole stays a hole instead of becoming a zero, and a group state — which carries no component at all — still says per field where it was folded. The reader keeps typed handles into the batch rather than expanding the state into rows and rebuilding it, so the merge above it reads the arrays where they lie. Signed-off-by: discord9 <discord9@163.com> * feat(common-query): pack and unpack aggregate-state series frames The packer takes what a partial aggregate produces — one group, its carried positions and its per-field state — and the unpacker hands back the row with typed handles into the batch. That asymmetry is deliberate: export columnarises once, import must not rebuild the state to read it. Validation mirrors the evaluated pair: the declaration is parsed, contiguity required, the empty batch decoded before the stream is returned, and kind, component order, grid and field count are each checked with the part that disagreed named. Signed-off-by: discord9 <discord9@163.com> * feat(promql): run native partial and final aggregates An aggregate node used to compute only `Complete`, refusing the stages a split aggregate needs. It now folds evaluated chunks into kernel state for `Partial` and merges that state into values for `Final`, so the compiler's `Partial -> Final` plan has a runtime behind it. The executor gains a third chunk kind: an input leaf accepts raw, evaluated or state chunks according to what its binding produces, never guessing from a schema, and a state chunk cannot be handed to a node that reads values. The state binding now carries the aggregate kind and its ordered components, derived from the kernel's own state fields, so a sum state cannot be merged as an average one. The sibling router refuses aggregate state explicitly for now; the node that routes it follows in its own change. Signed-off-by: discord9 <discord9@163.com> * feat(promql): route aggregate state by its group labels A state row is one group, complete as one producer wrote it: there is nothing to chain and nothing to merge, so the existing repartition node — whose invariants are all about assembling a key's chain and waiting for every input to close — is the wrong place for it. This node does one thing: send each row to the output its canonical labels name, so every contribution of a group reaches one owner and the merge above sees all of them. The read belongs to the outputs that were claimed, not to the plan, so dropping every output stops it and returns what it reserved. Signed-off-by: discord9 <discord9@163.com> * feat(promql): execute partial aggregates as state frames A fused node now reads its inputs by what they produce — raw samples, evaluated values, or aggregate state — instead of inferring the kind from a schema, and a partial root answers state rather than rows: the executor's state chunks are packed into the aggregate-state frame the merge above consumes. The temporary partial-fold bridge goes with it: no marker field, no side channel fold, no compatibility accessor. A tree that folds on the side is exactly what this change exists to remove, and the aggregate merge that replaces it is a separate node. The state path derives its mode and its binding from one reading of the root, so the two cannot disagree, and no production path panics on a malformed tree. Signed-off-by: discord9 <discord9@163.com> * feat(promql): split native aggregates into partial and final trees The previous commit made a `Partial` aggregate export its state, but left `AggregateFinalize` in place: it still called `FusedSeries::try_new_partial` and `partial_fold`, which that commit deleted, so the crate did not compile. Replace the bridge with the two trees that run a split aggregate: - `native/split.rs` owns the cut. It moves out of `aggregate_finalize.rs` unchanged in structure, and `Split::into_trees` binds the two halves: the prefix under a `Partial` aggregate answering aggregate state, and a `Final` aggregate consuming that state with the suffix above it. - The suffix is narrower than the old merge admitted. Only point-wise maps and a binary operation with a constant side may sit above the cut; `topk`, the classic-histogram functions, a second aggregate and a vector matching are declined, because the groups a `Final` aggregate owns are split across partitions and each of those reads a whole vector. - `FusedSeries` gains `try_new_state_export` (a `Partial` node declares its own state frame schema rather than taking a caller's), `split_aggregate` (declining shapes become `None`, not an error), `reads_aggregate_state` and the state-input path of `to_execution_plan_with_parallelism_option`, which routes state through `StateRepartitionExec` and lets a cross-series tree read state from more than one partition. - The runtime executes the `Final` stage: the guard that refused it is gone. `AggregateFinalize`, `aggregate_exchange` and their tests are deleted; nothing in the crate referenced the exchange module. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): route aggregate state through the series repartitioner Replace the dedicated state-repartition node with the existing series repartition execution: state batches are routed by their canonical labels without being rebuilt as StateSeries, and each row slice is charged for the Arrow wrappers it newly allocates while the parent buffers stay charged once. Carry a producer's state charge with the chunk it hands over, so the exporter that packs rows across polls remains accountable for the batch it still holds instead of releasing it at handoff. Read a Final aggregate's value and tag columns positionally, which is the order the split's output schema is built in, and name them through the legacy expression only at the outer boundary; a bare sum over a metric whose value column is named after the whole expression no longer fails planning. Pack aggregate-state frames in one pass over positions and validity instead of scanning the contributing rows for every position. Cover the changed behaviour with execution tests for bare global and grouped sums over one- and two-value-field tables, for the handed-over state charge, and for the export lifecycle on drain and early drop. Signed-off-by: discord9 <discord9@163.com> * fix(promql): bound tree depth on both sides and validate merge inputs The depth limit was decode-only and set to a number nobody had measured. A 129-deep tree could therefore be built, serialized and shipped, while the receiver refused it; and in CI the boundary test itself aborted with a stack overflow, because a 2 MiB thread was asked to build, serialize, decode and drop a chain of that depth, and the recursive drop of `Box<NativeNode>` is not covered by a guard that only wraps `Deserialize`. Measured instead of guessed. A probe walked build, serialize, decode, validate and drop at increasing depth inside a named 2 MiB thread: 32 through 128 completed, 160 aborted. The cap is now 64, half the highest passing depth, and the same number is enforced in `NativeTree::try_new` and in the deserializer. Verified through the real constructor: a 64-node tree round-trips, and a 65-node tree is refused when it is built as well as when it is decoded, so sender and receiver accept the same set. The boundary test runs on a named thread, which is how a future failure will identify itself. The merge node accepted any single input. `with_exprs_and_inputs` rebuilt `AggregateFinalize` over whatever it was handed, on the reasoning that a rewritten input only changes where the state comes from. Nothing checked that it was state at all -- and until now nothing could produce a counterexample, because a values-producing producer could not be serialized. The serializer was standing in for a validation that belonged to the node. `with_exprs_and_inputs` now rejects an input whose schema is not `state_schema(binding.tags)`, so substituting a computed-values producer fails at the logical node with a message naming the expected schema, rather than depending on an encoding limitation. `AggregateFinalizeExec` already had the same check at physical planning; the serializer stays permissive. The three placement tests in `commutativity.rs` asserted the old refusal as their mechanism -- an ordinary fused node "fails to encode", a partial producer "cannot be pushed down". Those expectations were the encoding limitation wearing a test's clothes. They now assert the responsibilities that remain: an ordinary fused node round-trips through the encoder, a partial producer rides to the region while its finalize stays on the frontend, and a values producer handed to a merge is refused by the schema contract. The test-only `NPRQ` envelope is gone. It framed trees with its own version, a 1 MiB cap and its own rejections, none of which production had, so `native_tree_envelope_rejects_version_size_trailing_and_truncation` was proving a helper's protocol while the PR description claimed a production guarantee. Trees are tested through plain postcard, transport through the real `FusedSeries` interface. The redundant clone-and-revalidate in `FusedSeries::deserialize` is also gone: `NativeTree::Deserialize` already routes through `try_new`. Signed-off-by: discord9 <discord9@163.com> * style: let taplo order the workspace dependency entry `taplo format --check` runs over the whole workspace in CI, not over the two manifests I had checked. It sorts the `[workspace.dependencies]` table, so `postcard` belongs in alphabetical position inside the external-dependencies block rather than immediately after its comment. Local checking was narrower than the CI command, which is why this only surfaced there. Signed-off-by: discord9 <discord9@163.com> * style: place postcard in the sorted position the review asked for Inside the external-dependencies block, `taplo` was willing to leave the entry before the `axum` comment, because a comment line ends a reorder group. The review asked for the alphabetical position, which `taplo format --check` also accepts, so use it. Signed-off-by: discord9 <discord9@163.com> * docs(promql): name the real reason a multi-source name grouping declines The comments still said the fused node's declared columns cannot name `__name__`. Since #10106 a discovered source declares it, so a nameless selector's `sum by (__name__)` survives; what actually declines is a tree reading more than one *named* source, where the table name is the metric name and no source declares the tag. Same conditions, accurate reason. Signed-off-by: discord9 <discord9@163.com> * test(promql): find the topk producer by behaviour, not by node count `native_aggregate_topk_over_global_partitions` asserted that the whole plan holds exactly one fused node. Once native frames cross the region boundary a plan legitimately holds the frontend root and the region producers at the same time, so that count is no longer an invariant. The test now selects the segmented producer and the ranking node by what their plans do: at least one producer contains `segment` and none of them runs `topk`, and exactly one node across the fused and final captures ranks. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>
Setup tests for multiple storage backend
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
Take s3 for example. You need to set your S3 bucket, access key id and secret key:
# Settings for s3 test
GT_S3_BUCKET=S3 bucket
GT_S3_REGION=S3 region
GT_S3_ACCESS_KEY_ID=S3 access key id
GT_S3_ACCESS_KEY=S3 secret access key
Run
Execute the following command in the project root folder:
cargo test integration
Test s3 storage:
cargo test s3
Test oss storage:
cargo test oss
Test azblob storage:
cargo test azblob
Setup tests with Kafka wal
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
GT_KAFKA_ENDPOINTS = localhost:9092
Setup kafka standalone
cd tests-integration/fixtures
docker compose -f docker-compose.yml up kafka
Setup tests with etcd TLS
This guide explains how to set up and test TLS-enabled etcd connections in GreptimeDB integration tests.
Quick Start
TLS certificates are already at tests-integration/fixtures/etcd-tls-certs/.
-
Start TLS-enabled etcd:
cd tests-integration/fixtures docker compose up etcd-tls -d -
Start all services (including etcd-tls):
cd tests-integration/fixtures docker compose up -d --wait
Certificate Details
The checked-in certificates include:
ca.crt- Certificate Authority certificateserver.crt/server-key.pem- Server certificate for etcd-tls serviceclient.crt/client-key.pem- Client certificate for connecting to etcd-tls
The server certificate includes SANs for localhost, etcd-tls, 127.0.0.1, and ::1.
Regenerating Certificates (Optional)
If you need to regenerate the etcd certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate-etcd-tls-certs.sh
# Or generate in custom location
./scripts/generate-etcd-tls-certs.sh /path/to/cert/directory
If you need to regenerate the mysql and postgres certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate_certs.sh
# Or generate in custom location
./scripts/generate_certs.sh /path/to/cert/directory
Note: The checked-in certificates are for testing purposes only and should never be used in production.