mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
discord9
23afa9697e
* feat(query): place the canonical merge above the regions
A canonical producer merges the pieces of one label set that its own input
partitions hold, and the canonical router FusedSeries::to_physical_plan inserts
above it collects them from exactly those partitions. In process the producer
reads the region's own scans, so the region holds every piece its router has to
see and that placement is the only one it has.
Over the wire the producer reads RawSeriesSource children instead - the wrapper
that reconciles fragments several regions hold. Classifying that shape
commutative shipped producer -> router as the region sub-plan, one producer and
one router per region, each collecting only its own region's pieces, so a label
set a time partition spread over several regions was answered from one region's
share. Classify a canonical producer by its own contract only: commutative when
every child is a bare scan, non-commutative otherwise, and never fall through to
the per-series rules, which describe what a node computes and say nothing about
where the pieces of one label set are. The analyzer then cuts at the raw source,
so the frontend holds
consumer -> SeriesRepartitionExec(CanonicalLabels) -> producer
-> [raw router] -> MergeScan(RawSeriesSource(TableScan))
and the canonical router collects each label set from the producers of every
region, while the fragments of each series are still reconciled below the
producer by the raw-primary-key router the interleaved raw input asks for.
Admission follows the placement: the in-process route still declines a canonical
merge over a partitioned table, because a region-local producer is the only
placement it has, and the remote route now admits it. Acceptance is a real
two-datanode test over a three-partition metric table: the rows equal an
unpartitioned control's, the producer and its canonical router are frontend
nodes above the merge scans, and a collision two regions hold reports the
Prometheus duplicate-labelset error that no single region can see.
Signed-off-by: discord9 <discord9@163.com>
* test(promql): partition the canonical acceptance table on a tag
GreptimeDB has no implicit time partitioning: a table has more than one
region only from a declared partition, and metric tables are normally
partitioned on a tag. The acceptance fixture partitioned on the time index
instead, which made the test read as if time partitioning were the everyday
case. Partition on `host` in three ranges - `early`/`first`/`late`/`second`
still land in different regions, so the merge pair and the collision pair keep
their cross-region geometry - and rename the test accordingly.
A partition on a primary-key column that excludes the time index also proves
the regions hold disjoint series, so the two assertions that described
fragmented series state what this fixture actually proves instead: the merge
scan publishes whole series (`contiguous`), and no raw-key repartition sits
below the frontend producer because only the label set crosses regions.
Signed-off-by: discord9 <discord9@163.com>
* refactor(query): ask the transport where a canonical merge runs
`canonical_boundary_region_node` inferred "this execution reads the whole
table" from the node's children: a canonical producer whose children were all
bare scans was admitted inside a region sub-plan. Over the wire every selector
input is a `RawSeriesSource`, so the inference happened to agree with the rule
that matters, but a wire plan that ever held a bare scan under a producer would
have been admitted silently and answered a label set from one region's share.
The analyzer already holds the fact. `in_process_region_transport` is what it
was built with, and it is exactly the condition under which a region read is
the whole table. Pass it down to `Categorizer::check_plan` and
`check_extension_plan` and require it instead of inspecting child shapes.
Guard the hole the inference hid: a canonical producer over a bare table scan
classifies `Commutative` in process and `NonCommutative` on the wire.
Signed-off-by: discord9 <discord9@163.com>
* test(promql): cover a fragmented canonical merge too
A partition on a primary-key column proves the regions hold disjoint series,
so the tag-partitioned fixture could not exercise the other geometry the
frontend merge has to read: fragments. In a metric table a tag is always a
primary-key column and a field must be Float64, so the only declared partition
that proves nothing about where a series lives is the time index.
Run the same two fixture rows over both geometries in one loop. The tag case
still publishes whole series and has no router below the producer; the time
case declares interleaved and asserts exactly one RawPrimaryKey reconciling
router between the merge scans and the frontend producer. Both keep the
unpartitioned control's rows, the frontend placement of the producer and its
canonical router, and the cross-region duplicate-labelset error.
Signed-off-by: discord9 <discord9@163.com>
* Revert "test(promql): cover a fragmented canonical merge too"
This reverts commit 8ff9f43031.
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Setup tests for multiple storage backend
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
Take s3 for example. You need to set your S3 bucket, access key id and secret key:
# Settings for s3 test
GT_S3_BUCKET=S3 bucket
GT_S3_REGION=S3 region
GT_S3_ACCESS_KEY_ID=S3 access key id
GT_S3_ACCESS_KEY=S3 secret access key
Run
Execute the following command in the project root folder:
cargo test integration
Test s3 storage:
cargo test s3
Test oss storage:
cargo test oss
Test azblob storage:
cargo test azblob
Setup tests with Kafka wal
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
GT_KAFKA_ENDPOINTS = localhost:9092
Setup kafka standalone
cd tests-integration/fixtures
docker compose -f docker-compose.yml up kafka
Setup tests with etcd TLS
This guide explains how to set up and test TLS-enabled etcd connections in GreptimeDB integration tests.
Quick Start
TLS certificates are already at tests-integration/fixtures/etcd-tls-certs/.
-
Start TLS-enabled etcd:
cd tests-integration/fixtures docker compose up etcd-tls -d -
Start all services (including etcd-tls):
cd tests-integration/fixtures docker compose up -d --wait
Certificate Details
The checked-in certificates include:
ca.crt- Certificate Authority certificateserver.crt/server-key.pem- Server certificate for etcd-tls serviceclient.crt/client-key.pem- Client certificate for connecting to etcd-tls
The server certificate includes SANs for localhost, etcd-tls, 127.0.0.1, and ::1.
Regenerating Certificates (Optional)
If you need to regenerate the etcd certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate-etcd-tls-certs.sh
# Or generate in custom location
./scripts/generate-etcd-tls-certs.sh /path/to/cert/directory
If you need to regenerate the mysql and postgres certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate_certs.sh
# Or generate in custom location
./scripts/generate_certs.sh /path/to/cert/directory
Note: The checked-in certificates are for testing purposes only and should never be used in production.