mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-30 07:26:56 +00:00
Ning Sun
3131cdbcb6
* fix: bound decompressed request size and close memory-admission gaps for compressed requests The request-memory accounting only bounds and charges the encoded bytes on the wire, but a compressed body can expand far beyond that during decompression, so tiny requests could allocate disproportionate frontend memory before any protobuf validation or quota charge. - Handler-level decompression (Prometheus remote read/write v1+v2, Loki) now enforces a hard 512 MiB decoded-size cap, checked before any output buffer is allocated, and charges the decoded bytes to the shared ServerMemoryLimiter, holding the permits for the lifetime of the decompressed buffer. - gRPC requests with transport compression reserve the configured max_recv_message_size before tonic decompresses, so the decoding phase is admitted against max_in_flight_write_bytes; the later per-message charge is skipped to avoid double accounting. - The HTTP memory-limit middleware keeps its upfront Content-Length charge but now also accounts the bytes actually streamed beyond it, so chunked requests and understated Content-Length headers no longer bypass the aggregate quota. - Routes that decompress request bodies via RequestDecompressionLayer (InfluxDB, OTLP, Loki, Splunk, Elasticsearch, pipelines, dashboards) now charge the decompressed bytes as handlers consume them: the global middleware marks Content-Encoding requests, and a route-local accounting layer inside the decompression layer charges the decoded stream. Plain requests are skipped to avoid double-counting. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on memory admission guard lifetimes and 429 mapping - Retain the gRPC pre-decode reservation for the whole request: the extensions holding the guard were dropped when the request was consumed (into_inner), releasing the reservation while per-message charges stayed skipped. Both the unary and streaming handlers now clone and hold the reservation marker for the duration of request handling. - Retain the HTTP body permits across the handler: the AccountedBody wrapper and the request extensions are dropped once the extractors finish collecting the body, before the handler is done with the decoded data. Both accounting middlewares now keep their own accounting handle alive across next.run(req).await. - Return a ChargedBuffer from the Loki snappy decompressor so the reservation outlives the decompressed bytes through the caller's protobuf decoding, matching the Prometheus path. - Map mid-stream quota exhaustion to 429 instead of the generic body error (400): the accounting flags exhaustion and the middlewares rewrite the extractor rejection, so clients can distinguish backpressure from malformed input. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve the in-flight body acquisition before trying a new one A parked acquisition in AccountedBody::charge always belongs to the currently buffered frame, so it must be resolved before any new acquisition is tried. Letting the fast-path try_acquire succeed while a waiter is parked left the waiter alive, and its late completion would then be credited with a later frame's byte count, under-reserving memory relative to what was marked charged. Adds a regression test that reproduces the misattribution: with the buggy ordering the test delivers a body the quota cannot cover; with the fix the over-quota frame waits for its own acquisition and times out. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com>
Setup tests for multiple storage backend
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
Take s3 for example. You need to set your S3 bucket, access key id and secret key:
# Settings for s3 test
GT_S3_BUCKET=S3 bucket
GT_S3_REGION=S3 region
GT_S3_ACCESS_KEY_ID=S3 access key id
GT_S3_ACCESS_KEY=S3 secret access key
Run
Execute the following command in the project root folder:
cargo test integration
Test s3 storage:
cargo test s3
Test oss storage:
cargo test oss
Test azblob storage:
cargo test azblob
Setup tests with Kafka wal
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
GT_KAFKA_ENDPOINTS = localhost:9092
Setup kafka standalone
cd tests-integration/fixtures
docker compose -f docker-compose.yml up kafka
Setup tests with etcd TLS
This guide explains how to set up and test TLS-enabled etcd connections in GreptimeDB integration tests.
Quick Start
TLS certificates are already at tests-integration/fixtures/etcd-tls-certs/.
-
Start TLS-enabled etcd:
cd tests-integration/fixtures docker compose up etcd-tls -d -
Start all services (including etcd-tls):
cd tests-integration/fixtures docker compose up -d --wait
Certificate Details
The checked-in certificates include:
ca.crt- Certificate Authority certificateserver.crt/server-key.pem- Server certificate for etcd-tls serviceclient.crt/client-key.pem- Client certificate for connecting to etcd-tls
The server certificate includes SANs for localhost, etcd-tls, 127.0.0.1, and ::1.
Regenerating Certificates (Optional)
If you need to regenerate the etcd certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate-etcd-tls-certs.sh
# Or generate in custom location
./scripts/generate-etcd-tls-certs.sh /path/to/cert/directory
If you need to regenerate the mysql and postgres certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate_certs.sh
# Or generate in custom location
./scripts/generate_certs.sh /path/to/cert/directory
Note: The checked-in certificates are for testing purposes only and should never be used in production.