* feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN COPY FROM STDIN WITH (FORMAT promwrite [, version 2] [, <options>]) ingests one Prometheus Remote Write request body over the postgres wire protocol — a GreptimeDB-specific, binary COPY format (wire format code 1). The payload is byte-for-byte the body a remote-write client POSTs to /v1/prometheus/write, compression included: buffered until CopyDone and completed through the exact ingestion core that endpoint runs. To share that core, the HTTP handlers' ingestion bodies are extracted into axum-free functions (ingest_remote_write_v1/v2 plus the context extension stamping), and the HTTP routes become thin wrappers — their behavior is unchanged, and the copy layer runs the identical decode, validation, conversion, batched/prepared write paths, per-batch write-progress accounting and context extensions, under the connection's regular postgres context. The WITH options carry what the HTTP endpoint takes as its content-type/-encoding and query parameters; session-provided values (database, user) are not repeated: - version 1|2 (default 1), matching the content-type's proto - zstd, matching Content-Encoding: zstd (snappy by default) - physical_table, matching the ?physical_table= parameter - pipeline_name/pipeline_version, matching the pipeline headers (v1 only; for v2 they are accepted and ignored, exactly like the HTTP route ignores them) Table-less like the other ingestion formats; one COPY statement is one bounded request (shared 64 MiB limit); the row count reports written rows (v1) or written samples plus histograms (v2). The flush enforces the frontend's shared request-memory quota (max_in_flight_write_bytes) — the same limiter the HTTP ingestion protocols use, so COPY cannot bypass the aggregate budget. Wiring: PostgresServer::with_prom_remote_write carries the handler, metric-engine flag, validation mode and shared memory limiter. e2e: v1 body chunked across CopyData boundaries; v2 samples with readback; physical_table option routing; garbage-body decode failure with connection reuse; a tiny-quota regression where a compressed body whose decompressed size exceeds the budget fails like the HTTP route. Signed-off-by: Ning Sun <sunning@greptime.com> * fix(servers): propagate COPY body reservations and test write accounting Signed-off-by: Ning Sun <sunning@greptime.com> * fix(postgres): share Prometheus logical batching with COPY ingestion Signed-off-by: Ning Sun <sunning@greptime.com> * refactor(postgres): narrow Prometheus COPY scope and share write outcomes Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
8.8 KiB
frontend — Agent & Contributor Guide
Navigation aid for src/frontend. Keep it short and point to code. Paths are
relative to the repo root.
Repo-wide rules that apply here: .agents/architecture-invariants.md.
What this crate does
Frontend is the request entry point and orchestration layer. It accepts
multi-protocol requests (gRPC, HTTP, MySQL, PostgreSQL, InfluxDB, OTLP, Jaeger,
Prometheus, OpenTSDB), checks permissions, parses/plans SQL, and dispatches:
reads go to the query engine (query crate), writes go to the inserter/deleter
(operator crate).
Boundary with servers: the servers crate implements the wire protocols and
network I/O; frontend provides the business logic by implementing handler
traits (SqlQueryHandler, GrpcQueryHandler, InfluxdbLineProtocolHandler,
...). In standalone mode the standalone crate bridges the frontend to an
embedded datanode RegionServer; in distributed mode the frontend talks to
remote datanodes via operator/client.
Module map
| Module | Path | Purpose |
|---|---|---|
instance |
src/frontend/src/instance.rs |
Instance: the core handler; implements SqlQueryHandler, PrometheusHandler, etc. |
instance/builder |
src/frontend/src/instance/builder.rs |
FrontendBuilder assembles Instance from its dependencies |
instance/grpc |
src/frontend/src/instance/grpc.rs |
GrpcQueryHandler: insert/delete/query/promql over gRPC |
instance/schema_selection |
src/frontend/src/instance/schema_selection.rs |
SQL database selection and authorization for sessions without a selected schema |
instance/region_query |
src/frontend/src/instance/region_query.rs |
Routes distributed region reads to datanodes |
instance/exemplar |
src/frontend/src/instance/exemplar.rs |
Resolves and authorizes selector branches before bounded exemplar-table scans |
instance/* |
src/frontend/src/instance/ |
Per-protocol handlers (influxdb.rs, promql.rs, otlp/, jaeger.rs, logs.rs, prom_store.rs, ...) |
frontend |
src/frontend/src/frontend.rs |
Frontend lifecycle wrapper (FrontendOptions, start/shutdown) |
server |
src/frontend/src/server.rs |
Services: builds and wires the protocol servers |
heartbeat |
src/frontend/src/heartbeat.rs |
Heartbeat to metasrv; handles suspend / cache invalidation |
service_config |
src/frontend/src/service_config/ |
Per-protocol option structs |
Shared frontend cache counters live in src/common/frontend/src/metrics.rs,
including caches owned by pipeline and protocol handlers.
Request lifecycles
-
SQL query (
instance.rs):do_query_innerhandles parsing, interceptors, permission checks, timeout/cancellation, and delegates planning/execution toStatementExecutor. Distributed scans enter throughregion_query.rs. MySQL sessions without a selected database validate explicit scopes before planning or executing SQL, including prepared statements; unqualified targets fail without falling back topublic. -
Exemplar query (
instance/exemplar.rs): authorize all selector branches before callingPromQueryInterceptor::pre_execute_exemplarsfor each resolved exemplar table, with that table's schema context, before scanning it. -
Logical-table batch DDL (
instance.rs): dedicated SQL handler → bounded parse/interception → all-member permissions → operator CREATE preparation → one logical-table batch procedure. Keep pre-submission validation side-effect free. -
Insert (
instance/grpc.rs):handle_inserts/handle_row_inserts→check_permission→operator'sInserter(schema validation, optional auto-create, partition routing, meter admission) → localRegionServer(standalone) or RPC to datanodes (distributed).FrontendBuilderconfigures the operator write dispatcher with frontend retry options for distributed inserts, deletes, and bulk writes. Arrow bulk inserts pass the request channel toInserterand check meter admission for each nonempty batch. -
Finite ingestion requests split internally admit their total rows per database before dispatch (
operator::insert::admit_write/admit_row_insert_batches). The returned context covers chunks and derived writes while preserving WCU accounting and the original protocol channel. -
Internal gRPC listeners mark requests with
Channel::Internalin middleware (server.rs), including requests handled by Enterprise Flight wrappers. -
Flight bulk insert (
instance/grpc.rs): initializes on the first batch after the lazy schema handshake; checks permissions and reconciles missing columns throughInserteronce per stream, then reuses the refreshed table. -
OTLP trace lookup tables (
instance/otlp/trace_ingest.rs):Instanceowns a shared cache of confirmed service/operation writes, which bypass batching.otlp.trace_aux_cache_sizesets its estimated byte budget at startup; zero disables it. Async main writes retain completion results; a frontend task writes auxiliary rows only for confirmed chunks before populating the cache. Cold async trace requests reserve a shared auxiliary slot before main writes, bounded separately by the table batcher'smax_inflight_requests; confirmed cache hits bypass this admission. The slot lasts through auxiliary completion. These tables are ingestion-managed; manual mutation requires restarting serving frontends. -
OTLP metrics (
instance/otlp.rs): authorize source tables before schema preflight can reject points or expose their schemas. Skip and count exemplars when metric-engine ingestion is disabled or the source is an ordinary table, retaining metric rows. Authorize retained exemplar writes before accessing the exemplar destination or reserving expansion memory. Reserve exemplar expansion with the transport's memory limiter before building rows; retain the permit throughWriteProgressuntil writes complete. Expansion must not wait while the request holds transport memory permits. -
Logical-table batching (
instance/logical_batcher.rs):Servicesinitializes one shared batcher for opted-in Prom remote-write (HTTP and PostgreSQL COPY) and OTLP metric-engine writes. Downstream routers can initialize it when enabling replacement endpoints.Instance::handle_otlp_metric_row_insertsshares OTLP eligibility and dispatch between converters and falls back for incompatible tables. The schema adapter holds a weak instance reference to avoid an ownership cycle. -
Table batching (
instance/builder.rs): protocol entry points opt in throughQueryContext. The primary inserter prepares eligible ordinary-table writes forservers::batcher::table::TablePendingRowsBatcher. A separate execution-only inserter, with no batcher attached, sends the prepared bulk writes to datanodes, avoiding recursive batching. The batcher handles successful-write Flow notifications.
Public surface
Instance(instance.rs) — the business-logic container.Frontend(frontend.rs) — lifecycle wrapper aroundInstance+ servers + heartbeat.- Created from
cmd:src/cmd/src/frontend.rs(distributed) andsrc/cmd/src/standalone.rs(standalone, with embedded datanode).
When you change X, also touch Y
servershandler traits: a new/changed protocol handler requires the matchingimplhere.operatorInserter/Deleter orqueryQueryEngine API: update the call sites ininstance.rs/instance/grpc.rs.session::QueryContext: new context fields thread through most handlers.sqlstatements: new statement kinds need handling inquery_statement.
Testing
cargo nextest run -p frontend
Gotchas
- Keep the frontend/servers split straight: wire format and network live in
servers; permissions, planning, and routing live here. - A statement whose tables are known only at execution time (
COPY DATABASE) resolves them first, authorizes every resolved table, then executes exactly the authorized set. Do not authorize such a statement with empty targets. - Built-in permission action names are a contract with external
PermissionCheckerimplementations. Define each as a constant insrc/auth/src/permission.rsand list it inALL_ACTIONS; do not build them from string literals at call sites. A new action or access-mode requirement on an existing statement or endpoint is a user-visible permission change. - Standalone vs distributed diverge in datanode access (the
standalonecrate's localRegionServeradapter vsNodeClientsRPC), MetaClient usage, and whether heartbeat matters. The builder defaults local invalidation to its layered cache registry; callers may supply a custom invalidator.
Maintenance contract
Update this file when you add a protocol handler, change the query/insert
lifecycle, or change how Instance is constructed or wired to servers.