Files
greptimedb/src/frontend/AGENTS.md
T
Ning SunandLei, HUANG b30e002cef feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN (#10092)
* feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT promwrite [, version 2] [, <options>])
ingests one Prometheus Remote Write request body over the postgres wire
protocol — a GreptimeDB-specific, binary COPY format (wire format code
1). The payload is byte-for-byte the body a remote-write client POSTs
to /v1/prometheus/write, compression included: buffered until CopyDone
and completed through the exact ingestion core that endpoint runs.

To share that core, the HTTP handlers' ingestion bodies are extracted
into axum-free functions (ingest_remote_write_v1/v2 plus the context
extension stamping), and the HTTP routes become thin wrappers — their
behavior is unchanged, and the copy layer runs the identical decode,
validation, conversion, batched/prepared write paths, per-batch
write-progress accounting and context extensions, under the
connection's regular postgres context.

The WITH options carry what the HTTP endpoint takes as its
content-type/-encoding and query parameters; session-provided values
(database, user) are not repeated:
- version 1|2 (default 1), matching the content-type's proto
- zstd, matching Content-Encoding: zstd (snappy by default)
- physical_table, matching the ?physical_table= parameter
- pipeline_name/pipeline_version, matching the pipeline headers
  (v1 only; for v2 they are accepted and ignored, exactly like the
  HTTP route ignores them)

Table-less like the other ingestion formats; one COPY statement is one
bounded request (shared 64 MiB limit); the row count reports written
rows (v1) or written samples plus histograms (v2). The flush enforces
the frontend's shared request-memory quota (max_in_flight_write_bytes)
— the same limiter the HTTP ingestion protocols use, so COPY cannot
bypass the aggregate budget. Wiring: PostgresServer::with_prom_remote_write
carries the handler, metric-engine flag, validation mode and shared
memory limiter.

e2e: v1 body chunked across CopyData boundaries; v2 samples with
readback; physical_table option routing; garbage-body decode failure
with connection reuse; a tiny-quota regression where a compressed body
whose decompressed size exceeds the budget fails like the HTTP route.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(servers): propagate COPY body reservations and test write accounting

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): share Prometheus logical batching with COPY ingestion

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): narrow Prometheus COPY scope and share write outcomes

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2026-11-16 02:45:34 +00:00

8.8 KiB

frontend — Agent & Contributor Guide

Navigation aid for src/frontend. Keep it short and point to code. Paths are relative to the repo root.

Repo-wide rules that apply here: .agents/architecture-invariants.md.

What this crate does

Frontend is the request entry point and orchestration layer. It accepts multi-protocol requests (gRPC, HTTP, MySQL, PostgreSQL, InfluxDB, OTLP, Jaeger, Prometheus, OpenTSDB), checks permissions, parses/plans SQL, and dispatches: reads go to the query engine (query crate), writes go to the inserter/deleter (operator crate).

Boundary with servers: the servers crate implements the wire protocols and network I/O; frontend provides the business logic by implementing handler traits (SqlQueryHandler, GrpcQueryHandler, InfluxdbLineProtocolHandler, ...). In standalone mode the standalone crate bridges the frontend to an embedded datanode RegionServer; in distributed mode the frontend talks to remote datanodes via operator/client.

Module map

Module Path Purpose
instance src/frontend/src/instance.rs Instance: the core handler; implements SqlQueryHandler, PrometheusHandler, etc.
instance/builder src/frontend/src/instance/builder.rs FrontendBuilder assembles Instance from its dependencies
instance/grpc src/frontend/src/instance/grpc.rs GrpcQueryHandler: insert/delete/query/promql over gRPC
instance/schema_selection src/frontend/src/instance/schema_selection.rs SQL database selection and authorization for sessions without a selected schema
instance/region_query src/frontend/src/instance/region_query.rs Routes distributed region reads to datanodes
instance/exemplar src/frontend/src/instance/exemplar.rs Resolves and authorizes selector branches before bounded exemplar-table scans
instance/* src/frontend/src/instance/ Per-protocol handlers (influxdb.rs, promql.rs, otlp/, jaeger.rs, logs.rs, prom_store.rs, ...)
frontend src/frontend/src/frontend.rs Frontend lifecycle wrapper (FrontendOptions, start/shutdown)
server src/frontend/src/server.rs Services: builds and wires the protocol servers
heartbeat src/frontend/src/heartbeat.rs Heartbeat to metasrv; handles suspend / cache invalidation
service_config src/frontend/src/service_config/ Per-protocol option structs

Shared frontend cache counters live in src/common/frontend/src/metrics.rs, including caches owned by pipeline and protocol handlers.

Request lifecycles

  • SQL query (instance.rs): do_query_inner handles parsing, interceptors, permission checks, timeout/cancellation, and delegates planning/execution to StatementExecutor. Distributed scans enter through region_query.rs. MySQL sessions without a selected database validate explicit scopes before planning or executing SQL, including prepared statements; unqualified targets fail without falling back to public.

  • Exemplar query (instance/exemplar.rs): authorize all selector branches before calling PromQueryInterceptor::pre_execute_exemplars for each resolved exemplar table, with that table's schema context, before scanning it.

  • Logical-table batch DDL (instance.rs): dedicated SQL handler → bounded parse/interception → all-member permissions → operator CREATE preparation → one logical-table batch procedure. Keep pre-submission validation side-effect free.

  • Insert (instance/grpc.rs): handle_inserts / handle_row_inserts → check_permission → operator's Inserter (schema validation, optional auto-create, partition routing, meter admission) → local RegionServer (standalone) or RPC to datanodes (distributed). FrontendBuilder configures the operator write dispatcher with frontend retry options for distributed inserts, deletes, and bulk writes. Arrow bulk inserts pass the request channel to Inserter and check meter admission for each nonempty batch.

  • Finite ingestion requests split internally admit their total rows per database before dispatch (operator::insert::admit_write / admit_row_insert_batches). The returned context covers chunks and derived writes while preserving WCU accounting and the original protocol channel.

  • Internal gRPC listeners mark requests with Channel::Internal in middleware (server.rs), including requests handled by Enterprise Flight wrappers.

  • Flight bulk insert (instance/grpc.rs): initializes on the first batch after the lazy schema handshake; checks permissions and reconciles missing columns through Inserter once per stream, then reuses the refreshed table.

  • OTLP trace lookup tables (instance/otlp/trace_ingest.rs): Instance owns a shared cache of confirmed service/operation writes, which bypass batching. otlp.trace_aux_cache_size sets its estimated byte budget at startup; zero disables it. Async main writes retain completion results; a frontend task writes auxiliary rows only for confirmed chunks before populating the cache. Cold async trace requests reserve a shared auxiliary slot before main writes, bounded separately by the table batcher's max_inflight_requests; confirmed cache hits bypass this admission. The slot lasts through auxiliary completion. These tables are ingestion-managed; manual mutation requires restarting serving frontends.

  • OTLP metrics (instance/otlp.rs): authorize source tables before schema preflight can reject points or expose their schemas. Skip and count exemplars when metric-engine ingestion is disabled or the source is an ordinary table, retaining metric rows. Authorize retained exemplar writes before accessing the exemplar destination or reserving expansion memory. Reserve exemplar expansion with the transport's memory limiter before building rows; retain the permit through WriteProgress until writes complete. Expansion must not wait while the request holds transport memory permits.

  • Logical-table batching (instance/logical_batcher.rs): Services initializes one shared batcher for opted-in Prom remote-write (HTTP and PostgreSQL COPY) and OTLP metric-engine writes. Downstream routers can initialize it when enabling replacement endpoints. Instance::handle_otlp_metric_row_inserts shares OTLP eligibility and dispatch between converters and falls back for incompatible tables. The schema adapter holds a weak instance reference to avoid an ownership cycle.

  • Table batching (instance/builder.rs): protocol entry points opt in through QueryContext. The primary inserter prepares eligible ordinary-table writes for servers::batcher::table::TablePendingRowsBatcher. A separate execution-only inserter, with no batcher attached, sends the prepared bulk writes to datanodes, avoiding recursive batching. The batcher handles successful-write Flow notifications.

Public surface

  • Instance (instance.rs) — the business-logic container.
  • Frontend (frontend.rs) — lifecycle wrapper around Instance + servers + heartbeat.
  • Created from cmd: src/cmd/src/frontend.rs (distributed) and src/cmd/src/standalone.rs (standalone, with embedded datanode).

When you change X, also touch Y

  • servers handler traits: a new/changed protocol handler requires the matching impl here.
  • operator Inserter/Deleter or query QueryEngine API: update the call sites in instance.rs / instance/grpc.rs.
  • session::QueryContext: new context fields thread through most handlers.
  • sql statements: new statement kinds need handling in query_statement.

Testing

cargo nextest run -p frontend

Gotchas

  • Keep the frontend/servers split straight: wire format and network live in servers; permissions, planning, and routing live here.
  • A statement whose tables are known only at execution time (COPY DATABASE) resolves them first, authorizes every resolved table, then executes exactly the authorized set. Do not authorize such a statement with empty targets.
  • Built-in permission action names are a contract with external PermissionChecker implementations. Define each as a constant in src/auth/src/permission.rs and list it in ALL_ACTIONS; do not build them from string literals at call sites. A new action or access-mode requirement on an existing statement or endpoint is a user-visible permission change.
  • Standalone vs distributed diverge in datanode access (the standalone crate's local RegionServer adapter vs NodeClients RPC), MetaClient usage, and whether heartbeat matters. The builder defaults local invalidation to its layered cache registry; callers may supply a custom invalidator.

Maintenance contract

Update this file when you add a protocol handler, change the query/insert lifecycle, or change how Instance is constructed or wired to servers.