Files
greptimedb/src/frontend/AGENTS.md
T
shuiyisong 0a93810c23 fix: improve exemplar handling and batch memory admission (#10363)
* chore: update README

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): reserve memory for table batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): prefix the labels column

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): hide exemplar table from metadata

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): reconcile source tags before writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): prune tags and retain source IDs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): record slow queries

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars when source writes fail

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): skip exemplars when source writes fail

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: account for held memory in batch admission

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: config comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: simplify exemplar and memory tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: deduplicate exemplar setup

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-16 11:55:20 +00:00

9.6 KiB

frontend — Agent & Contributor Guide

Navigation aid for src/frontend. Keep it short and point to code. Paths are relative to the repo root.

Repo-wide rules that apply here: .agents/architecture-invariants.md.

What this crate does

Frontend is the request entry point and orchestration layer. It accepts multi-protocol requests (gRPC, HTTP, MySQL, PostgreSQL, InfluxDB, OTLP, Jaeger, Prometheus, OpenTSDB), checks permissions, parses/plans SQL, and dispatches: reads go to the query engine (query crate), writes go to the inserter/deleter (operator crate).

Boundary with servers: the servers crate implements the wire protocols and network I/O; frontend provides the business logic by implementing handler traits (SqlQueryHandler, GrpcQueryHandler, InfluxdbLineProtocolHandler, ...). In standalone mode the standalone crate bridges the frontend to an embedded datanode RegionServer; in distributed mode the frontend talks to remote datanodes via operator/client.

Module map

Module Path Purpose
instance src/frontend/src/instance.rs Instance: the core handler; implements SqlQueryHandler, PrometheusHandler, etc.
instance/builder src/frontend/src/instance/builder.rs FrontendBuilder assembles Instance from its dependencies
instance/grpc src/frontend/src/instance/grpc.rs GrpcQueryHandler: insert/delete/query/promql over gRPC
instance/schema_selection src/frontend/src/instance/schema_selection.rs SQL database selection and authorization for sessions without a selected schema
instance/region_query src/frontend/src/instance/region_query.rs Routes distributed region reads to datanodes
instance/exemplar src/frontend/src/instance/exemplar.rs Resolves and authorizes selector branches before bounded exemplar-table scans
instance/* src/frontend/src/instance/ Per-protocol handlers (influxdb.rs, promql.rs, otlp/, jaeger.rs, logs.rs, prom_store.rs, ...)
frontend src/frontend/src/frontend.rs Frontend lifecycle wrapper (FrontendOptions, start/shutdown)
server src/frontend/src/server.rs Services: builds and wires the protocol servers
heartbeat src/frontend/src/heartbeat.rs Heartbeat to metasrv; handles suspend / cache invalidation
service_config src/frontend/src/service_config/ Per-protocol option structs

Shared frontend cache counters live in src/common/frontend/src/metrics.rs, including caches owned by pipeline and protocol handlers.

Request lifecycles

  • SQL query (instance.rs): do_query_inner handles parsing, interceptors, permission checks, timeout/cancellation, and delegates planning/execution to StatementExecutor. Distributed scans enter through region_query.rs. MySQL sessions without a selected database validate explicit scopes before planning or executing SQL, including prepared statements; unqualified targets fail without falling back to public.

  • Exemplar query (instance/exemplar.rs): authorize all selector branches before resolving logical schemas or calling PromQueryInterceptor::pre_execute_exemplars. Exact metric selectors project the union of current logical-table tags; broad or unresolved selectors retain all tags. Responses include the stored __table_id and __tsid as string labels. The process ticket retains the configured slow-query timer through collection; its query text includes the expression and inclusive nanosecond scan bounds.

  • Logical-table batch DDL (instance.rs): dedicated SQL handler → bounded parse/interception → all-member permissions → operator CREATE preparation → one logical-table batch procedure. Keep pre-submission validation side-effect free.

  • Insert (instance/grpc.rs): handle_inserts / handle_row_inserts → check_permission → operator's Inserter (schema validation, optional auto-create, partition routing, meter admission) → local RegionServer (standalone) or RPC to datanodes (distributed). FrontendBuilder configures the operator write dispatcher with frontend retry options for distributed inserts, deletes, and bulk writes. Arrow bulk inserts pass the request channel to Inserter and check meter admission for each nonempty batch.

  • Finite ingestion requests split internally admit their total rows per database before dispatch (operator::insert::admit_write / admit_row_insert_batches). The returned context covers chunks and derived writes while preserving WCU accounting and the original protocol channel.

  • Internal gRPC listeners mark requests with Channel::Internal in middleware (server.rs), including requests handled by Enterprise Flight wrappers.

  • Flight bulk insert (instance/grpc.rs): initializes on the first batch after the lazy schema handshake; checks permissions and reconciles missing columns through Inserter once per stream, then reuses the refreshed table.

  • OTLP trace lookup tables (instance/otlp/trace_ingest.rs): Instance owns a shared cache of confirmed service/operation writes, which bypass batching. otlp.trace_aux_cache_size sets its estimated byte budget at startup; zero disables it. Async main writes retain completion results; a frontend task writes auxiliary rows only for confirmed chunks before populating the cache. Cold async trace requests reserve a shared auxiliary slot before main writes, bounded separately by the table batcher's max_inflight_requests; confirmed cache hits bypass this admission. The slot lasts through auxiliary completion. These tables are ingestion-managed; manual mutation requires restarting serving frontends.

  • OTLP metrics (instance/otlp.rs): authorize source tables before schema preflight can reject points or expose their schemas. Skip and count exemplars when metric-engine ingestion is disabled or the source is an ordinary table, retaining metric rows. Authorize retained exemplar writes before accessing the exemplar destination or reserving expansion memory. Reserve exemplar expansion with the transport's memory limiter before building rows; retain the permit through WriteProgress until writes complete. Expansion must not wait while the request holds transport memory permits. Confirm source metric writes before inserting their exemplars; unrelated metric writes remain concurrent and their completions must still be observed.

  • Logical-table batching (instance/logical_batcher.rs): Services initializes one shared batcher for opted-in Prom remote-write (HTTP and PostgreSQL COPY) and OTLP metric-engine writes. Downstream routers can initialize it when enabling replacement endpoints. Instance::handle_otlp_metric_row_inserts shares OTLP eligibility and dispatch between converters and falls back for incompatible tables. The schema adapter holds a weak instance reference to avoid an ownership cycle.

  • Table batching (instance/builder.rs): protocol entry points opt in through QueryContext. The primary inserter prepares eligible ordinary-table writes for servers::batcher::table::TablePendingRowsBatcher. A separate execution-only inserter, with no batcher attached, sends the prepared bulk writes to datanodes, avoiding recursive batching. The batcher handles successful-write Flow notifications. FrontendBuilder creates the write memory limiter shared by conversion admission and the protocol servers wired by Services. Conversion admission includes the request's live transport and retained decoder reservations in its fit check; previously queued Arrow batches remain transient contention.

Public surface

  • Instance (instance.rs) — the business-logic container.
  • Frontend (frontend.rs) — lifecycle wrapper around Instance + servers + heartbeat.
  • Created from cmd: src/cmd/src/frontend.rs (distributed) and src/cmd/src/standalone.rs (standalone, with embedded datanode).

When you change X, also touch Y

  • servers handler traits: a new/changed protocol handler requires the matching impl here.
  • operator Inserter/Deleter or query QueryEngine API: update the call sites in instance.rs / instance/grpc.rs.
  • session::QueryContext: new context fields thread through most handlers.
  • sql statements: new statement kinds need handling in query_statement.

Testing

cargo nextest run -p frontend

Gotchas

  • Keep the frontend/servers split straight: wire format and network live in servers; permissions, planning, and routing live here.
  • A statement whose tables are known only at execution time (COPY DATABASE) resolves them first, authorizes every resolved table, then executes exactly the authorized set. Do not authorize such a statement with empty targets.
  • Built-in permission action names are a contract with external PermissionChecker implementations. Define each as a constant in src/auth/src/permission.rs and list it in ALL_ACTIONS; do not build them from string literals at call sites. A new action or access-mode requirement on an existing statement or endpoint is a user-visible permission change.
  • Standalone vs distributed diverge in datanode access (the standalone crate's local RegionServer adapter vs NodeClients RPC), MetaClient usage, and whether heartbeat matters. The builder defaults local invalidation to its layered cache registry; callers may supply a custom invalidator.

Maintenance contract

Update this file when you add a protocol handler, change the query/insert lifecycle, or change how Instance is constructed or wired to servers.