* chore: update README Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(ingest): reserve memory for table batches Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): prefix the labels column Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): hide exemplar table from metadata Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): reconcile source tags before writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(exemplar): prune tags and retain source IDs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): record slow queries Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): skip exemplars when source writes fail Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): skip exemplars when source writes fail Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: account for held memory in batch admission Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: config comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: simplify exemplar and memory tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: deduplicate exemplar setup Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>
9.6 KiB
frontend — Agent & Contributor Guide
Navigation aid for src/frontend. Keep it short and point to code. Paths are
relative to the repo root.
Repo-wide rules that apply here: .agents/architecture-invariants.md.
What this crate does
Frontend is the request entry point and orchestration layer. It accepts
multi-protocol requests (gRPC, HTTP, MySQL, PostgreSQL, InfluxDB, OTLP, Jaeger,
Prometheus, OpenTSDB), checks permissions, parses/plans SQL, and dispatches:
reads go to the query engine (query crate), writes go to the inserter/deleter
(operator crate).
Boundary with servers: the servers crate implements the wire protocols and
network I/O; frontend provides the business logic by implementing handler
traits (SqlQueryHandler, GrpcQueryHandler, InfluxdbLineProtocolHandler,
...). In standalone mode the standalone crate bridges the frontend to an
embedded datanode RegionServer; in distributed mode the frontend talks to
remote datanodes via operator/client.
Module map
| Module | Path | Purpose |
|---|---|---|
instance |
src/frontend/src/instance.rs |
Instance: the core handler; implements SqlQueryHandler, PrometheusHandler, etc. |
instance/builder |
src/frontend/src/instance/builder.rs |
FrontendBuilder assembles Instance from its dependencies |
instance/grpc |
src/frontend/src/instance/grpc.rs |
GrpcQueryHandler: insert/delete/query/promql over gRPC |
instance/schema_selection |
src/frontend/src/instance/schema_selection.rs |
SQL database selection and authorization for sessions without a selected schema |
instance/region_query |
src/frontend/src/instance/region_query.rs |
Routes distributed region reads to datanodes |
instance/exemplar |
src/frontend/src/instance/exemplar.rs |
Resolves and authorizes selector branches before bounded exemplar-table scans |
instance/* |
src/frontend/src/instance/ |
Per-protocol handlers (influxdb.rs, promql.rs, otlp/, jaeger.rs, logs.rs, prom_store.rs, ...) |
frontend |
src/frontend/src/frontend.rs |
Frontend lifecycle wrapper (FrontendOptions, start/shutdown) |
server |
src/frontend/src/server.rs |
Services: builds and wires the protocol servers |
heartbeat |
src/frontend/src/heartbeat.rs |
Heartbeat to metasrv; handles suspend / cache invalidation |
service_config |
src/frontend/src/service_config/ |
Per-protocol option structs |
Shared frontend cache counters live in src/common/frontend/src/metrics.rs,
including caches owned by pipeline and protocol handlers.
Request lifecycles
-
SQL query (
instance.rs):do_query_innerhandles parsing, interceptors, permission checks, timeout/cancellation, and delegates planning/execution toStatementExecutor. Distributed scans enter throughregion_query.rs. MySQL sessions without a selected database validate explicit scopes before planning or executing SQL, including prepared statements; unqualified targets fail without falling back topublic. -
Exemplar query (
instance/exemplar.rs): authorize all selector branches before resolving logical schemas or callingPromQueryInterceptor::pre_execute_exemplars. Exact metric selectors project the union of current logical-table tags; broad or unresolved selectors retain all tags. Responses include the stored__table_idand__tsidas string labels. The process ticket retains the configured slow-query timer through collection; its query text includes the expression and inclusive nanosecond scan bounds. -
Logical-table batch DDL (
instance.rs): dedicated SQL handler → bounded parse/interception → all-member permissions → operator CREATE preparation → one logical-table batch procedure. Keep pre-submission validation side-effect free. -
Insert (
instance/grpc.rs):handle_inserts/handle_row_inserts→check_permission→operator'sInserter(schema validation, optional auto-create, partition routing, meter admission) → localRegionServer(standalone) or RPC to datanodes (distributed).FrontendBuilderconfigures the operator write dispatcher with frontend retry options for distributed inserts, deletes, and bulk writes. Arrow bulk inserts pass the request channel toInserterand check meter admission for each nonempty batch. -
Finite ingestion requests split internally admit their total rows per database before dispatch (
operator::insert::admit_write/admit_row_insert_batches). The returned context covers chunks and derived writes while preserving WCU accounting and the original protocol channel. -
Internal gRPC listeners mark requests with
Channel::Internalin middleware (server.rs), including requests handled by Enterprise Flight wrappers. -
Flight bulk insert (
instance/grpc.rs): initializes on the first batch after the lazy schema handshake; checks permissions and reconciles missing columns throughInserteronce per stream, then reuses the refreshed table. -
OTLP trace lookup tables (
instance/otlp/trace_ingest.rs):Instanceowns a shared cache of confirmed service/operation writes, which bypass batching.otlp.trace_aux_cache_sizesets its estimated byte budget at startup; zero disables it. Async main writes retain completion results; a frontend task writes auxiliary rows only for confirmed chunks before populating the cache. Cold async trace requests reserve a shared auxiliary slot before main writes, bounded separately by the table batcher'smax_inflight_requests; confirmed cache hits bypass this admission. The slot lasts through auxiliary completion. These tables are ingestion-managed; manual mutation requires restarting serving frontends. -
OTLP metrics (
instance/otlp.rs): authorize source tables before schema preflight can reject points or expose their schemas. Skip and count exemplars when metric-engine ingestion is disabled or the source is an ordinary table, retaining metric rows. Authorize retained exemplar writes before accessing the exemplar destination or reserving expansion memory. Reserve exemplar expansion with the transport's memory limiter before building rows; retain the permit throughWriteProgressuntil writes complete. Expansion must not wait while the request holds transport memory permits. Confirm source metric writes before inserting their exemplars; unrelated metric writes remain concurrent and their completions must still be observed. -
Logical-table batching (
instance/logical_batcher.rs):Servicesinitializes one shared batcher for opted-in Prom remote-write (HTTP and PostgreSQL COPY) and OTLP metric-engine writes. Downstream routers can initialize it when enabling replacement endpoints.Instance::handle_otlp_metric_row_insertsshares OTLP eligibility and dispatch between converters and falls back for incompatible tables. The schema adapter holds a weak instance reference to avoid an ownership cycle. -
Table batching (
instance/builder.rs): protocol entry points opt in throughQueryContext. The primary inserter prepares eligible ordinary-table writes forservers::batcher::table::TablePendingRowsBatcher. A separate execution-only inserter, with no batcher attached, sends the prepared bulk writes to datanodes, avoiding recursive batching. The batcher handles successful-write Flow notifications.FrontendBuildercreates the write memory limiter shared by conversion admission and the protocol servers wired byServices. Conversion admission includes the request's live transport and retained decoder reservations in its fit check; previously queued Arrow batches remain transient contention.
Public surface
Instance(instance.rs) — the business-logic container.Frontend(frontend.rs) — lifecycle wrapper aroundInstance+ servers + heartbeat.- Created from
cmd:src/cmd/src/frontend.rs(distributed) andsrc/cmd/src/standalone.rs(standalone, with embedded datanode).
When you change X, also touch Y
servershandler traits: a new/changed protocol handler requires the matchingimplhere.operatorInserter/Deleter orqueryQueryEngine API: update the call sites ininstance.rs/instance/grpc.rs.session::QueryContext: new context fields thread through most handlers.sqlstatements: new statement kinds need handling inquery_statement.
Testing
cargo nextest run -p frontend
Gotchas
- Keep the frontend/servers split straight: wire format and network live in
servers; permissions, planning, and routing live here. - A statement whose tables are known only at execution time (
COPY DATABASE) resolves them first, authorizes every resolved table, then executes exactly the authorized set. Do not authorize such a statement with empty targets. - Built-in permission action names are a contract with external
PermissionCheckerimplementations. Define each as a constant insrc/auth/src/permission.rsand list it inALL_ACTIONS; do not build them from string literals at call sites. A new action or access-mode requirement on an existing statement or endpoint is a user-visible permission change. - Standalone vs distributed diverge in datanode access (the
standalonecrate's localRegionServeradapter vsNodeClientsRPC), MetaClient usage, and whether heartbeat matters. The builder defaults local invalidation to its layered cache registry; callers may supply a custom invalidator.
Maintenance contract
Update this file when you add a protocol handler, change the query/insert
lifecycle, or change how Instance is constructed or wired to servers.