common.rs held the wire messages, what a node sends and when, and what
a remote scan's caller tracks. Each now has its own module:
- message: CustomMessage and Metrics, the protocol between nodes.
- node_messages: NodeMessages, NodeRole and MetricsInfo, the sending side.
- remote_scan: RemoteScanMetrics, RemoteScanState and NodeProfiles, the
receiving side.
FlightMessage, the decoder's output, moves into decoder. Code moves
without changes; callers import from the new modules.
- RemoteScanExec keeps its scan stats, peak memory, partial error and
node profiles in one RemoteScanState instead of four fields, four
builders and four arguments; a cluster leader's NodeRole reuses it.
- DistributeAnalyzeExec builds its batch in a method that reads the
node's name and address from the profile.
- metric_value no longer returns an Option: convert_metrics already
drops the timestamps it skipped.
- Replace stale and multi-line comments.
Replace the flat PreCustomMessage list with NodeMessages: what every
node sends (its scan stats, memory peak and, for EXPLAIN ANALYZE, its
profile) plus a NodeRole saying whether it is a follower or a super
cluster's cluster leader, which also forwards what its remote scan
collected. first() and last() list what goes out before the first batch
and after the last one, so is_early_emit and the three remote-scan
visitors are gone. The messages on the wire are unchanged.
PreCustomMessage variants held Option values and get_custom_message
skipped None at encode time. The handler now pushes a message only when
it has a value, so each variant carries exactly what it sends.
Followers now always send their profile with EXPLAIN ANALYZE, and the leader
builds the four text columns from it, so the flat text metrics, the MetricsRef
forwarding and the SearchInfo.analyze_profile proto field are gone. The flag
only decides whether the leader adds the profile column.
ProfileMetric gains `display`, DataFusion's own formatting of the value, and
metrics are aggregated and ordered the way the text plan does, so the rendered
text matches DataFusion's. Memory gauges are now bytes, and `detail` keeps the
operator line unchanged.
ProfileInputs is either the operators that ran on the node or, for a
remote scan, the nodes it called; the two lists can no longer both be
set. It is flattened, so the JSON still has a children or a remote key;
a leaf now writes children: [].
The remote decoder already takes each node's profile out before storing
its metrics in cluster_metrics, so the entries MetricsRef forwards never
carry one.
The leader reads req.analyze_profile and passes it as an argument, so the
AnalyzeProfile session-config extension and its setup in
SearchContextBuilder are gone. DataFusion plans the outer AnalyzeExec
outside the physical optimizer rules, so this rewrite cannot move into a
rule.
- Drop the per-node error, first-batch time and peak memory recording from
the remote decoder; the profile keeps the node's plan, its stream counters
and whether it was skipped.
- Drop profile fields nothing reads: region, fetch_ns, assigned_files on a
remote node, the per-metric max, and the profile version.
- Use DataFusion's one_line display for the operator detail instead of a
local wrapper.
- NodeProfileRecorder now owns its node's profile and publishes it when
dropped; rows, batches, bytes, fetch/decode time and scan stats come from
the counters FlightDecoderStream already keeps, so the per-batch lock and
the duplicate tallies are gone. A disabled (default) recorder publishes
nothing, which removes the Option checks at every call site.
- build_profile walks the plan in the search crate and reads RemoteScanExec
directly; the flight crate keeps only the profile types.
- DistributeAnalyzeExec passes the profile start time as an Option instead
of a (start, duration) tuple.
With query.analyze_profile set, followers attach a structured profile to
their metrics message, each RemoteScanExec records per-node timing, rows,
bytes and scan stats, and DistributeAnalyzeExec adds a JSON `profile`
column on the first row. Without the flag the output stays the four text
columns, so plain EXPLAIN ANALYZE does not grow.
haohuaijin
Loakesh Indiran
Shrinath Rao
Manul PatelandManul
uddhavdaveandgithub-actions[bot]
Harsh Mahajan
Neha Pandgithub-actions[bot]
Hengfei Yang
Subhra264andgithub-actions[bot]
Jason LeiandClaude Opus 5.5
Sai Nikhilandgithub-actions[bot]