Files
greptimedb/src/flow/AGENTS.md
T
discord9 7c7132ea65 refactor(flow): execute streaming flows with DataFusion (#8976)
* test(mito2): cover regex inverted index pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(flow): execute streaming flows with DataFusion

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(flow): remove legacy streaming runtime

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): avoid retrying stateless sink inserts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): align stateless writes with sink schema

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): reject stale stateless source schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): validate stateless flow routing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* Revert "test(mito2): cover regex inverted index pruning"

This reverts commit 79e96ac745.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): preserve source timestamps in stateless flows

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): address stateless review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): recover stateless flows after schema changes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): serialize schema rebuilds and validate retained sources

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): verify streaming recovery and schema changes through SQL

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): cool down failed schema rebuilds

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): explain legacy aggregate recreation requirements

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): adapt stateless provider downcast to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-17 06:44:01 +00:00

3.1 KiB

flow — Agent & Contributor Guide

Navigation aid for src/flow. Keep it short and point to code. Paths are relative to the repository root.

Repo-wide rules that apply to here: .agents/architecture-invariants.md.

What this crate does

Flownode provides two flow profiles:

  • Streaming mode: a single stateless DataFusion query profile. Each mirror insert is materialized as a transient input table, executed against the retained logical plan, and written directly to the sink through batching_mode::frontend_client::FrontendClient.
  • Batching mode: splits data into time windows and periodically runs aggregation SQL through the frontend, writing results back to a sink table.

Users cannot select a mode directly: flow_type is a reserved internal option. StatementExecutor::determine_flow_type in src/operator/src/statement/ddl.rs owns current mode selection. FlowDualEngine defaults missing internal flow_type metadata to batching for compatibility. Read those paths before changing routing rules.

Module map

Module Path Purpose
engine src/flow/src/engine.rs FlowEngine lifecycle contract
adapter src/flow/src/adapter.rs, src/flow/src/adapter/ Stateless streaming execution, dual-engine dispatch, shared source schema/default normalization, and sink creation
batching_mode src/flow/src/batching_mode.rs, src/flow/src/batching_mode/ BatchingEngine, scheduling, time windows, frontend client, and checkpoints
repr src/flow/src/repr.rs Shared row and relation schema representations
server src/flow/src/server.rs gRPC Flow service and flownode builders
heartbeat src/flow/src/heartbeat.rs Reports flownode state/stats to metasrv

Flow metadata lives in common-meta, not here: src/common/meta/src/key/flow/ and src/common/meta/src/ddl/create_flow.rs.

Data flow

Frontend → Flownode (gRPC) → FlowService (server.rs) → FlowDualEngine (adapter/flownode_impl.rs) routes by FlowType:

  • Streaming: mirror rows are normalized against the source schema, evaluated by the retained stateless DataFusion plan, and sent to the sink via FrontendClient.
  • Batching: marks dirty windows; a task later runs aggregation SQL via the frontend client and writes the sink table.

FlowType::Streaming and mirror routing are shared contracts; preserve both when changing dispatch. Preserve batching behavior and the shared schema, default, and sink-creation helpers.

Public surface

  • gRPC Flow service in src/flow/src/server.rs.
  • FlowEngine trait in src/flow/src/engine.rs.
  • Started from the cmd crate via FlownodeBuilder / FlownodeInstance.

Testing

cargo nextest run -p flow

Stateless streaming tests are under src/flow/src/adapter/stateless.rs.

Gotchas

  • Batching and streaming differ in latency and state. Confirm the selected mode before reasoning about a flow.
  • Streaming is finite and stateless per mirror request; it has no worker graph, background runtime, or replay state.
  • Internal flow timestamps and table time columns are separate contracts.