mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-06-03 05:40:40 +00:00
ba15a9c056
* feat: support defer_on_missing_source for pending flow creation Add `defer_on_missing_source` flow option that allows creating flows even when source tables do not yet exist. The flow enters a pending state and is automatically activated when source tables become available. Key changes: - New `FlowStatus::PendingSources` and fields in `FlowInfoValue` for unresolved source table names and last activation error - `defer_on_missing_source` create-time-only option: stripped from runtime/flownode `CreateRequest` but preserved in metadata for SQL round-trip (`SHOW CREATE FLOW`, `information_schema.flows`) - `CreateFlowProcedure` creates pending metadata when sources are missing and `defer_on_missing_source=true`; falls back to `FlowType::Batching` for missing-source flows - `PendingFlowReconcileManager` in meta-srv periodically checks pending flows and activates them when source tables resolve - `ActivatePendingFlowProcedure` handles activation: allocates peers, creates flows on flownodes, updates metadata, invalidates cache - `OR REPLACE` properly handles pending<->active transitions, including peer allocation and flownode flow teardown - `FlowMetadataAllocator::alloc_peers` for peer allocation at activation time - Validated flow options: only `defer_on_missing_source` allowed; unknown options rejected - Known issue: standalone mode does not support flownodes, so pending flow flush/sink behavior covered only in distributed sqlness; operator and meta unit tests cover activation logic Tests: - operator `determine_flow_type_for_source_state` (3 passed) - common-meta `create_flow` (19 passed) including replacement - common-meta `activate_flow` (4 passed) - meta-srv `flow` (11 passed) - sqlness: `flow_pending` covers create/replace/round-trip Signed-off-by: discord9 <discord9@163.com> * chore: simplify pending flow PR scope Reduce PR #8124 to the metadata-only MVP after complexity review. Changes: - Remove automatic activation procedure and meta-srv reconcile wiring - Remove activation tests and activation-only metadata fields - Reject cross-state pending<->active `OR REPLACE` transitions for now - Keep pending metadata creation and SQL round-trip behavior - Allow `DROP FLOW` for pending flows without routes - Reduce flow_pending sqlness to metadata/round-trip/drop coverage only Deferred follow-ups are documented locally in `.tmp/tasks/pending-defer-semantics/deferred-followups.md` and intentionally not committed. Tests: - `cargo test -p operator determine_flow_type_for_source_state` - `cargo test -p common-meta create_flow` - `cargo test -p common-meta drop_flow` - `cargo sqlness bare --test-filter flow_pending --bins-dir /mnt/nvme_rust/rust-targets/pending_defer/debug` Signed-off-by: discord9 <discord9@163.com> * test: cover pending flow metadata edge cases Signed-off-by: discord9 <discord9@163.com> * test: fix pending flow metadata test lint Signed-off-by: discord9 <discord9@163.com> * docs: document pending flow metadata fields Signed-off-by: discord9 <discord9@163.com> * chore: more sleep when test Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>
25 lines
694 B
SQL
25 lines
694 B
SQL
CREATE FLOW pending_without_defer
|
|
SINK TO pending_sink
|
|
AS SELECT val FROM pending_source;
|
|
|
|
CREATE FLOW pending_with_defer
|
|
SINK TO pending_sink
|
|
WITH (defer_on_missing_source = true)
|
|
AS SELECT val FROM pending_source WHERE val > 10;
|
|
|
|
SHOW CREATE FLOW pending_with_defer;
|
|
|
|
SELECT
|
|
flow_definition,
|
|
source_table_ids,
|
|
source_table_names,
|
|
flownode_ids,
|
|
options LIKE '%"defer_on_missing_source":"true"%' AS has_defer_option,
|
|
options LIKE '%"flow_type":"batching"%' AS has_flow_type_option
|
|
FROM INFORMATION_SCHEMA.FLOWS
|
|
WHERE flow_name = 'pending_with_defer';
|
|
|
|
DROP FLOW pending_with_defer;
|
|
|
|
SELECT flow_name FROM INFORMATION_SCHEMA.FLOWS WHERE flow_name = 'pending_with_defer';
|