Files
windmill/backend/migrations/20260721082249_add_amqp_trigger.up.sql
68debab877 feat(triggers): add AMQP (RabbitMQ) trigger via lapin (#10230)
* feat(triggers): add AMQP (RabbitMQ) trigger using the lapin library

Fixes WIN-2214

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(triggers): defer AMQP cross-workspace deploy pending utils-internal publish

Revert the amqp_trigger additions to the shared windmill-utils-internal
TriggerDeployKind and the frontend cross-workspace deploy adapter: the
frontend installs the published npm package, which lacks the new kind
until a release is cut. AMQP create/edit/delete/list/sync/capture are
unaffected (they use local types); only cross-workspace deploy/merge of
AMQP triggers waits on the package bump. Also document the at-most-once
ack in the consumer loop.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(triggers): address AMQP review — at-least-once ack, workspace cascade, contracts

- ack AMQP deliveries only after successful dispatch; nack+requeue on failure
- add ON DELETE CASCADE workspace FK so amqp_trigger rows are cleaned on
  workspace deletion (and the listener stops)
- fix the /amqp_triggers/test OpenAPI body and add amqp_trigger to
  WorkspaceDiffRow.kind
- register AMQP in the generated workspace trigger tool (create_trigger)
- drop banned $bindable defaults on optional props in the config section
- add build_uri unit tests (encoding, ports, vhost)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(triggers): stop AMQP poison-message loop and reconnect on transient drops

Chaos testing against a live RabbitMQ broker showed the previous
nack(requeue) + immediate re-poll spun a tight redelivery loop (~1000
critical-error reports/sec) on a poison message, and any connection blip
permanently disabled the trigger (lapin has no built-in reconnect).

- on dispatch failure: nack+requeue then stop consuming; the listener
  framework re-lists the trigger after its ping goes stale (~15s), backing
  redelivery off to that cadence instead of a tight loop (verified: rate
  dropped from ~1000/s to ~1 per ~26s, message preserved)
- on connection/stream error: stop and let the framework reconnect instead
  of disabling; persistent failures are still disabled via get_consumer
  (verified: a forced connection close now auto-reconnects and resumes)
- finish the AI create-trigger action wiring for AMQP: add amqp to
  CreatedResourceTriggerKind, the action-card registry, and the drawer
  registry so the result card renders and its "Open" action works

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(triggers): complete AMQP frontend registries and defer merge rows

- add amqp to capturableTriggerTypes (so AmqpCapture mounts), the Runs
  jobTriggerKinds filter, and CLOUD_DISABLED_TRIGGER_TYPES
- wire AMQP into global AI chat mode: TRIGGER_KINDS, the request union,
  writeTriggerSchema, triggerServices, and the draft adapter
- stop emitting actionable AMQP fork-comparison rows (revert amqp_trigger
  from TRIGGER_OR_SCHEDULE_TABLES) since cross-workspace deploy is deferred
  until windmill-utils-internal is published — avoids a deploy that fails
  with "Unknown kind: amqp_trigger"
- use design-system TextInput instead of raw <input> in the config section

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(triggers): complete AMQP session/draft registries and constrain prefetch

- add amqp to the session-deploy, draft-compare, preview-router, and
  copilot workspace-item registries so AMQP drafts/deploys/nav/path
  resolution work
- include amqp_count in the MoveDrawer attached-trigger rename warning
- replace the raw prefetch <input> with a design-system TextInput bounded
  to an integer 1-65535 (backend u16) and block save on invalid values

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(triggers): make AMQP disconnect/reconnect consistent with the Kafka trigger

lapin, like rdkafka, has no transparent reconnect, so the AMQP listener now
mirrors the Kafka trigger's explicit reconnect loop instead of relying on the
framework re-list (which disabled the trigger once get_consumer failed on a
sustained outage):

- get_consumer returns cheaply; consume owns a (re)connect loop that retries
  with a 30s backoff, reports a critical error every 10 failed attempts, and
  reports a recovered critical error once it reconnects — never disabling the
  trigger on a connectivity failure
- a consumer/stream error breaks out to reconnect rather than disabling
- dispatch failure still nacks+requeues (at-least-once) with a short backoff
  to avoid a tight poison-message loop, keeping the connection alive

Verified against a live RabbitMQ broker: killing the broker keeps the trigger
enabled and retrying (attempt N), and restarting it auto-reconnects (logs
"reconnected after N attempts") and resumes dispatch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(triggers): complete AMQP capture registries and constrain prefetch contract

- add the 'amqp' case to triggerKindToTriggerType so opening the AMQP editor
  from a capture button no longer throws "Unknown TriggerKind: amqp"
- register AmqpIcon in CaptureTable's icon map and add an AMQP entry to the
  script/flow CaptureButton menu
- bound the OpenAPI prefetch_count to an integer 1-65535 (matches the Rust
  u16) and regenerate clients/prompts
- require a non-empty exchange name when the exchange binding is enabled
- build_uri: fall back to "/" on a blank vhost and bracket IPv6 hosts (+ tests)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(triggers): wire AMQP into pipeline graph, git-sync, and preprocessor types

- asset_graph: discover attached amqp_trigger rows and emit an AMQP TriggerEdge
  so AMQP triggers render (and can be opened/deleted) on the data-pipeline canvas
- frontend pipeline graph: add amqp to NativeTriggerKind, the add-trigger menu,
  node presentation, event-trigger set, annotation keywords, and the
  editor/service registrations
- git-sync: add the amqp_trigger include pattern (+ test) so an AMQP git-sync
  deployment stages only its .amqp_trigger.* file, not an unrelated same-path object
- preprocessor starters: add the AMQP event to the generated TS/Python/PHP
  trigger event types (kind/payload/exchange/routing_key/queue_name/redelivered/
  delivery_tag)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(triggers): finish AMQP pipeline/parser wiring, prefetch validation, source lists

- fix a stray edit that corrupted the pre-existing MqttTriggerEditor import
  ($lib/... path) in PipelineTriggerEditors.svelte
- reject prefetch_count = 0 server-side in validate_config (RabbitMQ treats 0
  as unlimited) and defensively skip basic_qos(0) in build_consumer (covers
  the capture path that bypasses CRUD validation)
- recognize `// on amqp` in the canonical parser (TriggerSpec::Amqp) and add
  amqp to the CLI non-autorun/event-trigger sets so a pipeline cascade never
  runs an AMQP-only node as a manual root without an event
- add amqp to the preprocessor intro lists and both pipeline AI instructions

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(triggers): reject zero AMQP prefetch in all paths and finish guidance lists

- extract a shared validate_amqp_options used by both CRUD validate_config
  and build_consumer, so capture configs (which bypass CRUD validation) also
  reject prefetch 0 instead of silently connecting with an unlimited buffer
  (+ unit tests for 0/1/65535/None)
- add AMQP to the main script-writing preprocessor-sources prompt and the CLI
  triggers-skill guidance list

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(triggers): de-duplicate AMQP prefetch comment and fix GET response text

- keep the zero-prefetch rationale only on the shared validate_amqp_options
  doc; drop the redundant call-site comments
- correct the getAmqpTrigger OpenAPI 200 description ("deleted" -> "retrieved")

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to aaa6cb89b05b76139252c64f057e53b94d12ac60

This commit updates the EE repository reference after PR #680 was merged in windmill-ee-private.

Previous ee-repo-ref: 5da5fd65aca9594b2611837a52e4677b544b0380

New ee-repo-ref: aaa6cb89b05b76139252c64f057e53b94d12ac60

Automated by sync-ee-ref workflow.

* chore(migrations): consolidate the four AMQP migrations into one

The table and the three enum ADD VALUE statements (trigger_kind, job_trigger_kind,
draft_kind) are one atomic feature. ALTER TYPE ... ADD VALUE runs inside the
migration transaction on PG >= 14 (Windmill's minimum) since the amqp_trigger
table doesn't reference those enum types, so they can share a single migration
instead of four. Verified applying cleanly in a single transaction on a fresh DB.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-07-21 15:10:29 +00:00

84 lines
5.0 KiB
SQL

-- Add up migration script here
CREATE TABLE amqp_trigger (
amqp_resource_path VARCHAR(255) NOT NULL,
queue_name VARCHAR(255) NOT NULL,
exchange JSONB NULL,
options JSONB NULL,
path VARCHAR(255) NOT NULL,
script_path VARCHAR(255) NOT NULL,
is_flow BOOLEAN NOT NULL,
workspace_id VARCHAR(50) NOT NULL,
edited_by VARCHAR(50) NOT NULL,
edited_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
extra_perms JSONB NOT NULL DEFAULT '{}',
server_id VARCHAR(50) NULL,
last_server_ping TIMESTAMPTZ NULL,
error TEXT NULL,
error_handler_path VARCHAR(255) NULL,
error_handler_args JSONB NULL,
retry JSONB NULL,
mode TRIGGER_MODE NOT NULL DEFAULT 'enabled'::TRIGGER_MODE,
permissioned_as VARCHAR(255) NOT NULL,
labels TEXT[] NULL,
PRIMARY KEY (path, workspace_id),
FOREIGN KEY (workspace_id) REFERENCES workspace(id) ON DELETE CASCADE
);
CREATE INDEX idx_amqp_trigger_labels ON amqp_trigger USING gin (labels) WHERE labels IS NOT NULL;
GRANT ALL ON amqp_trigger TO windmill_user;
GRANT ALL ON amqp_trigger TO windmill_admin;
ALTER TABLE amqp_trigger ENABLE ROW LEVEL SECURITY;
CREATE POLICY admin_policy ON amqp_trigger FOR ALL TO windmill_admin USING (true);
CREATE POLICY see_folder_extra_perms_user_select ON amqp_trigger FOR SELECT TO windmill_user
USING (SPLIT_PART(amqp_trigger.path, '/', 1) = 'f' AND SPLIT_PART(amqp_trigger.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.folders_read'), ','))::text[]));
CREATE POLICY see_folder_extra_perms_user_insert ON amqp_trigger FOR INSERT TO windmill_user
WITH CHECK (SPLIT_PART(amqp_trigger.path, '/', 1) = 'f' AND SPLIT_PART(amqp_trigger.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.folders_write'), ','))::text[]));
CREATE POLICY see_folder_extra_perms_user_update ON amqp_trigger FOR UPDATE TO windmill_user
USING (SPLIT_PART(amqp_trigger.path, '/', 1) = 'f' AND SPLIT_PART(amqp_trigger.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.folders_write'), ','))::text[]));
CREATE POLICY see_folder_extra_perms_user_delete ON amqp_trigger FOR DELETE TO windmill_user
USING (SPLIT_PART(amqp_trigger.path, '/', 1) = 'f' AND SPLIT_PART(amqp_trigger.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.folders_write'), ','))::text[]));
CREATE POLICY see_own ON amqp_trigger FOR ALL TO windmill_user
USING (SPLIT_PART(amqp_trigger.path, '/', 1) = 'u' AND SPLIT_PART(amqp_trigger.path, '/', 2) = (select current_setting('session.user')));
CREATE POLICY see_member ON amqp_trigger FOR ALL TO windmill_user
USING (SPLIT_PART(amqp_trigger.path, '/', 1) = 'g' AND SPLIT_PART(amqp_trigger.path, '/', 2) = any((select regexp_split_to_array(current_setting('session.groups'), ','))::text[]));
CREATE POLICY see_extra_perms_user_select ON amqp_trigger FOR SELECT TO windmill_user
USING (extra_perms ? (select concat('u/', current_setting('session.user'))));
CREATE POLICY see_extra_perms_user_insert ON amqp_trigger FOR INSERT TO windmill_user
WITH CHECK ((extra_perms ->> (select concat('u/', current_setting('session.user'))))::boolean);
CREATE POLICY see_extra_perms_user_update ON amqp_trigger FOR UPDATE TO windmill_user
USING ((extra_perms ->> (select concat('u/', current_setting('session.user'))))::boolean);
CREATE POLICY see_extra_perms_user_delete ON amqp_trigger FOR DELETE TO windmill_user
USING ((extra_perms ->> (select concat('u/', current_setting('session.user'))))::boolean);
CREATE POLICY see_extra_perms_groups_select ON amqp_trigger FOR SELECT TO windmill_user
USING (extra_perms ?| (select regexp_split_to_array(current_setting('session.pgroups'), ','))::text[]);
CREATE POLICY see_extra_perms_groups_insert ON amqp_trigger FOR INSERT TO windmill_user
WITH CHECK (exists(
SELECT key, value FROM jsonb_each_text(extra_perms)
WHERE SPLIT_PART(key, '/', 1) = 'g' AND key = ANY((select regexp_split_to_array(current_setting('session.pgroups'), ','))::text[])
AND value::boolean));
CREATE POLICY see_extra_perms_groups_update ON amqp_trigger FOR UPDATE TO windmill_user
USING (exists(
SELECT key, value FROM jsonb_each_text(extra_perms)
WHERE SPLIT_PART(key, '/', 1) = 'g' AND key = ANY((select regexp_split_to_array(current_setting('session.pgroups'), ','))::text[])
AND value::boolean));
CREATE POLICY see_extra_perms_groups_delete ON amqp_trigger FOR DELETE TO windmill_user
USING (exists(
SELECT key, value FROM jsonb_each_text(extra_perms)
WHERE SPLIT_PART(key, '/', 1) = 'g' AND key = ANY((select regexp_split_to_array(current_setting('session.pgroups'), ','))::text[])
AND value::boolean));
-- Enum values for the new trigger kind. ALTER TYPE ... ADD VALUE runs inside the
-- migration transaction on PG >= 14 (Windmill's minimum) as long as the value
-- isn't used in the same transaction — the amqp_trigger table above does not
-- reference these enum types.
ALTER TYPE TRIGGER_KIND ADD VALUE IF NOT EXISTS 'amqp';
ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'amqp';
ALTER TYPE draft_kind ADD VALUE IF NOT EXISTS 'trigger_amqp';