mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
feat: add auto_commit option to Kafka triggers with advanced UI badges (#8317)
* feat: add auto_commit option to Kafka triggers with manual commit API
Add ability to disable auto-commit on Kafka triggers so users can
manually commit offsets after processing messages. This prevents
message loss when processing fails.
Changes:
- Add `auto_commit` column to kafka_trigger table (default true)
- Add POST /kafka_triggers/commit_offsets/{path} endpoint using
BaseConsumer with manual assign() to avoid rebalance
- Enrich trigger_info payload with partition and offset fields
- Conditionally commit based on auto_commit setting
- Add auto-commit toggle to frontend Kafka trigger config
- Add commitKafkaOffsets helpers to Python and TypeScript SDKs
- Add integration tests for auto_commit DB defaults
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: use DB-based pending commits for kafka manual offset commit
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: pass trigger_path to all v2 preprocessors, secure commit_offsets endpoint, fix commit semantics
- Add trigger_path to v2 preprocessor event for all trigger types (kafka, nats, sqs, mqtt, gcp, postgres, websocket, http, email)
- Secure commit_offsets endpoint: infer trigger from job token (OptJobAuthed) instead of requiring trigger path parameter
- Fix auto_commit: only commit offset after successful job push
- Fix pending commits: commit offset+1 (Kafka semantics) and use CommitMode::Sync
- Update TS/Python clients and frontend preprocessor templates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add advanced section badges and reorganize kafka trigger settings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove dead wm_trigger assertions from kafka e2e test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* sqlx
* refactor: remove unused advancedCollapsed state from all trigger editors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update ref
* chore: update ee-repo-ref to ed2c9d360e6fab866b9744cc79f50038d1fc7152
This commit updates the EE repository reference after PR #452 was merged in windmill-ee-private.
Previous ee-repo-ref: 5b31116a1d5a042c6a780732901cfd89584d1773
New ee-repo-ref: ed2c9d360e6fab866b9744cc79f50038d1fc7152
Automated by sync-ee-ref workflow.
* fix: use path-based auth for kafka commit_offsets endpoint
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update ee-repo-ref to fcd3ea52b0cc94fbe1159baf662a38da947456de
This commit updates the EE repository reference after PR #457 was merged in windmill-ee-private.
Previous ee-repo-ref: b3a5c33c92cb1b2caf7a65986d71da291ff72a35
New ee-repo-ref: fcd3ea52b0cc94fbe1159baf662a38da947456de
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
This commit is contained in:
@@ -2842,3 +2842,28 @@ def _run_workflow(func, checkpoint: dict, input_args: dict):
|
||||
"""Synchronous wrapper that runs the workflow coroutine to completion
|
||||
or until it suspends."""
|
||||
return _asyncio.run(_run_workflow_async(func, checkpoint, input_args))
|
||||
|
||||
|
||||
@init_global_client
|
||||
def commit_kafka_offsets(
|
||||
trigger_path: str,
|
||||
topic: str,
|
||||
partition: int,
|
||||
offset: int,
|
||||
) -> None:
|
||||
"""Commit Kafka offsets for a trigger with auto_commit disabled.
|
||||
|
||||
Args:
|
||||
trigger_path: Path to the Kafka trigger (from event['wm_trigger']['trigger_path'])
|
||||
topic: Kafka topic name (from event['topic'])
|
||||
partition: Partition number (from event['partition'])
|
||||
offset: Message offset to commit (from event['offset'])
|
||||
"""
|
||||
_client.post(
|
||||
f"/w/{_client.workspace}/kafka_triggers/commit_offsets/{trigger_path}",
|
||||
json={
|
||||
"topic": topic,
|
||||
"partition": partition,
|
||||
"offset": offset,
|
||||
},
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user