dieriba
b49bed2b08
Merge branch 'main' into dieri/sharding-v0
2025-10-01 20:24:42 +02:00
Ruben Fiszel
e2e3ae9f02
fix: fix job duration unwrap crash
2025-10-01 15:35:09 +00:00
dieriba
8f0ff61a0a
Merge branch 'main' into dieri/sharding-v0
2025-09-29 17:22:07 +02:00
Ruben Fiszel
db4bc7ee69
fix: fix agent_workers completed job back-compatibility deserialization
2025-09-26 15:27:21 +00:00
hugocasa
8f4fef9804
feat(backend): job result stream optimization ( #6673 )
...
* feat(backend): job result stream optimization
* get offset locally instead of from db
* fix: agent worker result stream
* update ref
* nit
* remove foreign key on job
* fix build
2025-09-26 11:26:50 +00:00
dieriba
9741a28b56
big
2025-09-23 00:58:55 +02:00
Ruben Fiszel
f71f9b0894
feat: load for loop jobs timeline directly from for loop flow status ( #6646 )
2025-09-19 17:32:09 +00:00
20f4086a77
feat: add worker_group_job_stats table for efficient job metrics aggregation ( #6527 )
...
* feat: add worker_group_job_stats table for job metrics aggregation
- Add new table with hour timestamp, worker group, script lang, workspace_id, job count and total duration
- Workers accumulate stats in memory and update hourly via sum aggregation
- Monitor.rs cleans up rows older than 60 days periodically
- Stats are flushed on worker shutdown to prevent data loss
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* all
* all
* ee-repo-ref
* nits
* nits
---------
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev >
2025-09-08 17:27:58 +00:00
hugocasa
958e8af782
feat: ai agent steps ( #6393 )
...
* feat: ai agent steps base
* better backend and graph
* feat: anthropic, log viewer
* nit
* fix(frontend): hide tool nodes from timeline
* move ai agent actions from flow status to flow status module
* nits and workspace/hub scripts support
* tmp ref
* fix merge
* feat: display agent tools status in the graph
* fix reactivity
* fix flow status
* nit
2025-08-20 22:40:57 +00:00
+1
161dbbc6d6
feat: add native result streaming ( #6242 )
...
* feat: add stream output feature to SSE job updates
Adds stream_output field to JobUpdate struct that extracts log lines
starting with '[wm_stream]:' from job logs. Regular logs now exclude
stream lines, which are captured separately for specialized handling.
- Added stream_output: Option<String> field to JobUpdate struct
- Created extract_stream_output_from_logs() function to filter stream lines
- Modified get_job_update_data() to use stream extraction logic
- SSE clients now receive both new_logs and stream_output in job updates
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com >
* feat: rename stream_output to stream and handle newlines as \n
- Renamed `stream_output` field to `stream` in JobUpdate struct
- Updated extract_stream_output_from_logs to extract_stream_from_logs
- Changed stream output to join with literal \n instead of actual newlines
- Stream lines are properly excluded from regular new_logs
🤖 Generated with [Claude Code](https://claude.ai/code )
Co-Authored-By: Ruben Fiszel <rubenfiszel@users.noreply.github.com >
* decision tree nits
* push ee ref
* push ee ref
* fix: fix id renaming in apps
* remove duplicate caching (#6285 )
* feat: migrate audit log ids to bigints (blocking migration for EE)
* fix(mcp): add proper check for mcp routes (#6282 )
* add proper check for mcp routes
* cleaner
* apply to flow
* fix add checks scopes
---------
Co-authored-by: dieriba <dieriba.pro@gmail.com >
* chore(main): release 1.514.0 (#6283 )
* chore(main): release 1.514.0
* Apply automatic changes
---------
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com >
* fix: pin tokio to 1.46.1 and aws-sdks-ts
* pin rustls to 0.23.29 + pin aws-sdk
* chore(main): release 1.514.1 (#6288 )
* chore(main): release 1.514.1
* Apply automatic changes
---------
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com >
* fix: improve docker logs collection in docker mode
* support $res: string in form inputs of arrays
* fix import nit
* fix: fix DynSelect
* nits
* fix: resource-type-ts-parser (#6289 )
* fix: resource types as arg in typescript handle imported defined types
* Update nix flake (#6291 )
* merge
* Small UI fixes (#6294 )
* fix step history not refreshing with staticInputs
* fix array of obj not showing up in json editor in test this step
* datatable scales correctly in DisplayResult and scrolling is much more usable
* avoid next button disapearing and changing layout / hurting ux
* nits
* fix bug when renaming module A to B then module C to A, C takes the schema of A
* fix bug with comments in sql repl
* fix aggrid theme randomly not loading
* bindable script
* better delete button in db manager
* property select doesnt exist
* fix all warnings
* delete $flowStateStore[id] on delete
* feat(cli): generate cursor rules on init (#6270 )
* create cursor rules on init
* change gen
* add missing resource-type command
* add resource type command in guidance
* add schema option
* revert
* nit
* nit
* add flow guidance
* nit
* chore(main): release 1.515.0 (#6292 )
* chore(main): release 1.515.0
* Apply automatic changes
---------
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com >
* fix: improved logs for script
* nits logs
* chore(main): release 1.515.1 (#6295 )
* chore(main): release 1.515.1
* Apply automatic changes
---------
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com >
* merge
* even more indexer tracings
* add more tracing logs
* feat: prevent too large results (>500Mb) from OOMing database
* nit naming
* feat: add CA certificate update at startup via environment variable (#6280 )
* feat: add CA certificate update at startup via environment variable
Add support for running 'update-ca-certificates' at binary startup
when RUN_UPDATE_CA_CERTIFICATE_AT_START environment variable is set to "true".
- Check for RUN_UPDATE_CA_CERTIFICATE_AT_START env var on startup
- Execute update-ca-certificates command if env var is set to "true"
- Log success/failure appropriately with tracing
- Continue startup even if CA certificate update fails
- Non-blocking implementation with proper error handling
Fixes #6279
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com >
* refactor: extract CA certificate update logic into separate function
Extract the CA certificate update logic from windmill_main() into a
dedicated update_ca_certificates_if_requested() function for better
code organization and maintainability.
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com >
* improvements
---------
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com >
Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com >
Co-authored-by: Alexander Petric <alex@windmill.dev >
* fix: indexer collection of job logs before indexing (#6300 )
* Add flume as dependecy for indexer
* Update ee-repo-ref
* Remove flags from cargo.toml
* Update ee-repo-ref
* Update ee-repo-ref
* fix rust sdk build error (#6305 )
Signed-off-by: pyranota <pyra@duck.com >
* fix broken audit logs filter (#6304 )
* rename to from to
* goto fix
* default to false if field not present operator settings (#6301 )
* git sync UI improvements (#6303 )
* ui improvements round 1
* modal cleanup
* init
* UI refactor
* UI cleanup + refactor
* legacy cleanup
* success model -> github actions, non-ee warnings
* sqlx
* npm check
* ee warning everywhere
* last comments
* formatting
* no hardcoded theme
* claude review improvemenets
* fix: no process relative imports for scripts with codebase
* fix: sqs oidc authentication disconnect #6307
* handle metadata for new scripts happen after commit
* handle_deployment_metadata in a task
* nits
* chore: add windmill-utils-internal package (#6299 )
* add utils package
* naming
* cleaning
* add docs
* remove log
* use autogenerated types
* remove old
* fix
* cleaning
* add docs
* chore(main): release 1.516.0 (#6298 )
* chore(main): release 1.516.0
* Apply automatic changes
---------
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com >
* merge
* indexer improvements
* upgrade tantivy to 0.24.2
* use tantivy fork
* nit warnings
* fix oss build
* improve indexer
* chore: use windmill-utils-internal for cli (#6297 )
* add utils package
* naming
* cleaning
* simplify assignPath
* rename old files
* same for locks
* create on confirm
* default true
* use replaceinlinescripts from utils
* use extractscriptfromflows
* make it compile
* cleaning
* use argsigtojson
* fix
* fix missing await
* cleaner
* cleaning
* cleaning
* use in frontend
* add docs
* testing
* remove log
* use autogenerated types
* remove old
* fix
* cleaning
* adapt usage
* draft
* better build script
* fix build
* revert to default creation
* add docs
* remove and rename
* make everything work
* add await
* only if not installed
* add vs code setting
* add to publish action
* fix bc
* safer use of sep
* fix
* do not rename on push
* no publish on release
* use published package on frontend
* nit
* Add dependencies to run sqlx prepare to nix flake (#6309 )
* feat(cli): wmill-lock.yaml v2 for easier git merge diffs
* merge
* merge
* all
* all
* rm warnings
* fix styling on aichatinput (#6312 )
* fix: use with_capacity back presusre for tantivy directory multipart writes (#6313 )
* use with capacity for tantivy directory multi part uploads
* Update ee repo ref
* Update ee-repo-ref
* Update ee-repo-ref
* chore(main): release 1.517.0 (#6310 )
* chore(main): release 1.517.0
* Apply automatic changes
---------
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com >
* fix typo on cli build (#6314 )
* cleanup
* feat(utils): add flow.yaml validation function (#6316 )
* add validateflow function
* cleaner code
* preprocess json
* cleaning
* create specific package
* cleaning
* add tests
* fix: cleanup concurrency_counter automatically + remove orphans keys automatically
* fix: add disabled support to resource picker in schema forms
* fix: add wm_labels to tracing spans
* all
* merge
* all
* fix: delete empty git connection (#6318 )
* fix checks
* bun handling
* all
* all?
* all
* all
* update
* all
* update
* check
* fix history
* all
* all
* all
* Remove leftover debug tracing statements
- Remove commented debug trace in jobs.rs for stream output
- Remove commented debug trace in result_stream.rs for stream processing
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com >
* fix test
* all
* handle iter
* fix
---------
Signed-off-by: pyranota <pyra@duck.com >
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com >
Co-authored-by: Ruben Fiszel <ruben@windmill.dev >
Co-authored-by: centdix <40307056+centdix@users.noreply.github.com >
Co-authored-by: dieriba <dieriba.pro@gmail.com >
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com >
Co-authored-by: wendrul <53628737+wendrul@users.noreply.github.com >
Co-authored-by: Diego Imbert <70353967+diegoimbert@users.noreply.github.com >
Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com >
Co-authored-by: Alexander Petric <alex@windmill.dev >
Co-authored-by: pyranota <92104930+pyranota@users.noreply.github.com >
2025-08-06 22:40:42 +00:00
Ruben Fiszel
d45ddecf8f
fix: add wm_labels to tracing spans
2025-08-01 12:24:00 +00:00
Ruben Fiszel
4b9683f146
feat: prevent too large results (>500Mb) from OOMing database
2025-07-30 12:55:39 +00:00
Ruben Fiszel
34ccc8e0f7
fix: add error name and message to job_postprocessing span
2025-07-22 18:12:42 +00:00
dieriba and Ruben Fiszel
c4178c05be
feat: periodic worker group script ( #6159 )
...
* feaat periodic script
* feat periodic script
* fix unused
* check minimun time
* fix unused import
* update repo ref
* update ref
* nits and update repo ref
* add missing checks
* fix
* improve UX
* improve UX
* improve UX
* fix
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev >
2025-07-21 22:57:56 +00:00
Ruben Fiszel
8323707b07
fix: empty branches in branch one now return the previous_result
2025-07-21 11:23:22 +00:00
Ruben Fiszel
2e1b6c1947
fix: prevent worker not exiting if special case of same worker job
2025-07-15 15:32:26 +00:00
Ruben Fiszel
e99d6261f6
nit import
2025-07-13 14:38:59 +00:00
Ruben Fiszel
daeab7077a
fix: throttle job pull when bg processor takes too long
2025-07-13 14:05:22 +00:00
Ruben Fiszel
e6301702f5
fix: prevent workers from being stuck on kill signal
2025-07-13 11:05:46 +00:00
835645643e
feat: Better tracing for audit logs, including a graph to visualize them ( #6078 )
...
* Migrate audit log page to svelte 5
* Add email and span cols to audit table
* Add token_prefixs to audit logs (into AuditAuthorable trait)
* Add audit logs graph (wip)
* Add audit span on push and jwt
* Unify same job audit into the same audit span
* Improve the graph visually
* Fix typo
* functioning graph with svelte issue
* Fix leak
* feat: migrate AuditLogsTable from DataTable to VirtualList for performance
- Replace DataTable component with VirtualList for handling thousands of rows
- Migrate to Svelte 5 runes ($props, $bindable, $derived, $state)
- Implement flattenLogs() for virtual scrolling with grouped date headers
- Add sticky indices and dynamic height calculation
- Update parent component to use callback prop pattern instead of events
- Preserve all existing functionality: filtering, selection, pagination
- Follows RunsTable.svelte implementation pattern
Resolves performance issues when displaying large audit log datasets.
🤖 Generated with [Claude Code](https://claude.ai/code )
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com >
* Fix remaining virtual list issues
* WIP graph
* Fix chart styling
* Fix npm check
* Fix missing audit_span arguments
* Update sqlx
* use varchar 255 for email as in other tables
* Remove syntax inconsistency
* Match struct with ee crate
* Update ee-repo-ref.txt
* Update worker_flow.rs
* Remove redefinition of trait to prevent shadowing
* Re add trait on oss but only when no `private` flag
---------
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com >
Co-authored-by: GitHub Action <action@github.com >
Co-authored-by: Ruben Fiszel <ruben@windmill.dev >
2025-07-03 21:02:45 +00:00
f2526571a3
feat: ssh repl like direct to workers hosts machine ( #5809 )
...
* feat(backend): add repl for worker
* feat(frontend): add repl to interact with worker and missing packages
* nits
* Update backend/windmill-worker/src/result_processor.rs
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* Update backend/windmill-worker/src/result_processor.rs
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
* Update frontend/src/lib/components/WorkerRepl.svelte
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
* fix typo
* fix cd
* nits
* refactoring
* feat: handle other agent worker to also access repl feature, fix bugs
* update repo ref
* nits and match function args
* nits
* typo
* remove struct AgentWorkerData
* update repo ref
* nits
* impl new for authed client and revert to async
* update ref
* updage ee repo ref
* fix missing method/imports small bugs
* update ee repo ref
* update .sqlx
* test
* clean wait for interactive_shell future
* free call stack
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-06-03 18:41:14 +02:00
Diego Imbert
0e316239dd
EE Refactor ( #5844 )
...
* app compiles with every ee substituted
* Replace all oss files content
* Revert "Replace all oss files content"
This reverts commit ea4017d59f
2025-06-02 22:12:33 +02:00
Ruben Fiszel
5b96bccedd
feat: add aws oidc support for instance s3 storage ( #5810 )
...
* backend
* iterate
* all
* all
* all
* iterate
* revert
* all
* add tracing to get of authed client
* all
* all
* lal
* all
* update
* fix
* push
* all
* all
* revert
* frontend
* fix checks
* avoid deadlock
* safer
* fix
* fix
2025-05-25 14:03:38 +02:00
Ruben Fiszel
59afa493fa
fix: improve handling of custom concurrency key/tag with preprocessors ( #5762 )
...
* fix
* rm spurrious diff
* handle failing preprocessors
* nits
* update
* update
2025-05-17 10:52:28 +02:00
Ruben Fiszel
e87d4f3c1a
fix: avoid deadlocks in sending completed job to result processors ( #5742 )
...
* buffer
* unbounded and bounded
* fix
* buffer
* impl clone
* flume it all
* update
* update
* update
2025-05-15 09:13:09 +02:00
Ruben Fiszel
a3c76fb10c
fix: improve error display of nativets exceptions
2025-05-11 11:35:31 +02:00
Ruben Fiszel
12ba15c928
improve benchmarks
2025-04-12 11:09:21 +00:00
Ruben Fiszel
d5186da271
allow multiple workers on agent mode ( #5607 )
2025-04-11 21:19:29 +00:00
Ruben Fiszel
63fa499015
feat: agent workers v2 using http ( #5588 )
2025-04-10 00:56:11 +00:00
Ruben Fiszel
3edca4bc91
perf: optimize number of queries needed for job run ( #5504 )
...
* optPerf
* update sqlx
* update sqlx
2025-03-24 20:12:26 +01:00
Ruben Fiszel
880db319e8
feat: filter by worker + backend perf opt ( #5489 )
...
* all
* all
* all
* refactor
* sqlx
* ref
* impr
* fix
* fix
* fix
* ee private
2025-03-19 09:55:53 +01:00
Ruben Fiszel
d58ca9b395
fix: prevent monitoring task to die without sending killpill ( #5472 )
2025-03-12 11:51:53 +01:00
Ruben Fiszel
c8a959691c
feat: limit the number of times a job can be restarted (3) after loss of pings
2025-02-25 16:35:29 +01:00
Ruben Fiszel
2c67e84abe
fix(cli): improve dependency job error message (logs in result)
2025-02-20 01:52:18 +01:00
Lucas Abel
4a744ff313
backend: move bench.rs from common to worker crate ( #5253 )
2025-02-10 11:39:59 +01:00
Lucas Abel
d51c3080f7
backend: use v2 tables through views where possible (v2 phase 3) ( #5119 )
2025-02-06 12:43:24 +01:00
Ruben Fiszel
809242987e
improve errors II
2025-02-05 16:18:20 +01:00
Ruben Fiszel
32b75b8d84
improve error messages with locations
2025-02-05 15:56:58 +01:00
Lucas Abel
bded6028f1
backend: pass cargo check --all-features ( #4946 )
2024-12-18 12:37:31 +01:00
Lucas Abel
af5cca1b00
feat(cache): re-work job results cache ( #4898 )
2024-12-11 12:01:48 +01:00
Ruben Fiszel
fb510f33fc
fix tracing
2024-12-07 17:17:25 +01:00
Ruben Fiszel
6d047449e2
feat: add otlp support ( #4869 )
...
* all
* all
* add otel
* add otel
* update docker-image
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
2024-12-07 16:29:20 +01:00
Ruben Fiszel
bb9dd7d368
simplify warn_after_seconds
2024-12-04 18:24:31 +01:00
Ruben Fiszel
b534ef6074
improve transparency for slow queries
2024-12-04 17:30:58 +01:00
Ruben Fiszel
6768e5bbbd
fix: retry on inserting completed job ( #4784 )
...
* fix: retry on pushing next scheduled job of schedule
* all
* improve error handling
* update sqlx
2024-11-25 01:33:44 +01:00
Ruben Fiszel
89456f502f
remove rsmq
2024-11-24 11:04:54 +01:00
Ruben Fiszel
8c6f5a320b
propagate exit code in error result when relevant
2024-11-08 16:37:58 +01:00
Ruben Fiszel
6206f5e75e
improve suspend_first frequency behavior
2024-09-28 15:50:12 +02:00
Ruben Fiszel
f189aaad5a
fix: improve suspend_first behavior and frequency
2024-09-28 15:46:50 +02:00
Ruben Fiszel
da969da96f
improve benchmarking tools ( #4450 )
2024-09-27 10:48:04 +02:00
dieriba
Ruben Fiszel
hugocasa
Lucas Abel
Ruben Fiszel
