Commit Graph

1209 Commits

Author SHA1 Message Date
Ruben Fiszel 2e1b6c1947 fix: prevent worker not exiting if special case of same worker job 2025-07-15 15:32:26 +00:00
Ruben Fiszel e99d6261f6 nit import 2025-07-13 14:38:59 +00:00
Ruben Fiszel 88c2950a67 mend 2025-07-13 14:05:59 +00:00
Ruben Fiszel daeab7077a fix: throttle job pull when bg processor takes too long 2025-07-13 14:05:22 +00:00
Ruben Fiszel e7e27259ee nsjail skip_setsid for pg propagation + faster exit 2025-07-13 11:48:01 +00:00
Ruben Fiszel b292452477 nit tracing 2025-07-13 11:15:43 +00:00
Ruben Fiszel e36d8b00e4 nit behavior for invalid keys 2025-07-13 11:13:06 +00:00
Ruben Fiszel e6301702f5 fix: prevent workers from being stuck on kill signal 2025-07-13 11:05:46 +00:00
Alexander Petric 9a8bed128f fix: worker symlink dir + path fixes + npm postinstall on windows (#6167) 2025-07-11 22:51:48 +02:00
HugoCasa 328ef605ad feat: triggers error handler and retry (#6138)
* feat: http triggers error handling and retries

* feat: handle sync http triggers

* feat: triggers retry and error handling

* feat: frontend and preprocessor

* fix build

* minimum allow dead code

* check workspace tags in schedules

* fix flow preprocesor + hub urls

* update ee ref

* nit

* nit

* final nits

* schedule UI nits

* migrate workspace settings to svelte 5
2025-07-11 18:55:23 +02:00
Ruben Fiszel bfb2277ff1 fix: audit log truncation fix 2025-07-11 16:23:18 +00:00
Diego Imbert 433341b295 feat: assets as a primary concept (#6125)
* assets migration

* parse assets (duckdb)

* iterate on assets

* S3 object Preview

* remove pagination

* filterText

* better occurence list

* tweak

* assets in JobPreview

* clone impl

* AssetsDetectedBadge

* improve DbManagerButton + asset dropdown button

* edit resource btn

* warning when incorrect resource

* +Resource in DuckDB

* +S3 Object editor bar

* nit fix rename

* flow asset badge

* More Generic OnChange

* Highlight assets used in modules

* Show occurence count in flow

* Better UX, avoid moving parts

* nit

* Asset nodes

* move to dedicated Asset ctx

* fix layoutNodes not handling first assetsMap

* explore asset btn in flow asset node

* correct offset

* single computeAssetNodes function

* Fix y positioning of nodes with assets

* resource editor

* write mode node (ui)

* accessType in ctx + fix insert button positioning

* right positioning when mixing read and write nodes

* right positioning when mixing R and W assets

* Better layout fix algorithm

* listAssetsByUsage and asset nodes on transitive usages

* refactor + remove linkAssets

* Refactor to allow for custom R/W modes

* AssetsDropdownButton in flow script editor

* R/W/RW selection and changes node pos in flow

* layoutNodes doesnt need recompute now

* fix wrong assumption that nodes recompute when assets change

* r/w/rw multi toggle

* MultiToggle cool animation + clearable

* rename + 1px nit

* remove mini toggle button group, use ToggleButtonGroup

* Combinator parser that detects R / W asset context

* nit fix missing flex-1

* missing order by

* better ui indication for access type

* special x offset case when only one asset node for clarity

* parse getResource in TS with swc ecma parser

* support load and write s3 detection in TS

* Python asset parser

* support wmill api calls without special $res: or s3:// syntax

* detect out of context asset uris python

* do not use access type override when not ambiguous in flow graph

* parse_assets match case in rust

* AsRef<str> refactor

* From impl

* Save flow assets

* Save script asset usages + fixes + save fallback access types

* asset sub icon

* max total asset node width to avoid overlap

* small refactor

* don't parse comments in duckdb assets

* fix assets clearing on parse error

* fix script asset save in wrong place

* load initial asset fallback access types

* support variables

* ui fixes

* Support S3Object as URI in TS client

* support new syntax in python client

* Support +S3Object in EditorBar for TS and python

* Reduce resource requests in assets page

* import windmill client when necessary

* update s3Types.d.ts

* nit fix

* Show input resources and s3 objects as assets

* improve asset icons

* DarkModeObserver refactor

* asset page tabs

* Moved resource variables and s3object pages to assets tabs

* fetch resource usages

* Get variables usages

* move assets usage dropdown to component

* Revert "move assets usage dropdown to component"

This reverts commit 622ea4ab12.

* Revert "Get variables usages"

This reverts commit b11ced4e29.

* Revert "fetch resource usages"

This reverts commit aa5187ad4b.

* Revert "Moved resource variables and s3object pages to assets tabs"

This reverts commit 4430487be4.

* Revert "asset page tabs"

This reverts commit dacc2f0da5.

* move assets usage dropdown to component

* asset icon in asset pages

* tooltip

* details

* Storage selector in S3 File Picker

* make edge less opaque

* Refactor computeAssetNodes to separate in and out nodes

* AssetsOverflowedNode

* nits

* fix assets not being parsed in flows sometimes

* show asset kind and resource_type

* ui nits

* support res:// in duckdb

* add banner for old deployments

* Fix permissionning

* fix broken disable /enable all

* assets page view permission for operators

* Disable ExploreAssetButton for operators

* asset kind as subtitle

* do not spam getResource in assets page. prob. revert fail

* update assets page on workspace change

* reload storage names on ws change

* delete assets on archive / deletion

* sqlx prepare

* missing update when updating user

* add indexes on asset

* better message

* missing loadInit: false

* dead code

* use transaction

* typo

* update package.json

* update package.json

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-07-11 09:56:53 +00:00
Ruben Fiszel 5c4b6e7b6f nits 2025-07-10 13:07:05 +00:00
Ruben Fiszel b83aca30d2 feat: use process groups to improve zombie job handling (#6157)
* useProcessGroups

* nit
2025-07-10 08:49:56 +00:00
Ruben Fiszel 270d6a9a88 nit sleep between pull timeouts 2025-07-09 13:09:49 +00:00
Ruben Fiszel 8cbb88757d timeout on job pull iuf necessary 2025-07-09 13:08:44 +00:00
Alexander Petric 4482e9d86a fix: correct paths and no symlink for windows (go) (#6139)
* fix: correct paths and no symlink for windows (go)

* fix: improve Windows Go executor compatibility

- Fix path handling inconsistency - use consistent double backslashes
- Replace hardcoded Windows paths with dynamic system path detection
- Add missing env_clear() call for mod init command
- Create helper functions to reduce Windows environment setup duplication
- Use SYSTEMDRIVE and TMP/TEMP environment variables for better compatibility

Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com>

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
2025-07-08 06:52:44 +00:00
wendrul 835645643e feat: Better tracing for audit logs, including a graph to visualize them (#6078)
* Migrate audit log page to svelte 5

* Add email and span cols to audit table

* Add token_prefixs to audit logs (into AuditAuthorable trait)

* Add audit logs graph (wip)

* Add audit span on push and jwt

* Unify same job audit into the same audit span

* Improve the graph visually

* Fix typo

* functioning graph with svelte issue

* Fix leak

* feat: migrate AuditLogsTable from DataTable to VirtualList for performance

- Replace DataTable component with VirtualList for handling thousands of rows
- Migrate to Svelte 5 runes ($props, $bindable, $derived, $state)
- Implement flattenLogs() for virtual scrolling with grouped date headers
- Add sticky indices and dynamic height calculation
- Update parent component to use callback prop pattern instead of events
- Preserve all existing functionality: filtering, selection, pagination
- Follows RunsTable.svelte implementation pattern

Resolves performance issues when displaying large audit log datasets.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* Fix remaining virtual list issues

* WIP graph

* Fix chart styling

* Fix npm check

* Fix missing audit_span arguments

* Update sqlx

* use varchar 255 for email as in other tables

* Remove syntax inconsistency

* Match struct with ee crate

* Update ee-repo-ref.txt

* Update worker_flow.rs

* Remove redefinition of trait to prevent shadowing

* Re add trait on oss but only when no `private` flag

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-07-03 21:02:45 +00:00
Alexander Petric dd91089436 improve graphql error reporting (#6092) 2025-07-01 15:45:04 +02:00
windmill-internal-app[bot] c51e128920 fix: add support for GCS object storage (#6083)
* AI: Updates to files (run 15923369101)

* all

* ee ref

* unneeded

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
2025-06-30 20:11:57 +00:00
Ruben Fiszel bb110985e2 make OUTSTANDING_WAITING_TIME_TS configurable 2025-06-28 19:06:55 +02:00
Alexander Petric 835f1d2ec9 fix: add windows paths to uv install to find git/ssh (#6063) 2025-06-26 08:26:59 +02:00
Ruben Fiszel 900c8edd7b fix: ignore type only imports when computing ts lockfiles 2025-06-25 14:29:52 +02:00
HugoCasa 29f6fab60c fix: pwsh skip already installed modules outside of cache (#6037) 2025-06-24 12:41:54 +02:00
Ruben Fiszel b3b6c53430 remove WM_OIDC_JWT from context vars 2025-06-20 12:12:38 +02:00
Ruben Fiszel 86eb9074cc feat: blacklist remote agent worker token (#5985) 2025-06-20 12:07:33 +02:00
claude[bot] c4adaeeabd fix: improve piptar upload - sequential uploads via background task queue (#5994)
* feat: improve piptar upload with sequential background task queue

Replace simultaneous piptar uploads with sequential processing via background job queue while keeping dependency installation parallelized.

Key changes:
- Add PiptarUpload job kind and database migration  
- Queue piptar uploads instead of using tokio::spawn
- Implement job handler for sequential S3 uploads
- Maintain parallel dependency installation as requested

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* Remove PiptarUpload job kind migration files

Refactoring piptar uploads to use tokio channel instead of Windmill jobs.
Migration files are no longer needed.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* Refactor piptar uploads from Windmill jobs to tokio channel

Replace the complex job queue infrastructure for piptar uploads with a simple 
tokio channel approach as requested. This maintains sequential upload behavior
while keeping dependency installation parallel.

Key changes:
- Add PIPTAR_UPLOAD_CHANNEL global channel for sequential processing
- Replace JobPayload::PiptarUpload with simple channel send
- Remove PiptarUpload from JobKind enum and all job handling code
- Remove job dispatcher case from worker.rs
- Simplify upload logic while maintaining same functionality

Benefits:
- Reduced complexity by removing unnecessary job infrastructure
- Sequential uploads without blocking dependency installation
- Better separation of concerns

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* Update worker.rs

* Update python_executor.rs

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-06-19 15:32:21 +02:00
Ruben Fiszel 47d1ef0f1c fix additional python paths for windows 2025-06-18 18:24:54 +02:00
Ruben Fiszel 0f63d03093 dind compile 2025-06-17 08:14:11 +02:00
Ruben Fiszel 38eb71bdf5 fix: always rm containers in docker mode 2025-06-17 02:32:49 +02:00
Ruben Fiszel 26bec054a3 fix: flow steps use their tags if any specific when used as subflow 2025-06-17 02:27:24 +02:00
pyranota 0b89260540 feat(go): local go.mod (#5929)
* feat(go): local go.mod

* readability refactor

* remove dbg!

* ignore module

* remove space

Signed-off-by: pyranota <pyra@duck.com>

* Update backend/windmill-worker/src/go_executor.rs

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-06-13 10:17:31 +02:00
Ruben Fiszel e8836a393a fix: improve concurrent job parallelism performance 2025-06-09 16:59:07 +02:00
pyranota a2eac4ae37 nit: fix windows build (#5896) 2025-06-07 03:35:54 +02:00
pyranota 22b2f4988d fix: never consider minor version for global site packages (#5893) 2025-06-07 03:35:28 +02:00
pyranota ec552d5ef6 fix(python): account instance version when cli deploy and local lockfile (#5894)
* fix: do not ignore instance python version with cli deploy and local lockfile

* ellipsis
2025-06-06 17:49:59 +00:00
pyranota a0b302d2c5 fix: cannot parse INSTANCE_PYTHON_VERSION (#5874) 2025-06-05 16:56:30 +02:00
pyranota 50a5c1f56a fix(python): assign PATCH version to python runtime only when needed (#5866)
* fix

* clean up

* nit

* fix integration tests

* nit

* another nit

* remove dublicate test
2025-06-04 17:15:46 +02:00
dieriba f2526571a3 feat: ssh repl like direct to workers hosts machine (#5809)
* feat(backend): add repl for worker

* feat(frontend): add repl to interact with worker and missing packages

* nits

* Update backend/windmill-worker/src/result_processor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/result_processor.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/WorkerRepl.svelte

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* fix typo

* fix cd

* nits

* refactoring

* feat: handle other agent worker to also access repl feature, fix bugs

* update repo ref

* nits and match function args

* nits

* typo

* remove struct AgentWorkerData

* update repo ref

* nits

* impl new for authed client and revert to async

* update ref

* updage ee repo ref

* fix missing method/imports small bugs

* update ee repo ref

* update .sqlx

* test

* clean wait for interactive_shell future

* free call stack

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-06-03 18:41:14 +02:00
Diego Imbert 0e316239dd EE Refactor (#5844)
* app compiles with every ee substituted

* Replace all oss files content

* Revert "Replace all oss files content"

This reverts commit ea4017d59f.

* delete all ee

* hide all _ee files under private flag

* hide every oss stuff when private flag set

* pub use *

* gitignore and substitute script

* pub mod for ee needed for ee repo

* small mistakes

* remove oidc_oss impl

* ee ref (temp)

* ee ref

* fix --all-features selecting private in OSS CI

* ee repo ref

* allow unused
2025-06-02 22:12:33 +02:00
pyranota ed61d97700 feat(rust): shared build directory (#5610)
* feat: rust incremental compilation

* update dockerfiles

* fix compilation error

* add windows flags

* init

* polishing

* update

* return mount-point

* Update backend/windmill-worker/src/rust_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* share worker dir if nsjail off

* final refactor

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-05-29 22:11:35 +02:00
Diego Imbert fdefd4be93 feat: duckdb sql lang support (#5761) 2025-05-27 15:52:57 +02:00
Ruben Fiszel 5b96bccedd feat: add aws oidc support for instance s3 storage (#5810)
* backend

* iterate

* all

* all

* all

* iterate

* revert

* all

* add tracing to get of authed client

* all

* all

* lal

* all

* update

* fix

* push

* all

* all

* revert

* frontend

* fix checks

* avoid deadlock

* safer

* fix

* fix
2025-05-25 14:03:38 +02:00
pyranota 26222539e6 feat(python): inline script metadata (PEP 723) (#5712)
* make resolver

* more updates

* fix build

* fix raw_dependencies job type

* compat with http agent workers

* refactor

* rename

* more refactor

* cleanup

* more tests

* fix s3

* small fixes

* more fixing

* fix endpoint

* nit: update comment

* update ee ref

* update ee ref

* update ee ref

* implement safer `list_available_python_versions`

* add tracing to get of authed client

* internal: Trigger claude when commenting with /aider (#5783)

* add claude instructions files

* call claude too when using aider

* fix

* add draft for linear claude integration

* fix build

* update ee ref

* ignore versions <=3.9

* fix windows build

* correct versions filter

* fix windows build (this time for real)

* inject error to debug CI

* update CI

* undo debug of CI

* fix tests

* remove outdated comment

* update ee repo ref

* Update ee-repo-ref.txt

* Update backend/parsers/windmill-parser-py-imports/src/lib.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update InstanceSetting.svelte

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: centdix <40307056+centdix@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-05-24 02:13:47 +02:00
Ruben Fiszel af74653b7f improve err message 2025-05-22 19:48:26 +02:00
Ruben Fiszel 7c24fbcef2 fix: improve docker mode unexpected exit handling 2025-05-22 19:37:47 +02:00
Ruben Fiszel 3f825ec77f reduce the pip resolution cache duration 2025-05-22 18:47:02 +02:00
Ruben Fiszel e3e25daee7 fix 2025-05-22 14:12:10 +02:00
Ruben Fiszel d662e18f97 add more labels to traces 2025-05-22 12:07:21 +02:00
Ruben Fiszel dfd8c4cd2a more verbose docker wait errors 2025-05-22 01:07:29 +02:00