Commit Graph

346 Commits

Author SHA1 Message Date
Ruben Fiszel 4ae8cad4d4 fix: schedules do not accept 5 units cron syntax on update/create anymore 2025-02-20 17:59:32 +01:00
Ruben Fiszel 60a3cc2e65 make benchmarks even more stable 2025-02-18 08:43:03 +01:00
Ruben Fiszel a034c3ff28 fix: improve que job indices for faster performances 2025-02-17 21:26:46 +01:00
Lucas Abel ecbec1fe50 fix(backend): improve schedule queries plan to leverage indices better for performance (#5273)
* backend: fix schedule queries plan

* rework

* fix skipped as non success

* update sqlx

* update

* all

* fix npm run check

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-02-12 16:34:03 +01:00
Ruben Fiszel 5f46d79f26 fix: fix concurrent limit jobs non restarting 2025-02-10 11:33:19 +01:00
Ruben Fiszel 7bf5e381ac fix: worker name in job + better timeout handling for same_worker jobs (#5248) 2025-02-08 22:20:06 +01:00
Lucas Abel 17a86e6907 backend: fix workflow as code (#5239)
* backend: improve `/get_job_update` after v2

* backend: insert missing `workflow_as_code_status` on completion

also insert `flow_status` from so we can remove the query when `_duration` is above 500

* backend: fix workflow_as_code after v2

* backend: add `workflow_as_code` worker test
2025-02-07 12:27:43 +01:00
Lucas Abel 6ded2c598e backend: use v2 tables through views where possible (v2 phase 3) (#5119) 2025-02-06 12:43:24 +01:00
Lucas Abel 8374377958 backend: check queries to queue and completed_job where possible (v2 phase 2) (#5125) 2025-02-06 12:39:32 +01:00
Lucas Abel cecc19dafc backend: migrate job tables to v2 schema (v2 phase 1) (#5084) 2025-02-06 12:37:11 +01:00
Ruben Fiszel 098cd72840 fix uv install for nsjail (#5222) 2025-02-06 00:00:33 +01:00
Ruben Fiszel dd3fb0f955 improve error messages for internal err 2025-02-05 16:32:01 +01:00
dieriba 900a8d675e feat: postgres triggers (#4860)
* feat: init database triggers

* feat:  wip: database_triggers

* feat:  add database triggers front view

* feat: 🚧 database_triggers

* feat: 🚧 add definition in yaml, updated backend code and added migration

* feat: 🚧 updated migration file, update openapi.yml, updated database_triggers page and backend function

* fix: struct rust

* feat: 🚧 update migrate, database trigger backend function fixed

* feat: 🚧 add resource picker front, update backend function

* feat: 🚧 edit inner database inner, update triggers

* feat: 🚧 database_triggers

* feat: 🚧 update openapi yaml, prettied websocker trigger

* feat: 🚧 database_triggers

* feat: 🚧 add resource module, update variable file, working on main loop for database_trigger

* feat: 🐛 working sqlx query

* feat: 🚧 fix query with sqlx, added main loop

* feat: 🚧 add new column database_trigg

* feat: 🛂 run jobs works

* feat: 🚧 handling slot name and replication

* feat: restructring triggers, decoding trigger message on work

* feat: 🚧 database_trigger

* feat: 🚧

* feat: 🚧 converter done, work on custom script

* feat: 🚧 multiple trigger

* feat: 🚧 adding new argument function

* feat: 🚧 database_trigger

* feat: 🚧 add generate template for front, update script picker

* feat: 🚧 template script fix bug, work on restructing backend logic

* feat: 🚧 update autogenerated script, add persistence state for template script

* feat: 🚧 update structure client

* feat: 🚧 rewrited crud function

* feat: 🚧 added publication handler

* feat: 🚧 new ui finished

* feat: 🚧 added slot function hanlder finish front ux

* feat:  ux improvement done, backend logic done

* feat: 🐛 fix where clause

* feat:

* feat:

* chore: update .sqlx and remove empty package

* fix: update publication and remove unneccessary print

* feat: finish converter to json, remove save button, remove unused crate

* feat: update migration for database trigger, fixed query, fixed frontend error with missing feature on back

* chore: update .sqlx

* chore: update sqlx

* chore: .sqlx

* chore: add unused

* fix: use right database resource in back and front

* fix build

* refactor:

* fix sqlx

* nits: retrieve all script from database trigger editor

* fix: template script object, fix route name

* merge main

* feat: rework ux

* feat/finish ux and update backend logic

* feat: database_trigger

* feat: database_trigger

* chore: update sqlx

* chore: fix ci

* nits: fix worker.rs

* feat: database_trigger

* Update frontend/src/lib/components/triggers/database/DatabaseTriggersPanel.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* feat:

* fix ci

* nits: database to postgres

* nits: fix .sqlx

* feat: add link to original code and open tab functionallity

* nits: update .sqlx

* nits: fix request type and yaml file

* nits: add abel suggestion

* fix: update .sqlx

* fix: update .sqlx

* nits: add comment to hex.rs

* nits: remove features, rename database to postgres name

* nits: replace database name by postgres

* nits: database name to postgres

* nits: update .sqlx

* nits: update .sqlx

* nit

* nit

* nits

* nit

* nits

* nits

* nits

* nits

* fix sqlx

* nits: update .vscode

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-01-24 15:12:41 +01:00
Lucas Abel 7591edea4c backend: always prioritize flow steps (#5108)
When a flow start, his flow steps have lower priority than already scheduled jobs, hence steps are only executed when the queue is drained which considerably increase flow execution time when queue size is significant.
This add a very low priority to flow steps in order to prioritize finishing a started flow over new jobs.
2025-01-21 09:27:40 +01:00
HugoCasa 175379fbe9 feat: on behalf of (#5058)
* feat: on behalf of

* fix

* revert mistakke

* temp ee ref

* renaming + CLI

* sqlx

* nits + ee ref
2025-01-14 18:32:25 +01:00
HugoCasa a2f05d0e36 prevent cancelling schedule next tick + remove need of flow_status inside add_completed_job (#5022) 2025-01-13 14:26:28 +01:00
HugoCasa a6f78aed81 fea(backend): return 500 on sync webhook/route error (#5049)
* fea(backend): return 500 on sync webhook/route error

* fix sqlx
2025-01-13 12:25:06 +01:00
Lucas Abel 3f0a818fa3 backend: do not force get_schedule_opt to be within a transaction (#5019) 2025-01-06 09:36:57 +01:00
Lucas Abel dede390fab fix(backend): fix has_failure_module only looking for raw_flow (#5018) 2025-01-06 09:21:17 +01:00
Lucas Abel 12d18beb32 cache: better cache system (#4951)
* cache: consider source as untrusted

* cache: better scoped cache

* cache: non-lazy flow value parse error

* fixup! cache: consider source as untrusted

* fixup! cache: consider source as untrusted

* fixup! cache: better scoped cache
2024-12-20 11:19:57 +01:00
Lucas Abel dada363097 fix: fix redeploying flows with attached schedules 2024-12-18 18:56:30 +01:00
Lucas Abel 8502ff8225 pass cargo check (#4949) 2024-12-18 16:21:37 +01:00
Lucas Abel e32bcd3383 backend: pass cargo check --all-features (#4946) 2024-12-18 12:37:31 +01:00
Ruben Fiszel a857314f59 fix: fix relative imports in cached flow scripts 2024-12-18 11:34:07 +01:00
Lucas Abel dab7b39917 feat(cache): remove persistent raw values from queue (#4866)
* feat(cache): remove persistent raw values from queue

* test: add tests and fix discovered regressions

* fix csharp executor after rebase

* invalidate script cache after locking

* fix lite flow version race

* fix flow dependencies lock race

* update version flag to `v.1.440`
2024-12-16 17:10:16 +01:00
HugoCasa aaae1ea642 feat: add multipart/form-data support (#4927)
* feat: add multipart/form-data support

* nits
2024-12-15 10:53:12 +01:00
Ruben Fiszel 75ec668276 nit 2024-12-15 01:00:44 +01:00
Lucas Abel c430c95886 cleanup is_flow usage and impl (#4923) 2024-12-13 18:08:01 +01:00
Ruben Fiszel bd56ab8d5f fix: flow node default tag 2024-12-13 17:51:08 +01:00
HugoCasa a8d95434cc feat: accept direct file upload for webhook/http (s3) (#4903)
* feat: accept direct file upload for webhook/http (s3)

* nit

* nit
2024-12-12 00:33:19 +01:00
Lucas Abel ed71efa6da fmt: re-format Rust leftovers (#4897) 2024-12-11 08:57:00 +01:00
Ruben Fiszel 8a5ba115c6 feat: add otlp support (#4869)
* all

* all

* add otel

* add otel

* update docker-image

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update
2024-12-07 16:29:20 +01:00
Lucas Abel ef08fc8e29 feat: add db storage for app inline scripts (#4837) 2024-12-06 13:37:45 +01:00
Ruben Fiszel 19946deb82 simplify warn_after_seconds 2024-12-04 18:28:28 +01:00
Lucas Abel 0a65b90049 feat(cache): implement flow node caching (#4808)
* feat(cache): implement flow node caching

* feat(cache): implement script caching

* feat(cache): improve cache
2024-11-29 09:35:00 +01:00
Lucas Abel 4e52e54314 feat(backend): store flow inner modules into flow_node table (#4778) 2024-11-27 16:08:31 +01:00
Ruben Fiszel e3ee592a2f nit 2024-11-25 23:32:14 +01:00
Lucas Abel ce121fcf37 feat(backend): implement flow scripts (#4748)
* feat(backend): implement flow scripts

* conditionally resolve reduced flow value

* conditionally use flow scripts based on the min worker version

* grant all to `flow_{version_lite,node}`

* do not silently discard errors

* env: add flow script feature flag

* flow script on cloud automatically

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-11-25 23:20:51 +01:00
Ruben Fiszel e54e8ab405 fix: retry on inserting completed job (#4784)
* fix: retry on pushing next scheduled job of schedule

* all

* improve error handling

* update sqlx
2024-11-25 01:33:44 +01:00
Ruben Fiszel 48d1f61307 improve backoff for schedules 2024-11-24 18:48:01 +01:00
Ruben Fiszel fd7bf27f27 remove rsmq 2024-11-24 11:04:54 +01:00
Ruben Fiszel 9c6502b17b fix: retry on pushing next scheduled job of schedule 2024-11-24 08:52:51 +01:00
Alexander Petric 54b419d8ab feat(schedule): support for extended cron syntax (#4754)
* init

* croner with fall back on scheduling failure

* adding packages

* fallback mechanism v1

* removing debug console log

* improving cargo organization

* queue cargo.toml

* use workspace cargo-tz

* test next occurrence when creating new croner schedule

* toggle label

* treat legacy edit requests where no cron_version is provided as v1

* fix edit schedule
2024-11-21 19:24:38 +01:00
Lucas Abel 002f0b92ea backend: rework raw values fetching logic (#4770) 2024-11-21 13:59:42 +01:00
Lucas Abel b5f411a8ce feat(backend): move some static fields out of job tables (#4689)
* feat(backend): move some static fields out of job tables

* backend: re-use `flow_value` if any and avoid clone

* update minimal version to 1.427.0
2024-11-19 11:27:55 +01:00
Alexander Petric 5b39f8f0ba feat(monitoring): workspace critical alerts (#4684)
* critical alert ui

* updating ui, backend logic

* revert

* type check fix npm

* checking out cli files from main

* moving alert icon

* adding sqlx mock data

* more sqlx changes

* feat(frontend): nodes from flow can be connected directly in expr input through a plug icon (#4652)

* Add flow prop picker

# Conflicts:
#	frontend/src/lib/components/propertyPicker/PropPicker.svelte

* fix unwanted copy

* cleaning

* Fix unset context

* move button and always display input

* fix unwanted proppicker display

* update

* update

* clean all

* clean all

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>

* replace hide/show with toggle

* adding mutable setting and navigation to settings to configure channels

* merge fix

* ee non ee changees

* auto-acknowledge when muted

* pr comments

* fix bad log

* user inner modal component

* update unaknowledge alerts after acknowledging from modal

* aknowledge -> acknowledge

* format

* adding backend support for workspace critical alerts

* immediately check for alerts

* immediately check for alerts

* adding openapi changes

* simplify loading of superadmin/ee

* update modal logic

* frontend logic update

* sqlx prepare

* adding mute functionality for workspace critical alerts + show alerts button for instance and workspace settings

* npm / rust warnings & errors

* reverting non-mac cargo toml

* reverting non-mac cargo toml

* mute toggles in critical alert ui

* ui polish

* adding acknowledged_workspace column

* sqlx prepare

* make sure we wait for all stores to be loaded

* adding workspace mute logic to report_critical alert

* auto ack recovered alerts

* toggle workspace as superadmin and critical alerts in logs menu

* adding critical alert button if width < 786px

* refresh on change

* Move notification to logs (#4698)

* Move notification to logs

* removing unused button

---------

Co-authored-by: Alexander Petric <petric.al@gmail.com>

* tailwind typo

* auto ack on mute

* don't change deref

* sqlx prep

* avoid renaming db column, keep acknowledged instead of acknowledged_global

* hide critical alert menu when not ee

* if workspace muted, also acknowledge global when workspace set

* removing save button for mute setting and improve ux/responsiveness

* fix: deployment callbacks have a concurrency limit of 1 on same path

* sqlx prep

* ee-repo ref

* z-[9999] for modal

---------

Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-11-15 09:48:48 +01:00
Ruben Fiszel 9cdda95003 fix: deployment callbacks have a concurrency limit of 1 on same path 2024-11-14 21:15:26 +01:00
Lucas Abel d7b688d352 nit: generalize usage of has_failure_module (#4706) 2024-11-14 08:49:12 +01:00
Guilhem cc1d3f2193 feat(frontend): improve and simplify scheduled poll flows (#4560) 2024-11-04 23:18:09 +01:00
HugoCasa 62d88436b2 feat: flow step skipping (#4461) 2024-09-30 16:30:59 +02:00