Commit Graph

1004 Commits

Author SHA1 Message Date
Ruben Fiszel 8073e5daeb fix: better handle already completed jobs cases 2025-10-08 10:06:15 +00:00
hugocasa e1d2eb2870 feat: support for protected private hub (#6762)
* feat: support for protected private hub

* nits

* nits
2025-10-08 09:33:28 +00:00
Alexander Petric ae8d37fc34 feat: add dynamic skip for schedules (#6739)
* feat: add dynamic skip for schedules

* npm check

* accidental rename

* fixing tests

* Update SQLx metadata

* simplify api / queries

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2025-10-08 09:18:15 +00:00
dieriba 27d7809ef9 feat: parallel loop as expr (#6743)
* ok

* revert

* backward compatible

* nits

* nits

* perf

* ok

* nits
2025-10-08 06:57:22 +00:00
Alexander Petric c9dae9580a fix: correct otel log levels (#6772)
* fix: correct otel log levels

* oss
2025-10-08 06:47:05 +00:00
Diego Imbert 258b275f9b fix: better ducklake setup (#6763)
* stash

* Much better ducklake setup UX

* nits

* mistake

* sqlx prepare
2025-10-07 07:00:00 +00:00
centdix 820437c897 chore(flow): clean flow conversation code (#6755)
* use own folder for memory

* fixes

* better chat interface

* fix export tab

* move in folder

* dont show flow graph if chat mode

* fix

* fix too long title

* fix user message

* fix

* fix

* remove from server

* cleaner

* cleaning

* cleaning

* cleaning
2025-10-06 15:10:29 +00:00
Ruben Fiszel 0bc822dd10 warn for slow queries in push scheduled job 2025-10-03 17:30:40 +00:00
hugocasa 3907c9f951 feat: end user email env var (#6750)
* feat: end user email env var

* nit

* nits
2025-10-03 17:27:58 +00:00
centdix 047420e5ad feat(flow): add option to turn flow into chat (#6658)
* add toggle option + chat interface

* backend impl

* draft

* put info in schema

* Revert "backend impl"

This reverts commit c534eeb49986424e2c12e2c5642be4e17ba380d1.

* chat interface in flow input

* cleaning

* add logic for running flow + styling

* handle historic args

* fix frontend changes

* add tables

* add conv list

* add endpoints

* adapt frontend

* list message logic

* save message in db

* save response in db

* cleaning

* better migrations

* refresh on new conv

* better logic for messages

* nit

* genere conversation uuid from frontend

* store chat mode info in flow status

* better ui for chat

* collapse chat

* ui

* infinite scroll on convs

* infinite scroll on messages

* fix ui

* new chat entry on new

* cleaning

* change setting logic

* fix test logic from flow input

* move toggle to input

* add warning modal when enabling chat mode

* add summary and explanation on inline script

* add hint for chat mode on user_message desc

* show chat message instead of input in graph

* add warning for triggers

* one logo when not expanded

* use infinitelist for conversations

* add warning when deployment in progress

* full width button

* better icon for menu

* better input + nits

* put toggle in action

* use waitjob

* cleaning

* cleaning

* scroll on new + cleaning

* use enum

* fix logic

* full screen

* cleaning

* exit on updatesqlx error

* Update SQLx metadata

* fix

* cleaning

* add for wait result endpoint

* add missing drop

* delete cascade

* fix: use macro version of query_as in flow_conversations.rs

Use sqlx::query_as! macro instead of query_as function for compile-time
SQL validation and better type safety

Co-authored-by: centdix <centdix@users.noreply.github.com>

* fix: update comment to clarify conversation message update condition

The comment now accurately reflects that the update happens when
it's a flow and it's done (flow_is_done)

Co-authored-by: centdix <centdix@users.noreply.github.com>

* fix: only parse chat_input_enabled if conditions are met

Move the parse_chat_input_enabled() call inside the condition check
to avoid unnecessary parsing when the flow is not done or unsuccessful

Co-authored-by: centdix <centdix@users.noreply.github.com>

* fix: use the same transaction for conversation creation

Pass transaction to get_or_create_conversation_with_id instead of
creating a new one, ensuring all operations are atomic

Co-authored-by: centdix <centdix@users.noreply.github.com>

* fix: remove update trigger and handle updated_at in application code

Remove the database trigger that automatically updates conversation
timestamp and instead update it explicitly when creating messages.
This gives better control and consistency.

Co-authored-by: centdix <centdix@users.noreply.github.com>

* Update SQLx metadata

* cleaning

* feat(aiagent): handle memory (#6719)

* implement memory

* s3 logic for memory

* fix typo

* much cleaner

* cleaning

* cleaning

* only if chat

* display nit

* nit

* fix stack overflow

* cleaning

* use len arg from input

* cleaning

* change order

* delete memory when conv deleted

* cleaning

* nit

* show description in expr mode

* opti

* opti

* updatee ref

* store string as simple string

* use markdown

* do not wait for deletion

* add delete loading

* fix logic

* fix markdown

* Update ee-repo-ref.txt

* Update SQLx metadata

* fix in test interface

* nit

* nit

* fix layout

* use memory_id to store memory

* shorter description

* rls + grant

* fix text overflow

* extract output from res

* cleaning

* handle streaming

* cleaning

* fix tool error

* nit

* update ref

* fix

* Update SQLx metadata

* nit

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: centdix <centdix@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-10-03 14:04:21 +00:00
Diego Imbert f35c9ffe63 S3 Proxy better errors + DuckDB S3 fix (#6740)
* Support for MinIO and other S3 impls in S3 Proxy

* Nice S3 Proxy error messages in DuckDB executor

* nit

* useless code

* super nit

* ee repo ref
2025-10-03 11:06:46 +00:00
hugocasa 526dfd7237 feat(backend): allow multiple workspaces in jwt (#6714)
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-09-30 20:56:52 +00:00
Ruben Fiszel d382ea7c8b feat: support esm mode for codebase bundles (#6709)
* rawAppsS3

* make fn common

* all

* merge

* nit

* fix ingress

* all

* all

* all

* all
2025-09-30 14:45:30 +00:00
hugocasa 4bbbeb956f feat: powershell private repo support (#6684)
* feat: powershell private repo support

* nits
2025-09-29 13:24:41 +00:00
hugocasa 8f4fef9804 feat(backend): job result stream optimization (#6673)
* feat(backend): job result stream optimization

* get offset locally instead of from db

* fix: agent worker result stream

* update ref

* nit

* remove foreign key on job

* fix build
2025-09-26 11:26:50 +00:00
Pyra ed806bf9d0 fix(backend): rework dependency_map handling (#6598)
* v0

Signed-off-by: pyranota <pyra@duck.com>

* optimize relocks

* make it work with relative relative imports

Signed-off-by: pyranota <pyra@duck.com>

* use fallback

Signed-off-by: pyranota <pyra@duck.com>

* remove dbg and todos

Signed-off-by: pyranota <pyra@duck.com>

* future proof a bit

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* more cleanup

Signed-off-by: pyranota <pyra@duck.com>

* remove final TODO

Signed-off-by: pyranota <pyra@duck.com>

* do not use bytemuck

Signed-off-by: pyranota <pyra@duck.com>

* optimize hashing

Signed-off-by: pyranota <pyra@duck.com>

* implementation 1

Signed-off-by: pyranota <pyra@duck.com>

* almost v0

Signed-off-by: pyranota <pyra@duck.com>

* v0

Signed-off-by: pyranota <pyra@duck.com>

* add comments and use fallback

Signed-off-by: pyranota <pyra@duck.com>

* call dissolve for apps

Signed-off-by: pyranota <pyra@duck.com>

* add comms

Signed-off-by: pyranota <pyra@duck.com>

* refactor v0 (partially tested + dirty)

Signed-off-by: pyranota <pyra@duck.com>

* finishing

Signed-off-by: pyranota <pyra@duck.com>

* remove TODO

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

* silence unused argument

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* implement rebuild_map endpoint

Signed-off-by: pyranota <pyra@duck.com>

* update windmill api client

Signed-off-by: pyranota <pyra@duck.com>

* almost finish with tests

Signed-off-by: pyranota <pyra@duck.com>

* add proper testing

Signed-off-by: pyranota <pyra@duck.com>

* remove unused fixtures

Signed-off-by: pyranota <pyra@duck.com>

* Update SQLx metadata

* partial cleanup

Signed-off-by: pyranota <pyra@duck.com>

* Update backend/windmill-worker/src/scoped_dependency_map.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-common/src/scripts.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* evil doings

Signed-off-by: pyranota <pyra@duck.com>

* more cleanup

* Update SQLx metadata

* more cleanup

Signed-off-by: pyranota <pyra@duck.com>

* fixing CI

Signed-off-by: pyranota <pyra@duck.com>

* remove python from default features

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: Pyra <92104930+pyranye@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-09-24 13:40:39 +00:00
Ruben Fiszel ad38298840 nit compile 2025-09-22 15:05:13 +00:00
Ruben Fiszel 7a13e9e988 feat: split RUST_LOG into RUST_LOG and RUST_LOG_STDOUT 2025-09-22 14:36:15 +00:00
Ruben Fiszel f6dd78cb11 fix: improve reliability of exits in case graceful handler didn't exit as expected 2025-09-22 10:27:57 +00:00
Ruben Fiszel f71f9b0894 feat: load for loop jobs timeline directly from for loop flow status (#6646) 2025-09-19 17:32:09 +00:00
hugocasa 993baf46bd feat(backend): flow streaming (#6520)
* feat(backend): flow streaming

* all streaming languages + sync api

* sqlx

* fix build

* UI and nits

* nit

* feat: stream last flow step

* sqlx

* nit

* use get for stream endpoints + add snippet in UI

* refactor

* nits

* Update backend/windmill-worker/src/common.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* nits

---------

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-09-18 15:49:35 +00:00
centdix 709a937ac2 fix(aiagent): fix endpoint for azure (#6633)
* fix ai agent for azure

* cleaning

* add parentheses
2025-09-18 13:22:24 +00:00
wendrul 9325f5636c fix: Don't reencrypt secrets on workspace forking (#6622)
* Do not reencrypt variable when performing a workspace fork

* Update SQLx metadata

* Update SQLx metadata

* Remove unused

---------

Co-authored-by: GitHub Action <action@github.com>
2025-09-17 16:47:27 +00:00
Diego Imbert 673b4d2a4c Advanced S3 permissionning (#6617)
* Advanced permission rules UI

* stash

* first iteration for s3 rule parsing

* Move to glob based approach

* cache expiry

* fix popover positioning

* ee

* unused imports

* forgot windmill_uploads

* Check S3 permissions for apps

* nit

* typo

* ee repo ref

* forgot get_workspace_s3_resource_and_check_paths in oss
2025-09-17 12:15:33 +00:00
dieriba e98bde6be6 feat: allow operator to use script/flow with dynselect input (#6616)
* done

* update sqlx

* fix openapi spec
2025-09-17 10:56:39 +00:00
centdix 20f48e6ded feat(ai agent): handle images in ai agent (#6572)
* add in frontend

* draft openai handling

* upload to s3

* simpler output

* return s3 directly if any

* low quality

* implement for gemini

* handle imagen model

* handle image input

* cleaning

* remove base64 from output

* cleaning

* fix timeout

* handle openrouter

* remove log

* allow image input when creating image

* cleaning

* increase stack size

* inline everything

* revert stack size

* cleaning

* fix for openai

* better mime type

* add descriptions
2025-09-15 07:51:09 +00:00
Ruben Fiszel a9f1e1d186 fix nits worker group stats 2025-09-11 09:19:09 +00:00
Ruben Fiszel de273ebb5d warn for variable values delays 2025-09-09 22:16:51 +00:00
Ruben Fiszel 45898ef5a3 fix: variable perf and oauth refresh improvements (#6567)
* all

* improve

* improve

* improve

* improve
2025-09-09 22:03:53 +00:00
hugocasa 36bbde6239 feat: email triggers (#6548)
* feat: email triggers

* Change down migration to drop email_trigger table

* email triggers UI

* bug fix

* Apply suggestion from @ellipsis-dev[bot]

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* cli and git sync

* Revert "cli and git sync"

This reverts commit 220fd50d13.

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-09-09 21:04:23 +00:00
claude[bot] 20f4086a77 feat: add worker_group_job_stats table for efficient job metrics aggregation (#6527)
* feat: add worker_group_job_stats table for job metrics aggregation

- Add new table with hour timestamp, worker group, script lang, workspace_id, job count and total duration
- Workers accumulate stats in memory and update hourly via sum aggregation
- Monitor.rs cleans up rows older than 60 days periodically
- Stats are flushed on worker shutdown to prevent data loss

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* all

* all

* ee-repo-ref

* nits

* nits

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-09-08 17:27:58 +00:00
dieriba c210a404e0 feat: timeout as expression (#6509)
* done

* add ctx and flow input

* backward compatible

* fix typo

* fix
2025-09-08 15:42:21 +00:00
Ruben Fiszel 83fff3a590 fix: fix db pool corruption by nativets logs 2025-09-06 14:15:18 +00:00
Ruben Fiszel a5f4d23ca7 better permissions issues user facing errors 2025-09-05 10:15:14 +00:00
Ruben Fiszel 1611404e95 fix flow push schedule on flow update (#6538)
* fix flow schedule

* fix flow schedule
2025-09-05 10:09:17 +00:00
centdix c0dbbe942c feat(flows): allow all providers for ai agent steps (#6529)
* add ui for structured output

* implement backend logic for openai models

* simulate having non required props

* cleaning

* implement logic for anthropic

* cleaning

* cleaning

* cleaning

* avoid name clash

* return object

* focus new field when adding

* fix condition

* small opti

* use box raw value

* avoid unnecessary parsing + return error if parsing fails

* update ui to show all providers

* cleaning

* adapt backend

* fix

* styling fix

* fix for anthropic with openrouter

* nit styling

* avoid race condition

* fix merge

* fix

* fix import

* make one component for whole provider selection

* fix resource logic

* rename

* clean

* remove console log

* fix bad copy
2025-09-05 08:11:23 +00:00
Ruben Fiszel 0647202a4f fix: check perms more tightly for running jobs (#6532)
* all

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* all

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* nits
2025-09-05 06:07:45 +00:00
dieriba f10cac1c4b feat: conditional retry workflow (#6461)
* feat retry

* fix

* fix

* fix and nits

* fix query

* fix test

* nits and perf

* update sqlx

* once

* remove

* update sqlx
2025-09-03 20:24:38 +00:00
Diego Imbert 420be5a1f8 feat: use S3 Proxy for duckdb instead of direct connection (#6505)
* s3 proxy works with get (no auth yet)

* nit

* support s3:// syntax

* Support s3:// syntax and fix vite api proxy normalizing double slashes in URI

* s3 checks authed

* nit

* PUT works

* delete file works

* Derive the JWT signature from the backend

* Authorize s3 correctly (JWT signature is never sent in cleartext)

* convert object store error to wmill error for correct status code

* stash

* fix

* POST first request proxy works

* s3 put for duckdb

* factor out direct proxy code

* Fix Issue with backend proxy and wrong signature due to Host header mismatch

* Add _default_ syntax to solve URI normalization issues with signing

* restricted to user paths toggle

* user path restriction works !

* change restriction to allow

* fix

* factor out code

* better permissions UX in object storage settings

* Revert to restrict_to_user_paths

* check permissions in old s3 api

* DuckDB now uses S3 Proxy and no longer needs LFS query

* implement todo

* fix hardcoded w_id

* s3 proxy size limit

* s3_proxy is ee

* nit

* add Google Cloud Storage as option to secondary storage

* GCS secret in duckdb

* fix toolchain compile

* Remove user permissions for v0

* fix ci 2

* fix CI OSS

* fix missing feature flag

* fix unused warning

* integration test fails bc rustc 1.85.0

* ee ref

* fix ci ...

* update ee ref
2025-09-03 15:03:44 +00:00
dieriba f3fd1e90b0 feat: refactor trigger crud (#6472)
* base

* base crate

* websocket updated

* crud refactored

* fix and nits

* fix compiler warning, nits and update repo ref

* fix oss

* fix compilation

* update ref

* fix

* update feature

* listener base

* add listener

* refactor logic done and implemented for postgres

* fix capture

* websocket

* implem for all triggers

* update sqlx,repo ref and fix

* fix oss

* unify struct fix websocket

* nits and fix oss runtime axum error

* perf cache query

* update .sqlx

* update .sqlx

* fix

* fix unused

* fix

* fix

* nits and fix http handler update endpoint

* update .sqlx

* update repo ref

* nits

* fix

* update ref

* fix

* update .sqlx
2025-09-02 11:28:06 +00:00
hugocasa fbd942f179 fix(backend): retrieve root_job for MiniPulledJob + fix root job for flow jobs (#6490)
* feat(backend): retrieve root_job for MiniPulledJob + fix root job for flow jobs

* nit

* nits

* set root_job only if additional information meaning if not equal to innermost or parent

* nit

* feat: add root job env var

* nits

* fix build

* nit

* fix build

* nit

* sqlx
2025-09-02 09:44:09 +00:00
claude[bot] db36a323a7 feat(perf): add 60-second local cache for variables and resources(#6511)
* feat: add 60-second cache for variables and resources with PostgreSQL invalidation

- Add new var_resource_cache module with 60-second TTL
- Implement PostgreSQL NOTIFY/LISTEN for immediate cache invalidation
- Cache get_variable() for non-secret, non-encrypted reads
- Cache get_resource_value() for all read operations
- Add database triggers on variable/resource table changes
- Initialize cache system in main.rs after database connection
- Add Clone derive to ListableVariable for cache compatibility

Performance benefits:
- Avoids database queries on cache hits
- Immediate invalidation ensures data consistency
- Selective caching respects security constraints

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: move cache to API layer with allow_cache query param

- Move variable/resource cache from windmill-common to windmill-api
- Add allow_cache query parameter to variables and resources endpoints
- Follow raw script cache pattern with timestamp + value structure
- Create proper database migration for notification triggers
- Include encrypted values in cache when appropriate
- Only activate caching when explicitly requested

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* refactor: move cache to API layer with allow_cache query param

Move PostgreSQL LISTEN logic to main.rs following established pattern
- Remove custom listener initialization from cache module
- Add cache invalidation channels to centralized notification handler
- Simplified cache module to only handle cache operations
- Follow raw script cache pattern for notification handling

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
2025-09-02 07:48:03 +00:00
Ruben Fiszel 16912b484d feat: cache relative imports (#6504)
* all

* all

* update
2025-09-01 14:55:08 +00:00
claude[bot] 8f46066a64 feat(perf): add 300-second local cache for variable crypt retrieval (#6483)
* feat: add 60-second cache for workspace key retrieval

This implements a cache with 60-second staleness for the get_workspace_key
function to reduce database queries for workspace encryption keys. The cache
follows the same pattern as the existing CUSTOM_ENVS_CACHE but with a
shorter expiration time.

Requested by @rubenfiszel

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* Extend cache staleness from 60 to 300 seconds

* feat: add cache invalidation notifications for workspace keys

Add PostgreSQL LISTEN/NOTIFY mechanism to invalidate workspace key cache
across all servers and workers when workspace keys change.

- Add database migration with trigger function for workspace_key changes
- Add notification handler in main.rs to remove from WORKSPACE_KEY_CACHE
- Follow same pattern as existing workspace environment cache invalidation
- Ensures distributed cache consistency for workspace encryption keys

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* finish

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
2025-08-31 12:19:59 +00:00
Ruben Fiszel 9041bb9c7a add TAGS_ARE_SENSITIVE env variable 2025-08-30 10:26:24 +00:00
hugocasa 2ce11cea3b prevent past due team plans to overuse (#6487)
* prevent past due team plans to overuse

* nit

* update ee ref
2025-08-28 16:56:57 +00:00
pyranota 006f32602c fix: fix relative imports cache invalidation (#6468)
* fix: new script on deps job for secondary scripts

Signed-off-by: pyranota <pyra@duck.com>

* make it work (dirty)

Signed-off-by: pyranota <pyra@duck.com>

* reduce db calls

* remove `triggered_by_relative_import`

Signed-off-by: pyranota <pyra@duck.com>

* add comment to common_dependency_path

Signed-off-by: pyranota <pyra@duck.com>

* add fallback to old behavior

Signed-off-by: pyranota <pyra@duck.com>

* remove TODOs

Signed-off-by: pyranota <pyra@duck.com>

* pass deployed hash to git sync handler function

Signed-off-by: pyranota <pyra@duck.com>

* fix ci

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
2025-08-27 17:44:35 +00:00
Ruben Fiszel 86f41ffcde minor nits fix 2025-08-27 15:08:58 +00:00
Ruben Fiszel a245f70179 not_found_if_none displays location 2025-08-27 13:08:52 +00:00
Ruben Fiszel 97ed4a539b nits cleanup + faster script index #6450 2025-08-23 22:42:39 +00:00