Commit Graph
2278 Commits
Author SHA1 Message Date
a45c9ca525 feat: Ducklake native support (#6268)
* upgrade duckdb

* basic ducklake works

* ducklake works with custom db catalogs

* fix: pwsh skip already installed modules outside of cache (#6037)

* improve query performance of user stats

* separate ducklake_catalog db

* ducklake settings

* DucklakeSettings frontend

* Ducklake ws settings saved in backend

* fetch ducklake catalog resource

* Ducklake works with configured s3 storage

* Ducklake as asset

* ducklake asset icon

* Fix duckdb array and object args not working properly (#6254)

* Fix bug with comments in duckdb

* Avoid multiple queries when doing ATTACH ducklake

* trunc sig no longer needed now that comments are trimmed

* cache DuckdbConnectionSettingsResponse

* duplicated code

* transform_attach_ducklake contributes to duckdb_connection_settings_cache

* eliminate the need for used_storages

* nit

* cleaner management of the bigquery credentials file

* DBManagerDrawer refactor to prepare for Ducklake

* get ducklake schema

* implement delete for ducklake

* load column metadata for ducklake

* Select query works for ducklake, basic db explorer works !

* duckdb count query

* Support all db ops for ducklake

* clean migrations

* SQL repl for Ducklake

* fix broken database studio

* nit

* assert function

* Ducklake in Editor Bar

* default ducklake syntax + allow extra args

* DucklakeCatalogWizard UI

* nit + remove extra $

* modal when databases do not exist

* cannot be windmill

* Ducklake works safely with instance database

* Avoid sending instance db credentials on network

* resource leak security

* remove fetch_attach_db_conn_str

* prevent instance pg password leak

* hide asset usage count when not available

* case unsensitivity duckdb

* warnings

* disable instance catalog

* use shorthand syntax when inserting with EditorBar

* Instance ducklake catalog is now safe to use

* use safer argon2 pwd

* update package json parsers

* update package json

* better msgs

* tooltips

* disable explore button until saved

* nit

* fix warnings

* better ducklake_user password management

* nit

* Sanitize passwords from errors in ducklake

* DisplayResult broken in job result

* remove superadmin requirement to check databases_exist

* duckdb_connection_settings_v2_inner

* Ducklake works on agent worker (finally)

* ci

* #[allow(dead_code)]

* fix openapi missing response

* Separate +Database button for DuckDB in EditorBar

* Fix dropdown in ducklake settings

* Attempt to fix migration race condition in CI

* update sqlx failing for some offline queries

* avoid temp password for ducklake_user

* nits

* ducklake settings nits

* update duckdb default script

* fix sql repl resetting text on refresh

* avoid pgcrypto extension

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-08-06 13:55:36 +00:00
dieribaandRuben Fiszel 2a20329c4b feat: dynamic select in flow (#6315)
* update

* add dyn select for flows

* nits

* fix typo

* done

* update .sqlx

* better

* nits

* fix

* nits

* nits

* fix

* improvements

* update

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-08-06 07:20:03 +00:00
Ruben Fiszelandrubenfiszel 13c33bcc4a chore(main): release 1.518.2 (#6323)
* chore(main): release 1.518.2

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-08-05 07:30:34 +00:00
Ruben Fiszelandrubenfiszel ef9f90bb94 chore(main): release 1.518.1 (#6322)
* chore(main): release 1.518.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-08-04 22:17:08 +00:00
Ruben Fiszelandrubenfiszel 09f8eb5381 chore(main): release 1.518.0 (#6317)
* chore(main): release 1.518.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-08-02 14:08:00 +00:00
Alexander Petric c528c366f6 fix: delete empty git connection (#6318) 2025-08-01 16:35:40 +00:00
Ruben Fiszelandrubenfiszel 45828ec575 chore(main): release 1.517.0 (#6310)
* chore(main): release 1.517.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-31 18:13:54 +00:00
Ruben Fiszelandrubenfiszel 1d73e73899 chore(main): release 1.516.0 (#6298)
* chore(main): release 1.516.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-31 00:15:05 +00:00
Ruben Fiszel 2ad9d31a11 handle metadata for new scripts happen after commit 2025-07-30 23:20:22 +00:00
Ruben Fiszel 0e9bb06ce1 fix: no process relative imports for scripts with codebase 2025-07-30 22:50:22 +00:00
Alexander Petric 10cc170d70 git sync UI improvements (#6303)
* ui improvements round 1

* modal cleanup

* init

* UI refactor

* UI cleanup + refactor

* legacy cleanup

* success model -> github actions, non-ee warnings

* sqlx

* npm check

* ee warning everywhere

* last comments

* formatting

* no hardcoded theme

* claude review improvemenets
2025-07-30 21:36:32 +00:00
Diego Imbert b0495d5757 default to false if field not present operator settings (#6301) 2025-07-30 20:47:57 +00:00
Ruben Fiszelandrubenfiszel 5be5059754 chore(main): release 1.515.1 (#6295)
* chore(main): release 1.515.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-29 22:21:14 +00:00
Ruben Fiszel a65e2316df nits logs 2025-07-29 21:01:38 +00:00
Ruben Fiszel fe55834604 fix: improved logs for script 2025-07-29 20:51:43 +00:00
Ruben Fiszelandrubenfiszel 0974ae8884 chore(main): release 1.515.0 (#6292)
* chore(main): release 1.515.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-29 19:41:05 +00:00
Ruben Fiszelandrubenfiszel ee6805600a chore(main): release 1.514.1 (#6288)
* chore(main): release 1.514.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-28 15:48:28 +00:00
Ruben Fiszel 6024edd701 pin rustls to 0.23.29 + pin aws-sdk 2025-07-28 15:18:54 +00:00
Ruben Fiszelandrubenfiszel 893d57dc91 chore(main): release 1.514.0 (#6283)
* chore(main): release 1.514.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-28 12:30:26 +02:00
centdixanddieriba 544360b02a fix(mcp): add proper check for mcp routes (#6282)
* add proper check for mcp routes

* cleaner

* apply to flow

* fix add checks scopes

---------

Co-authored-by: dieriba <dieriba.pro@gmail.com>
2025-07-28 10:11:36 +00:00
Ruben Fiszel 99285caf87 cleanup warnings 2025-07-26 13:48:53 +00:00
Ruben Fiszel 9a62ac3912 remove warnings 2025-07-26 13:28:51 +00:00
dieribaandRuben Fiszel f2df6cbc99 improve: add retry logic for sqs oidc fetch credentials (#6275)
* improve

* update dependency

* push ee ref

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-07-26 13:25:25 +00:00
Ruben Fiszelandrubenfiszel 93f8ce4e08 chore(main): release 1.513.1 (#6272)
* chore(main): release 1.513.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-25 18:11:43 +00:00
centdix 82ac687d24 fix(mcp): allow mcp scopes to run scripts and flows (#6278)
* allow mcp scopes to run scripts and flows

* fix

* fix
2025-07-25 17:16:03 +00:00
Ruben Fiszelandrubenfiszel fccb5770f4 chore(main): release 1.513.0 (#6269)
* chore(main): release 1.513.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-24 23:24:00 +00:00
Ruben Fiszel 026f2990bf fix warning 2025-07-24 12:16:14 +00:00
iqdecay b09b0ff792 feat: enable workspace exclusion in custom tags (#6263)
* feat: enable tags that exclude workspaces

* feat: frontend tooltip

* fix: use method to check if tag applies

* refactor: change conversion method

* fix: change operator comparison

* remove redundant tests
2025-07-24 12:11:29 +00:00
Ruben Fiszelandrubenfiszel b91f977938 chore(main): release 1.512.0 (#6260)
* chore(main): release 1.512.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-23 13:49:23 +00:00
dieribaandHugoCasa 8dae223c69 feat: local type references parsing support for main function args (#5995)
* add base struct

* feat resolve interface and type declarion in entrypoint param's function

* nits

* fix reset dependencies

* update package

* fix handle infinite recursion

* add depth level and handle enum for referenced type

* nits

* nits

* nits

* perf

* fix

* done

* fix schema form cache inconsistency

* fix default type and nits

* remove

* update Object typ for parser

* one level ref from from parent when resolving types and use format for resource

* update cli and use resource type

* nits

* update parsers

* fix: use specific parser versions

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
2025-07-23 10:05:32 +02:00
Ruben Fiszelandrubenfiszel 9dbb2946e7 chore(main): release 1.511.0 (#6250)
* chore(main): release 1.511.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-22 19:57:31 +00:00
596280ab6d fix: scopes-run-and-webhook-token-display (#6259)
* fix

* update sqlx

* Update backend/windmill-api/src/scopes.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix perfect match

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-07-22 20:57:39 +02:00
dieribaandRuben Fiszel d626b75995 feat: periodic worker group script (#6159)
* feaat periodic script

* feat periodic script

* fix unused

* check minimun time

* fix unused import

* update repo ref

* update ref

* nits and update repo ref

* add missing checks

* fix

* improve UX

* improve UX

* improve UX

* fix

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-07-21 22:57:56 +00:00
Alexander Petric 2d696576df oauth client credentials: allow to override token url at resource level (#6233)
* oauth client credentials: allow to override token url at resource level

* ee ref
2025-07-21 20:08:56 +00:00
Ruben Fiszelandrubenfiszel 6708f59ba9 chore(main): release 1.510.1 (#6247)
* chore(main): release 1.510.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-21 16:07:26 +00:00
dieriba 1036488b3b fix: improve ssh agent worker naming (#6211)
* remove ssh suffix

* use hostname tags

* feat

* feat

* update sqlx

* use hostname

* use worker prefix

* nits
2025-07-21 13:43:25 +00:00
Ruben Fiszelandrubenfiszel 4d3af1a321 chore(main): release 1.510.0 (#6238)
* chore(main): release 1.510.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-20 18:57:12 +00:00
Ruben Fiszel b047f97bc3 fix: prevent loading script by hash if not permissioned 2025-07-19 23:54:01 +00:00
Ruben Fiszel d990133097 feat: use sse for flow status updates 2025-07-19 14:42:26 +00:00
Ruben Fiszelandrubenfiszel 0c9886848d chore(main): release 1.509.2 (#6237)
* chore(main): release 1.509.2

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-19 11:08:46 +00:00
Ruben Fiszelandrubenfiszel 3abab0d4f5 chore(main): release 1.509.1 (#6236)
* chore(main): release 1.509.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-19 10:01:11 +00:00
Ruben Fiszelandrubenfiszel e6de1811e6 chore(main): release 1.509.0 (#6213)
* chore(main): release 1.509.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-19 00:13:09 +00:00
a91a7fc741 feat: implement SSE for job updates polling (#6174)
* feat: implement SSE for job updates polling

- Add /getupdate_sse/:id endpoint for real-time job updates via Server-Sent Events
- SSE streams job status, logs, progress, and flow status updates
- Auto-stops streaming when job completes
- Frontend uses EventSource with graceful fallback to polling on errors
- Reduces server load and improves real-time responsiveness
- Update OpenAPI spec with new SSE endpoint definition

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* all

* all

* v1

* rename test job loader

* iter2

* nit

* all

* all

* improve

* upgrade

* try catch assets json parse

* only display loader of custom component if custom component is rendered

* Fix required properties (#6221)

* whitelabelling nits

* remove fdsfs (#6222)

* add refresh on worker tag select

* whitelabel nits

* merge

* merge

* done

* nit fix

* all

* app db

* all

* all

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Tammo Ippen <tammo.ippen@posteo.de>
Co-authored-by: Diego Imbert <70353967+diegoimbert@users.noreply.github.com>
2025-07-18 22:48:51 +00:00
Diego Imbert b3f75c6d0e Assets refactor (#6217)
* Moved logic to FlowAssetsProvider

* Remove assetsMap in flow

* do not parse everything on mount + only check for missing assets fields

* add assets field in backend

* remove fallbackAccessTypes

* better structure and less queries / parsing

* Fix assets not showing when pulling raw_flow from jobs

* flow assets ctx for job run

* Fix transitive assets fetching

* Fix input args asset node

* enablePathScriptAndFlowAssets flag

* edit btn for variable

* untrack refresh

* move parseInputArgsAssets

* Assets tab in runs

* Update FlowStatusViewerInner to svelte 5 + fix asset sync bug

* avoid toast error on bad resource

* fetch res metadata for input arg asset

* Job assets viewer in run page

* r/w selector

* remove indigo badge

* store alt_access_type state in ScriptEditor

* Don't parse assets in flow script editor

* Add alt_access_type in backend

* show Read as selected by default to avoid giving the feeling of having made a decision

* keep alt_access_type when reparsing in flow raw scripts

* Remove variable asset kind, and save assets for scripts

* remove all backend asset parsing

* R/W/RW selector button nits

* fix insert into assets not saving alt access type

* support named arguments in python asset parser

* improve asset usage drawer R/W indicator

* update legacy $res: syntax

* reactivity issue

* remove last variable asset stuff

* sqlx prepare

* tooltip explainer

* deprecated variable asset nit
2025-07-17 22:15:01 +00:00
dieribaandHugoCasa 7620c68855 feat: granular token scopes (#6093)
* base

* add scopes in the UI

* remove legacy scope, unified create token into a components

* fix layout, convert old scope to new scope

* update scope

* update ui, and clear scope

* remove desc

* almsot there

* fix path

* fix delete and scope resource path matching

* update scope

* update scope and error message

* nits and fix logic

* nits and fix

* added multiple resource and jobs scope for script and flow

* add check_scope for endpoint

* clean front and improve fronetend code

* fix resource validation logic and backward compatibility with old scope

* fix frontend state and scope checks logic

* update scopes

* fix height

* nits: better_naming

* fix route

* fix add missing import

* fix import and move fn

* update repo ref

* fix import

* fix query and nits

* nits

* fix ,missing import

* fix

* revert add admin protection

* handle run action correctly

* fix

* add check scopes to some endpoint

* fix and nits

* nits

* remove unused import

* nits

* add chevron when domain is exapanded

* fix border bottom

* nits adds resource path button

* nits

* fix

* nits

* nits

* nits

* fix merge

* fix

* UI nits

* update repo ref

* add lazy static

* update scopes

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
2025-07-17 15:18:40 +00:00
pyranotaandellipsis-dev[bot] 1199e9eea5 feat(cli): make flow generate-locks respect raw requirements (#6105)
* build: add claude-code to nix shell

Signed-off-by: pyranota <pyra@duck.com>

* stage progress

* stage files

* make it work

* stage work

Signed-off-by: pyranota <pyra@duck.com>

* make it work + refactor logic a bit

Signed-off-by: pyranota <pyra@duck.com>

* small cleanup

Signed-off-by: pyranota <pyra@duck.com>

* rename `local_lockfiles` to `rawReqs`

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* fix transpilation to deno error

Signed-off-by: pyranota <pyra@duck.com>

* update sqlx

Signed-off-by: pyranota <pyra@duck.com>

* cleanup features.default

Signed-off-by: pyranota <pyra@duck.com>

* follow up rename

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* overwrite flow.yaml on `flow generate-locks`

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* remove invalid code generated by claude

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* fix typo

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* remove double quotes

Signed-off-by: pyranota <pyra@duck.com>

* use async to write file

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-07-17 14:01:28 +00:00
Ruben Fiszelandrubenfiszel 16333db200 chore(main): release 1.508.0 (#6207)
* chore(main): release 1.508.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-16 23:05:38 +00:00
Ruben Fiszelandrubenfiszel 050d9050e0 chore(main): release 1.507.2 (#6202)
* chore(main): release 1.507.2

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-16 13:26:45 +00:00
Ruben Fiszelandrubenfiszel 6d034d2996 chore(main): release 1.507.1 (#6199)
* chore(main): release 1.507.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-16 10:37:50 +00:00
Ruben Fiszelandrubenfiszel 0d59ccfc96 chore(main): release 1.507.0 (#6189)
* chore(main): release 1.507.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-07-15 17:18:16 +00:00