Commit Graph

94 Commits

Author SHA1 Message Date
Diego Imbert 433341b295 feat: assets as a primary concept (#6125)
* assets migration

* parse assets (duckdb)

* iterate on assets

* S3 object Preview

* remove pagination

* filterText

* better occurence list

* tweak

* assets in JobPreview

* clone impl

* AssetsDetectedBadge

* improve DbManagerButton + asset dropdown button

* edit resource btn

* warning when incorrect resource

* +Resource in DuckDB

* +S3 Object editor bar

* nit fix rename

* flow asset badge

* More Generic OnChange

* Highlight assets used in modules

* Show occurence count in flow

* Better UX, avoid moving parts

* nit

* Asset nodes

* move to dedicated Asset ctx

* fix layoutNodes not handling first assetsMap

* explore asset btn in flow asset node

* correct offset

* single computeAssetNodes function

* Fix y positioning of nodes with assets

* resource editor

* write mode node (ui)

* accessType in ctx + fix insert button positioning

* right positioning when mixing read and write nodes

* right positioning when mixing R and W assets

* Better layout fix algorithm

* listAssetsByUsage and asset nodes on transitive usages

* refactor + remove linkAssets

* Refactor to allow for custom R/W modes

* AssetsDropdownButton in flow script editor

* R/W/RW selection and changes node pos in flow

* layoutNodes doesnt need recompute now

* fix wrong assumption that nodes recompute when assets change

* r/w/rw multi toggle

* MultiToggle cool animation + clearable

* rename + 1px nit

* remove mini toggle button group, use ToggleButtonGroup

* Combinator parser that detects R / W asset context

* nit fix missing flex-1

* missing order by

* better ui indication for access type

* special x offset case when only one asset node for clarity

* parse getResource in TS with swc ecma parser

* support load and write s3 detection in TS

* Python asset parser

* support wmill api calls without special $res: or s3:// syntax

* detect out of context asset uris python

* do not use access type override when not ambiguous in flow graph

* parse_assets match case in rust

* AsRef<str> refactor

* From impl

* Save flow assets

* Save script asset usages + fixes + save fallback access types

* asset sub icon

* max total asset node width to avoid overlap

* small refactor

* don't parse comments in duckdb assets

* fix assets clearing on parse error

* fix script asset save in wrong place

* load initial asset fallback access types

* support variables

* ui fixes

* Support S3Object as URI in TS client

* support new syntax in python client

* Support +S3Object in EditorBar for TS and python

* Reduce resource requests in assets page

* import windmill client when necessary

* update s3Types.d.ts

* nit fix

* Show input resources and s3 objects as assets

* improve asset icons

* DarkModeObserver refactor

* asset page tabs

* Moved resource variables and s3object pages to assets tabs

* fetch resource usages

* Get variables usages

* move assets usage dropdown to component

* Revert "move assets usage dropdown to component"

This reverts commit 622ea4ab12.

* Revert "Get variables usages"

This reverts commit b11ced4e29.

* Revert "fetch resource usages"

This reverts commit aa5187ad4b.

* Revert "Moved resource variables and s3object pages to assets tabs"

This reverts commit 4430487be4.

* Revert "asset page tabs"

This reverts commit dacc2f0da5.

* move assets usage dropdown to component

* asset icon in asset pages

* tooltip

* details

* Storage selector in S3 File Picker

* make edge less opaque

* Refactor computeAssetNodes to separate in and out nodes

* AssetsOverflowedNode

* nits

* fix assets not being parsed in flows sometimes

* show asset kind and resource_type

* ui nits

* support res:// in duckdb

* add banner for old deployments

* Fix permissionning

* fix broken disable /enable all

* assets page view permission for operators

* Disable ExploreAssetButton for operators

* asset kind as subtitle

* do not spam getResource in assets page. prob. revert fail

* update assets page on workspace change

* reload storage names on ws change

* delete assets on archive / deletion

* sqlx prepare

* missing update when updating user

* add indexes on asset

* better message

* missing loadInit: false

* dead code

* use transaction

* typo

* update package.json

* update package.json

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-07-11 09:56:53 +00:00
HugoCasa 23d624aa23 fix: error handling for S3 file loading in py and ts clients (#6124) 2025-07-02 21:28:13 +00:00
Ruben Fiszel 9c2f6a757f fix: consistency of root job propagation fixing cases where runFlow in scripts would fail 2025-06-20 20:24:53 +02:00
HugoCasa b9e879618b feat: signed s3 objects (#5593)
* feat: accept signed s3 objects for s3 file keys in apps + sign endpoint and helpers

* Update backend/windmill-api/openapi.yaml

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix build

* Update python-client/wmill/wmill/client.py

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* nti

* fix build

* fix build

* presigned

* Update frontend/src/lib/components/apps/editor/settingsPanel/InputsSpecEditor.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/apps/editor/settingsPanel/InputsSpecEditor.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix build

* b

* fix sqlx

* nit

* typo

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-04-11 18:03:40 +02:00
pyranota 04ffbf8c26 fix: remove typings_extensions from python sdk
* fix: make python-client compatible with python >= 3.13

* remove completely

* remove sys import
2025-03-03 14:56:52 +01:00
Le Phat Tan 50607c7625 feat: python-client now support mocked api via WM_MOCKED_API_FILE env (#5372)
* feat: python-client now support mocked api via `WM_MOCKED_API_FILE` env

* fix: typo
2025-02-25 16:39:32 +01:00
Alexander Petric 149d5fb3e1 feat: teams workspace scripts (#5238)
* backend and clients

* adding teams_team_name to workspace_settings

* openapi.yaml adding teams workspace settings endpoints

* teams workspace settings frontend

* workspaces router

* build ce

* ee gate

* ee oauth

* update client

* workspace error handler

* remove log

* ce

* point to new hub scripts

* updating hubPaths

* updating hubPaths

* cleanup

* merge

* schedule teams error

* polish, reactivity

* sqlx compilewarning

* sqlx migrate

* make it build

* router

* fix

* remove sqlx workaround

* Update ee-repo-ref.txt

* simplify some logic

* sqlx

* latest ee ref

* latest ee ref

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-02-13 16:53:57 +01:00
Alexander Petric 63b096851e interactive slack approvals: more improvements / nits (#5012)
* enum -> enums, client function docs

* advanced -> suspend -> form
2025-01-04 01:48:25 +01:00
Alexander Petric 2832b887a5 interactive slack improvements (#5001)
* refactor / rust style

* add detailed description to default message

* default args, dynamic enums

* sqlx prep

* cancel / close behavior
2025-01-03 16:27:03 +01:00
Alexander Petric 6308bf0dcb feat: interactive slack approvals (#4942)
* feat: interactive slack approvals

* move form creation logic to backend

* adding python client

* polish messages

* date time picker and default values

* Initial commit

* Initial commit

* refactor

* cleanup

* cleanup

* cleanup

* cleanup, treating numbers/integers/booleans

* adding slack to ts dependencies

* sqlx prepare

* gitignore, adding package.lock

* ellipsis comments

* Update typescript-client/client.ts

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* refactor to move everything serverside + modal

* sqlx prep

* reverting slack dependency

* python client update

* fix build errors

* Update pyproject.toml

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2024-12-20 21:47:45 +01:00
Ruben Fiszel 2a78359af7 fix: add countCompletedJobs api 2024-11-14 20:44:29 +01:00
Ruben Fiszel 237307ba18 nits client 2024-11-08 15:43:03 +01:00
Ruben Fiszel 3ea12f1821 fix(python-client): fix small break params of write_s3_file 2024-11-08 15:35:41 +01:00
HugoCasa dc8bd6d2b5 feat: http custom routes for static assets (#4666)
* feat: http custom routes for static assets

* update ee ref + fix build

* fix build

* nit
2024-11-07 20:16:14 +01:00
Ruben Fiszel f39b16f3bb fix(python-client): improve get_job_status for running jobs 2024-10-16 20:09:20 +02:00
pyranota d3e0b1c636 feat: Allow setProgress and getProgress from within the script (#4400)
* Allow setting progress explicitly from script body.

This feature exposes:
 * `getProgress`
 * `setProgress`
 * `incProgress`

API in TypeScript client (python is coming soon).

NOTE: Progress cannot be out of range 0..100 and cannot decrease.

With exposed APIs there is also UI changes, so progress can be shown for individual jobs as well.

For optimization reasons, jobs start to ask for progress only after N-seconds of execution.

* feat: Add `shell.nix`

If you dont have anything but nix, dont worry, run nix-shell in root, or activate with direnv and get all needed dependencies

NOTE: You will still need docker

* feat: Add `dev.nu` to typescript client

Little helper function, allowing developer to work on ts client easier.

To use:
  `./dev.nu watch`

Now add import of windmill in body of your script and `//nobundle` on top of the file

Edit ts client in your favourite editor and hit save. Script will do the rest.

* Cleanup files

* Fix: Failed to deserialize query string: missing field `get_progress`

* perf: Implement non-naive polling mechanism for getting job progress

* Add independant delay for getProgress

Problem in `TestJobLoader`:

There should be 2 delays:

    One until we find our first progress (every 5s)
    Once we found our first progress, we can do it every second

* nit: Use `query_scalar!` instead of `query_as`

* Fix: Sql error, no rows returned by a query that expected to return at least one row

* refactor: Remove global CSS for JobProgressBar

* Change UI for progress of flow subjobs

* Replace `Step 1` with `Running` in ProgressBar for individual jobs

* Remove `incProgress`

incProgress is not very usefull and error-prone

* perf: Set metric only for jobs that are actually using it

(https://github.com/windmill-labs/windmill/pull/4373#discussion_r1759843773)

* Offload registering progress from clients to server

* Add `jobId?` argument to typescript-client's `setProgress` and `getProgress`

Allows to set progress of other jobs and flows,
if jobId specified, than flow id will be inferred automatically.

Could be used by SDK.

* Add `Error::MetricNotFound` for better error handling

* Fix: Make `JobProgressBar` display in red when failed

* Add persistant progress bar

Now you can reload the page after job is done and progress will be still there

* Allow succeeded individual job's progress bar stick to 100%

* Add python support 

* nit: Remove usage of undefined variable in python-client

* Add `async` in ts client (for error handling)

* nit(frontend): Remove unused import

* Dont load JobProgressBar when it is not needed

* nit: npm check fix

* cargo sqlx prepare

* fix sqlx

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-18 01:16:19 +02:00
Ruben Fiszel b5a9111d07 fix: improve runFlowAsync and run_flow_async default behavior + time formatting of scheduled for 2024-09-11 23:52:02 +02:00
Ruben Fiszel 2b3e86e341 feat: add vim support for monaco/webeditor 2024-08-26 22:31:23 +02:00
Ruben Fiszel 5c278f3390 fix: fix python client for S3Object 2024-06-03 17:20:29 +02:00
Ruben Fiszel 0e7a25c0bd feat: support multiple object storage + parquet_csv + polars -> datafusion" (#3853)
* multiple storage

* all

* rm symlinks

* all

* all
2024-05-31 11:59:51 +02:00
HugoCasa 126459f972 feat: improve permissioned audit logs (#3799)
* fix: increase token label size in db

* Revert "Revert "feat:  improve premissioned audit logs (#3793)" (#3798)"

This reverts commit afae33f5f8.

* feat: add username to email function

* fix: nit

* fix: add docstring to global func
2024-05-23 17:23:50 +02:00
Ruben Fiszel 9f6e886570 fix(python-client): improve error message for wait_job 2024-04-29 12:10:29 +02:00
Ruben Fiszel 3c2652241f fix python client set_user_state 2024-03-26 19:09:56 +01:00
Ruben Fiszel 249c155856 fix python client set_user_state 2024-03-26 18:57:54 +01:00
Ruben Fiszel 6b842b3c53 feat: add set_flow_user_states and get_flow_user_states api and sdk support 2024-03-26 17:31:17 +01:00
Ruben Fiszel 7f10a891c3 fix: improve workflow as code python client 2024-03-05 19:16:10 +01:00
Ruben Fiszel 3ed2874c6d fix: improve workflow as code python client 2024-03-05 18:42:08 +01:00
Ruben Fiszel 426c2a87b5 feat: workflow as code with timelines 2024-03-05 12:45:24 +01:00
Ruben Fiszel f89f5d6e1c fix: workflow as code python sdk improvement 2024-03-04 23:23:41 +01:00
Ruben Fiszel b2521b7d2c fix: workflow as code api improvement 2024-03-04 20:45:04 +01:00
Ruben Fiszel ff3aa8ab06 feat: workflow as code v0 2024-03-04 20:19:34 +01:00
Ruben Fiszel 9b257a974b modify python signature 2024-02-24 18:51:33 +01:00
Guillaume Bouvignies 42740b0924 fix: Improve python writeS3File perf (#3182) 2024-02-08 17:57:42 +01:00
Ruben Fiszel 7db2501cc1 fix: add get_root_job_id 2024-02-06 00:17:50 +01:00
Guillaume Bouvignies 81263a9502 fix: Python buffered reader (#3136)
* fix: Python buffered reader

* use bytes generator only for buferred reader
2024-02-02 10:23:04 +01:00
Ruben Fiszel 5362e903d6 fix(python): fix python reader 2024-02-02 01:21:20 +01:00
Ruben Fiszel cdec1c22ad fix(python): fix python reader 2024-02-02 01:12:01 +01:00
Guillaume Bouvignies 0003673e8d feat: migrate s3 client to object_store (#3116)
* feat: migrate s3 client to object_store

* remove multipart from open API

* Remove multipart in favor of a single stream

* progress report

* add progress reader on server side

* small nit fix

* fix read chunk

* Fix TS and python SDK

* Fix download button

* Fix download button object viewer

* fix list

* Better errors

* export loadS3FileContent

* revert changes SDK

* fix browser

* small file list unavailable fix

* Old endpoints throws informative error messages

* Typescript SDK uses raw fetch

* update python SDK

* Error if uploaded file > 50Mb

* revert python SDL changes

* Update python SDK method docs

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-02-01 16:03:30 +01:00
Guillaume Bouvignies 2e6835f610 feat: Download s3 file as stream in Python and TS (#3099) 2024-01-29 14:49:08 +01:00
Guillaume Bouvignies 00881013ec fix: Improvement for Python write_s3_file (#3079) 2024-01-25 18:10:47 +01:00
Guillaume Bouvignies ef76a90e34 fix: Better UI for S3 download and S3 TS SDK endpoints (#3065)
* Better UI for S3 download and S3 TS SDK endpoints

* gitignore typescript client node_modules

* USe Windmill BE upload endpoint for TS SDK

* Use WM backend endpoint in Python upload SDK endpoint

* revert changes

* Add expiration for Python

* Add downaload endpoint

* Add toggle for public S3 resource

* revert changes to package.json

* Add link to doc page

* fix unauthorized bug
2024-01-25 15:34:40 +01:00
Guillaume Bouvignies 4f4b59c77a feat: Download button for s3 files (#3059)
* feat: Download button for s3 files

* Add S3 load and write endpoint to Python and TS SDK

* fix
2024-01-22 21:40:30 +01:00
Ruben Fiszel c744d61da6 fix: oidc token generation endpoint GET -> POST 2024-01-17 11:15:39 +01:00
Ruben Fiszel f38ab0cb82 fix: expose getIdToken in python-client 2024-01-17 11:04:29 +01:00
Ruben Fiszel a24645ee00 feat: OIDC support (#3017)
* oidc

* update

* feat: add support for multiselect in python

* d

* d

* progress

* all

* all

* all
2024-01-16 23:32:26 +01:00
Ruben Fiszel b094cfa085 fix: add ability to set secret variable from python 2024-01-13 13:02:36 +01:00
Guillaume Bouvignies 51d7912e22 S3 functions in Python SDK cannot return None (#2954) 2024-01-03 17:26:25 +01:00
Guillaume Bouvignies 539312bb7e fix: Update S3 TS code snippets and Python SDK for Polars 0.20.X (#2911)
* fix: update S3 TS code snippets

* use S3Object type from SDK

* update python snippets as well

* Update python SDK for Polars 0.20.X
2023-12-22 08:39:35 +01:00
Guillaume Bouvignies 82d9238b35 chore: add verify to Windmill python object for httpx (#2900)
* chore: add `verify` to Windmill python object for httpx

* move it before get_client
2023-12-21 02:38:16 +01:00
Guillaume Bouvignies fd55c3d8e3 feat: S3 objects are now typed in Python and TS SDK (#2878)
* feat: S3 objects are now typed in Python and TS SDK

* fix ts and python SDK after testing

* cleanup
2023-12-20 09:16:33 +01:00