mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
fix/hook-tests-worktree-root
811 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ruben Fiszel
|
0258f3f81b |
perf: unblock workers before the API router is built (#10711)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
c9ddddda1b |
log the settings a failed read left unapplied (#10709)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 hugocasa
|
53eb94659b |
feat(telemetry): extend feature-usage tracking beyond AI features (#10681)
* feat(telemetry): extend feature-usage tracking to long-tail features Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: describe telemetry as product feature usage rather than AI usage Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(telemetry): trim disclosure copy and drop unused pick origin Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(telemetry): count trigger fires per run and key hub picks from hub data Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(telemetry): slugify hub keys and order both writers' upserts Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(telemetry): key native trigger adoption by service so it matches fires Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref for native trigger adoption fix Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(telemetry): move feature-usage collection into the ee crate * docs: point feature-telemetry at the moved registry and rust writer * docs: correct the trigger-fire gate comment to match measured step counts * docs: put the private-build caveat on the verification step * chore: update ee-repo-ref to f079db9e7962a413b349c4ff8036080894f30771 This commit updates the EE repository reference after PR #725 was merged in windmill-ee-private. Previous ee-repo-ref: 055adb80416f9339c9a28ae7fbaeadad30d74959 New ee-repo-ref: f079db9e7962a413b349c4ff8036080894f30771 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
30f5d2e766 |
perf: declare a settings pass instead of reading one setting at a time (#10698)
* perf: read global_settings once per settings-load pass `initial_load` reads several dozen settings back to back, one `SELECT value FROM global_settings WHERE name = $1` each: 50 serialized round trips before a worker is ready, 32 before a server is. On localhost that is ~20ms and invisible; against a real database it is 50x the RTT per process start, which `EXIT_AFTER_N_JOBS` turns into a per-job cost. `with_global_settings_snapshot` reads the whole table (12 rows on a typical instance) into a tokio task-local, and `load_value_from_global_settings` serves from it. Scoping it to the task is what keeps the single-setting reload paths correct: a `notify_global_setting_change` event for one key runs outside any scope and still reads the database, so a live settings change reaches a running worker as before. Agent workers hold an HTTP connection with no snapshot to take and are unchanged. `load_smtp_config` and `reload_custom_tags_setting` had their own inline copies of the same query; they go through the shared loader so they land in the snapshot too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: state the snapshot contract on the reader and the query `load_value_from_global_settings` is called from ~10 crates and one of them writes a setting then immediately re-reads it through `reload_custom_tags_setting`; say on the function itself that a scope, when one is installed, serves the read and leaves `db` unused. The query comment claimed the table is a handful of rows. It is not bounded that way: `workspace_dependencies_map_rebuilt:<workspace_id>` adds a row per workspace and never removes it. Those dynamically named rows are also why the snapshot fetches the whole table instead of the wanted names, so state that as the reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: bound the settings snapshot and keep it out of two reads Three review findings, all real: The snapshot fetched the whole table, which is not bounded by the settings that exist: `workspace_dependencies_map_rebuilt:<workspace_id>` adds a row per workspace with no cleanup path, and no settings pass reads one. It now fetches only statically named rows, and reads of a `<prefix>:<id>` name skip the snapshot and go to the database. Correctness does not rest on that naming convention — a colon-free dynamic name would simply be in the snapshot and still answered correctly — only the bound does. A snapshot query that failed inside an enclosing snapshot awaited the body bare, so its reads were served by the outer snapshot rather than falling through as documented. The task-local carries an explicit bypass state and the failure path scopes it. `reload_jwt_secret_setting` decided whether to generate-and-upsert the JWT secret from a snapshot-served read, so a replica booting alongside another could overwrite the secret it had just generated and invalidate its tokens. That read goes through the new `load_value_from_global_settings_fresh`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the snapshot query on the primary-key index `name NOT LIKE '%:%'` bounded the rows returned but not the work: a leading wildcard cannot use the index, so Postgres read every row anyway. Against 50k dynamically named rows it plans as a seq scan of 516 buffers whether or not seqscans are enabled — and worker connections disable them, so the plan was one the query shape forbade rather than one the planner chose. `name = ANY($1)` over an explicit list plans as a bitmap index scan, 7 buffers, bounded by the listed names rather than by table size. That list is also exactly the set the snapshot may answer from, so a name outside it falls through to the database instead of reading as unset: listing a setting is a performance choice, never a correctness one, which is what keeps the list safe to maintain by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: declare a settings pass instead of reading one setting at a time Replaces the prefetch-list snapshot with a pass the call sites build themselves. `SettingsPass` collects the reads `initial_load` will make as `(name, applier)` pairs, fetches them together, then replays the appliers in declaration order. Declaring is what makes the batch exact. The same `if server_mode` / `if *CLOUD_HOSTED` / `cfg` branches that used to guard a read now guard a declaration, so the fetch asks for what this process needs and nothing else, and there is no list of setting names to keep in sync with anything. Ordering is preserved end to end: appliers run in the order they were declared, and non-setting work in the middle of the sequence keeps its place as a step, so nothing moves and nothing runs twice. Steps that need several settings at once take them together. The batch distinguishes three states where a per-setting read only ever produced two at a given call site: - a value, - genuinely unset, which several settings must see in order to restore a default when the setting is cleared, - could not be read, which must leave the in-memory value alone. Collapsing this into "unset" would let one failed query reset workspace fairness and the queue caps across a cluster. Over HTTP the reads go out together rather than sequentially, so an agent worker's settings load costs one round instead of ~36, with no new endpoint. A setting an agent may not request still resolves to unset, as the per-setting call returned for it. `reload_*` keeps working per setting for the notify path, sharing its apply half with the pass. The wrappers no caller was left using are dropped. worker startup: 50 queries -> 2 (the batch, and jwt_secret which stays its own read so the pass cannot sit between reading it absent and upserting a replacement over another replica's). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: run the pass's non-setting steps in declaration order too Review round found the settings pass had a gap: the reads were declared but the work interleaved between them still awaited inline, so it all ran before `pass.run` applied anything. `manage_audit_partitions` therefore saw `AUDIT_LOG_RETENTION_DAYS` at its compile-time default rather than the configured value, and dropped every partition past that default. An instance keeping 30 days on CE lost the 14-to-30-day band on startup and on every full-reload tick. The `STORE_AUDIT_LOGS_S3` export anchor had the same cause: the gate read `false` before the setting applied, so an env-var-enabled export never anchored and its first tick skipped the rows committed before it. `action` exists so a step keeps its place in the sequence; every remaining inline await is now one, which fixes both and leaves no phase where a read can observe a value the pass has not applied yet. Two more from the same round: A batch that fails as a whole now falls back to per-setting reads. Skipping every applier preserves known-good state on a reload tick, but a starting process has none, and would have run on compile-time defaults until the next full reload twelve hours later. `FORCE_RUBY_REPOS` is honored again: the batched url-list path parsed without the `FORCE_` check its per-setting counterpart applied, so the override was silently dropped. `load_setting_value` never had one, so the third helper was never affected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: declare the object-store and worker-config steps in the pass too Two awaits were left running ahead of `pass.run`, so the settings they read were still at their compile-time defaults. The object-store reload is the one that matters: an AWS OIDC store mints its first token against an issuer built from `BASE_URL` (`oidc_ee.rs`), and with `OTEL_ENVIRONMENT` set nothing loads that before this pass does, so the store signed with the unset default, left `OBJECT_STORE_SETTINGS` empty and fell back to the ten-second retry while startup carried on. `reload_worker_config` calls `store_pull_query`, which reads the workspace fairness knobs. It happened to converge because the enabled flag re-stores the query when it changes, but it was reading defaults on the way there. Both are steps now, which is also what the earlier fix should have covered: the only await left outside a step is `pass.run` itself. Also from the same round: `fetch_settings_batch`'s doc comment had been stranded on the helper inserted above it, and the batch-failure fallback re-ran the same reads on an agent worker, where the batch already is the per-setting read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: point the setting-loader docs at functions that still exist `reload_setting` went with the other wrappers no caller was left using, but two doc links still referenced it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: decide the jwt secret in sql so the read can be batched `reload_jwt_secret_setting` generated a secret whenever its read came back absent or unparseable, and upserted it unconditionally. Two replicas booting against an empty row therefore each installed their own and rejected each other's tokens, and the same happened on a running cluster whenever the row was deleted or set to a non-string. Keeping the read next to the write kept the window narrow but never closed it, and it was the reason this one setting could not go through the settings pass. `get_or_create_jwt_secret` puts the decision in the statement instead: INSERT ... ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value WHERE jsonb_typeof(global_settings.value) <> 'string' RETURNING value First writer wins, a usable secret is never overwritten, and an empty RETURNING is how a caller learns another process's secret stands. The `WHERE` also keeps a normal startup from writing at all, which matters because `notify_global_setting_change` fires on every write to this table and an unconditional upsert would have made each start trigger a cluster-wide reload. Because the statement decides rather than the caller's read, a stale value is harmless and `jwt_secret` is now an ordinary declaration. Worker startup is a single batch round. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep a failed read from dropping a FORCE_ override or clearing a setting Two ways a read that did not succeed was being treated as an answer. A `FORCE_` override used to be checked before the read, so a failed read could not affect it. Moving that check into the parser put it behind a value arriving, and a failed read skips its applier, so a forced private registry fell back to the public index and a forced `settings.xml` was deleted from disk by the Maven step that follows it. Forced settings are declared as steps with no read now: the override outranks the database, so there is nothing to fetch and nothing to lose when a fetch fails. The setting loaders were passing `v.ok().flatten()` to their appliers, which turns a database error into "unset". Most appliers ignore `None`, but `apply_tag_per_workspace_workspaces` clears the workspace whitelist with it, making every workspace eligible for per-workspace tags, and `apply_fork_workspace_tag_append_fork_suffix` stores `false`. Both are also reached from the notify handlers, so a blip during a reload changed routing for the cluster. They take `?` now, as the code they replaced did by leaving the error arm empty, and the other five are converted with them so an applier that later grows a `None` branch cannot inherit the problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: route hub_api_secret through the FORCE-aware declaration `HUB_API_SECRET` lives in an `ArcSwap` rather than an `Arc<RwLock<_>>`, so it could not use `option_setting` and was declared by hand with a bare `setting` plus `parse_option_setting_value` — which is exactly the path that skips the `FORCE_` handling, so a failed read still dropped `FORCE_HUB_API_SECRET`. The rule now lives in `option_setting_with`, which takes the store closure and leaves `option_setting` a wrapper over it, so a setting held in something other than an `RwLock` reaches it too rather than having to reimplement it. The three remaining hand-written parses are `parse_setting_value`, which has no `FORCE_` handling to miss: `load_setting_value` never had the check either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
6d03784d4b |
fix: keep non traffic-serving processes out of coordinated restarts (#10694)
* fix: key server_heartbeat row on hostname so restarts reuse one row Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: trim announce_server_started doc to the durable constraints Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: only traffic-serving processes take part in coordinated restarts Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: name every non traffic-serving mode in the restart-gate comments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: narrow the restart-gate comments to claims that hold Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
22eadab67d |
perf: resolve the worker external IP in the background (#10697)
* perf: resolve the worker external IP in the background `run_workers` awaited `external_ip::get_ip()` — an HTTPS GET to hub.windmill.dev — before spawning any worker, so every worker process paid that round trip before its first job pull. Measured on a CE debug build it was 120-450 ms of a ~200-500 ms startup, and behind a firewall the call does not fail fast: it burns its whole 5 s connect timeout, on every process start. That cost is per-job under EXIT_AFTER_N_JOBS. The value is informational (it is only written to `worker_ping.ip`, which the workers list displays so users can whitelist the address), so nothing needs to wait on it. It now resolves into a process-wide cache off the startup path, and `WORKER_EXTERNAL_IP` supplies it explicitly for deployments that know their egress address or have no egress at all. Until it resolves the ping carries no IP, which `insert_ping_query` now COALESCEs so a reclaimed row keeps the address the previous process wrote instead of being blanked. The main loop reports the IP as soon as it lands rather than on the next periodic tick, so a short-lived process still records it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep unknown worker IPs out of the whitelist alert Review follow-ups: - `WhitelistIp` filtered only the `'unretrievable IP'` sentinel, so the `'NO IP'` one a pending or failed lookup now leaves in the row would be offered as an address to whitelist. It filters both. - Register `WORKER_EXTERNAL_IP` in `ENV_SETTINGS` so operators can confirm from the instance settings view that it took effect. - The worker tracked whether it had reported the IP by re-reading the cache after each ping rather than remembering what the ping carried, so a lookup landing mid-ping marked it reported without it reaching the row. The value is read once and threaded through `insert_ping` / `update_worker_ping_full`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: report a sentinel IP once the lookup has definitively failed Keeping the previous process's address on a reclaimed `worker_ping` row is right while the lookup is still in flight, but not once it has failed: the row would advertise an address nothing has confirmed, and the whitelist alert would offer it. A failed lookup now reports `UNKNOWN_IP`, leaving NULL to mean "in flight". `WORKER_EXTERNAL_IP` is rejected when longer than the `varchar(50)` column rather than panicking the worker on its initial ping, which is a hard failure. Adds the regression guard for the `ON CONFLICT` semantics: reverting to `ip = EXCLUDED.ip` would compile and blank every reclaimed row. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the agent initial ping acceptable to older servers An agent worker routinely runs against a server of a different version, and one predating the background lookup rejects an initial ping carrying no IP — which `run_worker` turns into a panic, so a newly upgraded agent would crash-loop against it. The not-resolved-yet case goes over the wire as the sentinel instead, and the server maps it back so a reclaimed row still keeps its address while resolution is pending. Also documents `ip` as the one conditional exception to `insert_ping_query`'s "only `started_at` and `jobs_executed` survive a restart", and adds `WORKER_EXTERNAL_IP` to the README env-var table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: deliver the resolved IP to servers that only take it at registration A server predating the background lookup applies `ip` from the initial ping only, and ignores it on the periodic ones. An agent registering before its lookup resolves would therefore keep the sentinel forever on such a server, where it used to report its real address. It registers a second time once the address is known, skipping that when the address is still unknown, when the server is reached over SQL and needs no second registration, or once a job has run, since registering clears the row's current job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: re-register the resolved IP even after a job has run Gating the second registration on "this process has not run a job yet" meant an agent that pulled queued work before its lookup resolved never delivered the address to a server that only takes one at registration. No job of the worker is in flight where that runs, so the gate bought nothing beyond the last job's id, which the next job refills. Documents the two cases where WORKER_EXTERNAL_IP stops being an optimisation and becomes the only way to report an address: an agent against such a server, and a process shorter-lived than the lookup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * revert: drop the WORKER_EXTERNAL_IP escape hatch Supplying the address by hand skips the hub lookup, which is not something to make easy. Resolving it in the background is what keeps it off the startup path; opting out of it is a separate decision this does not need to take. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: distinguish an IP never established from one that could not be retrieved `NO IP` was doing double duty: the column default for a row whose lookup has not resolved, and the marker for one that failed. An operator reading the workers list could not tell "not resolved yet" from "this instance cannot reach the hub", and the latter is the actionable one. A failed lookup now reports `unretrievable IP`, which is also what it reported before the lookup moved off the startup path. That leaves `NO IP` meaning only "no address established", which is what an agent sends while its lookup is in flight and what the server maps back to "unresolved" — so the wire sentinel no longer collides with the failure marker, and an agent delivers the failure to a server that only reads an IP at registration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
2fcce4526a |
feat: add EXIT_AFTER_N_JOBS worker mode for environment cleanup (#10671)
* feat: add EXIT_AFTER_N_JOBS worker mode for environment cleanup Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address review findings on the EXIT_AFTER_N_JOBS worker mode Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-2 review findings on EXIT_AFTER_N_JOBS Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-3 review findings on EXIT_AFTER_N_JOBS Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: bound WORKER_SUFFIX length and document the same-worker drain Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: validate the assembled worker name length Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
c09de594b6 |
feat: version resource values with history, diff and restore (#10596)
* feat: version resource values with history, diff and restore Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: record resource versions in a trigger so direct writes are covered Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: show the selected version's value and tighten history write access Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: gate resource version recording in trigger WHEN clauses Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: clear a resource's past versions, and address review nits Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: restore the displayed version and keep author attribution on pooled writes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: scope history to the selected workspace and gate clearing on ownership Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: gate restore on write access and clearing on the signed-in workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(frontend): share the version-history row between script and resource drawers Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: trim resource version history in the monitor sweep, not on write Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(frontend): match the script versions drawer shell for resource history Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(frontend): highlight version values instead of mounting monaco Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(frontend): match the script drawer's code preview presentation Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: rank version trim in one windowed pass instead of a correlated delete Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(frontend): treat the newest version as current by position Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: gate the resource version trim to an hourly sweep Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: unnest the version row action and correct the trim cadence docs * perf: cap the history listing and use sets for reference lookup * feat: warn when a resource is written more than 60 times a minute * fix: lower the resource write advisory to 20 per minute * fix: discard stale history loads and never diff against an unread value * fix: correct the write advisory boundary and document the eviction lock * fix: read history and the live value from one snapshot * refactor: read the drawer's diff baseline from versions, not the live resource * fix: open the history drawer with no version selected * fix: disarm the clear confirmation and clear the pane when the selection moves * fix: explain the missing diff and drop a guard that can no longer fire --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
c59b60c729 |
fix: keep the same_worker pin when a suspend ends without approval (#10552)
* fix: keep the same_worker pin when a suspend ends without approval A disapproved or timed-out approval gate hands the flow back through the UpdateFlow channel with unrecoverable = true. That flag means "the previous step's worker died", and it is read by six sites. Five of them happen to want what it does here, but continue_on_same_worker and continue_with_runners do not: the worker that ran the approval step is alive, so unpinning the error handler and routing it by tag breaks the ./shared contract of a same_worker flow and can land it on a worker group that cannot run it — the same defect #10551 fixed for the three producers that hand back a live flow. Replace the boolean with StepFailureKind so the suspend producer can say "worker alive, but this failure is not the module's to handle" instead of overstating a worker death. The failed module's error policy is deliberately still bypassed: the failure is recorded against the step the gate was holding back, which never ran, so its retry would re-open the gate and its continue_on_error would skip it outright (verified: the gated step is marked Failure with a nil job id and the flow jumps past it). suspend. continue_on_disapprove_timeout remains the way to continue past a gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(flow-editor): flag that continue on error does not cover the approval gate A resolved approval is recorded against the step the gate holds back, not the step carrying the suspend, so continue_on_error never sees it: the flow still stops on a disapproval or timeout. Point users at suspend.continue_on_disapprove_timeout, which is what actually continues past a gate, whenever both settings are on and that one is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
25084170d7 |
feat: make job subprocess oom_score_adj configurable (#10443)
* feat: make job subprocess oom_score_adj configurable via JOB_OOM_SCORE_ADJ Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: warn when JOB_OOM_SCORE_ADJ leaves no gap over the worker's own score Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * style: drop em dash from JOB_OOM_SCORE_ADJ doc comment Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: warn on any oom_score_adj gap too small to steer the OOM killer Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 Alexander Petric
|
e0d6dc1a19 |
fix: harden flow-orchestration token refresh (mint from job_perms) (#10419)
* fix: harden flow-orchestration token refresh (mint from job_perms) * refactor: address review nits on flow token refresh |
||
|
Ruben Fiszel
|
81b23a2ba0 |
feat: make the fork lineage the only deploy relationship (#10410)
* feat: make the fork lineage the only deploy relationship `workspace_settings.deploy_to` (2023) and `workspace.parent_workspace_id` (2025) both expressed "which workspace does this one deploy into". Fork creation and dev-workspace attach seeded both, but nothing kept them in agreement, so every reader picked one and they disagreed. Drop `deploy_to`. A migration folds surviving pairs into the lineage: a sole claimant on a target with no dev workspace becomes that target's dev workspace and keeps its own job tags, while many-to-one pairs become plain forks. Pairs that the lineage cannot express -- dangling target, self-reference, chain, mutual -- are reported and left unlinked. Job tags were never lineage-aware: `per_workspace_tag` mapped any parented workspace to its parent while `$workspace` interpolated the raw id, so a fork running a script tagged `<tag>-$workspace` produced a tag no worker serves and the job queued forever. Both paths now resolve to the nearest ancestor whose id an admin would provision workers for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: preserve unconvertible deploy links and sweep tag caches on reparent Review findings on the deploy_to unification: - convert chains instead of discarding them, and keep whatever the lineage cannot express in workspace_deploy_to_unmigrated so the down migration can restore it - ignore soft-deleted workspaces when choosing between a dev workspace and a plain fork; an archived claimant was demoting live pairs - mirror attach_dev_workspace's git-sync strip, which the migration skipped - sweep the tag cache over whole subtrees on rename and delete: tag resolution now walks ancestors, so a nested fork kept a tag nothing serves - call a dev workspace a dev workspace in the settings copy - redirect a root away from ?tab=deploy_to instead of rendering an empty target Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: detect lineage cycles and record archived links in the deploy_to migration Second review round on the unification: - detect cycles over the lineage as it would exist after conversion, not over the deploy_to graph alone: a root whose target was one of its own forks closed a loop that no deploy_to edge revealed - record an archived source's link instead of filtering it out entirely, which dropped it with the column - treat a fork whose deploy_to merely repeats its parent as redundant rather than reporting every pre-existing fork as unmigrated - read the row count from the lineage update rather than the git-sync one - sweep the tag cache when archiving a dev workspace, the last site that mutates is_dev_workspace without one Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: resolve $workspace on preprocessed flow tags regardless of $args Third review round on the unification: - a flow tag containing only `$workspace` skipped interpolation entirely on the preprocessed path, because the branch that ran it keys on `$args`. The raw tag was written back and named a queue no worker serves. Resolve `$workspace` before the branch and leave `$args` to it. - record the new table's foreign key in the schema summary - describe what the archive tag sweep actually does: the dev flag is cleared for any archived workspace, which is why it is unconditional Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the deploy_to leftovers table only when it holds something * fix: sweep tag caches on archive only where the dev flag actually changes * feat: broadcast lineage changes and walk ws_specific ancestors only - propagate tag-cache invalidation across processes over notify_events: the cache is per-process, so replicas kept resolving stale lineage for the TTL. The listener clears the whole cache rather than tracking ids, since a single mutation invalidates an unbounded set of descendants and lineage changes are rare admin actions. - narrow list_ws_specific_versions to ancestors: walking down as well made a root fan out over its entire live fork subtree, and each member costs an identity lookup plus an RLS switch and probe. Ancestors are bounded by the fork depth limit. - probe the leftovers table unqualified so rollback restores on a PG_SCHEMA install, where search_path is not public - drop the nativets client method for the removed edit_deploy_to endpoint Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: let a prod see its dev workspace in ws_specific, and stop the walk oscillating Descending into plain forks made a root fan out over its whole live fork subtree, but a dev workspace is the paired editable environment rather than a throwaway copy, so a prod should still see it. There is at most one per parent and attach rejects nested dev chains, so that edge stays bounded. The edges run both ways, so the recursion never converged: it bounced parent<->dev until the depth cap on every call, 33 rows for a two-member set. A visited-path guard ends the walk when nothing new is reachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep dev pairings unnested, gate the delete broadcast, cover the ws_specific walk Fifth review round: - a root that already owns a dev workspace no longer converts: linking it under its deploy target would leave that dev nested beneath a fork, the shape attach_dev_workspace refuses to create. The link is preserved instead. - broadcast a lineage change on delete only when descendants are orphaned. Deleting a leaf, which ephemeral fork churn does constantly, changes nobody else's resolution and was making every replica drop its whole tag cache. - call list_ws_specific_versions in a test. plpgsql defers everything past a raw parse to the first call, so replaying the migration only proved it parses. - use unwrap_or_default for the descendant sweeps, which run after the transaction has committed; a transient failure must not fail the request - trim the traversal comment to the four-line limit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cache the renamed tally query and clear instance alerts on conversion The integration test's query was never cached: `cargo sqlx prepare` without --all-targets skips test targets entirely, and renaming its fixture workspace changed the query text. Regenerated with --all-targets --features all_sqlx_features,private, which is what lets the EE-gated otel test compile. Also from review: - clear error_handler_fallback_to_instance_alerts on converted workspaces. Dispatch ignores it once a parent exists, but the settings page keeps submitting the stored true, which the API rejects on a fork. - restore the schema summary row to the file's name: columns format and put it back in alphabetical order Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: never cache an unresolvable tag workspace, and unadvertise the removed endpoint - lookup_tag_workspace cached a "no row" result as self-resolution. A rename resolves the new id before its row lands, so a fork could be pinned to its own wm-fork-* id -- which nothing serves -- for the whole TTL, and its schedules kept re-pushing onto that dead tag. Fall back for the call without caching, matching how the error path already behaved. - change_workspace_id swept its children but never itself. Sweep the new and old ids and broadcast unconditionally, since a rename always changes lineage. - openapi-deref.{json,yaml} are served to clients via include_str!, so they were advertising edit_deploy_to after it started 404ing. The audit-action enum keeps the entry: historical rows still carry it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: align the served YAML spec with the JSON one and correct two comments - the YAML deref lost the removed path but kept deploy_to on get_settings, so the two served specs disagreed. Both are now identical. - the rename-sweep comment blamed cached-unresolvable lookups, which the same commit stopped caching. The real reason is that workspace ids are reclaimable, so a new id can carry a previous occupant's resolution. - the instance-alert comment claimed the settings page submits the stored true and gets a 400. It hides the option on a fork and sends false; the hazard is the value outliving the pairing and re-enabling alerts after a detach. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 82da6cb2bafeda18acd6b70c599013a12117ecb0 This commit updates the EE repository reference after PR #694 was merged in windmill-ee-private. Previous ee-repo-ref: f9ddf6a75aa13d1c13a3d7216a361a96f75ca435 New ee-repo-ref: 82da6cb2bafeda18acd6b70c599013a12117ecb0 Automated by sync-ee-ref workflow. * fix: grant the deploy_to preservation table to the windmill roles * test: drop the one-shot migration tests, keep the ws_specific execution guard The two conversion tests replayed the migration against the fully-migrated schema, which is not how it runs -- in production it runs mid-sequence against the schema as of that point. A later migration touching workspace or workspace_settings would break them without breaking anything real, and sqlx checksums already freeze a released migration. They earned their keep finding the archived-claimant and nested-dev cases during development; there is nothing left for them to guard. list_ws_specific_versions is different: it is live, no caller exercises it, and plpgsql only parses a function body until first call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: invalidate a reclaimed fork id cluster-wide without flushing every entry Gating the delete broadcast on orphaned descendants stopped leaf churn flushing every replica, but fork ids are reclaimable: the deleting process invalidated locally while every other replica kept the old parent for the TTL, so a job pushed in a recreated fork routed to the previous parent's tag. The broadcast payload now carries meaning. A workspace id drops that one entry, used for leaf deletion where exactly one id changed what it denotes. The `*` sentinel drops everything, used for attach, detach, archive, rename and deletions that orphan descendants -- reshaping a subtree no single id names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: name the right broadcast for each invalidation case * docs: attach does invalidate the tag cache; the resolver walks the whole chain --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
4d3ff0299f |
feat: mark failed jobs as resolved so handled failures stop showing red (#10319)
* feat: mark failed jobs as resolved so handled failures stop showing red Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: constrain auto-resolve to the proven retry chain and honor resolved filter everywhere Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: apply resolved filter to queue-union, concurrency and delete paths, bound note Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: sweep resolutions on workspace delete, verify helper args, enforce UI limits Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: count resolution note in characters on both sides of the API Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: skip the queue lookup for cancel-all under the resolved-only filter Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: converge retry auto-resolution from either commit order, keep notes on re-resolve Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct the idempotency claim on the retry auto-resolve sweep Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: gate resolution notes and attribution behind enterprise, add note popover Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: hide resolution from operators, exclude flow steps, enforce EE licence at runtime Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: add job_resolution.automatic to the summarized schema Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: preserve stored attribution when re-resolving without a valid licence Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: condense the attribution-preservation comment to four lines Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: validate resolution notes by code point instead of a UTF-16 maxlength Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the resolution popover open when a note is rejected Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: offer to resolve the original failure after a successful re-run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: verify supersession server-side and stop re-runs overwriting notes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: apply tag scope to the superseding run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: exclude obscured cross-workspace runs from resolution actions Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
30d8104edc |
feat: alert on expired online license key (#10295)
* [ee] feat: alert on expired online license key Wire alert_on_online_license_expired into the periodic monitor loop (verify_license_key_f), server-mode gated so only servers report to the alerts table and critical channels. Add the ee_oss stub so the enterprise-without-private build still compiles. Closes the gap where an expired online (renewable) license force-set externally (env var / CI / k8s) halted all jobs with only a stdout tracing::error! and no critical alert. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: point ee-repo-ref at license-expired-alert EE branch Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 9e63619688588fda434dfa06ed585f050e75ee67 This commit updates the EE repository reference after PR #682 was merged in windmill-ee-private. Previous ee-repo-ref: 7847711567a9c66da6a5c85d176b8cacc5aa7a7f New ee-repo-ref: 9e63619688588fda434dfa06ed585f050e75ee67 Automated by sync-ee-ref workflow. * chore: bump ee-repo-ref to license-expiry alert review fixes Point at the EE follow-up (windmill-ee-private#683): cross-replica dedup via acquire_lock and no false recovery on malformed key replacement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to eb8733a36010a7509b438903e95edfa293da079b This commit updates the EE repository reference after PR #683 was merged in windmill-ee-private. Previous ee-repo-ref: 562a306d9b66624b28ff90958f8b45d8db5481b9 New ee-repo-ref: eb8733a36010a7509b438903e95edfa293da079b Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
f02df7fc45 |
feat(monitor): make between-steps zombie flows hand-recoverable (#10287)
* feat(monitor): make between-steps zombie flows hand-recoverable When a worker is OOM-killed mid state-transition, the flow is reaped as a between-steps zombie (children all success, module still InProgress). We do not auto-recover (a re-driven transition can OOM again), so instead: - Append actionable recovery guidance to the cancellation reason when the reaped step's state is derivable (every child a success completion): which step, iterations completed, raise memory then restart-from-step (UI + API). - Restart-from-step now reuses a zombie step verbatim (InProgress with all children successful) and restarts from the next step, so no completed child re-runs; downstream steps re-derive its result from flow_jobs on demand. - Cast flow_status ::text in the reaper query: reading the jsonb column as Box<str> included the binary version byte and silently failed FlowStatus parsing (disabling the restart-not-yet-started branch since the v2 migration). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): only reuse a between-steps zombie step that provably finished Address review findings on the zombie-restart reuse path: - Require structural completeness (FlowStatusModule::is_between_steps_complete): a serial for-loop / branch-all reaped mid-fan-out has an all-success prefix but unrun remaining iterations, so the cursor must sit on the last element; while-loops are never derivable (continuation is a post-iteration condition). Parallel containers preallocate all children, so success alone is conclusive. Shared by the monitor guidance and the restart resolution. - Decline reuse when the step carries stop_after_if / stop_after_all_iters_if: those predicates decide whether downstream steps run, and reuse would bypass them; such a step re-runs instead. - Decline reuse when the zombie step is the last module (advancing past it lands on the failure step); it falls back to the existing re-run path. - Unit tests for is_between_steps_complete and an integration test asserting a mid-iteration serial-loop zombie is re-run, not reused. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): align zombie recovery guidance with restart eligibility Address CI review findings: - Exclude skip_if / suspend / sleep (not just stop predicates) from reuse via FlowModule::allows_zombie_reuse, so a skipped/suspend-armed step is never synthesized as Success (which would strand a restart waiting on an approval it never armed). - The reaper does not load the flow definition, so it cannot know whether restart will reuse or re-run a given step; reword the guidance to state both outcomes (reuse where derivable, re-run for the flow's last step or one carrying a stop/skip condition, approval, or sleep) instead of promising "no re-run". - Make the mid-iteration regression test exercise the cursor-completeness guard: a downstream step makes the loop non-final, so reuse is prevented only by the guard; a truncated loop result would then fail the assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): never let zombie reuse swallow a nested restart request A nested restart (RestartedFrom.nested) descends into the restart step's child to re-run an inner step. For an eligible zombie BranchOne/Subflow the outer branch_or_iteration_n is None, so reuse fired, skipped the container, and the explicitly requested inner step never re-ran. Thread the presence of a nested chain into restarted_flows_resolution and decline reuse when set. Regression test added (RED without the guard: the nested target is reused instead of re-run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): don't auto-requeue preprocessor zombies as unstarted flows The ::text parse fix re-activated the "hasn't started yet, restart it" branch, but its `modules[0] == WaitingForPriorSteps` check also matches a flow whose preprocessor is still InProgress (step == -1, first module waiting). Requeuing such a flow re-runs the preprocessor, duplicating side effects / repeating the OOM. Gate the branch on FlowStatus::is_not_yet_started, which also requires the preprocessor (if any) to be WaitingForPriorSteps. Unit-tested. Also drop the numbered procedural narration from the happy-path test comments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): only emit restart guidance for restartable (deployed, top-level) flows The recovery guidance points operators at the run page's "Re-start from" button and the restart API, but both require a top-level deployed flow: a preview has no flow path (the button is hidden, the API 400s) and a subflow child restarts via its root, not itself. Gate the guidance on runnable_path IS NOT NULL AND parent_job IS NULL so previews/subflows keep the existing wording instead of being told to use a button/endpoint that isn't there. Verified end-to-end: a reaped preview gets no RECOVERY block, a reaped deployed flow does. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): gate recovery guidance on kind='flow' to match the restart surface Addresses review nit: a pathful editor preview (kind='flowpreview' with a runnable_path) satisfied the previous runnable_path check but the run page only renders the "Re-start from" button for kind='flow'. Match that condition exactly so previews/singlestepflow keep the plain wording. Verified end-to-end: a reaped pathful preview now gets no RECOVERY block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): disable zombie reuse for raw-flow (editor preview) restarts A JobPayload::RawFlow restart queues the request's current, possibly EDITED, definition, but restarted_flows_resolution validates reuse against the completed job's STORED definition. For an eligible preview zombie, editing the restart step and restarting from it would synthesize Success from the old children and skip the edit. Thread allow_zombie_reuse into the resolver (true only for JobPayload::RestartedFlow, which queues the stored definition) and decline reuse for raw-flow restarts. Regression test added (RED without the guard: the edited step is skipped and the old result is reused). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(sqlx): add offline cache for zombie_flow_recovery test queries The integration test's UPDATE v2_job_completed queries had no .sqlx entry, so the CI SQLX_OFFLINE build of the test failed to compile. Regenerated with --all-targets --features deno_core,quickjs to capture the test-target queries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(monitor): drop procedural narration from the raw-flow zombie test Per AGENTS.md (comments record constraints, not narration): remove the two step-describing comments the reviewer flagged; the test doc comment already carries the durable rationale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): restrict zombie reuse to monitor-reaped flows The reuse predicate matched the InProgress/all-children-success shape without checking provenance, so an ordinary force-cancel at the same boundary (a child succeeded before its parent transition landed) would also be reused, dropping the usual restart-from-step re-run. Gate reuse on canceled_by = 'monitor' (the username the zombie reaper cancels with). Regression test added (RED without the guard: a user-cancelled flow reuses the child instead of re-running it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(monitor): reuse zombie step on Some(0) too, so the run-page button works The run page's "Re-start from" button always sends branch_or_iteration_n = 0 (never omits it), but reuse only fired for None, so the exact UI path the recovery message points to would re-run the children instead of reusing them. Treat a whole-step restart (None or Some(0)) as reuse-eligible; Some(n>=1) keeps the explicit partial-container restart. Verified against the live EE restart API with branch_or_iteration_n=0: all loop-iteration child UUIDs are reused. Happy- path test now sends Some(0) to match the button. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
fa3644281f |
fix(monitor): diagnose zombie-flow OOM on the transition worker, not q.worker (#10286)
handle_zombie_flows() joined worker_ping on the flow's queue-row worker
(q.worker) only. In nested/subflow/forloop cases that worker is frequently
NOT the one that performed the flow's final state transition, so all OOM
detection ran against a healthy bystander and the diagnostic wrongly told
on-call to chase a deadlock / SIGQUIT.
Change 1: when q.worker itself looks healthy, additionally search worker_ping
for a plausible culprit — a different worker on the same pod (worker_instance)
or worker group whose last ping clusters around the flow's last_ping and has
since gone silent (the signature of a worker OOM-killed mid state-transition),
picking the one closest in time to the transition. If found, the message names
that worker and routes to the OOM explanation. The extra query runs only on the
rare zombie path and fails soft (warn + fall back) so it never blocks the cancel.
Change 2: reorder the healthy-q.worker hint to lead with "check whether a
different worker on the same pod/group was OOM-killed around {last_ping}",
keeping deadlock/SIGQUIT as the secondary possibility.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Ruben Fiszel
|
ddec2abbb3 |
feat(jobs): cap total queued jobs per workspace on cloud (#10218)
* feat(jobs): cap total queued jobs per workspace on cloud A workspace could flood the queue with an unbounded number of jobs across many concurrency keys and scripts (or keyless jobs), which the per-key cap from #10197 does not bound. Add a companion instance-wide ceiling on a workspace's total queued jobs. check_workspace_queue_cap rejects a push once the workspace has WORKSPACE_MAX_QUEUED_JOBS (default 20000, superadmin-configurable, 0 to disable) jobs queued, cloud-only and runtime-gated on CLOUD_HOSTED like the per-key cap. It runs on every push, so it applies even to premium workspaces and catches parallel for-loop floods. Jobs already queued still drain; only new pushes past the ceiling are rejected, so an in-flight flow only fails to push further work while at the ceiling. The setting loader self-gates on CLOUD_HOSTED so it is never loaded off cloud, from initial load or a settings-change reload. The depth count is bounded by the cap via LIMIT so a runaway backlog never costs an unbounded scan on the push path. * docs(jobs): note the workspace cap is a soft ceiling and the depth helper is count-only Records the two review points as constraints: the cap does not serialize admission (a soft ceiling by design, like the per-key cap), and workspace_queue_depth is pub only for the test, returns a count not job data, and leaves authorization to the caller. |
||
|
hugocasa
|
11fda89b52 |
feat(telemetry): generic feature-usage telemetry with AI session metrics (#10200)
* feat(telemetry): add generic feature_usage table and batched logging endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(telemetry): log AI session usage events and document them in telemetry settings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): use escape sequence instead of literal NUL bytes in buffer key Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): validate dimensions, decouple retention, keepalive flush Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): allowlist feature-usage dimensions and index retention scans Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): pin tool-name allowlist and deploy session attribution Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(telemetry): route AI chat usage through feature_usage and drop ai_chat_usage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(telemetry): slim dimension validation to registered kinds plus key shape Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): backfill ai_chat_usage into feature_usage before dropping it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): disclose provider and model identifiers in telemetry settings text Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): issue all flush chunks before awaiting so pagehide keeps them Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 6306c072a50937ea9af44a5bcf42345543207486 This commit updates the EE repository reference after PR #672 was merged in windmill-ee-private. Previous ee-repo-ref: 964f242a0eb44db7f7d26636cc8d76aeabea2b73 New ee-repo-ref: 6306c072a50937ea9af44a5bcf42345543207486 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
hugocasa
|
87be041c09 |
fix(git-sync): avoid percent-encoded colon in git-sync hub script path (#10213)
* fix(git-sync): avoid percent-encoded colon in git-sync hub script path The git-sync init/detection hub script slug contained a colon stored as `%3A` in the run-by-path URL. The generated API client re-encodes path params with encodeURI, turning `%3A` into `%253A` (double-encoding). Some hardened reverse proxies / WAFs reject double URL-encoding and return a bare 400 before the request reaches Windmill, breaking git-sync repository detection on those instances. The hub resolves scripts by numeric id and ignores the slug, so dropping the colon from the slug is behavior-neutral (same script, same id-keyed worker cache) while producing a colon-free run URL. Also force-cache GIT_SYNC_PULL_SCRIPT_PATH at build alongside LATEST_GIT_SYNC_SCRIPT_PATH so the backend-driven pull script is always baked into the image for airgapped workers, instead of relying on an incidental hubPaths.json overlap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for git-init slug match fix Pulls in windmill-ee-private#676 so the EE is_git_init_script check matches the colon-free git-init hub slug (GitHub App token grant for git-sync jobs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for git-init slug helper + test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to a3adea1ffb406e709cc480871df58fab6c51aca1 This commit updates the EE repository reference after PR #676 was merged in windmill-ee-private. Previous ee-repo-ref: cef4e008ef62dec434aa9bb3ec783db8aff6a1c1 New ee-repo-ref: a3adea1ffb406e709cc480871df58fab6c51aca1 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
71f2d47cb4 |
feat: cap queued jobs per concurrency key on cloud (#10197)
* feat: cap queued jobs per concurrency key on cloud * fix: close preprocessed-flow bypass and bound concurrency cap scan * fix: only cap concurrency keys with an active concurrent_limit * chore: only load concurrency key cap setting when cloud hosted * fix: reject queued-job import on cloud |
||
|
Ruben Fiszel
|
f6e36f862e |
chore(schedules): lower reconciler re-arm back-off cap to 8 passes (#10182)
Follow-up to #10179. The exponential back-off between reconciler re-arm retries of a persistently-failing schedule capped at 32 passes (~2.7h at the default 5-min reconcile cadence). Lower the cap to 8 (~40min) so a schedule fixed out of band (a lapsed license renewed, a bad cron corrected directly in the DB) auto-recovers within a few passes, while still cutting the retry rate sharply versus retrying every pass. Fixes via the UI/API re-arm immediately and are unaffected. Fixes WIN-2198 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
c82056cfde |
fix(schedules): stop disabling schedules on transient push errors (#10179)
* fix(schedules): stop disabling schedules on transient push errors A scheduled flow whose next-occurrence push failed after retry exhaustion used to be disabled, killing a healthy schedule over a transient DB blip (pool contention, statement timeout). Now that the unarmed-schedule reconciler exists (#10174), transient failures no longer disable: the current occurrence runs to completion and the reconciler re-arms the next occurrence once this run leaves the queue. In the flow schedule-push path after retry exhaustion we now branch on the error: QuotaExceeded/NotFound still disable (the schedule's own fault, and rearm_schedule would otherwise leave them enabled-yet-unarmed forever), while transient errors are only reported and the flow continues. The previous iteration returned a SchedulePushZombieError to force a zombie restart; that is removed, because zombie detection cancels (does not restart) same-worker flows, so it would have lost the current run of a same-worker scheduled flow. The now-obsolete SchedulePushZombieError type and its catch in worker.rs are deleted. Fixes WIN-2198 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(schedules): back off and surface repeated reconciler re-arm failures The unarmed-schedule reconciler retried a schedule that could not be re-armed on every pass, forever, logging only to the server. With the flow schedule-push path no longer disabling on non-transient errors, a persistently-broken push (bad stored cron/timezone/args, lapsed license key) now stays enabled and would spin in that loop silently. The reconciler now tracks consecutive re-arm failures per schedule: exponential back-off (2, 4, 8, … passes, capped) between retries so a broken schedule is not hammered, and after 3 consecutive failures it surfaces the cause once (records schedule.error + raises a critical alert) without disabling. Both reset the moment the schedule re-arms, which also clears the recorded error. Verified end-to-end on a running server: a flow schedule with a corrupted cron stays enabled, retries back off, the error is surfaced after the third failure, and it re-arms and clears the error once the cron is fixed. Fixes WIN-2198 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
9762089fcb |
fix(schedules): re-arm enabled schedules left with no queued occurrence (#10174)
* fix(schedules): re-arm enabled schedules left with no queued occurrence * fix(schedules): lock schedule row while re-arming and report outcome * fix(schedules): make reconcile lock cancellation-safe, re-check armed under lock Address review feedback on the schedule reconciler: - Use a transaction-scoped advisory lock (pg_try_advisory_xact_lock) instead of a session-scoped one. monitor_db runs under a 600s timeout; on cancellation a session lock on a pooled connection would be stranded, wedging reconciliation on every replica. An xact lock releases when its transaction is dropped. - rearm_schedule re-checks for a queued occurrence under the row lock and returns NoOp if already armed, closing the scan→lock window that could double-push across a cron boundary. Add a regression test. - Make reconcile_unarmed_schedules private (its only caller is in monitor.rs) and document its system-only contract. - Log the disable only after the guarded UPDATE actually disables the schedule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(schedules): never disable from reconciliation and cap re-arms per pass --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Alexander Petric
|
396fb1c475 |
feat(otel-tracing-proxy): trust internal endpoints with untrusted CAs (#10139)
* [ee] feat(otel-tracing-proxy): trust internal endpoints with untrusted CAs Add `insecure_upstream_hosts` and `upstream_ca_certs` to the HTTP Request Tracing settings so the OTEL tracing proxy can reach internal endpoints with untrusted or private-CA certificates while keeping them traced. Wires the two settings through the worker config and live reload, adds the inputs to the instance settings UI, and pulls in the rustls upstream-client deps (hyper-rustls/tokio-rustls/rustls/ rustls-native-certs/rustls-pemfile; hyper-http-proxy switched to its rustls feature). The proxy-side implementation lives in the companion EE PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref to otel_ca companion commit Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(otel-tracing-proxy): expose new fields in declarative config; bump ee-ref Addresses code-review findings: - Add `insecure_upstream_hosts` and `upstream_ca_certs` to the declarative `OtelTracingProxySettings` in instance_config.rs so operator/GitOps-managed installs can set them and reconciliation no longer drops values saved via the UI. - Restore the trailing newline on ee-repo-ref.txt and bump it to the companion EE commit carrying the strict host-matching / port-ordering fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 51e50629f48dbc4f5520a787b4bdfb76f4cd38d3 This commit updates the EE repository reference after PR #665 was merged in windmill-ee-private. Previous ee-repo-ref: 49f458e4446395e98915c220baa757ab3b2ed2d8 New ee-repo-ref: 51e50629f48dbc4f5520a787b4bdfb76f4cd38d3 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasa
|
51d8db6602 |
feat: automatic git-to-windmill sync (polling, webhooks, in-app PRs + checks) (#9552)
* docs: add design doc for automatic git-to-windmill pull sync
* docs: add migration plan and implementation phases to git-sync pull design
* feat(git-sync): add auto_pull settings schema and pull enqueue primitive
Adds AutoPullSettings/AutoPullMode/AutoPullStatus on GitRepositorySettings
(workspace_settings.git_sync JSONB), the GIT_SYNC_PULL_SCRIPT_PATH constant,
and should_pull/effective_poll_interval_s helpers with unit tests. Exports the
EE enqueue_git_pull_job primitive. Foundation for repo→Windmill auto-pull.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): poll repos and auto-pull new commits into the workspace
Phase 1 of automatic repo → Windmill sync. A monitor task (EE-licensed,
single-replica via advisory lock) git ls-remotes each auto-pull-enabled
repository ~every minute and enqueues a pull when the tracked branch moves,
reusing the {workspace_id}:git_sync concurrency key so pulls serialize with
in-flight push commits.
- windmill-store: background (no-authed) resolver get_git_repo_head_for_autopull
that resolves the repo resource (incl. $var: refs) and ls-remotes; GitHub-App
repos are skipped here and will sync via webhooks (phase 2).
- monitor.rs: poll/reconcile/persist with optimistic sha advance and failure
status; targeted jsonb update so concurrent settings edits aren't clobbered.
- edit_git_sync_repository: preserve server-owned auto_pull state on UI save.
- openapi: AutoPullSettings/AutoPullMode/AutoPullStatus + auto_pull field.
- frontend: per-repo "Automatically deploy changes from Git" toggle with last
sync status; demote the GitHub Actions link to an advanced CI option.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): wire webhook lifecycle + receiver; share reconcile logic
OSS side of phase 2 auto-pull webhooks:
- edit_git_sync_repository creates/removes the repo webhook on save (EE-gated,
best-effort → falls back to polling).
- monitor poller now delegates to the shared windmill_git_sync reconcile/persist
helpers (also used by the webhook receiver), removing duplicated logic.
- export the shared reconcile/persist/failure helpers; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref for phase 3 in-app PR creation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): show webhook vs polling status on the auto-pull toggle
When a repo has an active webhook (auto_pull.webhook_id set), the status line
reads "instant via webhook"; otherwise it reads the ~1-minute polling cadence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): post PR diff check on dry-run completion (phase 4)
Worker completion hook in process_completed_job: when a DeploymentCallback job
carrying the __git_sync_pr_check marker finishes, parse the dry-run SyncResponse
and patch the GitHub check run with the diff summary (success/neutral/failure).
Export enqueue_git_pull_dry_run; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref (drop unused GHES webhook_secret)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* revert(git-sync): defer phase 4 PR diff checks (OSS side)
Remove the worker completion hook that posted the PR check run, drop the
enqueue_git_pull_dry_run re-export and the orphaned sqlx cache, bump EE ref.
Phases 1-3 (polling, webhooks, in-app PR creation) are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "revert(git-sync): defer phase 4 PR diff checks (OSS side)"
This reverts commit
|
||
|
Ruben Fiszel
|
3bd9f05938 |
fix(alerts): identify server replica in low-disk alert + per-host dedup tag (#10143)
* fix(alerts): identify server replica in low-disk alert + per-host dedup tag The server-mode low-disk alert keyed its dedup tag on the mountpoint alone, so `simple_alert_helper` mapped every server replica onto a single alert row per mountpoint. With more than one replica that row flaps every monitor pass: a replica seeing low disk raises the alert while a replica seeing healthy disk recovers it. The alert text also could not say which replica tripped. The fix lives in windmill-ee-private (`low_disk_alerts` in windmill-common/src/ee.rs) and appends the hostname to both the message and the dedup tag, mirroring the worker branch. Also add a regression test pinning the server tag as per-host, and correct the monitor cadence comments: iterations are LISTEN_NEW_EVENTS_INTERVAL_SEC (10s by default), not 30s, so "~60s (2 iterations * 30s)" was wrong on both factors. * fix(alerts): widen healthchecks.check_type so per-host disk tags fit Alert tags embed a mountpoint and a hostname, both unbounded, but check_type was varchar(50). create_alert only logs the insert error while the notification still fires, so an overflowing tag re-alerts every monitor pass and never records recovery state. The server tag overflows for ordinary pod-length hostnames, and the existing worker tag already overflows for every tracked mount except "/". Widening the column fixes both; bounding the hostname would not, since the mountpoint alone can consume the budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 This commit updates the EE repository reference after PR #666 was merged in windmill-ee-private. Previous ee-repo-ref: ccd1e42cf6b2d051ca17074fbdf5b80a46cffe0f New ee-repo-ref: b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
f7eb5c460d |
fix(apps): invalidate cached app policy on change or deletion (GHSA-r5v4-cxh9-7qhq) (#10121)
* fix(apps): invalidate cached app policy on change or deletion (GHSA-r5v4-cxh9-7qhq) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(agents): keep PR tests and comments minimal and non-ephemeral Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
ff774c46bf |
feat: add per-workspace job-retention override (#10050)
* feat: add per-workspace job-retention override (EE) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 2ba6a2a75b6fc97858b306b2c98ada481e363c10 This commit updates the EE repository reference after PR #658 was merged in windmill-ee-private. Previous ee-repo-ref: e7fb36acd813cd717bcf05f5aafbf81de271d618 New ee-repo-ref: 2ba6a2a75b6fc97858b306b2c98ada481e363c10 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
f28ea9cb99 |
feat(db-health): add connection sizing guidance (#10014)
* feat(db-health): add connection sizing guidance The Database Connections panel showed current/max connections but gave no guidance on how to size max_connections for the deployment. Derive an estimate from the live worker fleet: each worker instance shares a pool sized DEFAULT_MAX_CONNECTIONS_WORKER + (workers - 1), and each server opens up to DEFAULT_MAX_CONNECTIONS_SERVER (both overridable via DATABASE_CONNECTIONS). The endpoint now returns live worker/instance counts, the default per-server and per-worker pool sizes, the estimated peak worker connections, the reserved superuser connections, and a recommended max_connections floor (workers + one server + 25% headroom). Servers do not ping worker_ping, so the recommendation assumes one server and exposes the per-server increment. The panel renders this as a sizing breakdown and warns when max_connections is below the recommended floor. Fixes WIN-2147 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(db-health): single source for pool-size constants + sizing tests Address review: db_connect.rs kept its own copies of DEFAULT_MAX_CONNECTIONS_* that duplicate the windmill_common constants the sizing guidance reads, so tuning the runtime pool size would silently leave the guidance stale. Re-export the windmill_common constants from db_connect.rs so there is one source of truth. Add unit tests for compute_connection_sizing covering the zero-fleet, single worker, multi-instance, and reserved-clamp cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(db-health): 20% headroom and 200-connection minimum floor Lower the sizing headroom from 25% to 20% and never recommend below 200 connections (postgres defaults to 100; cheap headroom for growth/bursts/psql). Update the guidance message and unit tests accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-health): honor DATABASE_CONNECTIONS in sizing recommendation Address Codex P1: the runtime caps every process's pool at DATABASE_CONNECTIONS when set (db_connect.rs), but the sizing guidance always used the default 50/5 pools. For a tuned deployment this under-estimated worker demand and could hide a genuine under-provisioning (e.g. DATABASE_CONNECTIONS=100 with 5 instances is 500 worker connections, not 25). compute_connection_sizing now takes the effective DATABASE_CONNECTIONS override (read the same way db_connect.rs reads it): when set, each worker instance and server pool is that value and the worker estimate is override * instances. The response exposes server_pool_size / worker_pool_size (effective) and database_connections_override; the panel renders both pool rows and labels them (default) vs (DATABASE_CONNECTIONS), and the message states which source is used. Adds a unit test for the override path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-health): exclude agent workers from connection sizing Agent workers reach the API over HTTP (MODE=agent, Connection::Http) and hold no postgres pool, but their pings still land in worker_ping (written server-side by /api/agent_workers/update_ping). Counting them inflated the connection estimate. Filter the fleet query by the worker-name prefixes: DB-connected workers use "wk-" (WORKER_NAME_PREFIX), agent workers use "ag-" (AGENT_WORKER_NAME_PREFIX). Only wk- workers/instances feed the estimate; ag- workers are counted separately and surfaced as context ("N agent workers excluded — they use HTTP, not postgres connections"). Adds a unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
91e1b087a2 |
feat(auth): add runtime NO_AUTH mode for authentication bypass (#9962)
* feat(auth): add runtime NO_AUTH mode for authentication bypass Adds a runtime `NO_AUTH` env flag that makes every request resolve as the `admin@windmill.dev` superadmin with no login required, so self-hosted deployments can front Windmill with their own authenticating gateway without building a dedicated `oss` (compile-time `no_auth`) binary. - `NO_AUTH` is honored in any build but is force-disabled when `CLOUD_HOSTED` is set, so the managed cloud always enforces real auth. - The existing compile-time `no_auth` feature keeps its always-on behavior (`cfg!(feature = "no_auth") || *NO_AUTH`), so `oss` builds are unchanged. - `Tokened` now yields a synthetic token in no-auth mode so handlers that require it (e.g. global_whoami, called by the frontend on load) resolve. - A loud startup banner warns when the mode is on; `HIDE_NO_AUTH_BANNER` silences it once the operator has deliberately deployed behind a gateway. Fixes WIN-2131 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(auth): dismissable NO_AUTH warning banner via global setting Replaces the HIDE_NO_AUTH_BANNER env flag with a UI warning banner that can be permanently dismissed for all users from within the running instance (not exposed in instance settings). - New `no_auth_banner_dismissed` global setting, only ever written by dismissing the banner itself. - `GET /api/settings/no_auth_banner` returns whether to show the banner (true only when NO_AUTH is active and it hasn't been dismissed). - NoAuthBanner.svelte renders a top-of-app warning in NO_AUTH mode; its dismiss button opens a confirmation modal, then writes the global setting via the existing setGlobal endpoint so it stays hidden for everyone. - The server still logs the startup NO_AUTH warning unconditionally. Fixes WIN-2131 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(auth): resolve NO_AUTH in AuthCache so all_runnables works Codex/Pi review flagged that `/api/users/all_runnables` still failed in NO_AUTH mode: `get_all_runnables` extracts `Tokened` and re-validates the request token per workspace via `AuthCache::get_authed`, which rejected the fabricated `"no_auth"` token (no matching DB row) with a 400. Short-circuit `AuthCache::get_opt_job_authed` (the resolver behind `get_authed`) to the admin superadmin in no-auth mode, so any direct cache caller resolves without a real token. Single-source the mode check and the synthetic identity via `is_no_auth()` / `no_auth_admin_authed()` and reuse them across the extractor, resolver, and login paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * revert(auth): drop the NO_AUTH dismissable UI banner The in-app banner added a GET /api/settings/no_auth_banner request to every instance load for little benefit. The startup log warning already surfaces that auth is bypassed to operators, so drop the banner, its endpoint, and the no_auth_banner_dismissed global setting entirely. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
5d7fb6deca |
feat(pipelines): asset freshness — fresh/stale badge (CE) + watchdog (EE) (#9909)
* feat(pipelines): passive asset freshness tracking on the graph Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(pipelines): drop dead freshness-enforcement stub, document query ordering Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): freshness watchdog (EE) — auto re-run stale producers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): watchdog review fixes — archived workspaces, badge kind parity, scan index Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): CI review — no singlestepflow in freshness, +N parity, completion-time fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): CI review — history completedAt, freshness/asset trigger UI metadata Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 6f5fe0f7f56696fbef5a8349da38496c32e71666 This commit updates the EE repository reference after PR #643 was merged in windmill-ee-private. Previous ee-repo-ref: 1f13380354bf591ae25a2c20d36917534bcc5459 New ee-repo-ref: 6f5fe0f7f56696fbef5a8349da38496c32e71666 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
84141add1d |
feat(pipelines): workspace duckdb macro libraries (// macros / // use) (#9890)
* feat(pipelines): parse duckdb macro-library annotations (// macros, // use) * feat(pipelines): duckdb macro registry tables + deploy-path validation and writes * feat(pipelines): inject workspace duckdb macros into consumer jobs at run time * feat(pipelines): surface macro libraries and lib-consumer edges in asset graph api * feat(frontend): macro-library nodes, lib-consumer edges and scaffold in pipeline graph * docs: mark dbt gap #7 (packages/macros) shipped via workspace macro libraries * fix(pipelines): review fixes - char-safe parsing, local macros win, fork clone, trust-model docs * feat(frontend): duckdb macro autocomplete + workspace macro explorer drawer * fix(pipelines): address CI review - use-setup retention, splice past local defs, orphan filter, full consumer rescan, index-keyed strip * fix(pipelines): inject provider library setup for implicitly-called macros too * fix(pipelines): rls-gate macro listing + honor library-level // use transitively * fix(pipelines): weave injected macros around local definitions by bind order * fix(pipelines): injected library setup always runs before user blocks * perf(pipelines): cache macro registry per workspace with notify-event invalidation * perf(pipelines): disable macro registry cache on cloud |
||
|
Ruben Fiszel
|
d15033cde6 |
fix: invalidate bun bundle cache on transitive relative-import changes (#9891)
* fix: invalidate bun bundle cache on transitive relative-import changes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: do not memoize transient fetch errors in bundle-key import cache Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: use regular comment on lazy_static block (deny unused_doc_comments) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: align bundle-key import version selection with loader content endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
5549bdc67a |
fix(debounce): never supersede a running debounce survivor (#9780)
* fix(debounce): never supersede a running debounce survivor
Companion to the windmill-ee-private change in upsert_debounce_key.
With debounce_args_to_accumulate + a concurrent_limit, a message arriving
while its debounce survivor is already running was marked completed/skipped
("Debounced Running by ...") and the running survivor deleted from the
queue, silently dropping accumulated elements. A slow step + concurrent
limit keeps the survivor running for a long window, so any arrival during
it was lost. The fix leaves a running survivor untouched and starts a fresh
debounce window for the late arrival.
Adds regression coverage in windmill-queue/tests/debounce_test.rs (push,
flow post-preprocessing, no-accumulation, committed-running, and
max-count-window cases) and refreshes the SQLx cache for the changed
upsert_debounce_key queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): add missing SQLx cache for test-only running-flag query
The cargo_test CI job compiles the test target with SQLX_OFFLINE=true; the
new regression tests use `UPDATE v2_job_queue SET running = true ...` which
was not in the offline cache (the library-only `cargo sqlx prepare` skipped
test targets). check_oss/check_ee passed because they don't build tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): harden running-survivor guard against concurrent arrivals
Companion to windmill-ee-private: switch the running-state check to a
correlated EXISTS on the post-conflict-lock holder so two late arrivals
racing after a survivor started running can't both spawn independent
windows (the row lock serializes them; the second debounces into the
first's fresh window).
Adds a concurrent regression test
(test_debounce_concurrent_arrivals_after_running_survivor) asserting
exactly one late arrival survives and the other is debounced, and refreshes
the SQLx cache for the updated upsert_debounce_key queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): serialize upsert per key (simpler, race-free)
Companion to windmill-ee-private: the running-survivor guard and batch
chaining are now protected by a per-key advisory lock instead of
snapshot-sensitive single-statement SQL. This closes a concurrent-arrival
data-loss race where a debounced late arrival's args could be dropped
because the batch lookup couldn't see the predecessor's just-committed
batch row.
Extends test_debounce_concurrent_arrivals_after_running_survivor to pull the
survivor and assert its accumulation includes BOTH racing late arrivals
(shared batch), and refreshes the SQLx cache for the rewritten queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): atomic upsert robust to concurrent pull-time key deletion
Companion to windmill-ee-private: keep upsert_debounce_key a single atomic
INSERT ... ON CONFLICT DO UPDATE so a chaining push cannot fail when the
worker pull path concurrently deletes the holder's debounce_key (the prior
read+UPDATE split could hit "no row updated"). Adds
test_debounce_push_races_key_deletion_by_pull (races a chaining push against
the key deletion 50x, asserts the push never errors) and refreshes the SQLx
cache.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(debounce): claim-based exactly-once batch consumption
Eliminates the rare duplicate/loss when two survivors land on one debounce
batch (a narrow push/pull race), without locking the worker pull hot path.
- migration: v2_job_debounce_batch gains consumed_at + consumed_by.
- pull side (maybe_apply_debouncing): instead of deleting the batch on consume,
a survivor atomically claims its own row + any unclaimed siblings (stamping
consumed_by = itself) and accumulates exactly the rows it claimed. A second
survivor of the same batch finds its row already consumed by another job and
runs empty (no duplicate); a re-pulled survivor recognizes its own prior claim
and keeps its accumulated args; a never-batched job (CE/legacy) keeps its own
args. Non-accumulate debounce paths still hard-delete their batch rows.
- complete_debounced_job (EE companion) never completes a running predecessor,
so its in-flight run is not killed (no loss); the claim then prevents the
duplicate the guard would otherwise allow.
- monitor: GC sweep deletes consumed batch rows past a 1h grace.
Together with the running-survivor guard this makes debounce accumulation
exactly-once. Adds tests: batch_consumed_exactly_once, repull_keeps_accumulated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): exhaustive edge cases + tighten consumed-batch GC grace
Tighten the consumed debounce-batch GC grace 1h -> 10min: per-op cost of the
claim is unchanged (an indexed mark is as cheap as the old delete), so the only
cost of retaining consumed rows is table growth, which a shorter grace bounds
under high-throughput debounce (a survivor that could still reference a row is
pulled long before 10min; GC is not correctness-critical since a re-pull whose
row was swept falls back to its persisted args).
Adds edge-case tests: never-batched keeps own args (CE fallback), concurrent
claim partitions a batch disjointly (exactly-once under real concurrency),
three survivors -> first takes all / rest run empty, non-accumulate debounce
hard-deletes its batch rows (no leak), and the GC sweep deletes only
past-grace consumed rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): port the #9781 regression case, flow-node guard, full-path bench
- Port the regression from #9781
(test_post_preprocessing_debounce_into_running_survivor_loses_message):
post-preprocessing survivor accumulates + runs, a later same-key message must
start a new batch (survive) not be folded into the running survivor. Exercises
the full EE path via jobs_ee::maybe_debounce_post_preprocessing.
- Add the third EE entry point's guard:
test_flow_node_debounce_running_survivor_not_superseded (maybe_debounce_flow_node).
- Add an #[ignore] full-source throughput bench (bench_debounce_full_path) driving
the real maybe_debounce + maybe_apply_debouncing end-to-end.
All debounce tests exercise the real jobs_ee implementation (run with
--features private,enterprise); none stub it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): scalar-arg accumulation + GC-then-repull no-loss
Close two accumulation edge gaps (both run on --features private,enterprise,
exercising the real jobs_ee path):
- accumulate bare-scalar values (the T | T[] union fallback): each scalar is
wrapped and accumulated into the survivor's list.
- GC reclaiming a survivor's consumed batch row before a re-pull must not lose
data: the re-pull finds no row and keeps its already-persisted accumulated
args (had_row=false fallback), rather than running empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): real-worker end-to-end accumulation test
Drives the full real path on --features enterprise,deno_core,private: push 3
same-key debounced flow jobs (real push() -> maybe_debounce collapses the
batch), a real worker pulls the survivor (real pull() -> maybe_apply_debouncing
claim+accumulate) and executes the deno flow, then asserts the executed result
is the full accumulated set [1,2,3] and the two superseded messages are skipped.
Complements the in-process unit tests with a genuine worker-execution run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): atomic claim+persist, GC only non-queued rows; reword comment
Address review findings:
- [P1] Claim and accumulated-args persist are now in one transaction. Before,
a crash between stamping batch rows consumed_by=self and the `UPDATE v2_job
SET args` could let a zombie re-pull see its own prior claim and keep only its
own args (dropping the siblings it had claimed). Wrapping claim + accumulate +
persist in a tx makes them commit together or roll back together (re-pull then
re-claims cleanly).
- [P1] GC of consumed batch rows now also requires the job to no longer be in
v2_job_queue. A consumed sibling can stay queued well past any time grace under
a concurrency limit / backlog; reclaiming its marker by age alone let its
eventual pull treat it as never-batched and re-run its item (a duplicate).
Keeping the row until the job leaves the queue preserves the "already consumed"
signal. Test extended with a still-queued consumed row that must survive GC.
- [P2] Drop "Customer" attribution from a test doc comment (AGENTS.md).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): emit accumulation log after committing the claim transaction
append_logs opened a second pool connection while the claim transaction (and its
batch row locks) were still held; under concurrent debounced pulls that risks
pool-exhaustion stalls/timeouts. Defer the log line until after tx.commit().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 6aabd7c5ce53b9153be05c3e7bc9a76eadb1a48a
This commit updates the EE repository reference after PR #631 was merged in windmill-ee-private.
Previous ee-repo-ref: 30d740e619fad219108ec4b4c6a9d67c1ab42d46
New ee-repo-ref: 6aabd7c5ce53b9153be05c3e7bc9a76eadb1a48a
Automated by sync-ee-ref workflow.
* fix(debounce): claim whole batch in one UPDATE (no deadlock); assert test setup
Both Codex (P1) and Claude (P2) flagged a deadlock: the claim used two writable
CTEs (claim_self then claim_rest), locking the self row before siblings, so two
survivors of the same batch pulled concurrently acquired row locks in opposite
order and PostgreSQL aborted one with deadlock_detected (a transient pull error
on exactly the two-survivors race this path handles).
Replace with a single `UPDATE ... WHERE debounce_batch = (...) AND consumed_at IS
NULL RETURNING id` that claims the whole batch: both transactions lock rows in
the same scan order, so one simply waits and re-evaluates under EvalPlanQual.
A `claimed_self` flag (EXISTS id = self in the claimed set) plus the `mine`
snapshot still distinguishes fresh-claim / consumed-by-other / own-re-pull.
Also assert add_survivor_to_batch_of actually inserts a row (rows_affected == 1)
so a mis-set-up test can't pass vacuously.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
Ruben Fiszel
|
aa098c70c0 |
perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes (#9786)
* perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: document delete_jobs auth contract and workspace-scope jobs_export purge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
12f92e3ab7 |
[ee] feat(backend): native script retry without one-step-flow wrapping (#9688)
* feat(backend): native script retry without one-step-flow wrapping Schedules and data pipelines that retry a single script previously wrapped it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a v2_job_status row, and UI projection complexity. This adds native retry on a plain JobKind::Script job. - RetrySettings: flatten Retry into a deduped retry_settings table, carried via the existing runnable_settings_handle (lazy, off the hot path). - push() materializes a bare-script-with-retry SingleStepFlow into a native Script job (gated on min-version + no handlers/retry_if). - add_completed_job re-pushes the next attempt on failure with backoff, tracking the attempt counter in v2_job_queue.extras and the chain via parent_job; schedule completion handlers fire only on the terminal attempt. - frontend: ScriptRetryChain shows the attempt chain on the run page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): native retry_if eval + per-occurrence schedule handlers Extends native script retry to the two cases that previously stayed on the one-step-flow path: - retry_if: evaluated natively on the failure path via a feature-gated windmill-jseval dep (quickjs) over the failure result + flow_input; push materializes such policies natively only when quickjs is available. - on_failure_times / on_recovery: apply_schedule_handlers now resolves each past scheduled occurrence's terminal status across its native-retry chain (root OR any parent_job=root child succeeded) and excludes the current occurrence, so the counting is per-occurrence rather than per-attempt. All scheduled-script retries now go native (schedule.rs gate removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): always materialize retry_if natively; unsupported without quickjs retry_if is evaluated by the worker (which always has quickjs), not the pusher, so gating materialization on the pusher's feature was wrong. The flow path was never a real fallback either — the flow runtime needs quickjs to evaluate retry_if too. retry_if now always goes native; on a worker without quickjs it is unsupported and fails closed (no retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): un-park asset-cascade (pipeline) retry Native retry resolves the blocker that parked pipeline retry: a retried subscriber is now a Script job (not a one-step flow / flow step), so it stays eligible for asset dispatch and can trigger its own downstream on recovery. - scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset edges (was dropped with a TODO warning). - asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job, so native-retry attempts dispatch on success while flow steps stay excluded. - tests: retry-bearing subscriber now dispatches as a native Script carrying the policy in runnable_settings_handle; native-retry attempt is eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): cap native retry interval, lazy result serialization, idempotent retry push Hardening from a self-review of the native retry path: - Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path (evaluate_retry); the exponential formula could otherwise schedule up to ~18h vs the flow path's 6h. - Serialize the failure result lazily (only when a retry_if policy needs it), so the common failure no longer pays the serialization on the failure path. - Push each retry with a deterministic id per (root, attempt). If a worker dies between enqueueing the retry and finalizing the current attempt, the reaper re-handles the attempt and lands here again — push rejects the duplicate id, so the retry is enqueued exactly once (no double-retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): defer schedule handlers idempotently on retry-push replay (review P1) Address local-review findings: - P1: retry_pending was derived from the retry push *result*, so on a worker crash + reaper replay the duplicate-id push returned Err → retry_pending flipped to false → apply_schedule_handlers fired for the non-terminal attempt (and the terminal attempt later fired them again). Pre-check whether the deterministic retry id already exists and report it as pending without re-pushing, so the handler-deferral invariant is crash-idempotent too. - P2: refresh the stale 'wrap the script in a one-step flow' comment in the asset-cascade retry push — it now materializes a native Script. - Add RetrySettings <-> Retry round-trip unit tests (clamping edges). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): native retry chain + per-occurrence status sqlx tests Close the two integration-test gaps flagged in local review: - chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts) and asserts crash-replay idempotency (the P1 fix: a replayed completion reports pending without double-enqueueing). - per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but- recovered occurrence counts as success, retries (parent_job set) are excluded from occurrence counting, and the current occurrence is excluded. - canceled_job_does_not_retry: cancellation wins over a pending retry. Runtime sqlx API (no .sqlx cache entry needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): exclude schedule handlers from the retry-attempt chain The retry chain listed all script children of the root by parent_job, but schedule completion handlers (on_failure/on_recovery/on_success) are also script children — when the occurrence has no retries, the handler's parent is the root itself, so a successful, never-retried job rendered a bogus 'Retries (1)' badge pointing at the handler. Filter children to re-runs of the same script (matching script_hash); real retries keep the root's hash, handlers run a different script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): surface schedule handlers on the run page Extend the run-page chain component with schedule completion handlers: - A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/ on_success runs that fired for that occurrence (found as children of the terminal attempt, identified by their synthetic created_by). - A handler's own run page now shows a 'Failure/Recovery/Success handler' label with a link back to the run it handled and its schedule. on_recovery and on_success share created_by, disambiguated by the recovery-only error_started_at arg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare An earlier `cargo sqlx prepare` on this branch ran before #8801's folder_default_permissioned_as test merged in, so it pruned the 3 query caches that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main. * fix(backend): only cascade assets from native retry attempts, not handlers (review P1) is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script child became asset-eligible — including schedule/error/recovery handlers (Script jobs with parent_job set and no flow_step_id). A handler that declares assets would then trigger a cascade the old parent_job IS NULL guard prevented. Gate parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as its chain parent (handlers run a different script). Runtime query, no sqlx cache. * fix(backend): cache the private-gated retry_setting asset-dispatch test query The same prepare-without-private that dropped the folder_default caches also pruned the cache for the retry_setting_dispatches_subscriber_as_native_script test query (asset_trigger_dispatch.rs:721). Regenerated with --features private. * fix(backend): exclude handler children from per-occurrence recovery (review) A scheduled occurrence's on_failure/on_success handler runs as a successful child (parent_job = occurrence), and the per-occurrence success EXISTS counted ANY successful child — so a failed occurrence whose error handler succeeded was marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in the merge) and on_failure_times counting. EE query now scopes the EXISTS to same-runnable children (only native retry attempts); regenerate sqlx cache + bump ee-repo-ref. native_retry_test gains a handler-child regression case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): scheduled-script retry is a native Script, not SingleStepFlow test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a native Script. Update both to assert kind='script' and that the retry policy is carried via runnable_settings_handle. * fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic) Address cubic CI review: - P1: the SingleStepFlow->native Script materialization dropped dedicated_worker, so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve it from the script row in push so the materialized Script keeps the dedicated tag. - P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the u32->i16 // retry count narrowing (scripts.rs) instead of wrapping. * fix(backend): use a retry-specific signal, not runnable equality (codex review) Address Codex CI review: - P1: is_native_retry_attempt treated any same-runnable parented Script child as a retry. WAC v2 inline children have that exact shape, so an inline child of an asset producer would cascade. Use a retry-specific signal instead: the job carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no flow_innermost_root_job. Apply the same flow_innermost guard to the EE per-occurrence EXISTS (WAC inline children must not count as a recovery). - P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate now resolves as 'retry pending' (re-check on the duplicate-id error) instead of flipping retry_pending to false and firing handlers early. - Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): explicit native_retry_attempt marker, drop heuristics Replace the per-site "is this a retry?" inference (parent_job + runnable match + flow_innermost / retry_settings) with one explicit marker: a sparse native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker also carries the attempt counter (previously in v2_job_queue.extras), so it's the single source of truth. - asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker. - EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost. - maybe_enqueue: reads/writes the marker (persistent) instead of queue extras. - Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap. - Eliminates handler / WAC-inline-child misclassification by construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex) The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but the periodic monitor.rs path deleted v2_job rows without it, orphaning markers. Add the same WHERE job_id = ANY(...) sweep there. * fix(backend): widen native_retry_attempt.attempt to integer (cubic) The smallint column was cast to/from u32 and could wrap a retry chain longer than i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32 attempt count, so no narrowing occurs on the maybe_enqueue read/write path. * feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT - Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi), computed from the native_retry_attempt marker. The run-page chain now filters retry attempts by is_retry instead of the script_hash heuristic, so WAC v2 inline children (same script, parent_job) no longer render as retries (codex). - Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep the plain catch-and-continue matching the other side-table deletes; the table is created by a startup migration so it always exists when cleanup runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but does not select is_retry, which errored with "no column found". Only the list endpoint populates the marker; #[sqlx(default)] lets every other query omit the column and default to None. * feat(backend): select is_retry in single-job GET too for consistency The list endpoint already exposes the marker; populate it on the single-job GET (both completed and queued variants) as well so a run loaded directly reflects its retry status. #[sqlx(default)] stays as a safety net for any other query. * feat(backend): reap orphaned native_retry_attempt markers via periodic sweep The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so direct job deletions (workspace/job delete, schedule clearing) would leave marker rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which must then be remembered for every future path), reap orphans in the periodic delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the job). The table is sparse so the anti-join drives off it and probes v2_job by PK — cheap. Retention still sweeps markers inline (keeps the table small so this stays cheap); a transient orphan is harmless (nothing reads is_retry for a gone job). * fix(frontend): include flow handlers in retry chain handler row (codex) Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...), whose handler job is a flow, not a script. The chain fetched children with jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts are still selected by is_retry and handlers by created_by, so both kinds surface. * fix(backend): carry concurrency/debouncing settings into native retries maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings ::default(), dropping the script/pipeline concurrency settings the failed job carried in its runnable_settings_handle. A retry of a concurrency-limited script then inserted no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and pass them in the payload, which push forwards to the materialized retry. Adds a regression test asserting the retry's handle resolves to the concurrency settings. * fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex) P1a (schedule.rs): the scheduled-retry materialization fetched the script's concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings ::default() into the SingleStepFlow payload, so the root attempt's handle held only the retry policy and the whole chain ran unbounded. Pass the fetched settings. Regression test asserts the root handle resolves to retry + concurrency. P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract — it is pub only for the integration test; the sole production caller is the worker completion path passing a DB-derived, already-authorized MiniCompletedJob. * docs(backend): attach native-retry auth contract to the function itself (codex) The doc block was merged with eval_retry_if's doc and bound to that function, leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if keeps its own doc; the native-retry + authorization contract now sits directly above maybe_enqueue_native_script_retry. * docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex) - Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since 1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the served spec couldn't see the field. Now current at 1.739.0. - schedule.rs: a retry_if gate is evaluated at failure time and fails closed without quickjs (no retry); it does not fall back to a flow path. - windmill-types jobs.rs: is_retry is selected by both the list and single-job GET endpoints (not list-only). * docs(backend): fix remaining stale retry_if/quickjs comments (codex) The retry_if block and the push materialization comments claimed push keeps retry_if on a flow path / the worker always has quickjs. The code always materializes native retry and the no-quickjs eval_retry_if path fails closed — correct the comments to that constraint. * docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex) - Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated and the job does not retry (no one-step-flow fallback). - jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence failure/recovery status across the retry chain, so the old 'restricted to schedules whose handlers don't need per-occurrence counting' claim is dropped. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
8912e21d15 |
perf(monitor): vacuum job_perms/job_result_stream right after each orphan sweep (#9753)
A customer's top-load query was the job_perms orphan sweep (cleanup_job_perms_orphaned: 6.9s mean, 41s max). The cost is discovery, not deletion (~2.1ms per row deleted): the NOT EXISTS anti-join seq-scans the whole job_perms heap to find a few orphans, and that scan tracks the heap's physical size. job_perms / job_result_stream_v2 get one row per job and are drained only by these per-cycle sweeps, so they churn hard — but the bulk vacuuming_tables() runs only ~hourly, so dead tuples bloat the heap between bulk vacuums. Reclaim right after each sweep instead: VACUUM (SKIP_LOCKED) the swept table when it deleted rows. Plain VACUUM (not FULL) takes only SHARE UPDATE EXCLUSIVE so concurrent job creates/reads proceed; the visibility map skips unchanged pages so repeated runs are cheap; SKIP_LOCKED means HA replicas don't pile up (one vacuums, the rest skip). Benchmarked ~7x: a bloated 268MB job_perms heap swept in 35ms vs 5ms vacuumed. Chosen over an autovacuum reloptions migration so the behavior is explicit and lives with the sweep it pairs with. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
e90b2be8fa |
perf(monitor): skip protected prefix in retention delete via cross-batch watermark (WIN-2088) (#9744)
The expired-job retention loop re-scanned the same oldest rows on every batch. When the oldest completed jobs are undeletable (children of a still-active root flow), the ORDER BY completed_at ASC scan walked that protected prefix on each of the up-to-20 batches, doing a v2_job PK lookup per row — quadratic in prefix size (measured ~9s/batch, ~180s/cleanup-cycle on a 1.5M-row prefix). Carry a completed_at watermark (max deleted) across batches and re-apply it as completed_at >= floor so each batch resumes past the already-processed prefix. Also skip the v2_job join entirely when no old root flow is active (the common case), since nothing is protected then. Measured: subsequent batches 9000ms -> 159ms; empty-set path 154 -> 36ms. The watermark only ever skips rows the current run already deleted, was protecting, or skip-locked — all deferred to the next run, identical to the unbounded scan's row set (verified: union of batched deletes == single delete, 0 diff). Mirrored in windmill-api-settings log_cleanup. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
31d9215e5a |
fix: bound orphan-cleanup drain rate with capped multi-batch loop (#9730)
Follow-up to #9727. The orphan cleanups (cleanup_job_perms_orphaned and cleanup_job_result_stream_orphaned_jobs) deleted at most one 100k batch per monitor iteration. Each statement stays short and lock-light, but a single batch per ~30s cycle caps the drain rate at ~100k/30s, so a large one-time backlog (tens of millions of rows) takes ~hours to clear. Loop the batched delete up to ORPHAN_CLEANUP_MAX_BATCHES (10) times per cycle, stopping early once a batch deletes fewer than ORPHAN_CLEANUP_BATCH_SIZE rows. Each DELETE remains bounded (≤100k, short locks, no long single statement), while per-cycle throughput rises to ~1M rows so backlogs drain ~10x faster. The per-cycle cap keeps monitor_db responsive. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
75bafabeee |
perf(monitor): hash active-root exclusion in retention delete (WIN-2088) (#9732)
* perf(monitor): hash active-root exclusion in retention delete The expired-job retention delete (delete_expired_jobs_batch) excluded jobs belonging to still-active root flows with `COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)`. That ScalarArrayOp is evaluated per candidate row as a linear scan of $3, so cost grows with the number of active root jobs. Express the exclusion as `NOT IN (SELECT u FROM unnest($3) u WHERE u IS NOT NULL)` instead. The subquery form lets Postgres build a one-time hashed SubPlan and apply it as a filter on the ordered index scan, giving O(1) membership per candidate while preserving the `ORDER BY completed_at ASC LIMIT` early termination. The `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids). Measured on a 2M-row synthetic v2_job_completed (batch LIMIT 20000, 5-run min): active roots | != ALL (before) | NOT IN hashed (after) -------------|-----------------|---------------------- 100 | 108 ms | 104 ms 1000 | 168 ms | 105 ms 10000 | 719 ms | 131 ms Both forms return identical row sets (verified via EXCEPT, 0 diff). Neutral at small active-root counts, ~5.5x faster when many flows are active. Relates to WIN-2088 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(monitor): apply hashed active-root exclusion to log_cleanup mirror windmill-api-settings/log_cleanup.rs::delete_expired_jobs_batch carries a byte-identical copy of the retention delete and shared its prepared-query cache. Updating only monitor.rs removed that shared cache entry and broke the SQLX_OFFLINE build of the mirror. Apply the same NOT IN (hashed SubPlan) rewrite so both copies converge on one cached query and the mirror gets the same speedup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
6d94865109 |
fix: optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries (#9727)
The job_perms/job_result_stream_v2 orphan cleanups in monitor_db used `NOT IN` anti-joins, `RETURNING job_id` + `fetch_all` (loading every deleted UUID into memory) and no batch limit. On high-throughput instances these tables can accumulate tens of millions of orphaned rows, so a single execution ran for ~298s; because monitor_db awaits each iteration, the cleanup ran effectively continuously, saturating DB I/O and starving audit partition creation. Rewrite both deletes as bounded `NOT EXISTS` anti-joins selecting `ctid` with a LIMIT 100000, executed via `.execute()` (using rows_affected instead of fetch_all). Each run is now fast and bounded, while the 30s monitor cadence is preserved so the tables keep draining promptly. Fixes WIN-2088 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
8dea38383f |
fix: prevent silent audit-partition outage via monitor watchdog + alert (#9729)
The monitor loop runs ~25 periodic tasks under a single join!, so any one stuck on a non-DB await (statement_timeout only bounds DB statements) freezes the whole loop indefinitely — silently halting audit-partition creation. Once the missing partition's date is reached, audit inserts fail; because login writes its audit row in the same transaction, that poisons the login tx and locks every user out. - Wrap monitor_db in a 600s timeout (> statement_timeout) so a stuck task can no longer freeze the loop; report a critical error and continue on timeout. - After creating partitions, verify the lookahead window is actually covered and raise a critical alert naming any missing partitions, turning a silent latent outage into an early page. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
8a0b0abead |
fix: ignore NotFound errors when deleting log files from object store (#9707)
* fix: ignore NotFound errors when deleting log files from object store Periodic and manual log cleanup delete log files from instance object storage. S3's DeleteObjects silently ignores missing keys, but GCS returns a 404 for each individual delete, which the object_store crate's default delete_stream surfaces as Error::NotFound. This produced noisy error/warning logs on every cleanup cycle even though the cleanup succeeded (DB records are removed regardless). Treat a NotFound delete as a successful no-op in both delete handlers: - monitor.rs: skip logging NotFound errors - log_cleanup.rs: count NotFound as deleted instead of an error Fixes WIN-2081 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: report 404 (already-absent) count in object store log cleanup Track delete calls that returned 404 (object already absent) separately from real deletes so operators can see how many of the attempted deletes were no-ops, instead of those numbers silently folding into s3_deleted. - monitor.rs: emit a final info summary per cleanup cycle: "N deleted, M already absent (404), K failed" (only when work occurred) - log_cleanup.rs: add s3_not_found to LogCleanupProgress (serde default for backward-compatible deserialization of in-flight rows), thread it through s3_bulk_delete and all call sites, and log a final summary on release - openapi.yaml + generated client + ObjectStoreConfigSettings.svelte: surface the 404 count in the manual cleanup status UI Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: import ObjectStoreError directly from object_store_reexports The object_store_reexports module already re-exports object_store::Error under the name ObjectStoreError, so `Error as ObjectStoreError` failed to resolve (no `Error` in that module). This compiles only behind the parquet feature, which the local dev `cargo watch` doesn't enable, so it was caught by CI's full-feature check rather than locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
7155a0bb96 |
feat: Data Pipelines alpha (#9193)
* feat: add workspace asset graph view Workspace-wide canvas of assets and their producer/consumer scripts, reachable from the assets page. Left-to-right layered layout via d3-dag sugiyama, rendered with @xyflow/svelte (same stack as the flow editor). GET /w/:ws/assets/graph returns deduped nodes + edges. Follow-ups: filters (kind/folder/search), node detail drawer, inline script edit from a clicked node. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * all * all * all * update * all * all * all * feat(pipeline): output-kind picker and per-(lang, output) templates Add a third stage to PipelineInsertMenu that asks what kind of asset the new script will produce (datatable / ducklake / s3 parquet / s3 object / none). The picked kind drives a real wmill SDK skeleton — typed datatable inserts, ducklake CREATE+INSERT, s3 parquet COPY, etc. — with the upstream asset auto-wired as the input source when added from an asset node. Reorder languages to bun → duckdb → python → sql so data-shaped languages surface first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * all * chore(main): release 1.693.4 (#8994) * chore(main): release 1.693.4 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * feat: ansible delegate_to_git_repo install_requirements, dynamic fields, --limit (#8997) * feat: ansible delegate_to_git_repo install_requirements, dynamic fields, --limit Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: include .yaml variants in collections/roles requirements lookup Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): only preserve case for raw-app runnableIds, not app/flow summaries (#9000) * fix(cli): only preserve case for raw-app runnableIds, not app/flow summaries PR #8940 stopped lowercasing in sanitizeForFilesystem to fix #8939, where a raw-app runnableId like CamelCaseTSRunnable produced a CamelCase YAML metadata file but a lowercased code file, making them desync and register as duplicate runnables on push. That fix overshot. sanitizeForFilesystem is also reached by newPathAssigner, which serves normal apps and flows where the input is the script's human summary ("Get Users Data") rather than an identifier. There the on-disk filename is the only artifact — there's no companion YAML to keep in sync — so lowercasing was the right behavior. Removing it changed both the on-disk filename and the !inline reference in app.yaml / flow.yaml from get_users_data.inline_script.ts to Get_Users_Data.inline_script.ts on the next pull, surfacing as unwanted case churn for users updating to 1.693.x. Add a preserveCase option to sanitizeForFilesystem (default false → lowercase). newRawAppPathAssigner opts in; newPathAssigner stays on the default. Update unit tests accordingly and add an end-to-end raw-app round-trip in raw_app_sync.test.ts that pushes a CamelCase backend runnable, pulls it back, and asserts both YAML and code file preserve case with no lowercase orphan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(cli): use readdir for exact-case orphan check on Windows The CamelCase round-trip test used fileExists("camelcasetsrunnable.ts") to assert no lowercase orphan was produced, which false-positives on Windows since the filesystem is case-insensitive and resolves the lookup to the existing CamelCaseTSRunnable.ts. Switch to readdir + toContain so the exact on-disk casing is compared identically on Linux and Windows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): wmill-lock.yaml auto-fill + --rehash-only + path-prefix dedup (#8978) * fix(cli): canonical lockfile hashes + lock upgrade migration to v3 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): use __app_hash subpath in rehash missing-entry check Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): run sync pull lockfile auto-fill regardless of changes Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: regenerate system prompts for new lock and rehash-only commands Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address review feedback on lock upgrade Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): drop v3 marker; always run fallback; fail-fast on unknown lockfile version Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): drop yaml-round-trip legacy hash variant; recover via --rehash-only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): include legacy hash in script push staleness warning check Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * revert(cli): drop canonical hash formula; keep raw-bytes hashing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * perf(cli): reuse change-tracker map for sync pull lockfile auto-fill Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address review feedback on rehash-only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(cli): pin lockfile hash + yaml format and cover regression cases Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(cli): byte-stable snapshot tests for flow.yaml format Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(cli): add app and script-metadata yaml snapshot fixtures Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address claude review on rehash-only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(cli): factorize script-path to remote-path derivation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address claude + cubic review (dry-run mutation, rehash short-circuit) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(cli): make rehash a subcommand and factorize fs walks Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): normalize line endings in yaml snapshot tests for windows ci Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address review feedback on rehash + auto-fill - Flat-layout scripts now clearGlobalLock before rehash write so legacy ./-prefixed duplicates get cleaned up (matches flow/app behavior). - Add MalformedLockfileError; sync pull auto-fill re-throws it alongside UnknownLockVersionError instead of silently warning + continuing. - Document the legacy step-removal false-negative in isFlowDirectlyStale / isAppDirectlyStale and the categorizeLocalFiles ignore-filter invariant. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * fix: use otel.status_message for OTLP Status.message on failed jobs (#8995) tracing-opentelemetry only recognizes otel.status_code and otel.status_message as fields that map to the OTLP Status proto. The previously-used otel.status_description fell through to the generic attribute recorder, leaving Status.message unset and preventing OTLP consumers from filtering spans on error status. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: route email trigger path through standard info channel (#8996) * docs(skill): document email triggers and S3 attachments Add an "Email triggers" section to the triggers skill covering the local-part config, the parsed_email/raw_email/email_extra_args payload, the URL-style extras convention, where to find trigger_path (only with a preprocessor, at event.trigger_path), and — most importantly — that binary attachments are uploaded to the workspace S3 bucket and surface as `{ s3: "windmill_emails/<job_id>/attachments/<filename>" }`. Scripts must use wmill.loadS3File / wmill.load_s3_file to read them. Also pulls EmailTrigger into the schema mappings so a real `email_trigger.schema.yaml` is generated, and adds Email/Azure to the trigger kinds list in the CLI agent guidance. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref for email trigger path fix Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 26184ab7a4aadfc529dcedf038aa08d36c7ad381 This commit updates the EE repository reference after PR #553 was merged in windmill-ee-private. Previous ee-repo-ref: 318a46897a605dc9be3817901f35ba5a99a0a525 New ee-repo-ref: 26184ab7a4aadfc529dcedf038aa08d36c7ad381 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> * update git sync version to 1.693.5 * fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe (#8999) * fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(pg): wrap encoder errors with arg context, add fallback test Followups on #8999 review: - Wrap rust-postgres "error serializing parameter N" failures with the arg name, JSON value kind, and asserted Postgres type plus a hint about an explicit cast — so users see actionable context instead of an opaque WrongType. - Drift-prevention meta-test: assert otyp_to_pg_type and convert_val agree on the Type for every recognised arg_t when the JSON value matches its natural Rust kind. Catches future drift if either side changes. - Integration test for the prepare + query_raw fallback path: confirms unrecognised arg_t (custom enum) is routed through prepare and the server-resolved type appears in the failure surface — flips into a test failure if a regression accidentally routes unrecognised types through query_typed_raw. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): add otyp_inferred flag + regex-based placeholder renumbering Two follow-ups from the review of #8999: 1. **Issue #1 (Number/Bool + explicit text decl in WHERE)** Add `Arg::otyp_inferred: bool` to the parser. The PG SQL parser sets it `true` only at the "no info → fall back to text" site (bare `$N`, no inline cast, no `-- $N (TYPE)` decl). All other arg sources keep it `false`. In `convert_val` this flag distinguishes: - explicit text-like target (`-- $1 (text)` or `$1::text`) — coerce `Bool`/`Number` → `Box<String>` so `WHERE text_col = $1` works (`text = text` operator). Pre-#8988 behaviour, restored. - parser-default text (bare `$N`) — bind the value's natural Rust type so the regression case (`Value::Bool` against a real `bool` column via `CAST AS bool`) keeps working. `Arg` is in `windmill-parser`; the new field has `#[serde(default)]` so persisted signatures stay backward-compatible. 2. **Issue #4 ($5/$50 substring rewrite collision)** Replace the per-index `String::replace` chain (which turned `$50` into `$10` when oidx=5 was processed first) with a single regex pass. `\d+` is greedy, so `$5` and `$50` match as distinct units; indices outside the mapping are left intact. 3. Tests: - parser: `test_parse_pgsql_otyp_inferred_flag` covers bare/inline- cast/decl/mixed shapes. - executor unit: `convert_val_bool_against_every_arg_t` and `convert_val_*_number_*` split each text-like target into explicit vs inferred expectations. - executor unit: `renumber_sparse_placeholders_no_collision`. - integration: `test_postgresql_arg_type_combinations` adds 4 cases covering decl(text)+Number/Bool in WHERE, bare $1+Bool, and sparse positional args ($5/$50). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg+sdk): enum support, extended String arms, position-aware $N rewrite, SDK quality Backend: 1. **`AnyTextValue` ToSql/FromSql wrapper**: vanilla `tokio_postgres`'s `ToSql for String` / `FromSql for String` reject `Kind::Enum` and `Kind::Domain` even though the wire format is plain UTF-8. The wrapper accepts those kinds in both directions. End result: explicit `$1::my_enum` / `CAST($1 AS my_enum)` casts now round-trip without the ugly `CAST($1::text AS my_enum)` workaround, AND `SELECT enum_col` results come back as JSON strings instead of erroring at the FromSql layer. 2. **#10 — Value::String → numeric/real/double/oid/bool**. Without these arms, a string-encoded value (`"3.14"`, `"true"`) for a non-text / non-temporal arg_t fell through to `Box<String> + TEXT`, which then failed at the server (no implicit cast text→numeric in expression context). Now strings are parsed into the matching native type with clear error messages on parse failure. 3. **Position-aware `$N` rewrite**: replaces the regex-based renumbering (which fixed the `$5/$50` substring collision but still walked through string literals and comments, mangling `'price: $5'` etc.) with a walk over `parse_pg_statement_arg_positions` — the same string/comment/dollar-quote-aware tokenizer used for index discovery. Adds `parse_pg_statement_arg_positions` to the parser's public API. SDK: 4. **BigInt support**: `JSON.stringify(BigInt)` throws. The SDK now stringifies bigints before serialisation; the executor accepts numeric strings into BIGINT arg slots via the existing `Value::String → INT8` parsing arm. SDK-side `inferSqlType` is split so `BigInt` always resolves to `BIGINT` (was reaching `Number.isInteger(BigInt)` which returns false → wrong default). 5. **Homogeneous array auto-tag**: `${[1,2,3]}` against an `int[]` column now emits `$1::BIGINT[]` instead of `$1::JSON`. Detection covers primitive types only (number / bigint / string / boolean); mixed or nested arrays still fall back to JSON. Mixed int/float widens to `DOUBLE PRECISION[]`. 6. **`.query()` positional bug**: previously the `.query()` method abused the template-tag builder, which appended `$N::TYPE` after the user's literal SQL string instead of binding by position (`SELECT $1, $2` became `SELECT $1, $2$1::BIGINT`). Now `.query()` builds the executor-shaped content directly: a `-- $N argN (TYPE)` declaration block followed by the user's SQL verbatim. Tests: - Parser: `test_parse_pg_statement_arg_positions_skips_strings_and_comments` asserts string literals, comments, and dollar-quoted blocks don't produce positions (so renumbering doesn't mangle them). - Executor unit: `renumber_sparse_placeholders_no_collision_no_string_mangling` uses the new position-aware path and includes string-literal + comment + `$$…$$` cases. Existing convert_val tests grow to cover new String→numeric/real/double/oid/bool arms. - Integration: `test_postgresql_arg_type_combinations` adds 13 cases (enum round-trip both directions, string→numeric/real/double/bool/oid, string-literal `$N` non-mangling). The prepare-fallback test now asserts SUCCESS (not failure) for enum encoding via AnyTextValue. - SDK: new `typescript-client/tests/sqlUtils.test.ts` (42 tests) exhaustively covering inferSqlType primitives + arrays, parseTypeAnnotation, datatable() template tag (with all the new shapes — BigInt, homogeneous arrays, RawSql, schema preamble), datatable().query() positional, and ducklake() shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): replace DISCARD ALL with curated reset (preserves typeinfo cache) Found while exhaustively probing custom-type DX: every cached-connection reuse was running `DISCARD ALL`, whose included `DEALLOCATE ALL` deallocates *all* prepared statements server-side — including the typeinfo statements that tokio_postgres caches per-Client to resolve custom enum / domain Oids. tokio_postgres still held `Statement` objects whose names the server had forgotten, so the next custom-type query failed with intermittent "prepared statement \"sN\" does not exist" errors. The failure was easy to reproduce: any sequence that forced typeinfo lookup for two different custom-type kinds on the same cached connection (e.g. enum followed by domain) would hit it. Replace `DISCARD ALL` with a curated reset that explicitly targets the state we actually care about, *without* touching prepared statements: RESET ALL — GUC parameters (search_path, application _name, statement_timeout, …) RESET SESSION AUTHORIZATION — undoes both `SET SESSION AUTHORIZATION` and `SET ROLE` (RESET ALL does NOT — these aren't GUC parameters, so without this an elevated role from a previous job would silently leak) UNLISTEN * — drops LISTEN registrations CLOSE ALL — closes open cursors Trade-off: temp tables, advisory locks (session-scoped), and user-created PREPARE statements may persist across cached-connection reuse — rare in datatable / PG-script workloads. tokio_postgres's typeinfo cache survives intact, so custom enum / domain queries are fast on subsequent reuse. Tests: - `test_postgresql_custom_types_on_cached_connection` — runs 10× alternating enum + domain queries on a cached connection. Pre-fix this failed with `prepared statement "sN" does not exist` after the first reuse; post-fix passes. - `test_postgresql_set_role_does_not_leak_across_cached_connection` — switches `SET ROLE` and `SET SESSION AUTHORIZATION` to a non-postgres role, then runs a follow-up job and asserts current_user/session_user are restored. Specifically catches the case where someone might switch back to `RESET ALL` alone (which doesn't cover SET ROLE / SESSION AUTHORIZATION) and silently introduce a permission-leak vector. - All existing session-isolation tests (`test_postgresql_cached_connection_resets_session`, `test_postgresql_single_worker_session_isolation`, `test_postgresql_100_jobs_cached`) continue to pass. Found via end-to-end probing of datatable / PG-script DX, not previously covered: the existing isolation tests only did `SET ROLE postgres`, the connecting user, so the leak was invisible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): address PR #8999 review (cubic + claude) cubic (P1, real bug): - `convert_vec_val` for `timetz` array asserted `Type::TIMETZ_ARRAY`, but chrono `NaiveTime` only encodes for TIME (same caveat as the scalar arm). Switch to `Type::TIME_ARRAY`; rely on PG's implicit `time→timetz` assignment cast at the column site. Add an explicit unit test. claude (#1, silent failure → explicit error): - `Bool` + explicit `(char)` / `(character)` decl previously silently bound BOOL, hoping the server would cast at the use site — but PG has no implicit `bool→char` and the resulting error ("operator does not exist: bool = char") was opaque. Now error at bind time with an actionable hint to use `bool` decl or pass the value as a "t"/"f" string. claude (#2, asymmetry doc): - Object/Array still coerce to text on `matches!(typ, Typ::Str(_))` (covers both explicit AND inferred-default text), unlike Bool/Number which key on `explicit_text_target`. The asymmetry is intentional (no implicit `jsonb → text` cast in expression context vs PG having implicit `bool/int → text` casts) — added a body comment so future maintainers don't try to "align" them. claude (#3, perf): - `parse_pg_statement_arg_indices` and `parse_pg_statement_arg_positions` walked the SQL tokenizer twice. Fold into a single pass that derives the index set from the position list. claude (#4, fmt drift): - `cargo fmt` over the parser crates I touched with perl scripts in the earlier commit (windmill-parser-{sql,bash,ts,go,php,java,csharp,nu,py, rust,graphql,yaml,r}). Net cosmetic. claude (#5, parseTypeAnnotation): - One-line caveat in the SDK's `parseTypeAnnotation` that the returned string is presence-only (e.g. `${x}::DOUBLE PRECISION` returns `"DOUBLE"`, `CAST(${x} AS int)` returns `"int)"` — neither matches a real PG type, but the only consumer just checks `!== undefined`). While here — discovered + fixed independently while exhaustively probing DX: - **Replace `DISCARD ALL` with curated reset** (`RESET ALL; RESET SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;`). DISCARD's `DEALLOCATE ALL` killed tokio_postgres' typeinfo cache, producing intermittent `prepared statement "sN" does not exist` errors on custom-type queries after cached-conn reuse. New regression tests: `test_postgresql_custom_types_on_cached_connection` and `test_postgresql_set_role_does_not_leak_across_cached_connection` (the latter catches the case where someone might switch back to `RESET ALL` alone and silently introduce a permission-leak vector — RESET ALL doesn't cover SET ROLE / SET SESSION AUTHORIZATION). - **ISO-8601 timestamp results** (`pg_cell_to_json_value`). Pre-fix `TIMESTAMP` was rendered with a space separator ("2024-01-15 10:30:00") and `TIMESTAMPTZ` with " UTC" suffix ("2024-01-15 10:30:00 UTC") — neither parseable by `date-fns parseISO`, JavaScript `new Date()` is lenient enough to handle them but several frontend `App*Input.svelte` components use parseISO and fail silently. Switched to ISO-8601 with `T` separator and `+00:00` offset; arg-parsing path still accepts the legacy " UTC" suffix for back-compat. Test coverage: - 17/17 unit (`pg_executor::tests`) - 9/9 integration (`backend/tests/worker.rs`, `test_postgresql_*`) - 27/27 parser (`windmill-parser-sql`) - 42/42 SDK (`typescript-client/tests/sqlUtils.test.ts`) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): bounded one-shot warning on numeric precision loss + ISO-8601 + NaN handling Found while probing PG-script DX with millions of numeric cells: 1. **Numeric precision-loss warning**: `numeric` results are still serialised as JSON Number (back-compat — switching to JSON String would silently break user code doing arithmetic on results), but we now detect `Decimal -> f64 -> Decimal` round-trip failure and emit a single job-log warning recommending a `::text` cast in the SQL. Bounded by `NUMERIC_PRECISION_CHECK_BUDGET = 256` cells per query (one atomic load + one fetch_sub on the hot path; first lossy value short-circuits to a single load thereafter). Worst-case overhead on a 1M-cell numeric-heavy query: ~25µs of checks + 5ns × N atomic loads (vs. ~100ms unbounded). 2. **ISO-8601 timestamps**: `pg_cell_to_json_value` previously returned `"2024-01-15 10:30:00"` (TIMESTAMP) and `"2024-01-15 10:30:00 UTC"` (TIMESTAMPTZ) — neither parseable by date-fns `parseISO`, which is what the apps `App*Input.svelte` components use, so timestamp values silently failed to round-trip into date pickers. Switch to ISO-8601 (`T` separator + `+00:00` offset) on the result side; arg-parser continues to accept the legacy `" UTC"`-suffixed format for back-compat. 3. **Float NaN / Infinity results**: `Number::from_f64` returns None for NaN / ±Inf, which `pg_cell_to_json_value` was raising as "invalid json-float" — failing the *entire* query if any cell held one of these special values. Now serialise them as JSON strings ("NaN", "Infinity", "-Infinity") and let the rest of the row come through. Arg-side: `s.parse::<f64>()` already accepts the same strings. Tests: - `decimal_fits_f64_losslessly_predicate` — covers fits / doesn't-fit cases for the precision-loss predicate. - `precision_check_budget_caps_per_query_overhead` — locks in the budget cap and the loss-flag short-circuit. - All 9 PG integration tests + 17 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): add pg_advisory_unlock_all to reset; warn on missing args; honor decl defaults While probing PG-script DX further found three more frictions: 1. **Advisory lock leak** (cubic P2): switching from `DISCARD ALL` to `RESET ALL; RESET SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;` meant session-scoped advisory locks (`pg_advisory_lock`) leaked across cached-connection reuse. Add `SELECT pg_advisory_unlock_all()` to the chain — `DISCARD ALL` covered this implicitly via `DISCARD PLANS / DEALLOCATE / pg_advisory_unlock_all` and we lost it in the switch. 2. **Missing-arg silent NULL**: an arg declared in the SQL (e.g. `-- $1 amount (numeric)`) but not provided in the args object was bound as NULL with no error / warning. Misspelling the key in the args object silently produced a row of NULLs — a notorious DX debugging trap. Now: collect the names of declared-but-missing args during dispatch and emit a single one-shot warning to the job logs at end-of-query naming each one. Bound NULL is preserved for back-compat. 3. **Declaration defaults ignored**: `-- $1 a (int) = 5` carries `arg.default = Some(Number(5))`, but the dispatch fell straight to NULL when the arg was missing. Now: respect the default — user-supplied value > declaration default > NULL. Also fixes the warning logic above (only warn for args that *don't* have a default). Tests: existing 19 unit + 9 integration pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): multi-word PG types with [] suffix lost the array-ness; array arms accept stringified values Two more frictions found while probing SDK end-to-end against a real datatable resource: 1. **Multi-word array types lose the [] suffix in the parser**. `transform_types_with_spaces` recognises aliases for "double precision", "character varying", "timestamp with time zone", etc. but its return type was `&'a str` — only the bare alias, never with a trailing `[]`. The `RE_CODE_PGSQL` regex's `\w+` captures stop at the first space, so the regex's own `(?:\[\])?` array-suffix branch sees only `"double"` (not `"double precision[]"`); the `[]` was silently lost. Result: `$1::double precision[]` (which the SDK now emits for homogeneous float arrays via the new auto-tag) routed through `Value::Array → Type::JSONB` and the server failed with "cannot cast type jsonb to double precision[]". Fix: switch `transform_types_with_spaces` to return `Cow<'a, str>` and re-check the trailing bytes after a multi-word match. If they start with `[]`, return `format!("{alias}[]")` — Owned. Single-word types and the no-match path keep returning Borrowed slices, so no allocation in the hot path. 2. **Array arms in `convert_vec_val` rejected stringified values for numeric / int* / bool / oid / real / double**. The scalar `convert_val` already parses strings into the matching native type for these arg_ts, but the array variant only accepted JSON-native counterparts. Sending `["1.5", "2.5", "3.5"]` against `$1::numeric[]` (e.g. via `unnest` for bulk loading, or `JSON.stringify(BigInt[])` round-trip) failed with "Mixed types in array". Now the array arms mirror the scalar ones — `as_<native>().or_else(|| as_str().and_then(parse))` — so both shapes round-trip cleanly. Tests: 19 unit + 9 integration pass; existing parser tests cover the multi-word array forms (the regex-cap behaviour didn't break for single-word types, and Cow plumbing is transparent to all callers). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(parsers): add otyp_inferred field to Arg literals in tests + 3 missed src files CI failures: the perl-driven sweep that added `otyp_inferred: false` to every `Arg { ... }` literal when I introduced the field in the parser schema covered `src/lib.rs` files but missed: - parsers/windmill-parser-bash/src/lib.rs (mass-edited but a later format pass un-applied a few sites) - parsers/windmill-parser-go/src/lib.rs (same) - parsers/windmill-parser-graphql/src/lib.rs (same) - parsers/windmill-parser-nu/tests/tests.rs (test file — not swept the first time) - parsers/windmill-parser-ts/tests/tests.rs (test file — same) Also tightened the regex to handle `oidx: None` without the trailing comma (some test files had the field as the last initialiser line). `cargo build --features <CI feature combo> --workspace --all-targets` is clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(sdk): Date → TIMESTAMPTZ; NaN / ±Infinity → string Two more frictions found while running the actual SDK end-to-end against a live datatable resource: 1. **JS `Date`** fell into the typeof "object" branch and was tagged `::JSON`. It worked accidentally for `${date}::timestamptz` via PG's `json → text → timestamptz` implicit cast chain, but `${date}` against a `timestamptz` column without a user-supplied cast bound the value as a JSON string and the comparison `timestamptz = json` failed. Now: `inferSqlType` recognises `Date` and tags `::TIMESTAMPTZ`; `serializeArgValue` emits `Date.toISOString()` so the executor's `Value::String → TIMESTAMPTZ` arm parses it cleanly. 2. **JS `NaN` / `±Infinity`** silently became NULL. `JSON.stringify(NaN)` returns `"null"` per the JS spec, so the value reached the executor as JSON null — the SDK's `::DOUBLE PRECISION` tag then bound a NULL double. Fix: detect non-finite numbers in `serializeArgValue` and stringify them as `"NaN" / "Infinity" / "-Infinity"`. The executor's `Value::String → FLOAT8` arm (`f64::from_str`) accepts these literals directly, and the result-side already renders the values as JSON strings (matching round-trip). SDK unit tests grow from 42 → 44 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(pg): integration coverage for multi-word arrays + stringified array elements Locks in the two array fixes from the previous commit (`fix(pg): multi-word PG types with [] suffix lost the array-ness`) with end-to-end cases in `test_postgresql_arg_type_combinations`: - `double precision[]`, `character varying[]`, `timestamp without time zone[]` — verifies the parser keeps the `[]` suffix after multi-word alias resolution. - `numeric[]` / `int[]` / `bool[]` from stringified primitives — verifies the array arms of `convert_vec_val` apply the same string-coercion the scalar arms do. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: fix indentation drift on otyp_inferred lines cargo fmt cleanup of leftover indentation where the perl-driven sweep that introduced the otyp_inferred field landed at the wrong column. No behaviour change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * feat: support assigning a worker tag to app inline scripts (#9002) * feat: support assigning a worker tag to app/raw-app inline scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: omit empty tag field from inline script raw_code payload Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * style: shrink tag popover width --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * feat(pipeline): 2-col picker, draft path edit, save-all + leave guard Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * all * all * update * fix(cli): forward HEADERS env var on every backend fetch call (#9075) Several `fetch()` callers in the CLI bypassed `OpenAPI.HEADERS` and skipped the `HEADERS` env var, causing requests to fail behind auth gateways like Cloudflare Access (same shape as #6421): - `pushScript()` `/scripts/create` and `/scripts/create_snapshot` — regressed in #8936 when the call switched from `wmill.createScript()` (SDK) to a raw `fetch` for the `skip_if_noop` query param. - Script preview `/jobs/run/preview_bundle`. - App dev `/jobs_u/getupdate_sse` SSE stream. - `wmill docs` `/api/inkeep`. All four now spread `getHeaders()` and call `detectAuthGatewayChallenge()` so a Cloudflare/SSO challenge surfaces a clear error instead of an opaque JSON parse failure. Adds `test/headers_env_var.test.ts`: spins up an auth-gateway proxy that 403s requests missing `CF-Access-Client-Id` / `CF-Access-Client-Secret` and otherwise reverse-proxies to the test backend, then runs `wmill sync push` of a fresh script through the proxy. Negative case (no `HEADERS` env) verifies the proxy actually gates; positive case asserts every request including `/scripts/create` reaches the backend with the headers attached. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): add --parallel flag to generate-metadata (#9074) * feat(cli): add --parallel flag to generate-metadata * fix(cli): validate --parallel input and harden flush ordering * perf(flows): skip flow_env DB+transform work when no resolution is needed (#9078) * fix(cli-tests): stabilize flow lock-gen race + Windows path (#9080) * fix(cli-tests): stabilize flow lock-gen race + Windows path Three CLI test failures on the latest main, all flaky on CI: 1. `Mixed Case Paths: pull and push flow with capitalized folder` and `Integration: Mixed scripts and flows with nonDottedPaths are idempotent`: flow create/update queues an async FlowDependencies job that fills inline-script lockfiles and rewrites flow.value. The tests pulled/pushed before the worker finished, so dry-run idempotency saw phantom `*.inline_script.lock` adds and `flow.yaml` edits. Added a `waitForFlowDependencyJob` helper that polls `/flows/get` for the latest `dependency_job` and `/jobs_u/completed/get` until it lands, and called it after each API/CLI flow write in both tests. 2. `HEADERS env var is forwarded on every CLI fetch` (Windows-only, added in #9075): the new test built the CLI entrypoint via `new URL("..", import.meta.url).pathname`, which yields `/C:/...` on Windows and `Bun.spawn` rejected before reaching the proxy, leaving `rejectedRequests.length` at 0. Switched to `fileURLToPath` + `node:path.join` to match `cargo_backend.ts`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli-tests): use /flows/deployment_status to actually wait for dep job CI reviewers (Claude, Codex) flagged the prior `waitForFlowDependencyJob` as a no-op: it read `flow.dependency_job` from `/api/w/{ws}/flows/get`, but `Flow` / `FlowWithStarred` (backend/windmill-types/src/flows.rs:20-60) do not include that field. The helper exited on the first iteration without polling. Switch to `/api/w/{ws}/flows/deployment_status/p/{path}`, which returns `{ lock_error_logs, job_id }`. `job_id` is the FlowDependencies UUID written into `deployment_metadata` in the same tx as the dep-job push (backend/windmill-api-flows/src/flows.rs:660-672 and :1275-1292), so by the time the create/update API call returns, the response carries the latest dep-job UUID. Then poll `/jobs_u/completed/get/{job_id}` as before. Local runtime for `mixed_case_paths.test.ts` jumps from ~9s to ~32s, confirming the helper now actually waits instead of returning immediately. The 404 short-circuit in `sync_pull_push.test.ts` still works — `get_deployment_status` returns 404 when the flow is absent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(flows): cache resolved flow_env per flow execution (#9079) * perf(flows): cache resolved flow_env per flow execution * perf(flows): tighten flow_env cache cap to 1024 and clarify memory note * perf(flows): don't cache transient flow_env resolution failures * chore(main): release 1.698.0 (#9076) * chore(main): release 1.698.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix: reject root-rooted paths in ansible playbook validator on windows (#9081) * fix(native-triggers): serialize Google channel renewal across replicas (#9060) * fix(native-triggers): serialize Google channel renewal across replicas `sync_all_triggers` runs every 5 minutes on every windmill-app replica with no leader election. Multiple replicas were each rotating the webhook token, creating a new Google watch channel, and racing the trigger UPDATE — leaving the loser's new token (in `token`) and channel (in Google) orphaned. Cloud was accumulating ~5 leaked tokens/week without the silent best-effort `delete_token_by_hash` ever logging a warning. Wrap each per-trigger renewal in a transaction and acquire the row with `SELECT … FOR UPDATE SKIP LOCKED`. Contending replicas skip the row instead of duplicating the work. The lock spans `rotate_webhook_token` → Google API call → `update_native_trigger_service_config` and is only released on commit. Re-checks `should_renew_channel` after acquiring the lock so a replica that committed seconds earlier doesn't trigger a duplicate renewal. The pattern matches existing batch-cleanup paths in `monitor.rs` (job-retention sweep) and other `FOR UPDATE SKIP LOCKED` call sites. Also logs at `debug!` when `delete_token_by_hash` finds no matching row, so future investigations can distinguish "deleted" from "not found" without changing the `Ok(false)` contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fixup! fix(native-triggers): serialize Google channel renewal across replicas * fixup! fix(native-triggers): serialize Google channel renewal across replicas fixup! fix(native-triggers): serialize Google channel renewal across replicas Address claude review: - #5: per-skip log info -> debug (expected outcome under SKIP LOCKED) - #2: warn moved out of delete_token_by_hash to the call site that knows the expected state (try_renew_channel_locked); other callers are race-prone and shouldn't warn - #3: NULL service_config now warns (anomalous case) - #4: post-Google-API DB-update + commit failures log distinctly so the channel-orphan case is grep-able Plus: add 14d expiry to Google webhook tokens via ServiceName::webhook_token_expiration, mint fresh ephemeral-webhook-{service}-{rd5} labels at create + rotate so the existing 'ephemeral-' filter excludes them from user-token email/critical-alert paths (no filter changes in 3 places). Orphans now self-clean via the existing expiry sweep in monitor.rs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fixup! fix(native-triggers): serialize Google channel renewal across replicas fixup! fix(native-triggers): serialize Google channel renewal across replicas Address second-round review: - Claude #1 (P2): username_override_from_label now strips the 'ephemeral-' prefix for ephemeral-webhook-* labels, so created_by stays webhook-{service}-{rd5} instead of changing to label-ephemeral-webhook-... (preserves audit/job-list filter compatibility) - Codex (P2): updated renew_channel doc — labels are no longer copied; rotate mints fresh ephemeral-webhook-google-{rd5} with 14d expiration - Claude #3 (optional): test_rotate_webhook_token now asserts the rotated Google token has an ephemeral-webhook-google-* label and a populated expiration Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fixup! fix(native-triggers): serialize Google channel renewal across replicas fixup! fix(native-triggers): serialize Google channel renewal across replicas Reconsider the previous fixup: stripping the 'ephemeral-' prefix made created_by no longer match token.label exactly, defeating the linking purpose. Just allowlist 'ephemeral-webhook-' alongside the other recognized webhook/email/ws prefixes — created_by becomes ephemeral-webhook-google-XXXXX, matching token.label exactly. The 'ephemeral-' substring also informs operators that this is a system-managed auto-expiring token vs a user-managed webhook trigger. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): bump svelte version in `wmill app new` template (#9084) * fix(cli): bump svelte version in `wmill app new` template The svelte5 template pinned `svelte` to `5.45.2`, but the Svelte compiler bundled in `wmill app dev` emits `$.delegated('click', ...)` calls. The `delegated` export was added later, so 5.45.2 doesn't have it — esbuild warns `Import "delegated" will always be undefined`, replaces the call with `void 0`, and the page crashes at first event-handler bind (white screen). Bump to `^5.55.5` so the compiler and runtime stay in sync. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(frontend): bump svelte version in raw_apps UI template Mirror the CLI fix: the UI's `Add raw app` flow scaffolds a package.json with `svelte: "5.45.2"`. That works today only because the bundled rolldown worker also pins 5.45.2 — when the worker is upgraded past 5.51.1, the compiler will emit `$.delegated()` and the runtime won't have it, producing the same white-page crash that hit the CLI. 5.55.5 still exports `event` (used by the current bundled compiler), so this is forward-compatible: it works with the 5.45.2 compiler now and won't break when the worker is upgraded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(flows): gate flow_env resolve on expr text and share cache with handle_flow (#9085) * feat: parse windmill_failure field to tag run as failure (#9073) * feat: parse windmill_failure field in job result to tag run as failure * feat: preserve top-level fields when windmill_failure tags a run as failure * fix: address review findings on windmill_manual_failure * refactor: rename windmill_manual_failure to wm_failure and add wm_* aliases * fix: prefer injected ManualFailure error over sibling name/message in OTel * fix: hide _ENTRYPOINT_OVERRIDE jobs from script/flow history panel (#9088) * fix(flows): populate error handler input args from failure picker (#9087) * fix(flows): populate error handler input args from failure picker * style(flows): fix indentation in failure-step branch * fix(python): verify wheel RECORD on cache pull/install, finalize piptar (#9090) The Python per-package dependency cache could persist an incomplete wheel extraction with `.valid.windmill` set, then propagate that broken artifact to every worker through the object store. Customer hit this on argon2-cffi==25.1.0 (missing argon2/_utils.py), and previously on botocore/httpx (truncated tars). Symptom is a runtime ImportError that looks like a missing dependency declaration rather than a Windmill bug. Three changes that together stop the propagation: 1. After `pull_from_tar`, parse the wheel's `<dist-info>/RECORD` and confirm every listed path exists on disk before writing `.valid.windmill`. On failure, wipe the directory and fall through to a fresh local install — the next install also self-heals the broken object-store entry by pushing a fresh tar. 2. After `uv pip install` succeeds, run the same RECORD check before queuing the piptar upload or writing `.valid.windmill`. A bad install never becomes the source of a broken tar in the object store. 3. Finalize the tar (`drop(tar.into_inner()?)`) before reading its bytes for upload, so we never push an unfinalized archive (no end-of-archive marker) to the object store. Verified with a 60-package end-to-end integration test (first-fill → clear-local-cache → re-pull-from-objectstore → corrupt-objectstore-tar → detect-and-self-heal). All 27 packages on the live test pulled cleanly, and the deliberately corrupted argon2-cffi tar was caught with the exact expected log line ("wheel RECORD lists files missing on disk: argon2/_utils.py") and replaced with a fresh tar. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(main): release 1.699.0 (#9082) * chore(main): release 1.699.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * feat(cli): auto-infer args for `wmill app push` (#9091) Run `wmill app push` from inside an app folder (e.g. `f/foo/my_app.app/`) with no args. The local path defaults to CWD, and the remote path is derived from CWD relative to `wmill.yaml`, with `.app`/`.raw_app`/ `__app`/`__raw_app` suffixes stripped. Either, both, or neither positional argument can be passed. Also resolves `file_path` against the user's original CWD before `resolveWorkspace` may chdir to the wmill.yaml root, so a relative `file_path` argument is interpreted from where the user invoked the command (previously it could resolve against the wrong directory). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * all * fix(pipeline): live-update graph for annotations and body assets * fix(pipeline): persist draft body edits across node switches * fix(pipeline): persist live writes per draft to keep output node fresh after switch * feat(pipeline): animate graph edges only while a runnable is executing * feat(pipeline): add run button on script nodes + recomputing hint on preview * feat(pipeline): compact preview layout, two-way Test/Run sync * fix(pipeline): test button cross-browser placement (no overflow trick) * style(log-viewer): replace took/mem-peak labels with timer/cpu icons * style(log-viewer): hyphenate Auto-scroll label and prevent wrapping * style(log-viewer): lowercase auto-scroll label, force vertical scrollbar * style(log-viewer): force horizontal scrollbar instead of vertical * fix(log-viewer): scope overflow-x to top bar so pre doesn't drive panel width * fix(pipeline): overlay live body-asset writes for persisted scripts too * fix(pipeline): persist inferred body assets at save so edges survive page reload * fix(pipeline): snapshot live draft writes at persist time so they survive reload * fix(pipeline): keep inferred body writes on the canvas across selection changes * fix(pipeline): untrack inferredWrites cache mutation to break effect loop * fix(pipeline): refetch asset graph after persisted-script save * feat(pipeline): optional AI prompt when creating a pipeline script * all * all * test: cover asset-trigger dispatch end-to-end through worker * feat(pipeline): split-button Test with optional downstream cascade * feat(pipeline): cascade option on graph Run + match button heights * style(pipeline): match caret bg/text to Test button's accent-secondary * feat(pipeline): split Run pill on graph node exposes cascade option * feat: live run activity + status badges in pipeline asset graph - folder-scoped queue poll lights up the downstream asset-trigger cascade (not just the launched script); zero requests at rest, catch-up for fast hops, auto-disarm when idle - per-runnable node badge: last-run status + session run count - animate unsaved/live-parsed edges (was unconditionally suppressed) - background-pane click no longer clears selection - run-bridge guarded so node selection/save no longer triggers a test Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: live activity log, optimistic badges, node-avoiding graph edges - collapsible folder activity log (PipelineEventLog): live job feed, polls only while open/active, slow idle cadence, capped + pruned - composable: observe mode + events list + run-count anchored to graph-open time (pre-existing history excluded) - optimistic node badge: launched script shows running instantly via the zero-latency activeRunnable hint, keeps the polled run count - activity pane height capped (min(18rem,40vh)) then scrolls - route asset-graph edges through sugiyama-computed waypoints so they go around nodes instead of under them; bezier fallback for adjacent-layer / draft-overlay edges Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: prefetch all folder script assets so graph is stable on load On pipeline load, eagerly infer body assets for every persisted folder script and seed the existing inferredWritesByPath overlay, instead of only filling it when a node is selected. Scripts whose persisted asset rows are missing (e.g. object-form writeS3File) now have their edges from first paint, so clicking a node no longer re-layouts the graph. One-shot per (workspace, base-graph) load, untracked map reads, generation-cancelled, pool-capped fetches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf: guard no-op poll re-layout; dedupe write-asset extraction - skip reactive ids/states/events reassignment when unchanged, so an idle poll tick no longer re-runs the full sugiyama layout every 3-6s - bound countedJobIds (rebuilt from eventsById in lockstep with prune) - extract shared extractWrites() helper, replacing 4 copy-pasted write-asset filter/map blocks in the pipeline page - compute activeRunnable node-id once, reuse for the active-edge set and the optimistic badge (flattened ternary); trim narrating docs Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: live read-lineage overlay for inferred body assets Renaming e.g. duckdb read_parquet('s3://...') / loadS3File now updates the asset->reader edge live instead of only after Save re-derives the persisted asset rows. - extractReads() (+ shared refsByAccess) mirroring extractWrites - inferredReadsByPath sticky cache, filled by handleAssetsChange and the load prefetch alongside writes - replace the write-only overlay loop with one overlayLineage(map, access) helper invoked for both 'w' and 'r' (net DRY) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: detect S3 assets passed as SDK object arg in ts parser Mirrors merged PR #9181 so feat/asset-graph-view is self-contained (local origin/main is stale and lacks it). Object/{ s3, storage } form of writeS3File/loadS3File is now detected, not only the bare s3:// string literal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regenerate wasm Cargo.lock + frontend package-lock Lockfile churn from local wasm-pack (asset target) + npm operations during the asset-graph work. No source/dependency-intent change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: revert to bezier graph edges; add parsing-assets hint The sugiyama-waypoint routing looked worse than the original; revert AssetGraphEdge/assetGraphLayout to the pre-routing bezier logic (same as the flow editor's BaseEdge) and drop the now-unused route plumbing from the canvas. Add a small 'Parsing assets…' hint shown while the load-time prefetch sweep is still inferring folder scripts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: extract pure resolveGraph merge + unit tests Move the ~230-line graphWithDraft precedence/merge (base < session- inferred < draft-seeded < open-script-live, +read/write/annotation overlays, +dedup) out of the 1648-line route into a pure, testable resolveGraph() module; the route's graphWithDraft is now a thin $derived. Behaviour extracted verbatim. 10 unit tests cover the precedence matrix. Phase 1 of the state/render split. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: graph controls top-right, lift minimap, hide Save when unchanged Controls -> top-right horizontal, no lock toggle; MiniMap !mb-10 so it clears the activity bar; hide the per-script Save button when the script is already at its latest save point (drafts still show Create). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: scope runtime-asset prune by id to spare static lineage rows prune_runtime_assets deleted by (workspace_id, path, kind) tuple, so trimming surplus usage_kind='job' rows for an s3 path also wiped the static usage_kind='script'/'flow' producer rows for the same path — silently breaking the asset-trigger cascade (fetch_producer_writes found no writes; downstream never dispatched; required band-aid re-syncs). Delete the surplus job rows by id instead; the inner query is already scoped to usage_kind='job'. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: don't re-pulse already-running jobs after they finish The catch-up pulse re-added a completed job to the active set if its start was within the (lagging) lookback window — even one we'd already animated the whole time it ran — keeping its edges lit ~a poll interval past completion (~5s after a 3.5s test). Track job ids seen in-flight and skip the pulse for them; it still fires for hops whose whole lifetime fell between two polls. Bound the set in lockstep with eventsById; cleared on dispose. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: don't catch-up-pulse the runnable launched from the graph If the poll never sampled a launched run's in-flight window, the catch-up pulse re-flashed its edges one tick after it correctly stopped (the page already animated it zero-latency via activeRunnable). arm(launchedId) records the launched runnable id; catch-up skips it. Cascade hops (other ids) still pulse. launchedIds cleared on stop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: nudge graph controls left to clear panel toggle Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: partition value resolver + asset-cascade propagation windmill-common/partition: pure resolver — time kinds (tz/format/start anchor) + dynamic $.a.b JSONPath; 9 unit tests. asset_dispatch: read the producer's resolved partition and thread it into every cascaded subscriber's args + trigger.partition, so a chain resolves once at the top. No migration (cascade needs no spec lookup). Stage 1+3 of pipeline partition runtime; run-start resolution is Stage 2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: show args form in compact pipeline preview when script has inputs AssetGraphDetailsPane keeps the compact (hideArgs) preview but, via a new previewPanel.argsAboveLogs flag, renders a compact SchemaForm between the floating Test button and the logs/result panel when the script declares inputs (e.g. a partitioned script needing a `partition` arg). The preview pane also grows ~18pts so the args form doesn't shrink logs/result. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: parser join-mode (`// trigger all`) + script_trigger.join_all Stage A: JoinMode{Any(default),All} + `// trigger any|all` directive in parse_pipeline_annotations; TriggerSpec::is_partition_bearing() (path contains {partition}); join_mode threaded through all 4 asset-parser crates (ts/py/sql/yaml). Stage B: reversible migration adds script_trigger.join_all; insert_script_trigger writes it; deploy path sets it from the parsed annotation. No reader yet (AND-join dispatch is the next stage) so runtime behaviour is unchanged. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: resolve pipeline partition at job execution time Stage C: in handle_code_execution_job, once the script content is loaded, parse the // partitioned annotation (free here) and resolve the concrete partition once — schedule fire-time (scheduled_for anchor, not wall-clock) for time kinds, triggering payload for dynamic. The value is injected into the in-memory args the body sees (via a shadowed job clone) and persisted back to v2_job.args so dispatch_asset_triggers propagates the same value down the cascade. Already-set (explicit/backfill/cascade) partitions are never re-resolved (run identity immutable); unresolvable partitioned runs fail with a clear error. Integration test exercises the full worker loop + cascade propagation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: AND-join barrier for partitioned pipeline subscribers Stage D: a // trigger all subscriber no longer fires on any input. New join_pending_inputs slot table keyed (workspace, subscriber, partition); fetch_subscribers now returns join_all and the dispatch loop records each partition-bearing input arrival, pushing the subscriber once only when every partition-bearing input it declares is present for that partition. Per-partition slots, cleared on fire (re-accumulate, no double-fire), skew-immune (unlike debounce). Case-3 guard: an unpartitioned producer or a reference (non-{partition}) input never fires a partitioned join. Integration test covers wait/fire/isolation/no-double-fire. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: opt-in // debounce for asset-cascade subscribers (parser + schema) Stage E1+E2. Parser: script-level // debounce <dur> + per-// on debounce=<dur> override (edge wins, else script default, else none = fan-out, unchanged); TriggerSpec::Asset carries the per-edge override; split_trailing_kv_opts separates the ref from trailing key=val opts. Schema/deploy: reversible migration adds script_trigger.debounce_s; parse_duration_secs (bare int or <n>s|m|h|d, fail-safe on garbage) resolves the effective per-edge window at deploy and writes it per row. No reader yet (dispatch wiring is E3) so runtime is unchanged. New unit tests for the parser directive and duration parsing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: apply opt-in debounce to asset-cascade subscriber dispatch Stage E3. fetch_subscribers now also returns debounce_s; push_subscriber builds real DebouncingSettings (delay + a (subscriber, partition) key, so distinct partitions never collapse and latest-in-window falls out) instead of ::default() when the edge opted in. Default stays no-debounce (fan-out — the prior deliberate behaviour, now overridable rather than reversed). Wiring test asserts the dispatched job carries the configured window/key and an undebounced edge carries none. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix: atomic AND-join gate + preserve resolved partition; drop scratch artifacts Addresses local-review findings before PR: - P1: record_and_check_join_slot was a non-atomic check-then-act on a pooled connection; concurrent completion of a subscriber's last two partition-bearing inputs on different workers could double-dispatch. Now one transaction guarded by a tx-scoped advisory lock keyed on (workspace, subscriber, partition) so the gate fires exactly once. - P2: the preprocessed-args overwrite in result_processor replaced args wholesale, dropping a partition resolved by resolve_partition_for_job; the UPDATE now preserves an existing persisted partition key. - P2: gate resolve_partition_for_job on a cheap code.contains check so non-pipeline script jobs skip the annotation scan on the hot path. - P2: remove 40 scratch screenshot PNGs, a flicker-debug script and a local scheduler lock accidentally committed; gitignore the lock. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: AND-join fires once under concurrent upstream completion Regression for the check-then-act race fixed by the advisory-locked transactional gate: releases N producer dispatches simultaneously via a barrier and asserts the AND subscriber is pushed exactly once and the slot is cleared. The invariant holds for the correct gate regardless of interleaving; a non-atomic regression fails it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: fuller partitioned join + multi-hop pipeline coverage Exercises a complex pipeline combining options end to end: two partitioned producers fanning into a // trigger all join, then a multi-hop downstream chain. Asserts the resolved partition propagates unchanged at every hop, chain depth increments per hop, the AND barrier fires exactly once, and a second partition opens an independent slot with no cross-partition bleed across the whole graph. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor: simplify pipeline code per review (dedup, single-parse, constant) - ParseAssetsOutput::new() collapses the 6-line annotation copy-paste across the 4 asset-parser crates to one call site. - asset_dispatch: parse the cascade trigger object once and pass it to the depth/partition readers instead of deserializing it twice; add a TRIGGER_ARG constant for the previously stringly-typed key (3 sites). - scripts deploy: drop a redundant debounce_default clone. No behavior change; 29 parser + 6 dispatch integration tests green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: reap abandoned AND-join slots after a TTL (default 60d, per-slot) join_pending_inputs slots are normally cleared when the join fires; partial slots whose inputs never all arrive (upstream removed/renamed, one-off dynamic partition key, permanent skew) would otherwise leak. windmill_queue::asset_dispatch::reap_stale_join_slots, called from the monitor's delete_expired_items loop, deletes a (workspace, subscriber, partition) slot only when its MOST RECENT row is older than JOIN_SLOT_TTL_SECS (60d) — per-slot, never per-row, so a legitimately slow join is not corrupted mid-accumulation. Conservative default; per-join configurable TTL via the annotation is a planned follow-up. Test covers stale-reaped / fresh-kept / mixed-slot-kept. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * update * feat: path-less native trigger markers + missing-trigger placeholder * feat: pipeline // tag and // retry annotations + dispatch_event log * fix: derive test-pane min from split-axis dimension (height in bottom layout) * feat: show last run logs/result when a script node is selected * fix: backfill asset rows from script.assets for pre-feature scripts * feat: job-id link + dispatch popover above script log/result * style: drop 'dispatched' label, keep just the check icon * fix: drop tag picker from pipeline script editor (set via // tag annotation) * Nicer UI * refactor: move google ai proxy handling to windmill-ai (#9260) * refactor: add ai proxy execution mode * refactor: move google ai proxy handling * refactor: share google ai request building * fix: early return should consider failure_module result (#9241) * fix(flows): flag noLogs jobs and lazily resolve them in log panel (#9099) * fix(flows): flag noLogs jobs and lazily resolve them in log panel * fix appending to flag * fix: preserve WM_LOGS_SKIPPED sentinel on SSE/replay completion pickMoreCompleteLogs resolved both sentinel and undefined to '', so the SSE completion event (whose job field is fetched .without_logs()) would clobber the sentinel placed by flagSkippedLogs. The module log panel then saw '' instead of the sentinel, defeating the lazy-resolve path. Also wire onLogsResolved on the OutputPickerInner inline LogViewer so a lazy resolve writes back to flowStateStore.previewLogs, matching ModulePreviewResultViewer and avoiding repeated fetches on remount. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(main): release 1.705.0 (#9229) * chore(main): release 1.705.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * chore: add playwright mcp for frontend verification (#9269) * feat: CLI datatable serve / psql (#9267) * feat(cli): add datatable list and run commands * feat(cli): render datatable query results as a table * feat(cli): serve datatables as a postgres-wire endpoint * feat(cli): add 'datatable psql' to launch psql against the proxy * feat(cli): route datatable serve by client-supplied database name * override database list + password option * fix: support extended queries in datatable serve * fix: correct cloud size threshold log and parse CLI descriptions with parens/trailing comma * refactor: extract raw_output envelope encoding into pg_raw_output module --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> * oom_adj nit * feat: add UV_PYTHON_INSTALL_MIRROR env and instance setting (#9271) * feat: add UV_PYTHON_INSTALL_MIRROR env and instance setting Allows operators to point `uv python install` at a private mirror of the python-build-standalone releases. Configurable via the `UV_PYTHON_INSTALL_MIRROR` env var or the `uv_python_install_mirror` instance setting, with the env var as the boot fallback and the instance setting taking precedence at reload. Fixes WIN-1966 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: hoist uv_python_install_mirror binding above sandboxing branch The non-sandboxed uv pip install branch referenced a binding that was only declared inside the sandboxed branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: neutral placeholder for uv_python_install_mirror The previous placeholder was the default public URL the setting is meant to redirect away from. A neutral example mirror URL is clearer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(indexer): tell admins when ingress routes search to wrong pod (#9274) * [ee] fix(indexer): tell admins when ingress routes search to wrong pod When the IndexReader is absent on the pod handling a search request but another pod is actively holding the indexer lock, the EE handler now returns a tailored error pointing at the ingress/load-balancer configuration instead of the generic "indexer not running" message. The indexer status endpoint reads the DB lock so it reports "running" from any pod, but search endpoints need the in-memory IndexReader that only exists on the lock holder. In multi-replica deployments this looks like the indexer is healthy but every search 404s. Companion: windmill-labs/windmill-ee-private#TBD Fixes WIN-1968. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to eb18d7b4c0e37fea3f6e1e2cc44e0fddd74ff817 This commit updates the EE repository reference after PR #586 was merged in windmill-ee-private. Previous ee-repo-ref: 7dd43d1850813071cc18ba49ba090583e7321f4b New ee-repo-ref: eb18d7b4c0e37fea3f6e1e2cc44e0fddd74ff817 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> * feat(cli): add `wmill init prompts` and custom override slot (#9266) * feat(cli): add `wmill init prompts` and custom override slot Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): replace init prompts with refresh prompts + AGENTS.md/AGENTS.cli.md split Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): dedupe claude skills via @-includes and add prompts freshness check Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): drop migration-choice flags from `refresh prompts` Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(cli): add 'Running and previewing local changes' section to AGENTS.cli.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): write full skill content to .claude/, drop @-include wrapper Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): reconcile CLAUDE.md the same way as AGENTS.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): address PR review nits — argv parsing, lazy import, comment detection, error propagation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add yolo mode for ai chat tools (#9258) * feat: add yolo mode for ai chat tools * nit * fix: align chat footer controls * feat: add ai chat autonomy modes * feat: add autonomy mode dropdown * fix: highlight yolo autonomy icon * fix: auto accept flow edits * fix: hide unsupported autonomy modes * fix: handle auto-accept flow editor races * fix(debugger): add non-root user support to Dockerfile (#9277) Mirrors the main Windmill Dockerfile pattern: creates a windmill user (UID/GID 1000) and makes cache/work directories world-writable so the image runs cleanly under Kubernetes securityContext.runAsNonRoot or runAsUser: 1000 without permission errors on Bun, pip, or windmill cache writes. Fixes WIN-1969 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ai): enforce RLS and scope check on user-supplied X-Resource-Path (#9276) * fix(ai): enforce RLS and scope check on user-supplied X-Resource-Path The AI proxy handler accepts an X-Resource-Path header to override the configured workspace AI provider. When supplied, the handler loaded the resource value from the resource table using the root DB pool with no resources:read scope check, so any authenticated workspace user could point X-Resource-Path at a restricted AI resource (e.g. one in a folder they cannot read) and the proxy would use that resource's provider credentials for the outbound AI request. For user-supplied resource paths, now require resources:read:{path} scope and fetch the resource through user_db.begin(&authed) so RLS enforces the same folder/group boundary as the resource API. The RLS- scoped $var: resolution stays in place as defense in depth. The admin-configured workspace/instance ai_config path is unchanged. Fixes WIN-1971 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(ai): regression test for X-Resource-Path RLS enforcement Cover all four cases: - non-admin pointing X-Resource-Path at a restricted resource is rejected - non-admin pointing it at a resource they own still works - admin can point it at any resource - workspace-configured proxy flow (no X-Resource-Path) is unchanged Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add userdraft listing primitives (#9268) * feat: add userdraft listing primitives * fix: cancel stale userdraft discard writes * docs: remove global ai userdraft plan * feat(nsjail): optional disk-backed /tmp via instance setting (#9272) * feat(nsjail): optional disk-backed /tmp via instance setting * test(nsjail): unit-test tmp mount resolver and narrow visibility * refactor(nsjail): switch tmp backing to select + conditional UI * ui(nsjail): make tmpfs the visible default in /tmp backing select * fix(nsjail): refuse preexisting jail_tmp to block symlink escape * fix(nsjail): allow jail_tmp reuse on sequential nsjail calls Codex flagged that python/ruby/rust executors invoke nsjail twice per job_dir (install then run). The previous resolver treated any preexisting jail_tmp as hostile and silently fell back to tmpfs on the second call, so disk-backed mode never reached the main script run for those langs. Use symlink_metadata().is_dir() to distinguish a real directory left by an earlier call in the same job_dir (safe to reuse) from a symlink or other entity (still refused, as the codebase-tar escape requires). Also loosen the frontend visibility predicate: only hide nsjail settings when job_isolation is explicitly 'none' or 'unshare', so deployments that enable nsjail via DISABLE_NSJAIL=false with no DB setting can still see the controls. * chore(main): release 1.706.0 (#9270) * chore(main): release 1.706.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix(nsjail): gate unix-symlink test behind cfg(unix) for Windows build (#9280) The disk_backed_refuses_preexisting_symlink_at_jail_tmp test calls std::os::unix::fs::symlink directly, which doesn't exist on Windows targets. Without a cfg gate, `cargo check --tests` fails on Windows with E0433. Other symlink call sites in this crate (php_executor, bun_executor, rust_executor, etc.) already follow this pattern. Fixes WIN-1972 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Reduce slim image vulnerability surface (#9279) * Reduce slim image vulnerability surface * chore(docker): drop apt-get upgrade -y from slim images apt-get upgrade hurts build reproducibility (same Dockerfile + same commit at different times produces divergent images) and trips hadolint DL3005. The freshness it buys is dominated by simply rebuilding against the periodically-refreshed debian:bookworm-slim base image. The --no-install-recommends and apt-list cleanup wins are kept. --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev> * fix(git-sync): bump to hub/28234 with stateless gpg.program wrapper (WIN-1974) (#9282) * fix(git-sync): revert LATEST_GIT_SYNC_SCRIPT_PATH to hub/28230 to restore GPG-signed deploys (WIN-1974) hub/28231 (PR #9230) is the "thin" script that hands the actual `git commit` to the CLI's hidden `sync git-deploy`. The hub script still does the GPG setup (import key into a fresh GNUPGHOME, dummy `gpg -bsau` to warm the agent passphrase cache, then `git config user.signingkey` + `commit.gpgsign` locally), but the commit no longer runs in the same `git_push` flow — it runs minutes later inside the CLI after workspace API resolution, zip pull, file extraction, and lockfile autofill. By the time the spawned `git commit` asks gpg-agent for the cached passphrase, the cache state is no longer reliable (or the spawned `gpg` ends up talking to a fresh agent), so signing fails non-interactively with `gpg failed to sign the data`. hub/28230 is hub/28217's in-script logic rebuilt with windmill-cli@1.703.3: the GPG setup and the in-script `sh_run("git commit ...")` happen back-to-back in `git_push`, so the cache is always fresh. It preserves wm_deploy / fork branch behavior, the EE deployment-callback `main()` signature is unchanged, and the only min-version check in EE (`is_script_meets_min_version(28103)`) is comfortably below 28230 — so this revert is safe. Forward fix (separate PR): publish a new thin script that, alongside the existing GPG setup, writes a `gpg.program` wrapper using `--pinentry-mode loopback --passphrase-file` so signing is independent of the agent's cache state. Re-bump past 28231 then. Fixes WIN-1974 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(git-sync): check in source-of-truth for the next hub script (gpg.program wrapper) This is the script that will be published to hub.windmill.dev once verified on a customer GPG-signed deploy. It replaces hub/28231's agent-cache pre-warm (`gpg -bsau` with --passphrase) with a stateless gpg.program wrapper + chmod-600 passphrase file. Every git-invoked gpg call goes through the wrapper, which always uses --pinentry-mode loopback (and --passphrase-file when a passphrase exists). Signing no longer depends on gpg-agent having a cached passphrase by the time the CLI's `git commit` runs — which closes WIN-1974. Not wired in yet: LATEST_GIT_SYNC_SCRIPT_PATH stays on hub/28230 until this script is uploaded and the new hub id is known. This file is checked in so the diff is reviewable, future bumps have a source of truth, and a CLI regression test can `cat` it for fixture parity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(frontend): skip format/pattern validation for $var/$res/$jsonvar references in ArgInput A resource field with a `pattern` constraint (e.g. the gpg_key.private_key field, whose pattern enforces a `-----BEGIN PGP PRIVATE KEY BLOCK-----` prefix) rejects values like `$var:u/me/gpg-private-key` with an "invalid format" error in the resource editor — even though `$var:`/`$res:`/`$jsonvar:` are placeholders the backend resolves at runtime, not the actual string that needs to match the regex. Bail out of all format/pattern checks (email, ipv4, ipv6, uuid, custom pattern) when the value is one of these references. Required/numeric bounds/array checks still apply since they're shape-level, not regex. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(git-sync): bump LATEST_GIT_SYNC_SCRIPT_PATH to hub/28234 (gpg.program-wrapper fix) hub/28234 is the forward fix for WIN-1974: replaces hub/28231's agent-cache pre-warm (which became stale by the time the CLI's `git commit` ran) with a stateless `gpg.program` wrapper that uses `--pinentry-mode loopback` (and `--passphrase-file` when a passphrase exists) on every gpg invocation. Bundled CLI is windmill-cli@1.705.0. Verified via reproducer at /tmp/git-sync-diff/test-gpg-fix.sh: deliberately killing gpg-agent between GPG setup and `git commit` reproduces the customer's `gpg failed to sign the data` error verbatim under the old flow, and the wrapper signs through it. Holds for passphrase-protected keys, split-subkey [C]+[S] layouts, and unprotected keys. Drops the local source-of-truth copy (`hub-scripts/`) — hub is canonical now that 28234 is published. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(git-sync): drop verbose comment above LATEST_GIT_SYNC_SCRIPT_PATH The git history (this PR) carries the why; the constant name + value carry the what. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): wmill sync git-deploy stops committing; caller owns commit+push (#9284) Single contract for the deployment-callback path: the CLI does branch checkout + pull, the caller (hub script in production, test in test) does git add + commit + push. This restores the WIN-1974 invariant — GPG setup and `git commit` run back-to-back in the same process, so the agent's pre-warmed passphrase cache is still warm at sign time — without needing a `--skip-commit` flag for the hub case and a default "also-commit" for everything else. Same behavior in every call site. Changes: - sync.ts: drop the gitSyncDeployPush call from pull()'s deploy path (both the onlyCreateBranch fast-return and the post-pull commit). `gitSyncDeployPush` stays exported for any caller that wants the same commit/push semantics — just not invoked by the CLI subcommand. - gitsync_promotion.test.ts: e2e test now does its own git add + commit + push after `wmill sync git-deploy`, mirroring what the hub script does in production. Same regression coverage (wm_deploy branch created in Case A, main untouched; main updated in Case B, no new wm_deploy). CLI typecheck unchanged (two pre-existing TarAsZip errors at lines 2578/3307, present before this PR). All 743 unit tests still pass. The accompanying hub script (option-C — CLI for branch+pull, script for commit+push) lives at /tmp/git-sync-diff/sync-script-to-git-repo-windmill.option-C.ts. Once published, a follow-up bumps LATEST_GIT_SYNC_SCRIPT_PATH to its id. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bump git sync to 28236 * fix: fork compare visibility for non-admins and stale-token superadmins (#9283) * fix: use fork-scoped authed for fork visibility in compare_workspaces * test: add EE end-to-end repro for fork rename visibility * chore: restore concurrency_locks sqlx cache lost in cleanup * test: add regression for stale-superadmin-token fork visibility bug * chore: update sqlx cache for new test queries * chore(main): release 1.706.1 (#9281) * chore(main): release 1.706.1 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * feat: add wmill job rerun subcommand (#9275) * feat: add wmill job rerun subcommand * feat: add wmill job restart subcommand for flow restart-at-step * chore(system_prompts): point plugin skills sync at plugins/windmill/ (#9287) * chore(system_prompts): point plugin skills sync at plugins/windmill/ The plugin checkout's plugin folder is being renamed from `plugins/windmill-code-plugin/` to `plugins/windmill/` to shorten the slash-command namespace and align with the matching Cursor plugin layout. Paired with windmill-labs/windmill-claude-plugin#8. That PR must merge first so the next sync run finds the new folder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(system_prompts): update plugin-dir example to plugins/windmill Co-authored-by: centdix <centdix@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: centdix <centdix@users.noreply.github.com> * fix(cli): wmill sync pull updates wmill-lock.yaml for raw apps (#9289) * fix: flow recording teardown crash + rename package to @windmill-labs/components (#9288) * fix: guard against null recording during FlowRecordingReplay teardown Navigating away from a flow recording inside a workspace file-tree view threw `TypeError: Cannot read properties of null (reading 'flow')` from FlowGraphViewer once during the teardown tick. Svelte 5 compiles child component props as live getters that close over `$$props.recording.flow`. When `recording` flips to null on the parent's navigation, an outer `{#if !recording?.flow}` doesn't stop those getters from firing one more time as derived effects re-evaluate before the unmount lands — so the getter dereferences null and throws. Fix at the two layers where the deref actually happens: - FlowRecordingReplay: use `recording?.flow` at the binding sites (FlowViewer + graph-snippet FlowGraphViewer) so the compiler emits an optional-chained getter, and guard the snippet branch with `{:else if recording?.flow}` so it doesn't mount when there's nothing to show. - FlowGraphViewer: finish the optional chaining the rest of the file already used everywhere else (`flow?.value?.skip_expr`, `flow?.value?.cache_ttl`, `flow?.schema`). When the upstream binding returns undefined during teardown, the graph degrades to an empty frame instead of crashing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rename package to @windmill-labs/components - frontend/package.json: rename `windmill-components` → `@windmill-labs/components` - frontend/publish.sh: drop the in-place sed rename dance; the checked-in name now matches what's published, so `npm run package && npm publish` is enough - frontend/package-lock.json, system_prompts/auto-generated/prompts.d.ts: regenerated by `npm run package` under the new name Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * default script name * save logic * Keyboard nav * finish keynav * nits * CI fix * nit stop propagation * Merge branch 'main' into feat/asset-graph-view * commit * update * fix: cropped save button on small screens * progress * managed scheduled removed * all * progress * feat: add data upload pipeline trigger with auto S3 picker Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: avoid pane editor remount flicker when deploying a pipeline draft Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: show only the edited script's I/O in the asset graph, not the saved version's Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: derive script asset rows server-side at deploy Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: shared fixture corpus keeps annotation parsers in parity Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: dev-run draft pipeline chains, live badges, deploy drift warning Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: ungate cascade producers, squash pipeline migrations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: drop committed cli-sync fixtures and stray screenshots Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: show skip-asset-dispatch flag as badge instead of args row Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: pipeline view mode default with activity feed, drafts overlay chip Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: treat DROP TABLE as table-level write in sql asset parser Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: wmill datatable create + actionable sql extension error Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: ephemeral data-pipelines demo sync repo zip for handoff Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: wmill pipeline list/show renders the asset DAG in the terminal Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * nits * nits * nits * nits * fix: defer draft persist-back past the batch so discard sticks first click Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: band-reserving tidy-tree asset graph layout with join breakpoints Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route skip-layer and long graph edges around occupied columns Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: seed s3 template outputs with canonical leading-slash paths Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * all * feat: bundle data-pipeline drafts into the DB-backed user draft system Pipeline drafts were browser-only (localStorage `pipeline-<folder>`), so they didn't sync across devices, weren't server-visible, and never showed in the drafts list. Store them instead as one per-user `draft` row of a new `data_pipeline` kind, keyed at the folder (`f/<folder>/data_pipeline`), holding the same `{ drafts, activeDraftPath }` bundle. Stage 1 — backend kind: add `data_pipeline` to DRAFT_KIND (migration) and `UserDraftItemKind` (deployed_table=None, private). The list/update handlers and folder-path access check already cover a backing-table-less kind. Stage 2 — sync: add `GET /drafts/get_own/{kind}/{path}` so an editor with no deployed-overlay GET can load its own draft. The pipeline page now hydrates from the DB on mount (one-time localStorage import for in-flight drafts) and persists via UserDraftDbSyncer (debounce + optimistic-concurrency), keeping a localStorage crash mirror. Stage 3 — surface: the drafts review page renders the bundle as a "pipeline" row that opens `/pipeline/<folder>` (open-only; excluded from bulk deploy). Verified end-to-end in-browser: DB-seeded draft hydrates to "Edit (1)", edits persist back, and the row shows with Open pipeline / Discard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: pipeline Activity panel grouping, run↔graph highlight, deploy-conflict handling Activity panel (view mode): - Group cascade runs by the connected component of the asset-dispatch graph (new GET /jobs/asset_dispatch_edges over the dispatch_event table, incl. join_pending inputs), headed by the earliest originating run + its trigger, with a "+N" chip for joins fed by multiple triggers. - Success/failure count histogram with drag-to-filter brushing, an always-on time axis + per-bar tooltips, a Reset, and Last hour/24h/48h/7/30/90d ranges. - Node run-count/status badges now derive from the same merged historic+live events the panel shows (previously session-only). Run ↔ graph highlight: - Hovering a run row (or a group header → the whole cascade) rings the node(s), animates their incident edges, and borders the adjacent assets in the edge hue (blue write / gray read); expanding a run pins a soft-blue ring. - Switching edit→view re-surfaces the Activity feed. Deploy: - Live-content autosave for the open pipeline draft + an autosave indicator. - Re-saving a script now chains off the hash just created instead of a stale parent_hash (fixes the "lineage must be linear" error on a second save), and a genuine concurrent deploy opens a keep-mine / view-latest conflict modal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: pipeline editor badge requires asset-parse, not just main-function parse A pipeline script's asset lineage is load-bearing — a deploy that can't parse assets silently records no edges. The editor "parsable" dot only reflected inferArgs (the main function), so a body the asset parser rejects (e.g. a trailing `/////` in DuckDB) still showed green and deployed with empty lineage. ScriptEditor gains `requireValidAssets` (set by the pipeline pane); when on, the EditorBar badge is green only if BOTH the main function and inferAssets parse, with the tooltip distinguishing "Main function not parsable" / "Assets not parsable" / "Parsable". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: route asset-graph edges around nodes that sit in their path Edges could draw straight through an unrelated node (a join fan-out or long cross-component edge), making it ambiguous whether that node shared the input. AssetGraphEdge only saw its own endpoints, so it could only detour the near-vertical same-column skip case. The canvas now (once per layout, O(edges × nodes) — no per-frame cost) samples each edge's straight run against every non-incident node center and, on a crossing, passes a clear gutter lane to the edge via `data.detourX`; AssetGraphEdge routes the rounded-orthogonal detour through it. Verified: 0 edge↔node box crossings on the orders pipeline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: deploy pipeline drafts with freshly-inferred assets, not a stale snapshot "Save all" spread `...draft.script` into createScript, which carries a `assets` snapshot that isn't refreshed when the body is edited. So a renamed/removed output (e.g. an old `CREATE TABLE exciting_en32z9` later changed to `exciting_880909`) was re-deployed as a phantom write edge and lingered as an orphan asset on the graph — shown with no producer, and shifting position on click as the graph re-derived. saveDraft now re-runs inferAssets on the current body and passes the result as `assets`, overriding the snapshot — mirroring the per-pane save. The backend clears+reinserts from the sent set, so a re-deploy drops the stale rows. Verified: deploying with the fresh asset set removes the orphan from the graph. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: collect upstream reads from CTAS and CREATE VIEW in SQL asset parser `CREATE TABLE x AS SELECT … FROM y` (and `CREATE VIEW`) recorded only the write to x — the source read of y was silently dropped. Table-level reads are gathered in the `Statement::Query` arm via handle_table_with_joins; the generic table-factor visitor only picks up read-functions and string literals, not plain `FROM <table>` references. The AS-query of a CTAS isn't a `Statement::Query`, so its FROM tables were never walked. On the pipeline canvas this meant a `datatable://…` upstream consumed by a CTAS step showed no read node/edge — the step looked like it produced its output from nothing. Factor the Query arm's read collection into handle_query_reads and call it from the CreateTable (when it has an AS-query) and CreateView arms, balancing the cte_name_stack push in post_visit_statement. Updated the drop_then_create test (which had pinned the old drop-the-read behavior) and added CTAS + CREATE VIEW read coverage. Verified against the rebuilt asset wasm: the live editor now infers the read. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * update * updates * refactor: dedup asset-graph code, squash migrations, drop artifacts Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf: gate asset dispatch on a cached per-workspace producer set Cache the producer-path→writes map per workspace and invalidate it from the asset-clear paths via the notify_event polling system, so a top-level script/preview completion that isn't an asset producer costs an in-memory lookup instead of a per-completion query. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: remove dead unquote fn that failed backend check under -D warnings Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: green the frontend check (pin published wasm-asset, fix type errors) Pin windmill-parser-wasm-asset to the published 1.728.1 (was a file: link to a gitignored, CI-unbuilt pkg-asset). Exclude test files from svelte-check (the parity test reads a backend fixture via node:fs, which the browser app tsconfig has no @types/node for; vitest still runs them). Fix pre-existing branch type errors: drop the unsupported 2nd getScriptByPath arg, cast script.schema to Schema for inferArgs, coerce has_preprocessor to a definite boolean, and wrap the cancelJob handler so it isn't possibly-undefined. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: move pipeline partition resolution to ee-private (free-CE) Partition resolution becomes a private module (partition_ee in windmill-ee-private, hidden from the public repo) with an OSS no-op fallback (partition_oss); call sites resolve via the aliased windmill_common::partition. Not enterprise-gated — free to run in CE. Bumps ee-repo-ref to the ee branch carrying partition_ee. Verified building in default, private, and private,enterprise (offline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: move asset-cascade join/debounce/retry to ee-private (free-CE) Join barrier, debounce, and retry become the private windmill_queue::cascade module (cascade_ee in windmill-ee-private); OSS gets cascade_oss no-op fallbacks (plain OR fan-out). Core cascade stays public. Bumps ee-repo-ref. Verified default/private/private,enterprise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: skeleton enterprise pipeline freshness + backfill (TODO, ee-private) Gated windmill_common::pipeline_advanced (private; pipeline_advanced_ee) with OSS fallback; entry points return a clear not-implemented error. Deploy surfaces a TODO when a script declares // freshness. Bumps ee-repo-ref. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: repair asset_trigger_dispatch test after cascade carve-out + cache its queries Stage-2 moved reap_stale_join_slots to windmill_queue::cascade; update the integration test's import. Also commit the test's sqlx query cache (was never prepared with --tests, so SQLX_OFFLINE cargo test failed pre-existing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: invalidate producer-cache in asset dispatch tests (mirror deploy) The tests seed asset rows directly and run no notify poller, so the per-workspace producer cache went stale across tests → 0 dispatched. Clear it at the seed point, as a deploy would via notify_event. All 8 asset_trigger_dispatch tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to ba677ea142011462ad4dfe77e8375a6dd274cdef This commit updates the EE repository reference after PR #619 was merged in windmill-ee-private. Previous ee-repo-ref: 925c350cff55d3ea738d9e2e4098d9ce4bdda418 New ee-repo-ref: ba677ea142011462ad4dfe77e8375a6dd274cdef Automated by sync-ee-ref workflow. * test: disable producer cache in asset dispatch tests (isolated-DB safe) The .remove(WS) approach still raced: #[sqlx::test] gives each test its own DB but they share one workspace id, so the WS-keyed process-global cache clobbered across DBs under concurrent threads. Add an ASSET_PRODUCER_CACHE_DISABLED test hook and set it in the tests so every dispatch reads its own DB. 8/8 pass at --test-threads=10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: replace asset-cascade depth cap with cycle detection The hardcoded MAX_CHAIN_DEPTH=5 truncated legitimate deep pipelines (silently — the check returned before event logging). Replace it with per-edge cycle detection: carry the producer lineage in trigger.chain and skip only a subscriber already in the chain, recording a visible cycle_detected dispatch_event. Acyclic pipelines of any depth now cascade fully; a high MAX_CHAIN_LEN backstop guards against runaway. Tests + UI label updated; 8/8 pass at --test-threads=10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update dispatch_event reason examples (depth_cap → cycle_detected) Comment-only; the migration is idempotent and already in the potentially_stale self-heal list, so the checksum change re-applies cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: park cascade retry (P1 dead-end) + clear stale script_triggers on rename Two deploy-path fixes: - Retry is parked: a retried subscriber is wrapped in a SingleStepFlow, whose run is a flow step and ineligible for asset dispatch, so it would silently dead-end the cascade (P1). Stop persisting retry to script_trigger and warn at deploy; TODO(pipeline-retry) to re-enable once dispatch handles flow-wrapped producers. (Dispatch plumbing kept + still tested via direct seeding.) - Rename leaves stale script_trigger rows: clear was keyed on ns.path only, so old-path '// on' edges lingered and could trigger a script later recreated at that path. Also clear the old path on rename (assets already handled via the parent-hash clear). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> Co-authored-by: hugocasa <hugo@casademont.ch> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Arnaud <31803803+Araden14@users.noreply.github.com> Co-authored-by: Diego Imbert <diego@windmill.dev> Co-authored-by: centdix <40307056+centdix@users.noreply.github.com> Co-authored-by: Diego Imbert <70353967+diegoimbert@users.noreply.github.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Aldrin Jenson <aldrinjenson@gmail.com> Co-authored-by: centdix <centdix@users.noreply.github.com> |
||
|
Ruben Fiszel
|
38c0ccdf56 |
fix: refetch license key from settings when in-memory key is invalid (#9534)
* fix: refetch license key from settings when in-memory key is invalid Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: only record accepted license keys so rejected keys stay retryable Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
hugocasa
|
e8e0701a36 |
feat(api): add endpoint to update token label (#9474)
* feat(api): add endpoint to update token label Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(api): prevent renaming the session token label Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(api): restrict token-label edits to user tokens, not just session Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): edit token label in the edit modal instead of inline Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(api): reject relabeling tokens to reserved system-token names Centralize the is_user_token classifier in windmill-common and reuse it to reject labels colliding with system-token namespaces (ephemeral*, debugger-token, mcp-oauth-*), not just session. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(api): match ephemeral label case-insensitively and cap label length Align the canonical is_user_token, the SQL guard and the frontend mirror on a case-insensitive `ephemeral` match (so a token can't be relabeled to a casing the backend allows but the UI hides), reject labels over the VARCHAR(1000) column limit with a 400, and add unit tests for is_user_token. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
1727271e19 |
feat: sandboxed daemonless container runtime via '# sandbox <image>' (#9453)
* feat: add sandboxed docker v2 runtime via '# docker <image>' Run a container image as a subprogram of the job's own nsjail sandbox: extract the image rootfs with podman (rootless) and run it chrooted inside the job's nsjail, so the container inherits the job's confinement and is safe under nsjail / for untrusted code. Selected by '# docker <image>'; a bare '# docker' keeps the v1 (dind) path untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: default to daemonless docker (drop dind from compose, allow docker on cloud) docker-compose no longer ships the dind sidecar (v2 is daemonless: podman + nsjail in the worker); removed the dind service, DOCKER_HOST env, depends_on and volume. Removed the language-picker guard that blocked Docker scripts on the multi-tenant platform, now that v2 makes docker safe to run sandboxed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: select sandboxed container via # sandbox <image>; add pull policy + size guards - Surface moved from '# docker <image>' to '# sandbox <image>' (groups under the sandbox annotation; '# docker' stays v1-only, '# sandbox' stays nsjail-bash). - SANDBOX_IMAGE_PULL_POLICY (default 'newer') so moving tags don't go stale. - SANDBOX_IMAGE_MAX_SIZE_MB rejects oversized images before extraction. - SANDBOX_IMAGE_CACHE_MAX_MB best-effort LRU eviction of podman's image store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(sandbox): support # volume, honor nsjail tmp instance settings, v2 docker template - Thread shared_mount into the sandbox container nsjail config so '# volume' mounts (and the same-worker /tmp/shared folder) apply inside the container. - Use resolve_nsjail_tmp_mount_block for the container's /tmp so it honors the same nsjail_tmp_backing / nsjail_tmpfs_size_mb instance settings as other nsjail jobs. - docker-compose comment + the editor's Docker template now use '# sandbox <image>'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(sandbox): make image size/cache/pull-policy UI instance settings Convert SANDBOX_IMAGE_* from worker env vars to DB-backed instance settings (sandbox_image_max_size_mb, sandbox_image_cache_max_mb, sandbox_image_pull_policy), hot-reloaded via the same mechanism as nsjail_tmpfs_size_mb and configurable in #superadmin-settings. No worker restart needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(sandbox): windmill-managed registry — default registry + private auth Two new instance settings: - sandbox_image_default_registry: prepended to unqualified image refs (alpine -> <registry>/alpine); fully-qualified refs untouched. - sandbox_registry_auth: docker/podman auth.json blob written to a per-job authfile (0600, removed with the job) and passed to podman --authfile for private registries. Both hot-reloaded and configurable in #superadmin-settings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sandbox): protobuf-safe proto_str escaper, atomic 0600 authfile, registry tests Addresses local-review P2s: proto_str now emits valid protobuf octal escapes for control/non-ASCII bytes (not Rust \u{..} that nsjail would reject); the registry authfile is created 0600 atomically (no world-readable window); add a registry_qualified table test + a non-ASCII proto_str case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sandbox): P0 — deliver image env via nsjail envar:, never the launcher process env CI review (P0): the image's OCI Env (attacker-controlled keys+values) was applied to the nsjail launcher process via .envs(), so a hostile image could set LD_PRELOAD/ LD_LIBRARY_PATH/LD_AUDIT on nsjail itself and execute code as the worker outside the jail. Now the image env is rendered as proto-escaped 'envar:' directives (child-only) and nsjail's process env carries only windmill-trusted keys (reserved vars + proxy). Also: warn instead of silently bypassing the size guard on inspect failure; reset the eviction guard via a Drop guard (no stuck flag on panic/early-return). +render_envars test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sandbox): P0 symlink-write escape via rootfs script; P1 redact registry-auth logging CI review: - P0 (Codex): the body was written into the image-controlled rootfs as .windmill_docker_main.sh via write_file (follows symlinks) — a hostile image could plant that path as a symlink to a host file and capture the worker's write before nsjail starts. Now the body is passed straight to 'sh -c <body> sh <args>'; no file is written into the rootfs at all. - P1 (Codex): sandbox_registry_auth flowed through the generic setting loader which logs the value (raw auth.json credentials). Replaced with a secret-aware reload that loads directly and logs only a redacted 'configured=' message. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sandbox): redact sandbox_registry_auth in instance-settings write log too The settings API also logs 'Set global setting <key> to <value>' via format_setting_value; add sandbox_registry_auth to SENSITIVE_SETTINGS so the credential is redacted there as well as on reload. * fix(sandbox): don't silently disable cache eviction on podman images parse error Re-review (cubic/Claude P2): serde_json::from_slice(...).unwrap_or_default() meant any parse hiccup (e.g. podman omitting Size/Created via omitempty for a zero value, or schema drift) silently degraded to an empty Vec and disabled eviction with no log. Now Size/Created are #[serde(default)] (a missing omitempty key -> 0, not a whole-array parse failure) and a real parse error warns + breaks instead of being swallowed. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
f595787409 |
fix: invalidate relative-import cache when imported script changes (#9443)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
edf340c4d4 |
fix(security): re-pin cached hub scripts to CVE-patched versions (+ HUB_BASE_URL override for cache mode) (#9387)
* feat(cache): allow overriding hub base url via env in `cache` mode The `windmill cache hubPaths.json` prebuild step (run in the Dockerfile) never connects to the DB, so HUB_BASE_URL stays at its compiled default (https://hub.windmill.dev) — unlike server/worker modes which load it from the DB global setting. This made it impossible to point the prebuild cache step at a private or staging hub. Read HUB_BASE_URL from the environment at the start of cache_hub_scripts and store it into the existing HUB_BASE_URL ArcSwap (the same static the hub fetch functions read). No effect unless the env var is set and non-empty; server and worker modes are unchanged (they still use the DB setting). This also enables validating hub-script dependency changes end-to-end against a local fake-hub before pushing to the real hub. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): re-pin cached hub scripts to patched versions windmill-integrations#133 was merged and pushed to the hub, minting new versions with regenerated (CVE-free) lockfiles. Bump the hubPaths.json pins so the prebuild cache step (`windmill cache`) fetches the patched lockfiles instead of the old vulnerable ones (the hub serves each version_id immutably, so the old pins keep returning the vulnerable deps until bumped). - slackErrorHandler 19741 -> 28241 - slackRecoveryHandler 9080 -> 28239 - slackSuccessHandler 28220 -> 28240 - smtpReport 9086 -> 28242 - appReport 28076 -> 28243 (puppeteer screenshot script) - gitInitRepo 28219 -> 28229 (already-fixed hub version; pin was stale) Validated end-to-end against the real hub: `windmill cache` with these pins produces a clean cache_nomount/bun (axios 1.16.1, form-data 4.0.5, follow-redirects 1.16.0, nodemailer 8.0.10, ws 8.21.0, svelte 5.55.8, devalue 5.8.1; basic-ftp and ip-address no longer pulled). No vulnerable versions remain. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
59ab038d77 |
fix(monitor): cleanup stale server_heartbeat background_task_state rows (#9338)
`announce_server_started` writes a `server_heartbeat:{INSTANCE_NAME}` row
on each startup. INSTANCE_NAME is a fresh random string per process, so
the row is never updated again and a new row is inserted on every
restart, growing background_task_state unboundedly.
Add an hourly monitor task that deletes server_heartbeat:* rows older
than 7 days. Older rows cannot influence check_any_server_started (which
only considers heartbeats refreshed after the restart was initiated), so
they are safe to prune.
Fixes WIN-1990.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|